Improving DevOps Team Performance using DORA Metrics and CTO.ai

What’s the business value of DORA metrics, and how are organizations and teams using them? We recently organized a webinar on how you can improve performance in your organization using DORA Metrics and GitHub.  Let’s review the key takeaways in this post:

Why and How to Measure Software Development Performance?

Measuring software development performance has been a challenge for software developers; people have historically used measurements such as lines of code, man hours, test coverage, and story points. How do you measure developer output?

Metrics for the individual developers can actually be abused a lot and they’re counterproductive and they don’t really help with delivering value and impact to your customers. DORA metrics is a good metric system that helps to measure engineering effectiveness in your department. These metrics help you do what’s important, which is delivering value and impact to your customers and shortening the feedback loop. DORA metrics are the first engineering metrics that have a proven relationship with the ability to deliver good business outcomes.  The goal of using DORA metrics is to maximize your ability to iterate quickly while making sure that you’re not sacrificing quality.

Benefits of implementing DORA Metrics

DORA metrics make the process of software delivery more transparent and understandable, breaking it into pieces.

DORA metrics help teams to:

• spot the areas for improvement when it comes to the delivery process,
• understand what actions to take to streamline the process,
• deliver quicker, ensuring higher quality,
• ensure stability and accountability of a product
• make decisions based on data, not assumptions

How do DORA metrics enable improved Software Delivery Performance?

DORA metrics are a research and analysis driven set of measurements used to provide a clear approach to understanding the effectiveness of a software delivery process.  What’s great about DORA metrics is that they don’t attempt to measure productivity directly; instead, it’s focused on measuring the velocity and stability of the software delivery process in a team. At the heart of DORA are four key metrics divided into two groups: Throughput (or velocity) and Stability.

Throughput is measured by Deployment Frequency and Lead Time for Changes. Stability is measured by Change Failure rate and Mean Time to Recovery.

Throughput and Stability

Four metrics have a strong correlation with software delivery performance:

1. Deployment Frequency (DF)
2. Lead Time for Change (LTC)
3. Change Failure Rate (CFR)
4. Mean Time to Recovery (MTTR)

Let’s look at what these four metrics mean for you and your team and get a better understanding on how to apply them to your team.

These metrics all attempt to provide a standard definition for calculation, but no two teams are alike and as such, no two services will have the exact same definition of what is considered a failed deployment.

There are many common ways to define each metric, but the most important thing to remember is to have your team work together to set their definitions up ahead of time. This is the key to unlocking the most value from the metrics. This also prevents confusion and allows the team to understand the impact of their work on these metric calculations.

For example, one team may define a deployment failure as anytime a deployment fails in the CI/CD pipeline, while another team may only consider a failure to deploy in production environments worth noting. While the team at DORA have done a great job of clearly defining what they consider each metric to be, it may not work exactly the same for your team.

Key Takeaways

1. Define each metric together with the whole team ahead of time.
2. Use data you have available today, iterate over time.
3. Focus on the customer. The impact of code on customers should be the main focus.

DORA Analysis

The DORA framework categorizes teams into groups based on their cumulative responses, ranging from low performers to high performers, and sometimes Elite performers. You can get a quick assessment of where you stand on DORA metrics using the charts shown from the latest report below.

High vs Low Performers

After analyzing all the data, the DORA team felt data was best captured this year using three clusters instead of the four of previous years. The Elite category was removed for this reason. This could be due to many factors such as the global pandemic or the responses this year simply did not feel they were far enough along in their journey.

Whatever the reasons, the overall signals were still positive as the delivery performance overall compared to last year was still higher. The ceiling may have been lowered, but the floor has been raised.

DEMO

In this tutorial, you’ll learn how to send events from Circle CI and gather insights from it. Integrating CTO.ai Orbs with your Circle CI workflows, you will be able to send workflow events to CTO.ai to enable workflow metrics.

Prerequisites

• CTO.ai Account
• Circle CI Account configured with Orbs

With CTO.ai DORA Metrics, engineering teams can now track status, monitor deployments, and optimize their CI/CD Pipelines.

1. Sign up or log in to your CTO.ai Account

2. Install the CTO.ai CLI using npm install -g @cto.ai/ops

• Confirm if the CLI is installed by typing ops in your terminal.

3. Next, generate an API token using the ops generate:token command on your CLI.

• Enter your Client ID, Client Secret, and Password.

4. Generate your team ID using ops team:info

5. Next, add your TEAM ID and API TOKEN in your environment variables.

• Confirm if your Orb Security Settings is set to Yes in your Organization Settings.

6. Next, create your circle ci config file in your GitHub repository and attach the CTO.ai Circle CI Orbs configuration to your config.

version: 2.1
description: "Example showing how to send a workflow metric events to CTO.ai. Set CTOAI_EVENTS_API_TOKEN and CTOAI_EVENTS_API_TEAM_ID env vars in CircleCI settings."

orbs:
  cto-ai: cto-ai/cto-ai@1.0.2

jobs:
   events_example_job:
      docker:
         - image: cimg/base:stable
      steps:
        - cto-ai/event:
            token: ${CTOAI_EVENTS_API_TOKEN}
            team_id: ${CTOAI_EVENTS_API_TEAM_ID}
            image: app78df3bl2
            environment: staging
            event_name: deployment
            event_action: succeeded
            origin: circleci-orb

workflows:
   events_example_workflow:
      jobs:
         - events_example_job

7. Next, run and deploy your config on circle CI workflow. In your Circle CI dashboard, you’ll see the workflow status and pipeline jobs.

8. Back in your CTO.ai insights dashboard, you can see insights deployment status and events timeline. CTO.ai automatically collects and processes the events coming from your Circle CI workflows.

The Deployment Status and Event timeline let you track your deployment activity and fix builds faster for every Circle CI workflow on CTO.ai and measure all your successful and failed deployments, see your events success rates, how it changes over time, and identify your lengthiest Circle CI workflows.

• As discussed above, in your overview dashboard, you can measure the total ThroughPut and Stability of your workflows coming directly from Circle CI. The DORA metrics are calculated to measure insights whenever there’s a failure rate, successful deployments, recovery process, and generated failure codes from your Circle CI builds.

That’s it?

That’s it! There’s a lot you can do with CTO.ai, and tracking your deployments and workflows with CTO.ai is simply a ops publish . away.

Here’s the repo with a source code if you’d like to take it and run with it to your Circle CI workflows, you can go ahead and deploy it yourself.