Understanding DevSecOps KPIs – Part 1: Traditional DevOps Metrics
In this three-part series, we’re looking at the Key Performance Indicators that should be used when measuring the success of a DevSecOps project.
In this first blog post, we’ll introduce security to DevOps and build a foundation with Traditional DevOps Metrics.
DevOps Meets Security
DevOps emerged as a cultural phenomenon meant to bridge the gap between Development and Operations teams and effectively move software across the software supply chain from Ideation to Production. Over time, DevOps has evolved to become DevSecOps or sometimes SecDevOps. Regardless of what you call it, the spirit behind introducing the “Sec” into DevOps is to ensure that security is a critical attribute of DevOps workflows and automation, and should be considered at all times during solution development. The need to explicitly call out the importance of security in DevOps has become evident through the many security incidents and data breaches we have been hearing about for quite some time now.
At this point, it’s fair to assume that every organization has some flavor of DevOps / DevSecOps initiatives that are ongoing. What we see over our time helping organizations embrace and adopt DevSecOps is that most of these organizations lack a DevSecOps measurement strategy. The key performance indicators (KPIs) and Metrics for measuring their DevSecOps adoption and success are neither defined nor shared with teams.
You cannot evaluate progress if you don’t define what needs to be measured.
Building Your DevOps Metrics
Through these blog posts, our goal is to highlight key metrics to measure as they relate to DevSecOps. Besides the traditional DevOps metrics, we want to highlight ways to measure how well Security and Culture are integrated into your DevSecOps initiative. Yes, we believe there needs to be a way to measure culture as well because it is an essential function of DevSecOps.
Delving further into DevSecOps measurement, we hope to give you a set of metrics that allow you to further your team’s development. In the coming posts, we’ll first cover the traditional DevOps metrics, then focus on the critical security metrics that we would propose to measure, and finally move to measuring culture.
Why Metrics Are Important
The first step in building a list of metrics is to ask yourself why you want them. Building a comprehensive list of metrics is vital to taking a snapshot of your product. It’s checking the vitals across a number of areas to understand what is running well, and what needs help.
Just as important building a way to measure your metrics, is selecting which metrics you measure. Too much information inhibits your ability to focus on what matters.
Traditional DevOps Metrics
Before we move into the important security metrics we want to explore which metrics can prove essential to your foundational DevOps team. As DevOps culture continues to grow, countless metrics have been added, and are continuously added. We have chosen 11 metrics that we think are relevant and important to the majority of teams we have worked with.
| Cycle Time | The time it takes for a change (new feature or bug or enhancement) to move from ideation to production. The time it takes for development.The time it takes for testing to change.The time it takes for change to be deployed to production after QA signs off on the change. |
| Deployment Error Rate | (Number of deployments that had errors)——————————————————— * 100(Total number of Deployments) |
| Production Deployment Cost | (Total number of people on production deployments) * (hours until deployment is successful) * (average cost per hour) |
| Application outage rate by patching cycles | (Number of outages due to infrastructure patching)——————————————————————— * 100(Total number of patch cycles) |
| Sprint to Deployment Rate | (Number of sprints complete from the last deployment)———————————————————————- * 100(Number of deployments) |
| Unit Test Coverage | Automated test coverage of unit testing of your code base. Extremely important if testing pyramid is followed |
| Automated Regression Test Coverage | What percent of your regression testing is automated? |
| MTTR (Mean Time To Recovery) | The time it takes from identification of an incident to the time it takes to resolve the incident. |
| Automated Application Environment Provisioning | The amount of time taken to provision an entire application environment. IncludesCreation of virtual machines and configuration of virtual machines.Installation and configuration of middleware on the virtual machines. Deploying the required application. |
| Roll Back Rate | (Number of deployments that had to roll back)———————————————————————- * 100(Total number of deployments) |
| Time to Roll back | Amount of time it takes to roll back. |
Combined, these metrics provide a fantastic overview of your DevOps and give you a glimpse into areas that are working well, and areas that may need attention.
Pick the Metrics That Work Best For Your Organization
Every organization is different. When looking at building a report card for your product, it’s important to cater to the metrics that are important for your snapshot. Consider which ones are vital to a healthy product, and prioritize those.
Now that we have a basic understanding of DevOps KPIs, let’s look ahead. In part 2, we’ll be looking at the important security metrics of DevSecOps.
Questions about your DevSecOps? Looking for guidance as you prepare your next project? Reach out to us at [email protected]
0 Comments