There is no doubt that product teams need to measure their work in order to adjust development and business direction based on their results and to the reality that they expose.

Deciding on what to measure depends on what the team’s goal is. The case has been made that the ultimate goal is to reduce value investment debt at risk, basically implying that the sooner the team expunges this debt, the sooner they can adjust the product’s direction.

The most effective measurements are those that prove that the product has value to its consumers, or prove that the desired change in their behaviour has been achieved by the product team. Yet value is elusive and hard to define, and therefore measure. Here, there is an effort to measure the efficacy of the delivery pipeline so that at least we assure a quick way to production, and then focus on business value metrics.

From a technical perspective, subjects to be measured can span the following areas:

  • Configuration Management
  • Environments and Deployment
  • Continuous Integration
  • Data Management
  • Quality Assurance
  • Technical Architecture
  • Organizational Alignment
  • Transparency

Following is an attempt to define a set of metrics that are non-contextual, and that can be applied during the product’s development (code in motion), and semantic measurements for production-time metrics (code at rest).

Code in motion

Behavior-Driven Development measurements

  • % FPCA (first pass complete and accurate), seeking an upward trend.
  • Integration time for features, seeking a downward trend.

Unit & Integration Tests measurements

  • The number of unit tests, seeking an upward trend.
  • The number of lines covered by tests, seeking an upward trend.
  • The cyclomatic complexity, seeking a downward trend.
  • Ratio of functional lines of code to test lines of code (Ideally the number of test lines of code would be trending upwards).
  • The average number of test steps/length of tests, seeking a downward trend.
  • The number of automated scripts, seeking an upward trend.
  • The ratio of automated tests to manual tests, seeking an upward trend.
  • The percentage of stories covered by automated tests, seeking an upward trend.
  • The average testing time for stories, seeking a downward trend.
  • The total time spent in QA before a release, seeking a downward trend.
  • The total defect cycle time, seeking a downward trend.

Continuous Integration measurements

  • The build rate, seeking an upward trend.
  • The total components being built, seeking an upward trend.
  • The total number of tests being run, seeking an upward trend.
  • The total number of check-ins per day, seeking an upward trend.
  • The amount of integration testing time, seeking a downward trend.

Technical-debt reduction measurements

  • Technical metrics indicating code quality, seeking an upward trend.
  • The lead time to production, seeking a downward trend.

Product management measurements

  • Stand-up efficacy, and observe the increase in communication, collaboration and a reduction in misunderstandings.
  • The number of negative items brought up in Retrospective sessions, and observe their decrease.
  • The ratio of business priorities to technical issues in the backlog, and observe their rate of increase.
  • The efficacy of user-stories, and observe the increase in communication, collaboration and a reduction in misunderstandings and integration efforts. In addition, observe consistent invariability, and a reduction of inter-story dependencies.
  • The speed-of-delivery to production, and observe its rate of increase.
  • The average number of in-progress stories, seeking a downward trend.
  • The number of story carryover, seeking a downward trend.
  • The consistency of the number of right-sized stories put into production, looking for predictability.
  • The number of mid-sprint direction changes, seeking a downward trend.
  • The number of mid-sprint scope changes, seeking a downward trend.
  • The total story cycle time, seeking a downward trend.
  • The lead time to production, seeking a downward trend.
  • The number of failures after changes, seeking a downward trend.
  • The complexity of stories (batch-size), seeking a downward trend.
  • The ratio of roll-forward to rollback during outages, seeking an upward trend.

Teamwork measurements

  • The efficacy of pairing, and observe the increase in usage of a ubiquitous language, collaboration, and cross-training.
  • The time it takes to perform code reviews, seeking a downward trend.
  • The time for story implementation, seeking a downward trend.
  • The efficacy of story post-development review, and observe in-sprint defect rate decrease.
  • The number of blameless postmortems, seeking an upward trend.

Code at rest measurements

  • The number of outages due to infrastructure inconsistencies, seeking a downward trend.
  • The number of outages due to run-time dependencies, seeking a downward trend.
  • The number of passing semantic tests, seeking an upward trend.
  • The number of leaked defects, seeking a downward trend.
  • The MTTR values, seeking a downward trend.
  • The MTBF values, seeking an upward trend.

In conclusion

The measurements themselves will simply give the team a number. That number should be recorded to establish a baseline, being the basis for calculating a trend, showing the rate of increase or decrease. Once plotted, the team can decide on which area to focus in order to minimize debt at risk.

These measurements will vary based on team structure, its maturity, and the Value Stream. The product team should continuously monitor it to assure effective delivery as a basis for continuous learning and improvement using the KPIs listed above.