We have been familiar with the progress percentage, or percentage completion, which is usually tracked for a traditional project management system. Traditional approaches usually focus on task completion within a set schedule and cost.
However, when we use agile methodology, teams, customers and stakeholders see almost immediate results. We basically use prioritization techniques to build goals and objectives and adjust timeframes and effort to deliver a product that corresponds to schedule requirements. Thus, during agile transformation I personally lean towards KPI tracking, using which you can identify several traits. KPIs help to understand human behaviour, quality, efficiency, establish flow, set accurate work in progress limits and most importantly build transparency into the system of an organization.
First, a simple definition of the word "KPI" - This is not as per any book or literature, this is put in as simple terms as possible, which you can understand it.
KPI - A measurable value which demonstrates how effectively we are progressing towards business objectives
Now, as you know the simplest definition already, let's go a little deeper in the conversation on how to set these measurements up.
First Step - Ask your teams constructive and though provoking questions.
What are we measuring?
How are we able to measure it?
Are we on track with our releases?
How do we determine if the teams blocked?
How is the quality of the application?
Are we leaving a lot of technical debt behind?
What is the cost of our technical debt, how much effort will be needed to burn it to manageable limits?
How healthy is our backlog, do we have enough items ready for our teams?
Are we writing our stories too early?
These questions will help start a conversation in your teams about what is the need of the KPIs. You can also have a conversation around how you will gather data to measure these KPIs.
The tools you choose for work management will also be a limiting factor on what KPIs you can or cannot gather. Thus, it is important to keep those in mind as well. Recommended tools can be Jira, Rally, Version One etc.
KPIs and metrics help us measure certain things about our people and work that they are doing and the way how they are doing it, which help provide us with an analytical insight into what is happening around us.
I want to give you a simple guiding principle of how you can make sure that the agile metrics / KPIs that you are provisioning,
Thus, it is imperative that before a measure is used, it is
Discussed with the relevant people who are impacted by this.
Piloted before it is rolled out.
Abandoned if its dark side creates too much adverse performance.
When an organization transitions to Agile way of working, the measurement and reporting of what we do has to also change. You can no longer just use the "progress percentage" or "red, amber, green" metrics as a measurement of success. There are several things happening around that status, which we need to measure and report on.
As Spitzer says, "People will do what management inspects, not necessarily what management expects"
As consultants, coaches or scrum masters we spend so many hours discussing these metrics, making clever Excel files which work on the extracts taken from multiple systems and give us output using graphs of various kinds. We have to understand that the metrics we gather in any Organization revolves around three major things.
People - Deploying people effectively is the first step to build successful Agile teams. We have to ensure that people who are working on your projects are engaged and productive.
Projects - In certain organizations we only manage one project at a time, but this is not the necessary case in other organizations. As organizations grow, you will need to manage multiple complex projects simultaneously and for each of these projects you don't necessarily need to introduce additional KPIs all the time. You have to be able to make sure that your organization Is able to maintain visibility across all projects at any stage of the overall lifecycle.
Productivity - How fast we are is determined by how productive we are. Monitoring and measuring productivity is ongoing and continuous. As consultants or as an agile coach it has to be our responsibility that the tools and KPIs that we provide to the organizations help in increasing productivity over the long term.
Thus, our KPIs for Agile delivery should also allow us to measure our progress on these 3 things. More importantly on multiple levels in the organization. In any organization we can find clear 3 levels of hierarchy, line managers, senior management and project management office and executives.
Line managers are those people who operate daily with their teams to deliver projects successfully. Senior management is responsible for the roll out of the best practices and processes and tools in scaled environment. Executives basically care about the organization wide maturity, efficiency and agility. Thus, we have to make sure that the KPIs we provide and measure are matching on three levels.
Pro Tip: From the industry a typical KPI landscape will look something like this. Executive level. Organization level aggregations. Customer satisfaction Operational efficiency Workforce planning. Senior Management and PMO Benchmarking Teams Release Cycle Planning Program Velocity Program Burn-Up Capacity Management Line Managers Release Forecast Test Coverage Quality Debt Defect Density Red Build Graph Tech Debt
Some important questions which you can ask from your teams:
Which KPIs will tell us :
Are we on track? - Deadline vs realistic delivery date.
How reliable team commitments are? - Committed vs completed, estimation accuracy
Do we apply an efficient testing strategy? - open bugs over time, technical debt, defect containment.
Where are the bottlenecks? - flow, cycle time, touch time, improperly sequenced dependencies, blocked work in a sprint / release.
How to assess productivity of different streams including even vendors? - dedicated dashboards for teams and at each level.
I want to recommend some starting steps for you and your teams.
If you are using a sprint by sprint cadence
Ensure delivery predictability via sprint-by-sprint cadence
• Delivery Forecast via Burn up
• Velocity: Committed vs. Completed
• Sprint Plan Change
• Average velocity by sprints
• Backlog Health & Ready for Development
• Commitment rate
If you are using Kanban
Monitor the input & output flows of your service to ensure that SLAs are met & bottlenecks are avoided
• Average Lead and Cycle Time
• Reaction/Resolution Target Fulfilment • Throughput & Created vs Resolved
• Cumulative Flow Diagram
• First Time Right
• Time in Status
If you are working on a fix price assignment.
Keep a forecasted delivery date on track, control the scope creep, and keep quality on a high level
• Delivery Forecast via Burn up • Cumulative Flow Diagram
• Code Quality Summary
• Open Bugs Over Time
• Defect Containment • Time in Status
To keep track of your engineering practices and also to ensure that teams are following best practices.
Make sure your dev teams are fast and reliable, and having quality gates over code and CI/CD in green
• Unit Test Coverage
• Technical Debt
• Code Violations
• Code Base Change Trend • The Biggest Commits
• Most Frequently Modified Files
To make sure that your DevOps practices are taken care of and there is collaboration across the board.
Track and Boost the performance of your teams via faster lead time & increased deployment frequency
• CI/CD Pipeline Summary • Average build time
• Average build success
• Deployment Frequency
• Pipeline Mean Time to Recovery
• Average Pipeline Lead/Cycle Time