At a Single Glance – Interpreting and Acting Upon Agile Project Metrics

At a single glance.

That’s what it should take to know when something isn’t right on your projects.

Knowing that something is wrong with your project should be as easy as glancing at a visual indicator.  You shouldn’t have to interpret anything.  You shouldn’t have to be an expert in metrics or project analysis.  You should easily be able to tell that something isn’t right … with a single glance.

Monitor and control systems have been around since the dawn of innovation.  We see these types of systems all over industry – simple and intuitive displays representing the health of a system.  We see this in manufacturing, transportation, energy, and finance.  Of course, different environments use different ways of indicating problems – but all monitor and control systems have some commonalities – they not only inform you when something is “bad” but also when something “could be bad” or is “about to be bad” to ensure that you have an opportunity to do something about it.

This isn’t new in IT either.  15 years ago I remember filling out IBM status reports and at the top, sure enough there were the KPI indicators – Green, Amber, and Red.  Status reports mean well as ultimately, they were put in place to help detect problems sooner rather than later – with the hopes of allowing at a management level to help prevent a possible problem rather than dealing with the mess of the problem occurring.  This is why PM’s (especially those PMP or Prince2 based) are very focused on risk and issues – as these are communication tools that help us understand when risk is turning from potential to a reality.

On modern projects, especially those that use tooling, we should expect a much more real-time and transparent view of our projects.  We gather all sorts of metrics in on an agile project – these metrics providing us that necessary feedback loop required for continual improvement and alignment.

We capture things like velocity – which is the measure of the amount of value a team can produce over a period of time.  We represent progress in charts such as burndowns – that show the rate at which value is being produced against a plan (at different bucket scales such as iterations/sprints, releases, or the product itself).  Simply go to Google and search for “agile metrics” and you’ll see a LOT of information about what you can be measuring, what those measurements tell you, and what to do about it.

(go ahead.. click on this link – it will show you the results)

I’m a huge fan of metrics – I’m a huge number geek.  I feel very comfortable dealing with numbers, interpreting their meaning, and using them as a basis for action.  I commonly point to fancy charts to say “look.. this shows us we’re doing great”.  I love to get into the raw numbers, use pivot tables and graphs to really drill down into a project.

I’m not normal though.

What I realized a long time ago was that we actually needed “single glance” metrics – just like control stations in other industries.    Not everyone has the time and inkling to deal with raw data all day – or to continually interpret numbers and charts to understand if the system is meeting its goals.

On projects that I’ve been a part of we’ve always wanted ways to give us early “smell test” indicators.  We wanted something that would light up when something “smelled bad” on our project.  Perhaps a developer is going dark (meaning, they aren’t surfacing status and are hiding away getting work done), or the expectations we have set with our product owners/business may not be met (we don’t want to know after the fact that we’re not going to ship what we said we would ship).  We needed something that would give us an indicator when things were potentially bad.

Take a look at the following chart…

Now.. tell me.. is what this chart showing you good?  Is it bad?  If you’re mom looked at this chart – could she tell you if things are going well?

When you take a look at metrics on your project, you should consider ways in which you can know that a conversation needs to happen based on a very simple view of your projects – this is especially true when you are managing many projects.  Think of your environment like NASA’s Mission Control Center – you want to see valuable, meaningful information immediately that requires very little interpretation.  You don’t want the system to solve the problem – but you do want your control center to help interpret the conditions that help you make informed decisions as events happen – not after they happen when you don’t have any ability to do anything about it.

Typically, when you are alerted to the problem, you want to know right away what the consequences are.  Systems that alert you to a problem, should also help you understand the potential problems.  For example, if NASA’s onboard oxygen indicator says “low” – you might want to know immediately how low, but also, how much oxygen is left at the current rate of consumption.  Similarly, a good example on a project is an indicator that “the expectations you have set with your customer may not be met” (for example, the burndown rate of your iteration or sprint indicates that at your current team’s velocity you won’t get all of the sprint backlog items complete by the end of the sprint) you might want to know exactly what stories may not make it in order to have the appropriate well informed conversation with your team.

The need for these visual indicators also helps explain the rapid adoption of practices such as Kanban on IT projects – Kanban literally meaning “signal board” – leveraging visualization of flow and indicators such as work-in-progress problems (too much or too little) to help quickly raise the “problem” flag (I talk a little bit about Kanban here:  http://www.infoq.com/presentations/A-Dash-of-Kanban-Anyone)

In addition to the raw project data, you might also look for “subtle” pieces of information that tell you a lot.  Has a developer not checked in code for 5-6 days when normally they check in code every 1 or 2 days?  Has your team been updating their work?  Do they have too much work assigned to them across too many projects and just aren’t letting anyone know?  I consider these “soft metrics” – yet still extremely valuable in helping to detect problems early so that you and your team can have an opportunity to act.

Don’t think that having metrics and fancy charts will be the solution to your team problems.  You must truly understand what these metrics mean and more importantly how and when to act.  This is why I like using tools – as good tooling can help you.  Catching issues, even potential issues, as they happen gives you the insight to act – ultimately helping you reduce overall risk and more importantly help catch and eliminate waste.