Welcome to the last time you report on the number of projects your team delivered.
Are you still talking about how many projects your team delivered last year? Please don’t. You’re misrepresenting what your team, department, and organization delivered. Stop talking about projects completed and start talking about portfolio velocity.
At the beginning of each year, CIOs need to commit to deliver work. This could refer to a group of critical projects, features, buckets of work, or even just finishing certain carryover projects. Each year, we make these commitments, and each year we tell ourselves there must be a better way.
The fallacy of using projects as a metric for capacity
How does your team estimate their capacity for the year? The most common measure is the simplest: the number of projects completed. On the surface, this sounds reasonable. We have critical projects. If those projects are completed, we declare victory. There’s just one minor problem: this method is flawed.
Organizational growth presents huge opportunities, and it carries equal challenges. Living and dying by the number of projects your department delivers doesn’t work. Here’s an example of why:
Year One went great. There were 100 projects completed. The team is perceived as really delivering. In Year Two, the team struggled to deliver 50 projects. And, by Year Three, the team delivered only 25 projects. What happened? The budgets for the projects were doubled each year. Staff was added. There’s no excuse. The peppering of reasons—ranging from too much work to not enough time—fall on deaf ears.
This isn’t the full story. We need to add context. When the team was delivering 100 projects, they weren’t complex, averaging less than two months with budgets of under $50K. Companies mature as they grow. Year Two required additional roles beyond the original technical lead and developer. An architect modeled the solution. The security lead addressed authentication and authorization. The quality lead tested for external interfaces, which didn’t exist in Year One. A project manager managed the vendor relationship and procurement process.
In Year Three, complexity, effort and the budget increased. Projects required more rigor to ensure success working across time zones, engaging third parties, and with more business and technology complexity. Projects involved multiple departments, which required shared services—e.g., legal, procurement, infrastructure, service desk, etc.—and demanded an average of four to nine months of effort to deliver. Projects also required more funding for delivery given the increased number of participants.
We know that going from 100 to 50 to 25 projects delivered per year isn’t a story we want to tell. Our team has worked harder each year. We’re building a team of more competent staff. The organization has provided greater funding each year. We need to rewrite this story.
Agile velocity
Velocity is a great tool to measure the performance of a team. A team’s velocity is the amount of work a team delivers over a fixed period. In Scrum, velocity is commonly measured in “points,” also known as story points. The points measure the effort for a given story, i.e. a specific piece of work.
Consider three factors when estimating points: complexity, effort, and doubt.
Using a scale of 1 to 5, 1 is the least complex and 5 is the most complex. What matters is the relative value of the points. We could have chosen 1,000 to 5,000 or 100,000 to 500,000. What matters is that a 2 is twice the value of a 1. Agile teams can have trouble with absolute values. The concept of triangulation helps ensure consistency. Periodically during the estimating process, compare a 1-point and a 2-point story. Discuss whether the 2-point story is twice as complicated.
The estimation fallacy
In his book, The Black Swan: The Impact of the Highly Improbable, Nassim Taleb describes several lessons that impact how we estimate:
- Epistemic arrogance: people claim they saw it coming, e.g., we estimate well in general except for ‘’
- Narrative fallacy: missing cause and effect, people bend stories to fit current understanding, e.g., this project should be like that one
- Asymmetry: an unequal upside and downside, e.g., missing by a factor of 2 is easy
Michael Bolton provides a great example of project estimation fallacies with Black Swan-like events. In his example, a project is estimated to take 100 hours with each task requiring 1 hour. He added in some basic assumptions:
- 50 percent of tasks complete in 30 minutes, not 1 hour, i.e. the task is completed early
- 35 percent of tasks complete on time
- 15 percent of tasks experience bad luck.
Our 15 percent of “bad luck” tasks is broken out into the following assumptions:
- Little slips: 8 tasks slip to 2 hours
- Wasted mornings: 4 tasks slip to 4 hours
- Lost days: 2 tasks (one in 50) slips to 8 hours
- Black cygnets (baby black swans): 1 task slips to 16 hours, i.e. 1 in 100 tasks takes two days
Using this simple example, the probabilistically average project would come in 24 percent late. A project that was committed for Q4 would be delivered in Q1.
Introducing portfolio velocity
Our goal is to estimate the departmental capacity to deliver value. How do we measure this? Portfolio velocity.
Accuracy doesn’t matter, but consistency does. We can apply this new measure to estimate our departmental capacity. To calculate your portfolio velocity, perform the following steps:
- Select a project
- Estimate the complexity on a scale of 1 to 5, with 5 being the most complex
- Estimate the effort by estimating the number of months to complete the project ranging from 1 to 12
- Calculate the product by multiplying the complexity by the effort to determine the project velocity
- Sum the velocity for all projects to determine the portfolio velocity
- Calculate historical portfolio velocity for past years
- Determine reasonable growth rates on top of last year’s velocity
- Rank the projects in the portfolio 1 through n
- Draw an above and below the line based on your new forecasted velocity
By completing through step 5, you now have, in your hands, the forecasted demand. To determine if this velocity is reasonable, we need to conduct a brief exercise to discover the historical portfolio velocity by looking at prior years and conducting a similar exercise for projects from these years.
Let’s assume we determined that the velocity for Year One was 200 points, Year Two was 300 points, and Year Three is forecasted at 1200 points. Already our story is changing for the better. However, it’s clear our capacity isn’t going to increase four-fold.
Assuming moderate growth of 10 percent, we can forecast our Year Three velocity: 330 = 100 percent + (10 percent x 300). Using the ranked list, we can determine which projects will be delivered in Year Three and keep us at or slightly below our target portfolio velocity.
Estimation exploratory options
The 1-to-5 complexity scale works well. Alternatively, a base-2 sequence (0, 1, 2, 4, 8, 16, 32, 64, 128) or a Fibonacci-like sequence (0, 1, 2, 3, 5, 8, 13, 20, 40, 100) can help make the difference between projects more distinct.
However, humans are notoriously poor estimators, and I’ve found that teams more effectively estimate based on common orders of magnitude such as 1 to 5 or 1 to 10. It’s easier for a team to estimate the size of a mouse at 1 and an elephant at 5 versus a mouse at 1 and an elephant at 8 or a cargo ship at 64.
Measure what’s valuable
Too often, we get swept away by maintaining historical measures. Being accountable is good. Ensure you’re holding your team, department, and organization accountable for the right measures.
Make this year different—with hard commitments—by determining your portfolio velocity.