Blog

Sam Blog Image

Agile Metric Estimation

Estimation Dilemma

During the Egyptian revolution in 2011, thousands of people went to Tahrir Square in Cairo for mass demonstrations against the regime. The number of people in Tahrir Square was overwhelming enough but the variation of crowd estimation was even worse. The government news channels claimed only 8000 thousand in Tahrir Square while others in support of the revolution recorded 1 million 125 times the figure reported! 

Figure 1: Egyptian Revolution Tahrir Square
Figure 1: Egyptian Revolution Tahrir Square, credits: tahrirarchives

Politics aside, the fact is that crowd estimates can be quite variable. Estimates of the size of the crowd at the royal wedding of Prince William and Kate Middleton, for example, ranged from 500,000 to one million. At Obama’s inauguration ceremony, unofficial government estimates put the crowd at 1.8 million. Others gave estimates closer to one million.

Figure 2: William and Kate's wedding day crowds, credits: Jenny Goodall

Estimates can be very deceptive and this plays out in our day to day working life.  I recently discovered that one of the news channels used a metric estimation method to estimate how many people in Tahrir Square. They calculated the size of the venue Sand the maximum people per meter that could be gathered and discovered that the maximum capacity of Tahrir Square was actually only 500,000 people.

The same estimation issues occur frequently IT projects. It is not uncommon to have a variation of estimates on the same project when estimated by different people delivering the project based on factors such as;

  • Technical experience
  • Risk assessment skills
  • Clarity of the requirements from client
  • Assumptions
  • Business domain knowledge

Cone of Uncertainty

Researchers have compiled the last 60 years’ worth of software projects to measure the accuracy of the project estimates and plotted it over a chart as shown in Figure 3.  The same concept could be applied to agile projects as shown in Figure 4.

Figure 3: waterfall cone of uncertainty
Figure 4: Agile cone uncertainty

The chart shows that as the time progresses, the estimate gets better. So, if the initial estimate of the project is one year, then your estimate is accurate by plus or minus four. This means the project can be finished in four years or little as 3 months. However, as the project progresses, the estimate gets more accurate as result of the developing team become clearer about the product, business domain knowledge and more familiar with developing tools and technology used in the project. But our estimation will never be accurate 100% even during the last of stages of the project.

The uncertainty around how long exactly the project could take to develop and to be deployed into production causes a range of issues to the business such as:

  • Difficulty in predicting the project cost
  • Allocating resources
  • Planning training
  • Planning marketing events

How to Improve our Estimation

Split Big Project into Smaller Projects

Studies show that agile projects have a much better success rate – almost 3 times as compared to waterfall projects. However, adopting an agile approach in your project doesn’t guarantee high success rate in your project. As you can see in Figure 5 the success rate of large projects using agile approach is only 18% comparing to 58% success rate in smaller sized projects.

Figure 5: The resolution of all software projects from FY2011–2015 within the new CHAOS database, segmented by the agile process and waterfall method. The total number of software projects is over 10,000.

Large project will usually have a very wide cone of uncertainty, especially at the initial stage of the project. This makes the large project more likely to be challenged or even failed and delivered with much higher cost and time than the initial estimation.

One way of narrowing the uncertainty cone, therefore, is slicing larger projects into smaller ones, where each project is independent and more easily deployed to production, providing quicker business value to the organisation.

One Story Points Does Not Equal One Day

Many people get confused around the significance of story points in agile projects. Unfortunately, many development teams have the concept that of 1 story point is equal to 1 day. This often means that the project managers also assume 1 story point is equal 1 day which increases the cone of uncertainty instead of narrowing it down.

The purpose of using story points is to decouple time from effort and to use a different unit of the measurement to measure the effort from time. Story points will help to measure the team velocity later on but will not measure how long (in time) it will take to finish a specific feature or user story.

The Power of Relativity

Humans naturally don’t make great estimators. We tend to either be optimists or pessimists and very rarely realists. For example, if you were asked to estimate how long it will take us to walk up all of the buildings as shown in Figure 6 using the stairs

Figure 6: how long it will take us to walk up all of the buildings?

Consider you’ve never climbed these buildings before or aren’t sure how physically fit we are or what types of obstacles you might need to negotiate in the stairwells. Unfortunately, our estimate will never be accurate regardless of which approach we take. . It will be subject to plus or minus 4x based on our cone of uncertainty.

 

On the other hand, let’s assume the first building will take 100 story points to climb then what’s the estimate to climb the rest of buildings?

We can estimate the effort of climbing the other buildings as following:

Note all these estimated numbers are relative. Therefore, our estimates will be always relative and not absolute. So, if the first building took 1 hour to climb including rest time, then, how long it will take to climb the other buildings?

Using the Fibonacci Sequence Will Make You Better at Estimating User Stories

The Fibonacci sequence (1,2,5,8,13,21,34, 55) is based on the golden ratio and it’s important the team stick to the exponential sequence growth during the estimation sessions. The shorter the time span the more certainty. Longer tasks are more complex and has higher risk factor and time estimates are less precise. This will help the team to encapsulate the risk of large and complex tasks during estimation versus small tasks which carry less risk.

Stop Estimating and Use Metric

Let’s look at  a practical example of how to use agile metric approach to have better estimation, narrowing the cone of uncertainty and increasing the rate of successful projects.

Data to Collect

During each sprint we will need to collect data to facilitate more accurate agile estimation

Product backlog size

The backlog size will change during each sprint as the team will complete work items as well as items will be added or removed including bugs. The product backlog items could be calculated as following:

Committed Sprint Backlog

At the beginning of each iteration the team will be committed to complete selected items from product backlog. It’s important to record the sprint size which the team is committed to do.

Completed backlog items

The actual backlog item size that are completed during the sprint. This will be indication of how the team is performed during the iteration.

Cumulative team velocity

Measure team velocity during each iteration to record the fluctuations as the project moves. The team velocity could be measured as:

The initial velocity sprint will not be accurate, but as the project progress the velocity will become more accurate typically after the first 3 or 4 sprints

Item added to backlog

Item which was added or requested by business during the project

Item removed from backlog

Similar to item added to backlog, the business could request to remove a feature from backlog or replace it with other features

Bugs added to backlog

Regardless of the quality control you have in your project, there will always be bugs rate of which usually fluctuates during the project

Numbers of sprints to complete the project

To measure how many sprints are required to finish all backlog items and complete the project can be measured as follows

The initial velocity sprint will not be accurate, but as the project progress the velocity will become more accurate typically after the first 3 or 4 sprints

Item added to backlog

Item which was added or requested by business during the project

Item removed from backlog

Similar to item added to backlog, the business could request to remove a feature from backlog or replace it with other features

Bugs added to backlog

Regardless of the quality control you have in your project, there will always be bugs rate of which usually fluctuates during the project

Numbers of sprints to complete the project

To measure how many sprints are required to finish all backlog items and complete the project can be measured as follows

Allocated Percentage for New Bugs & Rework

As the project progresses, the bug rate and effort of rework or redevelopment usually increases. The percentage will be different for each project based on many factors, for example:

  • Existence of coding quality controls like unit test and integration testing
  • Length of project
  • Technology and framework updates
  • Quality of solution architect design of the project

Based on statistics, the percentage of effort to develop new user stores versus the effort of rework and bug fixing could look  something like Figure 7

Figure 7: Allocation % of bugs and rework through the project

Weighted sprints to complete

The weight sprint to complete is based on the allocation percentage of new bugs and rework.

Sprint Metrics Example

At the beginning of the project, let’s assume we have sum of 100 story points in backlog items. These items based on the initial project requirements and analysis of work which is required to build the project foundation. Then, our metrics could be something like the following

As shown in the previous table, note the following: 

  • Team velocity usually improves after the 3rd or 4th The team start becoming more familiar with the used technology with a better understanding of the business domain knowledge as project progresses.
  • The team velocity could be impacted if the team size kept changing every sprint. So, it’s important to have stable team to get accurate results
  • There are no hard rules to allocate percentage of new bugs and rework, and it will be different from team to team and from project to project.

Every Burndown Chart Has a Story to Tell​

The burndown chart is a great tool to help the team track progress, since it shows progress on a daily basis, it helps scrum master to predict if a team will be able to achieve the target. Also, it’s an indication of how the team is updating theirs tasks to ensure the accuracy of the data which will be collected during each sprint.

Ideal Chart

The graph below shows the ideal scenario where is actual and ideal lines are in sync and the team was able to finish all committed tasks at the end of sprint.

Figure 8: Ideal Chart

Early Finish

Actual curve finishing below ideal curve. So, if repeated then you might need to consider increasing the team velocity and commitment to more work.

Figure 9: Early Finish

Unable to Finish All Committed Work

Actual curve finishing above ideal curve. So, if repeated then you might need to consider decreasing the team velocity and commitment to less work.

Figure 10: Unable to Finish All Committed Work

Large peaks or valleys

Large peaks in the actual curve could means many things but the most common scenario is the team not updating their tasks regularly.

More Unexpected Work Discovered​

The jump in the actual burning curve could be because of more work discovered or more urgent bugs that are discovered during the sprint and must be resolved during the sprint. , if repeated then you might need to consider checking your quality control and increasing the allocation percentage of new bugs and rework.

Figure 11: More Unexpected Work Discovered

Stretched Curve

Team stretched toward end to meet the commitment​.

Figure 12: Stretched Curve

Inconsistent Curve

This could mean the team is not consistent through excessive meetings or not updating their tasks regularly.

Figure 13: Not Consistent Curve

Final Thoughts

The main idea of agile metrics is to narrow down the cone of uncertainty of the estimations and to notify the business as soon as possible for more accurate estimations. Importantly, metrics are just one part in building a team’s culture. They give quantitative insight into the team’s performance and provide measurable goals for the team.  This will help the team to be able to get unbiased estimations and to have more realistic estimations based on team effort.