Too many developers hated planning because the plan had never been of any personal benefit to them. Instead, plans were often used as weapons used against the developers
- Succeeding with Agile, Mike Cohn
Planning and estimating is often unpopular and causes a lot of conflict. A common approach is to plan and estimate up front and then occasionally re-plan at random times during the project. The idea is to 'get on with it' and try and meet the target dates, particularly when these are set in stone. The result is often schedule over-runs, poor quality releases and stressed-out development teams.
Agile takes a different approach, it plans to plan. The idea is that Scrum planning will be done at regular intervals and just before each iteration (i.e. sprint) is due to start. Please note this is a very Scrum focused post.
Why is this better?
Well, first of all, estimate accuracy decreases as the size of work being estimated increases. In typical development projects an estimate on a 1-day job can be +/- 50% accurate. Estimate errors are additive, so planning several months in advance results in a huge error range. It is surprising that projects ever come in on schedule, until you realise that development teams often compensate for the inaccuracy by adjusting quality and working longer hours.
Also, the more experience you have, the easier it is to estimate. If a development team has already built 5 portlets, the sixth portlet is easier to estimate for.
But estimating every iteration is an overhead. Why bother doing this, especially if you have a fixed deadline?
The answer is that it is much easier to correctly respond to the situation if you accurately know what the current situation is. You can still work the longer hours or cut quality, but it is a conscious decision that is discussed and planned for.
So what is the benefit for the development team? All developers have stories of working really hard on a project for months and still being critisised for not getting things done in time. This is not just a problem with initial estimates, it is more frequently a problem with re-estimating. In the course of the project, things change and there are usually integration and dependency issues. If you don't continually re-plan you are pretty much guaranteed to experience pain.
What is the agile approach to planning and estimating?
Agile planning is based on a pipeline of requirements (i.e. stories) that start out vague (epics) and get more detailed as they get closer to being worked on. The goal is to get just the right level of detail on a story just before it goes in to a planning meeting. In Scrum this works by starting with your stories (or epics) in the product backlog. Then adding a bit more detail and moving them in to the release backlog. Finally, you make sure there is enough detail in each story (including acceptance criteria) and then bring them in to the sprint planning meeting.
This process is called backlog grooming and is often missed out when teams implement agile. When it is missed, it tends to result in long and painful planning meetings. The best approach is usually to plan one or more backlog grooming sessions in to each sprint in order to prepare the stories for the next sprint. The aim is to make the planning meeting fast and routine.
Ideally the planning meeting only deals with stories. But in the real World there are often bugs, technical tasks (e.g. setup the new server) and technical debt issues. The various types of issue should be treated the same way, prioritised and estimated. The Product Owner makes the decisions on priority, but they should be heavily influenced by the technical team. For example the technical team might explain that a particular technical debt needs to be fixed so that they become more productive. That may make it a higher priority than a user story. When the Product Owner and technical team work well together this process is most productive.
Scrum has introduced the concept of story points
Traditionally work is estimated using time units (e.g. this task will take 2 days). But Scrum recognises that time estimates are relative to who does the work and any number of other factors. The concept of story points is based on relative sizing. You start by thinking about what is a typical small task. Then you relate other tasks to the typical small task. For example task A is small, task B is twice as big as task A. Some teams give these sizes such as small, medium, large, extra large. Other teams use Fibonacci numbers: 1, 3, 5, 8. What you use is unimportant as long as the concept of relative sizing is there.
But if you estimate in these timeless units, how do you then work out how long work will take? The answer is that you do this by learning how quickly your team works. For example, if your team can do 20 small tasks every 2 weeks this is a good guide to the team's velocity.
Of course it takes a while to work this out. That is why a newly formed Scrum team often struggles to estimate well. A stable, long-standing team can get very accurate at estimating.
It is important to keep in mind that the velocity of the team will change over time. If the teams gets more experienced or if team members are added or leave then the velocity will shift. Think of the velocity as a rolling average of the team's capability.
So is there no value in time based estimating?
Some Scrum teams use two rounds of estimating: starting with agile story points and then doing a more detailed round of time based estimates.
The benefits of this include:
- It can be easier to track progress on tasks
- You can put the time estimates in to an issue tracking system like JIRA and have the team log work against the task
- In theory an 8 hour task that has 4 hours of work logged against, will have 4 hours of work remaining
- Scrum teams can use the total number of hours as a sanity check that they have not overloaded their sprint
- The detailed discussion required to do time based task estimating may have value in iteself
It could be argued that this approach loses its usefulness when the story size is small. There is also a growing movement in agile to minimise the time spent estimating as it is seen as waste.