Our new project is off and running, but we have hit a few snags.  Before we get too alarmed, it is important to remember that it is Sprint 1, and if ever you're going to hit problems this is the time to do it, so they are sorted out by the end of the project.  Incidentally, this is what I see as one of Agile's best features - because each sprint is considered a mini-project in its own right, we get the chance to stand back and assess the situation before moving to the next phase of the project, rather than just ploughing on and hoping for the best.

Towards the end of this sprint, the burn-down seems to have flatlined - for every hour of work that seems to get done, someone experiences a problem elsewhere which causes production to seize up. 

My own analysis of the situation is just that there were too many unknowns at the beginning, and the tasks identified at the beginning of the sprint quickly became out-dated.  This can happen on any sprint, but normally it would only affect one area - this time it seemed to affect the majority of task lists and that's why it's had such a dramatic effect. 

What are the unknowns? 

For me, it's three things:

1. Developers being assigned to domains and/or codebases they are not familiar with.  Although there is some documentation available, it is easy to forget how complex the current system has become, even at a conceptual level.  Sometimes with the resources available then putting developers in unfamiliar areas can't be avoided - besides, learning is always a part of this job.
2. Unfamiliarity with new third party controls or tools.  Until you have looked at the control documentation in-depth and played around with it in your sandbox, you can only take a real guess as to how you would accomplish something with it, and therefore hazard an estimate.   Some controls might implement the entire feature just by calling a method, in others something you would EXPECT to be there, just isn't.  We tend to ask A LOT of our third-party controls, rarely do we find it acceptable to use just out-the-box, so this sometimes requires communication with the vendor support teams.
3. Software Development in general.  I don't think there's anything we can do about this.  It happens.  People will make the odd mistake - underestimating, overestimating, perhaps identifying the wrong approach to a problem.  The only thing you can do is learn from it and move on.

What are we going to do about it?

Well, if you want a highly accurate result then before we estimate ANYTHING, we should have a good, firm understanding of EXACTLY what you'll be doing over the next 3 weeks.  For this, you'd need to design the components and complete any prototyping - effectively implementing the feature - before this would be achieved.  This also is trapped in BDUF world, as opposed to the flexible, feature-driven delivery of the agile paradigm.

Personally, I don't think this is the way to go.  In fact, there are a couple (or more) arguments out there that question the usefulness of task level estimation full stop.  I'm not sure if I'd go that far, although it is an interesting argument.

My own proposal is one that I half-introduced in my last task breakdown, but didn't have the conviction to stick to it, although it made an appearance again in the retrospective.  The basic idea is that you identify as many tasks as you can that you're confident of, and estimate these as normal.  For those areas that you're unsure of - be it the domain, an area that you know will require significant design/architecture, or even familiarisation with a third-party control or codebase - then there should be two 'types' of task identified: Investigation & Implementation.

Investigation tasks are brief descriptions of work that you're just unsure of, for example "Investigate how to bind Control X to data", or "Investigate how to use this new tool" or "Investigate the refactoring approach I need to take".  An estimate is provided for these. 

The Implementation task is a traditional rough guess, probably based on the story points assigned, of how long you think it would take to implement the feature.  A rough guess.  This differs from normal breakdown estimates as you aren't identifying the tasks yet, and people should expect this estimate to be high-risk.

During the sprint, when the investigation tasks have been completed, the developer should then be at a stage where they have a much better idea what is going on.  'Proper' Tasks can then be identified, which take the place of the Implementation placeholder.  In addition, further Investigation tasks may be identified at this stage, and the cycle continues.

A Brief Example

In the above example, we start with an Investigation task at 7 hours and an Implementation task estimated at 14.  Once the first Investigation has been completed, Tasks 1, 2 and 3 will be identified and estimated with some degree of certainty, where Task 3 is another Investigation task from which Task 4 and 5 will be identified at a later stage.  As you can see, the first Implementation estimate had delivery of the feature within 14 hours - but the reality is it took at least 4 + 7 + 3 + 4 + 2 = 20.  A third more than the initial estimate. 

This gives the SCRUM-master (and other stakeholders) a much clearer visibility into the state of the sprint.  They can tell at-a-glance where the 'red-flag' areas are, and have confidence that when the implementation phase begins there are no more 'brick-walls' where a developer can lose days without realising it.

Honesty really is the best policy, even if sometimes the truth hurts.

But with this approach, how is time-boxing possible?

Calculating the exact number of tasks to fit into any given sprint isn't going to happen.  The estimates will either be under or over - the key here is to be agile.  With this approach, I would propose that no more than 50% of the tasks on the sprint should be Must-Haves - this allows us breathing space by letting tasks of a lesser priority slide to the next sprint, if necessary.  50% may seem like a large margin, but the rule of thumb for planning is always to double your initial estimates (and the very worst case scenario is that if a developer finishes early then the sprint can be ended or more work added from the backlog)

This approach also has the advantage that sprint turn-arounds are much quicker - you really shouldn't spend more than an hour or so planning for a 2 or 3-week sprint. 

Based on experience, I'm fairly confident the above approach could resolve a lot of the estimation and task identification issues that we have from time-to-time.  The important thing is that we have realised this early, and taking action to resolve it.