As an industry, we are terrible at estimates.
There are reasons for this, however. The chief reason is that we build things that do not exist in a physical sense. Lacking the constraints of the physical world, people on both sides of the table (customers and developers) can make fanciful assumptions about how much work is actually involved in accomplishing a given task.
Another reason is that the full scope of the project is not really known until relatively late in the project. Assuming a RUP-style process where there are overlapping phases of requirements-gathering, design, and implementation, it should be the case that your understanding of the project continues to develop over the course of the requirements-gathering period, which then informs your design, which informs your development. So there shouldn’t be any surprises at the end of the project. In practice, however, the people who gather the requirements for large software projects are generally not the same people who do the design and implementation. There may be limited communication among the teams. And in general, users and requirements analysts tend to not be experts at software development. So it often happens that the users and the requirements analysts meet for an ongoing period of time, write a requirements document, then hire a few developers to implement the requirements. So the newly-hired developers must learn the requirements while they’re writing the system. So the continuously-refined picture of the system that should theoretically happen never occurs. Worse, the requirements tend to be written with little input from technical experts
Following from the previous situation, overspecified requirements are another cause for surprisingly long schedules. Everyone has heard stories of $5000.00 hammers and $10,000.00 toilet seats. That’s the kind of thing that happens when requirements are specified in a vacuum.
What to do ? In my opinion, iterative development is a key part of the solution. The waterfall process needs to die and be exorcised from the world, as if it never even existed. It is purely evil. Projects should have a series of milestones, each of which results in a real executable system that does something. I’ve seen a lot of projects that give lip service to iteration, but are really just waterfall projects in disguise.
The first iteration should do almost nothing. Well, it should run on the target architecture, and it should implement a single use case that touches all the moving parts. It shouldn’t have a fully built-out user interface; an input box and an “OK” button are sufficient. It should be able to take an input, save it to whatever persistent store you’re using, retrieve it, and display it. Do that, and declare victory. You now know where all your development systems are, how to use them, and you know that they work together.
Next you should address non-functional requirements and architectural issues. You don’t have to have all the details locked down, but you need a big-picture understanding of the technologies you’re going to use. You need to get your security mechanisms in place, so that they’re in place through the rest of the development process. You need to have a rough idea of how many users you’re going to have, how heavy a usage level you’re going to encounter, and so forth, so that you can ensure that your infrastructure is sufficient to handle the demand. Cloud-based systems have less concerns with scalability and provisioning than more traditional systems where you have to buy your own hardware and set it up, but not every system is a good fit for the cloud.
Once you’ve got those things in place, you can be a lot more confident that you’ll be able to deliver a working system at the end of the project.
There’s more to say, but I’ll leave that for another day.