Why are software development schedules so hard to define?

Estimate of software development schedules
Software development is a process of continuous refinement.
Software development is -- by definition -- an act of discovery and invention.
A well-run software project attacks areas of highest variability first to narrow the cone as rapidly as possible.


To be able to make realistic estimate workload and start planning, we should have answers to these two CRUCIAL questions upfront:

1. What assumptions and preconditions have not been yet fulfilled?
You can NOT start to make any estimate of the volume of work and start planning development before the questions are answered!

2. Have your client CONFIRMED the future solution mock-up: interface layout and operating procedures, before the coding has started!?

Do NOT start coding before your client has CONFIRMED the future solution mock-up and functionality!
System Requirements and initial questions
BE CLEAR to your client -- you can not start to make any estimate before you have received clear answers from him and his team on your questions


---------------------------
Issues with estimations
---------------------------
Many unexpected things could happen that could disturb your development timelines and estimations. If you are scaling up the software system, creating a new feature or doing code refactoring to an existing system, you should know where everything is and what software does. If you are a good software engineer, your system should be designed to allow for such new introductions in the first place.
If you do a lot of monkey copy-pasting or even rewriting, then it is not a time estimation issue, it is a different problem - probably bad unreadable code.

----------------------
Bad pre-planning
----------------------
A lot of issues with wrong estimates come from the single problem that most software systems, even the people who wrote it can not give the straight time-lines because, they did a bad job of pre-planning.

They did not plan what could have happened next and not ask what would the client want next.

If we need to introduce some particular feature in the future, how can we create an expendable modular structure to fit future modifications and re-tailoring. As long as those exist, estimation is really manageable.

------------------
The art of start
------------------
What we need is to get at least some (elementary, small) part working first, so that we can tell that we are moving forward. For the start, we need to construct fully functional working base, which is incrementally extendible.
..................................

★ Deciding which parts of a software system to develop, reuse, re-tailor, lease, or purchase
★ Making legacy software inventory decisions: what parts to modify, phase out, outsource, etc


Iterative Development
The word "iterative" means that it involves repetition.
Iterative Development is a development approach that "cycles" through the development phases, from gathering requirements to delivering functionality in a working release.

---------------------------------------------------
Iterative & Incremental
---------------------------------------------------
The mission Project Apollo had was given with a high level Kennedy goal which says: "Build a system which can control the equipment to carry men to the moon and land them on the surface. The system must also be able to bring men safely back to Earth".
Which piece of the system can I design, code and deliver first to my customer and provide usable functionality?

In XP terms, the Project Apollo was series of spikes:
1. build a rocket engine that has enough thrust
2. build a rocket that can lob a man into space
3. build one that can lob a man into orbit
4. dock two space vehicles in orbit
5. land an unmanned one-way vehicle on the moon to check the environment
6. fly round the moon
7. test out the lander near the moon
etc

Apollo programme was, in itself, an incremental and iterative project. So, NASA has been an Agile organisation for much longer than many might give them credit for.
It wasn’t until Apollo 11 that they landed someone on the moon. Each mission, from 1 to 11 (as well as the numerous missions prior to that not carrying the Apollo name), was a process of experimentation and gradual improvement towards the end goal.
Think of each Apollo mission as a ‘release’, each of which improved upon the previous. Following the tragedy of Apollo 1 (where 3 astronauts lost their life) subsequent Apollo missions were unmanned progressing back to manned flights. Incrementally, missions progressed to achieve earth orbit, then lunar orbit and finally, landing on the moon.
Learning from each mission was used to improve and refine the portion of the journey achieved so far and to guide the next increment.

The mantras of IterativeDevelopment are:
★ Phases are Time-Boxed not Functionally-Boxed.
★ Test early, Test often.
★ Deliver early, Deliver often.
★ Production Quality.


Why are software development schedules so hard to define?

In classical engineering disciplines, like bridge building, if you are doing a project similar to other projects you have done before, using familiar tools and a familiar team, and if you are given firm, written requirements, then it should be possible to make reasonably good estimate.
On time, on budget - without the critical bugs in production is the holly gral of software development.

Getting us engineers to accurately estimate and determine the sequence of tasks to be completed is non an exact science.
For instance, customer X wants a feature A by some D-day deadline -- most of time it is just an unrealistic wish.
In implicit requirements there are "known unknowns" and "unknown unknowns."
Do NOT accept implicit or ambiguous requirements.

The KISS principle has helped me a great deal with my productivity over the years. Whenever I lose sight of the fundamental elements of a project, I remind myself to keep it simple. I don’t allow myself to add anything beyond the most basic elements of the project until I’m 100 percent finished with that simple, first deliverable.

It is impossible to estimate all complex development tasks accurately, but simple development tasks may be estimated with some precession.
Start small.
Then increment and iterate.
Ship some tiny but working product which can grow.
It is often NOT possible to schedule and enlist all necessary tasks on the road-map.

If you break the project down into its most granular subtasks and figure out exactly how each of them will function, then work through everything to ensure that you understand the implications of those pieces combining, and thoroughly review any new or not-fresh-in-your-mind technologies involved, and get every unknown out of the way, then you absolutely can provide a pretty darned accurate estimate. Of course, you've also done nearly all of the programming, leaving just the coding part, and you can't know in advance how long all of that estimating will take.
How do you schedule development time to implement this feature knowing that several bug fixes may be needed along the road?

It is difficult to plan a trip when the navigator is not sure of the destination and the driver doesn't know the road-map.
a. Define your goal: Marketing & business requirements freeze, MoSCoW style
b. Define your road-map -- requirement freeze, MoSCoW style
c. Define your data structures
d. Create independent software components

We are often asked to estimate projects that use new tools, technologies, where requirements are not frozen.
It's like asking how long will it take you to drive somewhere you've never been before, and you do not know your final destination.
You can give an guestimate based on distance, but you can't factor the amount of traffic during rush hour, or unplanned road-blocks.
There are an infinite number of things you can't know about the future.


It's not easy to know what exactly is required to do a product, and especially how to do it.
Some problems may arise. For example, if, to begin your work, you must install on your machine a fancy third party software component and the installation fails and the support is non-existent, you may spend weeks solving this issue.
Requirements are often not clear enough, but after analysing them at the first reading, you think wrongly that they are fine. Sometimes you can understand that the mean 'A', and, after starting to implement them, you would notice that they mean 'B' or even 'C'.
Customers like to change their requirements exactly when you've just finished the concerned part, and they really don't understand why are you requesting two more weeks and $5000 to do a tiny change, because they don't see that this tiny change requires to change other things, which require to change other dependencies, etc.

There are days when you can work for hours and proceed well, but there are days when, after writing few lines of code you are stuck.

Things become really bad when your estimate depends on other people. For example, in one 3 month project I had to wait months for a customer production data to finish my job. This customer took 3 months before delivering a piece of unusable data. Of course the project was late and frustrating.
Make the line, and refuse to accept the project if all project kick-off conditions are not clear or satisfied to some acceptable level.

Your estimate also depends on your customer. I had customers who spend weeks before answering their mail. It can really affect your schedule when you must wait for their answer (for example if you are asking them to precise this or that requirement).

What can you do?

Give a larger schedule. If you think you can do the job in two weeks, say you will deliver it in one month.
Be clear. If you rely on a test-case data, another developer, etc., say it. Instead of telling "I will deliver the solution in three months", say "I will deliver the product in two months after the verified test-case data are delivered."
Explain that if requirements change every day, the project will hardly be delivered in time.
Slice your schedule. Delivering components of a large project on time is easier.
Never give an estimate when you have to work on a source code of somebody else: you can never predict the source code problems and how much time you'll spend porting it to new platform or adding the new feature.

Psychologically, customers might be less angry when you said you will deliver something with some minor bugs in a week, but well tested and production-ready software in two months.
And if any additional bug will be spotted in production, you will fix it with no additional charge to customer.

You can just be depressed, or having temporary difficulties to concentrate, or having personal/family problems, or be too stressed by other customers or by some irritating project manager. There are situations in a team, when someone needs to be on leave, for joy or sickness.

The undesired situations happen to some extent with every programmer. It's just that it manifests itself in ways that aren't always obvious. One week working through a given problem might take 3 hours because you're super-sharp and "in the flow" while the next week it takes 3 days.


You cannot tell how long time they will take to solve some requirement until you have had a very careful look at them.

The most obvious reason why you engineers can't give you accurate estimates is that it's impossible.


In software, it's worse:
Estimates often become a commitment, so our judgement is modified.
Software development is -- by definition -- an act of discovery and invention.

There can be huge differences between the estimation of different engineers for the same task.
Uncertainties are very common, how much of us get stuck on a stupid bug or system crash destroying any apparent good estimation.
Human brain is limited. It has not been designed to estimate long running tasks.


So if someone asks for a delivery date, the best bet is to say "I am working on it, as soon as I get a clear picture, I will let you know how long it will take"?



Often it is easy to give an estimate, if you tend to work on the same kinds of problems for each project, with only minor changes.
If you referring to reasonably simple stuff, maybe one more RESTful service that basically just returns database table rows it is easy to give an estimate.


When I work on the same product, the problem sets change with ever release/iteration. I have never had to estimate something that took more than 6 months to complete.

I have learned a lot from this book: Software Estimation: Demystifying the Black Art
In short to get better estimation results we do this:
all but trivial tasks are estimated by more then one person (two in our case)
we divide task into smaller task. The more tasks you have the more likely your estimation will be good.

Estimate consensus:
We estimate worst, best and most likely scenario. Sometimes each of us estimates those tree scenarios totally differently. That means we have to talk and usually it turns out that one of us did not understand the task. Had that one person estimated the task alone it would be estimated wrong.

After work task is finished and our estimate was wrong we try to find reasons why. And we incorporate this knowledge into next estimation process.
So far this is the best process I have used for estimation of bigger tasks, i.e. I mean jobs that take from about 50-500 hours.

Note: This really only applies to projects where you bill by the hour versus a fixed/flat rate for some specified task.
I usually try to plan out my schedule so that it consists of essentially a bunch of SCRUM Sprints (whether using SCRUM or not). Up front when making the schedule I determine what the length each sprint will be and what the features will be for the project. Typically there are some features that have to be done first so I try to give a best estimate (not to be confused with optimistic) for those and any features that will be towards the end of the project will have generalized estimates. After mapping the features to sprints I try to add 1 to 2 sprints at the tail end of the project to account for features that slide right and for features that were overlooked in the original requirements gathering.

The key to this is that I make all of this transparent to the client upfront so they understand why the last two sprints are empty or sparsely populated.
At least to this point the client's I've worked with have liked this since they know that there is some cushion in the schedule/financials since most of them are aware that SW estimates tend to be less than concrete. They are also aware that if we don't need the last sprint or so then those are hours that we don't bill. With the transparency in how the schedule is built and the regular feedback on how progress is going during the execution of the project every client that I've done this with has been extremely satisfied.




Estimate all the non-development tasks:
Some optimistic developers notoriously forget to estimate all the non-development tasks like meetings and emails concerning the project, deployment, QA support, helpdesk support, research, documentation etc. Once we added these types of task to our estimation spreadsheet, our estimates got way better.


When it comes to time estimation for tasks which can take longer then a few hours I try my best to use this rules:
Don't ever try to predict when other people will finish their job if you happen to depend on it. Speak only for yourself.
First analyse the task, then estimate.
By analysing I mean at least writing down (and not trying to keep everything in you head!) a list of subtasks with estimation for each one of them.
If a task is complex enough, estimate the time for such analysis itself. Let the estimation be a separate process. You may also make sure your boss knows about that.
Don't estimate under pressure. Make it clear that it takes time to make a reasonable estimation and just tell them something like "I will name a date tomorrow by 11:00 when I'll finish analysing the task". I know some customers/managers can press hard, but they shall not pass. Put your busy face on and claim your time!
Ask a little more time than you think you'll need because you probably forgot to add a coffee break time (and other distractions) in your estimation.
For big tasks ask even more - there will probably be more than one coffee break.
If you really can't estimate (the task is too difficult/unfamiliar) or really unsure - ask someone capable for help with your analysis.

There's probably more rules than that, but I actually don't have a poster with this rules on my wall. I just formulated them now, but they come from my experience (so it may not work for you).

The only reliable way to schedule software development I can think of (but I haven't actually tried it) is Evidence Based Scheduling which is basically the Monte Carlo method used to count probability of a ship date based on historical records about tasks that you've accomplished before. It feels nice because it doesn't try to use any metrics other than estimated and actual time. However, it requires a lot of experience to split big tasks into smaller ones beforehand and you have to have a big set of historical data to make it work precise enough.


Developing software to meet functional requirements with acceptable quality, on planned schedule, and within budget is a target pursued by every software development organization.
Answers to these questions can sometimes discover unrealistic business targets coupled with too low budget and or poor planing.

NASA identified ten fundamental quality characteristics of software requirements specifications that are closely related to indicators of specification strengths and weaknesses, which are referred to as “quality indicators.” A quality requirements specification must be:
• Complete
• Consistent
• Accurate
• Modifiable
• Ranked
• Testable
• Traceable
• Unambiguous
• Verifiable
• Valid

Estimates often become deadlines.
Nobody want to miss a deadline and become headline!