John Moore asks: Why is it that the best bugs are found at the end of a release?
Posted by S4SD (Seth Morris) on 2009/05/04
I came across this on a LinkedIn group:
My summary of Moore’s answers:
- Features are delivered late and QA time is sacrificed to keep the date
- QA isn’t able to test whole-system issues like performance and scalability until the end
- QA spends too much of the up-front time doing planning instead of testing
- QA test plans don’t match the real world
- QA sticks to test plans robotically
- QA can’t find the bugs that customers can
Mr Moore has some interesting answers to a really good question; it’s worth a second consideration. Let’s start with the question itself. I notice three immediate, interesting things:
The term "best bugs" is a fun one. It isn’t clear from Moore’s post what qualifies as a "best1". Is it most subtle? Most careless/amusing? Easiest to fix? Most impact on the product? It could be any of these and retain the value of Moore’s post; it begs the very interesting question: What did you think of when you saw the term "best bugs"?
There is a presupposition of universality in the question: it isn’t scoped to one technology, language, problem domain or development methodology. Is it universally true that the "best bugs" are found at the end? Always? All of the "best bugs"? If this is true in your experience, or true in your experience with one team/company, do you think it speaks more to the nature of software development, to the current state of the industry, or to a specific individual’s (or team’s, or company’s)
methodology? Do you think any testing or development process methodologies or tools address this (or try to)?
For that matter, do you think it is an inherent property of software? Is it a bad thing? A good thing? How good? How bad?
In the list above2, the first two items are properties of the project itself: they’re project management issues. The other 4 answers are all ways of blaming QA’s methodology3. None consider the role that developers, project planning, and the general nature of programming play in the process.
I don’t mean to imply that John Moore blames every QA team in the world for making this a universal problem; he’s explicit that these are only some of the reasons for an observed phenomena, although he does describe these are the causes he’s seen most often4.
So let’s add some reasons that don’t implicate QA; everyone should get in on the fun.
Product Design, Marketing, Sales, etc:
- Product managers deliver incomplete specs and don’t finish the specs in a useful order; important features aren’t fully specced until very late in the lifecycle, leaving those features to be (re)written quickly at the end.
- Product management adds new features throughout the project, leading to features that don’t have the support of a full process of design, review, test planning, and inspection.
- Product managers never paper tested the design and product owners change the requirements when they actually see what they asked for, leading to quickly-turned-around rework.
- Sales or Marketing promise customers that they can see specific features early without the dev process being considered, restricting the ability either to front-load risk or front-load low-hanging fruit.
- Developers front-load risk and believe they can do the "easy stuff" at the end, without any planning.
- Developers take the easy stuff first and don’t give high-risk items attention until late in the process.
- As schedules tighten, development cuts corners on code inspection and documentation, leading to less well-thought-out work.
- Under the threat of a deadline, developers are likely to give less-tested code to QA, figuring they can save time on unit testing by risking the entire QA team’s time on a dead build.
- Work that requires technology new to the team (new languages, new APIs, etc.) has much higher risk and can’t be delivered until later in the project, when the developers have had some time to learn.
- Floundering developers are left on their own, causing their high-risk code to ship at the end or having it rewritten by more-senior developers at the end.
- "Cowboy" features and fixes are slipped in at the end, when developers are buttoning up features.
- Towards the end of a project, developers are focused on the next project and don’t give the current work their best effort.
- Any particular discipline in the process is likely to be pulled off on a random firedrill without the rest of the project understanding the full impact—often because no one but the executive could know how bad it is and the executive is in denial.
- Features that are pulled out for a future release become must-have features for the current cycle, often causing them to push ahead with incomplete (or no) planning.
- Project schedules are interleaved with no relationship between timeframe, risk, and importance—such as the most critical project delivering last, but taking resources from higher-risk projects that deliver first, causing those projects’ inevitable failures to impact the critical project and lead to last-minute attempt to catch up.
General Life in Software:
- QA find the easiest and most-visible bugs first; the subtle stuff takes time to discover, reproduce, and document.
- Simple bugs, particularly in the user interface, mask more interesting bugs in underlying code.
- Early in a project, QA is only likely to get code that developers have very high confidence in. As deadlines near, code has to be given to QA ASAP instead of sitting on a dev box until someone has time to go through it again.
- Interesting bugs are likely to cross functional boundaries, meaning they can’t be discovered until several functional areas are delivered (and working well enough to see them pipelined).
- At the end of a project, when deadlines are tight and the team is pushing hard, exhaustion overcomes the team, leading to bugs in the more complicates systems.
- At the end of a project, when deadlines are tight and the team is pushing hard, exhaustion overcomes the team, causing subtle bugs to take longer to fix and generate more false-fixes, further increasing their apparent complexity.
In short, everyone is susceptible to basic, human behaviors that lead to the "best bugs finish last" situation. It is fundamental to human nature and mitigating it is the #1 goal of every development methodology ever. It’s easy—and currently chic—to say that agile methods are focused on eliminating the "big ball of failure at the end of the project," and they can do a pretty good job of it, but don’t fall into the trap of thinking older (or newer) methodologies embrace failure. They have the same goal and make some reasonable attempt to reach it.
If I had to guesstimate, I’d say that development—in the roles of coders, inspectors, and estimators—is is responsible for the plurality of the last-minute "best bugs," with product management (in their role as "constant revisers of too-incomplete specs") being close behind. But really, all of these are a symptom of the fundamental problem of too many people trying to do too hard a job on too little time and with too little information. In other words, it’s all a symptom of human beings producing software in an entrepreneurial world.
We see it, we face it, we mitigate it, we plan for it, and we deal with it when it—inevitably—happens.
It’s what we do and it’s why we keep coming up with new ways to do it.
1) Moore’s post does say that these bugs are the "most interesting, and sometimes the most critical."
2) The list doesn’t match the bulleting or order in the original post; I have reordered them to group related items and I have flattened his nicely-nested bullets.
3) This could easily turn into a "developers versus QA" situation; let’s agree not to do that.
4) I certainly do believe that these are the causes Moore has seen most often. I do not believe these are the causes Moore has most often been exposed to. As a developer with a passion for SDLC and development process, I have realized that I am likely to see faults in QA’s methodology (without necessarily understanding the methodology well enough to be right) long before faults in areas I’m closer to, like dev process, coding, or design. I assume everyone, including Moore, has similar biases.
Listening to: Herb Alpert & the Tijuana Brass – Classics, Vol. 1 – Tijuana Taxi