TR Forums

Hi,

For those who do development for a living, how many defects / work hours is considered "bad" in your shop. The high muckety mucks are trying to reduce QA defects to 1 per 5000 project hours. To me this is insane and almost impossible to meet, so I am curious what goes on at other shops.

I don't have anything useful to add, but the phrase "muckety mucks" made me nearly spill my drink.

Hawkwing74 wrote:

For those who do development for a living, how many defects / work hours is considered "bad" in your shop. The high muckety mucks are trying to reduce QA defects to 1 per 5000 project hours. To me this is insane and almost impossible to meet, so I am curious what goes on at other shops.

Soo.... each programmer is allow one defect per 2.5 working years? How is a defect defined and how many do you have per project hour right now?

By defect they mean code defect as opposed to environment defect, requirement defect, etc.

In a project going in soon we have 2 defects and we have perhaps 500 hours assigned.

One was - defaulted a priority in the database to 1 rather than 0.
Other was - did not update the record's change user properly.

Both were very simple things fixed in less than a day but we have blown that metric.

Are these defects during development or after release? Alternatively, is this a released project or an ongoing one? I'm not in software development, so I don't know what's typical, but I would assume that defects would be higher in pre-release and higher in "ongoing" projects (stuff that is always live but you have to keep updating). Humans make mistakes...it happens, so just deal with them.

To give a different perspective, I work in the heavy truck industry, and we have defects/truck metrics, and defects can come from supplier errors, engineering errors, or plant operator errors. I think it's okay to have a couple defects per truck as long as they are minor. The main concern is making sure that the trucks don't need a bunch of rework after they roll off the line.

Yes, that's insane.

Artificial metrics like this invariably lead to unintended consequences. If they stick to their guns, they will get their wish, but not in the way they expect. The defect rate per work hour will indeed plummet; but it will be because the rate of production will fall to near zero as all resources get redirected to focus on defect prevention. There won't be any time left to write actual useful code, and without code you can't have defective code!

superjawes wrote:

Are these defects during development or after release? Alternatively, is this a released project or an ongoing one?

These are QA defects. Code has not moved to Production environment yet. So pre-release.

That's a hilariously low rate:
http://www.sei.cmu.edu/library/assets/s ... rofile.pdf

What they are saying about PSP is mostly snake-oil as usual, but look at their referenced study from JPL:

PSP sales pitch wrote:

In over six years of system testing, the Jet Propulsion Laboratory (JPL) found 196 defects in this 22,000 LOC system. While only 45 of these defects were judged to be fatal,

So, that's ~10 defects per kLOC, from JPL engineers/programming developing something for a spacecraft. The idea that regular developers are only going to have one defect in two-and-a-half years of working is nutso.

Unfortunately, your muckety-mucks are likely sniffing that same snake-oil (or something like it), and thus believe that with the appropriate "magic methodology" they can turn six years of testing/development into six weeks. (I'm not joking, the sales pitch I cited literally claims that!). And, then, of course, they won't even properly follow the magic mantras anyway, so even if you believe in the mumbo-jumbo they're still doing it wrong. (and that's how insultants making their living!)



The craziest thing is that "defect" can be awfully subjective to begin with, making attempts to quantify them trago-comically incoherent.

Hawkwing74 wrote:

superjawes wrote:

Are these defects during development or after release? Alternatively, is this a released project or an ongoing one?

These are QA defects. Code has not moved to Production environment yet. So pre-release.

So it's stupid. Gotcha.

superjawes wrote:

Hawkwing74 wrote:

superjawes wrote:

Are these defects during development or after release? Alternatively, is this a released project or an ongoing one?

These are QA defects. Code has not moved to Production environment yet. So pre-release.

So it's stupid. Gotcha.

Stupid is an understatement. The last project I was on that tracked metrics anywhere close to that level had a bug (defect) tracking system. Defects were classified severity 1-5. Severity 1 was a dead in the water type bug -- stuff stopped working, data got corrupted, etc. Severity 5 was for stuff like a random mis-spelling in a help field or something like that. In order to be considered production code, it had to have no sev 1 or sev 2 bugs outstanding. This was code for ground station processing for a NASA project. There probably more than a thousand sev 5 defects logged. As far as I know, there wasn't any effort being made to address any of them other than as a side effect of other work being done.

Depending on what is considered a defect, a simple typo every 2.5 years would bust the metric.

Defects per work hours is a useless metric by itself. I can go an entire year without any defects, but I might only produce 2000 lines of code. Or, I can churn our 50,000 lines of code in that same year that has 50 defects in it that each take a 8 hours of Q/A to find and retest after fix, and another 8 hours to fix. Which is more productive?

If you are only finding 1 defect per 5000 programmer hours, then you don't need a Q/A team. Defects are fine. As developer productivity goes up, the defect rate will go up. So long a productivity is increasing faster than defects, it is a net win.

This all assumes that you can accurately measure developer productivity with something as simply as lines of code produced per hour and defects per LOC. Hint: you can't.

Good luck and be on the look out for other stupid metrics driving management directives that signal its time to be considering a change of employers.

--SS

I would like to meet the developer that makes one code behavior mistake per 5000 work hours. And then punch him in the junk because he's making the rest of us look incompetent.

Yes, that is insane.

We only consider defects to be "bad" when it's obvious that a developer didn't even do the bare minimum of unit testing. We have so many requirement defects, that they tend to drown out coding defects anyway.

If the muckety-mucks start to penalize developers for this, I see a lot of turnover in your company's future. And productivity approaching zero, as JBI stated.

nerdrage wrote:

We have so many requirement defects, that they tend to drown out coding defects anyway.

This. It's not that the requirements are bad, it's just that they don't always account for every contingency.