Personal computing discussed

Moderators: SecretSquirrel, just brew it!

 
Hawkwing74
Silver subscriber
His Holy Gerbilness
Topic Author
Posts: 13798
Joined: Wed Aug 20, 2003 5:51 pm
Location: Streamwood, IL

Company policy on QA defect density

Thu Mar 12, 2015 8:15 am

Hi,

For those who do development for a living, how many defects / work hours is considered "bad" in your shop. The high muckety mucks are trying to reduce QA defects to 1 per 5000 project hours. To me this is insane and almost impossible to meet, so I am curious what goes on at other shops.
 
SuperSpy
Gold subscriber
Minister of Gerbil Affairs
Posts: 2226
Joined: Thu Sep 12, 2002 9:34 pm
Location: TR Forums

Re: Company policy on QA defect density

Thu Mar 12, 2015 8:43 am

I don't have anything useful to add, but the phrase "muckety mucks" made me nearly spill my drink. :lol:
Desktop: i7-4790K @4.8 GHz | 32 GB | EVGA Gefore 1060 | Windows 10 x64
Laptop: i7 740QM | 12 GB | Mobility Radeon 5850 | Windows 10 x64
 
Glorious
Gold subscriber
Grand Admiral Gerbil
Posts: 10283
Joined: Tue Aug 27, 2002 6:35 pm

Re: Company policy on QA defect density

Thu Mar 12, 2015 8:47 am

Hawkwing74 wrote:
For those who do development for a living, how many defects / work hours is considered "bad" in your shop. The high muckety mucks are trying to reduce QA defects to 1 per 5000 project hours. To me this is insane and almost impossible to meet, so I am curious what goes on at other shops.



Soo.... each programmer is allow one defect per 2.5 working years? :o How is a defect defined and how many do you have per project hour right now?
 
Hawkwing74
Silver subscriber
His Holy Gerbilness
Topic Author
Posts: 13798
Joined: Wed Aug 20, 2003 5:51 pm
Location: Streamwood, IL

Re: Company policy on QA defect density

Thu Mar 12, 2015 9:04 am

By defect they mean code defect as opposed to environment defect, requirement defect, etc.

In a project going in soon we have 2 defects and we have perhaps 500 hours assigned.

One was - defaulted a priority in the database to 1 rather than 0.
Other was - did not update the record's change user properly.

Both were very simple things fixed in less than a day but we have blown that metric.
 
superjawes
Gold subscriber
Gerbil Jedi
Posts: 1832
Joined: Thu May 28, 2009 9:49 am

Re: Company policy on QA defect density

Thu Mar 12, 2015 9:15 am

Are these defects during development or after release? Alternatively, is this a released project or an ongoing one? I'm not in software development, so I don't know what's typical, but I would assume that defects would be higher in pre-release and higher in "ongoing" projects (stuff that is always live but you have to keep updating). Humans make mistakes...it happens, so just deal with them.

To give a different perspective, I work in the heavy truck industry, and we have defects/truck metrics, and defects can come from supplier errors, engineering errors, or plant operator errors. I think it's okay to have a couple defects per truck as long as they are minor. The main concern is making sure that the trucks don't need a bunch of rework after they roll off the line.
On second thought, let's not go to TechReport. Tis a silly place.
 
just brew it!
Gold subscriber
Administrator
Posts: 49681
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: Company policy on QA defect density

Thu Mar 12, 2015 9:30 am

Yes, that's insane.

Artificial metrics like this invariably lead to unintended consequences. If they stick to their guns, they will get their wish, but not in the way they expect. The defect rate per work hour will indeed plummet; but it will be because the rate of production will fall to near zero as all resources get redirected to focus on defect prevention. There won't be any time left to write actual useful code, and without code you can't have defective code!
Nostalgia isn't what it used to be.
 
Hawkwing74
Silver subscriber
His Holy Gerbilness
Topic Author
Posts: 13798
Joined: Wed Aug 20, 2003 5:51 pm
Location: Streamwood, IL

Re: Company policy on QA defect density

Thu Mar 12, 2015 9:44 am

superjawes wrote:
Are these defects during development or after release? Alternatively, is this a released project or an ongoing one?

These are QA defects. Code has not moved to Production environment yet. So pre-release.
 
Glorious
Gold subscriber
Grand Admiral Gerbil
Posts: 10283
Joined: Tue Aug 27, 2002 6:35 pm

Re: Company policy on QA defect density

Thu Mar 12, 2015 9:51 am

That's a hilariously low rate:
http://www.sei.cmu.edu/library/assets/s ... rofile.pdf

What they are saying about PSP is mostly snake-oil as usual, but look at their referenced study from JPL:

PSP sales pitch wrote:
In over six years of system testing, the Jet Propulsion Laboratory (JPL) found 196 defects in this 22,000 LOC system. While only 45 of these defects were judged to be fatal,


So, that's ~10 defects per kLOC, from JPL engineers/programming developing something for a spacecraft. The idea that regular developers are only going to have one defect in two-and-a-half years of working is nutso.

Unfortunately, your muckety-mucks are likely sniffing that same snake-oil (or something like it), and thus believe that with the appropriate "magic methodology" they can turn six years of testing/development into six weeks. (I'm not joking, the sales pitch I cited literally claims that!). And, then, of course, they won't even properly follow the magic mantras anyway, so even if you believe in the mumbo-jumbo they're still doing it wrong. (and that's how insultants making their living!)

:P

The craziest thing is that "defect" can be awfully subjective to begin with, making attempts to quantify them trago-comically incoherent. :wink:
 
superjawes
Gold subscriber
Gerbil Jedi
Posts: 1832
Joined: Thu May 28, 2009 9:49 am

Re: Company policy on QA defect density

Thu Mar 12, 2015 9:55 am

Hawkwing74 wrote:
superjawes wrote:
Are these defects during development or after release? Alternatively, is this a released project or an ongoing one?

These are QA defects. Code has not moved to Production environment yet. So pre-release.

So it's stupid. Gotcha.
On second thought, let's not go to TechReport. Tis a silly place.
 
SecretSquirrel
Minister of Gerbil Affairs
Posts: 2296
Joined: Tue Jan 01, 2002 7:00 pm
Location: The Colony, TX (Dallas suburb)
Contact:

Re: Company policy on QA defect density

Thu Mar 12, 2015 10:24 am

superjawes wrote:
Hawkwing74 wrote:
superjawes wrote:
Are these defects during development or after release? Alternatively, is this a released project or an ongoing one?

These are QA defects. Code has not moved to Production environment yet. So pre-release.

So it's stupid. Gotcha.


Stupid is an understatement. The last project I was on that tracked metrics anywhere close to that level had a bug (defect) tracking system. Defects were classified severity 1-5. Severity 1 was a dead in the water type bug -- stuff stopped working, data got corrupted, etc. Severity 5 was for stuff like a random mis-spelling in a help field or something like that. In order to be considered production code, it had to have no sev 1 or sev 2 bugs outstanding. This was code for ground station processing for a NASA project. There probably more than a thousand sev 5 defects logged. As far as I know, there wasn't any effort being made to address any of them other than as a side effect of other work being done.

Depending on what is considered a defect, a simple typo every 2.5 years would bust the metric.

Defects per work hours is a useless metric by itself. I can go an entire year without any defects, but I might only produce 2000 lines of code. Or, I can churn our 50,000 lines of code in that same year that has 50 defects in it that each take a 8 hours of Q/A to find and retest after fix, and another 8 hours to fix. Which is more productive?

If you are only finding 1 defect per 5000 programmer hours, then you don't need a Q/A team. Defects are fine. As developer productivity goes up, the defect rate will go up. So long a productivity is increasing faster than defects, it is a net win.

This all assumes that you can accurately measure developer productivity with something as simply as lines of code produced per hour and defects per LOC. Hint: you can't.

Good luck and be on the look out for other stupid metrics driving management directives that signal its time to be considering a change of employers.

--SS
 
derFunkenstein
Gold subscriber
Gerbil God
Posts: 23832
Joined: Fri Feb 21, 2003 9:13 pm
Location: Comin' to you directly from the Mothership

Re: Company policy on QA defect density

Thu Mar 12, 2015 10:41 am

I would like to meet the developer that makes one code behavior mistake per 5000 work hours. And then punch him in the junk because he's making the rest of us look incompetent.
I do not understand what I do. For what I want to do I do not do, but what I hate I do.
 
nerdrage
Graphmaster Gerbil
Posts: 1305
Joined: Thu Mar 06, 2003 2:49 pm
Location: Raleigh, NC

Re: Company policy on QA defect density

Thu Mar 12, 2015 10:59 am

Yes, that is insane. :roll:

We only consider defects to be "bad" when it's obvious that a developer didn't even do the bare minimum of unit testing. We have so many requirement defects, that they tend to drown out coding defects anyway.

If the muckety-mucks start to penalize developers for this, I see a lot of turnover in your company's future. And productivity approaching zero, as JBI stated.
 
derFunkenstein
Gold subscriber
Gerbil God
Posts: 23832
Joined: Fri Feb 21, 2003 9:13 pm
Location: Comin' to you directly from the Mothership

Re: Company policy on QA defect density

Thu Mar 12, 2015 11:05 am

nerdrage wrote:
We have so many requirement defects, that they tend to drown out coding defects anyway.

This. It's not that the requirements are bad, it's just that they don't always account for every contingency.
I do not understand what I do. For what I want to do I do not do, but what I hate I do.

Who is online

Users browsing this forum: No registered users and 2 guests