What we're covering today
Where do errors come from?
How often do we make errors?
How costly are errors to fix?
How can we catch more errors?
What effects will increased (testing, code review, design) have on development time?
Methodology
- Survey a large company (Raytheon, AT&T, IBM, Motorola, etc)
- Determine defect rate (survey, interview, or automated collection)
- Determine tools used to catch errors (testing, code review etc)
Methodology (cont'd)
- Ask many programmers to write same program, w/ different emphases
- Give programmers a program w/ known errors, ask them to find/fix
Stop using the word "bug"
Examples of bugs
- Colony of ants infests your laptop
- Bird poops on computer & shorts the motherboard
- Termites chew through server cables, cause data center outage
Things that are not bugs
Syntax errors
Null pointer dereference
Misunderstanding requirements
Terminology has implications
Bugs are "random", acts of God
Errors can be measured
Error rates can be tracked
Errors can be reduced
50% of time is spent debugging, refactoring, reworking
Mills 1983, Boehm 1987, Cooper and Mullen 1993, Fishman 1996, Haley 1996
Room to improve
10x differences between pro programmers on:
- Size of completed program
- Speed to complete program
- Error rate
- Error detection rate
Room to improve, cont'd
Productivity variation between programmers
Sackman, Erickson, Grant, "Exploratory Experimental Studies Comparing Online/Offline Programming Performance", 1968
Room to improve, cont'd
Curtis, "Substantiating Programmer Variability", 1981
Room to improve, cont'd
Demarco and Lister, "Programmer performance and the effects of the workplace", 1985
Where do errors come from?
What kinds of errors?
-
18%-36% of errors are clerical errors
(Weiss 1975, Card 1987)
-
The 3 most expensive errors of all time - $1.6 billion, $900 million, $245 million - involved changing a single character in a correct program
What kinds of errors?
Most errors (~85%) can be fixed in a short period of time
Endres, "An Analysis of Errors and Their Causes in System Programs", 1975
Most errors are the programmer's fault
Other common error sources - changing requirements, communication breakdown, thin domain knowledge
What things are correlated with errors?
Unused variables (Card, Church, Agresti, 1986)
What things are correlated with errors?
High numbers of comments
What things are correlated with errors?
Complex control flow
(McCabe, "A Complexity Measure", 1976)
See: gocyclo
3+ layers of nesting
(Yourdon, "Managing the Structured Techniques: Strategies for Software Development in the 1990s", 1986)
How are errors distributed?
In general, 80% of the errors come from 20% of the code.
Also, 50% of errors from 5% of the code
Case study at IBM: 31 of 425 classes found to be error-prone.
After repair/refactoring, defects reported by customers were reduced by 90%
Capers Jones, "Software Assessments, Benchmarks, and Best Practices," 2000
How often do we make errors?
Best estimates: you will find 1-25 errors per 1000 lines of code
(McConnell, "Code Complete", 2003)
5-8 defects/hour during coding
(Humphrey, "Introduction to the Personal Software Process", 1997)
How many can we expect to find?
If you do it well: 70%
If you do it poorly: 20%
Effectiveness of different bug-finding tools
What the heck is an inspection?
What the heck is an inspection?
Michael Fagan (IBM), "Design and Code Inspections to Reduce Errors in Program Development", 1976
- 4 roles: Moderator, Author, Reviewer, Scribe
- Everyone prepares, brings notes
- Solutions not discussed
- Management isn't present
- Reviewers have checklists for points to cover
Inspections crush testing on effectiveness
Inspections crush testing
- Basili & Selby 1987: Code reading found 80% more faults/hour than testing
- Ackerman, Buchwald & Lewski 1989: 6x as much time to find errors with testing as inspections
- Kaplan 1995: 3.5 staff hours/error with inspections, 15-25 hours per error with testing
- Moore 1992: Microsoft spent 3 hours/error with inspections, 12 hours with testing
- Russell 1991: One hour spent on inspections avoided 33 hours of maintenance
What won't testing catch?
Changing requirements/lack of communication
Hard coded variable values
Unclear error messages (or typos)
Duplicated code
Inadequate comments
Leaving verbose logging turned on
If you do test:
Automate test procedure
50% of tests run manually were run incorrectly
Double check test code for errors
Tests as likely or more likely to contain errors than code (Weiland 1983)
Make errors hard to miss
Easy to miss erroneous output, use log.Fatal or panic in development
Review small changes
In one company, 55% of one line changes were incorrect
After code review was implemented, 2% were incorrect
Freedman and Weinberg, 1990
"Never debug standing up."
Gerald Weinberg
Pull request length
Example
Requirements change
Average change is 1-4% per month. (Jones 2000)
Store requirements in version control so changes are visible
Things you can measure:
- Compiler detected errors
- Total number of defects
- Errors per 1000 LOC
- Mean time between failures
- Defect severity
- Class/routine that caused defect
- Origin of the defect (Design confusion, syntax error, etc)
- Cost to correct defect (hours, $$$)
Cost of fixing defects at various stages
- Requirements defect is 5-10x as expensive once you've begun writing code
- Architecture defect is 10x as expensive once you've begun writing code
- Code is 10-25x as expensive after it's been released
- Requirements/architecture are 10-100x as expensive to fix after deploy
Fagan 1976, Dunn 1984, Shull 2002
Personal Error Tracking
Thanks!
Kevin Burke
These slides are available at:
←
→
/
#