Random Acts of Architecture

Wisdom for the IT professional, focusing on chaos that is IT systems and architecture.

Tag Archives: Quality assurance

Unit Testing: The 20/70/0 Rule

Leave a comment Posted by on November 4, 2013

20-70-0 Rule

Automated unit testing, one of the most important foundations of software quality, is still a struggle for many software development teams. Justifying the extra upfront time to business is difficult, particularly when the team is under deadline or resource pressure. Many teams give up when confronted by huge amounts of untested, untestable legacy code. However, avoiding or delaying unit testing hurts everyone.

Many misunderstand automated unit testing, making the message inconsistent or less convincing. For example, automated unit tests do not initially reduce the number or severity of code defects. Good developers should already manually test their code thoroughly, stepping through it in a debugger where possible. Good developers also manually check error conditions and corner cases.

Many concerns are also unfounded. For example, automated unit tests do not replace QA (testers). QA check software developers’ work and test at the functional level. Their different perspective can help write better automated unit tests, too.

Many complain about brittle unit tests only to find brittle “unit tests” are usually functional or integration tests, such as calling web services on external systems or accessing a shared database. Since these are not segregated, the unpredictable actions of others cause tests to fail.

Indeed, the biggest barrier to automated unit testing is software design. If a method or function cannot be unit tested, the design is incorrect. For systems outside the software development team’s control, see Michael Feather’s work on legacy code. Testable code tends to be better designed code, too.

The main benefit of automated unit tests is unit tests capture the expected behavior of a single unit of code, such as a method or function. These tests can be repeated quickly and regularly with little manual effort, identifying when code changes, refactoring or experiments break the expected behavior.

Software developers also forget important details as they move to other features or projects. Capturing expectations as automated unit tests retains this experience and knowledge.

Nothing mentioned above is novel. However, questions remain once agreement to add automated unit tests is reached. For example, how much unit testing do developers add? How much extra time is needed? How do you explain this to non-technical stakeholders? The 20/70/0 rule answers these questions:

First, spend 20% of development time writing automated unit tests. A day’s worth of testing each week is a good compromise. This is part of the development task, not extra effort. Otherwise non-technical stakeholders will demand skipping it when under pressure.

Second, aim for 70% code coverage. This excludes third-party or generated code, so make sure code coverage tools can exclude this. Interestingly, technical people tend to think this is high, especially if no automated unit tests exist. Less technical people ask why the remaining 30% cannot be covered.

Third, ensure 0 failing tests. Running automated unit tests after an automated build is a critical part of continuous integration. Fix failing tests immediately.

The first rule, 20% development time on tests, tells project managers and stakeholders the extra time to add initially. It also allows project managers to compare the up front costs with time savings later (ideally greater than 20%).

The second rule, 70% code coverage, tells developers what the team expects, particularly when code reviews highlight missing or poor unit tests. In an agile process, automated unit tests are part of “done” for development tasks.

Code coverage is an imperfect metric and heavily debated. Ideally, the team should target functional coverage. Behavior Driven Development (BDD) is one option. However, for a team without unit tests or a superior metric, code coverage is unambiguous, automatable and easy explained to less technical people.

The third rule, 0 failing tests, reinforces that quality is critical, especially to less technical people once again.

Software developers often get caught up in technical debate. Unit tests and quality are no different. However, projects can rarely wait for perfect understanding. The 20/70/0 rule is unambiguous and understandable, even to less technical people. Attaining it or, more specifically, the quality goal it represents is still a challenge but attaining it is now about metrics instead of gut feel and hand waving.

Software Development , , , , , , , ,

In Defense of Documentation

3 Comments Posted by on August 27, 2012

Anyone who has spent a modicum of time working with software developers learns that developers hate creating designs, use cases, “readmes”, code samples and similar technical documentation. Agile proponents point to the “working software over comprehensive documentation” principle and Lean proponents claim documentation is “waste” since it does not directly contribute to the finished product or customer value.

They have a point. Writing clear, understandable documents is hard and it takes time away from coding. Keeping them up-to-date when inevitable changes occur is similarly time-consuming, particularly large changes or redesigns. Consequently, many stop asking for documentation, exacerbating poor or absent documentation. Many developers argue “code is documentation”. It is always up to date and developers had to work to understand it. Why shouldn’t everyone else?

However, saying “code is the only documentation” is a fallacy. Well written and designed code and indicative automated tests can be good documentation but every project includes code the developers are not proud of. Software developers often think of themselves coding like athletes lining up for a race at the Olympics. They have spent years training and are lining up to compete under perfect conditions. However, frequently code is written when developers are tired, stressed or unfamiliar with the problem or tools and this produces less correct and readable code.

Also, and arguably more important, developers are not the only audience for technical documentation. QA (testers), localization, management and documentation writers (those that write the manuals and non-technical documentation) all need technical information about the product and usually cannot read code, at least not well enough to get the information they need time efficiently. Developers new to the product also need a starting point.

Part of the problem is that documentation, particularly in larger organizations, consists of word processed documents or E-mail. E-mail in particular is a poor choice since it is only available to the recipients and often gets lost. An internal Wiki is a better choice, since most maintain a change history, have centralized storage making the latest version easier to find, support access control, provide auditing and can notify users of changes. Wikis are not perfect, however, since their storage formats are often proprietary and they often lack features of modern word pressing tools.

Wikis also encourage splitting documentation from large, monolithic documents into smaller sections. Care must be taken to ensure consistency and information is easy to find but most Wikis support free text searches. This fits better into the Agile and Lean methodologies, where smaller documents are delivered when needed rather than complete documentation delivered up front.

Writing documentation forces developers to identify and empathize with their customers or other developers, encouraging them to simplify the product’s design, user interface, installation and/or configuration. Some techniques, such as Documentation Drive Development and Readme Driven Development, encourage starting from the documentation for this reason.

It seems that literacy has become a skill in the software development industry. We have fooled ourselves into thinking that developers are only productive if they are writing code and poor documentation is acceptable. As Albert Einstein said “If you can’t explain it simply, you don’t understand it well enough.” Writing documentation forces developers out of their warm, cosy technical silos and back into the user world, reaquainting themselves with the problems the software is trying to solve.

Software Development , , , , , , , ,

%d bloggers like this: