No idea how I found myself on this blog; however, it was an interesting read: Why Is 100% Test Coverage Easier To Achieve?
Although the translation is a little difficult to read he does an excellent job of pointing out the obvious benefits to 100% functional coverage. Let’s recap his main points…
1. First he mentions the obvious transition from “do we really need to test it” to the eventual “did we really need to write it”. By requiring 100% functional coverage on a code base you eliminate waste. Code that is not accessible via public interfaces is usually one of the first things that disappears. This pruning of dead-code can be very beneficial as the project ages.
2. The second benefit he discusses is that it reduces the likelihood for coverage to be reduced over time due to release pressure. For instance if it’s acceptable to have 20% of your code untouched by automated testing, then why is 21% so bad? or 23%? or 25%? or 35%? Release after release you lower your bar to meet expectations.
While these are certainly some of the key benefits, there are other benefits I enjoy by continuing to release the library at 100% functional coverage. So here are the things I would add to his original two points:
3. Over time it makes developers more conscious about over engineering a solution. Since they know that they will be required to test every method, they tend to keep their designs simple yet sufficient.
4. Rewrite anything, delete anything. One key problem that exists in partially covered code is that refactoring a low-level component can be dicey. Maybe all the pieces that depend on you have adequate coverage, and maybe they don’t. Making every method important to test makes it less possible that a re-factored piece of code doesn’t introduce bugs.
5. Totally saves time. If you have read this blog, you know I’ve writing a B+Tree. Among many things it relies on are storage layers and locking factories. By building and testing these in isolation first before integrating them with the intended use-case, I save tons and tons of time. A subtle bug in a lock would play havoc on multi-threaded code, how much time would it take you to figure that out the root cause of a concurrency issue was a race condition in a lock? This is why testing is important to me, I hate wasting my time.
6. Stuff just works… It seems like every time I take a reasonably complicated class from cursory usage coverage to 100% functional coverage I find something broken. When given an average coverage of 60-80% functional, this translates to only around half or 50% statement coverage. I don’t know about you but I can fit a whole lot of bugs in just half the code I write. Now the inverse, when you forcibly hit 100% functional your wind up with somewhere between 90~95% statement coverage.
For me and this library, most of this remaining uncovered code looks something like “throw new XxxException()” or “return -1″ etc. In fact, out of the 14,328 statements in the library only 710 are not touched by a unit test. Of those 710 lines there are only 263 after you throw out the set that contains ‘throw’, ‘catch’, ‘return’, ‘Trace’ and ‘Log’. Even reviewing manually the remaining 263 demonstrates that the majority of that is also error handling. So what’s the point? The point is that pushing to and keeping 100% functional coverage helps ensure that positive test cases are in place.
Disclaimer:
Does 100% functional coverage mean anything? No. Clearly, and we all know this to be true, making a function execute is not the same thing as testing it. Simply having a 100% functional coverage is meaningless without the discipline and maturity of all developers involved. If they are not 100% excited about 100% functional coverage then that goal will fail… one way or another. Even with disciplined developers bugs will still exist, that unfortunately doesn’t change.
However, if your whole team is excited about and serious about achieving 100% functional coverage it should become a check-in requirement. Just like you use Cruise Control to make sure every check-in builds and deploys, make certain your build fails if not at 100%. Remember it’s also important to find/write the tools up front to make locating the missing coverage quick and easy.