Application of Code Coverage to Model Based Testing

There’s something I want to get out of my system before I move on with the more meaty posts, and that is applying code coverage to model based testing.

In general code coverage must be the most abused quality metric in software testing. Take for example the often heard statement that once you hit 100% code coverage your system is completely tested. Of course from a coding perspective, yes – you have hit all code (and nothing exploded). But what does that mean from a testing perspective? You can have 100% code coverage, but it does not guarantee that the code is doing what you expected. Without verification on every line you cannot guarantee there are no bugs. Code coverage metrics often establishes a false sense of security.

Then, is code coverage useless? – no, actually it’s very useful, when it’s used correctly. Tying back to my original post on testing as risk minimization, code coverage will hint you at risks. If you have a class with zero code coverage, this means you have no tests exercising the class, and thus cannot say anything about its quality level, thus you have a gaping hole in your testing, which is to be considered a risk. Thus priority should be put on writing tests targeting this class. Code coverage can be utilized in this fashion to direct your effort to the most risky portions of the code.

The same holds true for model based testing. Obtaining high code coverage does not guarantee that your model is good – using the same argument, you do not know if the covered code is actually verified. However, if you obtain very low code coverage of the targeted part of the system under test, it is an indication that the model is disconnected from the actual system under test.

However, from my experience you should not expect to gain more than 40-60% coverage from a good model. The point here is that models are simplifications of the system under test and to obtain higher coverage percentage, you need to exercise corner cases of the system – usually this requires more specific test cases (e.g. boundary testing), and including these cases in the model is often more work than writing them stand alone.

From a code coverage perspective model based testing has the down side that the code coverage density of the tests is very low. By that I mean, that each test contributes very little to additional coverage. The reason for this is that model based testing performs combinatorial testing on the inputs, which results in many tests exercising the same paths, with only slight divergence between them. That means tests share a lot of code coverage. However, as you will no longer use code coverage as a performance metric for your test, this should not be a problem.