Contract Checks: Once and Only Once for Unit Testing (Part II)

In Part I of this post, I pointed out a problem with xUnit tests: it is difficult to obey Once and Only Once when the validation assertions are outside the method(s) being tested.  The solution, as I mentioned in my preceding post about contract checks, is to move as many assertions as make sense into contract checks. This has many benefits, most of which address the principles Stef was looking for (and the frustrations she encountered in applying them to testing Spatial's 3D InterOp product) in her post:

1. Once and Only Once – by turning validation code outside a method into contract code within the method, the validation code is only written once (inside the method). Every test which calls the method will pass through the validation code; there is no need for complicated architecture in the test system to avoid duplication.
2. Configuration Coverage – contract checks are performed whenever the method they check is called, including acceptance/system tests and runs of the actual application (if desired). In other words, they’re run as part of Stef’s “monolithic, mysterious test suite” (MMTS). In contrast, validation code is only called in the test(s) in which it is explicitly included by the test author(s). This means that contract checks are applied to many more data configurations than validation code, which in turn yields a greater return on the investment of writing the contract check.
3. (No Need For) Cleverness – one of the difficulties in designing good tests is thinking of weird corner cases of the input data; Stef’s experience was that no-one is clever enough to think of the full weirdness of realistic customer data. Because contract checks are run whenever the method is called, they can discover subtle bugs on data configurations that the developer wasn’t clever enough to think of (or energetic enough to set up). This is just a different way of saying that contract checks can take advantage of a MMTS.
4. Leveraging Bugfixes – Most companies have a set of tests to ensure that fixes for customer-reported bugs don’t regress. The good news is that these test the code on realistic customer data. The bad news is that each test covers only the particular data configuration which failed. If these tests were instead formulated as contract checks (by inserting a failing contract which succeeds afterthe bugfix), then the new contract checks will be run on other data configurations in other tests. This leads to an incremental process where each fixed bug contributes non-linearly to the overall robustness of the product, both by adding a contract test to the code and by adding another data configuration upon which many contract checks will be executed.
5. Defect Localization – one of the primary arguments for making unit tests small and simple is so that it is easy to isolate the functionality which is broken when the tests break. With contract checks, this is a non-issue – the debugger takes the developer to the method containing the first “test” failure. Stef talks about this point in the context of expressive tests that only test one thing.
6. Encapsulation – if you think about it, the argument “unit tests should be small, so that it’s easy to isolate which method is broken” is a subtle code smell indicating that putting the validation code outside the methods is a violation of encapsulation (which contract checks solve). Contract checks also avoid the temptation to make private data public so that the test code can get at it; as part of the method, the contract check has access to private data.

This is why I got so excited when I read Stef’s post – the frustrations she expressed lined up with my thoughts about the benefits of including contract checks as part of a unit (or system) testing strategy.

Before I go, I want to make one thing clear: I am not saying that xUnit is the source of the problems I’ve discussed above; it’s simply a very useful tool that makes it easy to write and run automated tests. Nor am I saying that contract checks are a silver bullet that will remove all complexity from the test code so that the techniques in Meszaros’ book will be unnecessary. What I am saying is that contract checking is an essential part of any testing strategy, and that developers who don’t use a contract checking methodology run the risk of running into many of the anti-patterns that Meszaros points out (and addresses) throughout his book.