About 18 months or so ago I wrote a post about how I’d seen tests written that were self-reinforcing (“Tautologies in Tests”). The premise was about the use of the same production code to verify the test outcome as that which was supposedly under test. As such any break in the production code would likely not get picked up because the test behaviour would naturally change too.
It’s also possible to see the opposite kind of effect where the test code really becomes the behaviour under test rather than the production code. The use of mocking within tests is a magnet for this kind of situation as a developer mistakenly believes they can save time [1] by writing a more fully featured mock [2] that can be reused across tests. This is a false economy.
Example - Database Querying
I recently saw an example of this in some database access code. The client code (under test) first configured a filter where it calculated an upper and lower bound based on timestamps, e.g.
// non-trivial time based calculations
var minTime = ...
var maxTime = ...
query.Filter[“MinTime”] = minTime;
query.Filter[“MaxTime”] = maxTime;
The client code then executed the query and performed some additional processing on the results which were finally returned.
The test fixture created some test data in the form of a simple list with a couple of items, presumably with one that lies inside the filter and another that lies outside, e.g.
var orders = new[]
{
new Order { ..., Timestamp = “2016-05-12 18:00:00” },
new Order { ..., Timestamp = “2018-05-17 02:15:00” },
};
The mocked out database read method then implemented a proper filter to apply the various criteria to the list of test data, e.g.
{
var result = orders;
if (filter[“MinTime”])
...
if (filter[“MaxTime”])
...
if (filter[...])
...
return result;
}
As you can imagine this starts out quite simple for the first test case but as the production code behaviour gets more complex, so does the mock and the test data. Adding new test data to cater for the new scenarios will likely break the existing tests as they all share a single set and therefore you will need to go back and understand them to ensure the test still exercises the behaviour it used to. Ultimately you’re starting to test whether can actually implement a mock that satisfies all the tests rather than write individual tests which independently validate the expected behaviours.
Shared test data (not just placeholder constants like AnyCustomerId) is rarely a good idea as it’s often not obvious which piece of data is relevant to which test. The moment you start adding comments to annotate the test data you have truly lost sight of the goal. Tests are not just about verifying behaviour either they are a form of documentation too.
Roll Back
If we reconsider the feature under test we can see that there are a few different behaviours that we want to explore:
- Is the filter correctly formed?
- Are the query results correctly post-processed?
Luckily the external dependency (i.e. the mock) provides us with a seam which allows us to directly verify the filter configuration and also to control the results which are returned for post-processing. Consequently rather than having one test that tries to do everything, or a few tests that try and cover both aspect together we can separate them out, perhaps even into separate test fixtures based around the different themes, e.g.
public static class reading_orders
{
[TestFixture]
public class filter_configuration
...
[TestFixture]
public class post_processing
...
}
The first test fixture now focuses on the logic used to build the underlying query filter by asserting the filter state when presented to the database. It then returns, say, an empty result set as we wish to ignore what happens later (by invoking as little code as possible to avoid false positives).
The following example attempts to define what “yesterday” means in terms of filtering:
[Test]
public void filter_for_yesterday_is_midnight_to_midnight()
{
DateTime? minTime = null;
DateTime? maxTime = null;
var mockDatabase = CreateMockDatabase((filter) =>
{
minTime = filter[“MinTime”];
maxTime = filter[“MaxTime”];
});
var reader = new OrderReader(mockDatabase);
var now = new DateTime(2001, 2, 3, 9, 32, 47);
reader.FindYesterdaysOrders(now);
Assert.That(minTime, Is.EqualTo(
new DateTime(2001, 2, 2, 0, 0, 0)));
Assert.That(maxTime, Is.EqualTo(
new DateTime(2001, 2, 3, 0, 0, 0)));
}
As you can hopefully see the mock in this test is only configured to extract the filter state which we then verify later. The mock configuration is done inside the test to make it clear that the only point of interest is the the filter’s eventual state. We don’t even bother capturing the final output as it’s superfluous to this test.
If we had a number of tests to write which all did the same mock configuration we could extract it into a common [SetUp] method, but only if we’ve already grouped the tests into separate fixtures which all focus on exactly the same underlying behaviour. The Single Responsibility Principle applies to the design of tests as much as it does the production code.
One different approach here might be to use the filter object itself as a seam and sense the calls into that instead. Personally I’m very wary of getting too specific about how an outcome is achieved. Way back in 2011 I wrote “Mock To Test the Outcome, Not the Implementation” which showed where this rabbit hole can lead, i.e. to brittle tests that focus too much on the “how” and not enough on the “what”.
Mock Results
With the filtering side taken care of we’re now in a position to look at the post-processing of the results. Once again we only want code and data that is salient to our test and as long as the post-processing is largely independent of the filtering logic we can pass in any inputs we like and focus on the final output instead:
[Test]
public void upgrade_objects_to_latest_schema_version()
{
var anyTime = DateTime.Now;
var mockDatabase = CreateMockDatabase(() =>
{
return new[]
{
new Order { ..., Version = 1, ... },
new Order { ..., Version = 2, ... },
}
});
var reader = new OrderReader(mockDatabase);
var orders = reader.FindYesterdaysOrders(anyTime);
Assert.That(orders.Count, Is.EqualTo(2));
Assert.That(orders.Count(o => o.Version == 3),
Is.EqualTo(2));
}
Our (simplistic) post-processing example here ensures that all re-hydrated objects have been upgraded to the latest schema version. Our test data is specific to verifying that one outcome. If we expect other processing to occur we use different data more suitable to that scenario and only use it in that test. Of course in reality we’ll probably have a set of “builders” that we’ll use across tests to reduce the burden of creating and maintaining test data objects as the data models grow over time.
Refactoring
While reading this post you may have noticed that certain things have been suggested, such as splitting out the tests into separate fixtures. You may have also noticed that I discovered “independence” between the pre and post phases of the method around the dependency being mocked which allows us to simplify our test setup in some cases.
Your reaction to all this may well be to suggest refactoring the method by splitting it into two separate pieces which can then be tested independently. The current method then just becomes a simple composition of the two new pieces. Additionally you might have realised that the simplified test setup probably implies unnecessary coupling between the two pieces of code.
For me those kind of thoughts are the reason why I spend so much effort on trying to write good tests; it’s the essence of Test Driven Design.
[1] My ACCU 2017 talk “A Test of Strength” (shorter version) shows my own misguided attempts to optimise the writing of tests.
[2] There is a place for “heavier” mocks (which I still need to write up) but it’s not in unit tests.
No comments:
Post a Comment