Man Cannot Live by Unit Testing Alone

Way back in May 2012 I wrote a blog post titled “Beware the Complacency Unit Testing Brings”. This was a reaction to a malaise that I began to see developing as the team appeared to rely more heavily on the feedback it was getting from unit tests. This in turn appeared to cause some “trivial” bugs that should also have been picked up early, to be detected somewhat later.

This post looks at a couple of other examples I’ve seen in the past of problems that couldn’t have be solved by unit testing alone.

Unit Tests Are Self-Reinforcing

Myself and a colleague once had a slightly tortuous conversation with a project manager about our team’s approach to testing. There was a “suggestion” that as the organisation began to make more decisions based on the results of the system we had built, the more costly “a mistake” could become. We didn’t know where this was coming from but the undertone had a suggestion about it of “work harder”.

Our response was that if the business was worried about the potential for losses in millions due to a software bug, then they should have no problem funding a few tens of thousands of pounds of hardware to give us the tools we need to automate more testing. To us, if the risks were high, then the investment should be too, as this helps us to ensure we keep the risks down to a minimum. In essence we advocated working smarter, not harder.

His response was that unit tests should be fast and easy to run, and therefore he questioned why we needed any more hardware. What he failed to understand about unit testing was its self-reinforcing nature [1]. Unit tests are about a programmer verifying that the code they wrote works as they intended it to. What it fails to address is that it meets the demands of the customer. In the case of an API “that customer” is possibly just another developer on the same team providing another piece of the same jigsaw puzzle.

As if to prove a point this scenario was beautifully borne out not long after. Two developers working on either side of the same feature (front-end and back-end) both wrote their parts with a full suite of unit tests and pushed to the DEV environment only to discover it didn’t work. It took a non-trivial amount of time of the three of us (the two devs in question and myself) before I happened to notice that the name of the configuration setting which the front-end and back-end were using was slightly different. Each developer had created their own constant for the setting name, but the constant’s value was different and hence the back-end didn’t believe it was ever being provided.

This kind of integration problem is common. And we’re not talking about junior programmers here either, both were smart and very experienced developers. They were also both TDD-ers and it’s easy to see how this kind of problem occurs when your mind-set is focused around the simplest thing that could possibly work. We always look for the mistake in our most recent changes and both of them created the mismatched constant right back at the very beginning, hence it becomes “out of mind” by the time the problem is investigated [2].

Performance Tests

Unit tests are about verifying functional behaviour, so ensuring performance is not in scope at that point. I got a nice reminder of this not long afterwards when I refactored a stored procedure to remove some duplication, only to send performance through the roof. The SQL technique I used was “slightly” less performant (I later discovered) and it added something like another 100 ms to every call to the procedure.

Whilst all the SQL unit tests passed with flying colours in it’s usual timescale [3], when it was deployed into the test environment, the process it was part of nosedived. The extra 100 ms in the 100,000 calls [4] that the process made to the procedure started to add up and a 30 minute task now took over 8 hours!

Once again I was grateful to have “continuous” deployments to a DEV environment where this showed up right away so that I could easily diagnose and fix it. This just echoes what I wrote about recently in “Poor Performance of log4net Context Properties”.

A Balance

The current backlash against end-to-end testing is well justified as there are more efficient approaches you can take. But we must remember that unit testing is no panacea either. Last year we had these two competing views going head-to-head with each other: Why Most Unit Testing is Waste and Just Say No to More End-to-End Tests. It’s hard to know what to do.

As always the truth probably lies somewhere in between, and shifts either way depending on the kind of product, people and architecture you’re dealing with. The testing pyramid gets trotted out as the modern ideal but personally I’m still not convinced about how steep the sides of it should be for a monolith versus a micro-service, or a thick client versus a web API.

What I do know is that I find value in all different sorts of tests. One size never fits all.

[1] This is one of the things that pair and mob programming tackles because many eyes help make many kinds of mistakes less common.

[2] Yes, I could also go on about better collaboration and working outside in from a failing system test, but this didn’t deserve any massive post mortem.

[3] Database unit tests aren’t exactly speedy anyway so they increased the entire test suite time by an amount of time that could easily have been passed off as noise.

[4] Why was this a sequential operation? Let’s not go there...

Tribalism or Marketing?

I had a brief conversation with Paulmichael Blassuci (@pblasucci) on Twitter after someone re-tweeted the following from him:

“Open Q: what will cause #SoftDev, as an industry, to stop thinking in tribal terms? (e.g. "scala dev", "linux guru", "sql server admin")”

Naturally I wanted to understand what it was he was observing, as in my experience this wasn’t the case, or perhaps I just didn’t understand what he was getting at. I’m not going to pretend that I have any significant grasp of psychology, philosophy, sociology or any (social) science for that matter, and so I expected this to quickly go way over my head. I probably only persevered because I’ve finally got around reading Gerry Weinberg’s seminal classic “The Psychology of Computer Programming” which he wrote way back in the 1970’s and has been on my reading list for far too long.

Twitter is not the best medium for holding any sort of proper conversation but I got enough out of it to start me thinking about how I describe myself, such as in the profiles I have on various sites like StackOverflow, my blog, Linked-In, etc. In all those cases I seem to do exactly what @pblasucci was observing – I appear to be pigeonholing myself with a specific subset of the programming community. But I too wondered why that was, as I don’t consciously try to feel closer to one group or another, although it’s possible I might have unconsciously chosen to try and distance myself from certain other groups.

What’s in a Name?

The debate about how to describe ourselves professionally seems to be never ending. Are we programmers, computer scientists, software developers, software engineers, solution architects, etc? At the recent “bake-off” I wrote about in “Choosing a Supplier – The Hackathon” the various teams all introduced themselves as software engineers and solution architects. When it came to our team we all just described ourselves simply as “a dev”.

When people outside the industry ask me what I do for a living I find it easiest to describe myself simply as “a programmer”. There is little point in being any more specific with them about which “area” I tend to work in unless they try and dig deeper. The person on the street probably hasn’t got the foggiest idea about the differences between developing video games, mobile apps, web sites, back office tools, etc. And given that a large part of my working life has been spent on financial middle-office and back-office type systems I don’t really even have a customer facing product that I could tangentially associate myself with [1].

My father-in-law wrote software back in the days when “programmers” were seen as just code monkeys that turned the analyst’s carefully worked out flowchart into computer code. When they started playing that role too they became known as analyst/programmers. Hence to avoid confusion (I was in my first job doing the whole lot: analysis, design, implementation, test, deployment, etc.) I said I was a “software engineer”. When I slip up on Twitter and say that I’m a “programmer” he still likes to remind me of my place in the software development hierarchy :o).

Going Freelance

To other people inside the industry I think I stopped being just “a” programmer and started qualifying it a bit more when I went freelance (i.e. contracting). At this point I stopped being interested in “the company” per-se or career progression and just wanted to get paid a decent wage for doing what I loved – programming.

As a contractor you are essentially seen as a mercenary (which is where the term “free-lance” comes from). You are primarily hired for your expertise in a particular language or technology and when the project using that finishes, so do you. You and the company part ways and move on to the next gig. Only, we all know software projects often have a lifetime somewhat longer than this simplified view suggests.

Perhaps in an ideal world we’d all just be “journeyman programmers” and would pick up whatever extra skills we needed on the job, whether they be big or small. This happens for permanent employees, but occasionally even for freelancers too. For instance, my introduction to the world of C# came about because the technology stack of the project I interviewed for switched from C++ to C# right at the last moment. Even though I was after a contract position they still asked me if I wanted to pick up C# and .Net on-the-fly. They were more interested in the experience I brought to the project and clearly thought I’d have no trouble learning the language and framework relatively quickly.

By-and-large though this doesn’t happen because the expectation is that you’re hiring a temporary worker that can “hit the ground running”. This means you’re already expected to be well versed in the primary language and toolchain (the so called “must haves”), but may not have much knowledge of the ancillary tooling. Knowledge of the problem domain is whole other ball game as that can often be traded off against someone’s technical abilities.

Marketing

And so whereas the programmer in the world of permanent employment (where the employer is happy to invest time and money in their education) probably thinks of themselves as more of a generalist, the independent programmer has less of a luxury and cannot. We have to perform a degree of marketing to ensure that we don’t get continually glossed over by the recruiters every time we switch contracts. Their searches (and those of the hiring organisation’s HR department) are often (at least initially) driven by simple keyword matches. These days the client can easily look you up too on the internet and, if you have a presence, it really helps if you fit the pigeon-hole they’re looking to fill. This is invariably described by the main programming language you have the most recent experience of [2].

This all becomes so much easier when you get older and have far more experience under your belt. When the mechanics of programming start falling into the realms of Unconscious Competence it’s a lot easier to focus on the problem you actually need to solve. Hence it’s easier for a client to take a punt on someone less well versed in one particular toolchain if they have have other experiences they can readily apply. But first you have to get passed the HR wall to even be considered for that, if you don’t have a way in via a recommendation.

More Than a Language

The need to define ourselves through the programming language with which we have the most recent experience seems a little simplistic, after all it’s just a language, right? Not really. A programming language probably always has, and still is, linked heavily with the entire toolchain to write, build, test and deploy the software. Compare the Java, Eclipse, Maven, JUnit toolset to its C#, Visual Studio, NUnit counterpart. The front-end world of HTML, CSS, JavaScript, Node, is another. An average programmer just doesn’t move from one world to another without taking some time to pick things up, and a large part of what needs picking up are the libraries that support each culture. As the ports of xUnit to CppUnit and JUnit to NUnit show, getting-by in a language is not the same as writing code idiomatically. That said the problem is probably eased somewhat these days as languages cluster around a “VM”, for example the JVM has Java, Scala, Groovy, etc. whereas .Net has C#, F#, PowerShell, etc. which reduces the impedance mismatches somewhat.

Conclusion

Hence I guess the outcome of this thought exercise is that I see the world through the eyes of the recruitment process. My life as a run-of-the-mill freelance programmer means that I generally describe myself based on the major traits that I expect clients will look for. Although I’ve dabbled in the likes of Python, Ruby, D, F#, Go, etc. over the years, I would never expect to be hired (as a contractor) to work on a production codebase using one of these languages as my skills are too weak [3]. Learning them has brought the benefits of a multi-paradigm education (other ways of seeing the world and solving problems) which ultimately makes me more marketable as a programmer. But this is still within the confines of my most defining skills – the ones that help me put (nice) food on the table.

Maybe though this is exactly what he meant by tribalism.

[1] I started out working on PC shrink-wrapped graphics applications, but that was a long time ago. More recently since moving away from finance I now have something more tangible once again to point the kids too (although it’s still far from being “rock-and-roll”).

[2] Interestingly my own GitHub repo has over a decade of C++ code in it but nothing for C# and .Net which has been my primary professional language for the last 5 years.

[3] Even so they can often be found in the tooling used within the build pipeline, system administration, analysis, etc. on the projects I find myself involved in.