Beware the Complacency Unit Testing Brings

One phrase that is always sure to raise the ire of any good honest developer when something breaks is:-

“well, it works on my machine”

This simple statement shows a complete disregard for any other sort of testing that you might need to do to ensure that your feature works correctly and is “done, done” not just almost done. But there is a new kid on the block when it comes to showing how little some people understand about software development that I’m beginning to hear with alarming regularity:-

“well, all the unit tests passed”

It seems that modern development practices have unknowingly created the Silver Bullet that Fred Brooks has always told us never existed! Apparently good unit test coverage and automated refactoring tools means that it’s highly unlikely that any bug would only show up during integration and system testing that those ideas are just old fashioned. Or, if not altogether outdated, then reduced to just a footnote in the product’s testing strategy on the basis that there is so much less value in them than unit testing.

Don’t get me wrong I can understand a genuine mistake caused by a seemingly unrelated change - accidents happen and it could be a fault of the design - but changing the configuration file for a service and then not even bothering to see if it starts up is just laziness. Yes, it does take time and effort to do more extensive testing in your sandbox but the feedback loop could still be fast and you won’t annoy your team mates when you cost them a day’s system testing because of a silly mistake.

The rule of thumb about not checking in code until the unit tests pass was designed to make you think about writing fast tests so that the barriers to testing are as low as possible, it was not expected to be used as a justification for short-circuiting the amount of testing you do.

My “Agile SQL Tweet” Slide

Before I put the PowerPoint slides for my ACCU conference talk on Database Development Using TDD up on my web site I agonised over whether to remove the 3rd slide or not. The slide in question just contains the following tweet from Allan Kelly:-

@chrisoldwood is the only person I know with a convincing Agile SQL story -- @allankelly

I felt that viewing the slide in isolation would only amplify the arrogance which could be attached to it and that wouldn’t have a positive effect. But at the same time I felt I should put up what was shown and accept that perhaps it may also raise the same questions that I had that caused me to put it in the first place.

The slide forms part of the prologue and was originally just a picture of myself to act as a backdrop whilst I spent a couple of minutes describing who I was and what I did as I’m not a database developer per-se, I’m mostly a C#/C++ service layer guy who has slipped into the SQL arena like many similar developers. As a consequence I have tried to work with the RDBMS in the same way that I would approach anything else, by using TDD and abstracting the client from the underlying implementation. So, rather than couple the client and database code tightly with client-side SQL[*] I have kept them decoupled to allow the database to develop (and therefore be refactored) independently where possible - I don’t expose public fields in my types[#] so why directly expose tables?

And then this tweet from Allan Kelly appeared. Until that moment I suspected I was the proverbial square peg trying desperately to fit into a round hole, but maybe, just maybe, I might not be barking mad after all. There are many benefits to TDD (and unit testing in particular) over and above proving the functional correctness of a piece of code, but I don’t understand why more people can’t see that.

So the slide should hopefully provoke a reaction that forces you to question what your database development tools and processes are and whether you’re getting the most out of them. If you are then great, it’s working for you, but if you have that uneasy feeling that you’re stymied at every turn because your database can’t evolve quickly enough then perhaps what you’re about to see (and hear, if you attended) will provide food-for-thought about what you could change.

Prior to this the most ‘positive’ reaction I’ve had to what I’ve been suggesting is:-

“I suppose it makes sense, I’ve just never seen it done like that before”

[*] I’m not just talking about explicit embedded SQL here where the SQL code is a string literal (or as a format string) but also where it is implicit as a by-product of a technology such as LINQ-to-SQL. One way or another the schema is likely to be too tightly coupled to the calling code and that means your database design will be harder to change as will remediating performance problems.

[#] Some would argue that a DTO (Data Transfer Object) does exactly that, but a modern RDBMS is so much more than just a data persistence mechanism. Use its power where it makes sense to.

My ACCU Conference Session - Database Development Using TDD

It’s been a busy few weeks as I’ve been preparing to present at the 2012 ACCU Conference in Oxford. This has become somewhat of a pilgrimage and for the second year running I’ve been accepted as one of the speakers; this time to talk about using TDD to develop databases. This was an excuse to piece together some of the ideas that I’ve been blogging about along with the experiences gained with my current client to present one way that can use tests to drive both the development and design of a database.

Session Details

The talk started by walking you through the pre-requisites needed to embrace TDD, such as your own development sandbox and looked at the primary testing mechanism[*], i.e. unit testing SQL. I also spent a fair bit of time covering the notion of a database’s public interface as I personally feel that the modern RDBMS gives you the tools to build an abstraction layer to help decouple the client from the data model to grant freedom of implementation[#].

Naturally I explained the principles of TDD and did a 20 minute live coding session to show how I would use them to develop a reporting style stored procedure. For this I used my own SS-Unit testing framework and SQL Server Express; switching rapidly between test code & production code. With the main body of the talk in the bag I turned to the opportunities that TDD (although really unit testing) opens up such as continuous integration, continuous deployment and schema refactoring.

The PowerPoint slides are available on my web site here.

Abstract

Just for the record this was the abstract that I submitted...

The modern day RDBMS is a complex product that offers so much more than just data persistence. The SQL language, with its vendor specific variants such as T-SQL, provides the ability to develop code in various forms to read, transform & write that data efficiently. This code requires constant testing right from its inception through its various incarnations until it is finally retired.

TDD is a technique that puts writing those tests at the front of the development process, whether that be because you’re writing new code or changing existing code. The knock-on effect of this approach is that your client-based perspective opens your eyes to potential variations in the implementation, and that is where the second ‘D’ in TDD turns from Development into Design. With a solid automated test suite and Continuous Integration under your belt too the world of refactoring opens itself up so that your database design can safely evolve.

This session looks at applying the same principles and disciplines used in other areas of system development to tame the ever increasing complexity that has arisen from the maturity of the RDBMS.

Photo

The above photo of yours truly comes courtesy of Mark Ridgwell. Here are the rest of his photos for Wednesday.

[*] TDD does not prescribe any particular type of testing but I believe that most people associate TDD with unit testing as it probably forms the lions share. I did point out that one of the benefits of a top-design and implementation is that you can start development with a system test and drill down from there.

[#] ORM tools like Entity Framework actually push you in the other direction which may be exactly what you want. But as I’ve said before, Enterprise culture has a habit of routing around nicely designed service layers and attempting to go straight to the heart of the data.

OK & Apply - A Failed UI Experiment?

As you may have guessed from my previous two posts, I have been paying far more attention than usual to how other IT professionals[*] work. The most recent one that caught my eye and has no doubt been done to death in UI circles is the OK and Apply buttons on dialogs. It’s funny to watch people first push the Apply button and then the OK one. The question is whether they don’t realise that pressing OK has addition affect of applying the changes, or that they don’t trust that just pressing OK is enough? I’m sure it’s the latter - a belt and braces approach.

I know how they are supposed to operate, but I suspect that’s only because I was doing a lot of UI work when Windows 95[#] appeared and so I was paid to know all the UX guidelines. Perhaps it’s time to take a trip down memory lane and visit the “Interface Hall of Shame” along with a quick Google search to find out if/when this UI faux pas was officially deprecated...

 

[*] I’m always looking to learn new tricks and techniques to be more effective and other IT professionals is obviously as good a place as any to look as this is (supposedly) their bread-and-butter.

[#] and Windows NT 3.5 with the shell upgraded to 4.0.

Copy-and-Paste Can Help Avoid Mistakes Too

In the software development world the term “copy-and-paste” is synonymous with bad practice. Web sites and books are filled with advice about how you should avoid duplication in code and maintain a Single Point of Truth (SPOT) or Don’t Repeat Yourself (DRY)[*]. One could almost be forgiven for ripping the X,C, & V keys out of the keyboard for fear of succumbing to such a sinful practice. And yet I do it all the time, way more than most of my colleagues it seems. Of course I’m not talking about duplicating code though, that would clearly be wrong...

Some months back we had a bug which caused quite a bit of head scratching as it was a subtle spelling mistake in a .cmd script (%valueData% instead of %valueDate%). Watching the developer at work intrigued me because where I would naturally have copy-and-pasted the variable name from a few lines below, he typed it out by hand. Just yesterday a colleague was surprised when they couldn’t see the files they were expecting in a folder and it turned out they had mistyped the folder path. What I found amusing was that the folder name was readily available in a text editor on the other screen (which itself was a copy of the wiki page I use for the same task) and so a quick copy-and-paste would have avoided the error.

The way I write wiki pages and release plans is very much with the copy-and-paste approach in mind. The textual description is like a comment in the code, there to advise the newbie and provide links to the supporting rationale and concepts. But what the experienced[#] developer wants is just a checklist of items, or perhaps a template that they can copy-and-paste onto the command line, into their text editor, run with their SQL tool, etc. Perhaps it’s just me, but once I step outside the woolly realm of writing English prose and into the precise world of programming I immediately look for that safety net that will help me avoid losing time due to “simple” mistakes, no matter how small. In some cases it may only save a minute or two, such as learning on the compiler, but in others it could be hours if the outcome is to bork system testing which then has a knock-on effect onto my team-mates.

Of course my way is far from perfect too and it is fraught with a different set of problems instead. Have you ever copy-and-pasted a command line from a Word document and found it doesn’t work? Did you then waste time, grumble constantly and gnash your teeth only to discover that Word had “intelligently” replaced a pair of en-dashes (--) with a single em-dash (—), or worse converted them from their ASCII to Unicode counterparts? They call it “smart formatting” but in this instance it’s anything but smart! Maybe that’s the reason why Microsoft eschews the “-“ command line switch convention in favour of “/”?

The second common mistake I run into is with copy-and-pasting from the very wiki pages that are designed to save people from making mistakes. Web browsers[+] like to be intelligent when you copy a block of text (such as an example command line) so that if you select an entire line they will “helpfully” terminate it with a newline. This has the wonderful effect of executing the command right after pasting it into a console window! No matter how slowly or carefully you select the text, the moment you reach that last character the selection box automagically expands to include the nauseating newline.

In the end mistakes are always going to happen, so choose your poison...

 

[*] Matthew Wilson even likes to combine the two and call such artefacts a “DRY SPOT” which creates a wonderful metaphor.

[#] “Newbie” and “experienced” in this context relate to their knowledge of the system, not how long they’ve been in the profession.

[+] No doubt this will incite the usual tirade of “why are you using a stupid browser like IE, get a ‘real’ browser!” comments. I’ve answered that before.

Don’t Let Your Tools Pwn You

Modern development tools such as IDEs and their associated multitude of plug-ins, such as ReSharper, can definitely improve productivity. But they can also reduce it too if welded without due care. In some cases it may not be yourself that suffers the loss in productivity but your team-mates because you were unaware of what that tool actually did.

The original impetus for this post was actually watching someone dig themselves into a right mess with ReSharper. I suspect they had got so used to accepting every pop-up that ReSharper threw at them that they didn’t notice when ReSharper had suggested the wrong change. The code didn’t compile now but it wasn’t obvious why as the wrong turn was some minutes ago and they were focused on the code they were writing now. After all, the previous code did compile so why could the mistake be back there?

Being “old school” I used the compiler output to point the finger rather than ReSharper’s advice and traced it back to the source. The earlier mistake was to mistype the inheritance of a class and then let ReSharper implement it automatically. This mistake initially swapped compile time errors for runtime errors (throw NotImplementedException is the default implementation) and unless you use a test-first approach the feedback loop between making and discovering the mistake grows rapidly.

The more recent impetus is another tale, once again involving ReSharper, that caused another person pain - me. As part of a large refactoring exercise one of my team mates had to split some core classes off into a common assembly for use by the new client application. They chose to use ReSharper for the job which makes perfect sense as refactoring is one of its main features. Sadly they didn’t know what ReSharper was also going to do at the same time...

These changes all happened in the background on the trunk and it wasn’t until I came to merge some last minute changes from our release branch that things started to get dicey. I was getting a lot of merge conflicts on the changes even though I knew that no one would have touched the code[*]. Well, I knew that the namespace at the top of the file would have changed as a consequence of the other person’s changes, but definitely no more. After looking through the Subversion log I could see that some whitespace has also changed and my gut reaction was that someone had also “fixed the formatting” at the same time as making another change. I didn’t spot this myself at first because the diff tool I originally used had “ignore whitespace changes” still set to ON after an earlier unrelated comparison! Unfortunately the piecemeal way the refactoring had been done also meant there was more noise in the SVN log[+] than a single commit.

Ultimately it took me an hour and half to convince myself that the merge was sound when it should have been trivial. I didn’t notice at the time but the unexpected whitespace changes had also destroyed the careful formatting of a table that needed to be kept in sync with a similar block in some SQL code[#]. This cost me another 30 mins or so to sort out. In retrospect I would have been better off reverting all his check-ins first, fixing the build and then merging mine in after. But that’s part of the problem with finding unexpected changes - you then go looking for the proverbial needle in the haystack that you’re convinced must exist.

My natural reaction to this second issue is that the original refactoring changes should have been diff’ed before check-in (as any change should) and then the problem would have been spotted and corrected at its source. The argument against doing that would no doubt be that they don’t have the time to diff 100 or so files and anyway the change was “automatic”. The question now is whether it would have taken that person more than 2 hours (the disruption caused to me, let alone others) to diff those files and spot that the tool was doing something unexpected? I sincerely doubt it, even doing it manually. And you could probably automate it to produce a unified diff that would clearly have shown up lots of unintended changes.

OK, so that’s only two issues, hardly enough to suggest we give up our modern tools and return to the stone age. And I’m not suggesting that. What I am suggesting is that you’re careful not to get drawn into relying on the tools so much that you cannot remember how do things “long hand”. Although I mainly use Visual Studio with ReSharper for day-to-day C# development I still like to use tools even as primitive as Notepad because they are fast, will spot issues other tools hide[$] and most importantly are even available on production hardware.

 

[*] It was the kind of legacy code you really don’t want to touch unless you absolutely have to. And if you do you want tests. But you can’t.

[+] Given the limitations of Subversion is was probably the right way to have done it as its Copy+Delete approach to moving and renaming files can get you into hot water, especially when mixed with branching.

[#] Yes, in an ideal world these two bodies of code would be generated from a single source, but life is never that easy in reality.

[$] The best way to spot a file that has a mixture of tabs and spaces is to open it in Notepad. Has anyone ever chosen 8 spaces-per-tab out of choice?

Diagnostic & Support User Interfaces

When people talk about user interfaces they are invariably talking about the face of a product or system that the end-users see and/or interact with. But within a complex system there are other user interfaces too that are not seen or interacted with by end-users at all; instead it is fellow developers and, more importantly, support staff that get to see this less salubrious side.

In a small well-established team of developers there will naturally be a lot of cross fertilisation meaning that they will be well versed in significant portions of the system. In contrast, a larger team will have skills spread across different disciplines and so from a support perspective it starts to approach the knowledge levels of a support team - far less. In my experience support teams do not get the luxury of devoting themselves to learning one system instead out; instead they invariably have to support a whole bunch of in-house systems written by different teams using different technologies and having different ideas about what is required to make a product or system “supportable”.

So what kinds of interfaces am I referring too? Well, the most traditional are status monitoring pages and custom admin tools; the kind of utilities built specifically for the purpose of administering and supporting the system. Due to the time constraints imposed and the lack of direct business value they are usually not given any real TLC but are just thrown together, presumably under the misguided assumption that it will never really be used except by them or their close colleagues. But these scripts and command line tools are not the only faces of the system you’ll see when doing support duties; before that you’ll probably be faced with hunting through the custom format log files, which may or may not be easily consumable. Stretching the analogy somewhat further you could encompass configuration files and the database schema as interfaces of sorts. They’re not graphical in themselves, but careful though about their structure can ensure that the off-the-shelf tools stand more chance of making interaction a more productive experience.

Anyway, here is my list of things to think about. I’ve not done any “production” UI programming for a few years and it’s been even longer since I read About Face (the 1st edition no less) so I’ll no doubt contradict some modern wisdom along the way. Your target audience may not be novice computer users, but they do have the clock against them and they won’t thank you for making an already difficult situation any harder.

Command-Line Tools

Server-side developers aren’t renowned for their user interface programming. After all, that’s why you have UX designers and developers whose major skills lie in the area of UI programming. But that doesn’t mean we have be to sloppy about the interfaces for the scripts and tools we do create. I’m not suggesting for a moment that we spend hours gold-plating our utilities, but I am proposing that you adhere to some of the common conventions that have been established over the years:-

Provide a help switch (-?, -h, --help) so that the user can see what the options are. Documentation, if you can even find it for a utility, can easily get out of step with the tool usage and so you need to be able to see which switch it is that you’re apparently missing, or that is no longer relevant for the task at hand. Even if you’re au fait with the tool you might just want to double-check its usage before going ahead and wiping out last years data.

Provide a version switch (-v, --version) so the user can check its compatibility, or if it’s become out of date. Many tools are installed on the application servers or alongside client binaries, but sometimes those tools are just “passed around” and then become available on some desktop share without you realising. You can even dump the version out on every invocation as the first line; this can be particularly useful if you’re capturing stdout when running it as a scheduled task.

Provide long switch names (--switch-name) by default and use them in documentation so it is obvious what the example command is doing. Short switch names are for advanced users and so should be considered the optional case. Positional parameters may save the user some typing but when you use concepts like integer based surrogate keys it’s not exactly obvious what an example may be suggesting:-

C:\> DC 1234 2001-02-03 1

compared to:-

C:\> CustomerAdmin delete --customer-id 1234
  --date 2001-02-03 --force

Use a consistent switch format. I’ve had to use utilities where one developer has made the switch names case-sensitive and another case-insensitive. Sometimes they use ‘/’ as the switch marker and other times it’s a ‘-‘. Often the case-sensitivity issue is down to lazy programming - using == or Equals() instead of the correct method. If you’re on Windows try and support both styles as even Microsoft’s developers aren’t consistent. Better yet download a 3rd party library that lets you use either style as there are plenty to choose from[#].

Return a meaningful result code. The convention (on Windows) is that zero means success and non-zero means failure or partial success. Some tools do provide more detailed result codes, but fundamentally you want assume that anyone trying to automate your tool on Windows is going to write something along these lines as a first-order approximation:-

CustomerAdmin.exe --do-something-funky
if errorlevel 1 exit /b 1

Sadly, 0 is also the default result code that most frameworks will return for an unhandled exception which really makes the whole process rather messy and means that you need to get a Big Outer Try Block in place in main() as part of your bootstrap code.

Support a read-only/test/what-if mode. Most of my support tools have grown out of test harnesses and other development tools and so I’ve naturally added switches that allow you to run them against production in a benign mode so that you can see what would happen. PowerShell has taken this idea forward formally with its common -WhatIf switch. When you’re not 100% sure of what the expected action is or how many items it will act upon this can be a life saver.

Most of the same comments apply to scripts, but they have the added advantage that they are deployed in source code form and so you have the ability to see the logic first hand. Of course when you’re in the thick of it you need to be able to trust the tool to do what it says it will.

Logging

When faced with a production issue, log files will often become the first port of call after you’ve established which process it is that has failed. But log files[+] often serve two different masters and the latter one can get forgotten once the development work has finished. During development it is probably the developer that spends the most time pouring over the output and therefore they have a habit of dumping every last bit of detail out. After the feature is complete though the main consumer then becomes the support team. What they don’t want to see is reams of output so that the proverbial wood is lost behind the trees.

Make sure your log messages make sense. This may seem obvious but spelling mistakes, whilst cute in a time of stability, are a distraction when you’re up to your eyes in it. Log messages should form a narrative that lead you up to the moment of interest, and that narrative should be written with the reader in mind - mostly likely a support engineer - someone who doesn’t know the code intimately. By all means be succinct, but ensure you’re not being ambiguous either.

Use the console output for high-level commentary, warnings and errors only. If extra output is required it should be enabled by an extra --verbose switch (which can easily be added temporarily under automation) or by having a companion log file with more detail. I tend to favour the second approach so that the detail is always there if needed but so that it’s not in your face the majority of the time. Remember we’re talking about in-house stuff here and so you have the luxury of controlling the environment in ways a 3rd party can often only dream of.

Keep to a low number of severity levels. If you have a large number of severity levels developers will not be able to log messages consistently. I recently saw a tweet that suggested you use only the following levels - FYI, WFT & OMG. If you put yourself in the support person’s shoes that’s actually a pretty good scale. Once the issue is triaged and the development team are involved the TRACE or DEBUG level output suddenly becomes of more use, but until then keep it simple.

Write the messages as text. A simple lined-based text format may seem very old-fashioned, but there are a plethora of tools out there designed to slice-and-dice text based output any way you want, e.g. grep, awk & sed. Plus you have modern GUI based Tail programs that support highlighting to really make those important messages shine, e.g. BareTail & LogExpert. Layer on top something like LogParser which supports a SQL like syntax and support for many formats out of the box and you’re in good shape to triage quickly. Also don’t underestimate the importance of NotePad as a tool of last resort.

Make the log message format simple. If you’re writing a separate log file include at least the date & time, and preferably the process ID (PID). If the tool is multi-threaded then the thread ID (TID) will be invaluable as well, along with the highest precision[*] of time (e.g. milliseconds or better) to spot potential race conditions. I’ve found the following format to be very easy to scan by eye and parse by tool as the left-hand side is of fixed width (the PID & TID are padded appropriately):-

<ISO date & time> <PID> <TID> <severity> <message...>

2001-01-01 01:02:03.456 1234 3456 INF Starting...
2001-01-01 01:02:04.123 1234 5678 ERR Stuff broke...

Enclose string values in some form of quotes or braces to make empty strings more visible. As I posted very recently, empty strings are a form of Null Object and so they have a habit of being silently propagated around before finally causing odd behaviour. Personally I like to enclose strings in single quotes because I often have to use those values in SQL queries and so then it’s already quoted for me. As an example I once spent far longer than necessary investigating an issue because the log message said this:-

Valuing trade ...

instead of:-

Valuing trade ''...

Include all relevant context, preferably automatically. Messages that just tell you something bad happened are useless without the context in which the operation is taking place. There is nearly always some higher-level context, such as a request ID or customer name that you can associate with the PID or TID to help you narrow down the data set. Debugging a complex or lengthy process just so you can find out the set of inputs to find that the data is dodgy is not a good use of time and costs the business money, especially if the relevant variables are in scope at the point of writing the error message![^]

Don’t spam the log, write a summary. If you have a process that deals with particularly dirty data, or you have a service that is temperamental try not to log every single indiscretion as a warning or error. Instead try and batch them up, perhaps by item count or time window and write a single message that gives a summary of the issue. At a high-level you’ll just be interested in seeing if the problem is getting worse and so using a tolerance is even better as once you have an idea of what “normal operation” is you can save the scary messages for when they’re really needed.

One of the reviews I do during system testing is to look at the log output. I try and put myself in the position of the support engineer and read the log messages to see if they make sense to me (I often only have a vague appreciation of the process in question). The thing I’m most looking out for is log spam which is usually solved by downgrading messages from, say, INFO to TRACE so that the main output is all about the big picture. It should contain virtually no warnings or errors to avoid support staff becoming desensitised by noise and when they are present they should hopefully be backed by more detailed diagnostics somewhere else.

 

[*] Don’t be fooled into thinking that because a datetime value can be formatted with a milliseconds part that it is actually that accurate. IIRC 16-bit Windows used to have a resolution of 50 ms whilst the NT lineage was better at 10 ms. If you wanted better accuracy than that you needed to resort to the Performance Counters API.

[#] My own free C++ Core library has a reasonably decent parser should you want to tap into the 3rd party option.

[+] I have mixed sentiments about the use of log files. On the one hand I feel they should largely be unnecessary because we should be able to anticipate, test for and handle most types of failure gracefully. But I’m also aware that in-house systems rarely have that level of robustness as a requirement and so we’re often forced into using it as a defensive measure against inadequate test environments and the other in-house external systems.

[^] I clearly remember spending a few hours at the weekend debugging a financial process that was failing with the message “Failed to correlate pairs”. What annoyed me most was that the developer had the two failing currency names in variables in scope but had not bothered to include them in the message! I fixed the error message first thing Monday morning...