Lured Into the foreach + lambda Trap

I had read Eric Lippert’s blog on the subject, I had commented on it, I had even mentioned it in passing to my colleagues the very same morning, but I still ended up writing:-

foreach(var task in tasks)
{ 
. . . 
ThreadPool.QueueUserWorkItem(o => ExecuteTask(task));
}

Then scratched my head as I tried to work out why the same task was suddenly being processed more than once, which was only an issue because the database was chucking seemingly random “Primary Key Violation” errors at me. Luckily I had only changed a few lines of code so I was pretty sure where it must lie, but it still wasn’t obvious to me and I had to fire up the debugger just to humour myself that the ‘tasks’ collection didn’t contain duplicates on entry to the loop as I had changed that logic slightly too.

Mentally the foreach construct looks to me like this:-

foreach(collection)
{
    var element = collection.MoveNext();
    . . .
}

The loop variable is scoped, just like in C++, and also immutable from the programmers perspective. Yes the variable is written before the opening ‘{‘ just like a traditional for loop and that I guess is where I shall I have to look for guidance until the compiler starts warning me of my folly. Back in the days before all C++ compilers* had the correct for-loop scoping rules a common trick to simulate it was to place an extra pair of { } ‘s around the loop:-

{for(int i = 0; i != size; ++i)
{
    . . .
}}

This is the mental picture I think I need to have in mind when using foreach in the future:-

{var element; foreach (collection)
{
    element = collection.MoveNext();
    . . .
}}

That’s all fine and dandy, but how should I rewrite my existing loop today to account for this behaviour? I feel I want to name the loop variable something obscure so that I’ll not accidentally use it within the loop. Names like ‘tmp’ are out because too many people do that to be lazy. A much longer name like ‘dontUseInLambdas’ is more descriptive but does shout a little too much for my taste. The most visually appealing solution (to me at least) has come from PowerShell’s $_ pipeline variable:-

foreach(var _ in tasks)
{
    var task = _;
    . . .
    ThreadPool.QueueUserWorkItem(o => ExecuteTask(task));
}

It’s so non-descript you can’t use it by accident in the loop and the code still visually lines up the ‘var task’ and ‘in collection’ to a reasonable degree so it’s not wholly unnatural. I think I need to live with it a while (and more importantly my colleagues too) before drawing any conclusions. Maybe I’ll try a few different styles and see what works best.

So what about not falling into the trap in the first place? This is going to be much harder because I’ve got a misaligned mental model to correct, but questioning if a lambda variable is physically declared before the opening brace (instead of the statement) is probably a good start. Mind you this presupposes I don’t elide the braces on single-line loops - which I do :-)

foreach(var task in tasks)
    ThreadPool.QueueUserWorkItem(o => ExecuteTask(task));

One final question on my mind is whether or not to try and write some unit tests for this behaviour. The entire piece of code is highly dependent on threads and processes and after you mock all that out there’s virtually nothing left but this foreach loop. It also got picked up immediately by my system tests. Still, I’ve always known the current code was a tactical choice, so as I implement the strategic version perhaps I can amortise the cost of refactoring in then.

 

[*Yes I’m pointing the finger squarely at Visual C++]

Integration Testing with NUnit

Back near the start of my current project I came across a number of tests in one of our unit test assemblies that touched the file-system. My gut reaction was to cry foul and point out that unit tests are not allowed to have any external dependencies. In fact just the other day on the accu-general mailing list someone was trying to locate the definition of a Unit Test that Kevlin Henney used in his ACCU 2008 Conference session:-

A unit test is a test of behaviour (normally functional) whose success or failure is wholly determined by the correctness of the test and the correctness of the unit under test. Thus the unit under test has no external dependencies across boundaries of trust or control, which could introduce other sources of failure. Such external dependencies include networks, databases, files, registries, clocks and user interfaces.

Test Code Structure

The author of the tests agreed that they weren’t really unit tests, but defended his position by saying that he just wanted an easy way to invoke his code. He was using Resharper* which allows you to run a unit test incredibly easy due to it having its only NUnit compatible test runner. Being new to NUnit I also didn’t know that you can categorise tests with string labels that can be passed to the test runner to ensure only a subset of the tests are run:-

[TestFixture, Category(“MyCategory”)]
public class MyClassTests
{
. . .

C:\> nunit-console /include=”MyCategory” MyAssembly.Tests.dll

He had created a number of categories including “UnitTest” and “IntegrationTest”. It felt wrong to include Integration Tests inside what had historically been our Unit Test assemblies so we agreed to split them out at a later date. This meant we could use the assembly itself as a coarse grained grouping mechanism and save NUnit categories for a more functional grouping.

It didn’t take long before it became apparent that we were heading into assembly hell as we had already decided to partition the code so that production logic sat in one assembly, the unit tests in another and a further one for related hand-crafted mocks and stubs that would be used by any test utilities. Adding an integration tests assembly as well would mean a ratio of 1 production assembly to 3 test assemblies. The Visual Studio solution was already creaking even with Solution Folders used to group components.

Instead we decided to keep a ratio of 1:1 with production and test assemblies and use his original test categories and then adjust the build scripts to only run a certain type of test from each test assembly. This also didn’t upset Resharper as it already provides support for running tests of different categories.

NUnit allows you to categorise tests either at the Test or TestFixture level. I didn’t like seeing the two mixed together in the same source file and found the Category attribute added quite a bit of extra noise to the individual tests so we elected to use separate fixtures for each category. This also means that should we decide to go back to using separate assemblies for unit and integration tests we can. So our fixture headings currently look like this:-

In MyClassUnitTests.cs

[TestFixture, Category(Test.Category.UnitTest)]
public class MyClassUnitTests
{
. . .
}

In MyClassIntegrationTests.cs

[TestFixture, Category(Test.Category.IntegrationTest)]
public class MyClassIntegrationTests
{
. . .
}

The nature of integration tests is such that you should be testing your interaction with some external dependency and therefore it is that dependency that more naturally lends itself to the test fixture name and avoids this kind of clash:-

[TestFixture, Category(Test.Category.IntegrationTest)]
public class TheirBackEndIntegrationTests
{
. . .

Of course this may work for a simple Gateway style component, like a web service with a couple of methods, but when you write tests for your Data Access Layer classes you may well just be writing tests that either use a mock or real database connection and so the distinction may appear to be only the test type and database dependency. Mind you, perhaps that’s a smell indicating your testing efforts are not optimal?

Test Configuration

By Kevlin’s definition there must be an external dependency and in the initial cases there was – the file-system. However looking forward we also had plenty of database code to come and it would be neat if we could test that the same way. We now do and I’ll describe that in a follow-up post.

Fortunately the nature of the components that we were initially testing were such that they serve up data from the file-system in a read-only manner. This meant that we could create a cut-down copy of the service data repository inside the VCS structure and always keep the folder layout and data format in step with the code. It also means that multiple simultaneous builds will not interfere with each other.

The build process uses some pretty run-of-the-mill batch files, so the question was how to pass the path of the test data folder via nunit-console.exe onto the test code assembly. I didn’t find any obvious mechanism during my searches – mostly people seemed to be trying to use the standard .Net .config mechanism and running into issues with AppDomains and suchlike. I decided that given the core folder structure was pretty much settled, and it’s already part of the VCS repository, what’s wrong with hard-coding a relative path?

const string REPOSITORY_ROOT = “..\..\..\IntegrationTests\Data\MyService”

Yes I know it feels unclean, but is it any different to hard-coding the same path in the .bat file that invokes nunit-console? At least it’s a relative path and I would hope that we would find out pretty quickly if this breaks or our Continuous Integration process is worthless.

This is all fine and dandy for read-only integration tests (which the vast majority of ours are at the moment) but what about if they also have to write to the file-system? For this we are using the Temp directory. The .Net API has methods for generating a temporary file with a unique name or you can use the Process ID and Thread ID if you want to something a little more user-friendly as this will still be safe with multiple simultaneous builds running. In retrospect I guess you could create a Temp folder within the VCS working copy and treat it like the ‘obj’ and ‘bin’ output folders. In Subversion you can use the ‘svn:ignore’ property to exclude a file or folder from being treated as unversioned and keep your “SVN Commit…” window clean.

Test Results

You still write your tests in the exact same was as unit tests, it’s just that you may have to be a little more creative in what it is that you are asserting. If you’re not round-tripping the data and cannot use Assert.AreEqual() you may instead just be satisfied with checking for a files existence and/or length:-

Assert.IsTrue(System.IO.File.Exists(filename));

Alternatively you can use the absence of exceptions to indicate success by finishing the test with Assert.Pass()** or Assert.IsTrue(true). Sometimes it’s just not possible to completely verify the results of this kind of test and the ROI to achieve that may be small; it may be worth putting that effort into System Tests instead. At the very least you’ll end up with an easy way to invoke the code under a debugger should the need arise.

Component Level Tests

Integration testing appears to be quite a loose term with the definition of ‘external’ being applied to everything from another class, to another team, to another organisation. In his book Code Craft, Pete Goodliffe describes Component Testing (p138) as something that sits between Unit and Integration Testing. Having not made this distinction in the past I’m not sure quite where the ‘external dependency’ line is drawn, but my guess is that it’s probably closer to Unit than Integration. Hopefully someone will add a comment that points out a really good book/article on testing as Code Complete’s chapter on Integration/System Testing talks a lot about methodology and little about the testing mechanics.

 

 

[*The reason why I’m procrastinating over installing Resharper will have to wait for another day, but Thomas Guest touched nicely on the subject just recently.]

[**It seems that Resharper doesn’t like Assert.Pass() - maybe I’m using it incorrectly? The uglier Assert.IsTrue(true) satisfies my need to always finish a test with an Assert.]

Cleaning up svn:mergeinfo Droppings

I’ve been using Subversion for about 6 months now and the daily grind of Update/Edit/Commit is a breeze with TortoiseSVN and Ankh. Admittedly my team only has a small codebase at the moment but performance is generally very good. However one area that Subversion is already struggling with is merging. I don’t think this is news to anyone who has used Subversion for any period of time (or the developers for that matter) but it does taint the experience a little. It’s also obviously unfair to compare a free VCS like Subversion with something Enterprisey like ClearCase (that comes with an Enterprise price tag to match), but I’ll do it anyway because my only other real yardstick is Visual SourceSafe…

The Merging Strategy

We are 3 iterations into a new system and although we have made no formal delivery yet, I have nonetheless treated the end of each iteration as a formal release so that I can get used to the branching, merging and labelling techniques in Subversion. I was already aware that Subversion does not treat branches and labels as first-class concepts like ClearCase, so was keen to explore Subversion’s model to discover its limitations as early as possible. We are following a run-of-the-mill development model, with continuous integration into the trunk and branching for release, but due to the odd mishap we have needed to cherry-pick changes off the trunk for the release and also cherry-pick changes off the release back to the trunk. The former is just an education problem as some of the developers adjust to working with branches, whilst the latter is a necessity because releases are often not 100% correct at the point of branching and need nursing to get them production ready. In the past with ClearCase cherry-picking changes caused no side effects but with Subversion it creates an ever growing trail that affects subsequent merges in unpleasant manner.

The svn:mergeinfo Property in Subversion

Subversion uses a special property on files and folders to record when a merge has taken place called “svn:mergeinfo”. The value of the property is a list of the branches (and associated range of revisions) that have been merged to date – irrespective of whether a physical change has been made to the item itself, e.g.

/branches/releases/1.0:1-1000
/branches/feature/cool_stuff:900-910
/branches/releases/2.0:1100-2000

Generally speaking it’s a good idea to merge using the root of a branch to ensure that you don’t miss anything (unless you’re cherry-picking changes of course). In Subversion this means that only the branch roots then need to have their svn:mergeinfo properties updated which keeps things clean and tidy. But if you cherry-pick a change Subversion then needs to maintain a merge list on that file for ever more, and more importantly for every future merge where that file is a child and therefore a potential candidate. The net outcome of this architecture is that you end up with a very noisy commit every time you merge because it is full of svn:mergeinfo property updates with the real code changes obscured.

ClearCase uses a similar technique (called hyperlinks) to achieve the same goal, but the difference is that it only records a link back to the single source version that contributed the changes – subsequent merges have no effect. Of course ClearCase is renowned for it’s slowness* and it’s possible that the Subversion architecture may improve the speed of merging at the cost of fidelity. The Revision Graph from TortoiseSVN doesn’t appear to have the wonderful merge arrows that the equivalent feature in ClearCase has and you can’t instigate merges from the TortoiseSVN Revision Graph – a feature I used heavily on a previous project - but maybe that says more about the quality of the Development Process than the tool…

Deleting the MergeInfo Properties

Now, if I interpret this post “Subversion merge reintegrate” correctly then the point of the mergeinfo property is so that Subversion knows what contributions have already been taken from other branches. The fact that Subversion keeps updating the upper revision in the property after each branch is merged reinforces my belief that this is an optimisation of some sort. In theory then you could remove those entries in the merginfo properties which reference dead branches. The only fly-in-the-ointment is that you probably won’t find a mergeinfo property with a single reference exactly because of the behaviour outlined above. So what about deleting the whole mergeinfo property on each file and folder?

C:\> svn propdel svn:mergeinfo –R

Obviously you can’t do this on the source branch as it’s the target of the branch that gets updated on a merge. So it would really have to be done on the trunk. If you know that you are never going to do a large scale merge again from any release branches (i.e. trunk is the only active branch) then I think wiping out all the svn:mergeinfo properties could be the quick solution. And you still have the ability to cherry-pick changes from an old branch should the need arise.

When I say all properties, there is one folder you should leave untouched – the root folder of the branch, e.g. /trunk or /branches/Release/0.1 or whatever. I believe that this advice is sound as long as you have not deleted any properties from files or sub-folders that have revisions outside the range specified at the root folder. The svn:mergeinfo property appears to be ‘inherited’ by its children so the act of deleting the property from the children doesn’t seem to stop Subversion from correctly inferring the merge set as long as a parent folder back up the tree has the right information.

Faking a Merge

One of the features I’ve seen in the TortoiseSVN UI that I’ve not yet used is the ability to record a merge without making any actual content changes. I’ve used similar features in the past when you have files you want to reconcile but the target version doesn’t physically need to change. The canonical example of this is a file that just contains the version number – it is different on every release branch but fixed to, say, 0.0.0 on the trunk. Merging these files will always cause a conflict so you only want to record the fact that the file has been ‘logically’ merged to ensure the entire merge is clean, i.e. no changes are left unaccounted for on the source branch.

I reckon this feature gives us a clean(ish) way to resolve the issue at strategic points in our development cycle – as long as we don’t perform long term development outside of the trunk. Here’s how I think it goes:-

1. Feature branches are created, then merged (reintegrated) at their roots and discarded. This means only the root folder needs to have its svn:mergeinfo up-to-date. Once discarded the merge information in the root serves no further purpose, but also causes no problems.
2. Release branches on the other hand may have changes cherry-picked off the trunk and other temporary branches and the reverse may also occur when the release is patched. This is where we start accumulating svn:mergeinfo properties on the child files and folders.
3. Once we reach a stable point where we no longer expect to take old contributions from our previous release branches we delete all the child svn:mergeinfo properties on the trunk (leaving one at the root) and record merges at the trunk root for each branch that we cherry-picked from. The net result should be the trunk root having merge records accurate up to today. Fixes made after today on the release branches should still show up in future merges as candidates.

I was going to add an additional condition on point 3 that there must be no open feature branches. My reservation is that Subversion may undo all this hard work when it comes to reintegrate a feature branch as there will be svn:mergeinfo properties on one branch but not the other. I haven’t done any experiments yet to see how Subversion handles this kind of property merging, but I will when a suitable opportunity arises.

It’s Easy To Google The Answer Once You Know It

When I first started looking into this issue, I didn’t really know what to search for and so didn’t find much. I then read bits of the SVN Book and started writing this post based on what I thought might be the answer. Once I’d finished it I went back and Googled “mergeinfo propdel” and what do you know? Yup, more posts about this issue than you can shake a stick at! I’m still publishing it though because I’ve not really seen one that explains why you can use svn propdel and many of them don’t mention keeping the mergeinfo intact at the branch root. In fact this is the most succinct answer I came across…

 

[*Personally I’ve never found the time waiting for ClearCase to generate the merge set on some 30,000 file views that excessive. At least not compared to the time you actually spend trying to decipher what code people have written to verify whether the merge is valid]

WCF Service Refusing Connections

I spent the majority of the day looking into a WCF issue with our system whereby the same request submitted in bulk against the same WCF service would fail indeterminately:-

System.ServiceModel.EndpointNotFoundException: Could not connect to http://localhost:8000/MyService. TCP error code 10061: No connection could be made because the target machine actively refused it 127.0.0.1:8000

I had noticed by log inspection that about 10 of the requests would succeed and the majority of the others would fail, this led me to believe that there was a default setting somewhere inside the WCF configuration that was having a throttling effect. When I finally got to the bottom of the issue I was somewhat annoyed with myself as it was a classic socket issue - not a WCF specific one. Therein I guess lies the problem with dealing with ever higher layers of abstraction – you begin to forget about the low-level details that underpin it (and which eventually leaks through). Fortunately my copy of Programming WCF Services by Juval Lowy arrived the other day so I had a chance to do some reading to see what possible setting could be having an effect.

ServiceThrottling “maxConcurrent*'”

Lowy dedicates an entire chapter to service request throttling. There are 3 settings:- maxConcurrentSessions, maxConcurrentCalls and maxConcurrentInstances described for the Service Throttling Behaviour that allow the maximum load on each service to be controlled. The effects of these settings are dependent on the transport and concurrency mode of the service, but it’s all spelt out clearly in the book. The most interesting of the lot for me was the number of concurrent sessions - which defaults to 10 under a Per-Session model which is what we are using.

I bumped the setting up to 1000 (way over the 100 needed to service my test harness load), but it had no effect. To humour myself I dropped all three settings to 1 to ensure it had some impact. It did. Oh.

NetTcpBinding “MaxConnections”

Back to the drawing board, or more accurately Google and the book. So convinced was I that I had discovered the cause that I never bothered to read on to the end of the chapter, where I would have discovered another setting that limits the maximum number of TCP connections for the binding:- maxConnections. This setting, which also has a default of 10, goes hand-in-hand with the previous settings as the smaller of it and maxConcurrentSessions becomes the effective throttle.

Once again I changed my service config only to become immediately disappointed as it also failed to have the desired result.

NetTcpBinding “listenBacklog”

Switching back to Google I read various posts about firewall issues which didn’t apply as I was running on the same machine. However I did start to pay closer attention to some of the settings that were being included in the App.config files. One in particular caught my eye – listenBacklog. Anyone who has done any raw sockets programming will know that when you create a server-side socket and start listening on it you need to specify how many pending connections you’re willing to buffer. I quickly Googled again to see what the default value was – yup another 10!

I quickly plugged my ridiculous value into the .config file and bingo! This time it worked. So, a new twist on a very old problem…

xUnit Style Database Unit Testing

Like many small teams I find myself in the position of not only writing framework and application code but also occasionally for the back-end database. In most teams I’ve worked in there’s generally been a developer that tends to do far more of the core database work than the others, such as writing triggers, handling indexes (or is that indices :-) and optimising our poorly written stored procedures. During my last assignment I started getting more involved in the development process of the database code as I wanted to try and help streamline it to iron out some of the integration and deployment issues we were having. My thoughts weren’t exactly greeted with open arms by the DBAs, but in my new role I’m finding that I have an opportunity to test out my ideas as they are being taken on board by the team with an interest in developing them further.

The Traditional Development Process

Database development has always appeared to be treated differently to normal application development. With application code you have to write the source, compile it, test it and then deploy it to the application servers (let’s ignore dynamic languages for now). With SQL you can write the source, deploy it very simply and then test it. What’s even easier is that you can extract the source from the database, make a change and redeploy without even saving the file anywhere else. Of course just because you can work this way, doesn’t mean you should. Some of the issues I’ve seen previously involved code which was running on the database but was not in the Version Control System (VCS) and regressions caused by code being written to work with the current data set instead of to a “specification”.

In my current role I wanted to see if we could adopt a more formalised process for database development. Essentially I want to see if we can treat database code almost exactly the same way as the application code:-

• The VCS acts as the master for all source code - you always start from what’s in the repository not a database
• Unit tests exist for the public interfaces
• Automated running of the unit tests on the build machine
• Automated packaging of the scripts on the build machine ready for deployment
• Automated deployment of the database code to a test environment

The last three should be relatively simple – just a bit of scripting, but what does it mean to unit test database code? When considering stored procedures and user-defined functions the answer is straightforward because they map onto traditional unit testing of subroutines & functions. But what about triggers, views and constraints? Also how should the running of the test suite fit into the wider context of the whole database development process? Is a more structured development process too detrimental because fights against traditional practices and tooling?

The Public Interface

When unit testing object-orientated code it is often asked if you should test private methods. I don’t believe so because they are implementation details and should be tested only as a by-product of exercising the public interface. If they are not invoked via the public interface, then what purpose do they serve (Reflection and Active Objects notwithstanding)? Trying to test private methods also leads you to exposing them or writing hacks to facilitate this goal which only ends up leading to the need to refactor tests when the implementation changes which seems anathema to the cause.

So what constitutes the Public Interface of a database? Obviously Stored Procedures and User Defined Functions fall into this category, but what about tables? This depends on your access model. If you’re allowing your client code full access to your tables then your schema changes can directly affect your clients, whereas if you follow the model of all access via Stored Procedures then they’re not public. Views are also public as they’re often created with the very purpose of encapsulating the base tables they draw from.

We have chosen this latter model and have agreed that all implementation details such as Constraints and Triggers should only be tested as a by-product of testing the Stored Procedures that drive them. For example, if the specification says that the SubmitOrder() procedure should default the order status to ‘Pending’, then it matters not whether that happens explicitly in the sp code itself, by a default value on the table or via an insert trigger – they all achieve the same externally visible goal.

The xUnit Model

With the decision made about what to test, the next step is finding a way to do it. The most popular unit testing model these days seems to be xUnit. In an OO language you implement a test as a class method that includes an Assert to perform a comparison after invoking the required behaviour. Each test only tests one thing, is not dependent on any other, should be free from side-effects and is carefully named to reflect the intended behaviour (nay specification). The model also supports the ability to define common pre and post-test code in methods called SetUp() and TearDown() at both the Test and Fixture (set of related tests) levels to remove repetitive code and make the test intent clearer.

So how can we map these concepts onto database code? The side-effect free behaviour is a no-brainer as we can just use transactions. Databases do not support a declarative model that allows you to use Attributes to indicate tests, fixtures etc, but they do have their own internal tables that can be used in a reflection-like manner to invoke procedures based on their names. For example you could use a prefix such as “test” or “utsp_” to separate unit test code from production code. Then to distinguish between the actual tests and the initialisation and cleanup functions you could use “<sp>_Should<expectation>” as an indicator of a test, and “<sp>_SetUp” and “<sp>_TearDown” for the other two. The fact that all test procedures take no arguments and return no values means that they can be invoked easily by reflection.

This leads to a SQL unit test script structured something like so:-

CREATE PROCEDURE testSubmitOrder_SetUp
AS
    INSERT Customer(CustomerId, CustomerName)
        VALUES(1, ‘TestCustomer’)
GO

CREATE PROCEDURE testSubmitOrder_TearDown
    . . .
GO

CREATE PROCEDURE testSubmitOrder_ShouldSetStatusPending
AS
    DECLARE @OrderId int
    EXEC SubmitOrder ‘TestCustomer’, ‘widget’, 2,
        @OrderId OUTPUT
    IF NOT EXIST (SELECT 1 FROM Order
                  WHERE OrderId = @OrderId
                  AND OrderStatus = ‘Pending’)
        RAISERROR (‘Order should be inserted with Pending status’, 16, 1)
GO

CREATE PROCEDURE testSubmitOrder_ShouldUpdateInventory
    . . .
GO

This leads you to a physical layout where one .sql file contains all the test code for a single Stored Procedure, much like a single source file containing all the tests for a class.

I’ll be honest and admit that the details of mocking and supporting a Fluent Interface style for asserting far exceeds my knowledge of SQL and databases. I know (because I’ve done it by mistake in the past!) that procedures of the same name can belong to different users, so perhaps this mechanism can be exploited? For the moment the style above serves our purpose.

Executing The Test Suite

My initial thoughts about the way that the test suite would be executed were somewhat grandiose. I thought that the Test Fixture could be used to only load the database objects needed to run the tests and therefore also provide a mechanism though which mocks could be injected. I fear that this is a maintenance nightmare in the making if the dependencies are not automated. The simpler option is to recreate the database from scratch and then run the unit test scripts on that. Because each unit test is responsible for its own test data we can avoid taking the “restore” option which ensures that we are guaranteed a truly clean database - one free from developer detritus.

The wonderful knock-on effect of formulating this process is that it can be executed on the build machine as part of the Continuous Integration process. SQL Server Express is just the ticket for the database role as it gives us all the features of it’s bigger brother, but can easily cope with the trivial workload we throw at it. This also feeds into the daily database development cycle as I can use a local installation of SQL Server Express as my sandbox where I only require the schema, objects and minimal test data because I’m focusing on the functional aspects of the problem, not performance testing.

This technique also gives us a path to the goal of continuous integration testing of database and application code and then continuous deployment to the system test environment. Here we can automatically rebuild a database from scratch, populate it with real data, deploy the code to the application servers and push the big green button. If this is done during the night, then by the morning we will have exercised a significant number of code paths and have a high degree of confidence that the environment is stable and ready for further integration/system testing.

Scalability

One of the conventions in unit testing is to reduce your dependencies so that the tests run really quickly, then you have no excuse not to run them at every opportunity. Unfortunately, due to the nature of the toolchain (an out-of-process database) running tests is not as instantaneous as with application code. Running the individual tests isn’t too expensive, but rebuilding the database from scratch can take a while so I would keep that for when you do a “Clean Build” prior to check-in.

TSQLUnit

As I mentioned in a previous post “Company-Wide IRC Style Chat & Messaging” I first started thinking about this some years ago, and a colleague at the time pointed me to TSQLUnit. I had a very brief look back then and it seemed pretty useful as it followed a similar model. I haven’t had time to go back and look at it in more detail. None of what I’ve said is rocket science though, if anything the hardest part is writing the Windows Batch files to trawl the VCS repository and invoke the relevant .sql scripts – a whole “FOR /R %%I” loop with a “SQLCMD –i %%I” body. Even PowerShell seems overkill for this job :-)

 

Phew, I didn’t expect this post to turn into War and Peace! At the moment this is working out really well for my current team, but I’m aware that as a greenfield project we currently don’t have a very large schema or number of objects. However, it’s going to be a large system in the end so it will be really interesting to see how all this scales up.

Coders at Work – Jamie Zawinski

I’ll be perfectly honest and admit that until Joel Spolsky’s “Duct Tape Programmer” blog, I’d never heard of Jamie Zawinski. Whilst reading Ayende Rahien’s blog the other day I find that he was also attributed with that memorable quote about Regular Expressions:-

Some people, when confronted with a problem, think "I know, I’ll use regular expressions." Now they have two problems.

Luckily my mum, ever in search of something to buy me for my birthday, checked my Amazon wish list and pulled an item from it - the book Coders at Work by Peter Seibel. The blurb states that this is a collection of interviews with some of the top programmers of our time. Now I consider myself to be pretty well read and yet I could only tell you who 4 of the 15 interviewees were, and one of those was only because his name came up at the Stack Overflow Dev Day last October. For the record Donald Knuth, Joe Armstrong & Simon Peyton Jones are the names I immediately recognised and Peter Norvig was the one name-checked during the Dev Day. Anyway I’m sure looking forward to finding out who these other “prominent” programmers are and maybe I’ll be able to work out why their names didn’t register.

My impression of Jamie from the blog posts by Joel Spolsky and Uncle Bob wasn’t at all favourable. I felt he was portrayed as a reckless hacker who paid little attention to the maintenance aspects of software development. I had a vision of a loner coder sat in an Ivory Tower churning out Netscape. Ok, maybe I’m exaggerating now :-) Of course soon into the book I realised this picture was far from the truth, in fact the more I read the more I noticed how much I have in common with him…

I started out working on shrink-wrap software and have since spent more time working in the enterprise arena. As Jamie points out these two worlds are very different and that can have a significant knock-on effect on the way that software is developed. In the dog-eat-dog shrink-wrap world your customer likely has a number of choices and if you don’t deliver someone else will. In stark contrast is the closed world of in-house software where your customer has only one choice – you - they can’t reasonably go anywhere else. In Jamie’s words:-

I know it’s kind of a cliché but it comes back to worse is better. If you spend the time to build the perfect framework…release 1.0 is going to take you three years to ship and your competitor is going to ship their 1.0 in six months and now you’re out of the game. You never shipped your 1.0 because someone else ate your lunch. Your competitor’s six-month 1.0 has crap code and they’re going to have to rewrite it in two years but, guess what: they can rewrite it because you don’t have a job anymore.

These statements put his later comments about unit testing (at Netscape) into a whole different light. I’m sure Uncle Bob would cite Yoda and use the quote “No! No different! Only different in your mind”, and in theory he is probably right. After my recent post “My [Unit] Testing Epiphany” it would be churlish of me to disagree entirely with Bob, but I can also see where Jamie is coming from.

I also agree with his sentiments about education. It’s extremely rare these days to come across a programmer that didn’t go to University and get a degree in Computer Science or some form of Engineering (I have a degree in Electronic System Engineering - but I sucked big time on the analogue side!). In the couple of cases I have come across where a decent candidate lacked a degree they fitted the mould of the classic 80’s bedroom coder and have themselves described how hard it is to be noticed when you don’t have a degree on your CV. I rated both these candidates highly for the type of work I do because they come from a time where resource management needed careful consideration. In the interview (pp 47) Siebel questions whether we can leave those days behind in favour of a more abstract view of resource management. Given the current attitude to Scaling-Out (more hardware) instead of Scaling-Up (optimised code) it seems that way, but as the need to reduce power consumption within the data centres (and therefore the cloud) becomes more important we may see greater importance placed on those skills.

The one part of the interview that did surprise me was his attitude towards Design Patterns. He said:-

Then there was another book that everybody thought was the greatest thing ever in that same period – Design Patterns - which I just thought was crap. It was just like, programming via cut and paste…Then in meetings they’d be tossing around all this terminology they got out of that book.

I honestly think that he has completely missed the point (or maybe I’m misunderstanding his answer). I’m sure that some developers do try and design software in a cookie cutter fashion and attempt to use every one of the GoF patterns, but by-and-large the single biggest contribution they have made is the introduction of a common vocabulary. They have also been instrumental in promoting the notion of “Program to an interface, not an implementation”. Maybe he uses common terms like Composite, Factory, Mediator, Proxy etc but doesn’t attribute their wide acceptance or visual connotations to the GoF. Fair enough, it doesn’t take a rocket scientist to guess what each of those terms mean, but there are a number of others that are less obvious and yet are also incredibly common, such as Double-Checked Lock, Scoped Guard, Monitor, Context Object and Data Mapper. I’d sure be interested to know how he avoids this aspect of Patterns.

I’ve only read one this one chapter so far but it has achieved it’s job of painting a far richer picture of the man that ended up becoming the centre of a row about software quality. It has also reminded me that there are different forces driving the software development process, like shrink-wrap vs enterprise and application vs framework, which pretty much guarantees that one size never fits all.

Process Explorer Failed to Display DLLs or Handles

If this was Mark Russinovich he would probably have titled this post “The Curious Case of Process Explorer…”, but I’m not him and it’s not nearly as interesting or challenging as the kinds of issues he blogs about…

The other day I was investigating an issue where our new C# based grid service was failing to run on any engines because of what appeared to be an issue with assembly paths. As a n00b to .Net and having done a little reading into AppDomains and how assembly paths and native paths are used I thought I’d have a look at the grid engine surrogate process with Process Explorer to inspect its Environment Block and also what DLLs it already has loaded. This was on a 64-bit server running Windows Server 2003. After firing up Process Explorer I duly opened the lower pain and switched it to DLL mode only to be presented with an empty list. I humoured myself by checking the sizes of the columns and switching to Handle mode, but the pane was definitely empty. I even clicked on a few other processes like the svchosts’s, but the result was the always the same - nought.

Then, purely by accident, I clicked on explorer.exe and it worked. Realising that this was one of the processes running in my Windows session I added the “Username” column to the top pane and sure enough the only processes I could inspect where my own. Most of the machines I’ve dealt with in the past have been 32-bit boxes except for a few Big Iron 64-bit application servers. Not sure If I’d run into a limitation of Process Explorer on 64-bit Windows, I decided to go old school and run the command-line based Handle tool instead. This time I got a more interesting result:-

Error loading driver: Access is denied

I thought they had granted me admin rights on the box, so again I thought I’d humour myself and check by running:-

C:\> NET LOCALGROUP Administrators

Sure enough I was an administrator. Time to hit the Search Engines…

I didn’t find anything that exactly matched my issue with Process Explorer (PE), but there were some posts about the Handle error message. One post said to check whether you have the SeLoadDriverPrivilege granted in any process. I checked PE itself, and no I didn’t have this in my token. The post then went on to describe how you can check if you should have this privilege by using secpol. I did and the Administrators group is indeed granted this right.

Casting my search wider I started searching for info on SeLoadDriverPrivilege in the hope that I might discover where this has affected other similar low-level diagnostic tools. What I found was lots of discussion about rootkits and other malware. Given the environment I was working in I found this scenario extremely unlikely, but at the same time I could imagine how the security software that was running on the server could also be using similar techniques exactly to stop this kind of infection.

Fortunately along the way I came across another post that mentioned running Process Explorer under the SYSTEM account – the post was actually about Vista and UAC, but the idea seemed worth a go. To do this you can use another Sysinternals tool “psexec” with the “–s” and “–i” switches. I must confess that in all the years I’ve been using psexec I’ve never once needed to use these switches. As the command-line help will tell you, –s runs the process under the SYSTEM account and –i allows it to interact with the desktop. You also need to remember to fully qualify the path to the child process you’re invoking if it’s not going to be on the SYSTEM account PATH, e.g.

C:\>C:\Tools\psexec –s –i C:\Tools\procexp.exe

Lo and behold it did the trick, I could see everything again in Process Explorer. Naturally I’d like to know if the whole SeLoadDriverPrivilege thing was a red herring or not, and if it isn’t, get some closure on why the privilege is missing in the first place – now that’s the kind of job Mark Russinovich would tackle…