Happy Birthday, Blog

Today sees the anniversary of my inaugural post (An apology to Raymond Chen) on this blog. That seems a highly appropriate moment to reflect and see if it’s turned out the way I hoped…

As I mentioned back in July, when my first two reviews were published in the ACCU Journal, I’ve found writing difficult, largely I guess because I’m out of practice. Writing this blog has certainly made a dramatic improvement to the speed at which I write. For example, last year my review of the ACCU 2009 Conference took me days to write – and I was on my sabbatical at the time! This year I did it on the train during my commute in a matter of hours. Ok, so I’ve not been one of those prolific bloggers that rattles out a piece every day, but I have tried to write a post a week. I realise that many of my posts are quite lengthy and given that I only use my daily commute as the source of time for it I reckon that’s not too shabby a rate. My commute time already has competition from reading, gaming and maintenance of my freeware codebase so it’s already a tight squeeze.

The one area I certainly didn’t expect to be writing about was C#. I was still a die hard C++ aficionado back in April last year and naturally assumed I’d be writing about C++ issues (if there are any left). It’s funny, but with all that time on my hands during my sabbatical I found it harder to know what to write about, whereas now I’m back working full-time the ideas keep flooding in. Once again I expect that part of it is down to the blogging experience, but I also suspect that I feel more confident about the topics I’m covering. I definitely expected to be sticking to very technical issues such as the recent ones involving WCF, but my current project is both greenfield and using an agile methodology and that has highlighted some very interesting dynamics which in turn has led to a new degree of consciousness about a number of software development issues.

Without a doubt the single biggest contribution blogging has made to me has been the clarity of thought that comes from the fear of “publishing and being damned”. Knowing that the moment I hit the ‘publish’ button my words will be broadcast out onto the Internet for all eternity where potential future employers will be able to see them ensures that I try to remain objective. In the last year there have been two posts that I started to write and ended up canning because I realised they were straw-man arguments. Conversely the mere act of documenting my experiences also leads to new questions that I’ve not considered in any real depth before. I have one post on Unit Test Naming Guidelines that I thought was all done-and-dusted until I met Steve Freeman and Nat Pryce and discovered that I was barking up the wrong tree. No doubt when I come to revise that post at a later date more questions will emerge…

The one thing I haven’t done is look at the stats in Google Analytics. I added a hit counter back at the start, mostly because it was easy, but I never expected anyone to actually read this stuff. The fact that there have been comments submitted (that aren’t just link spam) means that at least a couple of people have bothered to read my musings which is pretty satisfying. Now that a whole year has passed I feel tempted to take a peek and see if the number of hits has reached double figures yet.

Being Author and Editor means that I don’t have that fear of rejection you get with a ‘real’ publication, but I still have that fear of embarrassment to keep me on the straight and narrow. I’m quite contented at present to continue to build up a portfolio of posts that hopefully helps give me that edge we all need to ensure our own survival in the fast changing world of Software Development.

Object Finalized Whilst Invoking a Method

The JIT Compiler & Garbage Collector are wonders of the modern computer age. But if you’re an ex-C++ developer you should put aside what you think you might know about how your C# code runs because it appears their view of the world lacks some artificial barriers we’re used to. Deterministic Destruction such as that used in C++ has a double meaning, sort of. On the one hand it means that an object will be deleted when it goes out of scope, and yet on the other hand it also means that an object is only destroyed by an external influence[*]. Essentially this means that the lifetime of a root object or temporary is defined by the exiting of a scope which keeps life really simple. In the C# world scopes affect some similar aspects to C++ such as value type lifetimes and the visibility of local variables, but not the lifetime of reference types…

The Scenario

The C# bug that I’ve just been looking into involved an Access Violation caused by the finalizer thread trying to destroy an object whilst it was still executing an instance method. Effectively the object finalizer destroyed the native memory it was managing whilst it was also trying to persist that very same block of memory to disk. On the face of it that sounds insane. How can the CLR reach the conclusion that an object has no roots and is therefore garbage when you’re inside a method as surely at least the stack frame that is invoking the method has a reference to ‘this’?

Here is a bare bones snippet of the code:-

public class ResourceWrapper : IDisposable
{
    public void Save(string filename);
}

. . .

public class MyType
{
    public ResourceWrapper Data
    {
        get { return new ResourceWrapper(m_data); }
    }

    private byte[] m_data; // Serialized data in
                           // managed buffer.
}

. . .

public void WriteStuff(string filename)
{
    m_myType.Data.Save(filename);
}

Now, reduced to this simple form there are some glaring omissions relating to the ownership of the temporary ResourceWrapper instance. But that should only cause the process to be inefficient with its use of memory, I don’t believe there should be any other surprises in a simple single-threaded application. I certainly wouldn’t expect it to randomly crash on this line with an Access Violation:-

m_myType.Data.Save(filename);

The Object’s Lifetime

Once again, putting aside the blatant disregard for correct application of the Dispose Pattern, how can the object, whilst inside the Save() method, be garbage collected? I was actually already aware of this issue after having read the blog post “Lifetime, GC.KeepAlive, handle recycling” by Chris Brumme a while back but found it hard to imagine at the time how it could really affect me as it appeared somewhat academic. In my case I didn’t know how the Save() method was implemented, but I did know it was an incredibly thin wrapper around a native DLL, so I’ve guessed that it probably fitted Chris Brumme’s scenario nicely. If that’s the case then we can inline both the Data property access and the Save call so that in pseudo code it looks something like this:-

public void WriteStuff(string filename)
{
    ResourceWrapper tmp = new ResourceWrapper
                              (m_myType.data);
    IntPtr handle = tmp.m_handle;

    // tmp can now be garbage collected because we
    // have a copy of m_handle. 
    ResourceWrapper.NativeSaveFunction(handle,
                                       filename);
}

What a C++ programmer would need to get out of their head is that ‘tmp’ is more like a ResourceWrapper* than a shared_ptr<ResourceWrapper> - the difference being that its lifetime ends way before the end of the scope.

The Minimal Fix

So, if I’ve understood Chris Brumme’s article correctly, then the code above is the JIT Compiler & Garbage Collector’s view. The moment we take a copy of the m_handle member from tmp it can be considered garbage because an IntPtr is a value type, not a reference type, even though we know it actually represents a reference to a resource. In native code you manage handles and pointers using some sort of RAII class like shared_ptr with a custom deleter as each copy of a handle represents another reference, but within C# it seems that the answer is using GC.KeepAlive() to force an objects lifetime to extend past the use of the resource handle. In my case, because we don’t own the ResourceWrapper type, we have to keep the temporary object alive ourselves, which leads to this solution:-

public void WriteStuff(string filename)
{
    ResourceWrapper tmp = m_myType.Data;
    tmp.Save(filename);
    GC.KeepAlive(tmp);
}

From a robustness point of view I believe the KeepAlive() call should still be added to the Save() method to ensure correctness even when Dispose() has accidentally not been invoked - as in this case. Don’t get me wrong I’m a big fan of the Fail Fast (and Loud) approach during development, but this kind of issue can easily evade testing and bite you in Production. To me this is where a Debug build comes into play and warns you that the finalizer performed cleanup at the last minute because you forgot to invoke Dispose(). But you don’t seem to hear anything about Debug builds in the C#/.Net world…

The Right Answer

The more impatient among you will have no doubt been shouting “Dispose - You idiot!” for the last few paragraphs. But when I find a bug I like to know that I’ve really understood the beast. Yes I realised immediately that Dispose() was not being called, but that should not cause an Access Violation in this scenario so I felt there were other forces at work. If I had gone ahead and added the respective using() statement that would likely have fixed my issue, but not have diagnosed the root cause. This way I get to inform the relevant team responsible for the component of a nasty edge case and we both get to sleep soundly.

[*I’m ignoring invoking delete, or destructors or calling Release() or any other manual method of releasing a resource. It’s 2010 and RAII, whether through scoped_handle or shared_ptr or whatever, has been the idiom of choice for managing resources in an exception safe way for well over a decade]

Turning Unconscious Incompetence to Conscious Incompetence

I’m sure that I must have come across the Four Stages of Competence before, but it was when Luke Hohmann quoted the more humorous interpretation that it actually registered with me. Pete Goodliffe also brought up this topic up in his recent C Vu column and it’s got me thinking about how I turn Unconscious Incompetence to mere Conscious Incompetence. Oh yeah and the fact that I’m currently sitting in a garage whilst they drain the unleaded petrol from my diesel powered MPV brings the subject of incompetence to the forefront of my mind…

Luke Hohmann was quoting the “Known Knowns” statement from Donald Rumsfeld during an eXtreme Tuesday Club (XTC) meeting back in November 2009. He sketched a pie-chart with a tiny wedge to represent “what we know”, a slightly larger wedge for “what we know we don’t know” and the rest for “what we don’t know we don’t know”. His talk was about Innovation Games and his point was about where innovation occurs. As always it was an insightful session, not least because it got me thinking about how I’d go about reducing the size of the “what I don’t know I don’t know” slice[*].

I’d always thought that not knowing something in detail was not much better than not knowing it at all. But clearly there is value in it. After the better part of 15 years of C++ I don’t think it would be too modest of me to say that C++ was by-and-large in the “do know” region. My recent move to C# meant that I suddenly found myself drawing considerably on that somewhat larger “don’t know” section. Although for C++ developers the usefulness of MSDN Magazine vanished years ago as Microsoft drove headlong into promoting the New World Order that was .Net and C#, I kept subscribing and reading a significant portion of each issue as I felt that there was still an underlying message in there worth listening to. These days Jeremy Miller has a column dedicated to Patterns and Practices and James McCaffrey covers testing mechanisms, so some of those concepts are more readily digestible.

Still, the background noise of the articles has sat there at the back of my mind so that within the last 6 months I have easily been able to move to the C# world. Topics like Generics, Extension Methods, Lambdas and LINQ were high-up on the reading list as somehow I knew these were important. For GUI work I know there is ASP.Net, WinForms and WPF all vying for attention, and on the IPC front Sockets, DCOM and Named Pipes are all passé, and Indigo/WCF is the one-stop-shop. My recent WCF posts probably illustrates how I’m starting to cover the WCF angle, but the GUI stuff will just have to wait as I’ve no need to absorb that yet.

Clearly I knew more that just the names of many of these concepts so is that really fair to categorise them as Conscious Incompetence? Probably not. So how about a list of things that I know virtually nothing about at all except the name and a truly vague context:-

Hadoop – Something from Google related to Map/Reduce?
Scala/Groovy – Languages somehow related to Java and/or the JVM?
Maven/CMake – Stuff to do with building code?
Recursive Descent Parser – Compiler theory/technique?

Does it make sense to know this little about each of these topics? What if I’m actually wrong about the way I’ve categorized them? Does this fulfil the notion of Conscious Incompetence, or do I already know too much?

As each year passes it seems that keeping up with what’s going on in the world of Software Development is becoming more and more of an uphill struggle. There are so many languages and technologies that tuning into the relevant stuff is nigh on impossible (I could have said filtering out the noise, but so much of it appears interesting in one way or another that the term ‘noise’ feels disingenuous). My RSS reader is overflowing with blogs and articles demanding my attention, so after I’ve read the important stuff like The Old New Thing and The Daily WTF I start skimming the general purpose feeds like Slashdot Developers and Dr Dobbs. I say skim, but it’s pretty hard not to get drawn into reading the entire piece, and then of course you’ll follow a few links and have lost another hour or two of your life. That assumes the wife and kids have left you alone long enough for that to happen…

Therein lies the problem for me. I find it hard to leave many topics in the state of Conscious Incompetence. I want to know enough so that when the time comes my unconscious awareness ensures that I know where I need to go to discover more, but at the same time know little enough that I can still effectively filter out that which is most relevant from the daily bombardment of new innovations and revised practices. I guess I’m in need of a more Agile approach to learning.

 

[*I realised after having read the Wikipedia entries for these two topics more closely that I’m stretching the definition of the Four Stages of Competence to its limit by applying it to the acquisition of knowledge, but as Rumsfeld has shown, repeatedly using the words Known and Unknown would only make the prose more indecipherable]

Lured Into the foreach + lambda Trap

I had read Eric Lippert’s blog on the subject, I had commented on it, I had even mentioned it in passing to my colleagues the very same morning, but I still ended up writing:-

foreach(var task in tasks)
{ 
. . . 
ThreadPool.QueueUserWorkItem(o => ExecuteTask(task));
}

Then scratched my head as I tried to work out why the same task was suddenly being processed more than once, which was only an issue because the database was chucking seemingly random “Primary Key Violation” errors at me. Luckily I had only changed a few lines of code so I was pretty sure where it must lie, but it still wasn’t obvious to me and I had to fire up the debugger just to humour myself that the ‘tasks’ collection didn’t contain duplicates on entry to the loop as I had changed that logic slightly too.

Mentally the foreach construct looks to me like this:-

foreach(collection)
{
    var element = collection.MoveNext();
    . . .
}

The loop variable is scoped, just like in C++, and also immutable from the programmers perspective. Yes the variable is written before the opening ‘{‘ just like a traditional for loop and that I guess is where I shall I have to look for guidance until the compiler starts warning me of my folly. Back in the days before all C++ compilers* had the correct for-loop scoping rules a common trick to simulate it was to place an extra pair of { } ‘s around the loop:-

{for(int i = 0; i != size; ++i)
{
    . . .
}}

This is the mental picture I think I need to have in mind when using foreach in the future:-

{var element; foreach (collection)
{
    element = collection.MoveNext();
    . . .
}}

That’s all fine and dandy, but how should I rewrite my existing loop today to account for this behaviour? I feel I want to name the loop variable something obscure so that I’ll not accidentally use it within the loop. Names like ‘tmp’ are out because too many people do that to be lazy. A much longer name like ‘dontUseInLambdas’ is more descriptive but does shout a little too much for my taste. The most visually appealing solution (to me at least) has come from PowerShell’s $_ pipeline variable:-

foreach(var _ in tasks)
{
    var task = _;
    . . .
    ThreadPool.QueueUserWorkItem(o => ExecuteTask(task));
}

It’s so non-descript you can’t use it by accident in the loop and the code still visually lines up the ‘var task’ and ‘in collection’ to a reasonable degree so it’s not wholly unnatural. I think I need to live with it a while (and more importantly my colleagues too) before drawing any conclusions. Maybe I’ll try a few different styles and see what works best.

So what about not falling into the trap in the first place? This is going to be much harder because I’ve got a misaligned mental model to correct, but questioning if a lambda variable is physically declared before the opening brace (instead of the statement) is probably a good start. Mind you this presupposes I don’t elide the braces on single-line loops - which I do :-)

foreach(var task in tasks)
    ThreadPool.QueueUserWorkItem(o => ExecuteTask(task));

One final question on my mind is whether or not to try and write some unit tests for this behaviour. The entire piece of code is highly dependent on threads and processes and after you mock all that out there’s virtually nothing left but this foreach loop. It also got picked up immediately by my system tests. Still, I’ve always known the current code was a tactical choice, so as I implement the strategic version perhaps I can amortise the cost of refactoring in then.

 

[*Yes I’m pointing the finger squarely at Visual C++]

Integration Testing with NUnit

Back near the start of my current project I came across a number of tests in one of our unit test assemblies that touched the file-system. My gut reaction was to cry foul and point out that unit tests are not allowed to have any external dependencies. In fact just the other day on the accu-general mailing list someone was trying to locate the definition of a Unit Test that Kevlin Henney used in his ACCU 2008 Conference session:-

A unit test is a test of behaviour (normally functional) whose success or failure is wholly determined by the correctness of the test and the correctness of the unit under test. Thus the unit under test has no external dependencies across boundaries of trust or control, which could introduce other sources of failure. Such external dependencies include networks, databases, files, registries, clocks and user interfaces.

Test Code Structure

The author of the tests agreed that they weren’t really unit tests, but defended his position by saying that he just wanted an easy way to invoke his code. He was using Resharper* which allows you to run a unit test incredibly easy due to it having its only NUnit compatible test runner. Being new to NUnit I also didn’t know that you can categorise tests with string labels that can be passed to the test runner to ensure only a subset of the tests are run:-

[TestFixture, Category(“MyCategory”)]
public class MyClassTests
{
. . .

C:\> nunit-console /include=”MyCategory” MyAssembly.Tests.dll

He had created a number of categories including “UnitTest” and “IntegrationTest”. It felt wrong to include Integration Tests inside what had historically been our Unit Test assemblies so we agreed to split them out at a later date. This meant we could use the assembly itself as a coarse grained grouping mechanism and save NUnit categories for a more functional grouping.

It didn’t take long before it became apparent that we were heading into assembly hell as we had already decided to partition the code so that production logic sat in one assembly, the unit tests in another and a further one for related hand-crafted mocks and stubs that would be used by any test utilities. Adding an integration tests assembly as well would mean a ratio of 1 production assembly to 3 test assemblies. The Visual Studio solution was already creaking even with Solution Folders used to group components.

Instead we decided to keep a ratio of 1:1 with production and test assemblies and use his original test categories and then adjust the build scripts to only run a certain type of test from each test assembly. This also didn’t upset Resharper as it already provides support for running tests of different categories.

NUnit allows you to categorise tests either at the Test or TestFixture level. I didn’t like seeing the two mixed together in the same source file and found the Category attribute added quite a bit of extra noise to the individual tests so we elected to use separate fixtures for each category. This also means that should we decide to go back to using separate assemblies for unit and integration tests we can. So our fixture headings currently look like this:-

In MyClassUnitTests.cs

[TestFixture, Category(Test.Category.UnitTest)]
public class MyClassUnitTests
{
. . .
}

In MyClassIntegrationTests.cs

[TestFixture, Category(Test.Category.IntegrationTest)]
public class MyClassIntegrationTests
{
. . .
}

The nature of integration tests is such that you should be testing your interaction with some external dependency and therefore it is that dependency that more naturally lends itself to the test fixture name and avoids this kind of clash:-

[TestFixture, Category(Test.Category.IntegrationTest)]
public class TheirBackEndIntegrationTests
{
. . .

Of course this may work for a simple Gateway style component, like a web service with a couple of methods, but when you write tests for your Data Access Layer classes you may well just be writing tests that either use a mock or real database connection and so the distinction may appear to be only the test type and database dependency. Mind you, perhaps that’s a smell indicating your testing efforts are not optimal?

Test Configuration

By Kevlin’s definition there must be an external dependency and in the initial cases there was – the file-system. However looking forward we also had plenty of database code to come and it would be neat if we could test that the same way. We now do and I’ll describe that in a follow-up post.

Fortunately the nature of the components that we were initially testing were such that they serve up data from the file-system in a read-only manner. This meant that we could create a cut-down copy of the service data repository inside the VCS structure and always keep the folder layout and data format in step with the code. It also means that multiple simultaneous builds will not interfere with each other.

The build process uses some pretty run-of-the-mill batch files, so the question was how to pass the path of the test data folder via nunit-console.exe onto the test code assembly. I didn’t find any obvious mechanism during my searches – mostly people seemed to be trying to use the standard .Net .config mechanism and running into issues with AppDomains and suchlike. I decided that given the core folder structure was pretty much settled, and it’s already part of the VCS repository, what’s wrong with hard-coding a relative path?

const string REPOSITORY_ROOT = “..\..\..\IntegrationTests\Data\MyService”

Yes I know it feels unclean, but is it any different to hard-coding the same path in the .bat file that invokes nunit-console? At least it’s a relative path and I would hope that we would find out pretty quickly if this breaks or our Continuous Integration process is worthless.

This is all fine and dandy for read-only integration tests (which the vast majority of ours are at the moment) but what about if they also have to write to the file-system? For this we are using the Temp directory. The .Net API has methods for generating a temporary file with a unique name or you can use the Process ID and Thread ID if you want to something a little more user-friendly as this will still be safe with multiple simultaneous builds running. In retrospect I guess you could create a Temp folder within the VCS working copy and treat it like the ‘obj’ and ‘bin’ output folders. In Subversion you can use the ‘svn:ignore’ property to exclude a file or folder from being treated as unversioned and keep your “SVN Commit…” window clean.

Test Results

You still write your tests in the exact same was as unit tests, it’s just that you may have to be a little more creative in what it is that you are asserting. If you’re not round-tripping the data and cannot use Assert.AreEqual() you may instead just be satisfied with checking for a files existence and/or length:-

Assert.IsTrue(System.IO.File.Exists(filename));

Alternatively you can use the absence of exceptions to indicate success by finishing the test with Assert.Pass()** or Assert.IsTrue(true). Sometimes it’s just not possible to completely verify the results of this kind of test and the ROI to achieve that may be small; it may be worth putting that effort into System Tests instead. At the very least you’ll end up with an easy way to invoke the code under a debugger should the need arise.

Component Level Tests

Integration testing appears to be quite a loose term with the definition of ‘external’ being applied to everything from another class, to another team, to another organisation. In his book Code Craft, Pete Goodliffe describes Component Testing (p138) as something that sits between Unit and Integration Testing. Having not made this distinction in the past I’m not sure quite where the ‘external dependency’ line is drawn, but my guess is that it’s probably closer to Unit than Integration. Hopefully someone will add a comment that points out a really good book/article on testing as Code Complete’s chapter on Integration/System Testing talks a lot about methodology and little about the testing mechanics.

 

 

[*The reason why I’m procrastinating over installing Resharper will have to wait for another day, but Thomas Guest touched nicely on the subject just recently.]

[**It seems that Resharper doesn’t like Assert.Pass() - maybe I’m using it incorrectly? The uglier Assert.IsTrue(true) satisfies my need to always finish a test with an Assert.]

Cleaning up svn:mergeinfo Droppings

I’ve been using Subversion for about 6 months now and the daily grind of Update/Edit/Commit is a breeze with TortoiseSVN and Ankh. Admittedly my team only has a small codebase at the moment but performance is generally very good. However one area that Subversion is already struggling with is merging. I don’t think this is news to anyone who has used Subversion for any period of time (or the developers for that matter) but it does taint the experience a little. It’s also obviously unfair to compare a free VCS like Subversion with something Enterprisey like ClearCase (that comes with an Enterprise price tag to match), but I’ll do it anyway because my only other real yardstick is Visual SourceSafe…

The Merging Strategy

We are 3 iterations into a new system and although we have made no formal delivery yet, I have nonetheless treated the end of each iteration as a formal release so that I can get used to the branching, merging and labelling techniques in Subversion. I was already aware that Subversion does not treat branches and labels as first-class concepts like ClearCase, so was keen to explore Subversion’s model to discover its limitations as early as possible. We are following a run-of-the-mill development model, with continuous integration into the trunk and branching for release, but due to the odd mishap we have needed to cherry-pick changes off the trunk for the release and also cherry-pick changes off the release back to the trunk. The former is just an education problem as some of the developers adjust to working with branches, whilst the latter is a necessity because releases are often not 100% correct at the point of branching and need nursing to get them production ready. In the past with ClearCase cherry-picking changes caused no side effects but with Subversion it creates an ever growing trail that affects subsequent merges in unpleasant manner.

The svn:mergeinfo Property in Subversion

Subversion uses a special property on files and folders to record when a merge has taken place called “svn:mergeinfo”. The value of the property is a list of the branches (and associated range of revisions) that have been merged to date – irrespective of whether a physical change has been made to the item itself, e.g.

/branches/releases/1.0:1-1000
/branches/feature/cool_stuff:900-910
/branches/releases/2.0:1100-2000

Generally speaking it’s a good idea to merge using the root of a branch to ensure that you don’t miss anything (unless you’re cherry-picking changes of course). In Subversion this means that only the branch roots then need to have their svn:mergeinfo properties updated which keeps things clean and tidy. But if you cherry-pick a change Subversion then needs to maintain a merge list on that file for ever more, and more importantly for every future merge where that file is a child and therefore a potential candidate. The net outcome of this architecture is that you end up with a very noisy commit every time you merge because it is full of svn:mergeinfo property updates with the real code changes obscured.

ClearCase uses a similar technique (called hyperlinks) to achieve the same goal, but the difference is that it only records a link back to the single source version that contributed the changes – subsequent merges have no effect. Of course ClearCase is renowned for it’s slowness* and it’s possible that the Subversion architecture may improve the speed of merging at the cost of fidelity. The Revision Graph from TortoiseSVN doesn’t appear to have the wonderful merge arrows that the equivalent feature in ClearCase has and you can’t instigate merges from the TortoiseSVN Revision Graph – a feature I used heavily on a previous project - but maybe that says more about the quality of the Development Process than the tool…

Deleting the MergeInfo Properties

Now, if I interpret this post “Subversion merge reintegrate” correctly then the point of the mergeinfo property is so that Subversion knows what contributions have already been taken from other branches. The fact that Subversion keeps updating the upper revision in the property after each branch is merged reinforces my belief that this is an optimisation of some sort. In theory then you could remove those entries in the merginfo properties which reference dead branches. The only fly-in-the-ointment is that you probably won’t find a mergeinfo property with a single reference exactly because of the behaviour outlined above. So what about deleting the whole mergeinfo property on each file and folder?

C:\> svn propdel svn:mergeinfo –R

Obviously you can’t do this on the source branch as it’s the target of the branch that gets updated on a merge. So it would really have to be done on the trunk. If you know that you are never going to do a large scale merge again from any release branches (i.e. trunk is the only active branch) then I think wiping out all the svn:mergeinfo properties could be the quick solution. And you still have the ability to cherry-pick changes from an old branch should the need arise.

When I say all properties, there is one folder you should leave untouched – the root folder of the branch, e.g. /trunk or /branches/Release/0.1 or whatever. I believe that this advice is sound as long as you have not deleted any properties from files or sub-folders that have revisions outside the range specified at the root folder. The svn:mergeinfo property appears to be ‘inherited’ by its children so the act of deleting the property from the children doesn’t seem to stop Subversion from correctly inferring the merge set as long as a parent folder back up the tree has the right information.

Faking a Merge

One of the features I’ve seen in the TortoiseSVN UI that I’ve not yet used is the ability to record a merge without making any actual content changes. I’ve used similar features in the past when you have files you want to reconcile but the target version doesn’t physically need to change. The canonical example of this is a file that just contains the version number – it is different on every release branch but fixed to, say, 0.0.0 on the trunk. Merging these files will always cause a conflict so you only want to record the fact that the file has been ‘logically’ merged to ensure the entire merge is clean, i.e. no changes are left unaccounted for on the source branch.

I reckon this feature gives us a clean(ish) way to resolve the issue at strategic points in our development cycle – as long as we don’t perform long term development outside of the trunk. Here’s how I think it goes:-

1. Feature branches are created, then merged (reintegrated) at their roots and discarded. This means only the root folder needs to have its svn:mergeinfo up-to-date. Once discarded the merge information in the root serves no further purpose, but also causes no problems.
2. Release branches on the other hand may have changes cherry-picked off the trunk and other temporary branches and the reverse may also occur when the release is patched. This is where we start accumulating svn:mergeinfo properties on the child files and folders.
3. Once we reach a stable point where we no longer expect to take old contributions from our previous release branches we delete all the child svn:mergeinfo properties on the trunk (leaving one at the root) and record merges at the trunk root for each branch that we cherry-picked from. The net result should be the trunk root having merge records accurate up to today. Fixes made after today on the release branches should still show up in future merges as candidates.

I was going to add an additional condition on point 3 that there must be no open feature branches. My reservation is that Subversion may undo all this hard work when it comes to reintegrate a feature branch as there will be svn:mergeinfo properties on one branch but not the other. I haven’t done any experiments yet to see how Subversion handles this kind of property merging, but I will when a suitable opportunity arises.

It’s Easy To Google The Answer Once You Know It

When I first started looking into this issue, I didn’t really know what to search for and so didn’t find much. I then read bits of the SVN Book and started writing this post based on what I thought might be the answer. Once I’d finished it I went back and Googled “mergeinfo propdel” and what do you know? Yup, more posts about this issue than you can shake a stick at! I’m still publishing it though because I’ve not really seen one that explains why you can use svn propdel and many of them don’t mention keeping the mergeinfo intact at the branch root. In fact this is the most succinct answer I came across…

 

[*Personally I’ve never found the time waiting for ClearCase to generate the merge set on some 30,000 file views that excessive. At least not compared to the time you actually spend trying to decipher what code people have written to verify whether the merge is valid]

WCF Service Refusing Connections

I spent the majority of the day looking into a WCF issue with our system whereby the same request submitted in bulk against the same WCF service would fail indeterminately:-

System.ServiceModel.EndpointNotFoundException: Could not connect to http://localhost:8000/MyService. TCP error code 10061: No connection could be made because the target machine actively refused it 127.0.0.1:8000

I had noticed by log inspection that about 10 of the requests would succeed and the majority of the others would fail, this led me to believe that there was a default setting somewhere inside the WCF configuration that was having a throttling effect. When I finally got to the bottom of the issue I was somewhat annoyed with myself as it was a classic socket issue - not a WCF specific one. Therein I guess lies the problem with dealing with ever higher layers of abstraction – you begin to forget about the low-level details that underpin it (and which eventually leaks through). Fortunately my copy of Programming WCF Services by Juval Lowy arrived the other day so I had a chance to do some reading to see what possible setting could be having an effect.

ServiceThrottling “maxConcurrent*'”

Lowy dedicates an entire chapter to service request throttling. There are 3 settings:- maxConcurrentSessions, maxConcurrentCalls and maxConcurrentInstances described for the Service Throttling Behaviour that allow the maximum load on each service to be controlled. The effects of these settings are dependent on the transport and concurrency mode of the service, but it’s all spelt out clearly in the book. The most interesting of the lot for me was the number of concurrent sessions - which defaults to 10 under a Per-Session model which is what we are using.

I bumped the setting up to 1000 (way over the 100 needed to service my test harness load), but it had no effect. To humour myself I dropped all three settings to 1 to ensure it had some impact. It did. Oh.

NetTcpBinding “MaxConnections”

Back to the drawing board, or more accurately Google and the book. So convinced was I that I had discovered the cause that I never bothered to read on to the end of the chapter, where I would have discovered another setting that limits the maximum number of TCP connections for the binding:- maxConnections. This setting, which also has a default of 10, goes hand-in-hand with the previous settings as the smaller of it and maxConcurrentSessions becomes the effective throttle.

Once again I changed my service config only to become immediately disappointed as it also failed to have the desired result.

NetTcpBinding “listenBacklog”

Switching back to Google I read various posts about firewall issues which didn’t apply as I was running on the same machine. However I did start to pay closer attention to some of the settings that were being included in the App.config files. One in particular caught my eye – listenBacklog. Anyone who has done any raw sockets programming will know that when you create a server-side socket and start listening on it you need to specify how many pending connections you’re willing to buffer. I quickly Googled again to see what the default value was – yup another 10!

I quickly plugged my ridiculous value into the .config file and bingo! This time it worked. So, a new twist on a very old problem…