Codurance AI Hackathon

This is quite a long post (15 mins) and if you’re only interested in my final musings, and not the day itself, just skip to the last two sections.

Last Saturday I got to attend an event at Codurance’s offices in London around the use of AI based tooling to aid software delivery. I did not have much experience with these kinds of tools, in part because my current client uses their own in-house programming language [1]. Hence, my experiences to date have been limited to some messing around in VS Code with Copilot on some fairly simple C# katas.

Event Format

There were about twenty of us spread across a range of ages but, generally speaking, all fairly experienced programmers. We were split into two teams (A and B) and got to tackle two problems – one in the morning, and another in the afternoon. We got to spend 2 hours on each problem, with one team allowed to use AI tooling and the other not. Then, in the afternoon, we switched roles so that both teams got to try with and without AI assistance across the day. We also worked in pairs or threes. Both problems were very similar: effectively a web based service consisting of a backend API, that used some kind of database, and a frontend UI. If you’re thinking: an online clothing store / IMDB, then you’d be pretty close to the two exercises.

There were some questions about AI in the context of search engines as Google, for instance, puts its AI spin on the result. This was clarified as meaning no direct use of AI tooling for code generation and tooling suggestions. The non-AI teams had to rely on search engines / Stack Overflow, blog posts, docs, etc.

Another question (from yours truly) was about how much of a “hackathon” this really was, or whether we should approach it more like we were building something for real. The answer was that we should try and treat it more like a real project than something discardable like a prototype.

The Non-AI Exercise

I was in Team B, and for the morning session we weren’t allowed to use any AI tooling for our online clothing store. Ironically, trying to disable it to avoid biasing proved non-trivial, while still leaving the traditional IntelliSense and refactoring tools enabled. (Where does one end and the other begin…)

Our group of three had assembled because we all had C# in common. I’ve not done any non-trivial web UI work for twenty years, and it’s also been 5 years since I’ve done any production C# [1] so I wasn’t exactly up on the in-vogue architecture choices. Consequently I left it to the other two to suggest a Blazor based approach, which suited me as I had at least done some work with Razor 8 years ago, so was familiar with the underlying concepts.

The dataset we were given was hosted by Kaggle and came as a ZIP file that included a CSV file and some images. Trying to obtain this without having to sign-up to their service was another small barrier, eventually overcome when I unearthed the CURL instructions which thankfully didn’t require any authentication.

We all agreed to start by creating a walking skeleton that was a simple Blazor app. With some guidance from my colleagues, I used VS Code to knock that up and we were off with a basic web page in the browser.

We then looked at the dataset in a little more detail to see what shape it was in, and there was some discussion about next steps, such as pulling that entire CSV file in. That came off the back of the initial discussion about the model we should use to back the service. Yes, the categories aspect was an interesting modelling problem but I wanted to just to create a simple model with ID, Description, and Price, then hardcode a list of three items, and get that visible on the page to complete our skeleton. That idea was accepted and we got that working quite quickly.

At this point we all got a bit nervous as we hadn’t got any tests in our skeleton. Luckily one of us already had some experience with bUnit (a testing framework for Blazor), so we managed to get a test project and initial test in place relatively easily that checked for one of our hard-coded items being on the page.

This paved the way for starting to add new features in a test-first manner and, with what time we had left of the two hours, we started to work on the test for the product details page which comes from selecting an item on the products page. We had the failing test in place when the metaphorical whistle blew for half-time.

The AI Assisted Exercise

After lunch, where we had a chance to mingle and chat to others about their experiences so far, we tackled the IMDB clone with the addition of AI assistance.

We chose to remain in the same group, while others paired up with different people. We also thought it would be useful to use the same stack as that seemed like a useful comparison, although we did switch laptop and consequently from VS Code to JetBrain’s Rider. Once again there was some fiddling about with settings.

None of us had ever tried creating an entire project from scratch using an AI tool so we threw the README into the chat window prefixed by a sentence that asked it to create the entire solution and projects using Blazor as the stack.

The chat window started spewing out snippets of code which we then started to look at. I think we were all quite surprised about how plausible the code looked. We had some minor concerns about some of its design choices but it all looked pretty sane as a starting point. We hadn’t asked it to generate any tests, preferring instead to let this become an additional change later.

Trying to turn the suggested code in the chat window into an actual solution structure in Rider turned out to be our first time-sink. The UI was a little confusing and the laptop owner hadn’t toggled a setting which we later discovered allowed it to automatically create the files itself. Instead, to make progress in the meantime, we ended up creating the basic solution and project files ourselves, just like before. By the time we found the necessary setting we already had that aspect in place. However, once we enabled it we could at least take the suggestions for the model and Razor pages from the AI output.

This was the point when we discovered what looked plausible wasn’t entirely correct. For example, the code comment suggesting the relative path of the file didn’t match the file it created and it didn’t quite follow the Blazor convention. Hence, we moved a couple of files to follow the expected layout.

However, that said, when we built the code it worked first time and the web page appeared and looked totally sane. One of us remarked immediately that just getting the Bootstrap (UI) set-up right can be a real pain in practice and that alone was a real blessing.

Once we tried to interact with the page though we found it didn’t work. Cue the next time-sink, which was getting the filter feature to actually work. Naturally, we asked the AI tool what was wrong and it very confidently told us what was wrong and how to fix it. But, when I looked closely at the fix I pointed out that it had just re-written the existing code in a more complex manner but left it functionally the same. We went with it anyway but of course it didn’t work.

Another pointless suggestion from the AI tool about how to fix that didn’t inspire confidence at this point, so we resorted to using the debugger to see if the code was even being triggered. It wasn’t, although inspecting the DOM in the browser looked correct. I had raised a question earlier about session state management which didn’t initially sink in until a little later when our Blazor expert realised the page was missing some “interactive server” declaration at the top of the Razor page. Once that was added things started working.

It’s hard to know if the impedance mismatch was because we manually created the project files and the AI generated ones would have been correct for the code it was producing, or if it would still have been wrong.

Sadly, all this friction meant we had little time left to really explore the AI tool in the context of making changes to the existing codebase to add new features or refactor the code. We asked it a few cursory questions to see what it would suggest and it would always reply in a very confident way but would include changes that would revert some of our changes we made during our debugging attempts. I don’t know if that was a result of us not resetting the “context window” because those changes were ones it had originally suggested right at the beginning.

Reflections on the Experience

First up, I don’t know about how the others in my little group felt, but I was woefully underprepared for this day. With only two hours for each exercise we should have been in a position to hit the ground running and I just didn’t have enough experience with configuring or using the AI tool to make the most effective use of the little time available for the exercises.

Related to that, the problem domain wasn’t something I have any real experience of either. While this might have been useful for the “AI as a learning tool” aspect, I’m not sure it helps when trying to compare the two approaches.

With any day like this where you are working with other people you’ve never met before some element of “team dynamics” is going to distort the picture. My preference is to work in small steps, really small steps, and deciding whether to assert that or “go with the flow” is a much bigger side-quest than in an established team where you already know everyone’s favoured approach and politeness levels are well calibrated.

When you only have two hours there is a clear conflict between doing what you really would on a real project versus trying to explore the goals of the exercise. For example, due to my C# being a little rusty I wrote a classic POCO style class for the model instead using the more modern record types in C#, which I was aware of but had never actually used in practice. This prompted a short conversation about our feelings around immutability and anaemic models which is something a real team needs to resolve, but we simply don’t have time for that despite the “not really a hackathon” premise. Likewise, I have opinions on naming, design, etc. which I just have to put to one side this time.

However, I forgot this when writing our second test (in the first exercise) and naturally wrote a test name which opened a discussion about whether we were testing the navigation aspect or the state of the resulting page. Again, I think the approach to testing – the kinds of tests we want to write – is important in a real project. It would have been interesting to see what kinds of tests the AI tool would have generated as I see that being touted as a big use case, but given how poorly tests are commonly written I’m sceptical it has the corpus available to have “good taste”.

As I’ve already said above, I think we were all impressed with how much the tool churned out from the README, but are also aware that online stores and review sites are a well trodden path, with potentially a lot of example code to draw from. Other attendees got a lot further than we did, with one pair (using NodeJS) completing the exercise and having time to make up new features! In the post-exercise discussion one of that pair questioned how it would fair for, say, some embedded software where there are far less examples to train on.

The over confidence in the replies when we were trying to fix the code were scary. I think we were all sceptical and seeing it behave like this did not endear itself to us at all. Plus, trying to revert our code changes and only change the code it should need to when adding new features went against our agreed approach of only changing one thing at a time. Maybe we are being too rigid here and should cede more control in those early moments before we “hit production”?

We only got to work on producing code we had helped to write. It might have been interesting to have swapped at half-time and continue working on one of the other team’s AI generated codebases to see what it might be like to pick-up one of those as that is surely the future situation we’ll be finding ourselves in fairly soon.

Epilogue

I really enjoyed the day and am most grateful for the experience, but I don’t think I contributed that much personally to the experiment because of my current lack of knowledge around using these kinds of tools in practice.

I’ve only used Copilot integrated into VS Code on my laptop but would like to try some others based on the other group’s experiences, as long as they have tight integration into an IDE. I like baby steps and an “always be ready to ship” mentality which I’d feel uncomfortable giving up at this point in time. This is more like enhanced auto-complete based on what I’m already typing directly in the code than trying to write external prompts in a separate window which feels like unnecessary context switching when writing code.

Once I am more used to these new tools I’d like to try the exercise again and go “all in” and see what it chucks out, and then spend some significant time trying to work with that codebase to get a better feel for how much the AI tool wants to change each time. Automating the boilerplate stuff has to be worth exploring, but the question is how much leash to give it?

There is definitely a difference between building a disposable prototype and making changes in a large mature codebase. I think I’ve got more of a taste of the former now (coupled with my own fumblings beforehand) but I’m going to need more time to explore how this fits in with the latter as that’s my day job. At least I feel a lot more informed now than I did a week ago.

 

[1] My client has their own pure, functional, dynamic language which is used for much of the company’s reporting, scripting, and general gluing together of business workflows.

 

How Do I Test This?

I’ve no idea where I first came across the following premise for when starting a new code change:

“First ask yourself, how do I test this?”

I had assumed it was Steve Maguire’s Writing Solid Code or Debugging the Development Process as these are two of the earliest books I read as a professional developer. He has some excellent advice in “step through your code in the debugger” and “how could I have prevented / automatically detected this”. But it wasn’t either of those. No matter, I just like to remember my sources of inspiration.

Naturally, one shouldn’t take this advice too literally as there are other questions that really need answering first, such as whether this feature is even needed. The point is that once writing code is part of the solution, then thinking about how you’re going to test it should be right up there on your list. One behaviour that is common in those just starting out in their programmer journey is the desire to jump right in and start writing code without any thought for how they are going to answer the eventual question “does it work?”. And, by extension, “does it solve the customer’s problem?”

For a bug-fix, that might seem obvious, if the bug is easy to reproduce, though there is still the follow-up question of “have I broken anything else by accident?” Discovering up-front what you might break can have a sobering impact on your approach.

For new features the question can be much harder to answer, especially if the change is buried deep inside some complex system. It’s all very well being able to change some code and assume it works because the dry-run in your head suggests it will, but it’s another thing to be able to show that it works by running it in an observable way. This is doubly true when you factor in that famous quote from Edsger Dijkstra:

“Testing can prove the presence of bugs, but not their absence!”

Design for Testability

What both of these quotes try and convey is that testing requires a change in mindset, it’s not just some activity you throw in when it’s “code complete”. I’d hope in this modern era of software development that we’d all recognise the folly of believing that you can just bash out some code in a non-trivial codebase and assume it’ll work first time. Or even that you’ll obviously be able to see all the potential edge cases up front. (The average developer only spends 2-3 years in a role so we’re constantly putting ourselves back in the position of “new joiner”.)

To be clear, I’m not specifically talking about TDD (Test Driven Development) here, at least, not in the formal unit test / Red-Green-Refactor sense. I’m talking about it in a more general sense of how thinking about how you’re going to test something will affect how you approach the task at hand.

Plenty of what I do on a day-to-day basis does not have any formal automated tests produced as a by-product of it. For example, I am often heavily involved in the build, deployment, and support side of software delivery as much as writing code for the main product. While the latter will have a barrage of automated tests to help validate every change, the former often relies on an amount of manual testing. That’s because the cost benefit of introducing automated testing to a bunch of ad-hoc Bash glue scripts which rarely change is typically very high in comparison to the cost of testing them manually.

However, what makes the cost of testing them manually much lower is not the inherent nature of them, but the mindset used in how they were written – they were designed with testability in mind. What that means in practice is that instead of writing only the bare minimum to solve the problem at hand, some additional effort is made to allow it to be tested safely and quickly, both now and when future changes are made. It’s just another trade-off we have to be cognisant of and weigh up whether it’s worth the extra effort up-front – is being easier to test worth it? If so, how much easier is enough? (There is always an XKCD, and Is it Worth the Time along with Automation are well worth the time revisiting every now and then to remind yourself of the potential costs and pay-off, and delusions of grandeur.)

Risk / Confidence

When I say “easier” you shouldn’t just assume I mean “quicker” either. Testing is about risk, and developing confidence. The goal is to deliver the highest level of confidence with the lowest amount of effort. Printing out a log message saying that you would have deleted a particular file gives you confidence that you’d have deleted the correct file, but can’t point out that the entire process aborted because another process typically has it locked and now you might need to add logic to cater for retries and partial success. The principle of YAGNI (You ‘Aint Gonna Need It) is a strong force for reining in purely speculative requirements but can also be weaponised as an excuse for why anything outside the happy path was ignored.

My point here is that you consciously consider how you’re going to gain confidence and weigh up the options. Maybe when staring at that log message you’ll ask yourself “what would happen if that file is locked?” and consequently you go and adjust the code to allow you to target a different folder where you can create a locked file and test your hypothesis. Is that a temporary hack or do you introduce a command line switch / config setting [1]? If you’re going to keep hacking the code every time you test new changes then you run a higher risk of committing that hack by accident and breaking production [2].

Scope

We could very easily expand that original phrase “test this” to the more specific “test what happens when it fails”.  Only testing the happy paths is such a common malaise, although I suspect it is largely borne out of naivety rather than short-sightedness. The cost of debugging code that fails in a horrible way is huge, and frequently occurs as the result of a production incident when time is of the essence. Taking the time up-front to think about failure scenarios and test how they will manifest will really help reduce stress levels when the inevitable happens. (Stack Traces are a crutch we lean on far too often when a good error message could provide better value.)

Another tweak would be to replace the “I” with “we” in the original proposition. The person who finds the code difficult to test in future might still be you, or it might be a colleague. I don’t particularly like that analogy of writing code as if some psychopath will hunt you down later, and prefer instead to try and promote “paying it forward” as simply the right thing to do. The ultimate outcome is a healthier team, and codebase, and a faster, more sustainable delivery pace. Doing it right actually benefits the business in the long run.

Always be Testing

The question “how do I test this?” doesn’t necessarily stop being asked once the code has been delivered either. As one of my earlier posts (Validate in Production) shows, Dijkstra was right, and we need to be more pessimistic about our ability to get things right, first time, every time. Remember though there is a balance here. If you take Dijkstra at his word then you’ll never deliver anything as you’ll spend your entire time trying prove the absence of any bugs. (Formal Methods might be the exception here but I have no experience of that, or know anybody that has.)

It’s also only a short hop from that question to a closely related one – “how do I support this?” Ideally any separate support or operations team are already recognised as stakeholders in the system and their interests are explicitly factored in [3].

Taking this yet another step further leads you into the whole world of observability and ultimately around testing the business hypothesis of whether building that new feature delivered the benefits expected. Having that mindset where you can work backwards from “how to test the business impact” has yet more benefits that can be reaped.

Remember Overthinking is not Overengineering.

 

[1] See Testing Drives the Need for Flexible Configuration for further thoughts on that approach.

[2] This is one of the many things in my mental Commit Checklist which has come from making these mistakes myself in the past.

[3] Both Support-Friendly Tooling and the much older From Test Harness To Support Tool look at this angle.

Using CoPilot-Like Tools is Not Pairing

Ever since the rise of LLM based tools like ChatGPT and CoPilot there have been quips made about how great it makes as a pairing partner. The joke is because these tools are passive and won’t trash your code, or disagree with your approach, or smell of garlic.

I get the joke.

The problem for me is that these jokes are suggesting that using tools like CoPilot are akin to pairing, but they’re not. And, as a consequence, they are continuing to perpetuate misconceptions about what pairing is really about. This topic came up one Tuesday at a Cambridge Software Crafters meet-up when we were discussing mentoring & coaching and how to adopt pairing in companies that are hostile towards it, likely for these very reasons.

(If you’re after a longer read on this subject I can offer you my 2018 C Vu article To Mob, Pair, or Fly Solo, or if you’re a video kind of person then you might want to watch the talky version I gave at the ACCU 2024 Conference, also titled Mob, Pair, or Fly Solo.)

Tools like CoPilot are there to help you with the “mechanics” of writing code [1]. They complement the traditional tools like compilers and linters which help avoid a bunch of annoying problems that come with creating software as part of a solution. Your pairing partner may help you avoid these mistakes too, especially if you’re new to the language or tool, but that’s not primarily what they are there for.

Their input should focus on helping you to solve the actual problem. That might be at a design level, such as how best to represent the solution using common idioms or patterns, because they can see where it might be heading. They can also help come up with potential problems or edge cases that need to be factored in, and help decide how and when that might happen in the evolution of the feature you’re working on.

I’ve never paired in the strict navigator-driver sense, it’s always tended to be far more free-flowing, likewise when ensemble programming too. The keyboard tends to go back-and-forth with “type, don’t talk” being the trigger to switch, when it’s not blindingly obvious what my partner(s) are trying to convey.

Hence, in a TDD style loop the other half of the pair might already be thinking about the next test or where the design might be going for the refactoring step after the test passes. Sometimes one half is on a roll and it’s better to maintain momentum, other times both minds need to focus on the exact problem at hand, such as when the test fails unexpectedly. (If you both missed it then that is a smell worth exploring because something is clearly amiss somewhere.)

With modern IDEs and syntax highlighting the typist knows when there is a typo, they don’t need to be told by their partner. However the way they drive the IDE and other tools can often be a point of discussion as there are often many shortcuts and extensions that provide easy wins. High-end tooling often has an ability to try and point out other features you might find useful, and ChatGPT like tools could probably summarise related features in popular tools if asked.

Humans often remember things better when the correction happens at the point of triggering, and what your partner can see that your tooling can’t, is whether or not it’s an appropriate moment to make  a suggestion. There is a time and a place to share knowledge and part of the art of pairing is to read the mood and only interject if it’s likely to be beneficial now, or note it for a retrospective conversation later. It may even resolve itself automatically, later.

Sharing knowledge takes practice. For instance, sharing the right level of detail or techniques and idioms, to avoid overwhelming your partner while they are trying to focus on the problem at hand. The new generation of coding tools might be well versed in programming language details and common patterns but they don’t know anything about your organisation, team, or design principles. They might help with the small picture but they can’t help with the medium and bigger pictures, and that is the bit that makes software delivery hard. That is also the area where your partner should be adding the most value to the exercise.

Programming is so much more than just writing code, and programming in pairs and groups is about active participation to triangulate on a sustainable solution far quicker than working with passive tools and feedback.

 

[1] Maybe one day they’ll be able to consume your entire codebase, along with the VCS history, so that they can tell what’s old and what’s new and make better design suggestions too based on evolving trends. We’ll see…

 

Unpacking Code Ownership

This post was prompted by a document I read which was presented as a development guide. While the rest of it was about style, the section that particularly piqued my interest was one involving code ownership. For those of us who’ve been around the block, the term “code ownership” can bring with it connotations of protectionism. If you’ve never worked with people who are incredibly guarded about the code they write may I recommend my 2017 blog post Fallibility which contains two examples of work colleagues that erected a wall around themselves and their code.

While I initially assumed the use of the term was a proxy for accountability, some comments to my suggestion that Relentless Refactoring was an established practice in many teams hinted that there might be more to it than that. What came out of an online meeting of the team was that the term was carrying the weight of two different characteristics.

(I should point out that I’m always wary of this kind of discussion verging into bike-shedding territory. I like to try and ensure that language is only as precise as necessary, so when I suspect there may be confusion or suboptimal behaviour as a consequence, do I feel it’s worth digging deeper. In this instance I think “ownership” was referring to the following attributes and not about gatekeeping or protectionism for selfish reasons, e.g. job security.)

Accountability / Responsibility

When people talk about “owning your mistakes” what they’re referring to is effectively being accountable, nay responsible, for them. While there might be a legal aspect in the cases of Machiavellian behaviour, for the most part what we’re really after is some indication that changes were not made simply because “you felt like it”. Any code change should be justifiable which implies that there is an air of objectivity around your rationale.

For example, reformatting the code simply because you personally prefer a different brace placement is not objective [1]. In contrast, reformatting to meet the pre-agreed in-house style is. Likewise applying any refactoring that brings old code back in line with the team’s preferred idioms is inherently sound. Moreover, neither of these should even require any debate as the guide automatically confers agreement [2].

Where it might get more contentious is when it appears to be superfluous, but as long as you can justify your actions with a sense of objectivity I think the team should err on the side of acceptance [3]. The reason I think this kind of change can end up being rejected by default is when there is nothing in the development process to allow the status quo to be challenged. A healthy development process should include time for retrospection (e.g. a formal retrospective) and this is probably the place for further debate if it cannot quickly be resolved. (You should not build “inventory” in the form of open PRs simply because of unresolved conflict [4].)

One scenario where this can be less objective is when trying to introduce new idioms, i.e. experimental changes that may or may not set a new precedent. I would expect this to solicit at least some up-front discussion or proactive reviewing / pairing to weed out the obvious chaff. Throwing “weird” code into the codebase without consulting your teammates is disrespectful and can lead to unnecessary turf wars.

Being accountable also implies that you are mature enough to deal with the consequences if the decision doesn’t go your way, aka Egoless Programming [5]. That may involve seeing your work rejected or rewritten, either immediately or in the future which can feel like a personal attack, but shouldn’t.

Experience / Expertise

While accountability looks at ownership from the perspective of the person wanting to change the code, the flipside of ownership is about those people best placed to evaluate change. When we look for someone to act as a reviewer we look for those who have the most experience either directly with the code itself, or from working on similar problems. There may also be different people that can provide a technical or business focused viewpoint if there are both elements at play which deserve special attention, for example when touching code where the previous authors have left and you need help validating your assumptions.

In this instance what we’re talking about are Subject Matter Experts. These people are no more “owners” of the code in question than we are but that doesn’t mean they can’t provide useful insights. If anything having people unrelated to the code reviewing it can be more useful because you know they will have no emotional attachment to it. If the change makes sense feature-wise, and does it in a simple, easy to understand way, does anything else really matter?

These days we have modern tooling like version control products which, assuming we put the right level of metadata in, allow us to see the evolution of the codebase along with the rationale even when the authors have long gone. Ownership doesn’t have to be conferred simply because you’re the only one that remembers how and why the code ended up the way it did. This leads into territory around fear of change which is not a sustainable approach to software delivery. In this day-and-age “consulting the elders” should really be a last resort for times when the record of events is lost in the sands of time. Approval should be a function based on knowledge of the subject matter rather than simply years of service [6].

Shepherds, Not Owners

Ultimately what I find slightly distasteful about the term “shared ownership” is that it still conveys a sense of protectionism, especially for those currently “outside the team”.

From a metaphorical point of view what I think I described above is more a sense of shepherding. The desire should be to nurture contributors to understand the culture of the codebase and product to the extent that the conversations can focus on the essential, rather than accidental complexity.

I wonder if “shared mentorship” would work as a substitute?

 

[1] This is a good argument for using a standard code formatting tool as it can make these debates moot. 

[2] If the code is that performance sensitive it should not be touched without consultation then there should either be some performance tests or at a minimum some comments to make that obvious.

[3] The late Pieter Hintjens makes a compelling case in Why Optimistic Merging Works Better.

[4] There is where I favour the optimism of Trust, but Verify as an approach, or pairing / ensemble programming to reach early consensus.

[5] The Psychology of Computer Programming – Gerry Weinberg, 1971.

[6] One needs to be mindful of not falling into the Meritocracy trap though.

Planning is Inevitable

Like most programmers I’ve generally tried to steer well clear of getting involved in management duties. The trouble is that as you get older I think this becomes harder and harder to avoid. Once you get the mechanics of programming under control you might find you have more time to ponder about some of those other duties which go into delivering software because they begin to frustrate you.

The Price of Success

Around the turn of the millennium I was working in a small team for a small financial organisation. The management structure was flat and we had the blessing of the owner to deliver what we thought the users needed and when. With a small but experienced team of programmers we could adapt to the every growing list of feature requests from our users. Much of what we were doing at the time was trying to work out how certain financial markets were being priced so there was plenty of experimentation which lead to the writing and rewriting of the pricing engine as we learned more.

The trouble with the team being successful and managing to reproduce prices from other more expensive 3rd party pricing software was that we were then able to replace it. But of course it also has some other less important features that users then decided they needed too. Being in-house and responsive to their changes just means the backlog grows and grows and grows…

The Honeymoon Must End

While those users at the front of the queue are happy their needs are being met you’ll end up pushing others further down the queue and then they start asking when you’re going to get around to them. If you’re lucky the highs from the wins can outweigh the lows from those you have to disappoint.

The trouble for me was that I didn’t like having to keep disappointing people by telling them they weren’t even on the horizon, let alone next on the list. The team was doing well at delivering features and reacting to change but we effectively had no idea where we stood in terms of delivering all those other features that weren’t being worked on.

MS Project Crash Course

The company had one of those MSDN Universal licenses which included a ton of other Microsoft software that we never used, including Microsoft Project. I had a vague idea of how to use it after seeing some plans produced by previous project managers and set about ploughing through our “backlog” [1] estimating every request with a wild guess. I then added the five of us programmers in the team as the “resources” [2] and got the tool to help distribute the work amongst ourselves as best as possible.

I don’t remember how long this took but I suspect it was spread over a few days while I did other stuff, but at the end I had a lovely Gantt Chart that told us everything we needed to know – we had far too much and not enough people to do it in any meaningful timeframe. If I remember correctly we had something like a year’s worth of work even if nothing else was added to the “TODO list” from now on, which of course is ridiculous – software is never done until it’s decommissioned.

For a brief moment I almost felt compelled to publish the plan and even try and keep it up-to-date, after all I’d spend all that effort creating it, why wouldn’t I? Fortunately I fairly quickly realised that the true value in the plan was knowing that we had too much work and therefore something had to change. Maybe we needed more people, whether that was actual programmers or some form of manager to streamline the workload. Or maybe we just needed to accept the reality that some stuff was never going to get done and we should ditch it. Product backlogs are like the garage or attic where “stuff” just ends up, forgotten about but taking up space in the faint hope that one day it’ll be useful.

Saying No

The truth was uncomfortable and I remember it lead to some very awkward conversations between the development team and the users for a while [3]. There is only so long that you can keep telling people “it’s on the list” and “we’ll get to it eventually” before their patience wears out. It was unfair to string people along when we pretty much knew in our hearts we’d likely never have the time to accommodate them, but being the eternal optimists we hoped for the best all the same.

During that period of turmoil having the plan was a useful aid because it allowed is to have those awkward conversations about what happens if we take on new work. Long before we knew anything about “agility” we were doing our best to respond to change but didn’t really know how to handle the conflict caused by competing choices. There was definitely an element of “he who shouts loudest” that had a bearing on what made its way to the top of the pile rather than a quantitative approach to prioritisation.

Even today, some 20 years on, it’s hard to convince teams to throw away old backlog items on the premise that if they are important enough they’ll bubble up again. Every time I see an issue on GitHub that has been automatically closed because of inactivity it makes me a little bit sad, but I know it’s for the best; you simply cannot have a never-ending list of bugs and features – at some point you just have to let go of the past.

On the flipside, while I began to appreciate the futility of tracking so much work, I also think going through the backlog and producing a plan made me more tolerant of estimates. Being that person in the awkward situation of trying to manage someone’s expectations has helped me get a glimpse of what questions some people are trying to answer by creating their own plans and how our schedule might knock onto them. I’m in no way saying that I’d gladly sit through sessions of planning poker simply for someone to update some arbitrary project plan because it’s expected of the team, but I feel more confident asking the question about what decisions are likely to be affected by the information I’m being asked to provide.

Self-Organising Teams

Naturally I’d have preferred someone else to be the one to start thinking about the feature list and work out how we were going to organise ourselves to deal with the deluge of work, but that’s the beauty of a self-organising team. In a solid team people will pick up stuff that needs doing, even if it isn’t the most glamourous task because ultimately what they want is to see is the team succeed [4], because then they get to be part of that shared success.

 

[1] B.O.R.I.S (aka Back Office Request Information System) was a simple bug tracking database written with Microsoft Access. I’m not proud of it but it worked for our small team in the early days :o).

[2] Yes, the air quotes are for irony :o).

[3] A downside of being close to the customer is that you feel their pain. (This is of course a good thing from a process point of view because you can factor this into your planning.)

[4] See “Afterwood – The Centre Half” for more thoughts on the kind of role I seem to end up carving out for myself in a team.

Cargo Culting GitFlow

A few years back I got to spend a couple of weeks consulting at a small company involved in the production of smart cards. My team had been brought in by the company’s management to cast our critical eye over their software development process and provide a report on what we found along with any recommendations on how it could be improved.

The company only had a few developers and while the hardware side of the business seemed to be running pretty smoothly the software side was seriously lacking. To give you some indication of how bad things used to be, they weren’t even using version control for their source code. Effectively when a new customer came on board they would find the most recent and relevant existing customer’s version (stored in a .zip file), copy their version of the system, and then start hacking out a new one just for the new customer.

As you can imagine in a set-up like this, if a bug is found it would need to be fixed in every version and therefore it only gets fixed if a customer noticed and reported it. This led to more divergence. Also as the software usually went in a kiosk the hardware and OS out in the wild was often ancient (Windows 2000 in some cases) [1].

When I say “how bad things used to be” this was some months before we started our investigation. The company had already brought in a previous consultant to do an “Agile Transformation” and they had recognised these issues and made a number of very sensible recommendations, like introducing version control, automated builds, unit testing, more collaboration with the business, etc.

However, we didn’t think they looked too hard at the way the team were actually working and only addressed the low hanging fruit by using whatever they found in their copy of The Agile Transformation Playbook™, e.g. Scrum. Naturally we weren’t there at the time but through the course of our conversations with the team it became apparent that a cookie-cutter approach had been prescribed despite it being (in our opinion) far too heavyweight for the handful of people in the team.

As the title of this post suggests, and the one choice I found particularly amusing, was the introduction of VSTS (Visual Studio Team Services; rebranded Azure DevOps) and a GitFlow style workflow for the development team. While I applaud the introduction of version control and isolated, repeatable builds to the company, this feels like another heavyweight choice. The fact that they were already using Visual Studio and writing their web service in C# probably means it’s not that surprising if you wanted to pick a Big Iron product.

The real kicker though was the choice of a GitFlow style workflow for the new product team where there were only two developers – one for the front-end and another for the back-end. They were using feature branches and pull requests despite the fact that they were the only people working in their codebase. While the company might have hired another developer at some point in the future they had no immediate plans to to grow the team to any significant size [2] so there would never be any merge conflicts to resolve in the short to medium term! Their project was a greenfield one to create a configurable product instead of the many one-offs to date, so they had no regressions to worry about at this point either – it was all about learning and building a prototype.

It’s entirely possible the previous consultant was working on different information to us but there was nothing in our conversations with the team or management that suggested they previously had different goals to what they were asking from us now. Sadly this is all too common an occurrence – a company hires an agile coach or consultant who may know how to handle the transformation from the business end [3] but they don’t really know the technical side. Adopting an agile mindset requires the XP technical practices too to be successful and so, unless the transformation team really knows its development onions, the practices are going to be rolled out and applied with a cargo cult mentality instead of being taught in a way that the team understands which practices are most pertinent to their situation and why.

In contrast, the plan we put forward was to strip out much of the fat and focus on making it easy to develop something which could be easily demo’d to the stakeholders for rapid feedback. We also proposed putting someone who was “more developer than scrum master” into the team for a short period so they could really grok the XP practices and see why they matter. (This was something I personally pushed quite hard for because I’ve seen how this has played out before when you’re not hands-on, see “The Importance of Leading by Example”.)

 

[1] Luckily these kiosks weren’t connected to a network; upgrades were a site visit with a USB stick.

[2] Sadly there were cultural reasons for this – a topic for another day.

[3] This is debatable but I’m trying to be generous here as my expertise is mostly on the technical side of the fence.

Git is Not the Problem

Git comes in for a lot of stick for being a complicated tool that’s hard to learn, and they’re right, git is a complicated tool. But it’s a tool designed to solve a difficult problem – many disparate people collaborating on a single product in a totally decentralized fashion. However, many of us don’t need to work that way, so why are we using the tool in a way that makes our lives more difficult?

KISS

For my entire professional programming career, which now spans over 25 years, and my personal endeavours, I have used a version control tool (VCS) to manage the source code. In that time, for the most part, I have worked in a trunk-based development fashion [1]. That means all development goes on in one integration branch and the general philosophy for every commit is “always be ready to ship” [2]. As you might guess features toggles (in many different guises) play a significant part in achieving that.

A consequence of this simplistic way of working is that my development cycle, and therefore my use of git, boils down to these few steps [3]:

• clone
• edit / build / test
• diff
• add / commit
• pull
• push

There may occasionally be a short inner loop where a merge conflict shows up during the pull (integration) phase which causes me to go through the edit / diff / commit cycle again, but by-and-large conflicts are rare due to close collaboration and very short change cycles. Ultimately though, from the gazillions of commands that git supports, I mostly use just those 6. As you can probably guess, despite using git for nearly 7 years, I actually know very little about it (command wise). [4]

Isolation

Where I see people getting into trouble and subsequently venting their anger is when branches are involved. This is not a problem which is specific to git though, you see this crop up with any VCS that supports branches whether it be ClearCase, Perforce, Subversion, etc. Hence, the tool is not the problem, the workflow is. And that commonly stems from a delivery process mandated by the organization, meaning that ultimately the issue is one of an organizational nature, not the tooling per-se.

An organisation which seeks to reduce risk by isolating work (and by extension its people) onto branches is increasing the delay in feedback thereby paradoxically increasing the risk of integration, or so-called “merge debt”. A natural side-effect of making it harder to push through changes is that people will start batching up work in an attempt to boost "efficiency”. The trick is to go in the opposite direction and break things down into smaller units of work that are easier to produce and quicker to improve. Balancing production code changes with a solid investment in test coverage and automation reduces that risk further along with collaboration boosting techniques like pair and mob programming.

Less is More

Instead of enforcing a complicated workflow and employing complex tools in the hope that we can remain in control of our process we should instead seek to keep the workflow simple so that our tools remain easy to use. Git was written to solve a problem most teams don’t have as they neither have the volume of distributed people or complexity of product to deal with. Organisations that do have complex codebases cannot expect to dig themselves out of their hole simply by introducing a more powerful version control tool, it will only increase the cost of delay while bringing a false sense of security as programmers work in the dark for longer.

 

[1] My “Branching Strategies” article in ACCU’s Overload covers this topic if you’re looking for a summary.

[2] This does not preclude the use of private branches for spikes and/or release branches for hotfix engineering when absolutely needed. #NoAbsolutes.

[3] See “In The Toolbox - Commit Checklist” for some deeper discussion about what goes through my head during the diff / commit phase.

[4] I pondered including “log” in the list for when doing a spot of software archaeology but that is becoming much rarer these days. I also only use “fetch” when I have to work with feature branches.

Validate in Production

The change was reasonably simple: we had to denormalise some postcode data which was currently held in a centralised relational database into some new fields in every client’s database to remove some cross-database joins that would be unsupported on the new SQL platform we were migrating too [1].

As you might imagine the database schema changes were fairly simple – we just needed to add the new columns as nullable strings into every database. The next step was to update the service code to start populating these new fields as addresses were added or edited by using data from the centralised postcode database [2].

At this point any new data or data that changed going forward would have the correctly denormalised state. However we still needed to fix up any existing data and that’s the focus of this post.

Migration Plan

To fix-up all the existing client data we needed to write a tool which would load each client’s address data that was missing its new postcode data, look it up against the centralised list, and then write back any changes. Given we were still using the cross-database joins in live for the time being to satisfy the existing reports we could roll this out in the background and avoiding putting any unnecessary load on the database cluster.

The tool wasn’t throw-away because the postcode dataset gets updated regularly and so the denormalised client data needs to be refreshed whenever the master list is updated. (This would not be that often but enough to make it worth spending a little extra time writing a reusable tool for the job for ops to run.)

Clearly this isn’t rocket science, it just requires loading the centralised data into a map, fetching the client’s addresses, looking them up, and writing back the relevant fields. The tool only took a few hours to write and test and so it was ready to run for the next release during a quiet period.

When that moment arrived the tool was run across the hundreds of client databases and plenty of data was fixed up in the process, so the task appeared completed.

Next Steps

With all the existing postcode data now correctly populated too we should have been in a position to switch the report generation feature toggle on so that it used the new denormalised data instead of doing a cross-database join to the existing centralised store.

While the team were generally confident in the changes to date I suggested we should just do a sanity check and make sure that everything was working as intended as I felt this was a reasonably simple check to run.

An initial SQL query someone knocked up just checked how many of the new fields had been populated and the numbers seemed about right, i.e. very high (we’d expect some addresses to be missing data due to missing postcodes, typos and stale postcode data). However I still felt that we should be able to get a definitive answer with very little effort by leveraging the existing we SQL we were about to discard, i.e. use the cross-database join one last time to verify the data population more precisely.

Close, but No Cigar

I massaged the existing report query to show where data from the dynamic join was different to that in the new columns that had been added (again, not rocket science). To our surprise there were quite a significant number of discrepancies.

Fortunately it didn’t take long to work out that those addresses which were missing postcode data all had postcodes which were at least partially written in lowercase whereas the ones that had worked were entirely written in uppercase.

Hence the bug was fairly simple to track down. The tool loaded the postcode data into a dictionary (map) keyed on the string postcode and did a straight lookup which is case-sensitive by default. A quick change to use a case-insensitive comparison and the tool was fixed. The data was corrected soon after and the migration verified.

Why didn’t this show up in the initial testing? Well, it turned out the tools used to generate the test data sets and also to anonymize real client databases were somewhat simplistic and this helped to provide a false level of confidence in the new tool.

Testing in Production

Whenever we make a change to our system it’s important that we verify we’ve delivered what we intended. Oftentimes the addition of a feature has some impact on the front-end and the customer and therefore it’s fairly easy to see if it’s working or not. (The customer usually has something to say about it.)

However back-end changes can be harder to verify thoroughly, but it’s still important that we do the best we can to ensure they have the expected effect. In this instance we could easily check every migrated address within a reasonable time frame and know for sure, but on large data sets this might unfeasible so you might have to settle for less. Also the use of feature switches and incremental delivery meant that even though there was a bug it did not affect the customers and we were always making forward progress.

Testing does not end with a successful run of the build pipeline or a sign-off from a QA team – it also needs to work in real life too. Ideally the work we put in up-front will make that more likely but for some classes of change, most notably where actual customer data is involved, we need to follow through and ensure that practice and theory tie up.

 

[1] Storage limitations and other factors precluded simply moving the entire postcode database into each customer DB before moving platforms. The cost was worth it to de-risk the overall migration.

[2] There was no problem with the web service having two connections to two different databases, we just needed to stop writing SQL queries that did cross-database joins.

Feature Branches and Package Dependencies

For most of my programming career I have worked directly on the main integration branch (aka trunk / master) for day-to-day development. Release branches have featured occasionally at various clients, mostly to compensate for bureaucracy, and I once had the misfortune to work with project-level branches (see “The Cost of Long-Lived Feature Branches”) which was a merge nightmare. More recently I got to work in a team that used much shorter lived feature branches [1] and I got a reminder of the kinds of problems even they cause.

When the branch is confined to a single repository and that repo is for the delivered product then the only people who are affected by the changes are those working on the branch. (We’re not talking about the Cost of Delay here for the customer, this is about intra-team delays.) However once we start making changes outside the main repository, such as our library repos (nay packages) things get more complicated.

Changing Packages

Although in theory we could create a similar branch in the library repo the point of integration between the two codebases (product and library) is usually at a binary level, i.e. the package repository. The package manager almost certainly doesn’t deal in branches per-se, only published versions of a package [2].

Where things get even more complicated is when you have a few packages that all make use of some 3rd party library, e.g. a message queuing product, and you discover you need to upgrade that as part of your feature work too. If you go via the normal channels you’ll end up upgrading the dependency in the packages, publish them, and then pull those upgraded packages into your feature branch and carry on where you left off [3].

Dependent Changes

However, anyone working on another feature branch or even the trunk can no longer make an orthogonal change to those packages because pulling them in would likely create an impedance mismatch, unless they also duplicate the integration work done on the feature branch or cherry pick it. Ideally that would be a trivial merge but the very nature of feature branches is to work in isolation and therefore changes tend to get intertwined because the focus is on the feature itself, not the integration steps in-between.

Essentially until the new package versions are fully integrated any changes to them will be delayed, which if you’ve already started work means a blocker goes up on the board. In this instance, being used to trunk based development, I didn’t want to wait so instead reverted the upgrade to the 3rd party library, published it (with my changes), and then integrated it into the main product directly on the trunk.

Unfortunately this creates an extra burden for those on the feature branch as they will need to re-upgrade the package again before integrating their changes back into the trunk. Such is the price for working in isolation.

Small is Beautiful

One of the benefits of working in an Agile way using trunk based development is that it teaches you to focus of really small, but nonetheless valuable changes. A user story may be split across a number of commits and so we have to think about the way we’ll deliver the changes. Feature toggles help us to hide our work in progress but, as just described, occasionally we may need to make a more sweeping change.

In these scenarios we should be able to push the feature work temporarily onto the “stack”, make the package changes, and then pop the feature work and carry on. With trunk based development the work is woven in just like any other, but when using a feature branch you need to switch back to the trunk, perform the upgrade, then merge trunk into the feature branch before carrying on.

I believe that if you are able to make this work smoothly using a feature branch then you are almost certainly capable of making the changes directly on the trunk in the first place. Planning doesn’t stop at the release and sprint level, we also need to plan how we evolve the codebase at story and task level too to minimise disruption to others whilst also making progress on our own work.

 

[1] They only lasted a few days and were trying to move away from that where possible.

[2] In theory this is where sematic versioning comes into play as the breaking change demands a new major version. My change would then have been made for both major versions. I say “in theory” because In my experience this is not an approach commonly taken by enterprise development teams for internal libraries – the path of least resistance usually wins.

[3] Alternatively you may be able to hide the library’s dependency, assuming it’s backwardly compatible, by binding to it directly such as via a static library or merging assemblies. However, as with [2], it’s not the usual approach.

“Hello World” Stories

I’ve always tried really hard to fight against “technical stories”. These are supposedly user stories but which are really framed as a solution to a problem and really just technical tasks. In “Turning Technical Tasks Into User Stories” I looked at how it’s often possible to elevate these from an obvious solution to a problem back up to a problem which needs to be solved. At this point you may discover there are other, hopefully cheaper, solutions to the problem which have been missed in the original analysis either because things have changed or different people are doing the thinking.

On the flip-side there are occasionally times where, after having looked at a few related stories, it’s apparent that they all require the same underlying mechanism to work. One common solution to this is to bulk up the first story with the technical work and let the rest flow through as normal. This way you have no technical work on your backlog per-se as it’s all hidden in the stories.

Transparency

What I don’t like about this approach is that one story arbitrarily gets hit with a load of extra work, which, if you’re using historical data to stick a finger in the air for estimation of similar work later, skews the average somewhat. It also means that from a visibility perspective one story takes longer while the mechanism is being built.

One way I’ve found to address this has been to pull out the bare bones of the technical work into a “Hello, World!” story [1]. This story is framed around building the skeleton of the mechanism that will be used to drive the implementation of the subsequent features. The aim is keep the scope minimal enough that we avoid speculating while still delivering something which stands on its own two feet and remains clearly visible on the board.

Value Proposition

While the value to the end-user is in the eventual feature, the value in the mechanism is proving to the development team that the basic approach seems sound. With the skeleton built, the idiosyncrasies around each individual feature can then be dealt with appropriately at the right time and accounted for in the usual way.

To be clear this is not about doing a spike or building a prototype, although that may have happened earlier to gain the knowledge needed to undertake this piece of work. No, here we’re talking about building the bare bones of a real mechanism along with the most basic feature possible.

The reason I’ve called these “Hello World” Stories is probably self-evident, it alludes to the classic program many have chosen as their first – to write “Hello, World!” to the console. In this context the name is intended to conjure up simplicity and remind us that what we’re doing is delivering the minimum required to make the platform viable. We probably won’t literally write “Hello, World!” to the console, but it may a log message instead that we can then observe and monitor, or be a message on a queue that we can see discarded. Essentially whatever we can do to make its effects observable without wasting any real effort or leaving it partially complete.

Based on the classic INVEST acronym we should strive to make every unit of work: Independent, Negotiable, Valuable, Estimable, Small and Testable. By splitting it out from one of the arbitrary features it becomes more independent, negotiable, estimable and small which can be useful should short-term priorities change. And by extending the scope from a pure mechanism just a little bit further to the most trivial feature possible we make it more testable from a technical perspective, even if not from a product viewpoint. Most importantly, however, is it valuable in its own right? I think sometimes splitting the mechanism out gives value by making the I,N,E,S and T more tangible. In particular breaking work down into smaller deliverable units is often the most valuable practice even if occasionally the end-user has nothing initially to show for it.

Ultimately, I guess, I can’t ever remember anyone complaining they had broken their work down into pieces that were so small they were too visible.

 

[1] I’m sure there is an argument about this not being a “story” per-se but just a “task”. However I prefer to call it a story because our “Hello, World!” realization should have a grounding in the real world, even if it is more abstract than what the end-user will eventually receive.

[2] There is an assumption here that we’ve already decided we cannot or do not want to solve the dependent features in different ways, probably because it would be far more costly (in the long run) than briefly delaying them by building a common pillar.

Overthinking is not Overengineering

As the pendulum swings ever closer towards being leaner and focusing on simplicity I grow more concerned about how this is beginning to affect software architecture. By breaking our work down into ever smaller chunks and then focusing on delivering the next most valuable thing, how much of what is further down the pipeline is being factored into the design decisions we make today?

Wasteful Thinking

Part of the ideas around being leaner is an attempt to reduce waste caused by speculative requirements which has led many a project in the past into a state of “analysis paralysis” where they can’t decide what to build because the goalposts keep moving. By focusing on delivering something simpler much sooner we begin to receive some return on our investment earlier and also shape the future based on practical feedback from today, rather than trying to guess what we need.

When we’re building those simpler features that sit nicely upon our existing foundations we have much less need to worry about the cost of rework from getting it wrong as it’s unlikely to be expensive. But as we move from independent features to those which are based around, say, a new “concept” or “pillar” we should spend a little more time looking further down the backlog to see how any design choices we make might play out later.

Thinking to Excess

The term “overthinking” implies that we are doing more thinking than is actually necessary; trying to fit everyone’s requirements in and getting bogged down in analysis is definitely an undesirable outcome of spending too much time thinking about a problem. As a consequence we are starting to think less and less up-front about the problems we solve to try and ensure that we only solve the problem we actually have and not the problems we think we’ll have in the future. Solving those problems that we are only speculating about can lead to overengineering if they never manage to materialise or could have been solved more simply when the facts where eventually known.

But how much thinking is “overthinking”? If I have a feature to develop and only spend as much effort thinking as I need to solve that problem then, by definition, any more thinking than that is “overthinking it”. But not thinking about the wider picture is exactly what leads to the kinds of architecture & design problems that begin to hamper us later in the product’s lifetime, and later on might not be measured in years but even in days or weeks if we are looking to build a set of related features that all sit on top of a new concept or pillar.

The Horizon

Hence, it feels to me that some amount of overthinking is necessary to ensure that we don’t prematurely pessimise our solution and paint ourselves into a corner. We should factor work further down the backlog into our thoughts to help us see the bigger picture and work out how we can shape our decisions today to ensure it biases our thinking towards our anticipated future rather than an arbitrary one.

Acting on our impulses prematurely can lead to overengineering if we implement what’s in our thoughts without having a fairly solid backlog to draw on, and overengineering is wasteful. In contrast a small amount of overthinking – thought experiments – are relatively cheap and can go towards helping to maintain the integrity of the system’s architecture.

One has to be careful quoting old adages like “a stich in time saves nine” or “an ounce of prevention is worth a pound of cure” because they can send the wrong message and lead us back to where we were before – stuck in The Analysis Phase [1]. That said I want us to avoid “throwing the baby out with the bathwater” and forget exactly how much thinking is required to achieve sustained delivery in the longer term.

 

[1] The one phrase I always want to mean this is “think globally, act locally” because it sounds like it promotes big picture thinking while only implementing what we need today, but that’s probably stretching it too far.

Feeling Isolated

By and large I think I’ve been fairly lucky with my time as a contract programmer. Virtually all the teams I’ve worked in and systems I’ve worked on have been pretty decent. None of them are going to change the world but they’ve been enjoyable, which is probably why I’ve ended up working on them for a decent length of time [1].

I can only say “virtually all” because one contract sadly fell way short of the mark. Although I was technically part of a team it only really felt that way from a managerial perspective, even though we shared a codebase. I felt somewhat isolated both physically and mentally. Aside from the morning stand-up I could easily have gone the rest of the day without speaking to my teammates if I had chosen to do so.

Physical Isolation

I started the contract on a separate floor from the rest of my team with a couple of other recent joiners [2]. We were the only people on that floor with the air conditioning on full blast so we had to wear our coats in the afternoon to stay warm. None of the rest of my team had an office pass that could access the floor either, should they want to talk face-to-face while getting us up to speed.

Even when they moved us onto the same floor a month later we were still on the opposite side of the room. In the next desk shuffle I got to swap colleagues although they were working on an entirely separate area of the system with a totally different bunch of people so we had little need to collaborate per-se, only to make small talk. Also the two desks next to me only seemed to be used for a game of Tower of Hanoi by the office movers given how the occupants came and went.

Even my “customer”, at least, the one I knew about, because they were paying for the project, was situated in a different country and spoke a different language. Although their English was way better than any knowledge I have of a second language I quickly discovered why most communication was via email or IM instead of vocally.

Project Isolation

Being an enterprise scale organisation the work was all about projects, and who was sponsoring how many “resources”. Nowhere was this more apparent than the Scrum Board with its project-oriented swim-lanes. Each swim-lane had the names of the team members assigned to that project, and as the stand-up proceeded it walked down the board a project at a time with each member of the sub-team providing an update.

It was fairly apparent right from the moment I started, just by reading the body language of the team members, that there was often little real interest in what the rest of the team was doing. Those that did, cut across projects to some degree because they tended to nurse the build system, deployments and monitoring. A couple of team members never attended our stand-up because they already attended a different one that encompassed their project.

To be fair some of the apathy at the stand-up was almost certainly down to its excessive length. And with little reason for attending except to provide a status update for the managers it’s no surprise those mostly on the periphery zoned out. Sometimes the only common goal of the team seemed to be to not break the system.

Code Isolation

During my short stint I effectively had one feature to work on. There were a couple of other minor tweaks to begin with but ultimately my project was one feature (nay, user story) and it took 5 months to deliver. That one feature involved making a change in an area of the codebase that nobody else knew except one of the tech leads who I soon discovered was leaving. In fact, taking away his days off after the announcement of his departure, I effectively had 3 days for any handover.

Not only were there no docs to work from there were no tests either. The only real knowledge about how any of the service was expected to behave had left firmly inside the head of the author. This pretty much just left doing a spot of software archaeology with the VCS in the hope that the commit messages might contain some extra clues. Many features had been tracked in a feature tracking tool but there were not enough licenses to go round so I had to hassle a teammate to look things up. Even then it often wasn’t worth it as there were no useful details; it felt like the ticket was just there to “tick a box”.

The code relied heavily on the caller “doing the right thing” so any understanding only made sense if you already knew what the caller was supposed to do, and that relied heavily on knowledge of the problem domain and the organisation’s other systems. (At the interview I made it perfectly clear that I still knew little about the problem domain, despite the many years I have worked in it [3].)

Methodology Isolation

Ever since I had my epiphany [4] around testing all those years ago I have become a firm believer in TDD and automated testing as the preferred approach to the sustainable delivery of quality software. Being told early in the project that “you won’t have time to write tests”, despite being asked in the interview about what your approach is, did not bode well.

It soon became apparent that the previous approach had been to rush something out and rely on manual, end-to-end testing and the customer doing things “right”. Validation was almost entirely left to the underlying maths library and so bizarre errors manifested and needed investigating by the developers due to a lack of basic error handling and reporting [5].

With no way of knowing if I had broken anything, because I didn’t know for sure what anything was supposed to do, my only recourse was to write new code with tests and then refactor later when someone (potentially me) could be sure that it was safe to do so. For existing code that I had to change or understand I would write a barrage of tests first to try and ensure I didn’t accidentally break anything. In some cases it was hard to know what was “by design” and what was “by accident”.

Clearly not everyone took this approach, as you can see in “It Compiles, Ship It!”. My pessimism paid off though once the edge cases and little extras started appearing as I could turn around a fix or improvement (safely) in minutes due to my suite of automated unit and regression tests.

Environment Isolation

Sadly, despite my ability to push through changes quickly into the integration test environment, it still took weeks for them to actually appear in the production environment. When my first task, a handful of lines of boilerplate code, took 6 weeks to make it into production I assumed continuous delivery was not something they cared about.

On the contrary, for one aspect of the business, releases were very frequent. It was just that I was on the other side and due to some (IMHO) poor architecture and deployment decisions my part of the distributed system was tightly-coupled to another (major) system’s release cycle.

While it might seem great having my own integration test environment to play with, I ran into issues no one else knew about and I had no idea who was really using it and for what. Once again that information pretty much departed with the author.

Parting Thoughts

On reflection I have to look at my own behaviour first and ask myself whether I was at least partly responsible for feeling left out. Once we moved onto the same floor it was definitely easier to wander over and ask people questions, which I did. However when the response is “well I worked all this out by myself originally” and “that’s more than anyone ever gave me” I think it’s not entirely unfair to assume that knowledge sharing isn’t high on some people’s agenda.

I believe I was as welcoming as I normally am and was happy to help out where possible, given the limited knowledge I had acquired. I guess that culturally there was such a large drive for autonomy that the idea of just chatting about stuff to see what improvements in the system or process would be beneficial just wasn’t on the cards. A couple of times what should have been a constructive comment or question definitely came out of me more as a snide remark which is never a good sign. I’ve been trying hard to be more aware of any sarcasm, which unfortunately comes all too easily to me, and so not add to any unnecessary negativity but I know I failed a few times.

Ultimately I think it says a lot about an organisation that rejects your approach because “they are not a start-up” when your application of that approach has only ever been in large enterprises and none of them has ever had an issue with it before. On the contrary they have often been grateful for the insights and improvements that I’ve brought.

Maybe if I was a lot younger I’d not have known any better and stuck it out a bit more but these days I know it’s just not worth the effort. I feel comfortable that I left the place in a better state than I joined it by documenting various things and writing tests for the code I wrote. After a slightly rocky start my customer seemed pretty pleased with everything I delivered, which I guess is largely what matters most.

As ever, my main regret is leaving behind some people that I wish I could have gotten to know better. Maybe I will, in another life, one where the benefits of collaboration are more positively encouraged.

 

[1] Mostly my tenure has been measured in years, not months.

[2] Only one of which was left when I called it a day – the other two barely lasted a month or so.

[3] See “Problem Domain Expert or Technical Expert or Even Both” for more on this recurring theme.

[4] See “My [Unit] Testing Epiphany” and my more recent ACCU / Agile on the Beach talk “A Test of Strength” for what lead to my enlightenment.

[5] Poor error messages is a popular topic of mine, see “Terse Exception Messages”. Also “The Perils of DateTime.Parse()” covers one specific example.

Technical Debt – Conscious Competence

Once upon a time the term Technical Debt seemed to have a very clear meaning but over the last few years that has been diluted to generally mean any crap code or process which is holding back delivery. I’m sure any scholars of Wittgenstein will be at pains to point out that “meaning is use” and therefore if everyone uses it this way who am I to argue?

For me the canonical source of information on the technical debt metaphor comes from the wiki of the person who coined the phrase in the first place – Ward Cunningham. The entry on Technical Debt there suggests to me that the choice to enter into debt is a wholly conscious one, not the unconscious acts of a less professional bunch of programmers.

By way of example I thought I’d take the opportunity to write up one of those occasions where I’ve been involved with taking debt on (in the original spirit of the term) and how we dealt with it, to show where the distinction lies.

The Bug

Soon after going live with v1.0 of a new calculation system in a large financial organisation we discovered that a number of key counterparties were missing from the daily report. The report generator was a late addition and there were various other issues around it’s development and testing which muddied the waters somewhat but suffice to say that this wasn’t delivered as cleanly as the core system was. (You might consider the more recent meaning of the term to apply here.)

More importantly what transpired was that due to various mergers in the company’s history a few counterparties had the same “unique” code in different back-end systems. This wasn’t just news to my team (we were all recent hires) but also to quite a few people in the business too. Due to only dealing with a limited set of “books” the codes were always unique to them in their context, but our new system cut right across them all.

The Root Cause

The generation of calculations was ultimately based around a Cartesian product of two counterparties, however given that most of those were pointless there was an optimisation which used another source of data to reduce that by more than an order of magnitude.

This optimisation should have been fairly simple but due to a need to initially use some existing manually managed counterparty data to ease the cutover (so regression testing should then reconcile exactly) it was somewhat more complicated than first envisaged.

Our system was designed to use the correct source of data eventually, but do a reverse lookup for the time being. It might sound simple but the lookup actually involved multiple lookups using combinations of keys that had to make assumptions about which legacy back-end system might hold the related data. The right person who could explain how we could do what we needed to do correctly also seemed elusive; there were many people with “heuristics”, but nobody who knew for sure.

In total there were ~100 counterparties out of a total of ~15,000 permutations that suffered from this problem. Unfortunately a handful of those 100 had a significant effect on the “bottom line” and therefore the usefulness of the system as a whole was in doubt at that point.

Entering Into Debt

Naturally once we unearthed this clanger we had to decide how to tackle it. After getting our heads around what this all meant and roughly where in the code the missing logic probably needed to go we had to make a decision – do we try and fix the underlying issue right away or try and put a workaround in place (assuming that’s even possible) to mitigate the problem, at least temporarily.

We were all very aware of going down the dark road of putting a tactical fix in place because we’d all seen where that can lead. We had made a concerted effort over the 12 months required to build the system to refactor relentlessly [1] and squash any bugs as soon as possible. This felt like a backwards step.

On the positive side by adopting a Design for Testability approach in most parts of the code we had extra switches on our processes [2] that allowed us to make per-counterparty requests, usually for diagnostic purposes. Hence the workaround took the form of sticking the list of missing counterparties in a simple text file, then using a command prompt FOR loop [3] to read the file and invoke the tool in “single counterparty mode”. Yes it was a little slow due the constant restarting of the process but it was easy to surgically insert into the workflow with the minimum of testing or risk.

Paying Back the Debt

With the hole plugged for now, and an easy mechanism in place for adding any other missing counterparties – update the text file – we were in a position to sort out the root problem without feeling under pressure to get the system working correctly 100%, ASAP.

As you can probably imagine the real solution wasn’t easy, not least because it was one of a few areas of SQL code that didn’t have any unit tests and was a tangled web of tables and views which had grown organically in an attempt to graft the old and the new worlds together [4].

What Did it Cost?

If we assume that the gung-ho approach would have been to just jump in and start fixing the real code, then what did we lose by not doing that? It’s possible that the final fix was simple and a little more investigation may have lead to that solution instead.

In contrast, the risk is that we end up in one of those “have we fixed it or not” scenarios where we spend an indeterminate amount of time being “real close” to getting towards “done”. The old adage about the last 10% also taking 90% of the time springs immediately to mind. Instead we were almost positive we had a simple workaround that could be deployed and get the system running correctly enough in an estimable amount of time. I believe there is a lot of value in having that degree of confidence.

What I think was critical was being able to remove the pressure on finding the right solution as this gave us time to really consider what needed to be done. Any fix done under pressure is not going to be given the attention to detail that it probably deserves. You then run the risk of making the system worse and having an even deeper hole to dig yourself out of.

The customer does not care about strategic versus tactical decisions per-se, they just want the thing to work. We cared about the solution because we knew it would be a burden in the short term as everyone had to remember about the bit “grafted on the side”. The general trust the team had built up by keeping quality at the forefront meant that the business would be more willing to trust us to reconcile the problem appropriately when the time came.

Use Language With Care

I really hope the term Technical Debt doesn’t continue to get watered down even further as it’s a powerful concept which is incredibly useful in the right hands. We already have far too many words for “alternate implementation” that are pejoratives carrying an air of unprofessionalness about them. I would like this one to remain in the hands of the professionals so they can continue to have “grown up” conversations with their customers about when it’s appropriate to consider taking shortcuts for a short term business need without them rolling their eyes, yet again.

 

[1] See “Relentless Refactoring” for more thoughts around this (unfortunately) contentious topic.

[2] “From Test Harness To Support Tool”, “Building Systems as Toolkits” and “In The Toolbox - Home-Grown Tools” all look at the non-functional side of tooling.

[3] A batch file just wouldn’t be complete without a for loop, see “Every Solution Starts With ‘FOR /F’”.

[4] This one feature seemed destined to plague us forever, see “So Many Wrongs, But No Rights” for another tale of woe.