JezCon One

Last weekend a small group of fellow programmers met up just outside Birmingham for what I flippantly titled “JezCon”. It was a soul-filling weekend as it gave us an opportunity to discuss our chosen profession (and hobby in many cases) in surroundings where there was no pressure to engage in small-talk or feel uncomfortable because we were excluding our respective partners by “talking shop”. Not that there wasn’t plenty of social chit-chat too as the conversations naturally ebbed-and-flowed in and around the wider topic.

The Premise

JezCon is essentially the physical manifestation of what started out as “The Call” – a fortnightly video chat involving a bunch of long-time programmers whose primary connection is/was being a member of ACCU [1]. In particular we have all spent many late nights in the bar at the ACCU Conference and Meet-ups (and others, such as Pycon UK and NorDevCon) chewing the proverbial fat.

People typically refer to the time between talks at conferences as The Hallway Track as it’s the opportunity to socialise and network with like minds. Similarly the evening equivalent is The Pub/Bar Track as the attendees tend to swap tea & coffee for beer & cocktails [2].

The “Jez” (of JezCon fame) felt we should do this bit far more often, especially as since the start of Covid, and the consequential adoption of more remote working, we were spending less time directly socialising with our fellow professionals.

I’ll admit I’ve not been a regular attendee, but I have enjoyed the few I’ve managed to join in, and naturally jumped at the chance to spend a few days sharing a house for a more localised gathering.

The Format

As a group of nine it was possible to find a large house via AirBnB that we could all share. Unsurprisingly, Birmingham was the equidistant point for us all and so we all had a few hours of travelling but nobody had a ridiculously long journey. (Those at the edges of Britain are always a long way from everywhere…)

I packed my portable projector, some magic whiteboard paper, sharpies, Post-It’s, etc. just in case, but the property came with a large tv screen in the living room and that more than sufficed for what people had planned. Not that there was a formal schedule per-se, but a few people had put together some sessions which then gave the weekend a little bit of structure.

As the house came with a kitchen we had the opportunity to keep costs down by group cooking, and I was eternally grateful for the splendid meals that some of the group put together to feed us all. (I loathe cooking with a passion but am more than happy to make up for it by cleaning and doing the dishes, etc.)

The phrase “tea & coffee making facilities” didn’t exactly inspire me with confidence so my other “major” contribution was to pack our spare filter coffee machine which at least made up for the tiny cafetiere they provided. It’s not had that kind of a workout for quite a few years!

Ensemble Programming Session

My highlight of the weekend was undoubtedly a mob programming session where we got to tackle a hacked together JavaScript version of Snake written by one of the attendees. Our job was simply to “improve” it. (There were a few Machiavellian flaws for extra authenticity.)

With over 250 years of professional programming experience in the room (and probably another 75 before starting our respective careers) I was excited to see how this would play out. I’ve been fortunate enough to work in a few teams where ensemble programming was a key part of how we worked [3] and I’ve really missed this over the last few years as we’ve become more remote, both in the literal and figurative sense.

I’ve mostly stayed away from JavaScript and what experience I do have dates back to the turn of the Millennium so I was interested in how useful my knowledge of programming in general would be to the task at hand. With a background in curly brace languages (C, C++, Java, and C#) I knew the syntax would be fine, it was more about which high-level concepts are available, and can we use them to make the code more readable / malleable.

I think the instigator of the session only expected it to last a couple of hours, but we ended up doing six hours in total because we all got so invested in the process and the continued outcomes. Not everybody stayed for the entre six hours and people came and went as necessary – a free flowing mob – which highlighted the casual approach to the entire weekend.

One of the most interesting aspects of ensemble programming is the negotiation that takes place around the flow of changes. One participant was quite vocal when we moved too fast and they didn’t feel we were adding the relevant tests or taking small enough steps. My particular focus ended up being on naming and metaphors and using language to be more precise in our concepts. Yet another was looking more at the bigger picture and overall flow. There were definitely some voices that were louder than others, despite our best efforts to not dominate proceedings, and I hope everyone enjoyed the exercise as much as I did.

I think it’s fair to say that most of us aren’t exactly excited about where the industry is heading due to the invasive use of AI tooling, but it’s hard not to be at least slightly impressed by the equivalent refactoring that was produced by one attendee using such tooling in minutes rather than hours. (Of course one should not confuse the journey with the destination.)

Interlude

I hadn’t planned anything in particular, but as only one person had attended the ACCU conference this year, I decided to resurrect my “Slop Bucket” lightning talk to kill 10 minutes. This was my usual [4] IT related comedy one-liners, but this time with images for a change. (I also discovered Nagle’s Algorithm is less well known than perhaps it once was.)

Jez’s Favourite Pull Request

The session by the conference’s namesake was a little unusual but also very interesting. He presented his favourite PR, but opened with some simple code and a puzzle – why did this take hours to evaluate?

The code was a fairly simple conditional tree with some use of ternary expressions too, no loops or other obvious candidates that would typically create a performance issue of that magnitude.

This was in part a ruse because he wasn’t talking about the performance of evaluating in the traditional sense, he was referring to a tool he had been asked to work on which is used to evaluate the code using static analysis. I won’t spoil the punchline but the old adage about “the size of the fix being inversely proportional to the amount of time spent investigating” felt alive and well.

What was most interesting here was listening to someone talk about a problem domain I know little to nothing about. While it was good to see the PR in the end, the journey he took us on and the way he allowed us to play along and try and guess the issue by little clues was still very entertaining and engaging.

Time to Check-out

Although I’ve focused on those two sessions, they weren’t the only two over the weekend. However, they were the two which highlighted what it is that I enjoy most about programming and in particular how sociable it can be in the right setting.

Anyone who has ever read the works of Gerald Weinberg, and The Psychology of Computer Programming [5] in particular, will know all too well that programming has always really been as much about the people and the collaboration between them as the artefacts of the process.

We also had a fantastic conversation (yet again) about types and type systems (a topic which has always been close to my heart [6]) that stemmed from the ensemble programming session and has reminded me that I still need to pick up a book on Ada.

JezCon Two & Beyond?

Although we ended up with nine people, it should have been twelve, but sadly not everyone could make it; not least the person who probably put the most effort into making it happen in the first place!

Naturally he feels he missed out and so is already talking about when we can arrange a second gathering. (This also means it won’t fall foul of the classic naming problem caused by including a number in the title which then overpromises [7].)

I’m definitely up for more. It rekindled some feelings I had when I attended my first ever programming conference – ACCU 2008 in Oxford – and that’s got to be a good thing. Maybe the pragmatic use of AI will change the nature of our day-to-day jobs as we edge ever closer to retirement but I hope that there is still room in our lives to simply enjoy the art of code like we did that weekend [8].

EDIT: Our namesake has written up his own review here: jezcon1.

 

[1] See one of my first few blog posts back in 2009 – Who is the ACCU? – for a brief synopsis.

[2] This aspect of conferences has always been my favourite. My 2025 opening Afterwood Learning in Person looks at this in more detail.

[3] I first wrote this up back in 2018 as To Mob, Pair, or Fly Solo and later turned it into a conference talk you can watch here: Mob, Pair, or Fly Solo?

[4] If you’ve never experienced it, A Decade of Lightning Talks with Programming One-Liners will give you a feel :o).

[5] This book is almost as old as me, probably older if you consider when the research and observations that lead to it would have occurred.

[6] My 2012 post Primitive Domain Types - Too Much Like Hard Work? nicely captures my opinion on the lack of effort typically expended on thinking about types.

[7] Jez and I both attended The First International Conference on Software Archaeology way back in 2014 and sadly it has yet to be repeated :o(.

[8] Afterwood – Occupational Hazard

 

The Illusion of a One-Time Set-Up

One of the most laborious things about starting work at a new client / employer, can be getting your machine and user account configured so that you are able to work on their codebase.  For me, the acid test of whether I’m in a good place is being able to build the code and run the necessary test suites locally that I’ll be relying on for the early feedback loop. But that’s the bare minimum.

There are often plenty of tools to install and configure, and I’m not just talking about light vs dark mode, but team and organisation level stuff once you need to reach out across the network to dev, test, and production services. The number of permissions for your user account can be extensive if you’re working on a system that has lots of tentacles that reach out to message queues, databases, APIs, etc. especially when the organisation doesn’t use single sign-on everywhere.

One-Stop Shop

Sometimes this set-up process goes really smoothly and you can bootstrap yourself with very little effort, while other times it’s a long hard slog. In the corporate world there are typically more gates to go through as, for example, local admin rights are not conferred by default and your software choices are limited to what they grant access to. (See Getting Personal for a rant about where consistency in tooling actually matters.) Once you have access to the version control system, and the VCS tool installed, in theory you have the gateway to getting yourself set-up with a metaphorical “flick of the switch”…

If only that were always the case. Sadly it’s not unusual to be given little to nothing to work from. If the team uses a complex IDE like Visual Studio then it can often be assumed that once you have that installed then you’re 99% of the way there. This is only the true if you believe software developers only “write code”.

Maybe you are given a wiki page which helpfully lists many of the tools you need (to request from the company’s software portal) but probably neglects to tell you how you need to configure them based on the team’s typical workflow and development environments. The kinds of things I’m talking about here are local / shared databases (including drivers and DSNs, or containers), the various Azure emulators for storage and queues (or shared cloud hosted instances), shared AWS resources, upstream and downstream in-house dependent services, etc. Okay, you may not have the required access up-front but getting the approval signed should be the only burden, once you have that you don’t need to be fumbling around trying to work out how to make use of your new found powers.

Once Upon a Time

The reason typically given for not providing a better DX (developer experience) is that this is a “one-time setup” and the cost obviously isn’t worth it. But here’s the thing: while it may be a one-time set-up for that particular user and machine, it’s just one of many when you factor in all the places where this kind of workflow will be needed in practice.

Okay, so you don’t normally get people joining the team every few days, but configuring a local developer’s machine is the least interesting case, in my opinion. Where this blinkered one-time setup thinking really starts to cause a problem is once you factor in the build and deployment pipelines.

The whole “works on my machine” meme exists because it highlights the missing appreciation for what goes into turning a pending code change into a feature deployed across the real estate, or onto users desktops and/or phones. All the other machines that are required to build and run the same code and tests which you have on your machine also need to be set-up too.

Automation Friendly

While you might build a demo-able artefact on a developer’s machine, any release binaries or deployment packages will always be built in a “clean room” environment because a developer’s desktop is typically tainted with the results from experiments with ad-hoc code and tooling. While in the (not so) distant past we might have built and maintained the build and deployment pipeline servers, and dev, test, and production servers carefully by hand (aka snowflakes), those days should be long behind us. The rise in virtualisation and the purity gained from the “immutable infrastructure” movement means that the various steps in our once one-time set-up is now repeated, over and over again. This is even more apparent when the unit of delivery is an entire VM or container rather than just an application package. (Not seeing the similarities between how you build and test locally versus the entire delivery pipeline is a topic I covered way back in 2014 in Building the Pipeline - Process Led or Product Led?)

What this effectively boils down to is having an automation mindset. While the meme tells you to “automate all the things”, and this is a venerable goal, I’ve seen the pendulum swing too far so prefer the more pragmatic Automate Only What You Need To. Pedantry aside, the key point is that you think about how best to share the “process” you’re just about to discover. While it may be quicker for you to use a UI to perform this particular task now, if there is any possibility that other people or machines will need to perform it too, or it’ll be used as part of some automated process then it behooves you to spend a little bit of time looking at whether there is an automation-friendly approach which might be worth exploring instead.

MVP – Minimum Viable Process

Maybe you don’t have the time right now to write a nice little script that does “all the things” but have you considered whether there is an approach which at least leans into that? For example, instead of sharing a URL for downloading a tool that then has to be manually installed, see if it’s available via a package manager, which can later be scripted as part of the larger workflow.

While automation has always been Linux’s strong suit, Windows has improved hugely over the years such that many tasks which were once only accessible via the GUI can now be performed with a “one-liner”, if you ask your favourite search engine the right question. In essence instead of “how do I do X?” you need to append “from the command line” to access this Other World.

Console-ation Prize

Every time someone documents a process using a series of screenshots a kitten dies. Taking screenshots is labour intensive and far more likely to go out of date because vendors love to add new features and give their tools a facelift. In contrast a one-liner (or few lines) in a monospaced font on a wiki page is practically timeless in comparison and almost impossible for someone to mess up. It’s easy for someone else to then take on and turn into a script going forward.

You don’t need to be a DevOps kind of person to appreciate the simpler things in life, just someone who enjoys paying it forward when possible.

The Case of the Disappearing App

[Despite being almost 30 years ago, and the finer details being lost in the mists of time, the punchline has always stayed with me – it’s scar tissue I guess :o).]

I was contracting in a team which had taken over a codebase from another company, to try and improve the performance, as it was far below what was needed for even the smaller use cases they had in mind.

The front-end was a PowerBuilder based GUI and I was a C developer working on the in-process back-end library which was a scheduling engine (think: travelling salesman problem). To be more precise, the actual scheduling engine was written in Pascal which was then translated into C and statically linked into the back-end library which acted as the infrastructure code – reading & writing to the database, a large caching layer, providing the C API for PowerBuilder to interact with engine, etc.

One day there was a report from the QA team (which was effectively one guy – the same guy I mentioned before in A Not So Minor Hardware Revision) about the application just “disappearing”. There was no error message from the application, and no UAE / GPF style crash dialog from the OS either, which was Windows NT 4 at the time.

We could reproduce the problem fairly easily. It basically involved importing a new set of orders and running the scheduling engine. This was pretty much what we had been doing all along to profile the application and address the various performance issues we ran across. We had some nice test datasets that we kept using across the entire team and I suspect (now) he was probably using a newer one [1].

What was weird was that there no trace of the application terminating. If it was crashing Windows would have told us, and allowed us to attach the debugger. I even ran the application under the debugger and made sure I’d enabled “first chance exceptions” in case any Structured Exception Handling was being triggered and swallowed (it was C, not C++ so there would be no exceptions per-se). There was no mention in the event log either of anything unusual.

In the end I resorted to single stepping into the back-end and slowly drilling down into the code to see if I could work out if there was something in our code causing this. The core engine used a Tabu Search and called back out repeatedly to get various bit of data like product details, driving time between two points, etc. It was dying somewhere inside the core engine, but due to the cyclic nature of the execution it was hard to put a breakpoint anywhere useful. (This was Visual C++ 2.0 and conditional breakpoints had a habit of slowing things right down which added to the debugging burden as it didn’t trigger that quickly as-is.)

During my debugging attempts I began to notice that the memory footprint was spiking and had a suspicion that ultimately the cause was an out-of-memory problem. Native applications typically behave pretty badly when they run out of memory as it tends to lead to dereferencing a NULL pointer and the process unceremoniously crashing. But, as pointed out earlier the Windows crash handler was not being invoked so something else was going on.

Eventually I decided to put conditional breakpoints on the prologue of the CRT functions malloc() and calloc() in the hope that I could catch it returning a NULL pointer. It worked, and as I slowly walked back up the function stack I could see why the application simply disappeared – the error handling simply called abort()! Although this was quite an abrupt termination from a users perspective, from a Windows OS point of view the process had exited gracefully and therefore no crash handler was needed. Under the covers I suspect abort() called TerminateProcess() and so the PowerBuilder GUI would not have been able to catch the process’ impending doom either. If I remember correctly we found a couple of these calls to abort() and removed them so that the error could attempt to bubble back up the stack and at least the user would get notified one way or another of the unstable nature.

As an aside, this wasn’t the only memory issue we had on that project. Not counting the bizarre performance issue in A Not So Minor Hardware Revision we also suffered badly from heap fragmentation because this was before the days of the low-fragmentation heap. The knowledge I learned along the way debugging these, and other memory and heap issues elsewhere, paved the way for my first proper ACCU article Utilising More Than 4GB of Memory in a 32-bit Windows Process.

 

[1] I also seem to remember running Oracle locally so that each developer had their own database for testing which would have affected the overall memory footprint for the machine.