Thursday, 26 November 2015

The Cost of Not Starting

The idea of emergent design is uncomfortable to those at the top and it’s pretty easy to see why. Whilst there are no real physical barriers to overcome if the software architecture goes astray, there is the potential for some significant costs if the rework is extensive (think change of platform / paradigm / language). In times gone by there was a desire to analyse the problem to death in an attempt to try and ensure the “correct” design choices were made early and would therefore (theoretically) minimise rework.

In a modern agile world however we see the fallacy in that thinking and are beginning to rely more on emergent design as we make better provision for adapting to change. It’s relatively easy to see how this works for the small-scale stuff, but ultimately there has to be some up-front architectural choices that will shape the future of the system. Trying to minimise the number of up-front choices to remain lean, whilst also deciding enough to make progress and learn more, is a balancing act. But the cost of not actually starting the work and even beginning to learn can definitely be dear if the desire is to move to a newer platform.

A Chance to Learn

I recently had some minor involvement in a project that was to build a new, simple, lookup-style data service. Whilst the organisation had built some of these in the past they have been on a much older platform, and given the loose timescale at the project’s inception it was felt to be a great opportunity to try and build this one on a newer, more sustainable platform.

Essentially the only major decision to make up-front was about the platform itself. There had already been some inroads into both Java and .Net, with the former already being used to provide more modern service endpoints. So it seemed eminently sensible to go ahead and use it again to build a more SOA style service where it owns the data too. (Up to that point the system was a monolith where data was shared through the database.)

Due to there being an existing team familiar with the platform they already knew plenty about how to build Java-based services, so there was little risk there, aside from perhaps choosing a RESTful approach over SOAP. Where there would be an opportunity to learn was in the data storage area as a document-oriented database seemed like a good fit and it was something the department hadn’t used before.

Also as a result of the adapter-style nature of the work the team had done before they had never developed a truly “independent” service, so they had a great opportunity to try building something more original in an ATDD/BDD manner. And then there was the chance to make it independently serviceable too which would give them an initial data point on moving away from a tightly-coupled monolithic architecture to something looser [1].

Just Enough Design

In my mind there was absolutely no reason why the project could not be started based on the knowledge and decisions already made up to that point. The basic platform had been chosen and therefore the delivery team was known and so it would be possible to begin scheduling the work.

The choice of protocol and database were yet to be finalised, but in both cases they would be relying heavily on integrating a 3rd party product or library – there was little they had to write themselves. As such the risk was just in evaluating and choosing an approach, and they already had experience with SOAP and their existing database to fall back on if things didn’t pan out.

Admittedly the protocol was a choice that would affect the consumer, but the service was a simple data access affair and therefore there was very little complexity at this stage. The database was going to be purely an implementation detail and therefore any change in direction here would be of no interest to the consumers.

The only other design work might be around what is needed to support the various types of automated tests, such as test APIs. This would all just come out “in the wash”.

Deferring the Decision to Start

The main reason for choosing the project as a point of learning was its simplicity. Pretty much everything about it allowed for work in isolation (i.e. minimal integration) so that directions could be explored without fear of breaking the existing system, or development process.

What happened was that some details surrounding the data format of the 3rd party service were still up in the air. In a tightly-coupled system where the data is assumed to be handled almost verbatim, not knowing this kind of detail has the potential to cause rework and so it is seen as preferable to defer any decision it affects. But in a loosely-coupled system where we decide on a formal service contract between the consumer and producer that is independent of the underlying implementation [2], we have less reason to defer any decisions as the impact will be minimal.

As a consequence of delaying doing any actual development on the service the project reached a point well passed the Last Responsible Moment and as such a decision was implicitly made for it. The looming deadline meant that there was no time or resources to confidently deliver the project on time and so it was decided that it would be done the old way instead.

Cost versus Value

One of the reasons I feel that the decision to do it the old way was so easy to make was down to the cost based view of the project. Based solely on the amount of manpower required, it likely appears to be much cheaper to deliver when you’ve done similar work before and have a supply of people readily available. But that only takes the short-term cost into account – the longer term picture is different.

For a start it’s highly likely that the service will have to be rewritten on a newer platform at some point in the future. That means some of the cost to build it will be duplicated. It’s possible many of the same learning's could be done on another project and then leveraged in the rebuild, but what are the chances they’ll have the same deadline luxuries next time?

In the meantime it will be running on a platform that is more costly to run. It may only add a small overhead, but when you’re already getting close to the ceiling it has the potential to affect the reliably of the entire monolithic system. Being done on the old platform also opens the door to any maintenance being done using the “culture” of that platform, which is to tightly-couple things. This means that when the time finally comes to apply The Strangler Pattern it won’t just be a simple lift-and-shift.

Whilst it might be easy to gauge and compare the short-term costs of the two approaches it’s pretty hard to put a tangible value on them. Even so it feels as though you could make a judgment call as to whether doing it on a newer platform was “worth” twice or three times the cost if you knew you were going to be gaining a significant amount of knowledge about how to build a more sustainable system that can also be continuously delivered.

Using Uncertainty as a Driver

One of Kevlin Henney’s contributions to the book “97 Things Every Software Architect Should Know” discusses how we can factor uncertainty into our architecture and design so that we can minimise the disruption caused when the facts finally come to light.

In this particular case I see the uncertainty around the external data format as being a driver for ensuring we encapsulate the behaviour behind a service and instead formalise a contract with the consumer to shield them from the indecision. Whilst Kevlin might have largely been alluding to design decisions the notion “use uncertainty as a driver” is also an allegory for “agile” itself.

Eliminating Waste

There is undoubtedly an element of poetic justice in this tale. The reason we have historically put more effort into our analysis is to try and avoid wasting time and money on building the wrong thing. In this instance all the delays waiting for the analysis and design phases to finish meant that there was no time left to do it “right” and so we will in all likelihood end up generating more waste by doing it twice instead.

Also instead of moving forward the knowledge around building a more sustainable platform we now know no more than we do today, which means maintenance will continue to be more costly too, both in terms of time & money and, potentially more importantly, morale.

[1] Whilst a monolithic architecture is very likely to be tightly-coupled, it doesn’t have to be. The problem was not being monolithic per-se, but being tightly-coupled.

[2] Yes, it’s possible that such as change could cause a major re-evaluation of the tech stack, but if that happens and we had no way of foreseeing it I’m not sure what else we could have done.

Wednesday, 25 November 2015

Don’t Fail Fast, Learn Cheaply

The term “failing fast” has been around for a long time and is one that I’ve used since the early days of my career. When talking to other developers I’ve never had a problem with it, but using it with business folk has had a different reaction on occasion.

Defensive Programming

I first came across the term (I believe) when reading Steve Maguire’s excellent book “Writing Solid Code”. In it he describes how letting a process crash at the moment something really bad happens is often more desirable than trying to code defensively, as that just masks the underlying issue. Whilst it sucks for the user they stand less chance of even worse things happening, e.g. silent data corruption. I wrote about my own experiences with this type of coding in “The Cost of Defensive Programming”.

Resource Efficiency

The second context under which I met the “failing fast” term was when reading Michael Nygard’s fabulous book “Release It!” Here he was talking about avoiding queuing or doing work which was ultimately going to be wasteful. For example if you can’t acquire a resource because it is unavailable, it’s better to discover that early and fail then instead of waiting until the end at which point you need to throw work away. Once again I’ve told my own tale around this in “Service Providers Are Interested In Your Timeouts Too”.


The most recent use of “fail fast” I’ve encountered has appeared in relation to the delivery of software projects. In this guise we are talking about how to do just enough work to either prove or disprove that the project is in fact viable. At a smaller scale you could apply the same idea to a spike, which is often one part of a project and used to validate, say, a technical approach.

In essence what we’re saying is that if you’re going to fail, make sure you do it as quickly as possible. By delaying the work that will allow you to decide whether the idea is actually viable runs the risk of so much being done that you fall foul of the Sunk Cost Fallacy. Sander Hoogendoorn has a post titled “Failing fast” that talks about this idea in more detail.

Negative Connotations

As you can see the term has many uses and so I’ve found it quite natural when talking to fellow developers in any of these three contexts to say it – they’ve always understood the real meaning. However when talking to less technical people, such as business folk and higher level managers I’ve found that you can get a different reaction. Instead of latching onto the second word “fast”, they focus on the first word “fail”. What then happens is that the discussion turns into one about “why would we do something where we might fail?”. And “isn’t that a backwards step?”.

At this point you’ve now got explain that failing quickly is really not failing per-se, but actually a successful outcome from a business cost perspective. In essence you’ve already put yourself on the back foot and you’ve potentially lost your audience as they try and work out why this “agile” stuff is beneficial if it means you’re going to fail! Why wouldn’t you just do more analysis up front and avoid failing in the first place?

Learning Cheaply

A more positive sounding way of promoting the idea is instead to focus on the point of the exercise, which is to learn more about the problem. And if we flip the notion of “fast” around and turn it into something the business really understands, money, we can talk about saving it by spending less to get an answer to our question. Also, where failing feels like a backwards step, we generally consider learning to be a cumulative process and therefore it sounds like we’re always making some sort of progress instead.

It’s all just smoke-and-mirrors of course, but in a world where everyone is vying for the company’s money being a little careful with your language may just tip the scales in your favour.

Tuesday, 24 November 2015

Missing the Daily Commute by Train

I started programming professionally just over 20 years ago and in all that time I have mostly commuted to my place of work either by car or train. My first role, straight out of university, was at the height of the recession in 1992 and so I pretty much moved to wherever it was going to be. After that I started contracting and did a couple of stints where I commuted by car for an hour each-way which pretty much convinced me that commuting by car any distance was less than desirable. Whilst I had been car-sharing and enjoyed some very interesting chats with my fellow programmer passenger it was still tiring and no fun when stuck in traffic (which eventually became a regular occurrence).

During that time my wife had tried commuting into London by train for her first job and found it was quite palatable. Hence it felt as though I either took a contract on my doorstep (which was unlikely), we moved house, or I headed into London by train [1]. And so I spent the better part of the next 20 years commuting into London and its suburbs.

All Quiet on the Writing Front

You may have noticed that my writing activities have taken a serious nosedive over the last 6 months and that’s almost entirely due to me taking a contract that was once again almost on my doorstep. A chance to commute by car for only 20 minutes a day, and in the opposite direction to all the traffic (which was heading into Cambridge) felt like too good an opportunity to pass up. It wasn’t a hands-on development role like I’ve been doing for the past 2 decades but, frankly, given the short commute I was happy to try my hand at some pure consulting for a change. And I’m glad I did as I learned heaps of stuff in the process [2].

Having such a short commute by car has been an absolute delight. I’ve left the house in the morning, and the office in the evening, when I felt like it rather than to meet a timetable. And the short drive time has meant me spending more time at both ends of the day with the wife and kids [3]. I never quite made it home to enjoy dinner every day with them as getting out of the business park’s car park was a nightmare at 5 pm; but it was close.

That said it seems a little churlish to complain about the lack of time I’ve had to write either for this blog or the ACCU journals. Clearly I have had the time, in the evening for example, but I’ve (implicitly) chosen to spend it differently. As I look back over my career I now begin to understand some of the comments that my colleagues have made in the past around how “lucky” I was to have a regular train-based commute.

A Time to Learn

My journey into London consists initially of 45 minutes solid travel followed by “an amount” of time on the underground rail network which has been anywhere from around 10 to 30 minutes. That first solid block of time has been great for really getting my teeth into a spot of reading, gaming or writing (articles or code) as I nearly always get to sit down, even if it’s on the carriage floor. The underground stretch is “standing room only” but still perfectly fine for reading a paper journal, like MSDN or one of the ACCU publications, as they are easy to hold and manipulate even on a very crowded train.

In the early days when a development capable laptop cost in the region of “thousands of pounds” I spent most of my time reading books about C++, Design Patterns, Windows internals, etc. I also read a variety of journals that have long since gone out of print such as C++ Report, MSJ, CUJ, Application Development Advisor and Dr Dobbs. Pretty much the only one left from this era that’s still in print is MSJ, but now under the name of MSDN Magazine. Luckily the ACCU journals, which I only discovered when CUJ disappeared (circa 2005), are also still in printed form.

Deliberate Practice

There is a saying that goes:

In theory there is no difference between theory and practice. In practice there is.

And so I’ve spent plenty of time coding on the train too. The train line I travel on has never even had decent mobile phone reception and so the idea of using the Internet is somewhat laughable. But given that my background has mostly been writing applications and services in C++ this has hardly been a problem to date, and is, in my mind, even highly desirable (See “The Developer’s Sandbox”). Most of what you see on my web site and GitHub page is code that has been written in these little 45 minute stints to-and-from work. Occasionally I’ve done a little bit in the evenings or in the office when the tool has been used to actually help me with my day job, but I’ve never worked anywhere that provides “20% time” - even for permanent staff (and I’d never expect to as a freelancer either).

Habitual Behaviour

It shouldn’t have come as any real surprise that my non-work activities would fall by the wayside the moment that my commute disappeared. After all I’ve taken a few lengthy periods of time off between contracts in the past and despite my best efforts to get motivated and spend it productively I’ve instead found it easy to fritter it away (but in a really nice way, e.g. having time with the family).

It took me a long time to realise just how much structure I need in my life to get “other” things done. Whilst I’d like to believe that I don’t need this kind of formality I’m really just kidding myself. Just as I need my notebook (paper based, of course) by my side to make notes and not forget things, so I need some semblance of order throughout my day to help guide me.

As I write this I’m beginning to wonder how much of what I said in “Code by Day, Design by Night” actually describes my behaviour outside “work time”? I guess my commute means I’ve always had “20%” time, it’s just that it’s had to be on top of my 100% working day. Either way I now realise how valuable to my career that time actually is.

[As if to prove a point I’m only just proof-reading this and hitting the “publish” button now that I’m back commuting again…]

[1] Another choice would have been to go permanent again but I had just started to enjoy the freedom of freelancing and was reluctant to give that up again so quickly.

[2] Hopefully I’ll be filling these very pages with musings from that gig in the coming months.

[3] Let’s put aside for a moment the fact that working from home means a zero-minute commute. Also, given the disruption I caused just by being around when the kids are supposed to be getting ready for school, I’m not convinced my wife always saw it as a bonus :o).

Friday, 25 September 2015

Choosing a Supplier: The Hackathon

One would hope in this day-and-age that when hiring a candidate for a role as a developer the interview process would include some element of actually doing what it is we do - programming. This could be as simple as submitting an offline coding test, but better yet a face-to-face session where you pair with one of the people you’re likely to work with. Either way, just as an artist would expect to show their portfolio, we should expect to do something similar – not just write a CV and then talk about it.

If that’s the case for hiring a single developer, why shouldn’t choosing a software supplier follow a similar process, after all there is probably far more money at stake. And that’s really what the whole interview process is all about – it’s a form of risk management – you’re trying to find a supplier that you’re confident is going to deliver what you really need in a timely manner and for an acceptable cost.

Show Your Workings

The Hackathon, which probably sounds a bit too hipster for some people’s taste, is the embodiment of this idea – get the potential suppliers to show how they actually produce software, albeit on  a very small scale. My last two days have just been spent on one of these and it was great fun, if a little nerve-wracking.

The consultancy firm I’m currently working through (Equal Experts) has been involved in one of these before, quite recently, and they relish the chance to do this kind of selection process more as it plays to their strengths – delivering working software incrementally using modern development techniques [1].

By all accounts this particular one was slightly different to the previous one, although it still followed a similar outline. There are a number of very small teams (3) representing the different suppliers. Each team has a small number of people (~4) that covers whatever skills they think they’ll need based on the brief. In our case we had 3 devs and a UX person, but obviously we could also act in a lesser capacity as a BA, QA, Ops, etc. too to cover all the bases.

The overall structure was that we would spend a short period learning a bit about the company we were pitching to and about the problem they wanted us to tackle. Each team was then given a separate “war” room in which they could work for roughly two days before presenting back at the end.

Where in the previous Hackathon they got more freedom about what they wanted to tackle (i.e. a higher-level problem statement) this one had a more specific problem to be solved. The problem was also directly related to the actual problem they chosen supplier would eventually be asked to work on, which makes sense.

During the two days various people involved in the selection process would come round and visit us to see what we’d been up to and that would also give us an opportunity to ask any questions we had about the problem. If we really needed to we could have called upon The Business at any time to come and help us but their visits were frequent enough that it meant we never needed to go that far.

Our Approach

Naturally I can’t say anything about the problem itself, but suffice to say that it was far bigger than anything we could expect to build in just two days. However that didn’t stop us tackling it in the same way we would a real project. We did some up-front analysis to explore the problem domain a fair bit, make some key architectural decisions to decide what we were going to build, create a backlog with some stories on it and then start coding. We also littered the walls with sheets of magic whiteboard [2], post-it notes and index cards that covered our questions, answers, architecture, backlog, etc.

We were able to to do a little bit of work beforehand, such as creating a GitHub repo that we could all check we had access to, along with a dummy build in AppVeyor to cover the CI part. The guest Wi-Fi wasn’t brilliant [3] which meant pushing and pulling to/from GitHub was a bit laggy, but it was usable and the cloud based CI monitor was stable enough.

Despite it being unofficially described as a “Hackathon” we decided we would do the best we could to show how we worked in practice, rather than try and push out as much code as possible. Whilst we no doubt could have got more code working if we had cut corners on some of the other parts of process (e.g. analysis or testing) we would not have given a fair representation of what we do (IMHO). I’ve touched on this before in “Keeping the Faith Under Pressure” and I’m pleased with what we produced. I’m perfectly happy to call the code I wrote over the two days “production code”.

Day One

After doing our early analysis we started putting together the walking skeleton which was a text-book example of Conway’s Law as we initially kept out of each other’s way whilst we got the boilerplate stuff together. Of course this separation didn’t last long as by the end of the day the temporary assemblies and folders had all gone in a simple refactoring exercise so that we had a coherent codebase to build from.

We finished the day with our technology stack up and running (browser-based client + REST API), watched over by AppVeyor, and a very thin slice of functionality in play which we demoed to a couple of the stakeholders.

Day Two

The following day was essentially a few hours shorter to allow time for each team to present to the panel, that also meant we’d need to factor in some time to put the presentation together. In essence we only had ~5 hours to implement the next set of features and so got one of the stakeholders in first-thing to make a priority call so we knew where to focus our efforts during the morning.

During the time we three devs were building a working product our UX expert was building a clickable prototype to help explore the problem further ahead. This had the benefit of us being able to see the context in which the data was used and therefore we could better understand the bigger picture. In such a short timeframe it was perhaps difficult to see what beneficial effect it had on our design and thinking but what it did show clearly was how we work and how important we believe UX to be to the development process in general.

We stopped coding an hour before our presentation slot to give ourselves plenty of time to think about what we needed to say and show. Our UX expert had surreptitiously taken plenty of photos of us engaging with the stakeholders along with the evolving boards and backlog so that we could put together a compelling story about our approach.

We talked for 30 minutes about the process and architecture, and gave a demo of what we’d built to date. We then had about 15 minutes to answer questions from the panel, both about our approach and how we might tackle some of the questions we hadn’t yet answered in the code.

Each team got to present to the panel in isolation, but we all hung around until the end at which point we each did a very brief version of the presentation to the other teams. This was really interesting as up to that point we had no idea what the others were up to. For example we chose a thin-client whereas the other two chose thick-clients. We used post-its and a makeshift whiteboard for our notes and product backlog whilst another used an online tool and the third didn’t mention their approach.

Wrapping Up

Did the exercise achieve what it set out to do? As I’m not the client I have no idea what their final decision is or whether the eventual product has met their needs, because clearly it hasn’t been built yet. But I believe their behaviour suggested that they were pretty pleased with what they had seen from everyone. I think they seemed surprised that the three teams had behaved quite differently and so they probably got a lot more out of the exercise than anticipated as each team would probably have asked different questions and explored different avenues. Given that this was a real problem we were working on I’m sure there is a lot of value in that alone.

Personally I went into the process somewhat nervous. My current role is a real departure for me - it’s not a hands-on development role. As such I thought I might not be “match fit” even with 20 years of coding experience behind me. What I forgot though was that the point of the process was to be ourselves and write code as I would for real and that just came back naturally. Hopefully I added an equal amount of value to the team and gave as good an account of myself as the others appeared to.

So, is this is the way I think tenders should be done? Yes, but I’m just a developer :o). I did ask someone at Equal Experts about how it compared cost-wise for them given that the RFP (Request for Proposal) process can also be pretty time consuming too and he suggested it wasn’t that far off. Not having done any others I can’t say whether the client had a disproportionate number of people involved at their end but given the potential sums at stake I’m sure they saw it as a good investment. I certainly hope more companies do it in the future.

[1] Apologies if that sounded like a thinly-veiled sales pitch, it wasn’t mean to be. I was just trying to describe how they see themselves; I’m an associate, not an employee.

[2] Somewhat unfortunately the room we where in had frosted glass and the sheets of magic whiteboard didn’t really stick to it. Without any blu-tack or sellotape we had to use even more post-it notes to hold up the whiteboards!

[3] The Wi-Fi dropped out every now and then and had limited bandwidth which meant that any image-heavy web pages could take a while to load.

Monday, 7 September 2015

Stand-Up on the Beach

aotb-stand-up-lBack in April I performed a stand-up comedy routine as a lightning talk on an unsuspecting audience at the ACCU Conference (see my previous blog post The Daily Stand-Up). At this year's Agile on the Beach I got to have another go at it, but not as a lightning talk, this time it was going to form part of the pre-conference evening event (aka “pasty night”).

Whereas the ACCU conference is almost entirely about software craftsmanship, Agile on the Beach has mostly other streams covering all aspects of the software development process. As such I needed to come up with a different routine that catered for a much wider audience. Given its nature I swapped out many of the programming specific puns and replaced them with something (hopefully) more appropriate, i.e. more process related. Also there was a bonus track on continuous delivery this year so that meant I could throw in some relevant content there too.

Once again it seemed to go down pretty well, by which I mean the audience groaned appropriately :o). So for those of you unfortunate enough to have missed it, here is the set:

“Was the Tower of Pisa built using lean manufacturing?”

“Agile methods might be all the rage these days but I reckon the writing’s on the wall for Kanban.”

“Are cross-functional teams just a bunch of grumpy LISP programmers?”

“Some say Scrum’s sprint goals are necessary for motivation, but Kanban is also about cracking the WIP.”

“If you want to adopt Agile Release Trains do your developers need to use Ruby on Rails?”

“The last census had a box labelled ‘Religion’, so I put ‘TDD’.”

“When I’m working from home I like to get the kids involved in my coding; I call this ‘au pair programming’.”

“My team’s not really got the hang of this agile stuff – our successes are just stories and our fails are all epics.”

“If you have too many information radiators do you get scrumburnt?”

“The other day the product owner asked me why all our acceptance tests only covered happy paths. I said they’re rose tinted specs.”

“Agile’s a lot older than many people think – Dick Turpin was doing stand-up and deliver years ago.”

“If poor quality code results in technical debt, does that make bad programmers loan sharks?”

“Some say C# and Java programmers overuse reflection, but given the quality of their code I’d say they aren’t reflecting enough.”

“Is it me or are C# and Java so similar these days they’re harder to tell apart than The Munsters and The Adams Family?”

“Our system has five-nines reliability, it’s usually working about 45% of the time.”

“When working for New Scotland Yard do developers have to work on a special branch?”

“I really dig software archaeology.”

“As a baby I was brought up on Farley’s, and I’m bringing my children up on Farley’s too – at bedtime I read them a chapter from Continuous Delivery.”

“Are modern developers obese because they rely so heavily on syntactic sugar?”

“Is the removal of a dependency injection framework from a codebase known as ‘spring cleaning‘?”

“If you think keeping up with the Kardashians is hard, try JavaScript frameworks.”

“When it comes to drawing diagrams of micro-service architectures, I never know where to draw the line.”

“The other day I went to the dentist and he told me I had a scaling problem. I said that’s awkward as I’ve no room for any more teeth.”

“Our DR strategy is not so much active/passive as passive/aggressive – when it fails we sit around and tut loudly until someone fixes it.”

“Don’t upgrade your database as the SQL is never as good as the original.”

“I blame Facebook for the quality of modern SQL – young developers are so obsessed with LIKEs.”

“When Sherlock Holmes talks of a three-pipe problem does he mean it needs grep, sed, awk and sort?”

“I don’t understand why the police are profiling criminals – surely they shouldn’t be helping them to commit crimes more efficiently?”

“C++ has complexity guarantees – if you write C++, it’s guaranteed to be complex.”

“Some people are like ‘chars’ in C, they get promoted for no apparent reason.”

“Would our codebase be healthier if we only used natural numbers?”

“One of the hardest problems in computer science is dealing with nans – they always want you to fix their machines.”

“In my thirties I blew loads of cash on a monster gaming rig, I think I was suffering from a Half-Life crisis.”

“I forgot my password the other day so I tried a dictionary attack – I just kept hitting the administrator over the head with a copy of the OED until he let me in.”

“My wife and I have been together for many years so I thought I’d get a token ring, but it seems you can only get Gigabit Ethernet these days.”

“My son was being hassled at school to share his music on the internet with BitTorrent. I told him not to give in to peer-to-peer pressure.”

“When flying here I had to put my phone into airplane mode, at which point it assumed the crash position.”

“Are electronic cigarettes just vapourware?”

“I spent all day trying to upload a picture of Marcel Marceau but the server kept responding with ‘415 Unsupported Mime Type’.”

Wednesday, 29 April 2015

The Daily Stand-Up

At the ACCU Conference this year I managed to fulfil a lifelong ambition - do a stand-up comedy routine. OK, so it was only 5 minutes long (I did it as a lightning talk) and the audience and content was heavily programmer centric, but even so it seemed to go down a treat. A few people asked me to post the “set” that I did, so here it is.

The content all comes from my own tweets between 2009 and 2013 (except for a couple of more recent ones). Whilst this is the majority of the elapsed time I’ve been on Twitter, it only accounts for half of my tweets so far. This is because I was tweeting less than 100 times per-month at the beginning whereas I’m well over 400 per-month these days. Who says I’m an addict. It also means there should be enough content to do another routine next year (you have been warned).

I’d like to believe that it’s all my own work but the phenomena that is Cryptomnesia suggests this likely won’t be the case. Even if many are my own original thoughts it’s very likely others have been there well before me. Still I hope that you find some originality somewhere in this batch of 30 or so quips and puns.

If you’re trying to recreate the atmosphere of the original event then you’ll need to deliver these in a deadpan manner.

“My wife bought me some jigsaws of famous computer scientists for Christmas. She asked how I was getting on. I said it’s Turing complete.”

“I think Turing was a wicked mathematician and computer scientist. In fact I’d say he was da bombe.”

“Speaking of mathematicians, when they work-out do they focus on their abs?”

“Was the Tower of Pisa built using lean manufacturing?”

“I know agile is all the rage but I think the writing is on the wall for kanban boards.”

“Are cross-functional teams just a bunch of grumpy LISP programmers.”

“I’ve been writing more functional code lately. I recently tried a few numerical recipes with currying functions, but all I got was a NaN.”

“The recent census had a box marked ‘Religion’. So I put ‘TDD’.”

“Why are we surprised that Git developers are so fanatical when everything they do is dictated by a SHA?”

“I’m getting the hang of mocking. During the last code review I said ‘this is rubbish!’, ‘what were you thinking?’ and ‘are you an idiot?’”

“When I work from home I like to get the kids involved with my coding. I call it Au Pair Programming.”

“Just because it’s called code doesn’t mean it has to be cryptic.”

“If technical debt is taking shortcuts, does that make bad programmers loan sharks?”

“Our code is so bad we don’t say ‘here be dragons’, we say ‘here be weeping angels’ because every time you blink the code looks worse.”

“I blame Facebook for the quality of SQL produced by young developers. They’re obsessed with likes.”

“When I see ‘static’ used in multi-threaded code it makes my hair stand on end.”

“I like the actor model for handling concurrency, but I find it suboptimal due to the constant need to talk to their agents.”

“I’ve been refactoring my code towards a more single-entry/single-exit style. It worked out well in the beginning but now there’s less impact due to the law of diminishing returns.”

“Singletons are like onions, the moment you touch one you end up in tears.”

“Inexperienced programmers are like inexperienced swimmers, they rely too heavily on floats.”

“I’m not saying the C programming standard is loose but I reckon it should be called Subjective-C.”

“C++ comes with complexity guarantees. If you write C++ it’s guaranteed to be complex.”

“Our system has five-nines reliability. It usually works about 45% of the time.”

“When Sherlock Holmes talks about a three-pipe problem does he mean it requires grep, sed, awk and sort?”

“My daughter asked what ABBA stands for. I said 43,962.”

“Once I pulled into an overflow car park and found myself right back at the entrance.”

“I don’t understand the point of graph databases. There must be an easier way to draw a bar chart.”

“Isn’t it depressing that everything we create ends up in a folder called ‘bin’.”

“If a team mate tells you to use more lynx they mean you have a problem with body odour, not that you need to test you web site with a text-only browser.”

“If you’re going to write a résumé for a C++ job don’t forget to add your cv-qualifiers.”

“I’ve enlisted the help of Simon Cowell to help me manage my passwords. It’s called X-Factor authentication.”

“Our interfaces are less cohesive and more adhesive. Developers keep sticking new methods on the end.”

“Our codebase has so many try/catch blocks that it often feels like a game of throwns.”

“When Amazon start using drones to deliver parcels I’m going to mess with them by sticking a robots.txt on the front door.”

Monday, 26 January 2015

IIS Hosted Web API Not Caching Internal HTTP Responses

Executive Summary: Set “Load Profile” to true for the IIS application pool.

I had a fun afternoon last week trying to help a colleague work out why the responses to the internal HTTP requests from one of my team’s web APIs weren’t being cached by the .Net HTTP component. It appeared that this was all working fine in the pre-production environment, but not production itself.

Caching HTTP Responses

The code (which I’m not overly familiar with) uses one of the standard .Net HTTP request classes (WebRequest I believe) to make authentication calls to another internal, RESTful service. It was anticipated by the owner of this service that responses could be cached by the .Net framework thereby alleviating the need for our own service to manually implement a caching layer. Naturally the point of (briefly) caching the HTTP responses was to reduce load on the underlying authentication servers and they had made sure that their response headers were correctly configured to make this “all just work”.

In theory this was true, but in practice it turned out to be a little more complicated once the code as running under the real IIS worker process (w3wp.exe).

Administrator Rights

My colleague had already done a fair bit of analysis and experimentation with a spare production server and the wonderful Process Monitor tool from Sysinternals. By watching the w3wp.exe process whilst he sent requests with CURL he could see that the process was getting Access Denied errors when accessing parts of the Registry and file-system.

He had also tried configuring the IIS application pool to run under another non-service account, i.e. a support user’s account, and that appeared to work. Comparing the behaviours with Process Monitor for the two accounts he could verify that the problem was one of permissions, or so it seemed.

A natural response when seeing Access Denied problems is to grant the underprivileged account admin rights and that’s what he did and lo-and-behold things started working. However further experimentation suggested that this didn’t always fix the problem, which was mightily confusing.

At this point I joined the party...

Verifying Admin Rights

Past experience has taught me that just adding an account to the Administrators group is not always enough - it’s best to actually check the security attributes of the process in question to double-check that it really has become elevated. Conversely when removing rights I make the same check to ensure they’ve been revoked correctly.

For this check I used another of Sysinternals tools - Process Explorer. We hit an initial “Access Denied” for some process information [1] so I used the technique of starting it with “psexec -s -i” which was probably a bit of overkill (See “Process Explorer Failed to Display DLLs or Handles” for when this might be necessary). After right-clicking on w3wp.exe to show the process’s Properties and then switching to the Security tab I could see that the process was indeed a member of BUILTIN\Administrators.

It later transpired that the w3wp.exe process not correctly acquiring or relinquishing its power might have been the source of some of the earlier confusion. I too witnessed the removal of the account from the local Administrators group and after recycling the web site application pool found the w3wp.exe process still appeared to be a member. The process start time strongly suggested it had been restarted too when we recycled the app pool.

The Default Profile

We then switched back to Process Monitor and looked at the Registry and file-system access. The HTTP client was accessing various “Internet Settings” registry keys under the HKEY_USERS tree - HKU\.Default in particular. It was then reading and writing to an internet cache that was stored under “C:\Windows\Temp\Temporary Internet Files\...”.

I had always suspected that user profiles might have a role to play in this problem because I remembered back in the dim-and-distant past that the native WinInet library couldn’t (or perhaps shouldn’t) be used for making HTTP calls when running as a service. Microsoft later added another native component called WinHttp that could be used by services. IIRC the main problem back then was around configuring proxy settings, and whilst we weren’t suffering from that specific ailment this time, the registry keys it appeared to be accessing was pointing in the same direction.

We then revoked the admin rights from the service account, checked they had been revoked with Process Explorer, and then executed the test HTTP requests again so I could see whether the same registry and file-system access patterns occurred, and they did. A brief play with the REG command-line tool and DIR helped prove that, although the keys and folders existed, the unprivileged service account wouldn’t have access to them.


This suggested to me that the problem was likely down to the fact that the service account was being used by the w3wp.exe process without a user profile being loaded. As such, due to the lack of a profile, it was trying to do the same thing it would with just a default profile on hand, but it didn’t have the necessary rights to update the cache in the “C:\Windows\Temp\Temporary Internet Files” folders.

Changing the Application Pool setting for “Load Profile” to “true” seemed to fix the problem. We then applied this to another spare production server (after first proving that it was not behaving correctly) and that worked too.

Admin Rights Not Required

I think this tale highlights why the answer to many deployment issues can be solved by apparently making the service account a local administrator. The huge gain in power that right confers allows the process to read, and more importantly write, to so much of the system. Even if the correct answer is an entirely different setting (as in this case) taking the sledgehammer approach can get you running, but at a potentially disastrous cost [2].

Anyone doing service style deployments on Windows should know how to weld tools like NTRIGHTS and CACLS (or their more modern equivalents) to grant just the right amount of permissions for their processes.


[1] I’ve spent so many years supporting pre-Windows Vista systems that I still forget occasionally that I need to elevate admin tools. I usually start with a non-elevated command prompt for safety and then wonder later why a tool doesn’t work properly because I’ve forgotten that I’ve gone for safety first. Which of course is the whole point :-).

[2] If the service is compromised it might grant an intruder further access. Even if it’s an internal service putting it in a sandbox helps ensure it does what you think it should.