The Case of the Disappearing App

[Despite being almost 30 years ago, and the finer details being lost in the mists of time, the punchline has always stayed with me – it’s scar tissue I guess :o).]

I was contracting in a team which had taken over a codebase from another company, to try and improve the performance, as it was far below what was needed for even the smaller use cases they had in mind.

The front-end was a PowerBuilder based GUI and I was a C developer working on the in-process back-end library which was a scheduling engine (think: travelling salesman problem). To be more precise, the actual scheduling engine was written in Pascal which was then translated into C and statically linked into the back-end library which acted as the infrastructure code – reading & writing to the database, a large caching layer, providing the C API for PowerBuilder to interact with engine, etc.

One day there was a report from the QA team (which was effectively one guy – the same guy I mentioned before in A Not So Minor Hardware Revision) about the application just “disappearing”. There was no error message from the application, and no UAE / GPF style crash dialog from the OS either, which was Windows NT 4 at the time.

We could reproduce the problem fairly easily. It basically involved importing a new set of orders and running the scheduling engine. This was pretty much what we had been doing all along to profile the application and address the various performance issues we ran across. We had some nice test datasets that we kept using across the entire team and I suspect (now) he was probably using a newer one [1].

What was weird was that there no trace of the application terminating. If it was crashing Windows would have told us, and allowed us to attach the debugger. I even ran the application under the debugger and made sure I’d enabled “first chance exceptions” in case any Structured Exception Handling was being triggered and swallowed (it was C, not C++ so there would be no exceptions per-se). There was no mention in the event log either of anything unusual.

In the end I resorted to single stepping into the back-end and slowly drilling down into the code to see if I could work out if there was something in our code causing this. The core engine used a Tabu Search and called back out repeatedly to get various bit of data like product details, driving time between two points, etc. It was dying somewhere inside the core engine, but due to the cyclic nature of the execution it was hard to put a breakpoint anywhere useful. (This was Visual C++ 2.0 and conditional breakpoints had a habit of slowing things right down which added to the debugging burden as it didn’t trigger that quickly as-is.)

During my debugging attempts I began to notice that the memory footprint was spiking and had a suspicion that ultimately the cause was an out-of-memory problem. Native applications typically behave pretty badly when they run out of memory as it tends to lead to dereferencing a NULL pointer and the process unceremoniously crashing. But, as pointed out earlier the Windows crash handler was not being invoked so something else was going on.

Eventually I decided to put conditional breakpoints on the prologue of the CRT functions malloc() and calloc() in the hope that I could catch it returning a NULL pointer. It worked, and as I slowly walked back up the function stack I could see why the application simply disappeared – the error handling simply called abort()! Although this was quite an abrupt termination from a users perspective, from a Windows OS point of view the process had exited gracefully and therefore no crash handler was needed. Under the covers I suspect abort() called TerminateProcess() and so the PowerBuilder GUI would not have been able to catch the process’ impending doom either. If I remember correctly we found a couple of these calls to abort() and removed them so that the error could attempt to bubble back up the stack and at least the user would get notified one way or another of the unstable nature.

As an aside, this wasn’t the only memory issue we had on that project. Not counting the bizarre performance issue in A Not So Minor Hardware Revision we also suffered badly from heap fragmentation because this was before the days of the low-fragmentation heap. The knowledge I learned along the way debugging these, and other memory and heap issues elsewhere, paved the way for my first proper ACCU article Utilising More Than 4GB of Memory in a 32-bit Windows Process.

 

[1] I also seem to remember running Oracle locally so that each developer had their own database for testing which would have affected the overall memory footprint for the machine.

Fast Hardware Hides Many Sins

Way back at the beginning of my professional programming career I worked for a small software house that wrote graphics software. Although it had a desktop publisher and line-art based graphics package in its suite it didn’t have a bitmap editor and so they decided to outsource that to another local company.

A Different User Base

The company they chose to outsource to had a very high-end bitmap editing product and so the deal – to produce a cut-down version – suited both parties. In principle they would take their high-end product, strip out the features aimed at the more sophisticated market (professional photographers) and throw in a few others that the lower end of the market would find beneficial instead. For example their current product only supported 24-bit video cards, which were pretty unusual in the early to mid ‘90s due to their high price, and so supporting 8-bit palleted images was new to them. Due to the large images their high-end product could handle using its own virtual memory system they also demanded a large, fast hard disk too.

Even though I was only a year or two into my career at that point I was asked to look after the project and so I would get the first drop of each version as they delivered it so that I could evaluate their progress and also keep an eye on quality. The very first drop I got contained various issues that in retrospect did not bode well for the project, which ultimately fell through, although that was not until much later. (Naturally I didn’t have the experience I have now that would probably cause me to pull the alarm chord much sooner.)

Hard Disk Disco

One of the features that they partially supported but we wanted to make a little more prominent was the ability to see what the RGB value of the pixel under the cursor was – often referred to now as a colour dropper or eye dropper. When I first used the feature on my 486DX PC I noticed that it was a somewhat laggy; this surprised me as I had implemented algorithms like Floyd-Steinberg dithering so knew a fair bit about image manipulation and what algorithms were expensive and this definitely wasn’t one! As an aside I had also noticed that the hard disk light on my PC was pretty busy too which made no sense but was probably worth mentioning to them as an aside.

After feeding back to them about this and various other things I’d noticed they made some suggestions that their virtual memory system was probably overly aggressive as the product was designed for more beefier hardware. That kind of made sense and I waited for the next drop.

On the next drop they had apparently made various changes to their virtual memory system which helped it cope much better with smaller images so they didn’t page unnecessarily but I still found the feature laggy, and as I played with it some more I noticed that the hard disk light was definitely flashing lots when I moved the mouse although it didn’t stop flashing entirely when I stopped moving it. For our QA department who only had somewhat smaller 386SX machines it was almost even more noticeable.

DBWIN – Airing Dirty Laundry

At our company all the developers ran the debug version of Windows 3.1. enhanced mode with a second mono monitor to display messages from the Windows APIs to point out bugs in our software, but it was also very interesting to see what errors other software generated too [1]. You probably won’t be surprised to discover that the bitmap editor generated a lot of warnings. For example Windows complained about the amount of extra (custom) data it was storing against a window handle (hundreds of bytes) which I later discovered was caused by them constantly copying image attribute data back-and-forth as individual values instead of allocating a single struct with the data and copying that single pointer around.

Unearthing The Truth

Anyway, back to the performance problem. Part of the deal enabled our company to gain access to the bitmap editor source code which they gave to us earlier than originally planned so that I could help them by debugging some of their gnarlier crashes [2]. Naturally the first issue I looked into was the colour dropper and I quickly discovered the root cause of the dreadful performance – they were reading the application’s .ini file every time [3] the mouse moved! They also had a timer which simulated a WM_MOUSEMOVE message for other reasons which was why it still flashed the hard disk light even when the mouse wasn’t actually moving.

When I spoke to them about it they explained that once upon a time they ran into a Targa video card where the driver returned the RGB values as BGR when calling GetPixel(). Hence what they were doing was checking the .ini file to see if there was an application setting there to tell them to swap the GetPixel() result. Naturally I asked them why they didn’t just read this setting once at application start-up and cache the value given that the user can’t swap the video card whilst the machine (let alone the application) was running. Their response was simply a shrug, which wasn’t surprising by that time as it was becoming ever more apparent that the quality of the code was making it hard to implement the features we wanted and our QA team was turning up other issues which the mostly one-man team was never going to cope with in a reasonable time frame.

Epilogue

I don’t think it’s hard to see how this feature ended up this way. It wasn’t a prominent part of their high-end product and given the kit their users ran on and the kind of images they were dealing with it probably never even registered with all the other swapping going on. While I’d like to think it was just an oversight and one should never optimise until they have measured and prioritised there were too many other signs in the codebase that suggested they were relying heavily on the hardware to compensate for poor design choices. The other is that with pretty much only one full-time developer [5] the pressure was surely on to focus on new features first and quality was further down the list.

The project was eventually canned and with the company I was working for struggling too due to the huge growth of Microsoft Publisher and CorelDraw I only just missed the chop myself. Sadly neither company is around today despite quality playing a major part in the company I worked for and it being significantly better than many of the competing products.

 

[1]  One of the first pieces of open source software I ever published (on CiX) was a Mono Display Adapter Library.

[2] One involved taking Windows “out at the knees” – not even CodeView or BoundsChecker would trap it – the machine would just restart. Using SoftICE I eventually found the cause – calling EndDialog() instead of DestroyWindow() to close a modeless dialog.

[3] Although Windows cached the contents of the .ini file it still needed to stat() the file on every read access to see if it had changed and disk caching wasn’t exactly stellar back then [4].

[4] See this tweet of mine about how I used to grep my hard disk under Windows 3.1 :o).

[5] I ended up moonlighting for them in my spare time by writing them a scanner driver for one of their clients while they concentrated on getting the cut-down bitmap editor done for my company.

A Not So Minor Hardware Revision

[These events took place two decades ago, so consider it food for thought rather than a modern tale of misfortune. Naturally some details are hazy and possibly misremembered but the basic premise is still sound.]

Back in the late ‘90s I was working on a Travelling Salesman style problem (TSP) for a large oil company which had performance improvements as a key element. Essentially we were taking a new rewrite of their existing scheduling product and trying to solve some huge performance problems with it, such as taking many minutes to load, let alone perform any scheduling computations.

We had made a number of serious improvements, such as reducing the load time from minutes to mere seconds, and, given our successes so far, were tasked with continuing to implement the rest of the features that were needed to make it usable in practice. One feature was to import the set of orders from the various customer sites which were scheduled by the underlying TSP engine.

The Catalyst

The importing of orders required reading some reasonably large text files, parsing them (which was implemented using the classic Lex & YACC toolset) and pushing them into the database where upon the engine would find them and work out a schedule for their delivery.

Initially this importer was packaged as an ActiveX control, written in C and C++, and hosted inside the PowerBuilder (PB) based GUI. Working on the engine side (written entirely in C) we had created a number of native test harnesses (in C++/MFC) to avoid needing to use the PB front-end unless absolutely necessary due to its generally poor performance. Up until this point the importer appeared to work fine on our dev workstations, but when it was passed to the QA a performance problem started showing up.

The entire team (developers and tester) had all been given identical Compaq machines. Give that we needed to run Oracle locally as well as use it for development and testing we had a whopping 256 MB of RAM to play with along with a couple of cores. The workstations were running Windows NT 4.0 and we were using Visual C++ 2 to develop with. As far as we could see they looked and behaved identically too.

The Problem

The initial bug report from the QA was that after importing a fresh set of orders the scheduling engine run took orders of magnitude longer (no pun intended) to find a solution. However, after restarting the product the engine run took the normal amount of time. Hence the conclusion was that the importer ActiveX control, being in-process with the engine, was somehow causing the slowdown. (This was in the days before the low-fragmentation heap in Windows and heap fragmentation was known to be a problem for our kind of application.)

Weirdly though the developer of the importer could not reproduce this issue on their machine, or another developer’s machine that they tried, but it was pretty consistently reproducible on the QA’s machine. As a workaround the logic was hoisted into a separate command-line based tool instead which was then passed along to the QA to see if matters improved, but it didn’t. Restarting the product was the only way to get the engine to perform well after importing new orders and naturally this wasn’t a flyer with the client as this would happen in real-life throughout the day.

In the meantime I had started to read up on Windows heaps and found some info that allowed me to write some code which could help analyse the state of the heaps and see if fragmentation was likely to be an issue anyway, even with the importer running out-of-process now. This didn’t turn up anything useful at the time but the knowledge did come in handy some years later.

Tests on various other machines were now beginning to show that the problem was most likely with the QA’s machine or configuration rather than with the product itself. After checking some basic Windows settings it was posited that it might be a hardware problem, such as a faulty RAM chip. The Compaq machines we had been given weren’t cheap and weren’t using cheap RAM chips either; the POST was doing a memory check too, but it was worth checking out further. Despite swapping over the RAM (and possibly CPUs) with another machine the problem still persisted on the QA’s machine.

Whilst putting the machines back the way they were I somehow noticed that the motherboard revision was slightly different. We double-checked the version numbers and the QAs machine was one minor revision lower. We checked a few other machines we knew worked and lo-and-behold they were all on the newer revision too.

Fortunately, inside the case of one machine was the manual for the motherboard which gave a run down of the different revisions. According to the manual the slightly lower revision motherboard only supported caching of the first 64 MB RAM! Due to the way the application’s memory footprint changed during the order import and subsequent cache reloading it was entirely plausible that the new data could reside outside the cached region [1].

This was enough evidence to get the QA’s machine replaced and the problem never surfaced again.

Retrospective

Two decades of experience later and I find the way this issue was handled as rather peculiar by today’s standards.

Mostly I find the amount of time we devoted to identifying this problem as inappropriate. Granted, this problem was weird and one of the most enjoyable things about software development is dealing with “interesting” puzzles. I for one was no doubt guilty of wanting to solve the mystery at any cost. We should have been able to chalk the issue up to something environmental much sooner and been able to move on. Perhaps if a replacement machine had shown similar issues later it would be cause to investigate further [2].

I, along with most of the other devs, only had a handful of years of experience which probably meant we were young enough not to be bored by such issues, but also were likely too immature to escalate the problem and get a “grown-up” to make a more rational decision. While I suspect we had experienced some hardware failures in our time we hadn’t experienced enough weird ones (i.e. non-terminal) to suspect a hardware issue sooner.

Given the focus on performance and the fact that the project was acquired from a competing consultancy after they appeared to “drop the ball” I guess there were some political aspects that I would have been entirely unaware of. At the time I was solely interested in finding the cause [3] whereas now I might be far more aware of any ongoing “costs” in this kind of investigation and would no doubt have more clout to short-circuit it even if that means we never get to the bottom of it.

As more of the infrastructure we deal with moves into the cloud there is less need, or even ability, to deal with problems in this way. That’s great from a business point of view but I’m left wondering if that takes just a little bit more fun out of the job sometimes.

 

[1] This suggests to me that the OS was dishing out physical pages from a free-list where address ordering was somehow involved. I have no idea how realistic that is or was at the time.

[2] It’s entirely possible that I’ve forgotten some details here and maybe more than one machine was acting weirdly but we focused on the QA’s machine for some reason.

[3] I’m going to avoid using the term “root cause” because we know from How Complex Systems Fail that we still haven’t gotten to the bottom of it. For example, where does the responsibility for verifying the hardware was identical lie, etc.?

You Want Windows 98 Support in 2013?

A few weeks back I got an email from a chap in Australia who wanted to know if I could fix one of my DDE tools (DDE Command) to work under Windows 98 SE. After a quick check of the calendar to make sure I hadn’t entered a time warp I couldn’t help but be a little curious about where this mysterious Windows box was running and why it couldn’t be upgraded…

It turned out that this machine was fitted with a propriety ISA card (remember those?) for which support was discontinued in 1996! This chap seemed to be going through heroic efforts to keep it running by using Excel to grab data from the card via DDE. He then stumbled upon my command line tool which worked fine on Windows 7 (as it does right back to Windows 2000), but was failing on Windows 98 like this:-

C:\> ddecmd servers
ERROR: Failed to query DDE servers: DMLERR_DLL_NOT_INITIALIZED

I had not seen that kind of error before and it’s pretty fundamental too - the underlying DDEML library was apparently not initialised. All ddecmd commands seemed to report the same problem. Luckily I had more than an inkling of what it might be because the tool is pretty simple and my DDE classes have barely changed in over decade. Also I didn’t really fancy trying to cobble together a VM and source a Windows 98 licence just to help this chap out. One of the reasons I provide the source code to all my tools is exactly to allow someone to find their own (paid) support if I can’t accommodate them myself.

One of the key differences between the Windows 95/98/ME lineage and the NT/2K/XP/Vista/etc one is that the former is ANSI internally whereas the latter is Unicode. Naturally for backwards compatibility reasons the NT line can also run ANSI binaries too, with a slight overhead as it translates back and forth as required. One of the decisions I made back in February 2008 was to switch to Unicode builds by default; however the Windows build targets were still left as they were when I first ported my libraries to Win32 back in the mid ‘90’s!

#define WINVER         0x0400   //! Windows 95+
#define _WIN32_WINNT   0x0400   //! Windows NT 4.0+
#define _WIN32_WINDOWS 0x0400   //! Windows 95+
#define _WIN32_IE      0x0400   //! IE 4.0+

Luckily I have no need for anything more fancy, especially with my command line tools so it was a simpler matter of flicking the switch in Visual C++ (7.1, aka VS2003, is still my favoured version) and out popped an ANSI build. In the meantime because of the time lag between the UK and Australia I suggested to the chap that he download my GUI based DDE tool (DDEQuery) and try that. I’d remembered that this tool hasn’t been touched since 2005 and so the binary would have been an ANSI build anyway. Unsurprisingly, it worked. So I shipped off an ANSI build of DDE Command and it was “case closed”.

Every time I think it’s time to leave the past behind and get-with-the-program a question like this pops up and it feels great to still be able to support such an old OS. Although I still use VC++ 7.1 (on XP) for my personal C++ code, I still compile it with modern versions of VC++ and GCC to gain access to all that extra static analysis so that someone can just pick it up if needs be. My day job now consists of writing C# and although it’s currently .Net 3.5 I have my sights firmly set on .Net 4.5 and C# 5 as I’m no luddite.

Even though the corporate desktop standard is still Windows XP in many organisations and we now have to suffer all those annoying “you’re using an outdated browser version” banners, it feels good to know that there are others out there that have it so much worse.

A Not-So-Clever-Now String Implementation

Back around 1995 when I first started using C++ commercially[+] I picked up an excellent book by the late Paul DiLascia titled “Windows++”. It was about writing Windows applications in C++ and the book developed a framework along similar lines to the then fledgling MFC. I decided that writing my own C++ framework would be a great way to learn C++ and I (somewhat foolishly) decided that I would use a similar code style to MFC so that I could easily reuse any of my framework code in my day job (after making it production ready of course).

Reinventing the Wheel

The heart of any C++ class library back then was the String class. C++ was still evolving and my work involved using MSVC 1.52c (aka Visual C++ 1.5) which had no template[^] support so naturally I created my own ‘classic’ class that managed an ANSI char buffer:-

class CString
{
private: 
  int   bufsize;  
  char* buffer;      
  . . .
};

One of the problems with this class was that it was too easy to screw up[*] when using the printf() family of functions because it had a different memory layout to a plain char*, so if you forgot to invoke the moral equivalent of c_str() you could end up crashing the process because the bufsize member would be interpreted as a string pointer:-

{ 
  CString str = “chris”; 
  printf(“hello %s”, str); // BOOM!
}

Swapping the two members around would probably work for the single-string case above, but not for N arguments as the trailing bufsize member would become the next string:-

{ 
  CString first = “chris”, last = “oldwood”; 
  printf(“hello %s %s”, first, last); // Still BOOM!
}

Now, I felt that giving up the beauty of the printf() family was too much; I had just spent the last few years learning C and so I had the various format specifiers committed to memory. Windows also had its own variation, wsprintf(), that supported a few other useful extensions such as near/far pointers and the ANSI/Unicode build dependent variants of “%c” and “%s”.

Tempted by the Dark Side

One day whilst debugging, I stepped into some code that passed a MFC string directly into a printf() call and realised that it hadn’t gone bang. As I stepped into the printf() call I realised that the reason it had worked was because an MFC CString object had the same stack footprint as a raw char* pointer. When an object is passed into a varargs style function the object is passed by value, which in the case of a CString is effectively a single pointer. Also because the class only has a pointer based member it has no extra padding or alignment requirements – at least not on the x86 targets I was interested in.

I thought this was rather clever and set about changing my implementation so that it had similar behaviour. It hadn’t registered at first but I later realised the similarity with the implementation of the COM BSTR type which also stores the string length in an area preceding the string contents. And so I switched my implementation to this:-

class CString
{
private: 
  class CStringBuffer 
  {  
      int   bufsize; 
      char buffer[0]; 
  }; 

  char* string;  // Really CStringBuffer.buffer 
  . . .
};

So now I allocate a single chunk of memory for the header (CStringBuffer) + contents (N * sizeof(char)), but instead of storing a pointer to the CStringBuffer header as a member, I slide along the memory block and store a pointer to the first character of the string, i.e. &CStringBuffer.buffer[0]. I used an interesting Microsoft extension- the zero element array. This beast used to be quite common in the Windows headers and is a useful way to create a placeholder for a variable length array that follows a structure in a contiguous piece of memory. The alternative is the standards compliant “char buffer[1]”, but then that makes your sizeof() arithmetic a little more complicated as you have padding to account for along with the extra byte in the array. Of course to access the header structure without littering the code with casts I wrapped the necessaries in an inline method:-

CStringBuffer* CString::GetBuffer()
{ 
  return ((CStringBuffer*)string)-1;
}

All very ugly and switching from a C-style cast to a proper reinterpret_cast<> doesn’t make it any less unpleasant. Still, I happily lived in ignorant bliss for well over a decade until I decided to make a conscious effort to drop my CString class in favour of the standard string type std::string – at least for new libraries that is. The conduit was reading Matthew Wilson’s Imperfect C++ and coming across his STLsoft libraries. Suddenly I found a new impetus to deal with the impedance mismatch between MFC/ATL and Standard C++ (something I believe his new book may well be tackling).

A New Hope

The switchover went swimmingly well with my new Core and XML libraries showing the way forward whilst the old WCL, NCL and MDBL libraries became the epitome of the word ‘'legacy’. Where possible I added extra overloads to allow std:string’s to be used naturally but by-and-large the worlds were kept apart and the cost of the occasional implicit[#] conversion from a CString to a std::string was left for the profiler to find, should it ever show up.

Then I decided that where possible all new code, even in my legacy libraries, should adopt the proper string type and that I would slowly migrate code across. So I started making deeper and deeper refactorings (backed by my unit tests – where they existed) and still things looked good. Visual C++ 10 (aka VS2010) arrived on the scene and so a port was required to ensure my codebase was compatible and much to my surprise a unit test in my ANSI to Unicode conversion helper functions choked on an ASSERT inside the STL. The code was wrong, no doubt about it, but why hadn’t I see it before as I usually use STLport to keep me honest? I did a build with STLport 5.2 and it also whinged! So I fixed my iterator dereferencing bug and ran the entire suite to make sure I hadn’t entered into the murky waters of Undefined Behaviour anywhere else by accident and again it when BANG! One of my COM library tests was now failing! Running the test under the debugger revealed a more serious problem as the MS debug runtime started complaining about other matters…

Undefined Behaviour Strikes Back

It took a couple of debugging attempts but then I winced as I spotted my error. I had switched over a couple of helper functions from returning a CString to a std::string and the result of that function was in turn being passed to a varargs logging function like so:-

std::string SomeClass::toString()
{ 
  return . . .;
}
. . .
void Elsewhere::DoSomething()
{ 
  LOG(“Blah, blah, blah %s”, something.toString()); 
  . . .
}

Yup, there was no call to c_str()!

Then the true cost of my foolish design choice started to dawn on me as I realised that there was no obvious way to catch the problem at compile time because varargs functions are effectively type-less once you extend into the ‘variable’ part of the parameter list. Yes, it might be possible to do something with templates and macros but that would be non-trivial. Add in the fact that I have used varargs style functions to do all my string formatting, often in my exception handling, and you can guarantee that there are plenty of time bombs in the shape of rarely executed exception handlers just waiting to go off.

Another New Hope

Maybe it was fate, but I had always been meaning to get my Core library and its unit test suite to build and run under more than just Visual C++. So I started investigating the alternatives such as GCC and Digital Mars (see “GCC for the Visually C++ Impaired”) and soon I had both Core and XML working under GCC 3.x and 4.x via the Code::Blocks IDE – albeit only with the standard warning level.

After being distracted for some months I finally got back to playing with GCC as I had always intended on compiling my code with the highest warning settings available to see how much static code analysis GCC can do over-and-above what VC++ already does – an awful lot it turns out (a topic for another day). And, guess what one of the behaviours that it singles out? Yup, the passing of non-primitive types to printf() style functions! I’ve yet to dig deeper and see if it is varargs functions in general or whether it’s functions like printf(). Either way it spots code like this:-

{ 
  std::string text(“. . .”); 
  printf(“%s”, text);
}

Which is great because I now have a clear path forward for deprecating my CString type safely:-

1. Add a c_str() method to my CString type to provide compatibility with std::string
2. Start compiling[$] more of my codebase with GCC 4.x to identify and fix those bugs by appending a call to .c_str()
3. When a suitable refactoring moment arises switch code from my CString type to the standard string type
4. Test, test and test some more…

Epilogue

I often wonder whether it’s still worth maintaining my 15 year old C++ codebase of tools. Many of them were made obsolete years ago and so I have stopped fixing up the really old applications but the libraries live on – kept alive by each new utility that I write. I could start again from scratch but you don’t get that luxury in the real world so I’m inclined to try and refactor my way out. This way it remains a learning exercise – albeit one about refactoring legacy code – which sadly is what a lot of commercial C++ projects have become.

 

[+] I first used it at university some years earlier and at that time you used the ‘overload’ keyword to mark overloaded functions. I managed to get G++ (nay GCC) working on my Atari 1040 STE (with twin floppy drives) but it was not a pleasant experience. I already thought C was too bloated in comparison to Assembly Language and C++ was therefore even more pointless. Sooo naive…

[^] Not that I knew what a template was back then of course. I never really started to understand templates until the turn of the millennium and Template Meta-programming is still largely theory to me – that’s why there’s the Boost libraries!

[*] If you were lucky it was in a common code path and you got a UAE/GPF there and there; but often you weren’t and it became a time bomb if the code path was in an error handler that was called infrequently.

[#] Yes, I added a conversion operator to ‘const char*’ to avoid having two overloads on every text API method or littering my code with .c_str() style warts. Choose your poison…

[$] Sadly, although my Core and XML libraries link and I can therefore run the unit tests, I can’t do the same for my WCL, COM or MDBL libraries or projects dependent on them (i.e. everything). But that’s ok because it’s the static code analysis I’m after anyway.

DDE Is Still Alive & Kicking

Dynamic Data Exchange (DDE) is an ancient inter-process communication (IPC) mechanism carried over from the 16-bit Windows days. Post millennium Windows programmers are probably more used to a diet of COM, but there still seems to be life in the old dog yet. Maybe there are less people around to answer questions on DDE, or perhaps the other old timers are ignoring them in the hope they'll go away because I seem to have had more emails on the subject this year than ever.

I presume the reason the questions are still coming my way is because I have a number of freeware tools on my website aimed at working with DDE Servers that are still actively supported:-

• DDE Query is my oldest tool and is a GUI based utility for sending requests and creating advise loops on items. It was written as the main test harness for my DDE library.
• In contrast DDE Command is my most recent addition and is the console based counterpart to DDE Query. It also provides the ability to invoke XTYP_EXECUTE transactions.
• In between the two is my DDE COM Client - an automation compatible inproc COM component that can be used as a DDE Client for scripting scenarios, such as VBScript.
• The final, and second oldest utility I provide, is a Network Bridge to allow DDE Servers to be accessed remotely. It is entirely transparent to the client and server, unlike the built-in Windows NetDDE service.

These all use my own C++ DDE library which in turn is based on the DDEML C-API library that Microsoft provides. Under the covers DDE is just a message based protocol that uses Windows messages to encapsulate a connection between two applications – known as a Conversation. Data is passed by allocating it with the old GlobalAlloc() API, and the format is determined by either using a standard clipboard format, such as CF_TEXT or a custom one via RegisterClipboardFormat(). The fact that it is message based is also its biggest limitation as it means you cannot share data between machines – you can't even share data across desktops on the same machine. The only book I came across on the subject was Ole 2.0 and Dde Distilled by Al Williams, but it covers the topic pretty thoroughly.

Before working in finance the only time I had come across DDE was when writing an installer back the mid 90's. Under Windows 3.x you used DDE to communicate with Program Manager (the shell that today is known as Explorer) so that you could create a "Program Group" and the icons for your application. You can still see this legacy today if you fire up DDE Query and use the "Server | Connect…" option, where you will spot a server called PROGMAN with a single topic also called PROGMAN. If you open this conversation and request the item "Accessories" you will be shown a CSV formatted text block with the programs from the Accessories Start Menu folder.

Once I started working in finance I discovered that Excel was the traders tool of choice. Excel can pull data from a number of sources, but the legacy option for real-time data was DDE. The big providers like Reuters, Telerate & Bloomberg all provided tools to allow you to feed their financial data into a spreadsheet and the in-house software we developed followed the same architecture. Although COM was probably Microsoft's promoted technology, DDE felt far simpler to implement.

These days I don't work with that kind of real-time feed, but the questions I do get on DDE always seem to have RIC's in the examples, so I guess that it's the financial industry that is keeping this prehistoric mechanism alive.

A Potted History of My Windows Framework

I thought that before leaping into a few posts about my experiences porting my Windows Framework to 64-bit Windows and GCC that I would spend a few minutes explaining how it has come to be the way it is. If you've looked at the source code to my various libraries you'll see a mixture of Hungarian & Java'ish notations, plus some MFC'isms and two different comment styles. Some parts of the framework are very old and as I only spend part of my free time on it, it is in a way, treated very similarly to commercial code - new features often come first and consequently it has accrued a significant level of Technical Debt.

The oldest parts of the code actually date back 15 years or so to my first framework which was written in C as I started my professional career writing Windows 3.x applications in C. This was the Petzold era and the code was pretty much adapted straight from his famous book - Programming Windows.

However, this version didn't last very long as I soon got back into C++ again at work and discovered the late Paul Dilascia's book Windows++. Instead of starting from scratch though, I refactored most of the C library, which was written in an OO'ish style, into classes. The code also followed the house style of their C++ framework as I was used to it.

It was still just a framework without a purpose though until I went freelancing and got my first contract. I now had a need for an application to track my hours for billing and so Task Tracker was born. The contract involved maintenance on an MFC based application and during this time I realised that it would be beneficial if I could reuse code from my framework and potential applications in my professional work. I didn't want to be tied to the real MFC framework tough - after all the point of re-inventing the wheel was to learn, so I decided to refactor my core framework to be more like MFC in style so that moving code between work and home would be easier.

It turned out to be a good move as I continued to work on MFC based code for the next 10 years and I got to reuse some of my classes like CIniFile & CPath quite a lot. During that time I read the classic Design Patterns book and tried to apply some of it to my own MVC like framework as that was the one area of code that I wasn't going to reuse outside my own apps. I also went through an "everything should be a reference" phase as I couldn't decide on how best to deal with object lifetimes.

The main thing missing from my codebase though was use of the STL which was largely down to the fact that MFC doesn't use it and so it doesn't tend to be the default choice. Also I had written my own containers because Visual C++ 1.5 had no template support (and frankly I wouldn't have known what to do with it then anyway :-) and that had created a large body of legacy code to refactor.

My last contract, which although heavily COM/ATL based, was an entirely different beast however. They used STL extensively and was beginning to use Boost for more than just shared_ptr. They also used STLport instead of the Dinkumware offering and there was little to no use of Hungarian Notation. And there was not a line of MFC code in sight! However the biggest factor was the large number of ACCU members that worked there...

After joining the ACCU, I almost went back to basics and began to discuss quite fundamental issues, like style, testing, C++ conformance, tools etc and I began to realise that my personal codebase was somewhat of an embarrassment. So I decided it was time to try and modernise it by deprecating my own containers, switching to the more lightweight Doxygen style comments, using size_t consistently, etc. The most important change though was dropping my own variation of Hungarian Notation and using the more common camelCase convention.

Structurally I have started to split off parts of the Windows C++ Library into a new base class library called Core which I hope to make reasonably platform independent. The single biggest effect of this is that std::string has now become the default string class instead of my own MFC-like CString. My new XML library is the first to take advantage of this and I hope to ensure both libraries compile with GCC as well as Visual C++.

The changes are not just in the code, but also in the way I develop it. I have written my own simple Unit Testing framework which loosely follows the xUnit style and have adopted TDD when possible whilst making changes to all the libraries. Core has the most test coverage as it is the newest and was created using TDD from the start. The other libraries are gaining coverage as and when I make changes.

So, is it worth it? I've always viewed my framework as a playground - somewhere outside of work to play around and make mistakes - but more importantly to try out new things that I hope can then be fed back into my professional career to make me more valuable to a potential employer. I'm still learning heaps every time I refactor, so I guess it is :-)