My First Published Articles

Today the postman delivered the July 2009 issue of C Vu - the magazine of the ACCU. In it features two pieces by yours truly :-) Okay, so these aren't a couple of earth shattering discoveries about modern software development, but a couple of reviews. The longer review is of the 2009 ACCU Conference back in April, and the shorter is of the ACCU London meeting in May which featured Jeff Sutherland.

I guess the reason I'm more than a little pleased with myself is that I find writing hard. This is exemplified by the fact that I got U's for both my English Language and Literature O'Levels. For those of you unaware of the English school system, O'Levels were the exams you took at 16 - now called GCSE's. You were graded from A-E, with U covering everything below an E. The standing joke from my fellow classmates was that you got an E just for writing your name on the paper! An English language O'Level of grade C or above is pretty much required for any form of higher education in England so I had to retake until I got one... and I did the following year thankfully.

Once I left University I was glad to leave all that writing malarkey behind, I thought that Computer Programming was a great career where I would never have to write essays again. Somehow I managed to avoid any sort of formal writing for over a decade, with class descriptions in UML models being the largest paragraphs I wrote.

But this all changed when I switched jobs a few years ago. The company I joined required me to write some formal documentation for the Build System and other processes I had developed. The documents were painful to write but I actually found myself enjoying it, probably because I was writing about something that I actually liked and knew about. The distributed team structure also meant that email was the main form of communication and so constant writing started to become the norm.

I still find writing very hard and so I started this blog in the hope that it would help me improve my skills further. So far I'm still enjoying the process (probably because blogging allows me to talk about, well, me :-). Perhaps I would have got that English O'Level a little quicker if blogging had been around when I was at school...

Using Visual C++ Together With GCC - Some Alternatives

My last post described how I have started using GCC 4.4 to compile my codebase so that I gain a little more portability, but more importantly I get some extra static code analysis without spanking my wallet. My Heath Robinson-esque solution involves using the free Code::Blocks IDE as it can import Visual C++ projects and solutions with minimal fuss, they are easy to maintain, and it's only an extra .cbp and .workspace checked into the SCM repository - so it's not messy.

As I mentioned before this was a suggestion from fellow ACCU member Steve Love, but since then I have had a some discussions on the accu-general mailing list to see what alternatives there are...

I guess the most obvious alternative would be Eclipse + CDT (C/C++ Development Tooling) as this mirrors my current setup of an IDE + GCC. Given the volume of development in Eclipse it also probably makes the most sense, but I have not had a good experience with Eclipse or CDT in the past - probably because I've been spoilt by the simplicity of installing and using Visual C++. I recently tried the CDT 5.0.x release but once again got lost in the Eclipse menus trying to configure it for my MinGW installation. This was in stark contrast to Code::Blocks where I had it configured in minutes. I guess I shouldn't be surprised that the learning curve of a tool as powerful as Eclipse is going to be much steeper, but it's just too steep for my VC++ biased plan at the moment. I should also point out that I haven't even opened the manual, so cries of RTFM are entirely valid and I only have myself to blame.

Both the Code::Blocks and Eclipse solutions rely on a separate IDE to manage the build process which is annoying because of the additional maintenance required to keep them in sync with the master Visual C++ projects. So, is there a way to drive GCC directly from the Visual Studio IDE?

A little bit of Googling during my earlier research led to a tool called gnu2msdev that translates the output of GCC to the same format that CL produces so that the Visual Studio IDE can navigate to lines with compilation errors. From this I discovered a number of posts suggesting the use of a makefile project, but they were more about using Visual Studio as an expensive text editor rather than compiling the same source with two compilers. It feels like there might be something possible along these lines and I'm going to continue investigating in the background.

I've never used the Intel C/C++ compiler, but another fellow ACCU member, Anna-Jayne Metcalfe of Riverblade, who knows a thing or two about integrating with Visual Studio, pointed out that Intel have managed the tight integration, so perhaps it would possible to do the same with GCC? I've looked at the MSDN documentation and it looks somewhat complicated for now :-) However, the signs are that it will be somewhat easier in the future as Microsoft have been slowly separating the IDE and build system so that it can support other languages and toolchains. The VS Project Team Blog has a number of posts on the changes to Visual Studio extensibility in VS2010 that show what they are up to.

For the moment I'm sticking with the Code::Blocks approach, not least because the .cbp file format is simple and I reckon I could solve the sync'ing problem with a bit of script. However I'm still keen to get into Eclipse because there is so much development clout behind it. Hopefully by the time VS2010 becomes my baseline someone else would have already written the GCC integration and I'll get another free ride :-)

GCC for the Visually C++ Impaired

At this years ACCU conference Steve Love gave a talk titled Why Portable Code?. It covered far more than just toolchains and platforms, but it reminded me of a previous chat I had with Steve about portability after one of the recent ACCU London gatherings. I've always liked the idea of writing truly portable code, but quite frankly in the corporate waters where I swim Visual C++ is The Big Fish and any ideas about using alternate toolchains to satisfy personal desires around "writing quality code" would be seen as frivolous - irrespective of whether it has long term benefits or not.

In my discussion with Steve I explained that I hadn't even looked at a Makefile in a long time (according to my personal SourceSafe repository it's 1997) and I didn't feel that I could introduce any other build system just to enable portability checks. What I wanted initially was a quick way of taking a C++ codebase and just running it through A.N. Other compiler. It wouldn't have to link or run, just allow me to check compilation. Steve's suggestion was Code::Blocks, an Open Source, cross platform IDE, that was obviously also free. It turned out to be much better than I had hoped for and has also given me a partial answer to my previous blog 'Where Are the "Lite" Editions of Static Code Analysis Tools?'.

As you can see from my earlier post 'Building Visual C++ Projects From the Command Line', I am not adverse to command line tools, but the thought of trying to use another toolchain, especially one with a Unix heritage like GCC, didn't exactly fill me with joy. The last time I had used G++ was on a twin-floppy Atari STE circa 1990 and it needed a Unix like OS to run it. Googling didn't fill me with any more confidence as I started to read about Unix emulation layers like Cygwin and I didn't want to have to learn another 'Shell' - I just wanted to be able to run the compiler from a Windows command prompt. The answer again turned out to be pretty simple - MinGW - a Win32 port of GCC for building Windows applications. Feeling cocky I also downloaded the Digital Mars C/C++ compiler as well after noticing support for it in the Code::Blocks UI.

Now, I only wanted the IDE for the build system, not the text editor. So I created a simple Hello World C++ console application in Visual C++ and then switched to Code::Blocks to see if I could get it to build. After hunting though the menus I discovered how to configure a toolchain (it had detected VS2005 Express automatically, but not my VS2003 Professional) and noticed that it had an option to import a Visual Studio Solution or Visual C++ Project. I picked the former and played around in the UI to see what it had done. Amazingly I hit build and it worked... and ran too! I discovered how you switch compilers on a project and pointed it to my MinGW 3.4.5 installation and lo-and-behold it also worked (except for some warnings about unknown compiler switches which is an artifact of switching tools). I tried the Digital Mars compiler, after also downloading STLport, and it compiled, but didn't link (I discovered later that I had to specify the stlport .lib manually). I then got hold of the MinGW 4.4.0 release, but now it all went horribly wrong. However that was entirely my fault due to a lack of RTFM'ing the release notes. It was actually a blessing as I discovered the TDM GCC builds as an alternative to MinGW and it's "On-Demand Installer" worked out-of-the-box and better suits a heathen like me.

With my new found confidence I decided to take my Core library, which is the more platform agnostic part of my framework and give it a whirl with GCC 3.4.5 (I didn't have 4.4.0 working at that point). Once again I used the import feature of Code::Blocks to configure the build and it complained quite loudly at my code, which I kind of expected :-) But all the problems were quite simple and related to my use of Visual C++ specific #pragma's in the common build configuration header files. Once these were fixed the rest of the code was pretty sound - with GCC 3.4.5 that is. GCC 4.4.0 being much newer and stricter had a LOT more to say! I am going to blog about the changes I made because I was pleasantly surprised at how little I needed to change on the structural front to enable GCC support, but also the kinds of issues 4.4.0 has raised are squarely in the Static Analysis arena and not reported by any version of Visual C++ up to 9.0 (i.e. VS2008).

I feel somewhat annoyed that it has taken me so long to truly see the light. In Steve Love's talk he identified three stereotypes - The Ignorant, The Skeptic & The Zealot. I don't believe I'm any single one of those, as I've always agreed with the principle, but I probably share traits from all of them and add in a dose of procrastination for good measure. As I write this I'm just finishing up porting the oldest part of my framework, my Windows Class Library, and as soon as that is done I'll write some posts about the discoveries I've made.

Where Are the "Lite" Editions of Static Code Analysis Tools?

When I started out, the compiler I was using was set to build on warning level 2 (it was MS C600) which pretty much just told you if your code was well formed or not, and that was all I cared about then. Fortunately whilst working there the company discovered Writing Solid Code by Steve Maguire along with Code Complete by SteveMcConnell. One of the practices Steve Maguire suggests is cranking the diagnostic level up on your tools to maximum and leaving it there - pretty much a best practice these days. The net effect of this is to enable the static code analysis within the compiler to highlight your 'valid', but potentially dubious code, giving you a chance to fix it before it becomes a bona fide bug. Yes, sometimes it gets it wrong, but there is nearly always a way to rewrite the code to silence the compiler (preferable) or a sledgehammer in the form of a pragma to disable the warning for a small section of code.


But compilers are first-and-foremost seen as a build tool - the static code analysis abilities are a nice-to-have feature that has probably dropped out as a by-product. For full-on code analysis you apparently need an industrial strength tool, which quite probably comes at an industrial strength price. Now I'm sure these tools have taken a considerable investment and therefore demand a high price, which is fine for a big corporate client who can also afford the training required, but for smaller outfits and freelance developers the cost is prohibitive. Nowadays I can get started in C++ development using good quality free tools like Code::Blocks for the IDE and GCC for the compiler suite or even the all-in-one: Visual Studio Express. But what about when I need to take my skill further so that I can improve the quality aspects of my coding?


At various clients in the past I have used tools like BoundsChecker (now DevPartner Studio) and Rational PurifyPlus to check for memory leaks, buffer overruns, uninitialised variables etc. and they have been useful, but I don't feel it was the most efficient use of my time. They are a great way of tracking down a specific issue, but not at all the right tool for continuously sanity checking the codebase. On the other hand one of the oldest and potentially useful tools for doing this with C++ is probably PC-Lint (something I am keen to get my hands on - more so since going to Anna-Jayne Metcalfe's ACCU talk Taming the Lint Monster). And even Microsoft have got in on the act and added an /Analyze switch to its Visual C++ compiler - but only in the Enterprise edition.

Code Analysis (Static or Dynamic) is a noisy business, meaning that you need to be proficient in the tool to make the best use of it. But to do that requires some serious time with it and many businesses just wont be able to justify the cost of both the tool and time required for training. And this is normally where those "Lite" editions come into play. Freelancers (and hobbyist coders) like myself, who often have the ability to affect buying decisions would have considerably more weight if we were able to demonstrate the real benefits because they were a natural part of our toolset.

One client I worked at lost many man days due to uninitialised member variables, something I know at least one static code analysis tool points out. Personally I get annoyed when I see a bug that I know a tool could have prevented. What I really wanted to introduce into the build process was a step to run a short static code analysis job before the build and then later, as we got more proficient with the tool, add a weekly step to run a much deeper analysis. The stumbling block was that I could not even get a trial version of the product I wanted to use so that I could play in my own time to build a list of current defects to use to justify the cost.

Having the knowledge of certain products on my CV naturally helps make me more marketable, but more importantly, as a professional I feel I have another weapon in my arsenal to help ensure the quality of my code.

The Default 'Size' & 'Index' Type: size_t vs int

I started out writing applications in C on 16-bit Windows. The company I worked for favoured using the Windows SDK functions instead of the C-Runtime, so for example I would use lstrlen() instead of strlen(). Unfortunately, the Windows SDK is biased towards using int instead of size_t for parameters and return values that represent counts and indices. As I moved into working on MFC based applications the habit continued as MFC has it's own container types, such as CArray, which also favoured int, as does the popular common controls, like the CListCtrl. The final bias towards int was the use of -1 as an error code when using controls such as the combo and listboxes through constants such as CB_ERR and LB_ERR.

Unfortunately this habit got carried over into my own framework because I also had my own string and container classes and my wrapper facade is very thin in many places. So it wasn't much of an issue until I finally junked my clunky containers and started using the STL in preference...

All of a sudden the compiler was constantly complaining about signed/unsigned comparisons which would ripple through my code as loops where changed from,

for (int i = 0; i < v.Size(); ++i)

initially to,

for (int i = 0; i < v.size(); ++i)

and finally to,

for (size_t i = 0; i != v.size(); ++i)

(Changing the code structure to use iterators instead - the correct solution - was going to be an exercise for another day.)

I realised that a large part of my codebase was int-based instead of size_t-based and the number of uses of static_cast was growing and making the code even uglier so I decided to bite the bullet and go size_t across the board. The entire codebase isn't massive (160,000 LOC or 45,000 SLOC according to the excellent Source Monitor) and it took a few train journeys but it felt good. However one subsequent annoyance was with the comparisons to CB_ERR etc as they are just #define's for -1 so I followed the STL string type (which returns -1 as a result for some of the find() methods) and declared a global constant,

namespace Core
{
static const size_t npos = static_cast<size_t>(-1);
}

In retrospect I realise I should have declared an 'npos' in each class, such as CComboBox and CListBox, instead of globally, but there was quite a bit of code that just had "= -1" as an argument default and I got lazy trying to resolve all the issues quickly.

Now my codebase felt more wholesome because of the large scale refactoring of int to size_t and I felt more aligned to the C++ world. So I decided to see what happened when I ran the 64-bit cross compiler from the Platform SDK over it....

Yup, lots of errors about truncation from a 64-bit value to a 32-bit value (I always compile with /W4 /Wx) , i.e. conversions from a size_t to an int or DWORD! It appears that functions like GetWindowText() still traffic in int's and many of the I/O functions that specified sizes for buffers still use DWORD's. So, more static_casts went in.

And now the feeling of wholesomeness is gone again as I contemplate the impedance mismatch between the Windows API and the C/C++ standard libraries. My framework is largely a Wrapper Facade and is therefore thin, but I don't believe it should expose the use of int or DWORD to its clients. I also have use of my own 'uint' type in various places where I knew an int was undesirable, but that now also requires a cast when the source is a size_t. I even started a thread on one of the ACCU channels to see if other people felt that size_t was overused and if it would be a good idea to at least use an index_t typedef for [] operators and index style parameters.

For now the possibly excessive use of size_t stays whilst I chew the fat. I also want to deal with the nastiness that WPARAM and LPARAM has created in the framework due to the documentation using these types without describing what a suitable type would be (e.g. the Character Code - for WM_CHAR) as I want to use richer typedefs instead of the limited set the Windows API uses (e.g. WCL::TimerID instead of UINT_PTR).

Visual C++, the INCLUDE variable and the /USEENV switch

Back on my last contract I ported the codebase from VS2003 (VC++ 7.1) to VS2005 (VC++ 8.0). The port was pretty easy, but is wasn't until we started developing with VS2005 that we ran into a really nasty issue.

The system used STLport instead of the bundled STL for a number of reasons, and if you've used STLport with Visual C++ you'll possibly use a .cmd script, along with the devenv /useenv command line switch as a way of injecting the STLport paths into the #include chain before the standard MS ones. Here is an example of how that script built the include path and launched VC++,

. . .
set VCInstallDir=%VS80COMNTOOLS%..\..
set STLPORT=%DEV%\3rdParty\STLport-4.6.2
set BOOST=%DEV%\3rdParty\boost.1.33.0
. . .
set INCLUDE=^
%STLPORT%\stlport;^
%VCInstallDir%\VC7\include;^
%VCInstallDir%\VC7\atlmfc\include;^
%VCInstallDir%\VC7\PlatformSDK\include;^
%DEV%\OurCode\Lib;^
%BOOST%;^
. . .
start devenv.exe /useenv OurSolution.sln

Notice that the codebase also used the INCLUDE variable as the mechanism for finding it's own library code - the line "%DEV%\OurCode\Lib". These libraries were not 'static' in nature they were developed at the same time as the main code and so any library change was expected to cause the relevant dependencies to build immediately.

When I started using VS2005 for actual development I began to run into strange problems, which would manifest as bad builds - virtual functions calling the wrong code, memory corruption etc. Previous experience taught me to disable the "Minimal Rebuild" option and delete the .idb file, but they didn't seem to have any effect. Once my teammates joined me it became a real issue as we were wasting a considerable amount of time. We even talked about dropping VS2005 and going back to VS2003 until we could get our hands on VS2008.

Fortunately one of my colleagues, Sergey Buslov, was not going to be beaten so easily and after an exhaustive search came across a new setting in VS2005, called "External Dependencies". This resides in the same location as the other VC++ paths - "Tools ! Options" under the "Projects & Solutions ! VC++ Directories" section. This is where you normally configure the default Executable, Include, Lib etc paths. The "External Dependencies" list was exactly the same as the INCLUDE environment variable, and lo and behold if you cleared the list, all the dependency issues went away. The documentation only has this to say on the setting,

Exclude Directories
Directory settings displayed in the window are the directories that Visual Studio will skip when searching for scan dependencies.

I haven't seen any clarification yet, but I suspect that the Visual Studio team assume that you'll only put paths to 3rd party libraries in your INCLUDE variable. In my own work I have always had a single IncludeDependency that points to the root of my libraries source tree which is specified in every project so I didn't see this at home.

The real kicker though is that there is no obvious way of stopping Visual C++ from configuring the "External Dependencies" from the INCLUDE variable when using the /useenv command line switch! There is no EXTERNAL_DEPENDENCIES variable that I know of that would allow you to configure it appropriately (it does seem a very useful optimisation). In the end the hack Sergey came up with is really ugly :-) He found that if you modified the default paths, VS wouldn't interfere and inject the INCLUDE paths into it. So here are the instructions,

1. Open Visual C++ from the standard icon, NOT by invoking it with devenv /useenv or your script.
2. Open the "Tools ! Options" dialog and find the "External Dependencies" list.
3. Modify them by adding another dummy entry at the end. We just stuck the string "(null)" in there for want of something better.
4. Close Visual C++ and re-open it with your script and check that the "External Dependencies" list has not been hijacked.

At the time we hit this (06/2008) there was only one post I could find that seemed similar to our problem,

http://social.msdn.microsoft.com/forums/en-US/vcgeneral/thread/c447c0bd-bcdb-4e52-a8af-b1341ce3ad9f

I notice that it's also referenced in the following Microsoft Connect query, but closed as 'unreproducible',

http://connect.microsoft.com/VisualStudio/feedback/ViewFeedback.aspx?FeedbackID=274454

I'll see if I can open a new issue and maybe more information about this will come to light.

Building Visual C++ Projects From the Command Line

As a developer biased towards Windows applications I find that both as a professional and a hobbyist that Visual C++ is my day-to-day IDE. I still use VC++ 7.1 (aka VS2003) for my personal work as I ship source code as well as binaries and it's easier to upgrade projects and solutions to later editions of Visual C++ for compatability testing than to go backwards. With 6 class libraries (+ unit tests) and 12 applications to build, manually launching the solutions and interacting with the GUI is just tedious and obviously error prone. But a common question on the forums is whether it is possible to build projects using Visual C++ from the command line - sometimes I wonder if people take the "Visual" in Visual C++ too literally :-)

The starting point is firing up a command prompt and setting up the environment variables so that you can invoke Visual C++ to build a solution. Some editions of Visual C++ have a "Command Prompt" item listed in the External Tools - VS2008 does. If you look at the command for this you'll see that it runs a batch file called vcvars32.bat. But opening the IDE just to fire up a command prompt is madness. Fortunately MS added an environment variable from VC++ 7.0 to make life a little easier - VSxyCOMNTOOLS. This means you can open an arbitrary command prompt and run, say,

C:\> "%VS71COMNTOOLS%..\..\vc7\bin\vcvars32.bat"

Unfortunately the VC folder changed name with VC++ 8.0 to drop the version suffix, so it's,

C:\> "%VS80COMNTOOLS%..\..\vc\bin\vcvars32.bat"

And should you want to cross-compile 64-bit code on 32-bit windows, with VS2008 it's as easy as,

C:\> "%VS90COMNTOOLS%..\..\vc\bin\x86_amd64\vcvarsx86_amd64.bat"

That's still just a little too awkward for me to remember, so I have a .cmd file called SetVars.cmd that I use to setup the variables for whatever version of VC++ I'm testing with,

C:\> SetVars vc71

That covers setting up the enviroment variables. The next job is to invoke Visual C++ to build a solution. This can be done in one way under VC6.0-7.1 and two ways under VC8.0-9.0. The old way was to invoke DEVENV.com like this,

C:\> devenv /nologo /useenv "D:\Dev\my.sln" /build Debug

As of VS2005, you can now use the MSBUILD style tool called VCBUILD. Unfortunately this has a different command line format and cannot be used to build installer (.vdproj) projects. The equivalent command would be this,

C:\> vcbuild /nologo /useenv "D:\Dev\my.sln" "DebugWIN32"

Once again though I find this all a bit tedious and have wrapped this logic into another .cmd script called Build.cmd that just takes a solution filename. The choice of which tool to use, devenv or vcbuild, is based on the compiler that was configured by running the SetVars.cmd script.

C:\> Build "D:\Dev\my.sln"

The natural extension to this is building all my libraries, unit tests and applications from one command so I can see what I have broken. A straight FOR loop wrapped into another .cmd script called BuildAll.cmd helps reduce the clutter from this raw example,

C:\> FOR /R %I IN (*.sln) DO devenv /nologo /useenv %I /build Debug

The final script I use is for upgrading the projects and solutions from VS2003 to VS2005, VS2008 etc before building. By default I keep VS2003 version files in my source repository and then run the upgrade script followed by the build script to check compilation on later versions of Visual C++,

C:\> FOR /R %I IN (*.vcproj) DO vcbuild /upgrade %I
C:\> FOR /R %I IN (*.sln) devenv /upgrade %I

The one fly-in-the-ointment is that the Express editions of Visual C++ don't allow you to upgrade solutions - only projects. At least not from the command line. However you can do it through the GUI, which means invoking the IDE for each solution and manually clicking the wizard buttons!

C:\> FOR /R %I IN (*.sln) devenv /upgrade %I

The scripts are available on my website here.