Visual C++, the INCLUDE variable and the /USEENV switch

Back on my last contract I ported the codebase from VS2003 (VC++ 7.1) to VS2005 (VC++ 8.0). The port was pretty easy, but is wasn't until we started developing with VS2005 that we ran into a really nasty issue.

The system used STLport instead of the bundled STL for a number of reasons, and if you've used STLport with Visual C++ you'll possibly use a .cmd script, along with the devenv /useenv command line switch as a way of injecting the STLport paths into the #include chain before the standard MS ones. Here is an example of how that script built the include path and launched VC++,

. . .
set VCInstallDir=%VS80COMNTOOLS%..\..
set STLPORT=%DEV%\3rdParty\STLport-4.6.2
set BOOST=%DEV%\3rdParty\boost.1.33.0
. . .
set INCLUDE=^
%STLPORT%\stlport;^
%VCInstallDir%\VC7\include;^
%VCInstallDir%\VC7\atlmfc\include;^
%VCInstallDir%\VC7\PlatformSDK\include;^
%DEV%\OurCode\Lib;^
%BOOST%;^
. . .
start devenv.exe /useenv OurSolution.sln

Notice that the codebase also used the INCLUDE variable as the mechanism for finding it's own library code - the line "%DEV%\OurCode\Lib". These libraries were not 'static' in nature they were developed at the same time as the main code and so any library change was expected to cause the relevant dependencies to build immediately.

When I started using VS2005 for actual development I began to run into strange problems, which would manifest as bad builds - virtual functions calling the wrong code, memory corruption etc. Previous experience taught me to disable the "Minimal Rebuild" option and delete the .idb file, but they didn't seem to have any effect. Once my teammates joined me it became a real issue as we were wasting a considerable amount of time. We even talked about dropping VS2005 and going back to VS2003 until we could get our hands on VS2008.

Fortunately one of my colleagues, Sergey Buslov, was not going to be beaten so easily and after an exhaustive search came across a new setting in VS2005, called "External Dependencies". This resides in the same location as the other VC++ paths - "Tools ! Options" under the "Projects & Solutions ! VC++ Directories" section. This is where you normally configure the default Executable, Include, Lib etc paths. The "External Dependencies" list was exactly the same as the INCLUDE environment variable, and lo and behold if you cleared the list, all the dependency issues went away. The documentation only has this to say on the setting,

Exclude Directories
Directory settings displayed in the window are the directories that Visual Studio will skip when searching for scan dependencies.

I haven't seen any clarification yet, but I suspect that the Visual Studio team assume that you'll only put paths to 3rd party libraries in your INCLUDE variable. In my own work I have always had a single IncludeDependency that points to the root of my libraries source tree which is specified in every project so I didn't see this at home.

The real kicker though is that there is no obvious way of stopping Visual C++ from configuring the "External Dependencies" from the INCLUDE variable when using the /useenv command line switch! There is no EXTERNAL_DEPENDENCIES variable that I know of that would allow you to configure it appropriately (it does seem a very useful optimisation). In the end the hack Sergey came up with is really ugly :-) He found that if you modified the default paths, VS wouldn't interfere and inject the INCLUDE paths into it. So here are the instructions,

1. Open Visual C++ from the standard icon, NOT by invoking it with devenv /useenv or your script.
2. Open the "Tools ! Options" dialog and find the "External Dependencies" list.
3. Modify them by adding another dummy entry at the end. We just stuck the string "(null)" in there for want of something better.
4. Close Visual C++ and re-open it with your script and check that the "External Dependencies" list has not been hijacked.

At the time we hit this (06/2008) there was only one post I could find that seemed similar to our problem,

http://social.msdn.microsoft.com/forums/en-US/vcgeneral/thread/c447c0bd-bcdb-4e52-a8af-b1341ce3ad9f

I notice that it's also referenced in the following Microsoft Connect query, but closed as 'unreproducible',

http://connect.microsoft.com/VisualStudio/feedback/ViewFeedback.aspx?FeedbackID=274454

I'll see if I can open a new issue and maybe more information about this will come to light.

Building Visual C++ Projects From the Command Line

As a developer biased towards Windows applications I find that both as a professional and a hobbyist that Visual C++ is my day-to-day IDE. I still use VC++ 7.1 (aka VS2003) for my personal work as I ship source code as well as binaries and it's easier to upgrade projects and solutions to later editions of Visual C++ for compatability testing than to go backwards. With 6 class libraries (+ unit tests) and 12 applications to build, manually launching the solutions and interacting with the GUI is just tedious and obviously error prone. But a common question on the forums is whether it is possible to build projects using Visual C++ from the command line - sometimes I wonder if people take the "Visual" in Visual C++ too literally :-)

The starting point is firing up a command prompt and setting up the environment variables so that you can invoke Visual C++ to build a solution. Some editions of Visual C++ have a "Command Prompt" item listed in the External Tools - VS2008 does. If you look at the command for this you'll see that it runs a batch file called vcvars32.bat. But opening the IDE just to fire up a command prompt is madness. Fortunately MS added an environment variable from VC++ 7.0 to make life a little easier - VSxyCOMNTOOLS. This means you can open an arbitrary command prompt and run, say,

C:\> "%VS71COMNTOOLS%..\..\vc7\bin\vcvars32.bat"

Unfortunately the VC folder changed name with VC++ 8.0 to drop the version suffix, so it's,

C:\> "%VS80COMNTOOLS%..\..\vc\bin\vcvars32.bat"

And should you want to cross-compile 64-bit code on 32-bit windows, with VS2008 it's as easy as,

C:\> "%VS90COMNTOOLS%..\..\vc\bin\x86_amd64\vcvarsx86_amd64.bat"

That's still just a little too awkward for me to remember, so I have a .cmd file called SetVars.cmd that I use to setup the variables for whatever version of VC++ I'm testing with,

C:\> SetVars vc71

That covers setting up the enviroment variables. The next job is to invoke Visual C++ to build a solution. This can be done in one way under VC6.0-7.1 and two ways under VC8.0-9.0. The old way was to invoke DEVENV.com like this,

C:\> devenv /nologo /useenv "D:\Dev\my.sln" /build Debug

As of VS2005, you can now use the MSBUILD style tool called VCBUILD. Unfortunately this has a different command line format and cannot be used to build installer (.vdproj) projects. The equivalent command would be this,

C:\> vcbuild /nologo /useenv "D:\Dev\my.sln" "DebugWIN32"

Once again though I find this all a bit tedious and have wrapped this logic into another .cmd script called Build.cmd that just takes a solution filename. The choice of which tool to use, devenv or vcbuild, is based on the compiler that was configured by running the SetVars.cmd script.

C:\> Build "D:\Dev\my.sln"

The natural extension to this is building all my libraries, unit tests and applications from one command so I can see what I have broken. A straight FOR loop wrapped into another .cmd script called BuildAll.cmd helps reduce the clutter from this raw example,

C:\> FOR /R %I IN (*.sln) DO devenv /nologo /useenv %I /build Debug

The final script I use is for upgrading the projects and solutions from VS2003 to VS2005, VS2008 etc before building. By default I keep VS2003 version files in my source repository and then run the upgrade script followed by the build script to check compilation on later versions of Visual C++,

C:\> FOR /R %I IN (*.vcproj) DO vcbuild /upgrade %I
C:\> FOR /R %I IN (*.sln) devenv /upgrade %I

The one fly-in-the-ointment is that the Express editions of Visual C++ don't allow you to upgrade solutions - only projects. At least not from the command line. However you can do it through the GUI, which means invoking the IDE for each solution and manually clicking the wizard buttons!

C:\> FOR /R %I IN (*.sln) devenv /upgrade %I

The scripts are available on my website here.

Class Generator v2.0 Released

The other tool which has been in Beta for an eternity but now finally released is v2.0 of my Class Generator - a simple tool for generating the skeletons of classes, interfaces etc. This was always just an internal tool, but I have now added a manual so that the substitution parameters are now officially documented along with the configuration file format. Yes, I know that using C++ for something like this is overkill and you could probably knock up some VBScript to do the job in half the time, but that misses the point that my tools are not about the destination but the journey.

The Release and Debug binaries are on my website here. The source code will be added just as soon as I've labelled and packaged it.

Visual C++ Project Compare v1.0 Released

I've finally got around to releasing v1.0 of my tool for comparing Visual C++ project files to highlight inconsistencies in the settings between builds or other related projects. It's been in Beta since Xmas last year and the only change was to add the missing <Configuration> attributes to the comparison as it didn't show up mismatched "CharacterSet" settings for example.

The Release and Debug versions are already available and the source code will be released in a few days. The packages along with a screenshot and the manual are on my web site here in the Win32 section.

BCS Talk - An Introduction to F#

Last night I attended a BCS talk for the Advanced Programming Specialist Group by Don Syme of Microsoft Research about F#, a new language addition to the Visual Studio family as of VS2010, but also available as a separate download for VS2008. F# is a Functional Programming language at heart, but also has the ability to support both OO and traditional imperative styles as well, which is hugely important for interop with the other .Net languages. Don is the designer of F#, and although I went to Oliver Sturm's ACCU 2009 conference session Functional Programming in F#, I felt getting the lowdown on F# straight from the horses mouth was too good to miss. Oh yeah and a trip to the pub after to chat with fellow BCS members is always a worthy pursuit.

Don's talk lasted just under 90 mins and started by covering the background, such as the design goals; Simplicity, Economy & Programmer Productivity. The former allowed him to compare some verbose C# code and the equivalently succinct F# version. With it being a .Net language the JIT compiler should allow for efficient code generation, and the static typing also helps the IDE to use IntelliSense. Don pretty much skipped over any discussion of syntax and headed straight for the meat to show how you can write using different paradigms, giving some nice demonstrations such as graph plotting and web crawling. These also gave him a good opportunity to show how easily it interops with the rest of the .Net class libraries.

The reason I was aware of F# was because I had read an article about it in MSDN Magazine , and the pipelining and parallel processing aspects really caught my eye. Don had just enough time to take the original single-threaded web crawler example and show how it could be changed to use an asynchronous style, but without making the code unreadable, in fact there were hardly any changes at all.

I've never studied or done any Functional Programming, but this was a key theme at last years ACCU Conference [2008], and the change in hardware due to multiple-cores and NUMA architectures has prompted quite a bit of discussion around the mutability of data and it's effects on scalability. Also in this months C Vu magazine [ACCU], Andrei Alexandrescu in his article "The Case for D", touches quite a bit on this subject and explains the FP aspects of D which look very interesting. It looks like I can't ignore this area of Computer Science any longer.

A Potted History of My Windows Framework

I thought that before leaping into a few posts about my experiences porting my Windows Framework to 64-bit Windows and GCC that I would spend a few minutes explaining how it has come to be the way it is. If you've looked at the source code to my various libraries you'll see a mixture of Hungarian & Java'ish notations, plus some MFC'isms and two different comment styles. Some parts of the framework are very old and as I only spend part of my free time on it, it is in a way, treated very similarly to commercial code - new features often come first and consequently it has accrued a significant level of Technical Debt.

The oldest parts of the code actually date back 15 years or so to my first framework which was written in C as I started my professional career writing Windows 3.x applications in C. This was the Petzold era and the code was pretty much adapted straight from his famous book - Programming Windows.

However, this version didn't last very long as I soon got back into C++ again at work and discovered the late Paul Dilascia's book Windows++. Instead of starting from scratch though, I refactored most of the C library, which was written in an OO'ish style, into classes. The code also followed the house style of their C++ framework as I was used to it.

It was still just a framework without a purpose though until I went freelancing and got my first contract. I now had a need for an application to track my hours for billing and so Task Tracker was born. The contract involved maintenance on an MFC based application and during this time I realised that it would be beneficial if I could reuse code from my framework and potential applications in my professional work. I didn't want to be tied to the real MFC framework tough - after all the point of re-inventing the wheel was to learn, so I decided to refactor my core framework to be more like MFC in style so that moving code between work and home would be easier.

It turned out to be a good move as I continued to work on MFC based code for the next 10 years and I got to reuse some of my classes like CIniFile & CPath quite a lot. During that time I read the classic Design Patterns book and tried to apply some of it to my own MVC like framework as that was the one area of code that I wasn't going to reuse outside my own apps. I also went through an "everything should be a reference" phase as I couldn't decide on how best to deal with object lifetimes.

The main thing missing from my codebase though was use of the STL which was largely down to the fact that MFC doesn't use it and so it doesn't tend to be the default choice. Also I had written my own containers because Visual C++ 1.5 had no template support (and frankly I wouldn't have known what to do with it then anyway :-) and that had created a large body of legacy code to refactor.

My last contract, which although heavily COM/ATL based, was an entirely different beast however. They used STL extensively and was beginning to use Boost for more than just shared_ptr. They also used STLport instead of the Dinkumware offering and there was little to no use of Hungarian Notation. And there was not a line of MFC code in sight! However the biggest factor was the large number of ACCU members that worked there...

After joining the ACCU, I almost went back to basics and began to discuss quite fundamental issues, like style, testing, C++ conformance, tools etc and I began to realise that my personal codebase was somewhat of an embarrassment. So I decided it was time to try and modernise it by deprecating my own containers, switching to the more lightweight Doxygen style comments, using size_t consistently, etc. The most important change though was dropping my own variation of Hungarian Notation and using the more common camelCase convention.

Structurally I have started to split off parts of the Windows C++ Library into a new base class library called Core which I hope to make reasonably platform independent. The single biggest effect of this is that std::string has now become the default string class instead of my own MFC-like CString. My new XML library is the first to take advantage of this and I hope to ensure both libraries compile with GCC as well as Visual C++.

The changes are not just in the code, but also in the way I develop it. I have written my own simple Unit Testing framework which loosely follows the xUnit style and have adopted TDD when possible whilst making changes to all the libraries. Core has the most test coverage as it is the newest and was created using TDD from the start. The other libraries are gaining coverage as and when I make changes.

So, is it worth it? I've always viewed my framework as a playground - somewhere outside of work to play around and make mistakes - but more importantly to try out new things that I hope can then be fed back into my professional career to make me more valuable to a potential employer. I'm still learning heaps every time I refactor, so I guess it is :-)

Who is the ACCU?

Some diligent readers may have noticed that I described Matthew Wilson as "a fellow ACCU member" and are wandering what the ACCU is. Well it's not some kind of secret religious cult - on the contrary they also allow mere mortals like me and you to join! Rather than screw up the possibility of recruiting new members by trying to succinctly describe what the ACCU is about I'll leave it to the front page of the ACCU website (http://accu.org/):-

"ACCU is an organisation of programmers who care about professionalism in programming and are dedicated to raising the standard of programming."

Once upon a time the "CC" in ACCU stood for C/C++, but these days it is far more diverse. In one of the Lightning Talks at this years ACCU conference (http://accu.org/index.php/conferences/accu_conference_2009) there was a straw poll on the primary language used by the attendees and there was a considerable show of hands for Java and C#. Python also features quite regularly on "ACCU-General" which is the organisations main mailing list.

I'm a relative newcomer to ACCU (I only joined in 2006) and came to find out about it by working at one of the major investment banks that has quite a few members in its ranks. I was lamenting the loss of CUJ (The C/C++ Users Journal) and asked on the internal chat system where I was going to get my [printed] fix of C++. I was told that the ACCU publishes two journals, C Vu and Overload which are packed with top quality articles about all aspects of software engineering amongst other benefits. I thought I'd try it for a year and see how it pans out. I only wish I'd known about it when I started my career back in the early 90's!

At £35 per year, membership is cheaper than a subscription to CUJ, MSDN Magazine, TechNet etc. and in return you get a journal per month of quality articles, but more importantly, you get access via the mailing lists to some really smart cookies.

The highlight of the year is probably the ACCU conference, which is held in Oxford during April. This is 4 days of high-bandwidth communication with such luminaries as Andrei Alexandrescu, Nicolai Josuttis, Robert Martin, Kevlin Henney etc. During the day you listen to various speakers talking about all sorts of aspects of software engineering, but more importantly, during the evening (and most of the night :-) you get to speak to many of them one-to-one over a pint or two in the bar. Of course you also get to meet a whole bunch of other like minded souls and share experiences with them as well.

If you're still unsure that your £35 buys you excellent value for money, you can view back issues of Overload online here - http://accu.org/index.php/overloadonline.

See you soon on "accu-general"...