Mark Nelson » C/C++

C++11: Range-based for and auto

Mark Nelson — Sat, 07 Apr 2012 18:18:02 +0000

Two really handy features in C++11 are the range-based for statement and the auto type specifier. The former allows you iterate over collections using a much more compact form of expression, and the latter takes some of the headache out of the complex type declarations encountered in the standard library. Both of these features have been available in g++ since release 4.6, and are now present in Visual Studio 11, so you can start using them today. (auto typed variables are available in earlier versions of both compilers.) In this post I’ll give you a description of how these new features works, and show you a concrete example of the positive effects they can have on your programs.

The value of containers

It’s hard to overstate the value of the containers in the C++ standard library. With the addition of the hash-based containers in TR1, I rarely if ever find myself tempted to roll my own, or use a third party library. The flexibility and power of the library created by Alexander Stepanov does everything I need.

Despite the technical merit of the container classes, newcomers are often hesitant about completely embracing them. One of the main reasons has to be the conceptual drag imposed by the use of iterators as the primary means of accessing the objects they contain. It’s not that there is anything complicated about the concept, but the syntax can be more than just a little annoying. Let me illustrate it with an example.

Anagramania

The listing below is a C++ program that reads through the Scrabble dictionary and determines which set of letters generates the most anagrams. I’m using C++ circa TR1, in which I have access to the unordered associative containers, but I don’t take any shortcuts to try to simplify the syntax. (The fact that I can write this program in one screen of simple code is a nice testament to the quality of the container library.)

The logic for the program is simple. I use an unordered_multimap called counts to hold the count of all anagram families in the dictionary, with its key being the sorted value of the scrabble word. This means that all words that are anagrams of one another will have the same key. I use an unordered_multimap called words to hold the list of all words that are anagrams of that key. Each time I process a word, I increment a value in counts and I add a new value to words.

After the input processing is done, I can just iterate over counts from top to bottom, looking for the highest count. When I have gone through the entire map, I have the sorted key that generates the most anagrams. Using that key, I query an unordered_multimap for a range of results. It returns two iterators in a pair object, which I then use to iterate over the result set.

Even if you are familiar with the type system used by the containers and don’t make too many mistakes, just the magnitude of how much you have to type to get this to work is a bit of a downer. And the length of those type definitions doesn’t help make the concepts being used any clearer.

#include 
#include 
#include 
#include 
#include 
#include 

int main(int argc, char* argv[])
{
    std::ifstream data( "sowpods.txt" );
    std::unordered_map counts;
    std::unordered_multimap words;

    std::string s;
    while ( data >> s ) {
        std::string temp = s;
        std::sort(temp.begin(), temp.end() );
        counts[temp]++;
        words.insert( std::make_pair(temp,s) );
    }

    int max_count = -1;
    std::string max_string = "";
    for ( std::unordered_map::iterator ii = counts.begin();
          ii != counts.end();
          ii++ )
    {
        if ( ii->second > max_count ) {
            max_count = ii->second;
            max_string = ii->first;
        }
    }
    std::cout << "The maximum anagram family has " << max_count << " members:\n";
    std::pair< std::unordered_multimap::iterator,
	       std::unordered_multimap::iterator> range;
    range = words.equal_range( max_string );
    for ( std::unordered_multimap::iterator ii = range.first;
          ii != range.second;
          ii++ )
        std::cout << ii->second << " ";
    std::cout << std::endl;
    return 0;
}

Anagram finder circa TR1
Now let’s look at the two features that make major improvements to this program in C++11.

The auto Type Specification

The hard working committee members who hammered out the standard last year clearly listened to the millions of C++ programmers out there. While they were charting new waters for the language with things like move semantics and rvalue references, they were also making a lot of small changes that simply make the language a lot easier to work with. Maybe even a little more fun. The two things I find at the top of my list are the the use of auto type specifier and the for-range statement.

The auto keyword can be used in a number of different contexts, but in general it means that you can declare variables without having to enter a complete type. This solves some tricky problems for template programming, and it provides a convenience for awkward variable declarations. Most notably, it allows you to replace these two wordy lines of code:

    std::pair< std::unordered_multimap::iterator,
	       std::unordered_multimap::iterator> range;
    range = words.equal_range( max_string );

with this much simpler single line:

    auto range = words.equal_range( max_string );

In both cases, the type of range is the same – but by using the auto type specifier, we let the compiler replace all that typing with a bit of simple hand waving.

Bjarne Stroustrup has a good, concise explanation of auto on his C++11 FAQ, I recommend you spend the time to read it.

The Range-based for Statement

When working with standard library containers, one of the most common things we do is iterate over some or all of the container. This generally is done using a for or while loop with an iterator loop variable.

C++11 makes this type of iteration easier with new syntax injected into the for statement that has been around since 1969. The range-based for looks like this:

    for ( declaration : expression ) statement

In this new statement, expression can be an initializer list, an array, or an object that implements container semantics. This means that the object returns an iterator-like object from a begin() and end() methods, or via a call to begin() and end() functions in the current or std namespace.

The variable declaration is either a reference or value of the type of variable held in the container, array, or initializer list. The for loop is executed from the beginning of the container to the end, with statement executed once per value returned by the iterator.

Although this is a completely new language feature, I think most C++ programmers will be comfortable with it from the first time they are able – it makes those iterations over containers clean and concise.

Putting it to Use

Although it didn’t really cut down on my code size in a big way, I first made use of the range-based for in the loop that reads in the data from the scrabble dictionary. My new version of the loop is shown here:

for ( const std::string &s : std::istream_iterator( data ) )
{
    std::string temp = s;
    std::sort(temp.begin(), temp.end() );
    counts[temp]++;
    words.insert( std::make_pair(temp,s) );
}

The only big improvement here is that I was able to declare my string variable on first use, which is always my preference.

However, looking at this code, you might be wondering how it compiles. After all, the istream_iterator doesn’t have begin() or end() member functions.

That’s correct, and the reason it works is that I added a couple of convenience functions to my program that enable the use of this iterator type with the range-based for:

template
std::istream_iterator begin(std::istream_iterator &ii_stream)
{
    return ii_stream;
}

template
std::istream_iterator end(std::istream_iterator &ii_stream)
{
    return std::istream_iterator();
}

I made use of a similar set of template functions to enable the use of the new for statement in my final output statement. I now iterate over the discovered members of the anagram family with two easy-to-read lines:

for ( const auto &map_entry : words.equal_range( ii->first ) )
    std::cout << map_entry.second << " ";

Compare this to the TR1 code that does the same thing, and I think you will see the real value of both auto and range-based for.

Iterating over the values returned from a multimap is a common task, enabled it by these convenient template functions:

template
ITERATOR begin( std::pair &range )
{
    return range.first;
}

template
ITERATOR end( std::pair &range )
{
    return range.second;
}

When I first implemented the functions for my C++11 program, I was halfway expecting to find that this functionality had already been added to the standard library – they really make a big improvement for a small investment. But no, I couldn’t find them, so we will be using our own versions for the time being.

The Final Product

My much improved anagram finder is shown below. In addition to the use of range-based for and auto type declarations, I changed the way I find the maximum element in the container. Now that lambdas are part of the language, there is no excuse for not using the standard library algorithms, and this code gives an illustration of how that works as well.

#include 
#include 
#include 
#include 
#include 
#include 

template
ITERATOR begin( std::pair &range )
{
    return range.first;
}

template
ITERATOR end( std::pair &range )
{
    return range.second;
}

template
std::istream_iterator begin(std::istream_iterator &ii_stream)
{
    return ii_stream;
}

template
std::istream_iterator end(std::istream_iterator &ii_stream)
{
    return std::istream_iterator();
}

int main(int argc, char* argv[])
{
    std::ifstream data( "sowpods.txt" );
    std::unordered_map counts;
    std::unordered_multimap words;

    for ( const std::string &s : std::istream_iterator( data ) )
    {
        std::string temp = s;
        std::sort(temp.begin(), temp.end() );
        counts[temp]++;
        words.insert( std::make_pair(temp,s) );
    }
    auto ii = std::max_element( counts.begin(),
                                counts.end(),
                                [](const std::pair &v1,
                                   const std::pair &v2)
                                {
                                    return v1.second < v2.second;
                                }
                              );
    std::cout << "The maximum anagram family has " << ii->second << " members:\n";
    for ( const auto &map_entry : words.equal_range( ii->first ) )
        std::cout << map_entry.second << " ";
    std::cout << std::endl;
    return 0;
}

Anagram finder in C++11
If I move the four convenience functions into a utility header file, I think you’ll agree that the new version of the code implements my algorithm in a very clean and concise way. The new language improvements make a huge difference in readability and convenience.

Of course, these two features are just one small part of a huge new standard, but for right now, they are the ones I turn to the most. How about you? Let me know!

Visual Studio 11 and Modern C++

Mark Nelson — Tue, 13 Mar 2012 13:13:08 +0000

Despite some harsh words about Visual Studio 11, I’m finding that it makes my heart go pitter-pat every time I use it. Why? Because this early release is finally incorporating a decent set of long-awaited C++11 features. In this article I’ll show you how a little thing like a lambda can make a big difference in your coding style.

Microsoft and C++ – We Have History

Microsoft has a cyclic relationship with C++. In the early MFC days, the love was there big time – you had access to most of the system API using C++. However, around the turn of the millennium, Microsoft came under the Rasputin-like influence of Anders Hejlsberg and his beloved offspring, C#. Now it appears that maybe the pendulum is swinging back a bit, and C++ is no longer viewed as an afterthought. Great news.

Although Visual Studio 11 is a developer’s preview, Microsoft is saying that it is production ready – you can use this to create programs that are ready for release. In addition to touting a complete implementation of the C++11 standard library, an impressive list of language features have been turned on as well. (N.B. the path ahead is still long and arduous.)

Modern C++

Before even using Visual Studio C++ 11 to test a single line of code, I really appreciated reading Welcome Back to C++ (Modern C++), a manifesto that includes the following bullet points:

Modern C++ emphasizes:

Stack-based scope instead of heap or static global scope.

Auto type inference instead of explicit type names.

Smart pointers instead of raw pointers.

std::string and std::wstring types instead of raw char[] arrays.

Standard template library (STL) containers—for example, vector, list, and map—instead of raw arrays or custom containers.

STL algorithms instead of manually coded ones.

Exceptions, to report and handle error conditions.

Inline lambda functions instead of small functions implemented separately.

I feel that all of these changes result in safer code that is easier to read and maintain, without giving up the type-safety and efficiency that we love so much. Fully implementing these features either leans heavily on C++11 or requires it outright.

A Simple Example Using Naive C++98

It’s interesting to watch the evolution of code from C++ 98 to C++11 and see how it affects your code. You’ll see that the transformation can make it look like you are literally using a new programming language.

In this simple program, I’m taking a Scrabble rack of tiles and whipping through the Scrabble dictionary to find matches. Since it is a one-time call, I’m not storing the words, just doing a quick online comparison. In C++ 98, my code might have looked like this:

void find_matches( std::string rack, const std::string &filename )
{
    std::sort( rack.begin(), rack.end() );
    std::ifstream sowpods( filename.c_str() );
    std::string word;
    while ( sowpods >> word ) {
        std::string sorted = word;
        std::sort( sorted.begin(), sorted.end() );
        if ( sorted == rack )
            std::cout << word << " ";
    }
}

int main(int argc, char* argv[])
{
    find_matches( "etaionsr", "sowpods.txt" );
    return 0;
}

This works properly and I get what looks like correct output:

anoestri arsonite notaries notarise rosinate senorita

Classes Good, Templates Better

As people started to get more comfortable with templates and iterators, algorithms like this were commonly rewritten to take an range of iterators as input – much as the standard library algorithm functions do. This meant changing the function to a template function, but it did make it a lot more flexible. I could now call the function to operate on data from a file, just as before, but I can also now use any other container, or even an array as input:

template
void find_matches( std::string rack, ITERATOR ii, ITERATOR jj )
{
    std::sort( rack.begin(), rack.end() );
    for ( ; ii != jj ; ii++ ) {
        std::string sorted = *ii;
        std::sort( sorted.begin(), sorted.end() );
        if ( sorted == rack )
            std::cout << *ii << " ";
    }
}

int main(int argc, char* argv[])
{
    std::ifstream sowpods( "sowpods.txt" );
    find_matches( "etaionsr",
                  std::istream_iterator( sowpods ),
                  std::istream_iterator() );
    return 0;
}

More or less the same number of lines of code, but it is now generic.

Of course, just like with OOP, you need to take some care with template programming. Generic programming can’t be beat when it makes sense, but programmers have a particularly strong susceptibility to pro-innovation bias.

Using the Algorithms Library

Again, prodded by changing styles in the C++ world, my next step is to use a standard library algorithm to do the work. We’re told over and over that turning to the algorithms library allows you to use code that has been optimized to the n-th degree by the clever library teams.

In order to make this work, I have to call an algorithm with a predicate functor, seen below as class sorted_not_equal. Note also that I can’t use the logical function for this, which would be copy_if(). Why not? The committee forgot to put it in back in 1998, 2003, and 2005, a mistake that was fortunately remedied in C++11. So I have to use the inverse function, remove_copy_if(), and invert the logical sense of my functor:

class sorted_not_equal {
    std::string str;
public :
    sorted_not_equal( const std::string & test )
    {
        str = test;
        sort( str.begin(), str.end() );
    }
    bool operator()( std::string test )
    {
        sort( test.begin(), test.end() );
        return ( str != test );
    }
};

template
void find_matches( std::string rack,
                   INPUT_ITERATOR ii,
                   INPUT_ITERATOR jj,
                   OUTPUT_ITERATOR kk )
{
    std::sort( rack.begin(), rack.end() );
    std::remove_copy_if( ii, jj, kk, sorted_not_equal( rack ) );
}

int main(int argc, char* argv[])
{
    std::ifstream sowpods( "sowpods.txt" );
    find_matches( "etaionsr",
                  std::istream_iterator( sowpods ),
                  std::istream_iterator(),
                  std::ostream_iterator(std::cout,  "\n" ) );
    return 0;
}

Functors Not So Hot

So this new approach is supposed to soup up my code by taking advantage of the algorithms that come with the standard library. But if you look around at the code people have been writing for the past 10 years, you’ll find that this style is pretty common in textbooks and magazine articles, but no so much in the real world.

Why not? Well, it’s pretty obvious. The generic algorithms in the library need lots of predicate glue to make them useful, and the work to create these predicates is just a pain. My code is almost twice as long, and the functionality that took two lines of code earlier is now bloated into a complete class definition. It pollutes my namespace, takes up a lot of space, and has to be defined somewhere distant from where it is actually used. Not a win.

This is obviously a problem when you look at the history of Linux, C, and C++. An entire family of technologies and infrastructure was developed with the implicit goal of reducing the number of keystrokes programmers had to enter. (I’m kidding, but only somewhat.) Functors are a step in the wrong direction.

Lambda to the Rescue

So it is with much relief that C++11 delivers lambdas, which allow us to write short sweet predicates exactly where we need them, as shown in this C++11 version of the example:

template
void find_matches( std::string rack,
                   INPUT_ITERATOR ii,
                   INPUT_ITERATOR jj,
                   OUTPUT_ITERATOR kk )
{
    std::sort( rack.begin(), rack.end() );
    std::copy_if( ii, jj, kk,
                  [&rack](std::string str) ->bool
                  {
                      std::sort( str.begin(), str.end() );
                      return rack == str;
                  }
                );
}

int main(int argc, char* argv[])
{
    std::ifstream sowpods( "sowpods.txt" );
    find_matches( "etaionsr",
                  std::istream_iterator( sowpods ),
                  std::istream_iterator(),
                  std::ostream_iterator(std::cout,  "\n" ) );
	return 0;
}

Yes, I now have to get used to a new syntax for writing lambda functions – I think that was unavoidable. But my lambda function is short, it is quite easy to see exactly what it is doing, and it replaces a gangly and awkward functor class.

Best of all, I use the lambda exactly where I need it – as the predicate parameter to an algorithm used in the standard library. Locality rules.

Using Lambdas

Visual C++ 11 provides a great framework for experimenting with lambdas, as they are supporting the 1.1 definition that was ratified as part of the standard. If you want the gory details, I believe the working group’s proposal has essentially the same wording that went into the standard. For a detailed tutorial, Herb Sutter’s talk can’t be beat.

C++ – Where’s the Hate?

Mark Nelson — Tue, 28 Feb 2012 05:23:09 +0000

One thing I’ve become accustomed to over the years is that there are a lot of C++ haters. They have their reasons – some good, some bad – but they are never afraid of sharing their opinions. An article on Slashdot this week touting the release of the C++11 standard should have been a hotbed of language trash talk – instead, it was kind of a low key discussion of both the new language features and some retrospection about the language itself. Where have the haters gone?

The Epitome of C++ Hate

There really is no better example of C++ hate than the screed arising from the poison keyboard of Linus Torvalds back in 2007. One paragraph sums it up well:

C++ is a horrible language. It’s made more horrible by the fact that a lot of substandard programmers use it, to the point where it’s much much easier to generate total and utter crap with it. Quite frankly, even if the choice of C were to do *nothing* but keep the C++ programmers out, that in itself would be a huge reason to use C.

Linus asserts that C++ is actually sort of a honey-trap, pulling in the substandard programmers and keeping them out of decent C development.

Reading through the most coherent parts of his rant, it is my opinion that much of his ire is directed against a couple of real problems:

The early adopter’s fondness for Object Oriented Programming (OOP), which leads to objects everywhere – and particularly in places where they aren’t needed.
The more sophisticated user’s fondness for advanced library and language features found in places like boost.
My personal observations are obviously very subjective, but I feel like the excesses of OOP peaked back in the early part of the last decade. It’s been a while since I saw a program were a user did integer math using a type called Integer derived from something called Object.

These days I see OOP being heavily used for large libraries, and used internally in projects where it makes sense. Often “making sense” is as simple as using objects for RAII, which is really not OOP, just good resource management. Things like RAII are really just better C – providing some guarantees on resource usage that can’t be made when programming in straight C.

And best of all, the boost language features that Linus didn’t like in 2007 made their way into TR1, and finally into C++11. No longer will you need to worry about rebuilding boost for your compiler every time there is a new release of either product. From this point on, it should just work.

The Slashdot Crowd

So instead of seeing a bunch of comments in the vein of Linus’s, the recent Slashdot article contained virtually no blanket dismissals of C++ as something a sane programmer would use. The negative comments were much more constrained, maybe aimed at a pet peeve instead of the language as a whole:

printf() isn’t typesafe, but it’s a fuckton more readable than all that cout formatting stuff. Also, the fact that it’s not typesafe isn’t really an issue if you don’t suck – trivial unit testing will pretty much show any problems immediately. Besides, gcc/g++ is nice enough to warn you about egregious ones now.

I don’t think you’ll see a lot of people flaming C++, there just aren’t that many people that care one way or the other anymore.

I think some of the new features look nice but mainstream use has been shifting away from C++ for a while and I’m not sure I see these new features drawing many back…

Are the features useful? Yes, but they’re taking a complex language and slapping on yet more functionality. Some new C++ code syntax doesn’t even *look* like C++ anymore it’s so different. Not everyone is a C++ guru and the language is bad enough supporting so many different paths to the same implementation outcomes. This is just going to make staffing, testing, training, and code review that much worse trying to juggle yet another barrel-full of C++ “improvements”.

Why the Dropoff?

I think there are two reasons for the drop in C++ antagonism.

The first is the increasing diversity of languages in use today. Over the past 10 years, the most popular programming languages like Java, C, C++, and VB have all experienced drops in adoption. In 2002, the top four languages might have claimed almost 70% of the Tiobe Language Index. Today the top four represent less than 50%. Language selection has been democratized as the bottom tier of the top 10 gains representation in large markets where it makes sense.

In other words, it was a lot easier to focus your wrath on C++ when it represented 20% of the projects out there instead of the 8% of today.

The second is inevitable rationalization of the earlier adopters. C++ programmers in 1998 saw objects behind every tree. In 2003 they were building insanely complicated template metaprogramming frameworks.

All this crazy stuff settles down over the course of a few years as the C++ mainstream settles into paradigms for things that actually work sensibly. Templates for container classes? Sure. Templates for mathematical computation? Uh, no thanks.

Final Analysis

There is a lot of nice stuff in C++11, and nothing that is going to take years to figure out. A lot of it will lead to more readable code, and that’s always good.

The pace of change is pretty comfortable for me. These language features will be rolling out over the next few years, and as they drop into our favorite compilers, we will pick them up quickly.

I, for one, welcome our new ISO overlords.

Streambuf Iterators Are a Big Help

Mark Nelson — Mon, 06 Feb 2012 00:34:45 +0000

A few weeks back I was looking at the choice of whether to use iterators or streaming operations for I/O on my data compression code. I was bemoaning the fact that the C++ iterators that perform stream I/O use the insertion and extraction operators, making them unsuitable for binary data compression.

It looks like I spoke too fast.

The Problem

The canonical way to use iterators to go over an iostream is to use istream_iterator or its partner, ostream_iterator. In this short sample, I create a binary file with values 0 to 256, then try to read back the data using this iterator type:

#include 
#include 
#include 
using namespace std;

int main(int argc, char* argv[])
{
    ofstream temp_out("temp.dat", std::ios_base::binary );
    for ( int i = 0 ; i < 256 ; i++ )
        temp_out.put( (char) i );
    temp_out.close();
    ifstream temp_in("temp.dat", std::ios_base::binary );
    char last = -1;
    int count = 0;
    istream_iterator ii( temp_in );
    while ( ii != std::istream_iterator() ) {
        char c = *ii++;
        count++;
        if ( c != char(last+1) )
            cout << "Error on character number: " << (int) last << endl;
        last = c;
    }
    cout << "Count: " << count << endl;
    return 0;
}

Running this program gives the following output:

Error on character number: 8
Error on character number: 31
Count: 250

Due to the fact that the extraction operator uses whitespace as a delimiter, we get errors trying to read tab, line feed, vertical tab, form feed, carriage return, and space - which means we simply don't see six characters in the input file. This is not too bad when parsing ascii text, but when trying to do compression on binary data, it just won't work.

I figured this was the end of it until reader Fred Jardon chimed in with:

Isn’t istreambuf_iterator the iterator you’re looking for ?

http://cplusplus.com/reference/std/iterator/istreambuf_iterator/

It directly reads from the inner streambuf without using the extraction operator.

The opposite iterator exists: ostreambuf_iterator

Well, Fred is quite right, and it only serves to show my lack of depth when it comes to iostreams. The two classes, istreambuf_iterator and ostreambuf_iterator read directly from the underlying buffer, and don't use the extraction operator, which means they don't have the whitespace issues seen above. Changing just two lines fixes the problem:

    istreambuf_iterator jj(temp_in.rdbuf());
    while ( ii != std::istreambuf_iterator() ) {

Running that I get the pristine output I long for:

Count: 256

I'd like to tell you the ramifications of diving down into the object and working on the buffer, but I'm afraid this implementation details still elude me. There's a lot more going on in iostreams than in good old , and half of what I learned about it was wiped out in the standardization process. So be it.

In any case, I think the proper use of istreambuf_iterator tips the scale in favor of iterators for me, which reverses my previous thinking.

And many thanks to Fred for straightening out the mess.

Streams or Iterators?

Mark Nelson — Sat, 24 Dec 2011 18:21:11 +0000

When I updated my LZW reference code to use the latest C++ features, I abstracted my input and output functions using templates. Data was read and written using the iostreams paradigm, which requires simple classes that implement just a few functions. Would I have been better off using the iterator paradigm instead? The C++ algorithms library favors that method of processing data, and it can be both elegant and powerful. Which of the two paradigms is the right one for data compression?

The Conflict

General purpose data compression routines tend to be used on binary streams of data, either from files or in-memory objects. So what is the best general paradigm for input and output when compressing data?

You might analyze this problem by imagining that you need to write a binary copy routine.

template
void bcopy( INPUT_ITERATOR input, INPUT_ITERATOR eof, OUTPUT_ITERATOR output )
{
    while ( input != eof )
        *output++ = *input++;
}

This routine is particularly nice when you are performing a simple copy using pointers to memory – the generated code should be really efficient.

However, the iterator paradigm doesn’t work quite as well when you want to perform a binary copy of data in a file. I can make use of iterators that almost do the job:

 std::ifstream in( "input.txt", std::ios_base::binary );
 std::ofstream out("output.txt", std::ios_base::binary );
 bcopy( std::istream_iterator(in),
        std::istream_iterator(),
	std::ostream_iterator(out) );

But the bad news is that both istream_iterator and ostream_iterator use the insertion and extraction operators, which are really meant for whitespace-delimited textual data, not binary data. The copy routine shown here will not make a binary byte-for-byte copy of the input file.

So when using files, the stream approach seems to be the way to go:

template
void bcopy( INPUT_STREAM in, OUTPUT_STREAM out )
{
    char c;
    while ( in.get(c) )
        out.put(c);
}

If my files have been opened using the iostream classes, you can use this binary copy function without having to write any glue code – they already support the get and put methods, so this works right out of the box.

My Choice

If I’ve made up my mind that my data compression routine is going to use one of these two paradigms, it means I am going to have to write some glue code. If I choose the iterator-based approach, I need the equivalent of istream_iterator and ostream_iterator for binary files – and these aren’t in the standard library. If I choose the stream-based approach, I need efficient put() and get() members for blocks of memory. In some cases basic_stringstream might do the job, but not in all cases.

After dithering around with various solutions, I tentatively opted for the stream paradigm. I found the implementation for various sources of data to be fairly simple, and the interface is easy to understand. I don’t know if it’s the perfect choice, and I’ll keep experimenting, but for now it works for me. My abstraction of the LZW code still needs a lot of work, so it’s always possible I could rethink this at a later date.

I’d like to hear your thoughts – is there an obvious right answer to this question?

Automating Putty

Mark Nelson — Sat, 10 Dec 2011 12:11:15 +0000

Windows users who need a command line connection to another system via telnet or SSH are big fans of PuTTY. It's free, it has every feature you need, and it's reliable.

One thing many people would like to do is use PuTTY as a component in their program. Apparently this comes up so often enough that there is a FAQ entry dedicated to the topic. Alas, PuTTY does not have any sort of automation interface, so this goal has always been out of reach.

In this article I will show you how to work around this minor shortcoming. Creating a version of PuTTY that can be driven from a Windows program turns out to be an easy task. I'll demonstrate this with a small C++ program that shows exactly how to get this versatile program to do your bidding. My solution works for C++, but the changes I make should work well with any Windows software that can properly process a few messages.

Putting Together the Project

I'm using Visual Studio 2010 to build both my program and the modified version of Putty. I created the basic outline as follows:

Use the File|New|Project menu item to bring up the list of available project wizards.
Select MFC project, and enter a project name (I used the uninspired name PuttyDriver.)
I don't want the default MFC settings, so in the MFC App Wizard, select the Next button.
On the Application Type page of the wizard, change the Application Type to Dialog Based.
The project is ready to go at this point, you can click the Finish button and then build your initial project.

My driver program is only going to do one thing: direct putty to connect to the host of my choice, then log in using canned credentials. The resulting UI is shown below, and I am going to leave the very minor details of creating it up to the reader.

The driver program - a simple dialog-based MFC app

Adding Putty to the Project

The next step in this process is to add the Putty components to the project. I downloaded version 0.61 of the PuTTY source from the download page and extracted it to a separate folder. I then used Visual Studio's File|Add|Existing Project to add the compatible project file, Putty.dsp, found in /Windows/MSVC/Putty. Visual Studio has to convert this project to a version 10 project file, but it should do so with no problems.

I then right-clicked on the Putty project in Solution Explorer and renamed it to AutoPutty. Since this version of PuTTY will have some slightly different behavior, I don't want to confuse the executable I am creating with the real thing.

From Project|Project Dependencies, I set the PuttyDriver project to depend on AutoPutty - this insures that both projects get built when I build the entire solution.

My final change to the project is to modify the output directory for both Debug and Release versions of AutoPutty. I set the project to build the executable in the root directory of my PuttyDriver project - this will make it easy to find the executable when I need to launch it. I had to make this change in two places: Properties|Configuration Properties|General|Output Directory and Properties|Linker|Outuput File.

When you finally build the project, you'll find that current version of Microsoft's C++ compiler complain quite a bit about the use of functions like strcpy - Microsoft would like you to use safer replacement functions. You may choose to turn those errors off by defining _CRT_SECURE_NO_WARNINGS in the project file. While you are there, you should define SECURITY_WIN32 as well - it is required by Windows header sspi.h.

After a successful build you should find a copy of AutoPutty.exe in the root directory of your project, and it should run on your system and behave just like PuTTY.

Launching AutoPutty

If I'm going to have a PuTTY component in my PuttyDriver program, one of the first things I need is to be able to start and stop AutoPutty. So my first step in this project is to create the code that launches the program from PuttyDriver. The code below is inserted into the handler for the Start button:

UpdateData( true );
char path[MAX_PATH];
GetCurrentDirectory(MAX_PATH, path);
if ( path[ strlen(path) - 1 ] != '\\' )
    strcat_s( path, MAX_PATH, "\\" );
strcat_s( path, MAX_PATH, "AutoPutty.exe -ssh " );
strcat_s( path, MAX_PATH, m_HostName.GetBuffer() );
PROCESS_INFORMATION pi;
ZeroMemory(&pi, sizeof(pi) );
STARTUPINFO si;
ZeroMemory(&si, sizeof(si) );
si.cb = sizeof(si);
if ( CreateProcess( NULL, path, NULL, NULL, NULL, NULL, NULL, NULL, &si, &pi ) )
{
    Sleep( 1000 );
    BringWindowToTop();
}

This code assumes that AutoPutty.exe is in the current directory, and launches it with a command line telling it to connect to the host named in the dialog using ssh. Assuming that you have the project set up properly, pushing the start button should now start an independent copy of AutoPutty, which will behave identically to classic PuTTY.

Taking Ownership of AutoPutty

At this point I can successfully launch AutoPutty, but I can't really start calling this an integrated part of my main program, PuttyDriver. All I have done is set up a launcher for a separate executable.

The next step in the integration process is to establish PuttyDriver as the owner of AutoPutty's main window. Most Windows programmers are familiar with the traditional parent/child relationship between windows. That relationship is well understood, but I can't use it here - it doesn't work for two top level windows.

Setting PuttyDriver to be the owner (as opposed to the parent) of AutoPutty has the following effects, as explained here by Microsoft:

The owned window will always be above its owner in the z-order.
The system automatically destroys the owned window when the owner is destroyed.
The owned window is hidden when the owner is minimized.

The most straightforward way to set ownership of the window is to pass the owner's handle in the call to CreateWindow(), which means I will now make my first modifications to the PuTTY source code.

There are a number of ways to pass the owner handle to AutoPutty for use in the call to CreateWindow(), with the most obvious being to pass it on the command line. In the interest of minimizing changes to the existing PuTTY code base, I elected to pass it by creating an environment variable that holds the owner window handle. Since a child process inherits the parent's environment, this is a no-fuss way to get the data to AutoPutty.

I added the following code to the end of InitDialog() in PuttyDriver:

CString hwnd_text;
hwnd_text.Format( "%d", m_hWnd );
SetEnvironmentVariable("PUTTY_OWNER", hwnd_text );

This sets the environment variable for AutoPutty to find when it gets launched.

Now I come to the point where I am actually making changes to the PuTTY code. Fortunately, all of the changes needed for this program are confined to two files: terminal.c and windows/window.c. My first change is to window.c. This file contains the WndProc for the PuTTY window, and thus most of the rendering and control code for the GUI.

In order to establish the Owner/Owned relationship, I need to modify the code that calls CreateWindow(). I hoisted the function call into a block, added code to get the owner window handle, and inserted the handle into the call to CreateWindow():

{
    HWND owner_hwnd = 0;
    char buffer[ 132 ];
    if ( GetEnvironmentVariable( "PUTTY_OWNER", buffer, 132 ) )
        sscanf( buffer, "%d", &owner_hwnd );
    if ( owner_hwnd == 0 )
        MessageBox( NULL,
                    "AutoPutty did not find the handle for the "
                    "owner window, this is not going to work",
                    "Fail",
                    MB_OK );
    hwnd = CreateWindowEx(exwinmode, appname, appname,
                          winmode, CW_USEDEFAULT, CW_USEDEFAULT,
                          guess_width, guess_height,
                          owner_hwnd, NULL, inst, NULL);
}

At this point I've only modified one small block of code in the PuTTY source, but I'm well on my way to having it behave more like a component of PuttyDriver and less like an independent program. The ownership status means that the two programs only appear once on the taskbar, and will only appear once when you are pressing ALT-TAB to select a new active process. And they only produce a single entry in the Applications Tab of Task Manager.

The Communications Link

In order to achieve the automation that I am seeking, I also need to have two way communications between AutoPutty and the driver program. Since this is Windows, a natural choice for communications is to use native Windows messages. In order to do this, both programs need the Window handle of their opposite number.

I've already solved half of that problem through the ownership relationship established when I created the main window for AutoPutty. Now that it has set PuttyDriver as its owner window, I can get this window handle any place in the program through a simple function call:

HWND parent = GetWindow(hwnd, GW_OWNER);

But the reverse is not true - PuttyDriver does not know have a copy of the window handle for AutoPutty.

To remedy this situation, I added code to window.c that notifies its owner when it s created, and when it is destroyed. First I add this statement immediately after the call to CreateWindow():

if ( owner_hwnd )
   PostMessage( owner_hwnd, WM_APP, 0, (LPARAM) hwnd );

This tells PuttyDriver that the window is created, and gives it the handle to use for communications.

I also need to know when the window is closed, and I have to add that code two places in window.c - because Putty can be shut down two different ways.

Normally AutoPutty will shut down in response to a windows message. When this happens, I can count on a WM_CLOSE message being sent to the Windows Procedure. I add this code the existing handler for WM_CLOSE:

if (!cfg.warn_on_close || session_closed ||
    MessageBox(hwnd,
               "Are you sure you want to close this session?",
               str, MB_ICONWARNING | MB_OKCANCEL | MB_DEFBUTTON1)
    == IDOK) {
    HWND parent = GetWindow(hwnd, GW_OWNER);
    if ( parent )
        SendMessage( parent, WM_APP, 0, 0 );
    DestroyWindow(hwnd);
}

This lets PuttyDriver know that the window has been destroyed.

The original PuTTY code has an alternative method of shutdown. When it receives one of several possible network events, such as a telnet connection being broken, it calls PostQuitMessage(). When a program shuts down this way, it doesn't issue messages to destroys its windows - it relies on the O/S to destroy the windows when the process exists. As a result, I have to make a change in WinMain(), the main window procedure for PuTTY. This procedure extracts the messages sent to it using PeekMessage, and I add some code to handle the processing when a WM_QUIT message is sent:

if (msg.message == WM_QUIT) {
    HWND parent = GetWindow(hwnd, GW_OWNER);
    if ( parent )
        SendMessage( parent, WM_APP, 0, 0 );
    goto finished;	       /* two-level break */
}

Handling the AutoPutty Lifecycle Events

To keep track of the state of AutoPutty, I have to add a handler for WM_APP to PuttyDriver. It does two things when handling the incoming WM_APP event.

First, then handler stores the handle of the AutoPutty window - or sets the value to 0 when the window has been destroyed.

Second, it either enables or disables the button used to start up AutoPutty. Since this program can only manage one window at a time, I don't want to allow any inadvertent button pushes:

afx_msg LRESULT CPuttyDriverDlg::OnWmApp(WPARAM wParam, LPARAM lParam)
{
    m_PuttyWindow = (HWND) lParam;
    m_StartButton.EnableWindow( !m_PuttyWindow );
    return 0;
}

One final piece of bookkeeping is to make sure that the AutoPutty window is shut down when PuttyDriver shuts down. (The Windows documentation claims this happens automatically to owned windows, but it doesn't seem to be the case.)

void CPuttyDriverDlg::OnDestroy()
{
    CDialogEx::OnDestroy();
    if ( m_PuttyWindow )
        ::SendMessage( m_PuttyWindow, WM_CLOSE, 0, 0 );
}

Monitoring Input Traffic

Now that I have control over the lifetime of my AutoPutty window, it's time to take the next step in automation. My driver program needs to watch all the data coming in from the remote end so that it can take action on various types of input.

Depending on how you set up your connection, PuTTY can receive input data from a serial port, a Telnet connection, or an SSH connection. Fortunately the Windows version of PuTTY uses a standard handle-based interface to all three types of connections. The routine term_data() in terminal.c is called as data arrives, regardless of the source.

Since we are using the Windows API to communicate between processes, it makes sense to use the WM_COPYDATA message to send data to the parent program as it arrives. WM_COPYDATA is a good choice, as it takes care of marshalling the data between the two processes, which can add some complication to other solutions. The modified routine is shown below:

int term_data(Terminal *term, int is_stderr, const char *data, int len)
{
    HWND parent = GetWindow(hwnd, GW_OWNER);
    if ( parent ) {
        COPYDATASTRUCT cd;
        cd.dwData = (ULONG_PTR) 0xDEADBEEF;
        cd.cbData = len;
        cd.lpData = (PVOID) data;
        SendMessage( parent, WM_COPYDATA, (WPARAM) hwnd, (LPARAM) &cd );
    }

Receiving the Data

To receive this messages in PuttyDriver, I simply create a handler for WM_COPYDATA and start grabbing the data as it arrives. One important thing to note is that because AutoPutty has to use SendMessage() to send the data to its parent, it has to wait for PuttyDriver to finish processing the data until it can continue. This dictates a certain style of behavior on my part.

There are quite a few ways to skin this cat, and I'm keeping it very simple here. I'm using a deque container to hold the last 64 characters I've received. After each WM_COPYDATA message I received, I check to see if the current output snapshot ends in one of my trigger messages. If it does, I post the message number to myself for later processing, then return so that AutoPutty can continue its work.

The code I'm using here is doing something fairly simple: automating the login process by using the credentials that I've entered into the dialog box. That means the two strings I'm looking for are the login and password prompts. The resulting code is shown here:

BOOL CPuttyDriverDlg::OnCopyData(CWnd* pWnd, COPYDATASTRUCT* pCopyDataStruct)
{
    char *p = (char *) pCopyDataStruct->lpData;
    int len = pCopyDataStruct->cbData;
    if ( len >= 64 ) {
        p += len - 64;
        len = 64;
        m_Snapshot.clear();
    }
    while ( len-- )
        m_Snapshot.push_front(*p++);
    m_Snapshot.resize(64);
    static const char *needles[2] = { "login as: ", "password: " };
    for ( int i = 0 ; i < 2 ; i++ ) {
        int len = strlen( needles[i] );
        int j;
        for ( j = 0 ; j < len ; j++ ) {
            if ( needles[i][j] != m_Snapshot[len-1-j] )
                break;
        }
        if ( j == len )
            PostMessage( WM_APP+1, i, 0 );
    }
    return TRUE;
}

There is plenty of room for improvement in this routine, much of it depending on what type of automation you are going to be using in your program. Some obvious items would include the ability to add and remove triggers as the program progresses, and regular expression matching for triggers.

Driving PuTTY

This login program is now complete save for one detail: I need a way to send my responses back to AutoPutty.

The first part of this is pretty obvious - I just need to read the data from the dialog box and post it to AutoPutty with my useful WM_COPYDATA command. This happens in my WM_APP+1 handler:

afx_msg LRESULT CPuttyDriverDlg::OnWmAppPlusOne(WPARAM wParam, LPARAM lParam)
{
    UpdateData(TRUE);
    CString msg;
    switch ( wParam ) {
    case 0 :
        msg = this->m_UserId + '\r'; break;
    case 1:
        msg = this->m_Password + '\r'; break;
    }
    if ( this->m_PuttyWindow ) {
        COPYDATASTRUCT cd;
        cd.dwData = (ULONG_PTR) 0xF00DFACE;
        cd.cbData = msg.GetLength();
        cd.lpData = (PVOID) (const char *) msg;
        ::SendMessage( this->m_PuttyWindow,
                       WM_COPYDATA,
                       (WPARAM) this->m_hWnd,
                       (LPARAM) &cd );
    }
    return 0;
}

Sending this data to AutoPutty is fine, but right now the program doesn't do anything with that message. The final piece of work is to add a WM_COPYDATA handler to window.c.

Simply grabbing the data is easy enough - the data structure that accompanies the message contains a pointer to the data and a value indicating its length. However, I have two problems I have to solve before the data is actually sent out to the to whatever device AutoPutty is connected to.

First, I have to take into account the fact that PuTTY was written to use wide characters. My driver program was built using MultiByte characters, so we have a mismatch. This means I have to do a conversion of the data from one domain to the other. This is a two step process - I call MultiByteToWideChar() once to determine how much space I need, then I allocate a buffer and call it again.

The second thing I need to do is determine what to do with the data once I've converted it. PuTTY takes all terminal input and eventually passes through a function called luni_send(). Calling this function directly from the Windows procedure seems to work just fine.

The WM_COPYDATA handler I created looks like this:

case WM_COPYDATA :
{
    COPYDATASTRUCT *cd = (COPYDATASTRUCT *) lParam;
    int wsize = MultiByteToWideChar( CP_ACP,
                                     MB_PRECOMPOSED,
                                     (LPCSTR) cd->lpData,
                                     cd->dwData,
                                     NULL,
                                     0 );
    wchar_t *buf = (wchar_t *) calloc( wsize+1, sizeof(wchar_t) );
    MultiByteToWideChar( CP_ACP,
                         MB_PRECOMPOSED,
                         (LPCSTR) cd->lpData,
                         cd->dwData,
                         buf,
                         wsize + 1 );
    if (term->ldisc)
        luni_send(term->ldisc, buf, wsize, 0);
    free( buf );
}

At this point I have a working program - it connects to my designated host, and sends the username and password of my choice to the host, connecting me to the system.

I should add a note of caution here. Automating logins is a tempting time saver, but in general this is a really bad idea. Any time you hard code credentials into a program, you open the door to all sorts of new attacks on your system.

In my demo program, the user has to enter a name and password, so nothing is hardcoded, but even this adds security holes to a system. I encourage you to think of this as a demonstration only.

Demo of the program in action
For a better view, go to full screen and select 720p

Source Code

I've included the complete source code for PuttyDriver, the MFC project that controls AutoPutty. It was built with Visual Studio 2010, so you may have a little work to do if you backport it to earlier versions. My use of language features and classes should be compatible with much earlier versions - this is all very simple code.

Because PuTTY is always changing, I am not redistributing a snapshot of the version I used. Instead, I'm including before and after copies of the two source files I modified: window.c and terminal.c. If you build with Putty 0.61, you should be able to drop these two files right on top of the files included with the distribution and be on your way. With later versions of PuTTY you will have to perform an intelligent merge of the changes, which I hope will be a fairly effortless process.

Downloads

PuttyDriver.zip. The PuttyDriver source and project. You will need to add the PuTTY project to this solution as described in the article.
putty.zip. This contains the two PuTTY source files modified for this project. Both the original 0.61 source and my modified source are supplied. Executables are supplied as well, which may or may not work on your system.

VC++ 10 Hash Table Performance Problems

Mark Nelson — Mon, 28 Nov 2011 14:05:45 +0000

Microsoft has never been a slacker in the C++ department - they've always worked hard to provide a top-notch, compliant product. Visual Studio 10 supports their current incarnation, and for the most part it is up to their usual standards. It's a great development environment, and I am a dedicated user, but I have to give Microsoft a demerit in one area: their C++11 hash containers have some serious performance problems - so much that the Debug versions of the containers may well be unusable in your application.

Background

I first noticed the problem with unordered_map when I was working on the the code for my updated LZW article. I found that when running in the debugger, my program would hang after exiting the compression routine. A little debugging showed that the destructor for my hash table was taking a long time to run. And by a long time, I mean it was approaching an hour!.

This seemed pretty crazy. Destroying a hash table wouldn't seem to be a complicated task. I decided to see if I could come up with a reasonable benchmark. I wrote a test program that does a simple word frequency count. As a starter data set, I used the first one million white space delimited words in the 2010 CIA factbook, as published by Project Gutenberg. This data set yields 74,208 unique tokens.

I wrote a simple test rig that I used to test the word count program using four different containers:

unordered_map indexed by std::string
unordered_map indexed by std::string *
map indexed by std::string
map indexed by std::string *

The reason for testing with std::string * was to reduce the cost of copying strings into the hash table as it was filled, and then to reduce the cost of destroying those strings when the table was destroyed.

I ran tests against map expecting to see a pretty big difference in performance. Because map is normally implemented using a balanced binary tree structure, it has O(log(N)) performance on insertions. A sparsely populated hash table can have O(1) performance. By using fairly large data sets, I expected to see a big difference between the two.

I tried to eliminate a few obvious sources for error in my test function - and I used a template function so that I could use the same code on all the different container types:

template
void test( const DATA &data, const char *test_name )
{
  std::cout << "Testing container: " << test_name << std::endl;

#ifdef _DEBUG
  const int passes = 2;
#else
  const int passes = 10;
#endif
  double fill_times = 0;
  double delete_times = 0;
  size_t entries;
  for ( int i = 0 ; i < passes ; i++ ) {
    CONTAINER *container = new CONTAINER();
    std::cout << "Filling... " << std::flush;
    clock_t t0 = clock();
    for ( auto ii = data.begin() ; ii != data.end() ; ii++ )
      (*container)[*ii]++;
    double span = double(clock() - t0)/CLOCKS_PER_SEC;
    fill_times += span;
    entries = container->size();
    std::cout << " " << span << " Deleting... " << std::flush;
    t0 = clock();
    delete container;
    span = double(clock() - t0)/CLOCKS_PER_SEC;
    delete_times += span;
    std::cout << span << " " << std::endl;
  }
  std::cout << "Entries: " << entries
            << ", Fill time: " << (fill_times/passes)
            << ", Delete time: " << (delete_times/passes)
            << std::endl;
}

I didn't go overboard when it came to instrumenting this problem, I just used the timing functions built into the C++ library. On my Windows and Linux test systems, the values of CLOCKS_PER_SEC are both high enough that I'm not worried about granularity issues.

The First Results

I ran my test program in Visual C++ Release mode, using all the standard settings for a console application. For purposes of comparison, I ran the same program using g++ 4.6.1 on the same computer, booted up under Linux. For the set of 1,000,000 tokens, the results are shown below:

Task	VC++ 10 Release	g++ 4.6.1 -O3
Fill `unordered_map`	0.41s	.11s
Fill `unordered_map`	0.39s	0.14s
Destroy `unordered_map`	3.17s	0.01s
Destroy `unordered_map`	3.24s	0.004s
Fill `map`	0.83s	.53s
Fill `map`	0.88s	0.66s
Destroy `map`	.14s	0.01s
Destroy `map`	.07s	0.002s

There are a few interesting points to take away from these tests:

Microsoft's compiler is taking an exceptionally long time to destroy hashed containers - one order of magnitude greater than it took to create it, and two orders of magnitude greater than it takes g++ to do the same task.
It doesn't look like constructing and destroying the strings is a big factor. Both compilers have roughly the same performance with both std::string and std::string *. Microsoft's behavior is counterintuitive, as it takes longer to construct and destroy containers using the pointer.
The GNU compiler appears to be able to run through this exercise notably faster.

The time it takes to destroy the table is a concern - having a C++ program hang for over 3 seconds to destroy a modestly large data structure is a serious concern - particularly when the same task completes in a few milliseconds with g++.

The Pathological Results

These concerns are nothing compared to what I see when running in debug mode. Setting my Visual Studio project to Debug mode, then running the same test, yields the results shown here:

Task	VC++ 10 Debug
Fill `unordered_map`	17.41s
Fill `unordered_map`	17.08s
Destroy `unordered_map`	505.36s
Destroy `unordered_map`	505.99s
Fill `map`	13.29s
Fill `map`	13.15s
Destroy `map`	0.94s
Destroy `map`	0.18s

Those numbers are hard to believe. Destroying a hash table takes one millisecond when using g++. In VC++ 10, it takes almost ten minutes!

Worse, we suddenly see that hashed containers are slower than the containers built on red-black trees. Again, this just doesn't make sense.

The big problem with these numbers is that it means the debug mode of the compiler is effectively unusable for a lot of tasks. Regardless of how much testing it does, when it is this slow, it is just not useful.

A Workaround

I didn't invest the time to try debugging Microsoft's library, so I don't really know where the time is being spent. I did try a few things to speed things up, and I found one technique that helps a lot. Before including any Microsoft header files, try entering this single line in your source:

#define ITERATOR_DEBUG_LEVEL 0

With this definition in place, the delete times return to ball park of the times seen when running in release mode. Of course, you give up some debugging. I believe that an explanation of what this macro does might be found here.

In the final analysis, I think Microsoft has some serious work to to do here. The performance of their hashed containers, and to some lesser extent, the pre-C++11 associative containers, needs some serious examination. If the library is going to run this much slower than the competition, I need a good explanation why.

Source

HashTest.cpp

LZW Revisited

Mark Nelson — Tue, 08 Nov 2011 15:21:41 +0000

One of the first articles I wrote for Dr. Dobb's Journal, LZW Data Compression, turned out to be very popular, and still generates a fair amount of traffic and email over twenty years later.

One of the reasons for its popularity seems to be that LZW compression is a popular homework assignment for CS students around the world. And that audience sometimes found the article to be bit of a struggle. My code was modeled on the UNIX compress program, which was written in terse C for maximum efficiency. And sometimes optimization comes at the expense of comprehension.

By using C++ data structures I can model the algorithm in a much more straightforward way - the language doesn't get in the way of a clear implementation. And after 20 years of answering puzzled queries I think I can improve on the overall explanation of just how LZW works.

In this updated look at LZW, I will first give a description of how LZW works, then describe the core C++ code that I use to implement the algorithm. I'll then walk you through the use of the algorithm with a few varieties of I/O. Finally, I'll show you some benchmarks.

I'm hoping that this version of the article will be good enough to last for another 20 years.

LZW Basics

LZW compression works by reading a sequence of symbols, grouping the symbols into strings, and converting the strings into codes. Because the codes take up less space than the strings they replace, we get compression.

My implementation of LZW uses the C++ char as its symbol type, the C++ std::string as its string type, and unsigned int as its code type. The tables of codes and strings are implemented using unordered_map, the C++ library's hash table data structure. By using the native types and standard library data structures the representation in the program is straightforward and easy to follow.

Encoding/Decoding

Rather than jumping directly into a full implementation, I'm going to work my way up to LZW one step at a time.

The first step is getting a clear understanding of how the encoding and decoding process works. As I said earlier, LZW compression converts strings of symbols into integer codes. Decompression converts codes back into strings, returning the same text that we started with.

LZW is a greedy algorithm - it tries to find the longest possible string that it has a code for, then outputs that string. The code below is not quite LZW, but it shows you the basic idea of how a greedy encoder can work:

void encode( input_stream in, output_stream out )
{
  //
  // This hash table contains a list of codes, indexed
  // by the string that corresponds to the code.
  //
  std::unordered_map codes;
  //
  // There is presumably some code here that initializes
  // the dictionary with a set of codes based on whatever
  // algorithm we are implementing.
  //
  ...initialize the dictionary
  //
  // With codes in the dictionary, encoding is
  // now ready to begin.
  //
  std::string current_string;
  char c;
  while ( in >> c ) {
    current_string = current_string + c;
    if ( codes.find(current_string) == codes.end() ) {
      current_string.erase(current_string.size()-1);
      out << codes[current_string];
      current_string = c;
    }
  }
  out << codes[current_string];
}

The greedy encoder reads characters in from the uncompressed stream, and appends them one by one to the variable current_string. Each time it lengthens the string by one character, it checks to see if it still has a valid code for that string in the dictionary.

This continues until we eventually add a character that forms a string that isn't in the dictionary. So we then erase the last character from that string, and issue the code for the resulting string - the string from the previous pass through the loop.

The value of current_string is then initialized with the character that broke the camel's back, and the algorithm continues in the loop, building new strings until it runs out of input characters. At that point it outputs the last remaining code and exits.

As an example of how this would work, imagine I have the input stream ACABCA, and my code dictionary looks like this:

String	Code
A	1
B	2
C	3
AB	4
ABC	5

A sample dictionary

If you follow the algorithm above, you'll see that the code output has to be 1 3 5 1. If this wasn't a greedy algorithm, 1 3 4 3 1 would have been another valid output.

Decoding the stream in a system like this is very straightforward:

void decode( input_stream in, output_stream out )
{
  std::unordered_map strings;
  //
  // Initialize the code table with the same set of codes and strings
  // that the encoder used for your algorithm.
  //
  ...initialize the dictionary
  //
  // With codes in the dictionary, decoding is now
  // ready to begin.
  //
  unsigned int code;
  while ( in >> code )
    out << strings[code];
}

Remember, the decoder shown above is just a hypothetical sample - we're still working our way up to the full LZW decoder.

The LZW Encoder

The encoder shown above works okay, but there is one missing ingredient: management of the code dictionary. If you think about it, you'll see that we only achieve reasonable compression when we are able to build up longer strings and find them in the dictionary. Building a useful dictionary is referred to in the data compression world as modeling.

But our management of the dictionary is constrained by an important requirement: the encoder and decoder both have to be working with the same copy of the dictionary. If they have different dictionaries, the encoder might send a string that the decoder can't resolve.

Some data compression algorithms solve this problem by using a predefined dictionary that both the encoder and the decoder know in advance. But LZW builds a dictionary on the fly, using an adaptive method that ensures both the encoder and decoder are in sync.

LZW manages this in an effective and provably correct fashion. First, both the encoder and decoder initialize the dictionary with all possible single digit strings. For the compressor, that looks like this:

for ( unsigned int i = 0 ; i < 256 ; i++ )
    codes[std::string(1,(char)i)] = i;

This insures that we can encode all possible streams. No matter what, we can always break a stream down into single digits and encode these, knowing that the decoder has the same strings in its dictionary with values 0-255.

Then comes the key component of the LZW algorithm. If you go back to the greedy encoding loop above, you'll see that I keep adding input symbols to a string until I find a string that isn't in the dictionary. This string has the characteristic of being composed of a string that currently exists in the dictionary, with one additional character.

LZW then takes that new string and adds it to the dictionary, creating a new code. The strings are added to the table with code values that increment by one with each new entry.

The resulting code is just a slightly modified version of the encoder that I listed above. It still only outputs codes for values that are in the dictionary, but now the dictionary is being updated with a new string every time an existing code is sent:

void compress( input_stream in, output_stream out )
{
  std::unordered_map codes;
  for ( unsigned int i = 0 ; i < 256 ; i++ )
    codes[std::string(1,(char)i)] = i;
  unsigned int next_code = 257;
  std::string current_string;
  char c;
  while ( in >> c ) {
    current_string = current_string + c;
    if ( codes.find(current_string) == codes.end() ) {
      codes[ current_string ] = next_code++;
      current_string.erase(current_string.size()-1);
      out << codes[current_string];
      current_string = c;
    }
  }
  out << codes[current_string];
}

The code above constitutes a more or less complete LZW encoder. I've only made a couple of additions to the previous encoder:

The initialization of codes 0-255 with all possible single character strings.
The insertion of the newly discovered string into the string table, generating a new code.

(One item of note in this code: you might wonder why next_code is initialized to 257, when 256 is the first free code. This is because I reserve code 256 for an EOF marker. More on this in a later section.)

Just to make sure this all adds up, I'll walk through the steps the encoder takes as it processes a string from a simple two letter alphabet: ABBABBBABBA. There are a lot of steps shown below, but working through the process in detail is a great way to be sure you understand it:

Input Symbol	Action(s)	New Code	Output Code
A	read 'A' - set current_string to 'A' 'A' is in the dictionary, so continue
B	read 'B' - set current_string to 'AB' 'AB' is not in the dictionary, add it with code 257 output the code for 'A' - 65 set current_string to 'B'	257 (AB)	65 (A)
B	read 'B' - set current_string to 'BB' 'BB' is not in the dictionary, add it with code 258 output the code for 'B' - 66 set current_string to 'B'	258 (BB)	66 (B)
A	read 'A' - set current_string to 'BA' 'BA' is not in the dictionary - add it with code 259 output the code for 'B' - 66 set current_string to 'A'	259 (BA)	66 (B)
B	read 'B' - set current_string to 'AB' 'AB' is in the dictionary, so continue
B	read 'B' - set current_string to 'ABB' 'ABB' is not in the dictionary - add it with code 260 output the code for 'AB' - 257 set current_string to 'B'	260 (ABB)	257 (AB)
B	read 'B' - set current_string to 'BB' 'BB' is in the dictionary, so continue
A	read 'A' - set current_string to 'BBA' 'BBA' is not in the dictionary - add it with code 261 output the code for 'BB' - 258 set current_string to 'A'	261 (BBA)	258 (BB)
B	read 'B' - set current_string to 'AB' 'AB' is in the dictionary, so continue
B	read 'B' - set current_string to 'ABB' 'ABB' is in the dictionary, so continue
A	read 'A' - set current_string to 'ABBA' 'ABBA' is not in the dictionary - add it with code 262 output the code for 'ABB' - 260 set current_string to 'A'	262 (ABBA)	260 (ABB)
EOF	end of the input stream - exit loop current string is 'A' output the code for 'A' - 65		65 (A)

After processing string ABBABBBABBA, the output codes are 65,66,66,257,258,260,65. The dictionary at this point is:

String	Code
AB	257
BB	258
BA	259
ABB	260
BBA	261
ABBA	262

The dictionary generated for ABBABBBABBA
(Entries 0-255 not shown for brevity)

Looking at the above table, you can see a few interesting things happening. First, every time the algorithm outputs a code, it also adds a new code to the dictionary.

More importantly, as the dictionary grows, it starts to hold longer and longer strings. And the longer the string, the the more compression we can get. If the algorithm starts emitting integer codes for strings of length 10 or more, there is no doubt that we are going to get good compression.

As an example of how this works on real data, here are some entries from the dictionary created when compressing Alice's Adventures in Wonderland:

34830 : 'even\n'
34831 : '\nwith t'
34832 : 'the dr'
34833 : 'ream '
34834 : ' of Wo'
34835 : 'onderl'
34836 : 'land'
34837 : 'd of l'
34838 : 'long ag'
34839 : 'go:'

These strings have an average length of almost six characters. If we are writing the integer codes to a file using 16 bit binary integers, these entries offer the possibility of 3:1 compression.

The word adaptive is used to describe a compression algorithm that adapts to the type of text it is processing. LZW does an excellent job of this. If a string is seen repeatedly in the text, it will show up in longer and longer entries in the dictionary. If a string is seen rarely, it will not be the foundation for a large batch of longer strings, and thus won't waste space in the dictionary.

The LZW Decoder

The change made to the basic encoder to accommodate the LZW algorithm was really very simple. One small batch of code that initializes the dictionary, and another few lines of code to add every new unseen string to the dictionary.

As you might suspect, the changes to the decoder will be fairly simple as well. The first change is that the dictionary must be initialized with the same 256 single-symbol strings that the encoder uses.

Once the decoder starts running, each time it reads in a code, it must add a new value to the dictionary. And what is that value? The entire content of the previously decoded string, plus the first letter of the currently decoded string. This is exactly what the encoder does to create a new string, and the decoder must following the same steps:

void decompress( input_stream in, output_stream out )
{
  std::unordered_map strings;
  for ( int unsigned i = 0 ; i < 256 ; i++ )
    strings[i] = std::string(1,i);
  std::string previous_string;
  unsigned int code;
  unsigned int next_code = 257;
  while ( in >> code ) {
    out << strings[code];
    if ( previous_string.size() )
      strings[next_code++] = previous_string + strings[code][0];
    previous_string = strings[code];
  }
}

I won't do a walk-through of the the decoder - you should be able to take the codes output from the encoder, shown above, and run them through the decoder to see that the output stream is what we expect.

The important thing is to understand the logic behind the decoder. When the encoder encounters a string that isn't in the dictionary, it breaks it into two pieces: a root string and an appended character. It outputs the code for the root string, and adds the root string + appended character to the dictionary. It then starts building a new string that starts with the appended character.

So every time the decoder uses a code to extract a string from the dictionary, it knows that the first character in that string was the appended character of the string just added to the dictionary by the encoder. And the root of the string added to the dictionary? That was the previously decoded string. This line of code implements that logic:

    strings[next_code++] = previous_string + strings[code][0];

It adds a new string to the dictionary, composed of the previously seen string, and the first character of the current string. Thus, the decoder is adding strings to the dictionary just one step behind the encoder.

You might note one curious point in the decoder. Instead of always adding the string to the dictionary, it is only done conditionally:

if ( previous_string.size() )
  strings[next_code++] = previous_string + strings[code][0];

The only time that previous_string.size() is 0 is on the very first pass through the loop. And on the first pass through the loop, we don't have a previous string yet, so the decoder can't build a new dictionary entry. Again, the decoder is always one step behind the encoder, which is a key point in the next section, which puts the final touches on the algorithm.

The Catch

So far the LZW algorithm we've seen seems very elegant - that's a characteristic we associate with algorithms that can be expressed in just a few lines of code.

Unfortunately, there is one small catch in this perceived elegance - the algorithm as I've shown it to you has a bug.

The bug in the algorithm relates to the fact that the encoder is always one step ahead of the decoder. When the encoder adds a string with code N to the table, it sends enough information to the decoder to allow the decoder to figure out the value of the string denoted by code N-1. The decoder won't know what the value of the string corresponding to code N is until it receives code N+1.

This makes sense if you recall the key line of code from the decoder. It calculates the value of the string encoded by N-1 by looking at the string it received on the previous iteration, plus the first character of the current string. And that current string is the one that was sent after encoding N.

So how can this get us in trouble? The encoder is always one entry ahead of the decoder - it has entry N in its dictionary, and the decoder has entry N-1. So if the encoder ever sends code N, the decoder will look in its table and come up empty-handed, unable to do its job of decoding.

A simple example will show you how this can happen. Let's look at the state of the encoder after it has sent the first five symbols in a stream: ABABA:

Input Symbol	Action(s)	New Code	Output Code
A	read 'A' - set current_string to 'A' 'A' is in the dictionary, so continue
B	read 'B' - set current_string to 'AB' 'AB' is not in the dictionary, add it with code 257 output the code for 'A' - 65 set current_string to 'B'	257 (AB)	65 (A)
A	read 'A' - set current_string to 'BA' 'BA' is not in the dictionary, add it with code 258 output the code for 'B' - 66 set current_string to 'A'	258 (BA)	66 (B)
B	read 'B' - set current_string to 'AB' 'AB' is in the dictionary, so continue
A	read 'A' - set current_string to 'ABA' 'ABA' is not in the dictionary, add it with code 259 output the code for 'AB' - 257 set current_string to 'A'	259 (ABA)	257 (AB)

Now we are set for trouble. The encoder has symbol 259 in its dictionary, while the decoder has only gotten to 258. If the encoder were to send a code of 259 for its next output, the decoder would not be able to find it in its dictionary. Can this happen?

Yes, if the next two characters in the stream are BA, the next code output by the encoder will be 259, and the decoder will be lost.

In general, this can happen when a dictionary entry exists that consists of a string plus a character, and the encoder encounters the sequence string+character+string+character+string. In the example above, the value of string is A, and the value of character is B. After the encoder counters AB, it has string+character in the dictionary, so if the following sequence is ABABA, we will emit code N.

Whether this is likely to happen or not is not too important, what is important is that it most definitely can happen, and the decoder has to be aware of it. And it will happen repeatedly in the pathological case: a stream that consists of a single symbol, repeated on end.

The good news is that the problem is easily solved. When the decoder receives a code, and finds that this code is not present in its dictionary, it knows right away that the code must be the one that it will add next to its decoder. And because this only happens when we are encoding the sequence discussed above, the decoder knows that instead of using this value for that code:

    strings[next_code++] = previous_string + strings[code][0];

it can instead use this value:

    strings[ code ] = previous_string + previous_string[0];

The result of this is the insertion of just two lines of code at the start of the decompress loop, giving a loop that now looks like this:


while ( in >> code ) {
  if ( strings.find( code ) == strings.end() )
    strings[ code ] = previous_string + previous_string[0];
  out << strings[code];
  if ( previous_string.size() )
    strings[next_code++] = previous_string + strings[code][0];
  previous_string = strings[code];
}

And with that, you have a complete implementation of the LZW encoder and decoder.

Implementation

Now that I've shown you the algorithm, the next step is to take that code and add turn it into a working program. Without changing the algorithm itself, I'm going to take you through four different customizations that work as follows:

LZW-A reads and writes code values rendered in text mode, which is great for debugging. It means you can view the output of the encoder in a text editor.
LZW-B reads and writes code values as 16-bit binary integers. This is fast and efficient, and usually results in significant data compresion.
LZW-C reads and writes code values as N-bit binary integers, where N is determined by the maximum code size. Performing I/O on codes that are not aligned on byte boundaries complicates the code somewhat, but allows for greater efficiency and better compression.
LZW-D reads and writes code values as variable-length binary integers, starting with 9-bit codes and gradually increasing as the dictionary grows. This gives the maximum compression.

Before launching into these implementations, the code I showed above needs some minor tweaking to solve a couple of problems.

The first problem we have to deal with is the ever-expanding dictionary. In the algorithm I've presented, we keep adding new codes to the dictionary without end. This needs to be changed for a couple of reasons.

First, we don't have unlimited memory, so the dictionary simply can't grow forever. Second, practical experience shows that compression ratios don't improve as dictionary sizes grow without bound. As the dictionary grows, code sizes get larger and larger, and so they take up more space in the compressed stream, which can reduce compression efficiency.

To resolve this problem, I just add an additional argument to the encoder and decoder that sets the maximum code value that will be added to the dictionary. The function signatures now look like this:

void compress( input_string input,
               output_stream output,
               const unsigned int max_code = 32767 );
void decompress( input_string input,
                 output_stream output,
                 const unsigned int max_code = 32767 );

Implementing it means one small change in the encoder:

if ( next_code <= max_code )
  codes[ current_string ] = next_code++;

And a corresponding change in the decoder:

if ( previous_string.size() && next_code <= max_code )
  codes[ current_string ] = next_code++;

Input and Output

Finally, I need to give the algorithm a decent way to perform input and output - and this is where C++ offers a huge amount of help.

When writing generic compression code that you intend to use in multiple contexts, one of the more difficult things to deal with is I/O. People using your code might want to compress data in memory, stored in files, or streaming in from sockets or other sources. Some input data sources might be of unknown length (data coming from a TCP socket, for example), while others will be of a prescribed length. Back in the days of C, it was particularly difficult to make your compression code both generic, so it would work with all types of data streams, and efficient, so that I/O doesn't take any more time than it has to.

With the advent of C++, we have a new tool that can help in this quest - templates. Templates are designed to solve this problem in an efficient way, and I take advantage of this in my sample code. The code below shows the final version of the compressor and decompressor that are are used in all four versions of the implementation. There are two final changes made to the routines shown previously. First, both C++ functions are now function templates, parameterized on the the types being used for input and output. Second, the actual input and output is done through four newly introduced template classes:

template
void compress( INPUT &input, OUTPUT &output, const unsigned int max_code = 32767 )
{
  input_symbol_stream in( input );
  output_code_stream out( output, max_code );

  std::unordered_map codes( (max_code * 11)/10 );
  for ( unsigned int i = 0 ; i < 256 ; i++ )
    codes[std::string(1,i)] = i;
  unsigned int next_code = 257;
  std::string current_string;
  char c;
  while ( in >> c ) {
    current_string = current_string + c;
    if ( codes.find(current_string) == codes.end() ) {
      if ( next_code <= max_code )
        codes[ current_string ] = next_code++;
      current_string.erase(current_string.size()-1);
      out << codes[current_string];
      current_string = c;
    }
  }
  if ( current_string.size() )
    out << codes[current_string];
}

template
void decompress( INPUT &input, OUTPUT &output, const unsigned int max_code = 32767  )
{
  input_code_stream in( input, max_code );
  output_symbol_stream out( output );

  std::unordered_map strings( (max_code * 11) / 10 );
  for ( int unsigned i = 0 ; i < 256 ; i++ )
    strings[i] = std::string(1,i);
  std::string previous_string;
  unsigned int code;
  unsigned int next_code = 257;
  while ( in >> code ) {
    if ( strings.find( code ) == strings.end() )
      strings[ code ] = previous_string + previous_string[0];
    out << strings[code];
    if ( previous_string.size() && next_code <= max_code )
      strings[next_code++] = previous_string + strings[code][0];
    previous_string = strings[code];
  }
}

What exactly is the effect of implementing this algorithm using a pair of function templates, parameterized on the the types of the input and output objects? What this means is that you can call these compression routines with any type of I/O object you can throw at them. It can work with C++ iostreams, C FILE * objects, raw blocks of memory, whatever you want.

But there's a catch to that flexibility - you have to implement some basic I/O routines for whatever type you are using. Fortunately, this is not too hard.

The actual I/O that is done in the compression routines is defined by four template classes I created. These classes are defined in lzw_streambase.h. These classes don't have implementations, but they do define the methods you need to implement to work with the compressor and decompressor. The four classes are:

input_symbol_stream
ouput_symbol_stream
input_code_stream
output_code_stream

The first two classes are the symbol input and output classes. These are normally going to be very simple implementations, as they just have to read single characters to and from streams, while checking for errors or ends of streams. I use the same versions of these classes in all four implementations, so the code in lzw-a.h is unchanged in the other three header files.

The input_symbol_stream class has one member function: the extraction operator, which reads a character from the stream and returns a boolean true or false. You'll see later in this section that the implementation of this for types such as std::istream is trivial.

template
class input_symbol_stream
{
public :
    input_symbol_stream( T & );
    bool operator>>( char &c );
};

The output_symbol_stream class uses the insertion operator to write strings instead of individual characters - because that is what is stored in the dictionary. The C++ std::string class makes a perfectly good container for any variety of symbols, including binary data, and unlike the alternative vector, it comes with hash functions and iostream operators.

template
class output_symbol_stream
{
public :
    output_symbol_stream( T &  );
    void operator<<( const std::string &s );
};

The input_code_stream class reads codes, normally unsigned integers, from some type of stream. In my implementations, this class also returns false if it encounters the EOF_CODE in the stream of incoming codes. Removing the responsibility for EOF detection from the decompressor makes the code a bit simpler and more versatile.

The formatting of the integer is entirely up to the implementor, but the most common approach will probably be variable length codes ranging from 9 to 16 or so bits.

template
class input_code_stream
{
public :
    input_code_stream( T &, unsigned int );
    bool operator>>( unsigned int &i );
};

The output_code_stream class writes codes, usually unsigned integers, to some type of stream. Whatever class you implement for this function must agree with the implementation for input_code_stream.

template
class output_code_stream
{
public :
    output_code_stream( T &, unsigned int );
    void operator<<( const unsigned int i );
};

You can see that at the top of the compressor and decompressor, I instantiate objects of these types, then use the standard insertion and extraction operators to read and write from these objects.

LZW-A

In my sample windows program, I include lzw_streambase.h and lzw.h, which accounts for all of the code you have seen so far. I have the following lines that perform compression and decompression:

std::ifstream in( name, std::ios_base::binary );
std::ofstream lzw_out( temp_name_lzw, std::ios_base::binary );
compress( (std::istream &) in, (std::ostream&) lzw_out, pDlg->m_MaxCodeSize );
.
.
.
std::ifstream lzw_in( temp_name_lzw, std::ios_base::binary );
std::fstream out( temp_name_out,
                  std::fstream::in    |
                  std::fstream::out   |
                  std::fstream::binary );
decompress( (std::istream &) lzw_in, (std::ostream&) out, pDlg->m_MaxCodeSize );

If I try to build this project as-is, I get a nasty list of eight linker errors:

Visual Studio 10 Error Messages

If you have the fortitude to crawl through those link errors, you will see that what is missing are the implementations of the four classes parameterized on std::ostream and std::istream. Each of the four classes needs the implementation of a constructor and either an insertion or extraction operator. And with no class definitions at all, that adds up to eight missing functions. To get us started on performing actual LZW compression, I've created the first implementation of these four classes in lzw-a.h. Let's take a look at each of these in turn.

It's tempting to try to read characters using the ifstream extraction operator, as in m_impl >> c, but that operator skips over whitespace, so we don't get an exact copy of the input stream. Using get() works around this problem. Below is the complete definition of input_symbol_stream used in all four LZW implementations in this article:

template<>
class input_symbol_stream {
public :
    input_symbol_stream( std::istream &input )
        : m_input( input ) {}
    bool operator>>( char &c )
    {
        if ( !m_input.get( c ) )
            return false;
        else
            return true;
    }
private :
    std::istream &m_input;
};

Using the insertion operator to output strings seems to work properly, even when the strings contain binary data, so the implementation of the class used to output symbols is as simple as we could hope for. Again, this exact code is used in all four implementations in this article:

template<>
class output_symbol_stream {
public :
    output_symbol_stream( std::ostream &output )
        : m_output( output ) {}
    void operator<<( const std::string &s )
    {
        m_output << s;
    }
private :
    std::ostream &m_output;
};

LZW-A prints the text values of integers to the output stream, and reads them back in that format. This is not efficient at all, but it is a great aid in debugging. If you are having a problem with the algorithm, this provides a nice way to examine your stream. The implementation of this is very simple - just use the std::ostream insertion operator, and follow each code by a newline so it can be properly parsed on input, as well as be easily loaded into a text editor.

One important thing to notice in this class: the presence of a destructor that prints the EOF_CODE. Since this object goes out of scope as the compressor exits, this insures that every code stream will end with this special code. Putting the onus on the I/O routines to deal with EOF issues simplifies the algorithm itself. (It also means that you can implement versions of LZW that don't use an EOF in the code stream.)

template<>
class output_code_stream {
public :
    output_code_stream( std::ostream &output, const unsigned int )
        : m_output( output ) {}
    void operator<<( unsigned int i )
    {
        m_output << i << '\n';
    }
    ~output_code_stream()
    {
        *this << EOF_CODE;
    }
private :
    std::ostream &m_output;
};

The corresponding version of the input class just reads in the white-space separated codes. If there is an error or an EOF_CODE encountered in the stream, the extraction operator returns false, which allows the decompressor to know when it is time to stop processing.

template<>
class input_code_stream {
public :
    input_code_stream( std::istream &input, unsigned int )
        : m_input( input ) {}
    bool operator>>( unsigned int &i )
    {
        m_input >> i;
        if ( !m_input || i == EOF_CODE )
            return false;
        else
            return true;
    }
private :
    std::istream &m_input;
};

By including lzw-a.h along with the other two header files, I can now create a program that compiles, links, and is able to test the algorithm. Using my UNIX test program, I compress the demo string from earlier in this article, and I see the output as it is sent directly to stdout:

Compressing ABBABBBABBA

Fortunately, the output is identical to what was shown earlier, with the addition of the final EOF_CODE used to delimit the end of the code stream.

LZW-B

The header file lzw-b.h implements specialized classes that replace the text-mode output of the codes in lzw-a.h with binary codes stored in a short integer - two bytes.

The classes that read and write symbols are unchanged, but reading and writing codes has to change in order to do this new binary output.

Writing the codes to std::ostream as binary values requires breaking the integer code into two bytes and writing the bytes one at a time. There are more efficient ways to write the complete short integer in one function call, but they raise code portability problems, as we don't always know what order bytes will be written in.

Like the code stream output object in lzw-a.h, this version of the code output class has a destructor that outputs an EOF_CODE value:

template<>
class output_code_stream {
public :
    output_code_stream( std::ostream &output, const unsigned int )
        : m_output( output ) {}
    void operator<<( unsigned int i )
    {
        m_output.put( i & 0xff );
        m_output.put( (i>>8) & 0xff);
    }
    ~output_code_stream()
    {
        *this << EOF_CODE;
    }
private :
    std::ostream &m_output;
};

Reading the codes requires reading the two bytes that make up the short integer, then combining them. While reading, if the routine detects an EOF_CODE, it returns false, which tells the decompressor to stop processing. It also returns false if there is an error on the input code stream.

template<>
class input_code_stream {
public :
    input_code_stream( std::istream &input, unsigned int )
        : m_input( input ) {}
    bool operator>>( unsigned int &i )
    {
        char c;
        if ( !m_input.get(c) )
            return false;
        i = c & 0xff;
        if ( !m_input.get(c) )
            return false;
        i |= (c & 0xff) << 8;
        if ( i == EOF_CODE )
            return false;
        else
            return true;
    }
private :
    std::istream &m_input;
};

The most exciting thing about lzw-b.h is that you can now see data compression taking place. The figure below shows a sample run of this implementation against the Canterbury Corpus, a standard set of files used to test compression. A run with my Windows test program shows that the files are compressing quite nicely:

Compressing the Canterbury Corpus with lzw-b.h

LZW-C

The third I/O implentation, defined in lzw-c.h, writes binary codes like lzw-b.h, but with one crucial difference. Instead of being hard coded to 16 bit codes, lzw-c.h determines the maximum code size needed based on the maximum code value passed as an argument to compress() and decompress(). It then writes codes based on that width, which will normally be something in the range of 9-18 bits wide.

Since these values are not aligned with byte boundaries, there are some issues writing them to streams that expect to read and write bytes. However, it is definitely worth all the bit shifting, ORing, and ANDing, because when the size is 12 bites, we are going to save four bits per code when compared to using lzw-b.h. But every read and write potentially starts somewhere in the middle of a byte, so the I/O classes have to do some extra work - mostly involved with shifting bits to the correct position in the output stream.

Note that the code to read and write symbols is unchanged from lzw-a.h and lzw-b.h.

Many of the CS students who read my earlier article on LZW ran into a brick wall when they started trying to understand the code that performs I/O on codes of variable bit lengths. Obviously, writing 11 bit codes when your file system is oriented around eight-bit bytes involves a lot of bit twiddling, and I'm afraid that many novices are woefully deficient in this department. Not just in understanding the bitwise operators in C, such as shifting, masking, etc., but in understanding binary arithmetic in general.

That's why I've structured the code and this article a bit differently this time around. If the I/O operations in lzw-c.h and lzw-d.h are bewildering, well, no worries. They have absolutely nothing to do with the LZW algorithm itself. You can investigate and explore the algorithm completely using lzw-a.h and lzw-b.h, and just forget about the last two I/O implementations. They provide additional efficiency, but as I have said, have nothing to do with the algorithm itself.

Further, once you use lzw-a.h to debug and understand the algorithm, you can certainly plug in lzw-c.h and lzw-d.h and take advantage of their improved compression, even if you don't follow all the code.

It might be appropriate to add a sidebar or another section to explain the variable bit length I/O in detail, but this article is quite long already, and there are numerous other resources for the interested reader to explore the details. (But if you find yourself deficient in this area, you owe it to yourself to hit the books and get to the point where these operations make sense. This won't be the last time you need to understand bitwise operators.)

For those who are ready to tackle this more complicated I/O procedure, we will look first at the output_code_stream class. Here, the first thing to understand is that the constructor has to initialize the number of bits in the code. This value is calculated from the max_code parameter, and is stored in member m_code_size, where it is used frequently.

Next, the insertion operator. Output of codes proceeds as follows. Member m_pending_bits tells us how many bits are pending output while sitting in member m_pending_output. These bits are right justified, and the count will always be less than eight. When the new code is written, it is inserted into m_pending_output after being left shifted so it will be laid down just past the pending bits. After doing that, we presumably have some bytes to output - the exact number depends on various factors. The flush() routine is called, and it flushes all complete bytes out. When it completes, there can be anywhere from zero to seven bits still waiting to be output, and they will be right justified in m_pending_output.

In the destructor, we output an EOF_CODE, and then do a flush as well. But in this case, we flush all possible bits, not just the complete bytes. There are two good reasons for this. First, we don't care if the last bits that are flushed out are only part of a code - the code will be EOF_CODE, and that is the last one. And second, if we don't flush those final bits out in the destructor, they will never be sent to the output stream. This means the decoder will not see those bits, and we will most likely break the decompress process.

template<>
class output_code_stream
{
public :
    output_code_stream( std::ostream &out, unsigned int max_code )
        : m_output( out ),
          m_pending_bits(0),
          m_pending_output(0),
          m_code_size(1)
    {
        while ( max_code >>= 1 )
            m_code_size++;
    }
    ~output_code_stream()
    {
        *this << EOF_CODE;
        flush(0);
    }
    void operator<<( const unsigned int &i )
    {
        m_pending_output |= i << m_pending_bits;
        m_pending_bits += m_code_size;
        flush( 8 );
    }
private :
    void flush( const int val )
    {
        while ( m_pending_bits >= val ) {
            m_output.put( m_pending_output & 0xff );
            m_pending_output >>= 8;
            m_pending_bits -= 8;
        }
    }
    std::ostream &m_output;
    int m_code_size;
    int m_pending_bits;
    unsigned int m_pending_output;
};

Like the output code class, the input code class has to calculate the code size for this decompression based on the max_code value passed in the function call.

When an attempt is made to read a code, there must be a minimum of m_code_size bits in member m_pending_input. If there aren't, new bytes are read in one at a time, and inserted into m_pending_input after having been shifted left the appropriate amount. Once m_pending_input contains at least m_code_size bits, the code is extracted from m_pending_input using the appropriate mask, the count in m_pending_input is reduced, and m_pending_input is shifted right by m_code_size bits.

template<>
class input_code_stream
{
public :
    input_code_stream( std::istream &in, unsigned int max_code )
        : m_input( in ),
          m_available_bits(0),
          m_pending_input(0),
          m_code_size(1)
    {
        while ( max_code >>= 1 )
            m_code_size++;
    }
    bool operator>>( unsigned int &i )
    {
        while ( m_available_bits < m_code_size )
        {
            char c;
            if ( !m_input.get(c) )
                return false;
            m_pending_input |= (c & 0xff) << m_available_bits;
            m_available_bits += 8;
        }
        i = m_pending_input & ~(~0 << m_code_size);
        m_pending_input >>= m_code_size;
        m_available_bits -= m_code_size;
        if ( i == EOF_CODE )
            return false;
        else
            return true;
}
private :
    std::istream &m_input;
    int m_code_size;
    int m_available_bits;
    unsigned int m_pending_input;
};

The table below shows the results of a test run comparing LZW-B and LZW-C run with a maximum code of 4095. With this maximum value, all codes fit in a 12-bit integer. Since LZW-B will use a 16-bit integer to store the code values, and LZW-C will use 12-bits, there should be a 4:3 ratio between the ratio of the file sizes when compressed using the two algorithms, and this looks to be the case:

File Name	Original Size	Compressed LZW-B	Compressed LZW-C	Ratio
alice29.txt	152089	96428	72322	0.750
alphabet.txt	100000	4538	3404	0.750
asyoulik.txt	125179	83966	62975	0.750
bib	111261	71792	53845	0.750
bible.txt	4047392	2468326	1851245	0.750

Comparing 12-bit compression between LZW-B and LZW-C

It looks like things are working as expected.

LZW-D

The code in lzw-d.h represents the final and most efficient version of I/O for the LZW code streams. It builds on the code in lzw-c.h - at its core it is a variable bit-length I/O stream. However, there is one crucial difference from lzw-c.h: the code I/O in lzw-d.h starts at the smallest possible code size, nine bits, and increases the code size as needed, until it reaches the maximum value for this compression session. The maximum value is the parameter passed in to the invocation of compress() or decompress().

The logic behind this is pretty simple. Even if we are going to use 16-bit codes in an LZW program, when the program first starts, the maximum possible code the program can emit is 256, which only needs nine bits to encode. And each time we output a new symbol, that maximum possible code value only increases by one, which means that the first 256 codes output by the encoder can all fit in nine bits.

So the LZW-D encoder starts encoding using nine-bit code widths, and then bumps the value to ten as soon as the highest possible output code reaches 512. This process continues, incrementing the code size until the maximum code size is reached. At that point the code size stays fixed, as no new codes are being added to the dictionary.

The decoder follows exactly the same process - reading in the first code with a width of nine bits, then bumping to ten when the maximum possible input code reaches 512.

The code for this class is built on that from lzw-c.h, with some added complexity. Due to its increasing length, and the fact that it doesn't add too much to the discussion of LZW, I've omitted the listing, and instead refer you to the download available at the end of the article.

The Windows Test Program

When you develop compression code, there are a few different common tasks you are likely to want to perform:

Check your code for correctness, often through bulk testing.
Check your compression ratios against standard benchmarks.
Analyze your program's performance so as to make it more efficient and locate bottlenecks.

My Windows app is designed to help with all of these tasks. It basically allows you to select a single directory, set a maximum code size, then perform compression and decompression of all the files in the directory. An optional checkbox lets you include files in all directories under the test directory as well.

The application was built using Visual Studio 10, and it is a simple MFC Dialog-based application. It allows you to select a base directory, a maximum code size, and then compress all the files in that directory. If you select the recursion check box, you will also compress all the files in the entire tree of subdirectories below it.

Each file is compressed to a temporary location, then decompressed in a temporary location. The size of the compressed file is saved, and then a comparison is done to ensure that the original and expanded files are identical.

To help with data collection, after running a test, you can press the copy button and get the results of the test stuffed into your clipboard. Although it isn't visible in the display, the data stored in your clipboard includes the full path name of the original file, not just the basename.

This Visual Studio project takes advantage of a number of C++11 features, and as a result it will need some modification to work with earlier versions. Any version that supports unordered_map can be made to build without too many changes. And if you are going way back in time, you could replace unordered_map with map.

As shipped, the test program uses lzw-d.h. To use any of the other three other versions of I/O discussed in this article, just modify the include file selected at the top of LzwTestDlg.cpp. The figure below shows what the app looks like after running through some data:

The Windows test app after a test run

After pressing the copy button at the bottom of the dialog, you can paste the data into a spreadsheet and then crunch it to your heart's content:

Copying the data into a spreadsheet

The Linux Test Program

The LZW code is platform independent, and will build and run just fine on UNIX or Linux systems. The Linux test program, lzw.cpp, allows you to compress or decompress files from the command line. It builds just fine with g++ 4.5, as long as you use the -std=c++0x switch to turn on the latest language features. Compiling with earlier versions will require a few minor modifications.

The command line interface to the test program is not too complicated, and is probably best documented by looking at the usage output:

mrn@ubuntu:~/LzwTest$ g++ -std=c++0x lzw.cpp -o lzw
mrn@ubuntu:~/LzwTest$ ./lzw
Usage:
lzw [-max max_code] -c input output #compress file input to file output
lzw [-max max_code] -c - output     #compress stdin to file otuput
lzw [-max max_code] -c input        #compress file input to stdout
lzw [-max max_code] -c              #compress stdin to stdout
lzw [-max max_code] -d input output #decompress file input to file output
lzw [-max max_code] -d - output     #decompress stdin to file otuput
lzw [-max max_code] -d input        #decompress file input to stdout
lzw [-max max_code] -d              #decompress stdin to stdout
mrn@ubuntu:~/LzwTest$

Like the Windows test program, the command line program is built by default with lzw-d.h. Replacing this algorithm with any of the three others requires a minor change to the source code.

With the default build, the program produces output nearly identical to UNIX compress. The one difference is that UNIX compress monitors the compression ratio after the dictionary is full, and clears the dictionary if the ratio starts to deteriorate (which it almost always does.) I include a benchmark program that tests UNIX compress against the command line test program, and the results show that for small files, the file size is almost identical:

mrn@ubuntu:~/LzwTest$ ./benchmark.sh 65535 16 canterbury | head -n 15 | column -t
Filename                 Original-size  LZW-size  Compress-size
--------                 -------------  --------  -------------
canterbury/aaa.txt       33406          320       321
canterbury/alice29.txt   152089         62247     62247
canterbury/alphabet.txt  100000         3052      3053
canterbury/asyoulik.txt  125179         54989     54990
canterbury/a.txt         1              3         5
canterbury/bib           111261         46527     46528
canterbury/bible.txt     4047392        1417735   1377093
canterbury/book1         768771         317133    317133
canterbury/book2         610856         247593    251289
canterbury/cp.html       24603          11315     11317
canterbury/E.coli        4638690        1213579   1218349
canterbury/fields.c      11150          4963      4964
canterbury/geo           102400         77777     77777

You can see in this test that LZW-D and UNIX compress perform nearly identically for all but the largest files in the test sample. If I modify UNIX compress to not monitor compression ratios, the difference seen with larger files goes away:

mrn@ubuntu:~/LzwTest$ ./benchmark.sh 65535 16 canterbury | head -n 15 | column -t
Filename                 Original-size  LZW-size  Compress-size
--------                 -------------  --------  -------------
canterbury/aaa.txt       33406          320       321
canterbury/alice29.txt   152089         62247     62247
canterbury/alphabet.txt  100000         3052      3053
canterbury/asyoulik.txt  125179         54989     54990
canterbury/a.txt         1              3         5
canterbury/bib           111261         46527     46528
canterbury/bible.txt     4047392        1417735   1417735
canterbury/book1         768771         317133    317133
canterbury/book2         610856         247593    247593
canterbury/cp.html       24603          11315     11317
canterbury/E.coli        4638690        1213579   1213579
canterbury/fields.c      11150          4963      4964
canterbury/geo           102400         77777     77777

That provides some support for the notion that the algorithm shown here behaves properly.

Your Program

If you want to build your own program and use these classes, all you need is a C++11 compiler, or an earlier version and a willingness to make a few changes.

To use the classes, include in order lzw_streambase.h, one of the four implementation files for iostreams, preferably lzw-d.h, and finally, lzw.h. Because the significant code in these files is all implemented as template functions or classes, there is no library to include in your project, and no C++ source you have to compile separately.

All of the code in these header files has been hoisted into the lzw namespace, so you will either have to explicitly use the namespace when you invoke compress() and decompress(), or insert this line into your program:

using namespace lzw;

One thing to note about the I/O routines I have defined. The template functions are specialized on std::istream and std::ostream. If you innocently pass in an object such as an std::ifstream, you will get compile time errors. This is because C++ template matching is done on a very strict basis - the compiler won't generally try to figure out that std::ifstream is derived from std::istream, and use the existing class. So instead, you will need to cast your arguments to the types defined in the header files. (Or write your own implementations.)

Your rights to use this code are covered by my Liberal Code Use Policy. As I have mentioned before, this is teaching code, if you decide to use it in a production system, there are many optimizations you might want to perform.

Benchmarks

So how does LZW do when it comes to compression? LZW's original strength was its combination of good compression ratios with high speed compression. The UNIX compress program is still nice and fast, and Terry Welch's original application for LZW was in disk controllers. Because my program is a teaching program, it won't be nearly as fast as compress, but it's still useful to compare it to the de facto standard for lossless compression: the deflate algorithm.

We can compare LZW against deflate by a small modification of my benchmark script that uses gzip instead of compress. The table below shows the average compression ratios for the files in the canterbury corpus when compressed using maximum code widths of 15-18 bits. (The ratio is defined as 100*compressed_size/uncompressed_size, so 0% is perfect compression and 100% is no compression.)

gzip	LZW 15 bits	LZW 16 bits	LZW 17 bits	LZW 18 bits
32.7%	43.2%	42.6%	42.5%	42.3%

You can see that LZW does do a good job of compressing data, but the deflate algorithm used by gzip manages to squeeze an additional 10%, more or less, out of the files it compresses. The gap between LZW and deflate is larger on some types of files, and smaller on others, but deflate will almost always show a noticeable difference in compression ratios.

Variations

There are many variations on the code I've presented here that make sense.

One obvious change is to eliminate the special EOF_CODE used to delimit the end of the code stream. If the code stream is a file or other stream with an inherent EOF condition, there is no need for an EOF_CODE - simply reaching the end of the input stream will properly signal the end of the decoded material. Freeing up this one code will make a microscopically small improvement in the compression ratios of the product.

If you want to mimic the output of the compress program, you need to remove the EOF_CODE, and replace it with a CLEAR_CODE that has a value of 256. The compress program monitors the compression ratios it achieves after its dictionary is full, and when the ratio starts to decay, it issues the CLEAR_CODE. That code tells the decoder to clear its dictionary and make a fresh start with new nine-bit codes.

Once you get the hang of LZW, a good exercise to make sure you have it working properly is to create a GIF encoder and decoder. GIF uses LZW to losslessly compress images with a constrained palette, and after all these years is still somewhat of a standard on the web.

History

Usually the history lesson on an algorithm is at the start of the article, but this is a how-to piece, and I feel like the trip down memory lane is not as important as understanding how the algorithm works.

The roots of LZW were set down in 1978 when Jacob Ziv and Abraham Lempel published the second of their two seminal works on data compression, "Compression of Individual Sequences via Variable-Rate Coding". This paper described a general approach to data compression that involved building dictionaries of previously seen strings.

Ziv and Lempel's work was targeted at an academic audience, and it wasn't truly popularized until 1984 when Terry Welch published A Technique for High-Performance Data Compression. Welch's paper took the somewhat abstract Information Theory work of Ziv and Lempel and reduced it to practice in such a way that others could easily implement it.

UNIX compress was probably the first popular program that used LZW compression, and it very quickly became a standard utility on UNIX systems. The freely available code for compress was incorporated into ARC, one of the first archiving programs for PCs. In addition, the algorithm was used in the GIF file format, originally created by Compuserve in 1987.

LZW's popularity waned in the 1990s for two important reasons. First, Unisys began enforcing their patents that covered LZW compression, demanding and receiving royalties from various software companies. Not only did this make developers think twice about the liability they could incur while using LZW, it resulted in a general public relations backlash against using patented technology.

Secondly, the LZW algorithm was eclipsed on the desktop by deflate, as popularized by PKZIP. Not only did deflate outperform LZW, it was unencumbered by patents, and eventually had a very reliable and free open source implementation in zlib, a library written by a team lead by Marc Adler and Jean-loup Gailly. I don't know if there is any way to actually quantify this, but I think one could speculate that zlib is currently installed on more computer systems than any other software package in existence.

So LZW has settled down to an existence out of the limelight. It is still an important algorithm, used in quite a few file formats, and as this article shows, its simplicity makes it an excellent learning tool.

Downloads

LzwTest.zip - source for the Windows test app.
LzwExe.zip - The Windows test app executable.
lzw.tgz - source for the UNIX text app.

DNS Service Discovery On Windows

Mark Nelson — Tue, 25 Oct 2011 11:54:04 +0000

In a previous post I showed you how we use DNS Service Discovery in a product I work on for Cisco Systems. That project uses the Avahi browser, which does not have a Windows port. In this article, I'll show you how to perform service discovery on Windows using Apple's Bonjour SDK for Windows.

DNS-SD On Windows

Microsoft has been pushing us to use UPnP as our network discovery protocol, to the exclusion of all others. As a result, Windows ships with no support for DNS-SD - zip. And that might be the end of it, if it weren't for Apple's vested interest in having iTunes installed on every Windows machine in the world.

iTunes uses DNS-SD to share music catalogs across Local Area Networks - a natural choice with the native support in OS X for Zeroconf. Rather than isolate Windows users in an pocket UPnP universe, Apple chose to instead port the service discovery components of Bonjour to Windows, and install it with every copy of Windows. Thus, Windows and OS X users can happily share their iTunes libraries with no unusual calisthenics required.

To sweeten the deal, Apple has released a Bonjour SDK for Windows, which is currently shipping version 3.0 from their developer support site. In Apple's words:

The Bonjour SDK for Windows includes the latest version of Bonjour as well as header files, libraries, and sample code to assist developers in creating Bonjour enabled applications on Windows. The SDK has been updated with the Bonjour core that is bundled with iTunes 10.3.1. This release will bring all existing Bonjour functionality released in Mac OS X 10.7 into the Bonjour for Windows product.

It sounds pretty good, doesn't it?

An Emphasis on Kit

Installing the SDK is a decision-free breeze:

Installing the SDK

The installation may be easy, but the real dirty truth about the Bonjour SDK is that it is not much of an SDK at all. The developer interface to Bonjour services are packaged in a single DLL, and the SDK provides programmers with sample code that illustrates how to use some of the bindings in C, C#, Java, and VB. There is no documentation on the interfaces, no source for the Bonjour components, and the few samples don't begin to provide comprehensive coverage of the interface.

In other words, this is pretty much a code dump.

An Overview

For the most part, the Bonjour SDK interfaces follow a single pattern. Each request made to the API returns immediately, and gives you a reference that you can use to track the progress of your request. That reference can be converted to a file handle, and you can use the file handle to see when your request has some data to produce.

When your request has generated some data, and the Bonjour components are ready to deliver it to you, they do so by a callback mechanism - the Bonjour components make calls into your C or C++ program and provide the data you requested.

Most of the callbacks include a flag parameter. You can check the flag to see if there is any more data expected. If there isn't, you can delete the reference and you are then done with that particular call. Otherwise, you will have to wait for the request to be responded to when the Bonjour components get around to it.

An Example - Discover Service Types

I've written a demo program that browses the network for details on every service instance it can find, and presents the results in a tree format. The figure below shows the program running on my home network, where I have drilled down to get information about an instance of a print service.

The ServiceBrowser Sample Program

This program has to start by doing a top-level iteration of all the service types currently seen on my network. From my previous article, you might remember that I can use a special browse command to accomplish this. If you've installed the Bonjour SDK, you can run the dns-sd program and browse for service type _services._dns-sd._udp, which should give results something like this:

C:\Users\Mark>dns-sd -B _services._dns-sd._udp
Browsing for _services._dns-sd._udp
Timestamp     A/R Flags if Domain  Service Type  Instance Name
17:38:09.407  Add     3 13 .       _tcp.local.     _smb
17:38:09.407  Add     3 13 .       _tcp.local.     _printer
17:38:09.408  Add     3 13 .       _tcp.local.     _pdl-datastream
17:38:09.408  Add     3 13 .       _tcp.local.     _http
17:38:09.408  Add     3 13 .       _tcp.local.     _tivo-videos
17:38:09.409  Add     2 13 .       _tcp.local.     _csco-sb

Looking to do the same thing in my demo program, I turn to the header file dns_sd.h. This file ships with the Bonjour SDK and contains not only the API definition, but what passes for documentation. In that header file I see that there is a function called DNSServiceBrowse, and it looks like it does exactly what I want. In my sample program, my call to this routine is shown below:

DNSServiceRef client = NULL;
DNSServiceErrorType err;
err = DNSServiceBrowse( &client,
                        0,
                        0,
                        "_services._dns-sd._udp",
                        "",
                        IterateServiceTypes,
                        this );

You can get some exposition on each of the arguments I pass to the function from the header file, my brief comments on each are given below:

sdRef	Every call to the Bonjour API creates a new `DNSServiceRef` handle. It is initialized by the function call, and used later to retrieve the results.
flags	This parameter is not used in this version of the SDK.
interfaceIndex	This argument is used to select a specific network interface. For this particular function I want to browse on all available interfaces, so a value of 0 is used.
regType	The type of service being browsed for. Normally when you call `DNSServiceBrowse` you will use this parameter to specify a specific service type that you are interested in, such as `_http._tcp`. I'm using the special type of `_services._dns-sd._udp` in order to get a list of all published service types, not specific instances.
domain	By passing an empty string I tell the service that I want to see advertisements from all domains.
callback	The pointer to a callback function that will receive the responses to this request. The argument I pass in, `IterateServiceTypes`, is a static member of my MCF Dialog class.
context	The context variable is an opaque pointer type that is passed in to the Bonjour service. When it performs a callback, it will include a copy of the context pointer for the use of the callback function. I always pass in a pointer to my MFC Dialog class, so my callback functions have full access to the class members.

The important thing to note here is that the call to the API function doesn't return any important data. All I get back is an error code indicating that the request is being processed, and a DNSServiceRef handle that I use to track that progress.

The real action comes when my callback function is invoked. IterateServiceTypes is a static member of my Dialog class. Apple kept things simple by having one callback type for C and C++, which means no member functions. You could easily build shims to make it appear as though the DLL was calling member functions - it would just take a small modification to the code I'll show you here.

The function definition has to follow exactly the declaration given in dns_sd.h. My implementation starts like this:

void DNSSD_API CServiceBrowserDlg::IterateServiceTypes( DNSServiceRef sdRef,
                                                        DNSServiceFlags flags,
                                                        uint32_t interfaceIndex,
                                                        DNSServiceErrorType errorCode,
                                                        const char *serviceName,
                                                        const char *regtype,
                                                        const char *domain,
                                                        void *context )
{
	CServiceBrowserDlg *p = (CServiceBrowserDlg *) context;

It's worth walking through a look at each of the parameters in the callback:

sdRef	This is the same `DNSServiceRef` value that was created when the call was made to `DNSServiceBrowse`. You don't need to make any use of it in the callback, but it does provide a good way to correlate results with function calls, particularly if you are using a single callback function to process many different results.
flags	There are two important flag bits to check in this value. The first value, `kDNSServiceFlagsMoreComing` is used to indicate that there are definitely more callbacks coming. If that bit is cleared, there is no more pending data. The second flag bit, `kDNSServiceFlagsAdd` is used to indicate whether this service is being added or deleted. When you first start browsing, all the callbacks will be for services added. As services are added and removed from the system, additional callbacks will be generated with this bit both set and cleared.
interfaceIndex	In the callback, this index will be set to the index of the network interface where the advertisement was found. When it comes time to resolve this service, you need to pass in the correct index.
errorCode	If this value is not zero, the callback is indicating an error. As long as it is zero your code can process the input safely.
serviceName	This value contains the name of a discovered service - it is the whole point of the callback. Normally this will contain the name of an instance of a service. However, when browsing for the special name `_services._dns-sd._udp`, the instance name is actually a service type.
regtype	The type of the service - you may already know this information by the time you reach the callback, but if the callback is handling the results from multiple queries, it can be helfpul.
domain	The domain of the discovered service. Like the interface index, you need use the domain when you are attempting to resolve the service
context	A copy of the context variable passed in when the browse call was made.

If you look at the first line of code above, the first thing I do is cast the context pointer to its correct type, a pointer to my MFC Dialog class. Now I can make full use of all the members of the class, albeit via a call through a pointer instead of directly.

So what do I do with these services once I receive them? Well, for each service type that I find, I kick off a new browse process, looking for specific instances of the service. Just as an example, in my callback IterateServiceTypes, one of the callbacks returns a service type of _printer._tcp. In order to find all instances of this service, I have to call DNSServiceBrowse again, with that service name and the correct interface and domain. After inserting the service type into the tree, I make that call so I can start adding those instances:

HTREEITEM item = p->m_Tree.InsertItem( CA2T(service_type.c_str()), TVI_ROOT, TVI_SORT );
DNSServiceRef client = NULL;
DNSServiceErrorType err;
err = DNSServiceBrowse( &client,
                        0,
                        0,
                        service_type.c_str(),
                        "",
                        IterateServiceInstances,
                        context );

The key point to note about this call is that the callback function, IterateServiceInstances, is a different member function - one that expects to get the results of my browsing for instances of a specific service.

Driving the Callbacks

One thing I've skipped over so far - how do these callbacks actually get generated? Does the DLL asynchronously make calls into my code whenever events occur?

The Bonjour SDK lets your program control when callbacks occur by giving you the handle to the message pump. When you call DNSServiceProcessResult() with a single argument of a DNSServiceRef, you will generate a single callback message for the given reference. The callback will occur within the context of the call to DNSServiceProcessResult().

When you call DNSServiceProcessResult(), the Bonjour DLL will block if there are no messages ready to process. So how do you know when there are messages ready?

The indicator that messages are ready is given by a file descriptor associated with the DNSServiceRef. You can get a copy of the file descriptor by calling DNSServiceRefSockFD(), passing in a copy of the reference. When the file descriptor has data ready to read, you have callbacks pending. The easiest way to check this condition is to use the select() function, which can check multiple references in one fell swoop.

In my implementation of the callback message pump, I rely on an unordered_map called m_ClientToFdMap that contains a copy of all the DNSServiceRef references currently waiting for responses. I create the necessary data structure used by select(), then call it to get a list of all references that have callbacks pending. The core of this code looks like this:

int result = select(0, &readfds, (fd_set*)NULL, (fd_set*)NULL, &tv);
if ( result > 0 ) {
//
// While iterating through the loop, the callback functions might delete
// the client pointed to by the current iterator, so I have to increment
// it BEFORE calling DNSServiceProcessResult
//
    for ( auto ii = m_ClientToFdMap.cbegin() ; ii != m_ClientToFdMap.cend() ; ) {
        auto jj = ii++;
        if (FD_ISSET(jj->second, &readfds) )
            DNSServiceErrorType err = DNSServiceProcessResult(jj->first);
    }
}

This generates my callbacks efficiently, and because they are in the context of my main program's UI thread, I avoid a lot of unpleasant issues.

Threading Issues

My program manages the Bonjour callbacks in a fairly ugly fashion. When my browsing activity starts, I create a timer that fires once every 250 milliseconds. I process up to 10 callbacks in that timer call, then exit. This continues until there are no pending browser or resolution requests, at which time I kill the timer.

Depending on your use of DNS-SD, you may find that this is not as efficient as you like. If this is the case, you might find it useful to move your message pump code to a separate thread.

Once you do that, you can wait on all your callbacks by calling select with a long or infinite timeout. This has the effect of blocking your callback thread until it has actual work to do - resulting in a better use of CPU time.

There are some obvious downsides to this approach. Clearly you have to use some sort of locking mechanism on the data structures that are shared between your callback thread and the rest of your program. And the use of the select() statement with an infinite timeout is complicated by the possibility that you may be making or canceling browsing or resolution calls while your program runs.

A good way to deal with both of these problems is to invoke a socket-based message passing protocol between the callback thread and the other components of your program. If you restrict your interface to messages, you don't have to worry about locking access to shared data. And because you are using a socket for communications, your select() statement will be used to activate the thread when new messages arrive.

Character Sets

The days when DNS was limited to seven-bit ASCII characters are long gone. Service instances are encoded as UTF-8, and can use whatever Unicode characters they like. In the figure shown below, you can see the effects of that when I browse for instances of iTunes:

Character set problems in service instance names

You can see that OS X users have so-called curly quotes in their library instance names, and curly quotes are definitely outside the range of seven-bit ASCII. DNS-SD collects the names as UTF-8 encoded strings, and sends them to the console in that format.

By default, the Windows cmd.exe window doesn't render UTF-8 properly, but changing the code page to 65001 results in the correct rendering.

In my sample program, I deal with this with a two step approach. First, my program is built using the Unicode libraries, ensuring that I am able to render Unicode output properly. To conform with Microsoft's C++ paradigms, I use CString for all my Unicode strings, and wrap all my string literals in the _T() wrapper.

This works fine for my UI, but I can't use strings built of wchar_t to communicate with the Bonjour SDK - it expects eight bit characters with UTF-8 encoding. In m program I use the C++ std::string class everywhere where I am working with 8-bit characters that might be encoded in UTF-8. When it comes time to render one of those strings in my Unicode context, all I have to do is use the handy CA2T macro with the CP_UTF8 parameter, and things work properly.

Library Issues

The design of the Bonjour SDK imposes some uncomfortable restrictions on you when it comes to building your C or C++ program. Because you are linking directly to code found in the library dnssd.lib, you have to ensure that your program and that library link against the same version of the C run time library. And for the Bonjour SDK under Windows, this means you must link with the static, multithreaded, release version of the library.

You'll see the problem in this right away when you create an MFC project and try to build with dnssd.lib. By default, the project generator will probably have you using MFC in a shared DLL, and using the Multithreaded Debug DLL version of the C libraries. When you try to build like this, you will get some unpleasant error messages:

1>LINK : warning LNK4098:
         defaultlib 'msvcrtd.lib' conflicts with use of other libs;
         use /NODEFAULTLIB:library
1>LINK : warning LNK4098:
         defaultlib 'LIBCMT' conflicts with use of other libs;
         use /NODEFAULTLIB:library

A full featured SDK would provide libraries built for multiple scenarios, and you would pick the one of your choice depending on your build parameters. But with the Bonjour SDK, you don't get this choice, so you need to ensure that your project follows a few guidelines:

Under Configuration Properties/General, field Use of MFC needs to be set to Use MFC in a Static Library for both debug and release builds.
Under Configuration Properties/C++/Code Generation, field Runtime Library needs to be set to Multi-threaded (/MT) for both debug and release builds.
Under Configuration Properties/C++/Preprocessor, field Preprocessor Definitions the constant _DEBUG needs to be changed to NDEBUG for Debug configurations.

To build a project that uses the SDK, you will also need to add dns_sd.lib to your list of linker inputs, add dns_sd.h to your header files, and add the appropriate directories in the configuration under Configuration Properties/C++/General/ in field Additional Include Directories, and under Configuration Properties/Linker/General/ in field Additional Include Directories.

Overview Of the Demo Program

I've included the full source for a project that will build with Visual Studio 10, as long as you have the Bonjour SDK installed. It browses all available services on the network and displays the information about them in a tree form.

The program starts by kicking off a browser for _services._dns-sd._udp. The results are processed in member function IterateServiceTypes(). As each new service type is discovered, it is added to the tree, and a call to DNSServiceBrowse()is made to discover all instances of that service type. The callback for that browse call is member function IterateServiceInstances().

In IterateServiceInstances() I add the instance to the tree, then call DNSServiceResolve(). This function operates much like the browse function, but it actually gets the DNS record for the service. This record contains the host name, service port and a list of name/value pairs that a service can advertise as part of its record. You can see those values put to good work with service types like _ipp._tcp, in which printer parameters are exposed as part of service discovery.

ResolveInstance() is the callback routine that receives the information about the service instance. The host name, port, and name/value pairs are added to the tree, and then one final call is made to a Bonjour SDK entry called DNSServiceGetAddrInfo(). This function resolves the IP address for the given host name. The address is stuffed into the tree in callback function GetAddress().

Conclusion

DNS service discover is powerful tool, but Windows programmers might be put off by the lack of a nicely packaged SDK. Using this simple example program might be a good way to get comfortable with an SDK that gives you a powerful tool that provides a good multi-platform alternative to UPnP.

Sample program source:	ServiceBrowser.zip
Sample program executable:	ServiceBrowserExe.zip

Hash Functions for C++ Unordered Containers

Mark Nelson — Sat, 03 Sep 2011 20:00:24 +0000

The container classes included in the C++ standard library serve as good illustrations of both the strengths and the weaknesses of the language. The strengths are obvious: efficient, type-safe containers with performance guarantees suitable for a huge variety of applications. And the weaknesses? Compiler error messages that redefine the term useless, and documentation that makes a mockery of the word.

In this article I'll illustrate how you might bump into these problems using the unordered_map container, as well showing you how to work past the problems. By rights this basic hash map should be the first- or second-most used container in your arsenal, but if you are less than a C++ savant, you might find yourself ditching it out of frustration.

Hash Tables

It's a little bit of an embarrassment to the C++ community that it didn't have a hash table in the standard library until TR1 was published in 2005. In a perfect world the original standard should have contained hash map and hash set containers. But Alexander Stepanov didn't include these containers in the original Standard Template Library, and the standardization committee was reluctant to bless containers that didn't have a decent amount of mileage in the real world.

By 2005 there were enough non-standard implementations to allow the TR1 extension to confidently add four new template classes:

unordered_map
unordered_multimap
unordered_set
unordered_multiset

With basically the same semantics as their ordered counterparts (map, multimap, etc.) and the ideal O(1) performance afforded by hashed indexing, C++ finally had a feature that was basically table stakes for any high-level language created since the mid-1980s.

The container classes in general, and unordered_map in particular, are such a useful part of the language that we generally want to introduce them as early as possible to people learning C++. Admittedly, the underpinnings of template programming are out of the beginner's depth, but using the containers doesn't require much knowledge about templates beyond an understanding of a few syntax rules.

A simple piece of sample code that you might use to teach beginners how to use these containers is shown here:

//
// Compile this example with Visual Studio 2010
// or g++ 4.5 using -std=c++0x
//
#include 
#include 
#include 

using namespace std;

typedef string Name;

int main(int argc, char* argv[])
{
    unordered_map ids;
    ids["Mark Nelson"] = 40561;
    ids["Andrew Binstock"] = 40562;
    for ( auto ii = ids.begin() ; ii != ids.end() ; ii++ )
        cout << ii->first << " : " << ii->second << endl;
    return 0;
}

This example works great in the classroom or in the documentation page for unordered_map. But the new C++ programmer runs into trouble as soon as he or she steps outside the classroom and tries to implement a real-world example. A very common change to this program on its path to the real world will be in the use of a simple class or structure to hold the person's name. To keep it simple, I'll just assume we want to keep first and last names separate, and use the built-in pair class to hold the name:

//
// Compile this example with Visual Studio 2010
// or g++ 4.5 using -std=c++0x
//
#include 
#include 
#include 

using namespace std;

typedef pair Name;

int main(int argc, char* argv[])
{
    unordered_map ids;
    ids[Name("Mark", "Nelson")] = 40561;
    ids[Name("Andrew","Binstock")] = 40562;
    for ( auto ii = ids.begin() ; ii != ids.end() ; ii++ )
        cout << ii->first.first
        << " "
        << ii->first.second
        << " : "
        << ii->second
        << endl;
        return 0;
}

This seemingly small change generates seven errors in Visual C++, five in g++, and none of the errors point the user to the actual problem.

And what is the problem? It's actually a simple one: unordered_map doesn't know how to create a hash for the given key type of std::pair. Instead, the user is left to puzzle over things like this:

c:\program files\microsoft visual studio 10.0\vc\include\xfunctional(768):
 error C2440: 'type cast' :
 cannot convert from 'const Name' to 'size_t'

Or worse yet from g++:

/tmp/cc0B9FPH.o: In function `std::__detail::_Hash_code_base,
std::allocator >, std::basic_string, std::allocator > >,
std::pair, std::allocator >,
std::basic_string, std::allocator > > const, int>,
std::_Select1st, std::allocator >,
std::basic_string, std::allocator > > const, int> >,
std::equal_to, std::allocator >,
std::basic_string, std::allocator > > >,
std::hash, std::allocator >,
std::basic_string, std::allocator > > >,
std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, false>::_M_hash_code(
std::pair, std::allocator >,
std::basic_string, std::allocator > > const&) const'

The g++ compiler doesn't seem to actually produce a sentence describing the error, I think it figures no human could parse it anyway.

The inscrutability of compiler errors in template classes is hardly something new - I was complaining about it all the way back in 1995. And the C++0x committee tried to do something about it. Had the C++0x Concepts feature been accepted, the compiler might instead have issued an error message looked more like this:

hash_test.cpp(15): type 'Name' does not have a hash function

Unfortunately, Concepts did not make it, and we are stuck with error messages that are of no help at all.

RTFM

There are a couple of obvious places to try to figure out what these errors mean. Google would be one, and Visual Studio's class documentation pages would be another.

Just as an experiment, try putting yourself in the place of a novice, and execute a search on:

error C2440: 'type cast' : cannot convert from 'const Name' to 'size_t'

or show some sophistication and change your search to:

"error C2440: type cast : cannot convert from" to size_t

You will find some good clues, but probably no solutions to your problem. Much of the published code deals with pre-standard hash tables using boost implementations, or early g++ hash tables, which are not going to help.

But let's say you eventually figure out that you need to write a hash function for your Name class. All you need to know are the mechanics. Where do you find out what they are?

The logical place to do this would be in the Visual Studio unordered_map documentation page. This page has some good information in it, but nowhere does it address an undoubtedly common problem: how do I create a user-defined hash function for unordered_map?. And don't even thing about getting anything useful from the g++ documentation page.

These stumbling blocks are the kind of thing that give C++ a reputation as one of the most difficult languages to learn. To see the contrast, you might compare it to Java. The Java programmer won't ever run into this problem, because the language defines a default hash function in the base class Object. Any Java object may be used as a key in Java's HashMap generic class. While the base class definition may be far from optimal for many cases, it will at least work, and won't prevent the beginner from using the container in real programs.

C++ could have done the same thing when implementing the unordered containers, but it was constrained by both philosophy and language limitations. It would have been a real accomplishment to have overcome both, and to be honest, the people on the standards committee have to work hard enough as it is. Insisting on X-Ray vision and a cape for each member might be pushing it.

One Point of Light

So what is to be done? The language has structural problems that make it hard to issue good errors. Documentation is never as good as we like it. Are we stuck at the status quo?

While I'm not likely to change the Visual Studio documentation or the C++ compiler, I have found that one small article, like this one, that has good SEO terms to describe the problem, can help a lot of people. As an example, my next_permutation() article from 2002 still gets hundreds of readers every week, and I hope most of them walk away understanding how to use this function.

The same thing could end up being true for this article. I'll spend the rest of it showing you four good ways to define a hash function for use in unordered_map under C++0x, and with Google's help, it may end up providing the missing manual for this particular problem.

Method 1 - A simple function

You're used to seeing unordered_map declared with two template parameters. But a look at the help page shows that it actually takes five - the last three usually accept default values:

template,
    class Pred = std::equal_to,
    class Alloc = std::allocator > >
    class unordered_map;

The third parameter to the definition is a function object type that is used by the class to convert a key into a hash code. By default it is set to std::hash. Internally the unordered_map class calls operator() on an object of that type in order to get a hash code for a given key.

Note also that the several of constructors for unordered_map also take a default parameter which is an instance of this function object type.

So there are two places where we can provide some information about how to hash the key used in the unordered_map, but we normally don't fill in these items. The reason that we often don't is that the C++ standard library already defines instances of std::hash for commonly used types. So I can write a program that contains a line like this:

int main(int argc, char* argv[])
{
    cout << std::hash()("Mark Nelson") << endl;
    return 0;
}

Which when run, produces:

134514544

The problem, of course, is that the standard library did not implement a version of hash> - or any of the other infinite varieties of user-defined classes. However, since my new Name class is composed of two string objects, and the standard library knows how to hash string objects, I can create a pretty good hash function of my own that looks like this:

size_t name_hash( const Name & name )
{
    return hash()(name.first) ^ hash()(name.second);
}

As a general rule of thumb, if I have two hashes for independent variables, and I combine them using XOR, I can expect that the resulting hash is probably just as good as the input hashes.

Building this function is easy enough, but how do I actually use it with unordered_map? Although the solution is fairly simple, if you find templates confusing already, you aren't liable fumble your way through to the answer without an enormous amount of effort.

Basically, I have to modify my use of the map class in two places. First, I have to pass in a pointer to the hash function in the constructor of the map. The standard defines a constructor that takes an initial number of buckets and a hashing object as inputs. So the first step is to modify the declaration code to look like this:

unordered_map ids(100, name_hash );

This is only half the battle, however. The default implementation of unordered_map expects to be using a function object of type std::hash to calculate hashes, and that is not what I passed in to the constructor. So I also have to add a third template parameter to my declaration - a template parameter which matches the type of the function object I am passing in to the constructor.

Creating the proper template parameter to match this simple hashing function requires more than your usual level of library-fu. One way I've done this in the past is to wrap the function type inside the std::function template, which is defined in header . When you do this, your map will have a hasher object that is an instance of function. Upon initialization of the map, the function object will be assigned a pointer to name_hash, which can then be called via the interface to function.

The resulting declaration will then look like this:

unordered_map> ids(100, name_hash );

C++0x gives me a slightly easier way to do this, and the code below shows this simpler alternative. By using the decltype keyword, I can take the type of my hash function and pass it is as a template parameter. Not only is the code simpler this way, but I avoid defining one thing in two different places.

//
// This program uses a simple user-defined function
// to provide a hash function for use in unordered_map
//
// Compile this example with Visual Studio 2010
// or g++ 4.5 using -std=c++0x
//
#include 
#include 
#include 
#include 

using namespace std;

typedef pair Name;

size_t name_hash( const Name & name )
{
    return hash()(name.first) ^ hash()(name.second);
}

int main(int argc, char* argv[])
{
	unordered_map ids(100, name_hash );
	ids[Name("Mark", "Nelson")] = 40561;
	ids[Name("Andrew","Binstock")] = 40562;
	for ( auto ii = ids.begin() ; ii != ids.end() ; ii++ )
		cout << ii->first.first
                     << " "
                     << ii->first.second
                     << " : "
                     << ii->second
                     << endl;
	return 0;
}

Method 2 - A simple function defined inline

In some cases where your hash function is short and sweet, you might want to take advantage of the new C++0x support for lambda expressions. In this particular case, using a lambda to define your hash function lets you define the hasher right where you use it - which may or may not provide clarity for the program. In this particular case, the finished result looks like this:

//
// This program uses a simple user-defined function
// to provide a hash function for use in unordered_map.
// The function is passed in as a lambda expression to
// the unordered_map constructor.
//
// Compile this example with Visual Studio 2010
// or g++ 4.5 using -std=c++0x
//
#include 
#include 
#include 
#include 

using namespace std;

typedef pair Name;

int main(int argc, char* argv[])
{
    unordered_map>
    ids(100, []( const Name & name )
             {
                 return hash()(name.first) ^ hash()(name.second);
             } );
    ids[Name("Mark", "Nelson")] = 40561;
    ids[Name("Andrew","Binstock")] = 40562;
    for ( auto ii = ids.begin() ; ii != ids.end() ; ii++ )
        cout << ii->first.first
             << " "
             << ii->first.second
             << " : "
             << ii->second
             << endl;
    return 0;
}

As far as the compiler is concerned, this program will probably generate nearly identical code to the previous one, so it is unlikely that you should prefer one over the other for reasons of efficiency.

I don't use the lambda expression method for two reasons:

When using the lambda expression, I can't use decltype in the template class definition to get the type of the hash function object. This means I have to manually enter it, which means I'm manually synching one definition between two places in my code - always an opportunity for trouble.
As this is being written in 2011, lambda expressions are still unfamiliar to people, and aren't supported in a lot of compilers currently in use on production systems, so I save their use for places where they provide a marked improvement in either program structure or readability.

Your opinions may well differ.

Method 3 - A function object

Function objects are a way to package up functions so they can be called in a way that is often convenient to a library writer. So even though you might not have ever dreamed up the idea of creating a function object to provide a hash function to the library, this is the way unordered_map prefers things. My definition of a function object to use with this definition of Name is shown here:

struct hash_name {
    size_t operator()(const Name &name ) const
    {
        return hash()(name.first) ^ hash()(name.second);
    }
};

All I have done here is wrap up the hash function a class. And while that might not seem like much of a big deal, it allows a class to use this function when all it has is the class definition. When I define unordered_map using this function object, my code is a bit simpler:

unordered_map ids;

You can see that I still have to include a third template class parameter, but I don't have to pass in a reference to the function object in the constructor. This is because unordered_map keeps track of the class definition, and when it comes time to actually get a hash, it can simply construct the object on the fly and pass in the data. A sample (hypothetical) class that does this might have code like this:

template 
class HashMap {
...
void insert( HashKey &key, HashVal &val )
{
    size_t index = HashObject()( key );
...

Putting together for the sample program I've been using throughout, you get this code:

//
// This program uses a function object to define
// a hash function for use in unordered_map.
//
// Compile this example with Visual Studio 2010
// or g++ 4.5 using -std=c++0x
//
#include 
#include 
#include 
#include 

using namespace std;

typedef pair Name;

struct hash_name {
    size_t operator()(const Name &name ) const
    {
        return hash()(name.first) ^ hash()(name.second);
    }
};

int main(int argc, char* argv[])
{
    unordered_map ids;
    ids[Name("Mark", "Nelson")] = 40561;
    ids[Name("Andrew","Binstock")] = 40562;
    for ( auto ii = ids.begin() ; ii != ids.end() ; ii++ )
        cout << ii->first.first
             << " "
             << ii->first.second
             << " : "
             << ii->second
             << endl;
    return 0;
}

This method clearly has some nice features. In return for having to package your hash function inside a structure definition (and use the unfamiliar operator() ), you get the advantage of having an unordered_map declaration that is considerably simpler than the ones used in the previous two examples.

In terms of the generated code, this example is again probably going to generate nearly identical code to the previous examples, meaning that your preference towards using it should be based strictly on ease of coding, readability, and maintenance issues.

Method 4 - Specializing std::hash

When you use unordered_map with all the default class parameters, it tries to use a function object of type std::hash to create your hash keys. As discussed way back at the start of this program, this doesn't work in many cases because nobody bothered to create a specialization of the hash object for your specific class. In many cases, this is because your specific class had not been invented yet.

It stands to reason that if you are providing a class that will be used by other people, it might be nice to actually create the instances of the hash class for your class. When you do that, and include the definition in the header file that defines your class, people will be able to use your class as a key for any of the unordered containers without any additional work.

Defining a specialization of std::hash for your class is really no different than the method 3, shown immediately above. The only difference is that you have to use the name hash for your object, and you have to define it as a specialization of that template class, and finally, you have to hoist the whole thing into the std namespace.

//
// This program uses a specialization of
// std::hash to provide the function
// object needed by unordered_map.
//
// Compile this example with Visual Studio 2010
// or g++ 4.5 using -std=c++0x
//
#include 
#include 
#include 
#include 

using namespace std;

typedef pair Name;

namespace std {
    template <>
        class hash{
        public :
            size_t operator()(const Name &name ) const
            {
                return hash()(name.first) ^ hash()(name.second);
            }
    };
};

int main(int argc, char* argv[])
{
    unordered_map ids;
    ids[Name("Mark", "Nelson")] = 40561;
    ids[Name("Andrew","Binstock")] = 40562;
    for ( auto ii = ids.begin() ; ii != ids.end() ; ii++ )
        cout << ii->first.first
             << " "
             << ii->first.second
             << " : "
             << ii->second
             << endl;
	return 0;
}

As you can see, for the user of your class, this is the simplest of all possible solutions - no changes are needed to get the default versions of unordered_map to work properly. If you are going to be using this class in multiple places, this is probably the best solution. However, you need to be sure you aren't going to pollute the std namespace with a hash function that might conflict with those used by other classes.

One Last Source of Trouble

In addition to requiring a hash function, the unordered containers also need to be able to test two keys for equality. The canonical way for them to do this is with a version of operator==() defined at the global namespace. This is typically a function you are used to having to construct when creating new classes, but if you overlook it, you will be up against the same raft of incomprehensible compiler errors seen earlier in this article.

I didn't have to deal with it in this article because the standard library already defines this operator for std::pair. Of course, when using std::pair you also have to make sure you have an equality operator for T1 and T2.

Wrap-Up

This article showed you four different ways to define a hash function for use with a user-defined class and the unordered C++0x containers. Different people will choose one of the four methods for personal reasons. When it comes to performance, there is probably no reason to prefer one or the other.

None of the techniques described here are particularly difficult to implement, but it is always a surprise and a disappointment to see how hard it is to get this information out of the standard resources.

Mark Nelson » C/C++

C++11: Range-based for and auto

The value of containers

Anagramania

The auto Type Specification

The Range-based for Statement

Putting it to Use

The Final Product

Visual Studio 11 and Modern C++

Microsoft and C++ – We Have History

Modern C++

A Simple Example Using Naive C++98

Classes Good, Templates Better

Using the Algorithms Library

Functors Not So Hot

Lambda to the Rescue

Using Lambdas

C++ – Where’s the Hate?

The Epitome of C++ Hate

The Slashdot Crowd

Why the Dropoff?

Final Analysis

Streambuf Iterators Are a Big Help

The Problem

Streams or Iterators?

The Conflict

My Choice

Automating Putty

Putting Together the Project

Adding Putty to the Project

Launching AutoPutty

Taking Ownership of AutoPutty

The Communications Link

Handling the AutoPutty Lifecycle Events

Monitoring Input Traffic

Receiving the Data

Driving PuTTY

Source Code

Downloads

VC++ 10 Hash Table Performance Problems

Background

The First Results

The Pathological Results

A Workaround

Source

LZW Revisited

LZW Basics

Encoding/Decoding

The LZW Encoder

The LZW Decoder

The Catch

Implementation

Input and Output

LZW-A

LZW-B

LZW-C

LZW-D

The Windows Test Program

The Linux Test Program

Your Program

Benchmarks

Variations

History

Downloads

DNS Service Discovery On Windows

DNS-SD On Windows

An Emphasis on Kit

An Overview

An Example - Discover Service Types

Driving the Callbacks

Threading Issues

Character Sets

Library Issues

Overview Of the Demo Program

Conclusion

Hash Functions for C++ Unordered Containers

Hash Tables

RTFM

One Point of Light

Method 1 - A simple function