It looks like all my long years of studying data compression might be ready to pay off:

Hello Good Day,

This is Troop Emonds With regards to your Company i am sending this email Regards to order some( Compression Machine )I will like to know the type and sizes you have in stock and get me the sales price of one so that i will tell you the quantity i will be ordering, and if you accept credit card as a form of payment..

Hope to read from you soon about my order request……
With Kind Regards.
Troop

I just need to put together some compression machines, and then I’m set.

In my early years of learning about data compression, the book Text Compression by Timothy Bell, John Cleary, and Ian Witten was my resource of first resort. I was in Christchurch, New Zealand, recently and had a chance to meet Tim for the first time in person. Tim teaches at the University of Canterbury in Christchurch, and is Deputy Head of the Computer Science and Software Engineering department. I got a chance to ask him about his work in data compression as well as one of his new areas of interest, Computer Science education.

TB: I’ve sometimes wonder about this myself… during the early days of computing and especially personal computers, it took some time for the latest technology to reach us “down under”, so perhaps we were motivated to get more out of what we had rather than wait some months for a larger disk or new memory to arrive from overseas. When the Internet arrived we started with a very small pipe, so a good compression algorithm could do the equivalent to laying a second cable from NZ to the US – who can resist getting something for free?

MN: Since you wrote Text Compression back in the early 90s, I’d say the biggest development in lossless compression has been the Burrows-Wheeler transform. Is lossless text compression basically done? Are we left with just incremental improvements as processor resources increase?

TB: That seems to be the case; the only big improvements we’ve seen have turned out to be frauds — we even had one in NZ recently, where a Nelson man raised NZ$5.3 million for an impressive sounding method; he was convicted of fraud last year. The main indicator we have that we’re running out of steam (apart from a lack of new discoveries) is Shannon’s experiments on predicting text which gave a bound in the order of 1 bit per character for English text, and current methods are approaching this. Of course, there’s plenty of room for dealing with new kinds of data (for example, bioinformatics deals with massive amounts of data that we’re still trying to understand) and for finding better data structures and algorithms for performing the compression and decompression. Lossy compression is a whole different story…

A Change In Focus

MN: It looks like you are now dedicating a large amount of your time to establishing computer science as part of the basic curriculum in high school education, for students in the 15-18 age range. In many ways, this is as much a bureaucratic problem as an academic one. What motivated you to take it on?

TB: It’s been a problem that we’ve complained about for decades, and it’s been getting worse and worse as computing in schools has focussed increasingly on using computers and not preparing students to be developers. A lot of this can be attributed to bureaucracy – it’s hard to explain to government officials that putting word processors in every classroom isn’t the same as building a computationally literate society. As a result of some strategic lobbying done by others, a small window of opportunity opened for me to be on a group to advise our Ministry of Education, just over 3 years ago. The group managed to convince the officials that something useful could be done, and then we had to work very quickly to come up with a concrete proposal before the enthusiasm died down. This has happened rapidly; the advisory group first met in November 2008, and Computer Science started being taught in schools in February 2011.

MN: What have you been able to accomplish in New Zealand so far?

TB: Computer science (including programming, but also topics the involve understanding the importance of things like algorithms, HCI, programming languages and even compression) is currently available as part of computing courses for two of the three final years of our main high school graduation qualification, with all three years being covered from 2013. After that we would expect some of the introductory material to start filtering down to earlier classes, and for wider offerings as teachers become more confident in the subject. One of the biggest challenges has been preparing teachers, few of whom have significant experience in Computer Science. Many have embraced it enthusiastically, and the universities and others have done a lot of work to help them get up to speed. It’s been a wild ride doing it so quickly, but there have been some very pleasing outcomes.

MN: And how do things look in the rest of the world? Are there any obvious winners and losers at this point? Do you have any concise advice for the world?

TB: Computing in schools is a hot topic around the world; the UK have just announced a strong drive to introduce this sort of material to schools, and the US has people working hard to make it available to students. Israel and Korea have had computer science in schools for some time. We’re learning a lot about what is worth teaching, and what the best pedagogy is for the general classroom (most of our experience is for specialist students who have chosen the subject!) The New Zealand path of getting something going quickly with grass-roots support seems to be more effective than waiting for a top-down approach which could take years to develop and prepare teachers for, although it does make for a bumpy ride as problems are ironed out as we go along!

MN: This might be straying out of your area a bit, but do you see CS in a K-12 education setting having an effect on the representation of women in the STEM fields?

TB: Attitudes that affect representation definitely start at school, and to me the biggest goal of teaching CS in high school is not so much to prepare students for further study, but to enable them to find out what the subject is! School students rarely know what CS is, and even worse, it’s common for them to assume that it must be advanced word processing or some other dull area, and hence they avoid it. It’s particularly important for female students to have the opportunity to find out if it’s something that they might be good at, as the stereotypes associated with computing can make them assume that they shouldn’t consider it as a career.

MN: One final question, Tim. The whole world has seen the devastating damage Christchurch has suffered from the earthquakes in the last year. How has the University of Canterbury held up? Have you managed to maintain continuity in your academic calendar?

TB: It’s been quite a year! Thankfully our university has escaped the brunt of the earthquakes (most of the damage is some distance from the university), and we’ve managed to keep a full programme going despite being closed for three weeks for safety checks. Many students joined the “student volunteer army”, who helped with the cleanup in the damaged parts of town, and that was probably one of the most valuable experiences of their career! It hasn’t been without disruption as buildings need to be checked carefully, and some are still under repair, but with a bit of resourcefulness we managed to keep going (for a while I even delivered my classes in a restaurant while lecture theatres were being inspected) The city is now going through a massive program of redevelopment with some pretty creative ideas, and it’s an exciting time to be part of these changes.

Thanks to Dr. Bell for taking the time to share all this with us. My visit to his amazing homeland was a real treat, and the short time I got to spend with Tim in Christchurch was worth the trip all in itself.

When I updated my LZW reference code to use the latest C++ features, I abstracted my input and output functions using templates. Data was read and written using the iostreams paradigm, which requires simple classes that implement just a few functions. Would I have been better off using the iterator paradigm instead? The C++ algorithms library favors that method of processing data, and it can be both elegant and powerful. Which of the two paradigms is the right one for data compression?

The Conflict

General purpose data compression routines tend to be used on binary streams of data, either from files or in-memory objects. So what is the best general paradigm for input and output when compressing data?

You might analyze this problem by imagining that you need to write a binary copy routine.

template<class INPUT_ITERATOR, class OUTPUT_ITERATOR>
void bcopy( INPUT_ITERATOR input, INPUT_ITERATOR eof, OUTPUT_ITERATOR output )
{
    while ( input != eof )
        *output++ = *input++;
}

This routine is particularly nice when you are performing a simple copy using pointers to memory – the generated code should be really efficient.

However, the iterator paradigm doesn’t work quite as well when you want to perform a binary copy of data in a file. I can make use of iterators that almost do the job:

 std::ifstream in( "input.txt", std::ios_base::binary );
 std::ofstream out("output.txt", std::ios_base::binary );
 bcopy( std::istream_iterator(in),
        std::istream_iterator(),
	std::ostream_iterator(out) );

But the bad news is that both istream_iterator and ostream_iterator use the insertion and extraction operators, which are really meant for whitespace-delimited textual data, not binary data. The copy routine shown here will not make a binary byte-for-byte copy of the input file.

So when using files, the stream approach seems to be the way to go:

template<class INPUT_STREAM, class OUTPUT_STREAM>
void bcopy( INPUT_STREAM in, OUTPUT_STREAM out )
{
    char c;
    while ( in.get(c) )
        out.put(c);
}

If my files have been opened using the iostream classes, you can use this binary copy function without having to write any glue code – they already support the get and put methods, so this works right out of the box.

My Choice

If I’ve made up my mind that my data compression routine is going to use one of these two paradigms, it means I am going to have to write some glue code. If I choose the iterator-based approach, I need the equivalent of istream_iterator and ostream_iterator for binary files – and these aren’t in the standard library. If I choose the stream-based approach, I need efficient put() and get() members for blocks of memory. In some cases basic_stringstream might do the job, but not in all cases.

After dithering around with various solutions, I tentatively opted for the stream paradigm. I found the implementation for various sources of data to be fairly simple, and the interface is easy to understand. I don’t know if it’s the perfect choice, and I’ll keep experimenting, but for now it works for me. My abstraction of the LZW code still needs a lot of work, so it’s always possible I could rethink this at a later date.

I’d like to hear your thoughts – is there an obvious right answer to this question?

One of the first articles I wrote for Dr. Dobb’s Journal, LZW Data Compression, turned out to be very popular, and still generates a fair amount of traffic and email over twenty years later.

One of the reasons for its popularity seems to be that LZW compression is a popular homework assignment for CS students around the world. And that audience sometimes found the article to be bit of a struggle. My code was modeled on the UNIX compress program, which was written in terse C for maximum efficiency. And sometimes optimization comes at the expense of comprehension.

By using C++ data structures I can model the algorithm in a much more straightforward way – the language doesn’t get in the way of a clear implementation. And after 20 years of answering puzzled queries I think I can improve on the overall explanation of just how LZW works.

In this updated look at LZW, I will first give a description of how LZW works, then describe the core C++ code that I use to implement the algorithm. I’ll then walk you through the use of the algorithm with a few varieties of I/O. Finally, I’ll show you some benchmarks.

I’m hoping that this version of the article will be good enough to last for another 20 years.

LZW Basics

LZW compression works by reading a sequence of symbols, grouping the symbols into strings, and converting the strings into codes. Because the codes take up less space than the strings they replace, we get compression.

My implementation of LZW uses the C++ char as its symbol type, the C++ std::string as its string type, and unsigned int as its code type. The tables of codes and strings are implemented using unordered_map, the C++ library’s hash table data structure. By using the native types and standard library data structures the representation in the program is straightforward and easy to follow.

Encoding/Decoding

Rather than jumping directly into a full implementation, I’m going to work my way up to LZW one step at a time.

The first step is getting a clear understanding of how the encoding and decoding process works. As I said earlier, LZW compression converts strings of symbols into integer codes. Decompression converts codes back into strings, returning the same text that we started with.

LZW is a greedy algorithm – it tries to find the longest possible string that it has a code for, then outputs that string. The code below is not quite LZW, but it shows you the basic idea of how a greedy encoder can work:

void encode( input_stream in, output_stream out )
{
  //
  // This hash table contains a list of codes, indexed
  // by the string that corresponds to the code.
  //
  std::unordered_map<std::string,unsigned int> codes;
  //
  // There is presumably some code here that initializes
  // the dictionary with a set of codes based on whatever
  // algorithm we are implementing.
  //
  ...initialize the dictionary
  //
  // With codes in the dictionary, encoding is
  // now ready to begin.
  //
  std::string current_string;
  char c;
  while ( in >> c ) {
    current_string = current_string + c;
    if ( codes.find(current_string) == codes.end() ) {
      current_string.erase(current_string.size()-1);
      out << codes[current_string];
      current_string = c;
    }
  }
  out << codes[current_string];
}

The greedy encoder reads characters in from the uncompressed stream, and appends them one by one to the variable current_string. Each time it lengthens the string by one character, it checks to see if it still has a valid code for that string in the dictionary.

This continues until we eventually add a character that forms a string that isn’t in the dictionary. So we then erase the last character from that string, and issue the code for the resulting string – the string from the previous pass through the loop.

The value of current_string is then initialized with the character that broke the camel’s back, and the algorithm continues in the loop, building new strings until it runs out of input characters. At that point it outputs the last remaining code and exits.

As an example of how this would work, imagine I have the input stream ACABCA, and my code dictionary looks like this:

Decoding the stream in a system like this is very straightforward:

void decode( input_stream in, output_stream out )
{
  std::unordered_map<unsigned int,std::string> strings;
  //
  // Initialize the code table with the same set of codes and strings
  // that the encoder used for your algorithm.
  //
  ...initialize the dictionary
  //
  // With codes in the dictionary, decoding is now
  // ready to begin.
  //
  unsigned int code;
  while ( in >> code )
    out << strings[code];
}

Remember, the decoder shown above is just a hypothetical sample - we're still working our way up to the full LZW decoder.

The LZW Encoder

The encoder shown above works okay, but there is one missing ingredient: management of the code dictionary. If you think about it, you'll see that we only achieve reasonable compression when we are able to build up longer strings and find them in the dictionary. Building a useful dictionary is referred to in the data compression world as modeling.

But our management of the dictionary is constrained by an important requirement: the encoder and decoder both have to be working with the same copy of the dictionary. If they have different dictionaries, the encoder might send a string that the decoder can't resolve.

Some data compression algorithms solve this problem by using a predefined dictionary that both the encoder and the decoder know in advance. But LZW builds a dictionary on the fly, using an adaptive method that ensures both the encoder and decoder are in sync.

LZW manages this in an effective and provably correct fashion. First, both the encoder and decoder initialize the dictionary with all possible single digit strings. For the compressor, that looks like this:

for ( unsigned int i = 0 ; i < 256 ; i++ )
    codes[std::string(1,(char)i)] = i;

This insures that we can encode all possible streams. No matter what, we can always break a stream down into single digits and encode these, knowing that the decoder has the same strings in its dictionary with values 0-255.

Then comes the key component of the LZW algorithm. If you go back to the greedy encoding loop above, you'll see that I keep adding input symbols to a string until I find a string that isn't in the dictionary. This string has the characteristic of being composed of a string that currently exists in the dictionary, with one additional character.

LZW then takes that new string and adds it to the dictionary, creating a new code. The strings are added to the table with code values that increment by one with each new entry.

The resulting code is just a slightly modified version of the encoder that I listed above. It still only outputs codes for values that are in the dictionary, but now the dictionary is being updated with a new string every time an existing code is sent:

void compress( input_stream in, output_stream out )
{
  std::unordered_map<std::string,unsigned int> codes;
  for ( unsigned int i = 0 ; i < 256 ; i++ )
    codes[std::string(1,(char)i)] = i;
  unsigned int next_code = 257;
  std::string current_string;
  char c;
  while ( in >> c ) {
    current_string = current_string + c;
    if ( codes.find(current_string) == codes.end() ) {
      codes[ current_string ] = next_code++;
      current_string.erase(current_string.size()-1);
      out << codes[current_string];
      current_string = c;
    }
  }
  out << codes[current_string];
}

The code above constitutes a more or less complete LZW encoder. I've only made a couple of additions to the previous encoder:

• The initialization of codes 0-255 with all possible single character strings.
• The insertion of the newly discovered string into the string table, generating a new code.

(One item of note in this code: you might wonder why next_code is initialized to 257, when 256 is the first free code. This is because I reserve code 256 for an EOF marker. More on this in a later section.)

Just to make sure this all adds up, I'll walk through the steps the encoder takes as it processes a string from a simple two letter alphabet: ABBABBBABBA. There are a lot of steps shown below, but working through the process in detail is a great way to be sure you understand it:

More importantly, as the dictionary grows, it starts to hold longer and longer strings. And the longer the string, the the more compression we can get. If the algorithm starts emitting integer codes for strings of length 10 or more, there is no doubt that we are going to get good compression.

As an example of how this works on real data, here are some entries from the dictionary created when compressing Alice's Adventures in Wonderland:

34830 : 'even\n'
34831 : '\nwith t'
34832 : 'the dr'
34833 : 'ream '
34834 : ' of Wo'
34835 : 'onderl'
34836 : 'land'
34837 : 'd of l'
34838 : 'long ag'
34839 : 'go:'

These strings have an average length of almost six characters. If we are writing the integer codes to a file using 16 bit binary integers, these entries offer the possibility of 3:1 compression.

The word adaptive is used to describe a compression algorithm that adapts to the type of text it is processing. LZW does an excellent job of this. If a string is seen repeatedly in the text, it will show up in longer and longer entries in the dictionary. If a string is seen rarely, it will not be the foundation for a large batch of longer strings, and thus won't waste space in the dictionary.

The LZW Decoder

The change made to the basic encoder to accommodate the LZW algorithm was really very simple. One small batch of code that initializes the dictionary, and another few lines of code to add every new unseen string to the dictionary.

As you might suspect, the changes to the decoder will be fairly simple as well. The first change is that the dictionary must be initialized with the same 256 single-symbol strings that the encoder uses.

Once the decoder starts running, each time it reads in a code, it must add a new value to the dictionary. And what is that value? The entire content of the previously decoded string, plus the first letter of the currently decoded string. This is exactly what the encoder does to create a new string, and the decoder must following the same steps:

void decompress( input_stream in, output_stream out )
{
  std::unordered_map<unsigned int,std::string> strings;
  for ( int unsigned i = 0 ; i < 256 ; i++ )
    strings[i] = std::string(1,i);
  std::string previous_string;
  unsigned int code;
  unsigned int next_code = 257;
  while ( in >> code ) {
    out << strings[code];
    if ( previous_string.size() )
      strings[next_code++] = previous_string + strings[code][0];
    previous_string = strings[code];
  }
}

I won't do a walk-through of the the decoder - you should be able to take the codes output from the encoder, shown above, and run them through the decoder to see that the output stream is what we expect.

The important thing is to understand the logic behind the decoder. When the encoder encounters a string that isn't in the dictionary, it breaks it into two pieces: a root string and an appended character. It outputs the code for the root string, and adds the root string + appended character to the dictionary. It then starts building a new string that starts with the appended character.

So every time the decoder uses a code to extract a string from the dictionary, it knows that the first character in that string was the appended character of the string just added to the dictionary by the encoder. And the root of the string added to the dictionary? That was the previously decoded string. This line of code implements that logic:

    strings[next_code++] = previous_string + strings[code][0];

It adds a new string to the dictionary, composed of the previously seen string, and the first character of the current string. Thus, the decoder is adding strings to the dictionary just one step behind the encoder.

You might note one curious point in the decoder. Instead of always adding the string to the dictionary, it is only done conditionally:

if ( previous_string.size() )
  strings[next_code++] = previous_string + strings[code][0];

The only time that previous_string.size() is 0 is on the very first pass through the loop. And on the first pass through the loop, we don't have a previous string yet, so the decoder can't build a new dictionary entry. Again, the decoder is always one step behind the encoder, which is a key point in the next section, which puts the final touches on the algorithm.

The Catch

So far the LZW algorithm we've seen seems very elegant - that's a characteristic we associate with algorithms that can be expressed in just a few lines of code.

Unfortunately, there is one small catch in this perceived elegance - the algorithm as I've shown it to you has a bug.

The bug in the algorithm relates to the fact that the encoder is always one step ahead of the decoder. When the encoder adds a string with code N to the table, it sends enough information to the decoder to allow the decoder to figure out the value of the string denoted by code N-1. The decoder won't know what the value of the string corresponding to code N is until it receives code N+1.

This makes sense if you recall the key line of code from the decoder. It calculates the value of the string encoded by N-1 by looking at the string it received on the previous iteration, plus the first character of the current string. And that current string is the one that was sent after encoding N.

So how can this get us in trouble? The encoder is always one entry ahead of the decoder - it has entry N in its dictionary, and the decoder has entry N-1. So if the encoder ever sends code N, the decoder will look in its table and come up empty-handed, unable to do its job of decoding.

A simple example will show you how this can happen. Let's look at the state of the encoder after it has sent the first five symbols in a stream: ABABA:

Yes, if the next two characters in the stream are BA, the next code output by the encoder will be 259, and the decoder will be lost.

In general, this can happen when a dictionary entry exists that consists of a string plus a character, and the encoder encounters the sequence string+character+string+character+string. In the example above, the value of string is A, and the value of character is B. After the encoder counters AB, it has string+character in the dictionary, so if the following sequence is ABABA, we will emit code N.

Whether this is likely to happen or not is not too important, what is important is that it most definitely can happen, and the decoder has to be aware of it. And it will happen repeatedly in the pathological case: a stream that consists of a single symbol, repeated on end.

The good news is that the problem is easily solved. When the decoder receives a code, and finds that this code is not present in its dictionary, it knows right away that the code must be the one that it will add next to its decoder. And because this only happens when we are encoding the sequence discussed above, the decoder knows that instead of using this value for that code:

    strings[next_code++] = previous_string + strings[code][0];

it can instead use this value:

    strings[ code ] = previous_string + previous_string[0];

The result of this is the insertion of just two lines of code at the start of the decompress loop, giving a loop that now looks like this:

while ( in >> code ) {
  if ( strings.find( code ) == strings.end() )
    strings[ code ] = previous_string + previous_string[0];
  out << strings[code];
  if ( previous_string.size() )
    strings[next_code++] = previous_string + strings[code][0];
  previous_string = strings[code];
}

And with that, you have a complete implementation of the LZW encoder and decoder.

Implementation

Now that I've shown you the algorithm, the next step is to take that code and add turn it into a working program. Without changing the algorithm itself, I'm going to take you through four different customizations that work as follows:

• LZW-A reads and writes code values rendered in text mode, which is great for debugging. It means you can view the output of the encoder in a text editor.
• LZW-B reads and writes code values as 16-bit binary integers. This is fast and efficient, and usually results in significant data compresion.
• LZW-C reads and writes code values as N-bit binary integers, where N is determined by the maximum code size. Performing I/O on codes that are not aligned on byte boundaries complicates the code somewhat, but allows for greater efficiency and better compression.
• LZW-D reads and writes code values as variable-length binary integers, starting with 9-bit codes and gradually increasing as the dictionary grows. This gives the maximum compression.

Before launching into these implementations, the code I showed above needs some minor tweaking to solve a couple of problems.

The first problem we have to deal with is the ever-expanding dictionary. In the algorithm I've presented, we keep adding new codes to the dictionary without end. This needs to be changed for a couple of reasons.

First, we don't have unlimited memory, so the dictionary simply can't grow forever. Second, practical experience shows that compression ratios don't improve as dictionary sizes grow without bound. As the dictionary grows, code sizes get larger and larger, and so they take up more space in the compressed stream, which can reduce compression efficiency.

To resolve this problem, I just add an additional argument to the encoder and decoder that sets the maximum code value that will be added to the dictionary. The function signatures now look like this:

void compress( input_string input,
               output_stream output,
               const unsigned int max_code = 32767 );
void decompress( input_string input,
                 output_stream output,
                 const unsigned int max_code = 32767 );

Implementing it means one small change in the encoder:

if ( next_code <= max_code )
  codes[ current_string ] = next_code++;

And a corresponding change in the decoder:

if ( previous_string.size() && next_code <= max_code )
  codes[ current_string ] = next_code++;

Input and Output

Finally, I need to give the algorithm a decent way to perform input and output - and this is where C++ offers a huge amount of help.

When writing generic compression code that you intend to use in multiple contexts, one of the more difficult things to deal with is I/O. People using your code might want to compress data in memory, stored in files, or streaming in from sockets or other sources. Some input data sources might be of unknown length (data coming from a TCP socket, for example), while others will be of a prescribed length. Back in the days of C, it was particularly difficult to make your compression code both generic, so it would work with all types of data streams, and efficient, so that I/O doesn't take any more time than it has to.

With the advent of C++, we have a new tool that can help in this quest - templates. Templates are designed to solve this problem in an efficient way, and I take advantage of this in my sample code. The code below shows the final version of the compressor and decompressor that are are used in all four versions of the implementation. There are two final changes made to the routines shown previously. First, both C++ functions are now function templates, parameterized on the the types being used for input and output. Second, the actual input and output is done through four newly introduced template classes:

template<class INPUT, class OUTPUT>
void compress( INPUT &input, OUTPUT &output, const unsigned int max_code = 32767 )
{
  input_symbol_stream<INPUT> in( input );
  output_code_stream<OUTPUT> out( output, max_code );

  std::unordered_map<std::string, unsigned int> codes( (max_code * 11)/10 );
  for ( unsigned int i = 0 ; i < 256 ; i++ )
    codes[std::string(1,i)] = i;
  unsigned int next_code = 257;
  std::string current_string;
  char c;
  while ( in >> c ) {
    current_string = current_string + c;
    if ( codes.find(current_string) == codes.end() ) {
      if ( next_code <= max_code )
        codes[ current_string ] = next_code++;
      current_string.erase(current_string.size()-1);
      out << codes[current_string];
      current_string = c;
    }
  }
  if ( current_string.size() )
    out << codes[current_string];
}

template<class INPUT, class OUTPUT>
void decompress( INPUT &input, OUTPUT &output, const unsigned int max_code = 32767  )
{
  input_code_stream<INPUT> in( input, max_code );
  output_symbol_stream<OUTPUT> out( output );

  std::unordered_map<unsigned int,std::string> strings( (max_code * 11) / 10 );
  for ( int unsigned i = 0 ; i < 256 ; i++ )
    strings[i] = std::string(1,i);
  std::string previous_string;
  unsigned int code;
  unsigned int next_code = 257;
  while ( in >> code ) {
    if ( strings.find( code ) == strings.end() )
      strings[ code ] = previous_string + previous_string[0];
    out << strings[code];
    if ( previous_string.size() && next_code <= max_code )
      strings[next_code++] = previous_string + strings[code][0];
    previous_string = strings[code];
  }
}

What exactly is the effect of implementing this algorithm using a pair of function templates, parameterized on the the types of the input and output objects? What this means is that you can call these compression routines with any type of I/O object you can throw at them. It can work with C++ iostreams, C FILE * objects, raw blocks of memory, whatever you want.

But there's a catch to that flexibility - you have to implement some basic I/O routines for whatever type you are using. Fortunately, this is not too hard.

The actual I/O that is done in the compression routines is defined by four template classes I created. These classes are defined in lzw_streambase.h. These classes don't have implementations, but they do define the methods you need to implement to work with the compressor and decompressor. The four classes are:

• input_symbol_stream<T>
• ouput_symbol_stream<T>
• input_code_stream<T>
• output_code_stream<T>

The first two classes are the symbol input and output classes. These are normally going to be very simple implementations, as they just have to read single characters to and from streams, while checking for errors or ends of streams. I use the same versions of these classes in all four implementations, so the code in lzw-a.h is unchanged in the other three header files.

The input_symbol_stream<T> class has one member function: the extraction operator, which reads a character from the stream and returns a boolean true or false. You'll see later in this section that the implementation of this for types such as std::istream is trivial.

template<typename T>
class input_symbol_stream
{
public :
    input_symbol_stream( T & );
    bool operator>>( char &c );
};

The output_symbol_stream<T> class uses the insertion operator to write strings instead of individual characters - because that is what is stored in the dictionary. The C++ std::string class makes a perfectly good container for any variety of symbols, including binary data, and unlike the alternative vector<char>, it comes with hash functions and iostream operators.

template<typename T>
class output_symbol_stream
{
public :
    output_symbol_stream( T &  );
    void operator<<( const std::string &s );
};

The input_code_stream<T> class reads codes, normally unsigned integers, from some type of stream. In my implementations, this class also returns false if it encounters the EOF_CODE in the stream of incoming codes. Removing the responsibility for EOF detection from the decompressor makes the code a bit simpler and more versatile.

The formatting of the integer is entirely up to the implementor, but the most common approach will probably be variable length codes ranging from 9 to 16 or so bits.

template<typename T>
class input_code_stream
{
public :
    input_code_stream( T &, unsigned int );
    bool operator>>( unsigned int &i );
};

The output_code_stream<T> class writes codes, usually unsigned integers, to some type of stream. Whatever class you implement for this function must agree with the implementation for input_code_stream<T>.

template<typename T>
class output_code_stream
{
public :
    output_code_stream( T &, unsigned int );
    void operator<<( const unsigned int i );
};

You can see that at the top of the compressor and decompressor, I instantiate objects of these types, then use the standard insertion and extraction operators to read and write from these objects.

LZW-A

In my sample windows program, I include lzw_streambase.h and lzw.h, which accounts for all of the code you have seen so far. I have the following lines that perform compression and decompression:

std::ifstream in( name, std::ios_base::binary );
std::ofstream lzw_out( temp_name_lzw, std::ios_base::binary );
compress( (std::istream &) in, (std::ostream&) lzw_out, pDlg->m_MaxCodeSize );
.
.
.
std::ifstream lzw_in( temp_name_lzw, std::ios_base::binary );
std::fstream out( temp_name_out,
                  std::fstream::in    |
                  std::fstream::out   |
                  std::fstream::binary );
decompress( (std::istream &) lzw_in, (std::ostream&) out, pDlg->m_MaxCodeSize );

If I try to build this project as-is, I get a nasty list of eight linker errors:

Visual Studio 10 Error Messages

It's tempting to try to read characters using the ifstream extraction operator, as in m_impl >> c, but that operator skips over whitespace, so we don't get an exact copy of the input stream. Using get() works around this problem. Below is the complete definition of input_symbol_stream<std::istream> used in all four LZW implementations in this article:

template<>
class input_symbol_stream<std::istream> {
public :
    input_symbol_stream( std::istream &input )
        : m_input( input ) {}
    bool operator>>( char &c )
    {
        if ( !m_input.get( c ) )
            return false;
        else
            return true;
    }
private :
    std::istream &m_input;
};

Using the insertion operator to output strings seems to work properly, even when the strings contain binary data, so the implementation of the class used to output symbols is as simple as we could hope for. Again, this exact code is used in all four implementations in this article:

template<>
class output_symbol_stream<std::ostream> {
public :
    output_symbol_stream( std::ostream &output )
        : m_output( output ) {}
    void operator<<( const std::string &s )
    {
        m_output << s;
    }
private :
    std::ostream &m_output;
};

LZW-A prints the text values of integers to the output stream, and reads them back in that format. This is not efficient at all, but it is a great aid in debugging. If you are having a problem with the algorithm, this provides a nice way to examine your stream. The implementation of this is very simple - just use the std::ostream insertion operator, and follow each code by a newline so it can be properly parsed on input, as well as be easily loaded into a text editor.

One important thing to notice in this class: the presence of a destructor that prints the EOF_CODE. Since this object goes out of scope as the compressor exits, this insures that every code stream will end with this special code. Putting the onus on the I/O routines to deal with EOF issues simplifies the algorithm itself. (It also means that you can implement versions of LZW that don't use an EOF in the code stream.)

template<>
class output_code_stream<std::ostream> {
public :
    output_code_stream( std::ostream &output, const unsigned int )
        : m_output( output ) {}
    void operator<<( unsigned int i )
    {
        m_output << i << '\n';
    }
    ~output_code_stream()
    {
        *this << EOF_CODE;
    }
private :
    std::ostream &m_output;
};

The corresponding version of the input class just reads in the white-space separated codes. If there is an error or an EOF_CODE encountered in the stream, the extraction operator returns false, which allows the decompressor to know when it is time to stop processing.

template<>
class input_code_stream<std::istream> {
public :
    input_code_stream( std::istream &input, unsigned int )
        : m_input( input ) {}
    bool operator>>( unsigned int &i )
    {
        m_input >> i;
        if ( !m_input || i == EOF_CODE )
            return false;
        else
            return true;
    }
private :
    std::istream &m_input;
};

By including lzw-a.h along with the other two header files, I can now create a program that compiles, links, and is able to test the algorithm. Using my UNIX test program, I compress the demo string from earlier in this article, and I see the output as it is sent directly to stdout:

Compressing ABBABBBABBA

LZW-B

The header file lzw-b.h implements specialized classes that replace the text-mode output of the codes in lzw-a.h with binary codes stored in a short integer - two bytes.

The classes that read and write symbols are unchanged, but reading and writing codes has to change in order to do this new binary output.

Writing the codes to std::ostream as binary values requires breaking the integer code into two bytes and writing the bytes one at a time. There are more efficient ways to write the complete short integer in one function call, but they raise code portability problems, as we don't always know what order bytes will be written in.

Like the code stream output object in lzw-a.h, this version of the code output class has a destructor that outputs an EOF_CODE value:

template<>
class output_code_stream<std::ostream> {
public :
    output_code_stream( std::ostream &output, const unsigned int )
        : m_output( output ) {}
    void operator<<( unsigned int i )
    {
        m_output.put( i & 0xff );
        m_output.put( (i>>8) & 0xff);
    }
    ~output_code_stream()
    {
        *this << EOF_CODE;
    }
private :
    std::ostream &m_output;
};

Reading the codes requires reading the two bytes that make up the short integer, then combining them. While reading, if the routine detects an EOF_CODE, it returns false, which tells the decompressor to stop processing. It also returns false if there is an error on the input code stream.

template<>
class input_code_stream<std::istream> {
public :
    input_code_stream( std::istream &input, unsigned int )
        : m_input( input ) {}
    bool operator>>( unsigned int &i )
    {
        char c;
        if ( !m_input.get(c) )
            return false;
        i = c & 0xff;
        if ( !m_input.get(c) )
            return false;
        i |= (c & 0xff) << 8;
        if ( i == EOF_CODE )
            return false;
        else
            return true;
    }
private :
    std::istream &m_input;
};

The most exciting thing about lzw-b.h is that you can now see data compression taking place. The figure below shows a sample run of this implementation against the Canterbury Corpus, a standard set of files used to test compression. A run with my Windows test program shows that the files are compressing quite nicely:

Compressing the Canterbury Corpus with lzw-b.h

LZW-C

The third I/O implentation, defined in lzw-c.h, writes binary codes like lzw-b.h, but with one crucial difference. Instead of being hard coded to 16 bit codes, lzw-c.h determines the maximum code size needed based on the maximum code value passed as an argument to compress() and decompress(). It then writes codes based on that width, which will normally be something in the range of 9-18 bits wide.

Since these values are not aligned with byte boundaries, there are some issues writing them to streams that expect to read and write bytes. However, it is definitely worth all the bit shifting, ORing, and ANDing, because when the size is 12 bites, we are going to save four bits per code when compared to using lzw-b.h. But every read and write potentially starts somewhere in the middle of a byte, so the I/O classes have to do some extra work - mostly involved with shifting bits to the correct position in the output stream.

Note that the code to read and write symbols is unchanged from lzw-a.h and lzw-b.h.

Many of the CS students who read my earlier article on LZW ran into a brick wall when they started trying to understand the code that performs I/O on codes of variable bit lengths. Obviously, writing 11 bit codes when your file system is oriented around eight-bit bytes involves a lot of bit twiddling, and I'm afraid that many novices are woefully deficient in this department. Not just in understanding the bitwise operators in C, such as shifting, masking, etc., but in understanding binary arithmetic in general.

That's why I've structured the code and this article a bit differently this time around. If the I/O operations in lzw-c.h and lzw-d.h are bewildering, well, no worries. They have absolutely nothing to do with the LZW algorithm itself. You can investigate and explore the algorithm completely using lzw-a.h and lzw-b.h, and just forget about the last two I/O implementations. They provide additional efficiency, but as I have said, have nothing to do with the algorithm itself.

Further, once you use lzw-a.h to debug and understand the algorithm, you can certainly plug in lzw-c.h and lzw-d.h and take advantage of their improved compression, even if you don't follow all the code.

It might be appropriate to add a sidebar or another section to explain the variable bit length I/O in detail, but this article is quite long already, and there are numerous other resources for the interested reader to explore the details. (But if you find yourself deficient in this area, you owe it to yourself to hit the books and get to the point where these operations make sense. This won't be the last time you need to understand bitwise operators.)

For those who are ready to tackle this more complicated I/O procedure, we will look first at the output_code_stream<std::ostream> class. Here, the first thing to understand is that the constructor has to initialize the number of bits in the code. This value is calculated from the max_code parameter, and is stored in member m_code_size, where it is used frequently.

Next, the insertion operator. Output of codes proceeds as follows. Member m_pending_bits tells us how many bits are pending output while sitting in member m_pending_output. These bits are right justified, and the count will always be less than eight. When the new code is written, it is inserted into m_pending_output after being left shifted so it will be laid down just past the pending bits. After doing that, we presumably have some bytes to output - the exact number depends on various factors. The flush() routine is called, and it flushes all complete bytes out. When it completes, there can be anywhere from zero to seven bits still waiting to be output, and they will be right justified in m_pending_output.

In the destructor, we output an EOF_CODE, and then do a flush as well. But in this case, we flush all possible bits, not just the complete bytes. There are two good reasons for this. First, we don't care if the last bits that are flushed out are only part of a code - the code will be EOF_CODE, and that is the last one. And second, if we don't flush those final bits out in the destructor, they will never be sent to the output stream. This means the decoder will not see those bits, and we will most likely break the decompress process.

template<>
class output_code_stream<std::ostream>
{
public :
    output_code_stream( std::ostream &out, unsigned int max_code )
        : m_output( out ),
          m_pending_bits(0),
          m_pending_output(0),
          m_code_size(1)
    {
        while ( max_code >>= 1 )
            m_code_size++;
    }
    ~output_code_stream()
    {
        *this << EOF_CODE;
        flush(0);
    }
    void operator<<( const unsigned int &i )
    {
        m_pending_output |= i << m_pending_bits;
        m_pending_bits += m_code_size;
        flush( 8 );
    }
private :
    void flush( const int val )
    {
        while ( m_pending_bits >= val ) {
            m_output.put( m_pending_output & 0xff );
            m_pending_output >>= 8;
            m_pending_bits -= 8;
        }
    }
    std::ostream &m_output;
    int m_code_size;
    int m_pending_bits;
    unsigned int m_pending_output;
};

Like the output code class, the input code class has to calculate the code size for this decompression based on the max_code value passed in the function call.

When an attempt is made to read a code, there must be a minimum of m_code_size bits in member m_pending_input. If there aren't, new bytes are read in one at a time, and inserted into m_pending_input after having been shifted left the appropriate amount. Once m_pending_input contains at least m_code_size bits, the code is extracted from m_pending_input using the appropriate mask, the count in m_pending_input is reduced, and m_pending_input is shifted right by m_code_size bits.

template<>
class input_code_stream<std::istream>
{
public :
    input_code_stream( std::istream &in, unsigned int max_code )
        : m_input( in ),
          m_available_bits(0),
          m_pending_input(0),
          m_code_size(1)
    {
        while ( max_code >>= 1 )
            m_code_size++;
    }
    bool operator>>( unsigned int &i )
    {
        while ( m_available_bits < m_code_size )
        {
            char c;
            if ( !m_input.get(c) )
                return false;
            m_pending_input |= (c & 0xff) << m_available_bits;
            m_available_bits += 8;
        }
        i = m_pending_input & ~(~0 << m_code_size);
        m_pending_input >>= m_code_size;
        m_available_bits -= m_code_size;
        if ( i == EOF_CODE )
            return false;
        else
            return true;
}
private :
    std::istream &m_input;
    int m_code_size;
    int m_available_bits;
    unsigned int m_pending_input;
};

The table below shows the results of a test run comparing LZW-B and LZW-C run with a maximum code of 4095. With this maximum value, all codes fit in a 12-bit integer. Since LZW-B will use a 16-bit integer to store the code values, and LZW-C will use 12-bits, there should be a 4:3 ratio between the ratio of the file sizes when compressed using the two algorithms, and this looks to be the case:

LZW-D

The code in lzw-d.h represents the final and most efficient version of I/O for the LZW code streams. It builds on the code in lzw-c.h - at its core it is a variable bit-length I/O stream. However, there is one crucial difference from lzw-c.h: the code I/O in lzw-d.h starts at the smallest possible code size, nine bits, and increases the code size as needed, until it reaches the maximum value for this compression session. The maximum value is the parameter passed in to the invocation of compress() or decompress().

The logic behind this is pretty simple. Even if we are going to use 16-bit codes in an LZW program, when the program first starts, the maximum possible code the program can emit is 256, which only needs nine bits to encode. And each time we output a new symbol, that maximum possible code value only increases by one, which means that the first 256 codes output by the encoder can all fit in nine bits.

So the LZW-D encoder starts encoding using nine-bit code widths, and then bumps the value to ten as soon as the highest possible output code reaches 512. This process continues, incrementing the code size until the maximum code size is reached. At that point the code size stays fixed, as no new codes are being added to the dictionary.

The decoder follows exactly the same process - reading in the first code with a width of nine bits, then bumping to ten when the maximum possible input code reaches 512.

The code for this class is built on that from lzw-c.h, with some added complexity. Due to its increasing length, and the fact that it doesn't add too much to the discussion of LZW, I've omitted the listing, and instead refer you to the download available at the end of the article.

The Windows Test Program

When you develop compression code, there are a few different common tasks you are likely to want to perform:

• Check your code for correctness, often through bulk testing.
• Check your compression ratios against standard benchmarks.
• Analyze your program's performance so as to make it more efficient and locate bottlenecks.

My Windows app is designed to help with all of these tasks. It basically allows you to select a single directory, set a maximum code size, then perform compression and decompression of all the files in the directory. An optional checkbox lets you include files in all directories under the test directory as well.

The application was built using Visual Studio 10, and it is a simple MFC Dialog-based application. It allows you to select a base directory, a maximum code size, and then compress all the files in that directory. If you select the recursion check box, you will also compress all the files in the entire tree of subdirectories below it.

Each file is compressed to a temporary location, then decompressed in a temporary location. The size of the compressed file is saved, and then a comparison is done to ensure that the original and expanded files are identical.

To help with data collection, after running a test, you can press the copy button and get the results of the test stuffed into your clipboard. Although it isn't visible in the display, the data stored in your clipboard includes the full path name of the original file, not just the basename.

This Visual Studio project takes advantage of a number of C++11 features, and as a result it will need some modification to work with earlier versions. Any version that supports unordered_map can be made to build without too many changes. And if you are going way back in time, you could replace unordered_map with map.

As shipped, the test program uses lzw-d.h. To use any of the other three other versions of I/O discussed in this article, just modify the include file selected at the top of LzwTestDlg.cpp. The figure below shows what the app looks like after running through some data:

The Windows test app after a test run

Copying the data into a spreadsheet

The Linux Test Program

The LZW code is platform independent, and will build and run just fine on UNIX or Linux systems. The Linux test program, lzw.cpp, allows you to compress or decompress files from the command line. It builds just fine with g++ 4.5, as long as you use the -std=c++0x switch to turn on the latest language features. Compiling with earlier versions will require a few minor modifications.

The command line interface to the test program is not too complicated, and is probably best documented by looking at the usage output:

mrn@ubuntu:~/LzwTest$ g++ -std=c++0x lzw.cpp -o lzw
mrn@ubuntu:~/LzwTest$ ./lzw
Usage:
lzw [-max max_code] -c input output #compress file input to file output
lzw [-max max_code] -c - output     #compress stdin to file otuput
lzw [-max max_code] -c input        #compress file input to stdout
lzw [-max max_code] -c              #compress stdin to stdout
lzw [-max max_code] -d input output #decompress file input to file output
lzw [-max max_code] -d - output     #decompress stdin to file otuput
lzw [-max max_code] -d input        #decompress file input to stdout
lzw [-max max_code] -d              #decompress stdin to stdout
mrn@ubuntu:~/LzwTest$

Like the Windows test program, the command line program is built by default with lzw-d.h. Replacing this algorithm with any of the three others requires a minor change to the source code.

With the default build, the program produces output nearly identical to UNIX compress. The one difference is that UNIX compress monitors the compression ratio after the dictionary is full, and clears the dictionary if the ratio starts to deteriorate (which it almost always does.) I include a benchmark program that tests UNIX compress against the command line test program, and the results show that for small files, the file size is almost identical:

mrn@ubuntu:~/LzwTest$ ./benchmark.sh 65535 16 canterbury | head -n 15 | column -t
Filename                 Original-size  LZW-size  Compress-size
--------                 -------------  --------  -------------
canterbury/aaa.txt       33406          320       321
canterbury/alice29.txt   152089         62247     62247
canterbury/alphabet.txt  100000         3052      3053
canterbury/asyoulik.txt  125179         54989     54990
canterbury/a.txt         1              3         5
canterbury/bib           111261         46527     46528
canterbury/bible.txt     4047392        1417735   1377093
canterbury/book1         768771         317133    317133
canterbury/book2         610856         247593    251289
canterbury/cp.html       24603          11315     11317
canterbury/E.coli        4638690        1213579   1218349
canterbury/fields.c      11150          4963      4964
canterbury/geo           102400         77777     77777

You can see in this test that LZW-D and UNIX compress perform nearly identically for all but the largest files in the test sample. If I modify UNIX compress to not monitor compression ratios, the difference seen with larger files goes away:

mrn@ubuntu:~/LzwTest$ ./benchmark.sh 65535 16 canterbury | head -n 15 | column -t
Filename                 Original-size  LZW-size  Compress-size
--------                 -------------  --------  -------------
canterbury/aaa.txt       33406          320       321
canterbury/alice29.txt   152089         62247     62247
canterbury/alphabet.txt  100000         3052      3053
canterbury/asyoulik.txt  125179         54989     54990
canterbury/a.txt         1              3         5
canterbury/bib           111261         46527     46528
canterbury/bible.txt     4047392        1417735   1417735
canterbury/book1         768771         317133    317133
canterbury/book2         610856         247593    247593
canterbury/cp.html       24603          11315     11317
canterbury/E.coli        4638690        1213579   1213579
canterbury/fields.c      11150          4963      4964
canterbury/geo           102400         77777     77777

That provides some support for the notion that the algorithm shown here behaves properly.

Your Program

If you want to build your own program and use these classes, all you need is a C++11 compiler, or an earlier version and a willingness to make a few changes.

To use the classes, include in order lzw_streambase.h, one of the four implementation files for iostreams, preferably lzw-d.h, and finally, lzw.h. Because the significant code in these files is all implemented as template functions or classes, there is no library to include in your project, and no C++ source you have to compile separately.

All of the code in these header files has been hoisted into the lzw namespace, so you will either have to explicitly use the namespace when you invoke compress() and decompress(), or insert this line into your program:

using namespace lzw;

One thing to note about the I/O routines I have defined. The template functions are specialized on std::istream and std::ostream. If you innocently pass in an object such as an std::ifstream, you will get compile time errors. This is because C++ template matching is done on a very strict basis - the compiler won't generally try to figure out that std::ifstream is derived from std::istream, and use the existing class. So instead, you will need to cast your arguments to the types defined in the header files. (Or write your own implementations.)

Your rights to use this code are covered by my Liberal Code Use Policy. As I have mentioned before, this is teaching code, if you decide to use it in a production system, there are many optimizations you might want to perform.

Benchmarks

So how does LZW do when it comes to compression? LZW's original strength was its combination of good compression ratios with high speed compression. The UNIX compress program is still nice and fast, and Terry Welch's original application for LZW was in disk controllers. Because my program is a teaching program, it won't be nearly as fast as compress, but it's still useful to compare it to the de facto standard for lossless compression: the deflate algorithm.

We can compare LZW against deflate by a small modification of my benchmark script that uses gzip instead of compress. The table below shows the average compression ratios for the files in the canterbury corpus when compressed using maximum code widths of 15-18 bits. (The ratio is defined as 100*compressed_size/uncompressed_size, so 0% is perfect compression and 100% is no compression.)

Variations

There are many variations on the code I've presented here that make sense.

One obvious change is to eliminate the special EOF_CODE used to delimit the end of the code stream. If the code stream is a file or other stream with an inherent EOF condition, there is no need for an EOF_CODE - simply reaching the end of the input stream will properly signal the end of the decoded material. Freeing up this one code will make a microscopically small improvement in the compression ratios of the product.

If you want to mimic the output of the compress program, you need to remove the EOF_CODE, and replace it with a CLEAR_CODE that has a value of 256. The compress program monitors the compression ratios it achieves after its dictionary is full, and when the ratio starts to decay, it issues the CLEAR_CODE. That code tells the decoder to clear its dictionary and make a fresh start with new nine-bit codes.

Once you get the hang of LZW, a good exercise to make sure you have it working properly is to create a GIF encoder and decoder. GIF uses LZW to losslessly compress images with a constrained palette, and after all these years is still somewhat of a standard on the web.

History

Usually the history lesson on an algorithm is at the start of the article, but this is a how-to piece, and I feel like the trip down memory lane is not as important as understanding how the algorithm works.

The roots of LZW were set down in 1978 when Jacob Ziv and Abraham Lempel published the second of their two seminal works on data compression, "Compression of Individual Sequences via Variable-Rate Coding". This paper described a general approach to data compression that involved building dictionaries of previously seen strings.

Ziv and Lempel's work was targeted at an academic audience, and it wasn't truly popularized until 1984 when Terry Welch published A Technique for High-Performance Data Compression. Welch's paper took the somewhat abstract Information Theory work of Ziv and Lempel and reduced it to practice in such a way that others could easily implement it.

UNIX compress was probably the first popular program that used LZW compression, and it very quickly became a standard utility on UNIX systems. The freely available code for compress was incorporated into ARC, one of the first archiving programs for PCs. In addition, the algorithm was used in the GIF file format, originally created by Compuserve in 1987.

LZW's popularity waned in the 1990s for two important reasons. First, Unisys began enforcing their patents that covered LZW compression, demanding and receiving royalties from various software companies. Not only did this make developers think twice about the liability they could incur while using LZW, it resulted in a general public relations backlash against using patented technology.

Secondly, the LZW algorithm was eclipsed on the desktop by deflate, as popularized by PKZIP. Not only did deflate outperform LZW, it was unencumbered by patents, and eventually had a very reliable and free open source implementation in zlib, a library written by a team lead by Marc Adler and Jean-loup Gailly. I don't know if there is any way to actually quantify this, but I think one could speculate that zlib is currently installed on more computer systems than any other software package in existence.

So LZW has settled down to an existence out of the limelight. It is still an important algorithm, used in quite a few file formats, and as this article shows, its simplicity makes it an excellent learning tool.

Downloads

• LzwTest.zip - source for the Windows test app.
• LzwExe.zip - The Windows test app executable.
• lzw.tgz - source for the UNIX text app.

Newcomers to the world of data compression often stumble on this old idea in hopes of creating a novel and powerful algorithm. In a nutshell, the idea is to create an enumerative coding system that uses combinatorial numbering to identify a message, in hopes of providing a more compact representation . Unfortunately, these schemes always fail, for reasons that I’ll lay out in this article.

Combinations

To design a combinatorial algorithm that will compress files, you can think of the file as a series of integers. Since most of the files that you use are streams of bytes, consider each file to be a sequence of integers with values from 0 to 255.

If you go back to your first classes on probability and statistics, you might remember the definition of a combination. A combination is simply a way of selecting a number of things from a larger set. When you are trying to compress a file of bytes, the natural size of this set is 256.

Probability theory tells us that we can count the number of combinations of a given size using a pretty simple formula. If a set has n elements and we are choosing k at a time, the number of possible combinations is given by the formula n!/k!*(n-k)!. This number is also known as the binomial coefficient.

Just for a simple example, the number of different ways you can select three bytes out of a set of 256 is 2,763,520. In general, with a large set, most combinations are going to generate very large numbers. The exceptions will be for values of k that are either very small or very close to the size of the set.

Combinations are well ordered, so any instance of the three bytes has a specific number between 0 and 2,763,519. We can call this the combinatorial rank. This means I can identify any three byte sequence by a combination number.

Assuming all combinations are equal, we can use an optimal arithmetic coder to encode this number in lg(2,763,510) bits, roughly 21.4. That’s interesting, because the three bytes actually take up 24 bits, so maybe there is some savings to be found here.

The First Problem

Knowing the combinatorial rank is good, but it won’t let you reconstruct a compressed file on its own. The combinatorial rank gives you the set of bytes in the file, but doesn’t tell you the order of those bytes. If there are k bytes, they can be ordered in k! different permutations. So to fully describe the file, you need to encode the combination rank and the permutation number.

Encoding the permutation for your 3 byte file is going to take 2.58 bits, calculated as lg(3!). This makes the total needed to encode your three byte file 23.98 bits. Admittedly not a lot of savings, but it’s also non-zero.

Let’s look at the number of bits needed to encode a 20 byte message. The number of combinations of this length are roughly 2.8*10^29, which will take 97.8 bits to encode. 20! is roughly 2.43*10^18, which will take 61.1 bits to encode. The total comes out to 158.9 bits. Since we’re encoding 160 bits of information, there is clearly a greater savings.

As the message size increases, the savings start to grow. At a message length of 50 we save 7.5 bytes, at 75 bytes we save 16 bytes per message. The trend looks good. By the time you get to a message length of 100 bytes, you’re saving 32 bits per message – a compression of 4% for doing nothing but recoding!

The Second Problem

The second problem you encounter in the combinatorial system is that, by definition, a combination is composed of unique elements. So if you are compressing a three byte file, you can’t have any duplicate bytes. Is this a problem?

Your inclination is to hope not. You know that every compression scheme only works on a subset of files, so perhaps the combinatorial scheme can be developed to work on segments of files with no duplicates.

How likely are you to find a duplicate in a file of three bytes? You can start by enumerating the total number of files of that length: 256^3. And you know how many files there are with no duplicates: the combinatorial number times the number of permutations. So it’s a simple matter to calculate the probability that a message of length k has no duplicated bytes. The value will be n!/(n-k)!*256^k.

For a value of 3, we see that the probability of no duplicate bytes is .988 – this means you can compress almost every file by a fraction of a bit.

You’d like to think that you can look at pretty long stretches of data and expect a low probability of duplicates, but unfortunately you run into the Birthday Paradox. In the birthday paradox, you’re asked a question something like this: in a room of 23 people, what are the chances that two people share a common birthday? For most people, the answer, 50% or so, is non-intuitive.

Likewise, it means that a file with 100 bytes and no duplicates is such a rarity that it might as well never appear – the chances are less than one in a billion.

Facing the Music

You can see the problem here. We can compress very short sequences using a combinatorial system, but the savings are very small. Even so, we can compress most files. We can compress longer files for greater savings, but very few sequences will prove to be eligible.

It’s actually worse than that. Let’s work out the number on a hypothetical compressor. This compressor will use a combinatorial scheme to compress all files of 10 bytes. The compressor will look at the file, and if it has no duplicates, it will set a flag symbol in the output stream to be true, followed by the combinatorial number, followed by the permutation.

If the 10 byte file has duplicates, the compressor will generate a flag symbol of false, followed by the uncompressed data.

If this scheme gives us some savings, we can scale it up to operate on files of any size – we’ll just compress them in 10-byte chunks.

So let’s analyze the result. First, the number of files that will make it past the first test is pretty impressive: 83.695%. Each of these files will be compressed down to 79.743 bits. The remaining 16.305 percent will take exactly 80 bits in the output stream. So the overall size of our output file thus far is going to add up to 79.78519 bytes. Our algorithm is still in the black!

Unfortunately, we also need to account for the cost of the flag message. Using optimal coding, when the flag is true we are going to require .25679 bits. When it is false, optimal coding of the much rarer message will require 2.6 bits. Add in the cost of the flag, and the average output size goes up to a smidgen over 80 bits.

In other words, you lose.

Conclusion

The problem is a familiar one in data compression. Every time you come up with a way to encode a subset of files that saves some space, you find that all your savings are lost when you try to encode the files that aren’t part of the subset. Even using a single bit to flag special files as being incompressible is enough to wipe out your savings. It is the definition of Whac-A-Mole.

With combinatorial coding you will find that same rule to hold true as for all forms of data compression: It isn’t going to be a universal compressor that can reduce every file in size. The only reason it will be useful is if you have an input set of files that all have a common characteristic: a preponderance of streams where duplicates are rare. And odds are, this set of files will probably be compressible using some more reasonable algorithm.

The Pigeonhole Principle, also referred to as the Counting Theorem, is a handy tool for mathematicians, and naturally, computer programmers.

The loose version of this principle says “After placing n pigeons into m compartments, if n is greater than m, you will find that some compartment must contain more than one pigeon.”

Seems obvious, and perhaps it is, but at least in the world of data compression it must be trotted out from time to time in order to bludgeon dreams back to reality.

Impossible Compression

A common dream for the novice is the creation of a compressor that will reduce the size of all files. (Often touted as the ability to compress “random” data.) For example, Dr. Constant Wong of Recursiveware has been polishing his technique for compressing random data since 2003. And the USENET newsgroup comp.compression always has at least one thread dedicated to thrashing a new and eager theorist with a (flawed) idea.

The Pigeonhole Principle quickly puts this idea to rest. We know that if a file is of length n bits, there are 2ⁿ possible input files. If a compressor can reduce the size of every file, the number of possible output files is 2ⁿ-1. The Pigeonhole Principle tells us that the output of at least two file compressions have to be identical. And since they are identical, the decompressor cannot create two different output files.

And More

The Wikipedia has another nice example of the principle in use. Imagine that you have a party with n people attending. At random, people shake hands with one another as they mill about. At the end of the night, we check the number of unique individuals each person has shaken with. What are the odds that two people will have shaken hands with the same number of people?

The answer is of course that there will always be two people who have shaken the same number of hands. There are n-1 pigeonholes, and n pigeons, QED.

Don’t Go There

If you ever find yourself spiraling down the rabbit hole of impossible data compression, I urge you to grab the life jacket of the Pigeon Principle before you are lost. It will save you a lot of pointless effort, plus clue you in to the fact that there are two IBM employees with the same number of hairs on their head.

Jeff Bezos Hawks the Kindle

It’s still not clear whether electronic books are the wave of the future or a consumer products cul-de-sac. Technology continues to improve, and there are certainly lots of good reasons for a device like Amazon’s Kindle to be the leading edge of a major wave of adoption. A few of the more obvious arguments include:

• Reduced cost of distribution. The publishing industry wastes a lot of money printing and shipping books, and because of historical practices, creates huge numbers of books that never even get sold. Not very green, and a waste of money.
• Niche markets that can be very well-served. For example, high school and college students can replace those 30-pound backbacks with a 30-ounce tablet-sized device.
• Removal of barriers to publication. The news and magazine businesses are being revolutionized by self-publication in the form of blogs. Self-publication is possible in the printed book world, but it is still a rather awkward process. Publication to electronic format is presumably a trivial problem.
• Integration of information resources. A device like the Kindle allows you to consult the Internet, written reference materials, and your personal notes all from the same device, making it a true information portal.

But consumer acceptance is a fickle thing, so we don’t know if these rational arguments are going to fly. And of course, every writer who reviews a device like the Kindle or the equally capable Sony Reader feels compelled to write something along the lines of “But I just can’t imagine forgoing the pleasure of curling up on my couch with a good book.” I’m sure that when Gutenberg introduced movable type there were millions of industry reviewers posting notes to their blogs saying “The uniformity of the type is just esthetically unpleasing – I love it when I recognize the script of one of my favorite transcribing brothers.”

And then of course there’s the Steve Job’s money quote on the subject:

“It doesn’t matter how good or bad the product is, the fact is that people don’t read anymore, forty percent of the people in the U.S. read one book or less last year. The whole conception is flawed at the top because people don’t read anymore.”

So it’s really pretty hard to be sure just which way this is going to go.

Looking Beyond The Reader

Sony’s Reader

For electronic books to succeed, one thing is certain: the physical reading experience needs to match up well with the one we have right now for our printed media. In some ways this problem is already solved – readers like the products from Amazon and Sony are book-sized and lightweight, with displays that are doing their best to match the various good qualities of paper.

But there are still issues that need work. One of the most important is in the area of layout and markup. A presentation format such as HTML is designed to work with multiple display sizes, repositioning elements as needed. This doesn’t necessarily work so well with textbooks, magazines, etc., where graphic artists invest huge amounts of time and energy on positioning, font selection, and other esthetic issues.

Additionally, there is am staggering amount of material that is simply not in a format compatible with today’s electronic books. Even magazines being published today are not always ready for transfer to an eBook format, and there is of course a massive backlog of valuable material in the world that has never existed in digital format.

Into this void steps Robert Maxwell Case, who has developed a system called ReadAllOver, and is attempting to exploit it via his company, SeeAndBelieve.com. SeeAndBelieve.com has created a digital layout technology called ReadAllOver (the company seemingly has a love affair with awkward CamelCase constructions) that does a superior job of preserving the look of printed materials. The company web site gives a good demonstration of exactly what they are capable of doing – it is definitely worth your time to take a look.

I asked Robert if he could take the time to answer a few questions about ReadAllOver, and he was gracious enough to respond.

Questions With Robert Maxwell Case

Mark Nelson: Hi Robert. I just recently became aware of your company, SeeAndBelieve.Com, and your imaging system, ReadAllOver. Before we get into the details of your technology, can you tell me a little bit about the history of your company? How long have you been at it? Are you working mostly solo or do you have some help? What kind of background do you have that got you into your current work?

Robert Maxwell Case: Sure, Mark. I come from a background of being a full-time musician and a part-time graphic designer. Around 1991-92, I was unhappy with then-current digital halftoning routines and began experimenting on my own. So I’ve been at it 15-plus years.

Jimmy Kung (left) and Robert

I’m not a programmer, so I have had a succession of programming assistants. In recent years, they have come from my affiliation with the Computer Science department at Texas State University-San Marcos where I am a seven-year member of the Industrial Advisory Board. Currently SeeAndBelieve.Com, in addition to myself, has one full-time employee, Jimmy Kung, and several part-timers.

The first of my five U.S. patents (three issued, two pending) was filed in 1993 in response to some interest expressed by Steve Carlsen, developer of the .TIF graphics file format at Aldus (he’s now with Adobe Systems.)

MN: Can you give me a capsule summary of ReadAllOver? How does it differ from page layout systems like we see in web browsers or PDF viewers? Does it differ from the rendering systems used in Sony and Amazon’s current eBook readers?

RC: Well, ReadAllOver in a nutshell is a digital halftone-based graphics system suitable for eBooks. It renders on the screen a digital page with the “look and feel” of a printed page, with all included graphic elements, typography and images, placed precisely as the graphic designer intended. It differs from existing web browsers and .PDF viewers in that it relies less on text files and font metrics and instead places more emphasis on a simplified, highly-compressible bitmap image. In many respects, it is a “picture” of the page, with an ancillary text file.

The Sony and Amazon eBook readers are primarily text-based, offering a limited number of typefaces and few graphics. They both use the E-Ink subtractive screen and we think ReadAllOver’s halftone system can be tailored to enable a good fit with that screen.

MN: It looks like your technology emulates the halftone process used to render photographic images in newspapers and magazines. How does it improve on that process to achieve smaller file sizes? Do you have data showing the level of compression you get for specific images? And do you also render type as halftone images? That would seem a lot less efficient than treating type as marked-up text.

RC: That’s right, ReadAllOver does emulate the halftone process with one major difference, and that is that typography can be processed with it and not fall apart.

Samples of ReadAllOver output (Detail may not be representative, images were resized)

The big idea is that the output image starts as an interim monochromatic checkerboard pattern, beginning from gray. To simplify, we derive the output image by rendering local areas of the input image that are darker than checkerboard gray by turning corresponding output monochrome white pixels to black. Conversely, we render local areas of the input image that are lighter than checkerboard gray by turning corresponding output monochrome black pixels to white. The result is a checkerboard-based ordered dither that can be re-ordered for variable-length run-length compression.

We presently are comparing compression levels for specific images and plan to publish the results. So with typography, abrupt gray level changes such as font outlines fall on the black pixels of the interim checkerboard to render any font without reliance on font metrics or hinting. For example, ReadAllOver pages containing only typography are competitive in file size to marked-up systems, and with the additional attribute that our system is able to efficiently render any font, in any language, and any image, placed correctly. That’s exciting.

MN: I get digital delivery of some magazines already. For example, IEEE Spectrum is delivered using technology from Qmags. Do you think you can do a better job than they are already doing?

RC: Qmags has selected Adobe PDF as its preferred file format, so the comparisons I’ve made previously between ReadAllOver and PDF come into play here. Taking a quick look at Qmags’ file sizes, I would say that ReadAllOver is competitive and looks subjectively more “print-like.”

MN: Have you released any sample code or SDKs for people to work with your compression technology?

RC: Not at the present time … we have been too busy developing our prototype (you can see it at SeeAndBelieve.Com ).

MN: The idea of an electronic book reader has been floating around for a long time, but right now it seems like we’re finally seeing designs that are actually gaining some traction. There are still a lot of naysayers, however. What do you see in the future for the electronic book reader? Will it eventually do for reading what the iPod did for music? Or will it forever be a niche product that is stuck on the verge of popular success? And how hard will it be for you to get the manufacturers of eBook readers to adopt ReadAllOver?

RC: Mark, I do believe that an electronic book reader will achieve iPod-like popular success, and, hopefully, in the near-term. In my opinion, the more book-like the readers become, the closer we will get to that so-called tipping point. Then the public will recognize the added benefit of having access to any publication, including one’s own library, available whenever and wherever they desire. There are some of obstacles still to be overcome, like screen pixels that are pretty large, and wired and wireless transmission pipes that are pretty small.

I personally would like to see a reading device with 8-1/2″ x 11″ facing screens, about the size and weight of a coffee-table book, with graphics-intensive magazines and newspapers rendered on screen nearly indistinguishable from paper and ink. I certainly can imagine a college student not having to carry a backpack full of textbooks around campus.

Here’s my vision for our product: We’re hoping to make adapting to ReadAllOver as seamless and easy as possible. We envision three communities of ReadAllOver users:

1. The reader

2. The publisher/bookseller
3. The hardware manufacturer.

We plan to provide each of these groups with a cost effective, easy-to-use solution. The ReadAllOver Viewer will be free for readers. They’ll read .RAO files on dedicated eBook devices, and most likely on other devices that could emulate a book-like reading experience. The inherent look and feel of printed material should reduce eye strain and, of course, we plan to offer such extras as page flipping, content search, printing, etc.

Publishers and booksellers will use ReadAllOver Publisher, our media content production system. They’ll be able to convert content from scanned material, as well as from existing editing and page layout systems and standards, including Adobe InDesign and FrameMaker, Quark XPress, MS/Office, Open Office, etc. (and, of course, we’ll include a .PDF to .RAO converter.) This community will appreciate a built-in high level of content protection with ReadAllOver’s emphasis on bitmaps. Hopefully they won’t need more, but if they do, our product should easily be able to incorporate additional encryption and DRM mechanisms.

We also plan to collaborate with eBook (and other display device) manufacturers in order to provide built-in support for the ReadAllOver rendering system. We feel we can efficiently adapt our system to display components with limited available grayscale and color levels. We also hope to offer a fixed-bit-rate option where every delivered page is the same file size.

Conclusion

Thanks Robert, this is all interesting stuff. I can see advantages to ReadAllOver that we don’t get from layout systems like PDF or HTML, so perhaps you will be able to hammer out an effective market position. I don’t have an eBook reader yet, but I think when the technology reaches the point where I can have a color Kindle I’ll probably jump on board. I’m almost there now, but since the Kindle is perpetually sold out at Amazon.com, my dollars are still safely in my wallet.

The IEEE has announced its list of medal winners for 2007, and this year’s Richard Hamming medal was awarded to Dr. Abraham Lempel:

For pioneering work in data compression especially the Lempel-Ziv algorithm.

This is a timely award, because it comes on the 30th anniversary of the publication of the first of two seminal papers by Dr. Lempel and Jacob Ziv, his associate at Technion – Israel Institute of Technology.

In 1977 Ziv and Lempel published A Universal Algorithm for Sequential Data Compression, describing what would come to be known as the LZ77 algorithm. In 1978, they followed this with Compression of Individual Sequences via Variable-Rate Coding, which described what came to be known as the LZ78 algorithm.

Both of these algorithms use macro substitution to compress arbitrary data. By replacing a long sequence of bytes with a short macro, compression is achieved. Both algorithms build this library of macros dynamically, adjusting to the input text as it is read. They differ in the way they build the library of macros.

LZ78 was eventually reduced to practice by Terry Welch with the publication of the LZW algorithm, used in UNIX compress, GIF files, and elsewhere. LZ77 provided the core of the deflate algorithm, which is used in the Zip standard, arguably the dominant lossless compression algorithm since it was introduced by PKZIP in 1993.

It would be hard to overstate the impact of these two papers in the world of data compression. While there are other lossless algorithms that can compress as well as LZ77 and LZ78 (e.g. PPM), the two LZ algorithms have won the battle of speed and efficiency almost 20 years. Applications for the algorithms vary from desktop applications to graphics file formats to tape drive controllers. As an example of their influence, Citeseer lists 458 citations for the 1977 paper, and 293 for the 1978 paper.

Dr. Lempel is currently employed by HP, performing and directing research at HP Labs in Israel. He was kind enough to answer a few questions for this article.

Mark Nelson: In 1977 and 1978, you published two now-famous compression papers with Jacob Ziv that provided the background for some of today’s most effective and popular compression algorithms, such as the deflate standard used in Zip-compatible programs. At the time, did you imagine that your work would be so broadly used 30 years later?

Abraham Lempel: We were so excited with the results of our work, that we did not give too much thought to this question at the time. As more and more extensions and versions were published by other researchers, we recognized the seminal nature of this work and viewed our approach to lossless data compression as a long term method.

MN: The papers you published provide a mathematical and theoretical description of universal compressors, but don’t dig into implementation details. Do you have much personal interest in converting theory into practice, or do you get the most satisfaction from the research work that provides the foundation?

AL: Following the publication of 1977 and 1978 papers, we both participated in writing an invention disclosure which included all the details of a preferred embodiment implementation. This was done while I was on sabbatical at Sperry Research, and the 2 granted patents ended up as Sperry, now Unisys, Intellectual Property.

MN: Since your seminal work, I think the most significant advance in lossless compression has been the creation of block-sorting algorithms, as described by Burrows and Wheeler. Do you think there are any revolutionary new techniques on the horizon, or should we just expect minor incremental improvements?

AL: Hard to tell. When we worked on our method, the Huffman algorithm was considered the last word in lossless data compression. Now, the various LZ versions are treated as such. All this is in the context of text compression. I do expect radical progress in lossless image compression, which is a different beast from text.

MN: After a distinguished career at Technion – Israel Institute of Technology and almost 25 years with HP Labs, you’re at a point where many people would be thinking of retiring and slowing down. Is that in your near-term plans?

AL: I do plan to retire in 2008.

It is great to see the IEEE honor Dr. Lempel for his work. It would have been hard to understand how important this was in 1977. Today, with the benefit of 30 years hindsight, it clearly stands out as a masterpiece.

Reference Links

• Citeseer page for A Universal Algorithm for Sequential Data Compression (1977)
• Citeseer page for Compression of Individual Sequences via Variable-Rate Coding (1978)
• IEEE Bio Page that accompanied the Hamming Medal Announcement
• List of Hamming Medal Recipients on the IEEE site.
• Wikipedia entry for Abraham Lempel
• Wikipedia entry for Jacob Ziv
• HP Bio page for Abraham Lempel
• Wikipedia entry for LZW compression.
• My 1989 DDJ article on LZW data compression, including C source
• The zlib home page. zlib is the free and widely used library that implements the deflate algorithm using in Zip programs. deflate combines an LZ77-style algorithm with a Huffman coder.

It looks like Forgent’s long run of JPEG lawsuit revenue has now officially dried up. The Austin American Statesman has an article (registration required) that says the remaining JPEG lawsuits have all been settled, with a grand total of less than $8 million from somewhere fewer than 20 defendents.

Apparently, the whole JPEG lawsuit run that Forgent was on must have racked up a lot of bad karma for the company. Although they collected over $100 million over a few years, they had to spend over half of that on legal fees, and while that was going on, they seem to have taken their eye off the ball and forgotten about developing new products and business. Their low stock price has generated a delisting warning from the NASDAQ, and to top it off, a digital video patent they had high hopes for appears to have been invalidated as well.

For those of you who haven’t been keeping up on Forgent’s activities, the company has been generating fairly large sums of money since 2002 by suing users of JPEG technology, starting with camera companies and moving on to computer manufacturers, web publishers, and so on. It all came to an end this year with the patent office invalidated key portions of the patent. A good recap can be found in this Wikipedia article.

The eminently quotable Samuel Johnson had a very pragmatic view about writing, and was quoted by Boswell as having said:

No man but a blockhead ever wrote, except for money.

Personally, I think Johnson is pretty close to the mark on this one, but I will add one caveat. Ask any writer about their first book, and the thing they remember best is the thrill of seeing a volume with their name on it up on the shelf – money has nothing to do with it.

My first effort was The Data Compression Book, published all the way back in 1992, when there was a lot less interest in the field than there is now. It was unbroken ground, which meant there was room for an amateur in the field, and I nearly had it to myself.

With the help of DDJ Editor and mentor Jon Erickson, I convinced M&T Books I could do a creditable job on this book. They took me up on it, and believe me, the first time I walked into Taylor’s books in Dallas and saw three or four of these on the shelf, it was a thrill you can hardly imagine.

Fifteen years later, with a handful of other books behind me, I’m much more Johnson-esque and blase these days. Show me the money.

But every once in a while, something manages to pierce this hard-boiled shell and remind of what it was like to first see that book in 1992. This week it was an email out of the blue from somebody named Steve Johnson, who was kind enough to let me reprint his email in its entirety:

Dear Mark,

I have had a copy of The Data Compression Book in my possession for a long time. It was a book of instrumental importance to me when I was starting out a fledgling business long ago – in 1992 – which turned into an image compression company partly because of your book. Your book captivated me with the compression problem, and taught me the basics of information/coding theory (enough to become a dangerous dilettante). Out of that understanding sprang a company called Johnson-Grace Company, which created the first streaming online media ever. AOL used my algorithm to put pictures online in early 1994 when the world dialed in at 2400bps or 9600bps, and ‘digital pictures’ were as fantastic as radio in 1920. My little company went on in 1995 (around the time MSN was launched) and created ‘streaming sound’ and ‘moving pictures’ (slideshows with sound) and then simple telephony (still over dialup, now at 14.4kbps, we added a ‘talk’ button to AOL’s Instant Message box – and then subtracted it after seeing the challenge of creating a consumer grade experience over low bandwidth dial up!).

AOL bought my company in 1996, and the algorithm (AOL’s proprietary ‘ART’ format) still compresses billions of images everyday on the backend of their web delivery system. I owe a great deal to your book – for its clear accessible style, and excellent coverage of the subject. It quite simply taught me (an economist) how compression works, and I managed to put it to a use that solved an important problem.

So here’s why I’m writing (besides finally thanking you after all these years!).

My eldest daughter Emma graduated from high school earlier this year (we live in Boston) and has just commenced her freshman year at Oberlin College this fall. As a graduation gift, I’m presenting her with a bound set of my favorite books called “Dad’s Great Books,” of which I have included my rebound copy of Data Compression. It is no doubt one of the most influential books I’ve ever read, and I hope Emma cherishes the book as much as I did (she happens to be fascinated with information theory at the moment).

If it isn’t too much trouble to ask, I would greatly appreciate it if I could send this bound copy to you for your autograph and have you return the book in the FedEx envelope that I would include.

Please let me know, at your earliest convenience, as well as a good address to use for sending this your way.

Warm Regards,

Steve Johnson

When I read this email to my wife she had tears in her eyes, and yes, I was a bit verklempt myself.

Steve sent me a nicely bound copy of the book and I inscribed it to he and Emma (without even asking for an honorarium!), and with luck she’ll have it on her shelf in a few days, a curious souvenir of bygone times.

All in all a Hallmark moment, although there is a scary side to the whole thing. Steve comes from a background in Economics, knowns as the Dismal Science. Emma is interested in Information Theory, which has no nickname yet, but perhaps should be known as The Yet More Dismal Science.

Should this continue, I fear for Emma’s children. They’ll have to find a branch of mathematical science even more obscure, confusing, and impossible to explain at family functions. Will they be Set Theorists? Transfinite Algebrists?

Regardless of their intellectual or business pursuits, it seems inevitable that anyone from this line of people will be delightful to have around.

Thanks, Steve, for a reminder of what it was like to write that book in 1992, and best of all, to know that at least for you, it was exactly what I had hoped it would be.