Hello Good Day,

This is Troop Emonds With regards to your Company i am sending this email Regards to order some( Compression Machine )I will like to know the type and sizes you have in stock and get me the sales price of one so that i will tell you the quantity i will be ordering, and if you accept credit card as a form of payment..

Hope to read from you soon about my order request……
With Kind Regards.
Troop

I just need to put together some compression machines, and then I’m set.

Before this trip, I had not set foot on foreign soil more than one mile from the US border, so the experience of going to a foreign country was in itself new. This means I am compelled to share it with you.

Overall New Zealand made me feel very welcome. I would like to move to New Zealand. Barring that, visiting New Zealand as a tourist was a great experience.

I could write a detailed photoblog of our eighteen day journey, but this would be a lot like going to your Uncle’s house and watching his 90-minute DVD compilation of his trip to Norway – a bit tedious.

One thing I noticed as a tourist is that it is kind of hard to notice things that are there, but very easy to notice the things that are missing. So my detailed summary of the trip will give you the list of things that I noticed were not in New Zealand – at least not on my self-driven tour. (I’d give a link to the fine touring company if they would work out some sort of affiliate program, then you could do it yourself.)

Things that don’t appear to exist in New Zealand:

• Stop signs
• Iced tea
• Pennies
• Nickels
• Dollar bills
• Insane airport security (domestic only)
• Pickup trucks
• The purported monoculture of sheep
• Moas
• Cosmopolitans, or more generally, grown-up cocktails
• Hot weather
• Air-Conditioned hotel rooms
• Free Wireless
• Reasonably priced Wireless
• Horrifying public restrooms
• 4 lane highways
• Any sense of realism about building a country in a volcano/earthquake/tsunami free-fire zone

Pictures and Movies

A bunch of photos here.

Some very short videos here.

TB: I’ve sometimes wonder about this myself… during the early days of computing and especially personal computers, it took some time for the latest technology to reach us “down under”, so perhaps we were motivated to get more out of what we had rather than wait some months for a larger disk or new memory to arrive from overseas. When the Internet arrived we started with a very small pipe, so a good compression algorithm could do the equivalent to laying a second cable from NZ to the US – who can resist getting something for free?

MN: Since you wrote Text Compression back in the early 90s, I’d say the biggest development in lossless compression has been the Burrows-Wheeler transform. Is lossless text compression basically done? Are we left with just incremental improvements as processor resources increase?

TB: That seems to be the case; the only big improvements we’ve seen have turned out to be frauds — we even had one in NZ recently, where a Nelson man raised NZ$5.3 million for an impressive sounding method; he was convicted of fraud last year. The main indicator we have that we’re running out of steam (apart from a lack of new discoveries) is Shannon’s experiments on predicting text which gave a bound in the order of 1 bit per character for English text, and current methods are approaching this. Of course, there’s plenty of room for dealing with new kinds of data (for example, bioinformatics deals with massive amounts of data that we’re still trying to understand) and for finding better data structures and algorithms for performing the compression and decompression. Lossy compression is a whole different story…

A Change In Focus

MN: It looks like you are now dedicating a large amount of your time to establishing computer science as part of the basic curriculum in high school education, for students in the 15-18 age range. In many ways, this is as much a bureaucratic problem as an academic one. What motivated you to take it on?

TB: It’s been a problem that we’ve complained about for decades, and it’s been getting worse and worse as computing in schools has focussed increasingly on using computers and not preparing students to be developers. A lot of this can be attributed to bureaucracy – it’s hard to explain to government officials that putting word processors in every classroom isn’t the same as building a computationally literate society. As a result of some strategic lobbying done by others, a small window of opportunity opened for me to be on a group to advise our Ministry of Education, just over 3 years ago. The group managed to convince the officials that something useful could be done, and then we had to work very quickly to come up with a concrete proposal before the enthusiasm died down. This has happened rapidly; the advisory group first met in November 2008, and Computer Science started being taught in schools in February 2011.

MN: What have you been able to accomplish in New Zealand so far?

TB: Computer science (including programming, but also topics the involve understanding the importance of things like algorithms, HCI, programming languages and even compression) is currently available as part of computing courses for two of the three final years of our main high school graduation qualification, with all three years being covered from 2013. After that we would expect some of the introductory material to start filtering down to earlier classes, and for wider offerings as teachers become more confident in the subject. One of the biggest challenges has been preparing teachers, few of whom have significant experience in Computer Science. Many have embraced it enthusiastically, and the universities and others have done a lot of work to help them get up to speed. It’s been a wild ride doing it so quickly, but there have been some very pleasing outcomes.

MN: And how do things look in the rest of the world? Are there any obvious winners and losers at this point? Do you have any concise advice for the world?

TB: Computing in schools is a hot topic around the world; the UK have just announced a strong drive to introduce this sort of material to schools, and the US has people working hard to make it available to students. Israel and Korea have had computer science in schools for some time. We’re learning a lot about what is worth teaching, and what the best pedagogy is for the general classroom (most of our experience is for specialist students who have chosen the subject!) The New Zealand path of getting something going quickly with grass-roots support seems to be more effective than waiting for a top-down approach which could take years to develop and prepare teachers for, although it does make for a bumpy ride as problems are ironed out as we go along!

MN: This might be straying out of your area a bit, but do you see CS in a K-12 education setting having an effect on the representation of women in the STEM fields?

TB: Attitudes that affect representation definitely start at school, and to me the biggest goal of teaching CS in high school is not so much to prepare students for further study, but to enable them to find out what the subject is! School students rarely know what CS is, and even worse, it’s common for them to assume that it must be advanced word processing or some other dull area, and hence they avoid it. It’s particularly important for female students to have the opportunity to find out if it’s something that they might be good at, as the stereotypes associated with computing can make them assume that they shouldn’t consider it as a career.

MN: One final question, Tim. The whole world has seen the devastating damage Christchurch has suffered from the earthquakes in the last year. How has the University of Canterbury held up? Have you managed to maintain continuity in your academic calendar?

TB: It’s been quite a year! Thankfully our university has escaped the brunt of the earthquakes (most of the damage is some distance from the university), and we’ve managed to keep a full programme going despite being closed for three weeks for safety checks. Many students joined the “student volunteer army”, who helped with the cleanup in the damaged parts of town, and that was probably one of the most valuable experiences of their career! It hasn’t been without disruption as buildings need to be checked carefully, and some are still under repair, but with a bit of resourcefulness we managed to keep going (for a while I even delivered my classes in a restaurant while lecture theatres were being inspected) The city is now going through a massive program of redevelopment with some pretty creative ideas, and it’s an exciting time to be part of these changes.

Thanks to Dr. Bell for taking the time to share all this with us. My visit to his amazing homeland was a real treat, and the short time I got to spend with Tim in Christchurch was worth the trip all in itself.

The Conflict

General purpose data compression routines tend to be used on binary streams of data, either from files or in-memory objects. So what is the best general paradigm for input and output when compressing data?

You might analyze this problem by imagining that you need to write a binary copy routine.

template<class INPUT_ITERATOR, class OUTPUT_ITERATOR>
void bcopy( INPUT_ITERATOR input, INPUT_ITERATOR eof, OUTPUT_ITERATOR output )
{
    while ( input != eof )
        *output++ = *input++;
}

This routine is particularly nice when you are performing a simple copy using pointers to memory – the generated code should be really efficient.

However, the iterator paradigm doesn’t work quite as well when you want to perform a binary copy of data in a file. I can make use of iterators that almost do the job:

 std::ifstream in( "input.txt", std::ios_base::binary );
 std::ofstream out("output.txt", std::ios_base::binary );
 bcopy( std::istream_iterator(in),
        std::istream_iterator(),
	std::ostream_iterator(out) );

But the bad news is that both istream_iterator and ostream_iterator use the insertion and extraction operators, which are really meant for whitespace-delimited textual data, not binary data. The copy routine shown here will not make a binary byte-for-byte copy of the input file.

So when using files, the stream approach seems to be the way to go:

template<class INPUT_STREAM, class OUTPUT_STREAM>
void bcopy( INPUT_STREAM in, OUTPUT_STREAM out )
{
    char c;
    while ( in.get(c) )
        out.put(c);
}

If my files have been opened using the iostream classes, you can use this binary copy function without having to write any glue code – they already support the get and put methods, so this works right out of the box.

My Choice

If I’ve made up my mind that my data compression routine is going to use one of these two paradigms, it means I am going to have to write some glue code. If I choose the iterator-based approach, I need the equivalent of istream_iterator and ostream_iterator for binary files – and these aren’t in the standard library. If I choose the stream-based approach, I need efficient put() and get() members for blocks of memory. In some cases basic_stringstream might do the job, but not in all cases.

After dithering around with various solutions, I tentatively opted for the stream paradigm. I found the implementation for various sources of data to be fairly simple, and the interface is easy to understand. I don’t know if it’s the perfect choice, and I’ll keep experimenting, but for now it works for me. My abstraction of the LZW code still needs a lot of work, so it’s always possible I could rethink this at a later date.

I’d like to hear your thoughts – is there an obvious right answer to this question?

One thing many people would like to do is use PuTTY as a component in their program. Apparently this comes up so often enough that there is a FAQ entry dedicated to the topic. Alas, PuTTY does not have any sort of automation interface, so this goal has always been out of reach.

In this article I will show you how to work around this minor shortcoming. Creating a version of PuTTY that can be driven from a Windows program turns out to be an easy task. I'll demonstrate this with a small C++ program that shows exactly how to get this versatile program to do your bidding. My solution works for C++, but the changes I make should work well with any Windows software that can properly process a few messages.

Putting Together the Project

I'm using Visual Studio 2010 to build both my program and the modified version of Putty. I created the basic outline as follows:

1. Use the File|New|Project menu item to bring up the list of available project wizards.
2. Select MFC project, and enter a project name (I used the uninspired name PuttyDriver.)
3. I don't want the default MFC settings, so in the MFC App Wizard, select the Next button.
4. On the Application Type page of the wizard, change the Application Type to Dialog Based.
5. The project is ready to go at this point, you can click the Finish button and then build your initial project.

My driver program is only going to do one thing: direct putty to connect to the host of my choice, then log in using canned credentials. The resulting UI is shown below, and I am going to leave the very minor details of creating it up to the reader.

The driver program - a simple dialog-based MFC app

Adding Putty to the Project

The next step in this process is to add the Putty components to the project. I downloaded version 0.61 of the PuTTY source from the download page and extracted it to a separate folder. I then used Visual Studio's File|Add|Existing Project to add the compatible project file, Putty.dsp, found in /Windows/MSVC/Putty. Visual Studio has to convert this project to a version 10 project file, but it should do so with no problems.

I then right-clicked on the Putty project in Solution Explorer and renamed it to AutoPutty. Since this version of PuTTY will have some slightly different behavior, I don't want to confuse the executable I am creating with the real thing.

From Project|Project Dependencies, I set the PuttyDriver project to depend on AutoPutty - this insures that both projects get built when I build the entire solution.

My final change to the project is to modify the output directory for both Debug and Release versions of AutoPutty. I set the project to build the executable in the root directory of my PuttyDriver project - this will make it easy to find the executable when I need to launch it. I had to make this change in two places: Properties|Configuration Properties|General|Output Directory and Properties|Linker|Outuput File.

When you finally build the project, you'll find that current version of Microsoft's C++ compiler complain quite a bit about the use of functions like strcpy - Microsoft would like you to use safer replacement functions. You may choose to turn those errors off by defining _CRT_SECURE_NO_WARNINGS in the project file. While you are there, you should define SECURITY_WIN32 as well - it is required by Windows header sspi.h.

After a successful build you should find a copy of AutoPutty.exe in the root directory of your project, and it should run on your system and behave just like PuTTY.

Launching AutoPutty

If I'm going to have a PuTTY component in my PuttyDriver program, one of the first things I need is to be able to start and stop AutoPutty. So my first step in this project is to create the code that launches the program from PuttyDriver. The code below is inserted into the handler for the Start button:

UpdateData( true );
char path[MAX_PATH];
GetCurrentDirectory(MAX_PATH, path);
if ( path[ strlen(path) - 1 ] != '\\' )
    strcat_s( path, MAX_PATH, "\\" );
strcat_s( path, MAX_PATH, "AutoPutty.exe -ssh " );
strcat_s( path, MAX_PATH, m_HostName.GetBuffer() );
PROCESS_INFORMATION pi;
ZeroMemory(&pi, sizeof(pi) );
STARTUPINFO si;
ZeroMemory(&si, sizeof(si) );
si.cb = sizeof(si);
if ( CreateProcess( NULL, path, NULL, NULL, NULL, NULL, NULL, NULL, &si, &pi ) )
{
    Sleep( 1000 );
    BringWindowToTop();
}

This code assumes that AutoPutty.exe is in the current directory, and launches it with a command line telling it to connect to the host named in the dialog using ssh. Assuming that you have the project set up properly, pushing the start button should now start an independent copy of AutoPutty, which will behave identically to classic PuTTY.

Taking Ownership of AutoPutty

At this point I can successfully launch AutoPutty, but I can't really start calling this an integrated part of my main program, PuttyDriver. All I have done is set up a launcher for a separate executable.

The next step in the integration process is to establish PuttyDriver as the owner of AutoPutty's main window. Most Windows programmers are familiar with the traditional parent/child relationship between windows. That relationship is well understood, but I can't use it here - it doesn't work for two top level windows.

Setting PuttyDriver to be the owner (as opposed to the parent) of AutoPutty has the following effects, as explained here by Microsoft:

• The owned window will always be above its owner in the z-order.
• The system automatically destroys the owned window when the owner is destroyed.
• The owned window is hidden when the owner is minimized.

The most straightforward way to set ownership of the window is to pass the owner's handle in the call to CreateWindow(), which means I will now make my first modifications to the PuTTY source code.

There are a number of ways to pass the owner handle to AutoPutty for use in the call to CreateWindow(), with the most obvious being to pass it on the command line. In the interest of minimizing changes to the existing PuTTY code base, I elected to pass it by creating an environment variable that holds the owner window handle. Since a child process inherits the parent's environment, this is a no-fuss way to get the data to AutoPutty.

I added the following code to the end of InitDialog() in PuttyDriver:

CString hwnd_text;
hwnd_text.Format( "%d", m_hWnd );
SetEnvironmentVariable("PUTTY_OWNER", hwnd_text );

This sets the environment variable for AutoPutty to find when it gets launched.

Now I come to the point where I am actually making changes to the PuTTY code. Fortunately, all of the changes needed for this program are confined to two files: terminal.c and windows/window.c. My first change is to window.c. This file contains the WndProc for the PuTTY window, and thus most of the rendering and control code for the GUI.

In order to establish the Owner/Owned relationship, I need to modify the code that calls CreateWindow(). I hoisted the function call into a block, added code to get the owner window handle, and inserted the handle into the call to CreateWindow():

{
    HWND owner_hwnd = 0;
    char buffer[ 132 ];
    if ( GetEnvironmentVariable( "PUTTY_OWNER", buffer, 132 ) )
        sscanf( buffer, "%d", &owner_hwnd );
    if ( owner_hwnd == 0 )
        MessageBox( NULL,
                    "AutoPutty did not find the handle for the "
                    "owner window, this is not going to work",
                    "Fail",
                    MB_OK );
    hwnd = CreateWindowEx(exwinmode, appname, appname,
                          winmode, CW_USEDEFAULT, CW_USEDEFAULT,
                          guess_width, guess_height,
                          owner_hwnd, NULL, inst, NULL);
}

At this point I've only modified one small block of code in the PuTTY source, but I'm well on my way to having it behave more like a component of PuttyDriver and less like an independent program. The ownership status means that the two programs only appear once on the taskbar, and will only appear once when you are pressing ALT-TAB to select a new active process. And they only produce a single entry in the Applications Tab of Task Manager.

The Communications Link

In order to achieve the automation that I am seeking, I also need to have two way communications between AutoPutty and the driver program. Since this is Windows, a natural choice for communications is to use native Windows messages. In order to do this, both programs need the Window handle of their opposite number.

I've already solved half of that problem through the ownership relationship established when I created the main window for AutoPutty. Now that it has set PuttyDriver as its owner window, I can get this window handle any place in the program through a simple function call:

HWND parent = GetWindow(hwnd, GW_OWNER);

But the reverse is not true - PuttyDriver does not know have a copy of the window handle for AutoPutty.

To remedy this situation, I added code to window.c that notifies its owner when it s created, and when it is destroyed. First I add this statement immediately after the call to CreateWindow():

if ( owner_hwnd )
   PostMessage( owner_hwnd, WM_APP, 0, (LPARAM) hwnd );

This tells PuttyDriver that the window is created, and gives it the handle to use for communications.

I also need to know when the window is closed, and I have to add that code two places in window.c - because Putty can be shut down two different ways.

Normally AutoPutty will shut down in response to a windows message. When this happens, I can count on a WM_CLOSE message being sent to the Windows Procedure. I add this code the existing handler for WM_CLOSE:

if (!cfg.warn_on_close || session_closed ||
    MessageBox(hwnd,
               "Are you sure you want to close this session?",
               str, MB_ICONWARNING | MB_OKCANCEL | MB_DEFBUTTON1)
    == IDOK) {
    HWND parent = GetWindow(hwnd, GW_OWNER);
    if ( parent )
        SendMessage( parent, WM_APP, 0, 0 );
    DestroyWindow(hwnd);
}

This lets PuttyDriver know that the window has been destroyed.

The original PuTTY code has an alternative method of shutdown. When it receives one of several possible network events, such as a telnet connection being broken, it calls PostQuitMessage(). When a program shuts down this way, it doesn't issue messages to destroys its windows - it relies on the O/S to destroy the windows when the process exists. As a result, I have to make a change in WinMain(), the main window procedure for PuTTY. This procedure extracts the messages sent to it using PeekMessage, and I add some code to handle the processing when a WM_QUIT message is sent:

if (msg.message == WM_QUIT) {
    HWND parent = GetWindow(hwnd, GW_OWNER);
    if ( parent )
        SendMessage( parent, WM_APP, 0, 0 );
    goto finished;	       /* two-level break */
}

Handling the AutoPutty Lifecycle Events

To keep track of the state of AutoPutty, I have to add a handler for WM_APP to PuttyDriver. It does two things when handling the incoming WM_APP event.

First, then handler stores the handle of the AutoPutty window - or sets the value to 0 when the window has been destroyed.

Second, it either enables or disables the button used to start up AutoPutty. Since this program can only manage one window at a time, I don't want to allow any inadvertent button pushes:

afx_msg LRESULT CPuttyDriverDlg::OnWmApp(WPARAM wParam, LPARAM lParam)
{
    m_PuttyWindow = (HWND) lParam;
    m_StartButton.EnableWindow( !m_PuttyWindow );
    return 0;
}

One final piece of bookkeeping is to make sure that the AutoPutty window is shut down when PuttyDriver shuts down. (The Windows documentation claims this happens automatically to owned windows, but it doesn't seem to be the case.)

void CPuttyDriverDlg::OnDestroy()
{
    CDialogEx::OnDestroy();
    if ( m_PuttyWindow )
        ::SendMessage( m_PuttyWindow, WM_CLOSE, 0, 0 );
}

Monitoring Input Traffic

Now that I have control over the lifetime of my AutoPutty window, it's time to take the next step in automation. My driver program needs to watch all the data coming in from the remote end so that it can take action on various types of input.

Depending on how you set up your connection, PuTTY can receive input data from a serial port, a Telnet connection, or an SSH connection. Fortunately the Windows version of PuTTY uses a standard handle-based interface to all three types of connections. The routine term_data() in terminal.c is called as data arrives, regardless of the source.

Since we are using the Windows API to communicate between processes, it makes sense to use the WM_COPYDATA message to send data to the parent program as it arrives. WM_COPYDATA is a good choice, as it takes care of marshalling the data between the two processes, which can add some complication to other solutions. The modified routine is shown below:

int term_data(Terminal *term, int is_stderr, const char *data, int len)
{
    HWND parent = GetWindow(hwnd, GW_OWNER);
    if ( parent ) {
        COPYDATASTRUCT cd;
        cd.dwData = (ULONG_PTR) 0xDEADBEEF;
        cd.cbData = len;
        cd.lpData = (PVOID) data;
        SendMessage( parent, WM_COPYDATA, (WPARAM) hwnd, (LPARAM) &cd );
    }

Receiving the Data

To receive this messages in PuttyDriver, I simply create a handler for WM_COPYDATA and start grabbing the data as it arrives. One important thing to note is that because AutoPutty has to use SendMessage() to send the data to its parent, it has to wait for PuttyDriver to finish processing the data until it can continue. This dictates a certain style of behavior on my part.

There are quite a few ways to skin this cat, and I'm keeping it very simple here. I'm using a deque<char> container to hold the last 64 characters I've received. After each WM_COPYDATA message I received, I check to see if the current output snapshot ends in one of my trigger messages. If it does, I post the message number to myself for later processing, then return so that AutoPutty can continue its work.

The code I'm using here is doing something fairly simple: automating the login process by using the credentials that I've entered into the dialog box. That means the two strings I'm looking for are the login and password prompts. The resulting code is shown here:

BOOL CPuttyDriverDlg::OnCopyData(CWnd* pWnd, COPYDATASTRUCT* pCopyDataStruct)
{
    char *p = (char *) pCopyDataStruct->lpData;
    int len = pCopyDataStruct->cbData;
    if ( len >= 64 ) {
        p += len - 64;
        len = 64;
        m_Snapshot.clear();
    }
    while ( len-- )
        m_Snapshot.push_front(*p++);
    m_Snapshot.resize(64);
    static const char *needles[2] = { "login as: ", "password: " };
    for ( int i = 0 ; i < 2 ; i++ ) {
        int len = strlen( needles[i] );
        int j;
        for ( j = 0 ; j < len ; j++ ) {
            if ( needles[i][j] != m_Snapshot[len-1-j] )
                break;
        }
        if ( j == len )
            PostMessage( WM_APP+1, i, 0 );
    }
    return TRUE;
}

There is plenty of room for improvement in this routine, much of it depending on what type of automation you are going to be using in your program. Some obvious items would include the ability to add and remove triggers as the program progresses, and regular expression matching for triggers.

Driving PuTTY

This login program is now complete save for one detail: I need a way to send my responses back to AutoPutty.

The first part of this is pretty obvious - I just need to read the data from the dialog box and post it to AutoPutty with my useful WM_COPYDATA command. This happens in my WM_APP+1 handler:

afx_msg LRESULT CPuttyDriverDlg::OnWmAppPlusOne(WPARAM wParam, LPARAM lParam)
{
    UpdateData(TRUE);
    CString msg;
    switch ( wParam ) {
    case 0 :
        msg = this->m_UserId + '\r'; break;
    case 1:
        msg = this->m_Password + '\r'; break;
    }
    if ( this->m_PuttyWindow ) {
        COPYDATASTRUCT cd;
        cd.dwData = (ULONG_PTR) 0xF00DFACE;
        cd.cbData = msg.GetLength();
        cd.lpData = (PVOID) (const char *) msg;
        ::SendMessage( this->m_PuttyWindow,
                       WM_COPYDATA,
                       (WPARAM) this->m_hWnd,
                       (LPARAM) &cd );
    }
    return 0;
}

Sending this data to AutoPutty is fine, but right now the program doesn't do anything with that message. The final piece of work is to add a WM_COPYDATA handler to window.c.

Simply grabbing the data is easy enough - the data structure that accompanies the message contains a pointer to the data and a value indicating its length. However, I have two problems I have to solve before the data is actually sent out to the to whatever device AutoPutty is connected to.

First, I have to take into account the fact that PuTTY was written to use wide characters. My driver program was built using MultiByte characters, so we have a mismatch. This means I have to do a conversion of the data from one domain to the other. This is a two step process - I call MultiByteToWideChar() once to determine how much space I need, then I allocate a buffer and call it again.

The second thing I need to do is determine what to do with the data once I've converted it. PuTTY takes all terminal input and eventually passes through a function called luni_send(). Calling this function directly from the Windows procedure seems to work just fine.

The WM_COPYDATA handler I created looks like this:

case WM_COPYDATA :
{
    COPYDATASTRUCT *cd = (COPYDATASTRUCT *) lParam;
    int wsize = MultiByteToWideChar( CP_ACP,
                                     MB_PRECOMPOSED,
                                     (LPCSTR) cd->lpData,
                                     cd->dwData,
                                     NULL,
                                     0 );
    wchar_t *buf = (wchar_t *) calloc( wsize+1, sizeof(wchar_t) );
    MultiByteToWideChar( CP_ACP,
                         MB_PRECOMPOSED,
                         (LPCSTR) cd->lpData,
                         cd->dwData,
                         buf,
                         wsize + 1 );
    if (term->ldisc)
        luni_send(term->ldisc, buf, wsize, 0);
    free( buf );
}

At this point I have a working program - it connects to my designated host, and sends the username and password of my choice to the host, connecting me to the system.

I should add a note of caution here. Automating logins is a tempting time saver, but in general this is a really bad idea. Any time you hard code credentials into a program, you open the door to all sorts of new attacks on your system.

In my demo program, the user has to enter a name and password, so nothing is hardcoded, but even this adds security holes to a system. I encourage you to think of this as a demonstration only.

Source Code

I've included the complete source code for PuttyDriver, the MFC project that controls AutoPutty. It was built with Visual Studio 2010, so you may have a little work to do if you backport it to earlier versions. My use of language features and classes should be compatible with much earlier versions - this is all very simple code.

Because PuTTY is always changing, I am not redistributing a snapshot of the version I used. Instead, I'm including before and after copies of the two source files I modified: window.c and terminal.c. If you build with Putty 0.61, you should be able to drop these two files right on top of the files included with the distribution and be on your way. With later versions of PuTTY you will have to perform an intelligent merge of the changes, which I hope will be a fairly effortless process.

Downloads

• PuttyDriver.zip. The PuttyDriver source and project. You will need to add the PuTTY project to this solution as described in the article.
• putty.zip. This contains the two PuTTY source files modified for this project. Both the original 0.61 source and my modified source are supplied. Executables are supplied as well, which may or may not work on your system.

Unfortunately, setting up email on a Linux or UNIX system is not quite as automatic as it once was. In our class we use mailx and sendmail as tools to send files from background processes or cron jobs - but mailx will typically not work out of the box. In this post I'll discuss how to get it working on an Ubuntu 11 system.

Things Have changed

Back in the day if you wanted to send mail, you simply found a handy dedicated server that was accepting incoming SMTP connections. There were thousands, and they were undiscriminating, so this was no big deal.

The invention of spam ruined that.

Now any SMTP server you find is going to require you to go through a bit of a dance in order to authenticate and prove you are not a spammer. I started off this post with the intention of showing you how to use your gmail account to access Google's SMTP servers. The process was fairly arduous, as it involved creating a certificate authority, your own certificates, and then setting up the mail server to use this authentication.

While working on this, my son Joey recommended that I just set up a free account on one of several email gateway providers, such as SendGrid or MailJet. Both services will let you access their servers and send up to 200 emails a day for free.

I took him up on it and found the process to be much simpler than using gmail, so I'll pass along the setup procedure here.

Getting an Account

Obviously, SendGrid is in business to get you to purchase a commercial account so you can send thousands of emails a day from your web site. Accordingly, the don't go out of their way to advertise their free plan. If you go to their pricing page, you will find a little tiny link to the free plan hidden at the bottom.

Setting up an account is easy, but SendGrid insists that you have a web site. For automatic verification they will need to find your email address on the site. I opted for an alternate provisioning plan in which I created a page on my site with the phrase "Sendgrid".

Once you have an account, you have free access to the SendGrid SMTP servers for up to 200 outbound messages a day. So you are ready to configure your UNIX system to take advantage of it.

Ubuntu Configuration

Configuring Ubuntu 11 to send email is fairly painless. Using the Ubuntu Software Center, you can locate and install two packages: postfix and bsd-mailx. During the install of postfix, you will get dropped into a debconf window asking you some basic configuration questions:

Figure 1 - The initial configuration screen

• General configuration: Internet
• System mail name: dogma.net

That seemed to be all I needed for basic configuration.

Postfix Configuration

To configure postfix to use SendGrid was just a matter of adding a few lines to /etc/postfix/main.cf, using your SendGrid user name and password. Note that the file probably has an existing relayhost line, this one should replace it:

smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = static:username:password
smtp_sasl_security_options = noanonymous
smtp_tls_security_level = may
relayhost = [smtp.sendgrid.net]:587

After making the changes you should restart postfix so it reads the new config options. I also start watching the mail log file so I can see if there are any problems on first use:

sudo /etc/init.d/postfix restart
sudo tail -f /var/log/mail.log

A test message sent to my cell phone arrived as a text message in just one or two seconds, with the following log messages:

Jun 26 17:02:08 ubuntu postfix/pickup[21145]: 51A5E5E1DA6: uid=1000 from=<mark>
Jun 26 17:02:08 ubuntu postfix/cleanup[21336]: 51A5E5E1DA6: message-id=<20110627000208.51A5E5E1DA6@ubuntu>
Jun 26 17:02:08 ubuntu postfix/qmgr[21146]: 51A5E5E1DA6: from=<mark@dogma.net>, size=273, nrcpt=1 (queue active)
Jun 26 17:02:08 ubuntu postfix/smtp[21338]: 51A5E5E1DA6: to=<xxxxxxxx@txt.att.net>, relay=smtp.sendgrid.net[174.36.32.204]:587, delay=0.33, delays=0.04/0.02/0.23/0.04, dsn=2.0.0, status=sent (250 Delivery in progress)
Jun 26 17:02:08 ubuntu postfix/qmgr[21146]: 51A5E5E1DA6: removed

Moving On to Better Things

Now that postfix is properly configured, I can really start taking advantage of the mail infrastructure on my system. The next obvious step is to create a .forward in my home directory, and give it my external gmail address. That external address will now be the recipient of output from cron jobs, or from at or batch. It's nice to have the mail set up as in integral part of the O/S, and if you can just make it through a little bit of setup, it's all yours.

With a limit of 200 messages a day you can still make extensive use of outbound email for system monitoring - whether it is via text to your phone or huge messages being sent to an account used for storing log files. Either way, integral email is still a great feature, almost forty years after it first showed up in UNIX.

Background

I first noticed the problem with unordered_map when I was working on the the code for my updated LZW article. I found that when running in the debugger, my program would hang after exiting the compression routine. A little debugging showed that the destructor for my hash table was taking a long time to run. And by a long time, I mean it was approaching an hour!.

This seemed pretty crazy. Destroying a hash table wouldn't seem to be a complicated task. I decided to see if I could come up with a reasonable benchmark. I wrote a test program that does a simple word frequency count. As a starter data set, I used the first one million white space delimited words in the 2010 CIA factbook, as published by Project Gutenberg. This data set yields 74,208 unique tokens.

I wrote a simple test rig that I used to test the word count program using four different containers:

• unordered_map indexed by std::string
• unordered_map indexed by std::string *
• map indexed by std::string
• map indexed by std::string *

The reason for testing with std::string * was to reduce the cost of copying strings into the hash table as it was filled, and then to reduce the cost of destroying those strings when the table was destroyed.

I ran tests against map expecting to see a pretty big difference in performance. Because map is normally implemented using a balanced binary tree structure, it has O(log(N)) performance on insertions. A sparsely populated hash table can have O(1) performance. By using fairly large data sets, I expected to see a big difference between the two.

I tried to eliminate a few obvious sources for error in my test function - and I used a template function so that I could use the same code on all the different container types:

template<class CONTAINER, class DATA>
void test( const DATA &data, const char *test_name )
{
  std::cout << "Testing container: " << test_name << std::endl;

#ifdef _DEBUG
  const int passes = 2;
#else
  const int passes = 10;
#endif
  double fill_times = 0;
  double delete_times = 0;
  size_t entries;
  for ( int i = 0 ; i < passes ; i++ ) {
    CONTAINER *container = new CONTAINER();
    std::cout << "Filling... " << std::flush;
    clock_t t0 = clock();
    for ( auto ii = data.begin() ; ii != data.end() ; ii++ )
      (*container)[*ii]++;
    double span = double(clock() - t0)/CLOCKS_PER_SEC;
    fill_times += span;
    entries = container->size();
    std::cout << " " << span << " Deleting... " << std::flush;
    t0 = clock();
    delete container;
    span = double(clock() - t0)/CLOCKS_PER_SEC;
    delete_times += span;
    std::cout << span << " " << std::endl;
  }
  std::cout << "Entries: " << entries
            << ", Fill time: " << (fill_times/passes)
            << ", Delete time: " << (delete_times/passes)
            << std::endl;
}

I didn't go overboard when it came to instrumenting this problem, I just used the timing functions built into the C++ library. On my Windows and Linux test systems, the values of CLOCKS_PER_SEC are both high enough that I'm not worried about granularity issues.

The First Results

I ran my test program in Visual C++ Release mode, using all the standard settings for a console application. For purposes of comparison, I ran the same program using g++ 4.6.1 on the same computer, booted up under Linux. For the set of 1,000,000 tokens, the results are shown below:

There are a few interesting points to take away from these tests:

• Microsoft's compiler is taking an exceptionally long time to destroy hashed containers - one order of magnitude greater than it took to create it, and two orders of magnitude greater than it takes g++ to do the same task.
• It doesn't look like constructing and destroying the strings is a big factor. Both compilers have roughly the same performance with both std::string and std::string *. Microsoft's behavior is counterintuitive, as it takes longer to construct and destroy containers using the pointer.
• The GNU compiler appears to be able to run through this exercise notably faster.

The time it takes to destroy the table is a concern - having a C++ program hang for over 3 seconds to destroy a modestly large data structure is a serious concern - particularly when the same task completes in a few milliseconds with g++.

The Pathological Results

These concerns are nothing compared to what I see when running in debug mode. Setting my Visual Studio project to Debug mode, then running the same test, yields the results shown here:

Those numbers are hard to believe. Destroying a hash table takes one millisecond when using g++. In VC++ 10, it takes almost ten minutes!

Worse, we suddenly see that hashed containers are slower than the containers built on red-black trees. Again, this just doesn't make sense.

The big problem with these numbers is that it means the debug mode of the compiler is effectively unusable for a lot of tasks. Regardless of how much testing it does, when it is this slow, it is just not useful.

A Workaround

I didn't invest the time to try debugging Microsoft's library, so I don't really know where the time is being spent. I did try a few things to speed things up, and I found one technique that helps a lot. Before including any Microsoft header files, try entering this single line in your source:

#define ITERATOR_DEBUG_LEVEL 0

With this definition in place, the delete times return to ball park of the times seen when running in release mode. Of course, you give up some debugging. I believe that an explanation of what this macro does might be found here.

In the final analysis, I think Microsoft has some serious work to to do here. The performance of their hashed containers, and to some lesser extent, the pre-C++11 associative containers, needs some serious examination. If the library is going to run this much slower than the competition, I need a good explanation why.

Source

HashTest.cpp

One of the reasons for its popularity seems to be that LZW compression is a popular homework assignment for CS students around the world. And that audience sometimes found the article to be bit of a struggle. My code was modeled on the UNIX compress program, which was written in terse C for maximum efficiency. And sometimes optimization comes at the expense of comprehension.

By using C++ data structures I can model the algorithm in a much more straightforward way - the language doesn't get in the way of a clear implementation. And after 20 years of answering puzzled queries I think I can improve on the overall explanation of just how LZW works.

In this updated look at LZW, I will first give a description of how LZW works, then describe the core C++ code that I use to implement the algorithm. I'll then walk you through the use of the algorithm with a few varieties of I/O. Finally, I'll show you some benchmarks.

I'm hoping that this version of the article will be good enough to last for another 20 years.

LZW Basics

LZW compression works by reading a sequence of symbols, grouping the symbols into strings, and converting the strings into codes. Because the codes take up less space than the strings they replace, we get compression.

My implementation of LZW uses the C++ char as its symbol type, the C++ std::string as its string type, and unsigned int as its code type. The tables of codes and strings are implemented using unordered_map, the C++ library's hash table data structure. By using the native types and standard library data structures the representation in the program is straightforward and easy to follow.

Encoding/Decoding

Rather than jumping directly into a full implementation, I'm going to work my way up to LZW one step at a time.

The first step is getting a clear understanding of how the encoding and decoding process works. As I said earlier, LZW compression converts strings of symbols into integer codes. Decompression converts codes back into strings, returning the same text that we started with.

LZW is a greedy algorithm - it tries to find the longest possible string that it has a code for, then outputs that string. The code below is not quite LZW, but it shows you the basic idea of how a greedy encoder can work:

void encode( input_stream in, output_stream out )
{
  //
  // This hash table contains a list of codes, indexed
  // by the string that corresponds to the code.
  //
  std::unordered_map<std::string,unsigned int> codes;
  //
  // There is presumably some code here that initializes
  // the dictionary with a set of codes based on whatever
  // algorithm we are implementing.
  //
  ...initialize the dictionary
  //
  // With codes in the dictionary, encoding is
  // now ready to begin.
  //
  std::string current_string;
  char c;
  while ( in >> c ) {
    current_string = current_string + c;
    if ( codes.find(current_string) == codes.end() ) {
      current_string.erase(current_string.size()-1);
      out << codes[current_string];
      current_string = c;
    }
  }
  out << codes[current_string];
}

The greedy encoder reads characters in from the uncompressed stream, and appends them one by one to the variable current_string. Each time it lengthens the string by one character, it checks to see if it still has a valid code for that string in the dictionary.

This continues until we eventually add a character that forms a string that isn't in the dictionary. So we then erase the last character from that string, and issue the code for the resulting string - the string from the previous pass through the loop.

The value of current_string is then initialized with the character that broke the camel's back, and the algorithm continues in the loop, building new strings until it runs out of input characters. At that point it outputs the last remaining code and exits.

As an example of how this would work, imagine I have the input stream ACABCA, and my code dictionary looks like this:

Decoding the stream in a system like this is very straightforward:

void decode( input_stream in, output_stream out )
{
  std::unordered_map<unsigned int,std::string> strings;
  //
  // Initialize the code table with the same set of codes and strings
  // that the encoder used for your algorithm.
  //
  ...initialize the dictionary
  //
  // With codes in the dictionary, decoding is now
  // ready to begin.
  //
  unsigned int code;
  while ( in >> code )
    out << strings[code];
}

Remember, the decoder shown above is just a hypothetical sample - we're still working our way up to the full LZW decoder.

The LZW Encoder

The encoder shown above works okay, but there is one missing ingredient: management of the code dictionary. If you think about it, you'll see that we only achieve reasonable compression when we are able to build up longer strings and find them in the dictionary. Building a useful dictionary is referred to in the data compression world as modeling.

But our management of the dictionary is constrained by an important requirement: the encoder and decoder both have to be working with the same copy of the dictionary. If they have different dictionaries, the encoder might send a string that the decoder can't resolve.

Some data compression algorithms solve this problem by using a predefined dictionary that both the encoder and the decoder know in advance. But LZW builds a dictionary on the fly, using an adaptive method that ensures both the encoder and decoder are in sync.

LZW manages this in an effective and provably correct fashion. First, both the encoder and decoder initialize the dictionary with all possible single digit strings. For the compressor, that looks like this:

for ( unsigned int i = 0 ; i < 256 ; i++ )
    codes[std::string(1,(char)i)] = i;

This insures that we can encode all possible streams. No matter what, we can always break a stream down into single digits and encode these, knowing that the decoder has the same strings in its dictionary with values 0-255.

Then comes the key component of the LZW algorithm. If you go back to the greedy encoding loop above, you'll see that I keep adding input symbols to a string until I find a string that isn't in the dictionary. This string has the characteristic of being composed of a string that currently exists in the dictionary, with one additional character.

LZW then takes that new string and adds it to the dictionary, creating a new code. The strings are added to the table with code values that increment by one with each new entry.

The resulting code is just a slightly modified version of the encoder that I listed above. It still only outputs codes for values that are in the dictionary, but now the dictionary is being updated with a new string every time an existing code is sent:

void compress( input_stream in, output_stream out )
{
  std::unordered_map<std::string,unsigned int> codes;
  for ( unsigned int i = 0 ; i < 256 ; i++ )
    codes[std::string(1,(char)i)] = i;
  unsigned int next_code = 257;
  std::string current_string;
  char c;
  while ( in >> c ) {
    current_string = current_string + c;
    if ( codes.find(current_string) == codes.end() ) {
      codes[ current_string ] = next_code++;
      current_string.erase(current_string.size()-1);
      out << codes[current_string];
      current_string = c;
    }
  }
  out << codes[current_string];
}

The code above constitutes a more or less complete LZW encoder. I've only made a couple of additions to the previous encoder:

• The initialization of codes 0-255 with all possible single character strings.
• The insertion of the newly discovered string into the string table, generating a new code.

(One item of note in this code: you might wonder why next_code is initialized to 257, when 256 is the first free code. This is because I reserve code 256 for an EOF marker. More on this in a later section.)

Just to make sure this all adds up, I'll walk through the steps the encoder takes as it processes a string from a simple two letter alphabet: ABBABBBABBA. There are a lot of steps shown below, but working through the process in detail is a great way to be sure you understand it:

More importantly, as the dictionary grows, it starts to hold longer and longer strings. And the longer the string, the the more compression we can get. If the algorithm starts emitting integer codes for strings of length 10 or more, there is no doubt that we are going to get good compression.

As an example of how this works on real data, here are some entries from the dictionary created when compressing Alice's Adventures in Wonderland:

34830 : 'even\n'
34831 : '\nwith t'
34832 : 'the dr'
34833 : 'ream '
34834 : ' of Wo'
34835 : 'onderl'
34836 : 'land'
34837 : 'd of l'
34838 : 'long ag'
34839 : 'go:'

These strings have an average length of almost six characters. If we are writing the integer codes to a file using 16 bit binary integers, these entries offer the possibility of 3:1 compression.

The word adaptive is used to describe a compression algorithm that adapts to the type of text it is processing. LZW does an excellent job of this. If a string is seen repeatedly in the text, it will show up in longer and longer entries in the dictionary. If a string is seen rarely, it will not be the foundation for a large batch of longer strings, and thus won't waste space in the dictionary.

The LZW Decoder

The change made to the basic encoder to accommodate the LZW algorithm was really very simple. One small batch of code that initializes the dictionary, and another few lines of code to add every new unseen string to the dictionary.

As you might suspect, the changes to the decoder will be fairly simple as well. The first change is that the dictionary must be initialized with the same 256 single-symbol strings that the encoder uses.

Once the decoder starts running, each time it reads in a code, it must add a new value to the dictionary. And what is that value? The entire content of the previously decoded string, plus the first letter of the currently decoded string. This is exactly what the encoder does to create a new string, and the decoder must following the same steps:

void decompress( input_stream in, output_stream out )
{
  std::unordered_map<unsigned int,std::string> strings;
  for ( int unsigned i = 0 ; i < 256 ; i++ )
    strings[i] = std::string(1,i);
  std::string previous_string;
  unsigned int code;
  unsigned int next_code = 257;
  while ( in >> code ) {
    out << strings[code];
    if ( previous_string.size() )
      strings[next_code++] = previous_string + strings[code][0];
    previous_string = strings[code];
  }
}

I won't do a walk-through of the the decoder - you should be able to take the codes output from the encoder, shown above, and run them through the decoder to see that the output stream is what we expect.

The important thing is to understand the logic behind the decoder. When the encoder encounters a string that isn't in the dictionary, it breaks it into two pieces: a root string and an appended character. It outputs the code for the root string, and adds the root string + appended character to the dictionary. It then starts building a new string that starts with the appended character.

So every time the decoder uses a code to extract a string from the dictionary, it knows that the first character in that string was the appended character of the string just added to the dictionary by the encoder. And the root of the string added to the dictionary? That was the previously decoded string. This line of code implements that logic:

    strings[next_code++] = previous_string + strings[code][0];

It adds a new string to the dictionary, composed of the previously seen string, and the first character of the current string. Thus, the decoder is adding strings to the dictionary just one step behind the encoder.

You might note one curious point in the decoder. Instead of always adding the string to the dictionary, it is only done conditionally:

if ( previous_string.size() )
  strings[next_code++] = previous_string + strings[code][0];

The only time that previous_string.size() is 0 is on the very first pass through the loop. And on the first pass through the loop, we don't have a previous string yet, so the decoder can't build a new dictionary entry. Again, the decoder is always one step behind the encoder, which is a key point in the next section, which puts the final touches on the algorithm.

The Catch

So far the LZW algorithm we've seen seems very elegant - that's a characteristic we associate with algorithms that can be expressed in just a few lines of code.

Unfortunately, there is one small catch in this perceived elegance - the algorithm as I've shown it to you has a bug.

The bug in the algorithm relates to the fact that the encoder is always one step ahead of the decoder. When the encoder adds a string with code N to the table, it sends enough information to the decoder to allow the decoder to figure out the value of the string denoted by code N-1. The decoder won't know what the value of the string corresponding to code N is until it receives code N+1.

This makes sense if you recall the key line of code from the decoder. It calculates the value of the string encoded by N-1 by looking at the string it received on the previous iteration, plus the first character of the current string. And that current string is the one that was sent after encoding N.

So how can this get us in trouble? The encoder is always one entry ahead of the decoder - it has entry N in its dictionary, and the decoder has entry N-1. So if the encoder ever sends code N, the decoder will look in its table and come up empty-handed, unable to do its job of decoding.

A simple example will show you how this can happen. Let's look at the state of the encoder after it has sent the first five symbols in a stream: ABABA:

Yes, if the next two characters in the stream are BA, the next code output by the encoder will be 259, and the decoder will be lost.

In general, this can happen when a dictionary entry exists that consists of a string plus a character, and the encoder encounters the sequence string+character+string+character+string. In the example above, the value of string is A, and the value of character is B. After the encoder counters AB, it has string+character in the dictionary, so if the following sequence is ABABA, we will emit code N.

Whether this is likely to happen or not is not too important, what is important is that it most definitely can happen, and the decoder has to be aware of it. And it will happen repeatedly in the pathological case: a stream that consists of a single symbol, repeated on end.

The good news is that the problem is easily solved. When the decoder receives a code, and finds that this code is not present in its dictionary, it knows right away that the code must be the one that it will add next to its decoder. And because this only happens when we are encoding the sequence discussed above, the decoder knows that instead of using this value for that code:

    strings[next_code++] = previous_string + strings[code][0];

it can instead use this value:

    strings[ code ] = previous_string + previous_string[0];

The result of this is the insertion of just two lines of code at the start of the decompress loop, giving a loop that now looks like this:

while ( in >> code ) {
  if ( strings.find( code ) == strings.end() )
    strings[ code ] = previous_string + previous_string[0];
  out << strings[code];
  if ( previous_string.size() )
    strings[next_code++] = previous_string + strings[code][0];
  previous_string = strings[code];
}

And with that, you have a complete implementation of the LZW encoder and decoder.

Implementation

Now that I've shown you the algorithm, the next step is to take that code and add turn it into a working program. Without changing the algorithm itself, I'm going to take you through four different customizations that work as follows:

• LZW-A reads and writes code values rendered in text mode, which is great for debugging. It means you can view the output of the encoder in a text editor.
• LZW-B reads and writes code values as 16-bit binary integers. This is fast and efficient, and usually results in significant data compresion.
• LZW-C reads and writes code values as N-bit binary integers, where N is determined by the maximum code size. Performing I/O on codes that are not aligned on byte boundaries complicates the code somewhat, but allows for greater efficiency and better compression.
• LZW-D reads and writes code values as variable-length binary integers, starting with 9-bit codes and gradually increasing as the dictionary grows. This gives the maximum compression.

Before launching into these implementations, the code I showed above needs some minor tweaking to solve a couple of problems.

The first problem we have to deal with is the ever-expanding dictionary. In the algorithm I've presented, we keep adding new codes to the dictionary without end. This needs to be changed for a couple of reasons.

First, we don't have unlimited memory, so the dictionary simply can't grow forever. Second, practical experience shows that compression ratios don't improve as dictionary sizes grow without bound. As the dictionary grows, code sizes get larger and larger, and so they take up more space in the compressed stream, which can reduce compression efficiency.

To resolve this problem, I just add an additional argument to the encoder and decoder that sets the maximum code value that will be added to the dictionary. The function signatures now look like this:

void compress( input_string input,
               output_stream output,
               const unsigned int max_code = 32767 );
void decompress( input_string input,
                 output_stream output,
                 const unsigned int max_code = 32767 );

Implementing it means one small change in the encoder:

if ( next_code <= max_code )
  codes[ current_string ] = next_code++;

And a corresponding change in the decoder:

if ( previous_string.size() && next_code <= max_code )
  codes[ current_string ] = next_code++;

Input and Output

Finally, I need to give the algorithm a decent way to perform input and output - and this is where C++ offers a huge amount of help.

When writing generic compression code that you intend to use in multiple contexts, one of the more difficult things to deal with is I/O. People using your code might want to compress data in memory, stored in files, or streaming in from sockets or other sources. Some input data sources might be of unknown length (data coming from a TCP socket, for example), while others will be of a prescribed length. Back in the days of C, it was particularly difficult to make your compression code both generic, so it would work with all types of data streams, and efficient, so that I/O doesn't take any more time than it has to.

With the advent of C++, we have a new tool that can help in this quest - templates. Templates are designed to solve this problem in an efficient way, and I take advantage of this in my sample code. The code below shows the final version of the compressor and decompressor that are are used in all four versions of the implementation. There are two final changes made to the routines shown previously. First, both C++ functions are now function templates, parameterized on the the types being used for input and output. Second, the actual input and output is done through four newly introduced template classes:

template<class INPUT, class OUTPUT>
void compress( INPUT &input, OUTPUT &output, const unsigned int max_code = 32767 )
{
  input_symbol_stream<INPUT> in( input );
  output_code_stream<OUTPUT> out( output, max_code );

  std::unordered_map<std::string, unsigned int> codes( (max_code * 11)/10 );
  for ( unsigned int i = 0 ; i < 256 ; i++ )
    codes[std::string(1,i)] = i;
  unsigned int next_code = 257;
  std::string current_string;
  char c;
  while ( in >> c ) {
    current_string = current_string + c;
    if ( codes.find(current_string) == codes.end() ) {
      if ( next_code <= max_code )
        codes[ current_string ] = next_code++;
      current_string.erase(current_string.size()-1);
      out << codes[current_string];
      current_string = c;
    }
  }
  if ( current_string.size() )
    out << codes[current_string];
}

template<class INPUT, class OUTPUT>
void decompress( INPUT &input, OUTPUT &output, const unsigned int max_code = 32767  )
{
  input_code_stream<INPUT> in( input, max_code );
  output_symbol_stream<OUTPUT> out( output );

  std::unordered_map<unsigned int,std::string> strings( (max_code * 11) / 10 );
  for ( int unsigned i = 0 ; i < 256 ; i++ )
    strings[i] = std::string(1,i);
  std::string previous_string;
  unsigned int code;
  unsigned int next_code = 257;
  while ( in >> code ) {
    if ( strings.find( code ) == strings.end() )
      strings[ code ] = previous_string + previous_string[0];
    out << strings[code];
    if ( previous_string.size() && next_code <= max_code )
      strings[next_code++] = previous_string + strings[code][0];
    previous_string = strings[code];
  }
}

What exactly is the effect of implementing this algorithm using a pair of function templates, parameterized on the the types of the input and output objects? What this means is that you can call these compression routines with any type of I/O object you can throw at them. It can work with C++ iostreams, C FILE * objects, raw blocks of memory, whatever you want.

But there's a catch to that flexibility - you have to implement some basic I/O routines for whatever type you are using. Fortunately, this is not too hard.

The actual I/O that is done in the compression routines is defined by four template classes I created. These classes are defined in lzw_streambase.h. These classes don't have implementations, but they do define the methods you need to implement to work with the compressor and decompressor. The four classes are:

• input_symbol_stream<T>
• ouput_symbol_stream<T>
• input_code_stream<T>
• output_code_stream<T>

The first two classes are the symbol input and output classes. These are normally going to be very simple implementations, as they just have to read single characters to and from streams, while checking for errors or ends of streams. I use the same versions of these classes in all four implementations, so the code in lzw-a.h is unchanged in the other three header files.

The input_symbol_stream<T> class has one member function: the extraction operator, which reads a character from the stream and returns a boolean true or false. You'll see later in this section that the implementation of this for types such as std::istream is trivial.

template<typename T>
class input_symbol_stream
{
public :
    input_symbol_stream( T & );
    bool operator>>( char &c );
};

The output_symbol_stream<T> class uses the insertion operator to write strings instead of individual characters - because that is what is stored in the dictionary. The C++ std::string class makes a perfectly good container for any variety of symbols, including binary data, and unlike the alternative vector<char>, it comes with hash functions and iostream operators.

template<typename T>
class output_symbol_stream
{
public :
    output_symbol_stream( T &  );
    void operator<<( const std::string &s );
};

The input_code_stream<T> class reads codes, normally unsigned integers, from some type of stream. In my implementations, this class also returns false if it encounters the EOF_CODE in the stream of incoming codes. Removing the responsibility for EOF detection from the decompressor makes the code a bit simpler and more versatile.

The formatting of the integer is entirely up to the implementor, but the most common approach will probably be variable length codes ranging from 9 to 16 or so bits.

template<typename T>
class input_code_stream
{
public :
    input_code_stream( T &, unsigned int );
    bool operator>>( unsigned int &i );
};

The output_code_stream<T> class writes codes, usually unsigned integers, to some type of stream. Whatever class you implement for this function must agree with the implementation for input_code_stream<T>.

template<typename T>
class output_code_stream
{
public :
    output_code_stream( T &, unsigned int );
    void operator<<( const unsigned int i );
};

You can see that at the top of the compressor and decompressor, I instantiate objects of these types, then use the standard insertion and extraction operators to read and write from these objects.

LZW-A

In my sample windows program, I include lzw_streambase.h and lzw.h, which accounts for all of the code you have seen so far. I have the following lines that perform compression and decompression:

std::ifstream in( name, std::ios_base::binary );
std::ofstream lzw_out( temp_name_lzw, std::ios_base::binary );
compress( (std::istream &) in, (std::ostream&) lzw_out, pDlg->m_MaxCodeSize );
.
.
.
std::ifstream lzw_in( temp_name_lzw, std::ios_base::binary );
std::fstream out( temp_name_out,
                  std::fstream::in    |
                  std::fstream::out   |
                  std::fstream::binary );
decompress( (std::istream &) lzw_in, (std::ostream&) out, pDlg->m_MaxCodeSize );

If I try to build this project as-is, I get a nasty list of eight linker errors:

Visual Studio 10 Error Messages

It's tempting to try to read characters using the ifstream extraction operator, as in m_impl >> c, but that operator skips over whitespace, so we don't get an exact copy of the input stream. Using get() works around this problem. Below is the complete definition of input_symbol_stream<std::istream> used in all four LZW implementations in this article:

template<>
class input_symbol_stream<std::istream> {
public :
    input_symbol_stream( std::istream &input )
        : m_input( input ) {}
    bool operator>>( char &c )
    {
        if ( !m_input.get( c ) )
            return false;
        else
            return true;
    }
private :
    std::istream &m_input;
};

Using the insertion operator to output strings seems to work properly, even when the strings contain binary data, so the implementation of the class used to output symbols is as simple as we could hope for. Again, this exact code is used in all four implementations in this article:

template<>
class output_symbol_stream<std::ostream> {
public :
    output_symbol_stream( std::ostream &output )
        : m_output( output ) {}
    void operator<<( const std::string &s )
    {
        m_output << s;
    }
private :
    std::ostream &m_output;
};

LZW-A prints the text values of integers to the output stream, and reads them back in that format. This is not efficient at all, but it is a great aid in debugging. If you are having a problem with the algorithm, this provides a nice way to examine your stream. The implementation of this is very simple - just use the std::ostream insertion operator, and follow each code by a newline so it can be properly parsed on input, as well as be easily loaded into a text editor.

One important thing to notice in this class: the presence of a destructor that prints the EOF_CODE. Since this object goes out of scope as the compressor exits, this insures that every code stream will end with this special code. Putting the onus on the I/O routines to deal with EOF issues simplifies the algorithm itself. (It also means that you can implement versions of LZW that don't use an EOF in the code stream.)

template<>
class output_code_stream<std::ostream> {
public :
    output_code_stream( std::ostream &output, const unsigned int )
        : m_output( output ) {}
    void operator<<( unsigned int i )
    {
        m_output << i << '\n';
    }
    ~output_code_stream()
    {
        *this << EOF_CODE;
    }
private :
    std::ostream &m_output;
};

The corresponding version of the input class just reads in the white-space separated codes. If there is an error or an EOF_CODE encountered in the stream, the extraction operator returns false, which allows the decompressor to know when it is time to stop processing.

template<>
class input_code_stream<std::istream> {
public :
    input_code_stream( std::istream &input, unsigned int )
        : m_input( input ) {}
    bool operator>>( unsigned int &i )
    {
        m_input >> i;
        if ( !m_input || i == EOF_CODE )
            return false;
        else
            return true;
    }
private :
    std::istream &m_input;
};

By including lzw-a.h along with the other two header files, I can now create a program that compiles, links, and is able to test the algorithm. Using my UNIX test program, I compress the demo string from earlier in this article, and I see the output as it is sent directly to stdout:

Compressing ABBABBBABBA

LZW-B

The header file lzw-b.h implements specialized classes that replace the text-mode output of the codes in lzw-a.h with binary codes stored in a short integer - two bytes.

The classes that read and write symbols are unchanged, but reading and writing codes has to change in order to do this new binary output.

Writing the codes to std::ostream as binary values requires breaking the integer code into two bytes and writing the bytes one at a time. There are more efficient ways to write the complete short integer in one function call, but they raise code portability problems, as we don't always know what order bytes will be written in.

Like the code stream output object in lzw-a.h, this version of the code output class has a destructor that outputs an EOF_CODE value:

template<>
class output_code_stream<std::ostream> {
public :
    output_code_stream( std::ostream &output, const unsigned int )
        : m_output( output ) {}
    void operator<<( unsigned int i )
    {
        m_output.put( i & 0xff );
        m_output.put( (i>>8) & 0xff);
    }
    ~output_code_stream()
    {
        *this << EOF_CODE;
    }
private :
    std::ostream &m_output;
};

Reading the codes requires reading the two bytes that make up the short integer, then combining them. While reading, if the routine detects an EOF_CODE, it returns false, which tells the decompressor to stop processing. It also returns false if there is an error on the input code stream.

template<>
class input_code_stream<std::istream> {
public :
    input_code_stream( std::istream &input, unsigned int )
        : m_input( input ) {}
    bool operator>>( unsigned int &i )
    {
        char c;
        if ( !m_input.get(c) )
            return false;
        i = c & 0xff;
        if ( !m_input.get(c) )
            return false;
        i |= (c & 0xff) << 8;
        if ( i == EOF_CODE )
            return false;
        else
            return true;
    }
private :
    std::istream &m_input;
};

The most exciting thing about lzw-b.h is that you can now see data compression taking place. The figure below shows a sample run of this implementation against the Canterbury Corpus, a standard set of files used to test compression. A run with my Windows test program shows that the files are compressing quite nicely:

Compressing the Canterbury Corpus with lzw-b.h

LZW-C

The third I/O implentation, defined in lzw-c.h, writes binary codes like lzw-b.h, but with one crucial difference. Instead of being hard coded to 16 bit codes, lzw-c.h determines the maximum code size needed based on the maximum code value passed as an argument to compress() and decompress(). It then writes codes based on that width, which will normally be something in the range of 9-18 bits wide.

Since these values are not aligned with byte boundaries, there are some issues writing them to streams that expect to read and write bytes. However, it is definitely worth all the bit shifting, ORing, and ANDing, because when the size is 12 bites, we are going to save four bits per code when compared to using lzw-b.h. But every read and write potentially starts somewhere in the middle of a byte, so the I/O classes have to do some extra work - mostly involved with shifting bits to the correct position in the output stream.

Note that the code to read and write symbols is unchanged from lzw-a.h and lzw-b.h.

Many of the CS students who read my earlier article on LZW ran into a brick wall when they started trying to understand the code that performs I/O on codes of variable bit lengths. Obviously, writing 11 bit codes when your file system is oriented around eight-bit bytes involves a lot of bit twiddling, and I'm afraid that many novices are woefully deficient in this department. Not just in understanding the bitwise operators in C, such as shifting, masking, etc., but in understanding binary arithmetic in general.

That's why I've structured the code and this article a bit differently this time around. If the I/O operations in lzw-c.h and lzw-d.h are bewildering, well, no worries. They have absolutely nothing to do with the LZW algorithm itself. You can investigate and explore the algorithm completely using lzw-a.h and lzw-b.h, and just forget about the last two I/O implementations. They provide additional efficiency, but as I have said, have nothing to do with the algorithm itself.

Further, once you use lzw-a.h to debug and understand the algorithm, you can certainly plug in lzw-c.h and lzw-d.h and take advantage of their improved compression, even if you don't follow all the code.

It might be appropriate to add a sidebar or another section to explain the variable bit length I/O in detail, but this article is quite long already, and there are numerous other resources for the interested reader to explore the details. (But if you find yourself deficient in this area, you owe it to yourself to hit the books and get to the point where these operations make sense. This won't be the last time you need to understand bitwise operators.)

For those who are ready to tackle this more complicated I/O procedure, we will look first at the output_code_stream<std::ostream> class. Here, the first thing to understand is that the constructor has to initialize the number of bits in the code. This value is calculated from the max_code parameter, and is stored in member m_code_size, where it is used frequently.

Next, the insertion operator. Output of codes proceeds as follows. Member m_pending_bits tells us how many bits are pending output while sitting in member m_pending_output. These bits are right justified, and the count will always be less than eight. When the new code is written, it is inserted into m_pending_output after being left shifted so it will be laid down just past the pending bits. After doing that, we presumably have some bytes to output - the exact number depends on various factors. The flush() routine is called, and it flushes all complete bytes out. When it completes, there can be anywhere from zero to seven bits still waiting to be output, and they will be right justified in m_pending_output.

In the destructor, we output an EOF_CODE, and then do a flush as well. But in this case, we flush all possible bits, not just the complete bytes. There are two good reasons for this. First, we don't care if the last bits that are flushed out are only part of a code - the code will be EOF_CODE, and that is the last one. And second, if we don't flush those final bits out in the destructor, they will never be sent to the output stream. This means the decoder will not see those bits, and we will most likely break the decompress process.

template<>
class output_code_stream<std::ostream>
{
public :
    output_code_stream( std::ostream &out, unsigned int max_code )
        : m_output( out ),
          m_pending_bits(0),
          m_pending_output(0),
          m_code_size(1)
    {
        while ( max_code >>= 1 )
            m_code_size++;
    }
    ~output_code_stream()
    {
        *this << EOF_CODE;
        flush(0);
    }
    void operator<<( const unsigned int &i )
    {
        m_pending_output |= i << m_pending_bits;
        m_pending_bits += m_code_size;
        flush( 8 );
    }
private :
    void flush( const int val )
    {
        while ( m_pending_bits >= val ) {
            m_output.put( m_pending_output & 0xff );
            m_pending_output >>= 8;
            m_pending_bits -= 8;
        }
    }
    std::ostream &m_output;
    int m_code_size;
    int m_pending_bits;
    unsigned int m_pending_output;
};

Like the output code class, the input code class has to calculate the code size for this decompression based on the max_code value passed in the function call.

When an attempt is made to read a code, there must be a minimum of m_code_size bits in member m_pending_input. If there aren't, new bytes are read in one at a time, and inserted into m_pending_input after having been shifted left the appropriate amount. Once m_pending_input contains at least m_code_size bits, the code is extracted from m_pending_input using the appropriate mask, the count in m_pending_input is reduced, and m_pending_input is shifted right by m_code_size bits.

template<>
class input_code_stream<std::istream>
{
public :
    input_code_stream( std::istream &in, unsigned int max_code )
        : m_input( in ),
          m_available_bits(0),
          m_pending_input(0),
          m_code_size(1)
    {
        while ( max_code >>= 1 )
            m_code_size++;
    }
    bool operator>>( unsigned int &i )
    {
        while ( m_available_bits < m_code_size )
        {
            char c;
            if ( !m_input.get(c) )
                return false;
            m_pending_input |= (c & 0xff) << m_available_bits;
            m_available_bits += 8;
        }
        i = m_pending_input & ~(~0 << m_code_size);
        m_pending_input >>= m_code_size;
        m_available_bits -= m_code_size;
        if ( i == EOF_CODE )
            return false;
        else
            return true;
}
private :
    std::istream &m_input;
    int m_code_size;
    int m_available_bits;
    unsigned int m_pending_input;
};

The table below shows the results of a test run comparing LZW-B and LZW-C run with a maximum code of 4095. With this maximum value, all codes fit in a 12-bit integer. Since LZW-B will use a 16-bit integer to store the code values, and LZW-C will use 12-bits, there should be a 4:3 ratio between the ratio of the file sizes when compressed using the two algorithms, and this looks to be the case:

LZW-D

The code in lzw-d.h represents the final and most efficient version of I/O for the LZW code streams. It builds on the code in lzw-c.h - at its core it is a variable bit-length I/O stream. However, there is one crucial difference from lzw-c.h: the code I/O in lzw-d.h starts at the smallest possible code size, nine bits, and increases the code size as needed, until it reaches the maximum value for this compression session. The maximum value is the parameter passed in to the invocation of compress() or decompress().

The logic behind this is pretty simple. Even if we are going to use 16-bit codes in an LZW program, when the program first starts, the maximum possible code the program can emit is 256, which only needs nine bits to encode. And each time we output a new symbol, that maximum possible code value only increases by one, which means that the first 256 codes output by the encoder can all fit in nine bits.

So the LZW-D encoder starts encoding using nine-bit code widths, and then bumps the value to ten as soon as the highest possible output code reaches 512. This process continues, incrementing the code size until the maximum code size is reached. At that point the code size stays fixed, as no new codes are being added to the dictionary.

The decoder follows exactly the same process - reading in the first code with a width of nine bits, then bumping to ten when the maximum possible input code reaches 512.

The code for this class is built on that from lzw-c.h, with some added complexity. Due to its increasing length, and the fact that it doesn't add too much to the discussion of LZW, I've omitted the listing, and instead refer you to the download available at the end of the article.

The Windows Test Program

When you develop compression code, there are a few different common tasks you are likely to want to perform:

• Check your code for correctness, often through bulk testing.
• Check your compression ratios against standard benchmarks.
• Analyze your program's performance so as to make it more efficient and locate bottlenecks.

My Windows app is designed to help with all of these tasks. It basically allows you to select a single directory, set a maximum code size, then perform compression and decompression of all the files in the directory. An optional checkbox lets you include files in all directories under the test directory as well.

The application was built using Visual Studio 10, and it is a simple MFC Dialog-based application. It allows you to select a base directory, a maximum code size, and then compress all the files in that directory. If you select the recursion check box, you will also compress all the files in the entire tree of subdirectories below it.

Each file is compressed to a temporary location, then decompressed in a temporary location. The size of the compressed file is saved, and then a comparison is done to ensure that the original and expanded files are identical.

To help with data collection, after running a test, you can press the copy button and get the results of the test stuffed into your clipboard. Although it isn't visible in the display, the data stored in your clipboard includes the full path name of the original file, not just the basename.

This Visual Studio project takes advantage of a number of C++11 features, and as a result it will need some modification to work with earlier versions. Any version that supports unordered_map can be made to build without too many changes. And if you are going way back in time, you could replace unordered_map with map.

As shipped, the test program uses lzw-d.h. To use any of the other three other versions of I/O discussed in this article, just modify the include file selected at the top of LzwTestDlg.cpp. The figure below shows what the app looks like after running through some data:

The Windows test app after a test run

Copying the data into a spreadsheet

The Linux Test Program

The LZW code is platform independent, and will build and run just fine on UNIX or Linux systems. The Linux test program, lzw.cpp, allows you to compress or decompress files from the command line. It builds just fine with g++ 4.5, as long as you use the -std=c++0x switch to turn on the latest language features. Compiling with earlier versions will require a few minor modifications.

The command line interface to the test program is not too complicated, and is probably best documented by looking at the usage output:

mrn@ubuntu:~/LzwTest$ g++ -std=c++0x lzw.cpp -o lzw
mrn@ubuntu:~/LzwTest$ ./lzw
Usage:
lzw [-max max_code] -c input output #compress file input to file output
lzw [-max max_code] -c - output     #compress stdin to file otuput
lzw [-max max_code] -c input        #compress file input to stdout
lzw [-max max_code] -c              #compress stdin to stdout
lzw [-max max_code] -d input output #decompress file input to file output
lzw [-max max_code] -d - output     #decompress stdin to file otuput
lzw [-max max_code] -d input        #decompress file input to stdout
lzw [-max max_code] -d              #decompress stdin to stdout
mrn@ubuntu:~/LzwTest$

Like the Windows test program, the command line program is built by default with lzw-d.h. Replacing this algorithm with any of the three others requires a minor change to the source code.

With the default build, the program produces output nearly identical to UNIX compress. The one difference is that UNIX compress monitors the compression ratio after the dictionary is full, and clears the dictionary if the ratio starts to deteriorate (which it almost always does.) I include a benchmark program that tests UNIX compress against the command line test program, and the results show that for small files, the file size is almost identical:

mrn@ubuntu:~/LzwTest$ ./benchmark.sh 65535 16 canterbury | head -n 15 | column -t
Filename                 Original-size  LZW-size  Compress-size
--------                 -------------  --------  -------------
canterbury/aaa.txt       33406          320       321
canterbury/alice29.txt   152089         62247     62247
canterbury/alphabet.txt  100000         3052      3053
canterbury/asyoulik.txt  125179         54989     54990
canterbury/a.txt         1              3         5
canterbury/bib           111261         46527     46528
canterbury/bible.txt     4047392        1417735   1377093
canterbury/book1         768771         317133    317133
canterbury/book2         610856         247593    251289
canterbury/cp.html       24603          11315     11317
canterbury/E.coli        4638690        1213579   1218349
canterbury/fields.c      11150          4963      4964
canterbury/geo           102400         77777     77777

You can see in this test that LZW-D and UNIX compress perform nearly identically for all but the largest files in the test sample. If I modify UNIX compress to not monitor compression ratios, the difference seen with larger files goes away:

mrn@ubuntu:~/LzwTest$ ./benchmark.sh 65535 16 canterbury | head -n 15 | column -t
Filename                 Original-size  LZW-size  Compress-size
--------                 -------------  --------  -------------
canterbury/aaa.txt       33406          320       321
canterbury/alice29.txt   152089         62247     62247
canterbury/alphabet.txt  100000         3052      3053
canterbury/asyoulik.txt  125179         54989     54990
canterbury/a.txt         1              3         5
canterbury/bib           111261         46527     46528
canterbury/bible.txt     4047392        1417735   1417735
canterbury/book1         768771         317133    317133
canterbury/book2         610856         247593    247593
canterbury/cp.html       24603          11315     11317
canterbury/E.coli        4638690        1213579   1213579
canterbury/fields.c      11150          4963      4964
canterbury/geo           102400         77777     77777

That provides some support for the notion that the algorithm shown here behaves properly.

Your Program

If you want to build your own program and use these classes, all you need is a C++11 compiler, or an earlier version and a willingness to make a few changes.

To use the classes, include in order lzw_streambase.h, one of the four implementation files for iostreams, preferably lzw-d.h, and finally, lzw.h. Because the significant code in these files is all implemented as template functions or classes, there is no library to include in your project, and no C++ source you have to compile separately.

All of the code in these header files has been hoisted into the lzw namespace, so you will either have to explicitly use the namespace when you invoke compress() and decompress(), or insert this line into your program:

using namespace lzw;

One thing to note about the I/O routines I have defined. The template functions are specialized on std::istream and std::ostream. If you innocently pass in an object such as an std::ifstream, you will get compile time errors. This is because C++ template matching is done on a very strict basis - the compiler won't generally try to figure out that std::ifstream is derived from std::istream, and use the existing class. So instead, you will need to cast your arguments to the types defined in the header files. (Or write your own implementations.)

Your rights to use this code are covered by my Liberal Code Use Policy. As I have mentioned before, this is teaching code, if you decide to use it in a production system, there are many optimizations you might want to perform.

Benchmarks

So how does LZW do when it comes to compression? LZW's original strength was its combination of good compression ratios with high speed compression. The UNIX compress program is still nice and fast, and Terry Welch's original application for LZW was in disk controllers. Because my program is a teaching program, it won't be nearly as fast as compress, but it's still useful to compare it to the de facto standard for lossless compression: the deflate algorithm.

We can compare LZW against deflate by a small modification of my benchmark script that uses gzip instead of compress. The table below shows the average compression ratios for the files in the canterbury corpus when compressed using maximum code widths of 15-18 bits. (The ratio is defined as 100*compressed_size/uncompressed_size, so 0% is perfect compression and 100% is no compression.)

Variations

There are many variations on the code I've presented here that make sense.

One obvious change is to eliminate the special EOF_CODE used to delimit the end of the code stream. If the code stream is a file or other stream with an inherent EOF condition, there is no need for an EOF_CODE - simply reaching the end of the input stream will properly signal the end of the decoded material. Freeing up this one code will make a microscopically small improvement in the compression ratios of the product.

If you want to mimic the output of the compress program, you need to remove the EOF_CODE, and replace it with a CLEAR_CODE that has a value of 256. The compress program monitors the compression ratios it achieves after its dictionary is full, and when the ratio starts to decay, it issues the CLEAR_CODE. That code tells the decoder to clear its dictionary and make a fresh start with new nine-bit codes.

Once you get the hang of LZW, a good exercise to make sure you have it working properly is to create a GIF encoder and decoder. GIF uses LZW to losslessly compress images with a constrained palette, and after all these years is still somewhat of a standard on the web.

History

Usually the history lesson on an algorithm is at the start of the article, but this is a how-to piece, and I feel like the trip down memory lane is not as important as understanding how the algorithm works.

The roots of LZW were set down in 1978 when Jacob Ziv and Abraham Lempel published the second of their two seminal works on data compression, "Compression of Individual Sequences via Variable-Rate Coding". This paper described a general approach to data compression that involved building dictionaries of previously seen strings.

Ziv and Lempel's work was targeted at an academic audience, and it wasn't truly popularized until 1984 when Terry Welch published A Technique for High-Performance Data Compression. Welch's paper took the somewhat abstract Information Theory work of Ziv and Lempel and reduced it to practice in such a way that others could easily implement it.

UNIX compress was probably the first popular program that used LZW compression, and it very quickly became a standard utility on UNIX systems. The freely available code for compress was incorporated into ARC, one of the first archiving programs for PCs. In addition, the algorithm was used in the GIF file format, originally created by Compuserve in 1987.

LZW's popularity waned in the 1990s for two important reasons. First, Unisys began enforcing their patents that covered LZW compression, demanding and receiving royalties from various software companies. Not only did this make developers think twice about the liability they could incur while using LZW, it resulted in a general public relations backlash against using patented technology.

Secondly, the LZW algorithm was eclipsed on the desktop by deflate, as popularized by PKZIP. Not only did deflate outperform LZW, it was unencumbered by patents, and eventually had a very reliable and free open source implementation in zlib, a library written by a team lead by Marc Adler and Jean-loup Gailly. I don't know if there is any way to actually quantify this, but I think one could speculate that zlib is currently installed on more computer systems than any other software package in existence.

So LZW has settled down to an existence out of the limelight. It is still an important algorithm, used in quite a few file formats, and as this article shows, its simplicity makes it an excellent learning tool.

Downloads

• LzwTest.zip - source for the Windows test app.
• LzwExe.zip - The Windows test app executable.
• lzw.tgz - source for the UNIX text app.

DNS-SD On Windows

Microsoft has been pushing us to use UPnP as our network discovery protocol, to the exclusion of all others. As a result, Windows ships with no support for DNS-SD - zip. And that might be the end of it, if it weren't for Apple's vested interest in having iTunes installed on every Windows machine in the world.

iTunes uses DNS-SD to share music catalogs across Local Area Networks - a natural choice with the native support in OS X for Zeroconf. Rather than isolate Windows users in an pocket UPnP universe, Apple chose to instead port the service discovery components of Bonjour to Windows, and install it with every copy of Windows. Thus, Windows and OS X users can happily share their iTunes libraries with no unusual calisthenics required.

To sweeten the deal, Apple has released a Bonjour SDK for Windows, which is currently shipping version 3.0 from their developer support site. In Apple's words:

The Bonjour SDK for Windows includes the latest version of Bonjour as well as header files, libraries, and sample code to assist developers in creating Bonjour enabled applications on Windows. The SDK has been updated with the Bonjour core that is bundled with iTunes 10.3.1. This release will bring all existing Bonjour functionality released in Mac OS X 10.7 into the Bonjour for Windows product.

It sounds pretty good, doesn't it?

An Emphasis on Kit

Installing the SDK is a decision-free breeze:

Installing the SDK

In other words, this is pretty much a code dump.

An Overview

For the most part, the Bonjour SDK interfaces follow a single pattern. Each request made to the API returns immediately, and gives you a reference that you can use to track the progress of your request. That reference can be converted to a file handle, and you can use the file handle to see when your request has some data to produce.

When your request has generated some data, and the Bonjour components are ready to deliver it to you, they do so by a callback mechanism - the Bonjour components make calls into your C or C++ program and provide the data you requested.

Most of the callbacks include a flag parameter. You can check the flag to see if there is any more data expected. If there isn't, you can delete the reference and you are then done with that particular call. Otherwise, you will have to wait for the request to be responded to when the Bonjour components get around to it.

An Example - Discover Service Types

I've written a demo program that browses the network for details on every service instance it can find, and presents the results in a tree format. The figure below shows the program running on my home network, where I have drilled down to get information about an instance of a print service.

The ServiceBrowser Sample Program

C:\Users\Mark>dns-sd -B _services._dns-sd._udp
Browsing for _services._dns-sd._udp
Timestamp     A/R Flags if Domain  Service Type  Instance Name
17:38:09.407  Add     3 13 .       _tcp.local.     _smb
17:38:09.407  Add     3 13 .       _tcp.local.     _printer
17:38:09.408  Add     3 13 .       _tcp.local.     _pdl-datastream
17:38:09.408  Add     3 13 .       _tcp.local.     _http
17:38:09.408  Add     3 13 .       _tcp.local.     _tivo-videos
17:38:09.409  Add     2 13 .       _tcp.local.     _csco-sb

Looking to do the same thing in my demo program, I turn to the header file dns_sd.h. This file ships with the Bonjour SDK and contains not only the API definition, but what passes for documentation. In that header file I see that there is a function called DNSServiceBrowse, and it looks like it does exactly what I want. In my sample program, my call to this routine is shown below:

DNSServiceRef client = NULL;
DNSServiceErrorType err;
err = DNSServiceBrowse( &client,
                        0,
                        0,
                        "_services._dns-sd._udp",
                        "",
                        IterateServiceTypes,
                        this );

You can get some exposition on each of the arguments I pass to the function from the header file, my brief comments on each are given below:

The important thing to note here is that the call to the API function doesn't return any important data. All I get back is an error code indicating that the request is being processed, and a DNSServiceRef handle that I use to track that progress.

The real action comes when my callback function is invoked. IterateServiceTypes is a static member of my Dialog class. Apple kept things simple by having one callback type for C and C++, which means no member functions. You could easily build shims to make it appear as though the DLL was calling member functions - it would just take a small modification to the code I'll show you here.

The function definition has to follow exactly the declaration given in dns_sd.h. My implementation starts like this:

void DNSSD_API CServiceBrowserDlg::IterateServiceTypes( DNSServiceRef sdRef,
                                                        DNSServiceFlags flags,
                                                        uint32_t interfaceIndex,
                                                        DNSServiceErrorType errorCode,
                                                        const char *serviceName,
                                                        const char *regtype,
                                                        const char *domain,
                                                        void *context )
{
	CServiceBrowserDlg *p = (CServiceBrowserDlg *) context;

It's worth walking through a look at each of the parameters in the callback:

If you look at the first line of code above, the first thing I do is cast the context pointer to its correct type, a pointer to my MFC Dialog class. Now I can make full use of all the members of the class, albeit via a call through a pointer instead of directly.

So what do I do with these services once I receive them? Well, for each service type that I find, I kick off a new browse process, looking for specific instances of the service. Just as an example, in my callback IterateServiceTypes, one of the callbacks returns a service type of _printer._tcp. In order to find all instances of this service, I have to call DNSServiceBrowse again, with that service name and the correct interface and domain. After inserting the service type into the tree, I make that call so I can start adding those instances:

HTREEITEM item = p->m_Tree.InsertItem( CA2T(service_type.c_str()), TVI_ROOT, TVI_SORT );
DNSServiceRef client = NULL;
DNSServiceErrorType err;
err = DNSServiceBrowse( &client,
                        0,
                        0,
                        service_type.c_str(),
                        "",
                        IterateServiceInstances,
                        context );

The key point to note about this call is that the callback function, IterateServiceInstances, is a different member function - one that expects to get the results of my browsing for instances of a specific service.

Driving the Callbacks

One thing I've skipped over so far - how do these callbacks actually get generated? Does the DLL asynchronously make calls into my code whenever events occur?

The Bonjour SDK lets your program control when callbacks occur by giving you the handle to the message pump. When you call DNSServiceProcessResult() with a single argument of a DNSServiceRef, you will generate a single callback message for the given reference. The callback will occur within the context of the call to DNSServiceProcessResult().

When you call DNSServiceProcessResult(), the Bonjour DLL will block if there are no messages ready to process. So how do you know when there are messages ready?

The indicator that messages are ready is given by a file descriptor associated with the DNSServiceRef. You can get a copy of the file descriptor by calling DNSServiceRefSockFD(), passing in a copy of the reference. When the file descriptor has data ready to read, you have callbacks pending. The easiest way to check this condition is to use the select() function, which can check multiple references in one fell swoop.

In my implementation of the callback message pump, I rely on an unordered_map called m_ClientToFdMap that contains a copy of all the DNSServiceRef references currently waiting for responses. I create the necessary data structure used by select(), then call it to get a list of all references that have callbacks pending. The core of this code looks like this:

int result = select(0, &readfds, (fd_set*)NULL, (fd_set*)NULL, &tv);
if ( result > 0 ) {
//
// While iterating through the loop, the callback functions might delete
// the client pointed to by the current iterator, so I have to increment
// it BEFORE calling DNSServiceProcessResult
//
    for ( auto ii = m_ClientToFdMap.cbegin() ; ii != m_ClientToFdMap.cend() ; ) {
        auto jj = ii++;
        if (FD_ISSET(jj->second, &readfds) )
            DNSServiceErrorType err = DNSServiceProcessResult(jj->first);
    }
}

This generates my callbacks efficiently, and because they are in the context of my main program's UI thread, I avoid a lot of unpleasant issues.

Threading Issues

My program manages the Bonjour callbacks in a fairly ugly fashion. When my browsing activity starts, I create a timer that fires once every 250 milliseconds. I process up to 10 callbacks in that timer call, then exit. This continues until there are no pending browser or resolution requests, at which time I kill the timer.

Depending on your use of DNS-SD, you may find that this is not as efficient as you like. If this is the case, you might find it useful to move your message pump code to a separate thread.

Once you do that, you can wait on all your callbacks by calling select with a long or infinite timeout. This has the effect of blocking your callback thread until it has actual work to do - resulting in a better use of CPU time.

There are some obvious downsides to this approach. Clearly you have to use some sort of locking mechanism on the data structures that are shared between your callback thread and the rest of your program. And the use of the select() statement with an infinite timeout is complicated by the possibility that you may be making or canceling browsing or resolution calls while your program runs.

A good way to deal with both of these problems is to invoke a socket-based message passing protocol between the callback thread and the other components of your program. If you restrict your interface to messages, you don't have to worry about locking access to shared data. And because you are using a socket for communications, your select() statement will be used to activate the thread when new messages arrive.

Character Sets

The days when DNS was limited to seven-bit ASCII characters are long gone. Service instances are encoded as UTF-8, and can use whatever Unicode characters they like. In the figure shown below, you can see the effects of that when I browse for instances of iTunes:

Character set problems in service instance names

By default, the Windows cmd.exe window doesn't render UTF-8 properly, but changing the code page to 65001 results in the correct rendering.

In my sample program, I deal with this with a two step approach. First, my program is built using the Unicode libraries, ensuring that I am able to render Unicode output properly. To conform with Microsoft's C++ paradigms, I use CString for all my Unicode strings, and wrap all my string literals in the _T() wrapper.

This works fine for my UI, but I can't use strings built of wchar_t to communicate with the Bonjour SDK - it expects eight bit characters with UTF-8 encoding. In m program I use the C++ std::string class everywhere where I am working with 8-bit characters that might be encoded in UTF-8. When it comes time to render one of those strings in my Unicode context, all I have to do is use the handy CA2T macro with the CP_UTF8 parameter, and things work properly.

Library Issues

The design of the Bonjour SDK imposes some uncomfortable restrictions on you when it comes to building your C or C++ program. Because you are linking directly to code found in the library dnssd.lib, you have to ensure that your program and that library link against the same version of the C run time library. And for the Bonjour SDK under Windows, this means you must link with the static, multithreaded, release version of the library.

You'll see the problem in this right away when you create an MFC project and try to build with dnssd.lib. By default, the project generator will probably have you using MFC in a shared DLL, and using the Multithreaded Debug DLL version of the C libraries. When you try to build like this, you will get some unpleasant error messages:

1>LINK : warning LNK4098:
         defaultlib 'msvcrtd.lib' conflicts with use of other libs;
         use /NODEFAULTLIB:library
1>LINK : warning LNK4098:
         defaultlib 'LIBCMT' conflicts with use of other libs;
         use /NODEFAULTLIB:library

A full featured SDK would provide libraries built for multiple scenarios, and you would pick the one of your choice depending on your build parameters. But with the Bonjour SDK, you don't get this choice, so you need to ensure that your project follows a few guidelines:

• Under Configuration Properties/General, field Use of MFC needs to be set to Use MFC in a Static Library for both debug and release builds.
• Under Configuration Properties/C++/Code Generation, field Runtime Library needs to be set to Multi-threaded (/MT) for both debug and release builds.
• Under Configuration Properties/C++/Preprocessor, field Preprocessor Definitions the constant _DEBUG needs to be changed to NDEBUG for Debug configurations.

To build a project that uses the SDK, you will also need to add dns_sd.lib to your list of linker inputs, add dns_sd.h to your header files, and add the appropriate directories in the configuration under Configuration Properties/C++/General/ in field Additional Include Directories, and under Configuration Properties/Linker/General/ in field Additional Include Directories.

Overview Of the Demo Program

I've included the full source for a project that will build with Visual Studio 10, as long as you have the Bonjour SDK installed. It browses all available services on the network and displays the information about them in a tree form.

The program starts by kicking off a browser for _services._dns-sd._udp. The results are processed in member function IterateServiceTypes(). As each new service type is discovered, it is added to the tree, and a call to DNSServiceBrowse()is made to discover all instances of that service type. The callback for that browse call is member function IterateServiceInstances().

In IterateServiceInstances() I add the instance to the tree, then call DNSServiceResolve(). This function operates much like the browse function, but it actually gets the DNS record for the service. This record contains the host name, service port and a list of name/value pairs that a service can advertise as part of its record. You can see those values put to good work with service types like _ipp._tcp, in which printer parameters are exposed as part of service discovery.

ResolveInstance() is the callback routine that receives the information about the service instance. The host name, port, and name/value pairs are added to the tree, and then one final call is made to a Bonjour SDK entry called DNSServiceGetAddrInfo(). This function resolves the IP address for the given host name. The address is stuffed into the tree in callback function GetAddress().

Conclusion

DNS service discover is powerful tool, but Windows programmers might be put off by the lack of a nicely packaged SDK. Using this simple example program might be a good way to get comfortable with an SDK that gives you a powerful tool that provides a good multi-platform alternative to UPnP.

OnPlus

Cisco OnPlus is a cloud-based network management tool that helps resellers support their customers. The figure below shows you a typical view of a customer network from an OnPlus browser screen. (The customer in this case being my Dad.) OnPlus not only identifies the devices on the customer's network, it also performs configuration backups, firmware updates, and monitors network performance.

The OnPlus Topology View

The OnPlus Network Agent

• DNS Service Discovery
• DHCP Packet Inspection
• DNS Packet Inspection
• Windows Management Instrumentation
• Cisco Discovery Protocol, or CDP
• NETBIOS/SMB
• UPnP
• SLP
• Traceroute
• ARP

When it comes to locating devices made by my business unit at Cisco, the most useful protocols are DNS Service Discovery and Cisco Discovery Protocol. DNS Service Discovery provides all the information the inventory process needs to fully identify a device: its specific Product ID (more or less a model number), the version number of both the hardware and the firmware, its MAC address, and its IP address. (CDP provides a nearly, but not identical bundle of data.) This information is readily available because devices made by Cisco's Small Business Technology Group use DNS-SD to broadcast information using our proprietary service type: csco-sb.

A Quick Overview of DNS Service Discovery

So what exactly is DNS Service Discovery? If you're like me, you became aware of DNS-SD because Apple uses it as part of Bonjour. Bonjour is a proprietary implementation of Zeroconf, a set of technologies marked by by three key network components:

• Address assignment
• Service discovery
• Name resolution

The history of Zeroconf is a somewhat quixotic story, based around the shared idea that setting up small networks ought to be a painless and simple process. The components of Zeroconf provide a nice, vendor-agnostic way to set up networks in such a way that no consumer would ever have to manually assign an IP address, set up a DHCP server, or manually enter the address of a printer.

Apple has embraced this idea, with their implementation of Zeroconf called Bonjour, an Apple trademark. If you run iTunes on your Apple or Windows PC, you may well see that there are other users out there running iTunes who would be happy to share their collections with you. This happens more or less with no work on your part, and can be a really nice feature in a big office:

Sharing iTunes Libraries

daap Digital Audio Access Protocol (iTunes)
     Amandeep Jawa 
     Defined TXT keys: txtvers, Version, iTSh Version, Machine ID,
                       Database ID, Machine Name, Password

This is a pretty simple definition - let's see what it looks like on the network.

On my desktop Linux sytem, I have the avahi utilities installed. Avahi provides a nice suite of tools used to implement DNS-SD. I'll use the avahi-browse command to see what these daap services actually look like:

mark@ubuntu:~$ avahi-browse _daap._tcp -t
+   eth0 IPv4 Itunes NAS Server on nas                      iTunes Audio Access  local
+   eth0 IPv4 Denise___s Library                            iTunes Audio Access  local
+   eth0 IPv4 Mark___s Library                              iTunes Audio Access  local
mark@ubuntu:~$

If I ask avahi-browse to resolve the services, it will query the service provider for the details in the advertisement. A partial output is shown below:

mark@ubuntu:~$ avahi-browse _daap._tcp -r -t
+   eth0 IPv4 Itunes NAS Server on nas                      iTunes Audio Access  local
+   eth0 IPv4 Denise___s Library                            iTunes Audio Access  local
+   eth0 IPv4 Mark___s Library                              iTunes Audio Access  local
=   eth0 IPv4 Itunes NAS Server on nas                      iTunes Audio Access  local
   hostname = [nas.local]
   address = [192.168.1.165]
   port = [3689]
   txt = ["ffid=075abcc4" "Password=false" "Version=196610" "iTSh Version=131073"
          "mtd-version=svn-1676" "Machine Name=Itunes NAS Server" "Machine ID=BE8926F6"
          "Database ID=BE8926F6" "txtvers=1"]

From this information, I know have everything I need in order to connect to to a server and start playing music. I have a hostname, IP address, and a port, all of which can be used to access the service. Finally I have a txt record that contains an aribtrary set of name/value pairs, as defined in the service definition. The us of these fields is up to the creator of the service, and in this case most of them are self-evident.

Browsing the Network

We use the avahi toolkit in OnPlus to browse the network for devices. It is worth doing a little exploring on my home network to see what kind of information we get out of this process.

To get a high-level view, I can ask avahi-browse to query for a special service: _services._dns-sd._udp. When this browse request goes out on the network, all the active nodes using DNS-SD issue records detailing the types of services they support. The result on my home network looks like this:

mark@ubuntu:~$ avahi-browse _services._dns-sd._udp -t
+   eth0 IPv4 _udisks-ssh                                   _tcp                 local
+   eth0 IPv4 _workstation                                  _tcp                 local
+   eth0 IPv4 _ir-hvac-021                                  _tcp                 local
+   eth0 IPv4 _ir-hvac-020                                  _tcp                 local
+   eth0 IPv4 _ir-hvac-000                                  _tcp                 local
+   eth0 IPv4 _pdl-datastream                               _tcp                 local
+   eth0 IPv4 _printer                                      _tcp                 local
+   eth0 IPv4 _tivo-videos                                  _tcp                 local
+   eth0 IPv4 _readynas                                     _tcp                 local
+   eth0 IPv4 _smb                                          _tcp                 local
+   eth0 IPv4 _afpovertcp                                   _tcp                 local
+   eth0 IPv4 _rsp                                          _tcp                 local
+   eth0 IPv4 _daap                                         _tcp                 local
+   eth0 IPv4 _http                                         _tcp                 local
+   eth0 IPv4 _csco-sb                                      _tcp                 local
mark@ubuntu:~$

As you can see, there are a surprising number of DNS-SD services present. On my network, an explanation for each of the services is:

This special command to show me the services available is not actually used in OnPlus. Instead, we call avahi-browse with the -r and -p commands, asking it to do a full resolution on all discovered services.

Cisco Devices

The place where we get the most interesting results from avahi-browse is when we tell it to look specifically for instances of the cisco-sb service. That command produces output like this:

mark@ubuntu:~$ avahi-browse -r -t _csco-sb._tcp
+   eth0 IPv4 switch32026a                                  _csco-sb._tcp        local
+   eth0 IPv4 onplus005229                                  _csco-sb._tcp        local
=   eth0 IPv4 switch32026a                                  _csco-sb._tcp        local
   hostname = [sg200-26p.local]
   address = [192.168.1.168]
   port = [80]
   txt = ["hostname=sg200-26p" "serialNo=DNI1515005U" "MACAddress=44E4D932026A"
          "PIDVID=SLM2024PT V01" "fmVersion=1.1.1.8"
          "deviceDescr=26-port Gigabit PoE Smart Switch" "deviceType=Switch"
          "model=SG 200-26P"]
=   eth0 IPv4 onplus005229                                  _csco-sb._tcp        local
   hostname = [PLG1000F0AD4E005229.local]
   address = [192.168.1.167]
   port = [80]
   txt = ["accessType=http" "MDFID=Unassigned" "hostname=onplus005229"
          "serialNo=PLGF0AD4E005229" "MACAddress=F0:AD:4E:00:52:29"
          "PIDVID=Unassigned" "fmVersion=6.2.2.007" "deviceDescr=Cisco OnPlus Network Agent"
          "deviceType=Service Appliance" "model=PLG1000" "version=1.0"]
mark@ubuntu:~$

If you look at the first '=' records that is issued by avahi-browse, you can see that when it comes to discovery, I have really hit the jackpot. I've identified a device on my system that I can reach with a specific IP address. I have a MAC address that I can now use as a globally unique identifier. And I have the Cisco Product ID, the hardware version, and the firmware version, as well as the user-assigned host name and a friendly model name.

When I have information like this, it allows me to fill in the details in the topology map quite accurately. Better yet, since this is a Cisco device, the OnPlus appliance can now send it some queries to find out more network information. As an example, the switch's CAM table provides me with a list of devices and the ports they are attached to, which helps me fill in some of the details of the topology picture.

Processing This Data

If you are a programmer, the natural question you might be asking is how you access these service advertisements from inside your program. In the case of Cisco OnPlus, most of the code running the inventory task consists of PHP scripts. As far as I know, there are no bindings in PHP to DNS-SD services, and we elected not to try to invent that wheel.

Instead, we use PHP's popen() function to run instances of avahi-browse, collecting the output from the program and parsing it accordingly. We actually have three instances of the browser running at any time. Two are dedicated to Cisco-specific services, while the third looks at all other services. Even though other services might not give us as munch information as csco-sb, they still supply host names, MAC addresses, IP addresses, descriptions, and more, and we use whatever we can find.

These instances of avahi-browse run as independent discovery processes, collecting and storing data as it is seen on the network. The records that they collect are then used by the inventory process when it is periodically launched.

The DNS-SD discovery processes actually take an active role in the inventory scheme. When one of the avahi-based processes discovers a significant new device on the network, such as one advertising cisco-sb, it stores the data record and then triggers an early start to the inventory process. This allows OnPlus to respond to new devices on the network in something close to real-time.

Advertising

So let's say you decide you want to use DNS-SD for some form of network discovery. I've shown you how you can you discover network nodes that are advertising a service, but how do you actually perform that advertisment?

In the OnPlus Network we use the avahi package to advertise services as well as to find them. The avahi-publish command does the job, typically started as a daemon to run for the lifetime of the system. When executing avahi-publish, you will normally want to specify:

• A service type
• The name of the service instance
• A port to access the service
• Text records containing whatever name/value pairs you want to publish

One thing to note is that you can use DNS-SD to advertise things that don't necessarily map directly to a port. For example, if you just want to let everyone know of your existence, the port parameter might be irrelevant, but the name/value pairs could still be valuable.

As an example, I might create a photo sharing service that I advertise on the network using a service I'll call PhotoMark. To let everyone know about this, I execute the following command when my system starts up:

avahi-publish -s mark _PhotoMark._tcp 9999 "payload=.gif" "folder=/pictures"

Anyone on the network using DNS-SD could then see that I was advertising photo sharing with photos of type GIF accessible on my pictures folder. Of course, the details of the protocol are not included in the advertisement - that's outside the scope of DNS-SD.

Often a good choice for the service instance is to simply use the computer name:

avahi-publish -s `uname -n` _PhotoMark._tcp 9999 "payload=.gif" "folder=/pictures"

What's Next

In this article I showed you how we use DNS-SD on the Cisco OnPlus Network Agent. Mostly it involves calling the avahi command line tools and parsing their output with PHP scripts.

In my next post, I'll show you how to do the same thing on a Windows PC using the Apple Bonjour SDK. Unfortunately, Windows has not included DNS-SD support in the O/S, so instead of using the Win32 API to do service discovery, you will have to rely on some slightly less elegant methods. But when it comes to network discovery, functionality and interoperability trump elegance every day of the week.