Windows users who need a command line connection to another system via telnet or SSH are big fans of PuTTY. It's free, it has every feature you need, and it's reliable.

One thing many people would like to do is use PuTTY as a component in their program. Apparently this comes up so often enough that there is a FAQ entry dedicated to the topic. Alas, PuTTY does not have any sort of automation interface, so this goal has always been out of reach.

In this article I will show you how to work around this minor shortcoming. Creating a version of PuTTY that can be driven from a Windows program turns out to be an easy task. I'll demonstrate this with a small C++ program that shows exactly how to get this versatile program to do your bidding. My solution works for C++, but the changes I make should work well with any Windows software that can properly process a few messages.

Putting Together the Project

I'm using Visual Studio 2010 to build both my program and the modified version of Putty. I created the basic outline as follows:

1. Use the File|New|Project menu item to bring up the list of available project wizards.
2. Select MFC project, and enter a project name (I used the uninspired name PuttyDriver.)
3. I don't want the default MFC settings, so in the MFC App Wizard, select the Next button.
4. On the Application Type page of the wizard, change the Application Type to Dialog Based.
5. The project is ready to go at this point, you can click the Finish button and then build your initial project.

My driver program is only going to do one thing: direct putty to connect to the host of my choice, then log in using canned credentials. The resulting UI is shown below, and I am going to leave the very minor details of creating it up to the reader.

The driver program - a simple dialog-based MFC app

Adding Putty to the Project

The next step in this process is to add the Putty components to the project. I downloaded version 0.61 of the PuTTY source from the download page and extracted it to a separate folder. I then used Visual Studio's File|Add|Existing Project to add the compatible project file, Putty.dsp, found in /Windows/MSVC/Putty. Visual Studio has to convert this project to a version 10 project file, but it should do so with no problems.

I then right-clicked on the Putty project in Solution Explorer and renamed it to AutoPutty. Since this version of PuTTY will have some slightly different behavior, I don't want to confuse the executable I am creating with the real thing.

From Project|Project Dependencies, I set the PuttyDriver project to depend on AutoPutty - this insures that both projects get built when I build the entire solution.

My final change to the project is to modify the output directory for both Debug and Release versions of AutoPutty. I set the project to build the executable in the root directory of my PuttyDriver project - this will make it easy to find the executable when I need to launch it. I had to make this change in two places: Properties|Configuration Properties|General|Output Directory and Properties|Linker|Outuput File.

When you finally build the project, you'll find that current version of Microsoft's C++ compiler complain quite a bit about the use of functions like strcpy - Microsoft would like you to use safer replacement functions. You may choose to turn those errors off by defining _CRT_SECURE_NO_WARNINGS in the project file. While you are there, you should define SECURITY_WIN32 as well - it is required by Windows header sspi.h.

After a successful build you should find a copy of AutoPutty.exe in the root directory of your project, and it should run on your system and behave just like PuTTY.

Launching AutoPutty

If I'm going to have a PuTTY component in my PuttyDriver program, one of the first things I need is to be able to start and stop AutoPutty. So my first step in this project is to create the code that launches the program from PuttyDriver. The code below is inserted into the handler for the Start button:

UpdateData( true );
char path[MAX_PATH];
GetCurrentDirectory(MAX_PATH, path);
if ( path[ strlen(path) - 1 ] != '\\' )
    strcat_s( path, MAX_PATH, "\\" );
strcat_s( path, MAX_PATH, "AutoPutty.exe -ssh " );
strcat_s( path, MAX_PATH, m_HostName.GetBuffer() );
PROCESS_INFORMATION pi;
ZeroMemory(&pi, sizeof(pi) );
STARTUPINFO si;
ZeroMemory(&si, sizeof(si) );
si.cb = sizeof(si);
if ( CreateProcess( NULL, path, NULL, NULL, NULL, NULL, NULL, NULL, &si, &pi ) )
{
    Sleep( 1000 );
    BringWindowToTop();
}

This code assumes that AutoPutty.exe is in the current directory, and launches it with a command line telling it to connect to the host named in the dialog using ssh. Assuming that you have the project set up properly, pushing the start button should now start an independent copy of AutoPutty, which will behave identically to classic PuTTY.

Taking Ownership of AutoPutty

At this point I can successfully launch AutoPutty, but I can't really start calling this an integrated part of my main program, PuttyDriver. All I have done is set up a launcher for a separate executable.

The next step in the integration process is to establish PuttyDriver as the owner of AutoPutty's main window. Most Windows programmers are familiar with the traditional parent/child relationship between windows. That relationship is well understood, but I can't use it here - it doesn't work for two top level windows.

Setting PuttyDriver to be the owner (as opposed to the parent) of AutoPutty has the following effects, as explained here by Microsoft:

• The owned window will always be above its owner in the z-order.
• The system automatically destroys the owned window when the owner is destroyed.
• The owned window is hidden when the owner is minimized.

The most straightforward way to set ownership of the window is to pass the owner's handle in the call to CreateWindow(), which means I will now make my first modifications to the PuTTY source code.

There are a number of ways to pass the owner handle to AutoPutty for use in the call to CreateWindow(), with the most obvious being to pass it on the command line. In the interest of minimizing changes to the existing PuTTY code base, I elected to pass it by creating an environment variable that holds the owner window handle. Since a child process inherits the parent's environment, this is a no-fuss way to get the data to AutoPutty.

I added the following code to the end of InitDialog() in PuttyDriver:

CString hwnd_text;
hwnd_text.Format( "%d", m_hWnd );
SetEnvironmentVariable("PUTTY_OWNER", hwnd_text );

This sets the environment variable for AutoPutty to find when it gets launched.

Now I come to the point where I am actually making changes to the PuTTY code. Fortunately, all of the changes needed for this program are confined to two files: terminal.c and windows/window.c. My first change is to window.c. This file contains the WndProc for the PuTTY window, and thus most of the rendering and control code for the GUI.

In order to establish the Owner/Owned relationship, I need to modify the code that calls CreateWindow(). I hoisted the function call into a block, added code to get the owner window handle, and inserted the handle into the call to CreateWindow():

{
    HWND owner_hwnd = 0;
    char buffer[ 132 ];
    if ( GetEnvironmentVariable( "PUTTY_OWNER", buffer, 132 ) )
        sscanf( buffer, "%d", &owner_hwnd );
    if ( owner_hwnd == 0 )
        MessageBox( NULL,
                    "AutoPutty did not find the handle for the "
                    "owner window, this is not going to work",
                    "Fail",
                    MB_OK );
    hwnd = CreateWindowEx(exwinmode, appname, appname,
                          winmode, CW_USEDEFAULT, CW_USEDEFAULT,
                          guess_width, guess_height,
                          owner_hwnd, NULL, inst, NULL);
}

At this point I've only modified one small block of code in the PuTTY source, but I'm well on my way to having it behave more like a component of PuttyDriver and less like an independent program. The ownership status means that the two programs only appear once on the taskbar, and will only appear once when you are pressing ALT-TAB to select a new active process. And they only produce a single entry in the Applications Tab of Task Manager.

The Communications Link

In order to achieve the automation that I am seeking, I also need to have two way communications between AutoPutty and the driver program. Since this is Windows, a natural choice for communications is to use native Windows messages. In order to do this, both programs need the Window handle of their opposite number.

I've already solved half of that problem through the ownership relationship established when I created the main window for AutoPutty. Now that it has set PuttyDriver as its owner window, I can get this window handle any place in the program through a simple function call:

HWND parent = GetWindow(hwnd, GW_OWNER);

But the reverse is not true - PuttyDriver does not know have a copy of the window handle for AutoPutty.

To remedy this situation, I added code to window.c that notifies its owner when it s created, and when it is destroyed. First I add this statement immediately after the call to CreateWindow():

if ( owner_hwnd )
   PostMessage( owner_hwnd, WM_APP, 0, (LPARAM) hwnd );

This tells PuttyDriver that the window is created, and gives it the handle to use for communications.

I also need to know when the window is closed, and I have to add that code two places in window.c - because Putty can be shut down two different ways.

Normally AutoPutty will shut down in response to a windows message. When this happens, I can count on a WM_CLOSE message being sent to the Windows Procedure. I add this code the existing handler for WM_CLOSE:

if (!cfg.warn_on_close || session_closed ||
    MessageBox(hwnd,
               "Are you sure you want to close this session?",
               str, MB_ICONWARNING | MB_OKCANCEL | MB_DEFBUTTON1)
    == IDOK) {
    HWND parent = GetWindow(hwnd, GW_OWNER);
    if ( parent )
        SendMessage( parent, WM_APP, 0, 0 );
    DestroyWindow(hwnd);
}

This lets PuttyDriver know that the window has been destroyed.

The original PuTTY code has an alternative method of shutdown. When it receives one of several possible network events, such as a telnet connection being broken, it calls PostQuitMessage(). When a program shuts down this way, it doesn't issue messages to destroys its windows - it relies on the O/S to destroy the windows when the process exists. As a result, I have to make a change in WinMain(), the main window procedure for PuTTY. This procedure extracts the messages sent to it using PeekMessage, and I add some code to handle the processing when a WM_QUIT message is sent:

if (msg.message == WM_QUIT) {
    HWND parent = GetWindow(hwnd, GW_OWNER);
    if ( parent )
        SendMessage( parent, WM_APP, 0, 0 );
    goto finished;	       /* two-level break */
}

Handling the AutoPutty Lifecycle Events

To keep track of the state of AutoPutty, I have to add a handler for WM_APP to PuttyDriver. It does two things when handling the incoming WM_APP event.

First, then handler stores the handle of the AutoPutty window - or sets the value to 0 when the window has been destroyed.

Second, it either enables or disables the button used to start up AutoPutty. Since this program can only manage one window at a time, I don't want to allow any inadvertent button pushes:

afx_msg LRESULT CPuttyDriverDlg::OnWmApp(WPARAM wParam, LPARAM lParam)
{
    m_PuttyWindow = (HWND) lParam;
    m_StartButton.EnableWindow( !m_PuttyWindow );
    return 0;
}

One final piece of bookkeeping is to make sure that the AutoPutty window is shut down when PuttyDriver shuts down. (The Windows documentation claims this happens automatically to owned windows, but it doesn't seem to be the case.)

void CPuttyDriverDlg::OnDestroy()
{
    CDialogEx::OnDestroy();
    if ( m_PuttyWindow )
        ::SendMessage( m_PuttyWindow, WM_CLOSE, 0, 0 );
}

Monitoring Input Traffic

Now that I have control over the lifetime of my AutoPutty window, it's time to take the next step in automation. My driver program needs to watch all the data coming in from the remote end so that it can take action on various types of input.

Depending on how you set up your connection, PuTTY can receive input data from a serial port, a Telnet connection, or an SSH connection. Fortunately the Windows version of PuTTY uses a standard handle-based interface to all three types of connections. The routine term_data() in terminal.c is called as data arrives, regardless of the source.

Since we are using the Windows API to communicate between processes, it makes sense to use the WM_COPYDATA message to send data to the parent program as it arrives. WM_COPYDATA is a good choice, as it takes care of marshalling the data between the two processes, which can add some complication to other solutions. The modified routine is shown below:

int term_data(Terminal *term, int is_stderr, const char *data, int len)
{
    HWND parent = GetWindow(hwnd, GW_OWNER);
    if ( parent ) {
        COPYDATASTRUCT cd;
        cd.dwData = (ULONG_PTR) 0xDEADBEEF;
        cd.cbData = len;
        cd.lpData = (PVOID) data;
        SendMessage( parent, WM_COPYDATA, (WPARAM) hwnd, (LPARAM) &cd );
    }

Receiving the Data

To receive this messages in PuttyDriver, I simply create a handler for WM_COPYDATA and start grabbing the data as it arrives. One important thing to note is that because AutoPutty has to use SendMessage() to send the data to its parent, it has to wait for PuttyDriver to finish processing the data until it can continue. This dictates a certain style of behavior on my part.

There are quite a few ways to skin this cat, and I'm keeping it very simple here. I'm using a deque<char> container to hold the last 64 characters I've received. After each WM_COPYDATA message I received, I check to see if the current output snapshot ends in one of my trigger messages. If it does, I post the message number to myself for later processing, then return so that AutoPutty can continue its work.

The code I'm using here is doing something fairly simple: automating the login process by using the credentials that I've entered into the dialog box. That means the two strings I'm looking for are the login and password prompts. The resulting code is shown here:

BOOL CPuttyDriverDlg::OnCopyData(CWnd* pWnd, COPYDATASTRUCT* pCopyDataStruct)
{
    char *p = (char *) pCopyDataStruct->lpData;
    int len = pCopyDataStruct->cbData;
    if ( len >= 64 ) {
        p += len - 64;
        len = 64;
        m_Snapshot.clear();
    }
    while ( len-- )
        m_Snapshot.push_front(*p++);
    m_Snapshot.resize(64);
    static const char *needles[2] = { "login as: ", "password: " };
    for ( int i = 0 ; i < 2 ; i++ ) {
        int len = strlen( needles[i] );
        int j;
        for ( j = 0 ; j < len ; j++ ) {
            if ( needles[i][j] != m_Snapshot[len-1-j] )
                break;
        }
        if ( j == len )
            PostMessage( WM_APP+1, i, 0 );
    }
    return TRUE;
}

There is plenty of room for improvement in this routine, much of it depending on what type of automation you are going to be using in your program. Some obvious items would include the ability to add and remove triggers as the program progresses, and regular expression matching for triggers.

Driving PuTTY

This login program is now complete save for one detail: I need a way to send my responses back to AutoPutty.

The first part of this is pretty obvious - I just need to read the data from the dialog box and post it to AutoPutty with my useful WM_COPYDATA command. This happens in my WM_APP+1 handler:

afx_msg LRESULT CPuttyDriverDlg::OnWmAppPlusOne(WPARAM wParam, LPARAM lParam)
{
    UpdateData(TRUE);
    CString msg;
    switch ( wParam ) {
    case 0 :
        msg = this->m_UserId + '\r'; break;
    case 1:
        msg = this->m_Password + '\r'; break;
    }
    if ( this->m_PuttyWindow ) {
        COPYDATASTRUCT cd;
        cd.dwData = (ULONG_PTR) 0xF00DFACE;
        cd.cbData = msg.GetLength();
        cd.lpData = (PVOID) (const char *) msg;
        ::SendMessage( this->m_PuttyWindow,
                       WM_COPYDATA,
                       (WPARAM) this->m_hWnd,
                       (LPARAM) &cd );
    }
    return 0;
}

Sending this data to AutoPutty is fine, but right now the program doesn't do anything with that message. The final piece of work is to add a WM_COPYDATA handler to window.c.

Simply grabbing the data is easy enough - the data structure that accompanies the message contains a pointer to the data and a value indicating its length. However, I have two problems I have to solve before the data is actually sent out to the to whatever device AutoPutty is connected to.

First, I have to take into account the fact that PuTTY was written to use wide characters. My driver program was built using MultiByte characters, so we have a mismatch. This means I have to do a conversion of the data from one domain to the other. This is a two step process - I call MultiByteToWideChar() once to determine how much space I need, then I allocate a buffer and call it again.

The second thing I need to do is determine what to do with the data once I've converted it. PuTTY takes all terminal input and eventually passes through a function called luni_send(). Calling this function directly from the Windows procedure seems to work just fine.

The WM_COPYDATA handler I created looks like this:

case WM_COPYDATA :
{
    COPYDATASTRUCT *cd = (COPYDATASTRUCT *) lParam;
    int wsize = MultiByteToWideChar( CP_ACP,
                                     MB_PRECOMPOSED,
                                     (LPCSTR) cd->lpData,
                                     cd->dwData,
                                     NULL,
                                     0 );
    wchar_t *buf = (wchar_t *) calloc( wsize+1, sizeof(wchar_t) );
    MultiByteToWideChar( CP_ACP,
                         MB_PRECOMPOSED,
                         (LPCSTR) cd->lpData,
                         cd->dwData,
                         buf,
                         wsize + 1 );
    if (term->ldisc)
        luni_send(term->ldisc, buf, wsize, 0);
    free( buf );
}

At this point I have a working program - it connects to my designated host, and sends the username and password of my choice to the host, connecting me to the system.

I should add a note of caution here. Automating logins is a tempting time saver, but in general this is a really bad idea. Any time you hard code credentials into a program, you open the door to all sorts of new attacks on your system.

In my demo program, the user has to enter a name and password, so nothing is hardcoded, but even this adds security holes to a system. I encourage you to think of this as a demonstration only.

Source Code

I've included the complete source code for PuttyDriver, the MFC project that controls AutoPutty. It was built with Visual Studio 2010, so you may have a little work to do if you backport it to earlier versions. My use of language features and classes should be compatible with much earlier versions - this is all very simple code.

Because PuTTY is always changing, I am not redistributing a snapshot of the version I used. Instead, I'm including before and after copies of the two source files I modified: window.c and terminal.c. If you build with Putty 0.61, you should be able to drop these two files right on top of the files included with the distribution and be on your way. With later versions of PuTTY you will have to perform an intelligent merge of the changes, which I hope will be a fairly effortless process.

Downloads

• PuttyDriver.zip. The PuttyDriver source and project. You will need to add the PuTTY project to this solution as described in the article.
• putty.zip. This contains the two PuTTY source files modified for this project. Both the original 0.61 source and my modified source are supplied. Executables are supplied as well, which may or may not work on your system.

This article presents the algorithm that solves the Byzantine General’s Problem, as first described by Lamport, Pease, and Shostak in 1982 [1]. While Lamport’s algorithm is not particularly complex, programmers who aren’t used to working on distributed computation might find it difficult to implement. To accompany the explanation of the algorithm, I have included a C++ program designed for experimentation with the solution.

Introduction

The Byzantine General’s Problem is one of many in the field of agreement protocols. In 1982, Leslie Lamport described this problem in a paper written with Marshall Pease and Robert Shostak. Lamport framed his paper around a story problem after observing what he felt was an inordinate amount of attention received by Dijkstra’s Dining Philosophers problem [2].

This problem is built around an imaginary General who makes a decision to attack or retreat, and must communicate the decision to his lieutenants. A given number of these actors are traitors (possibly including the General.) Traitors cannot be relied upon to properly communicate orders; worse yet, they may actively alter messages in an attempt to subvert the process.

When we’re not dwelling in storybook land, the generals are collectively known as processes, the general who initiates the order is the source process, and the orders sent to the other processes are messages. Traitorous generals and lieutenants are faulty processes, and loyal generals and lieutenants are correct processes. The order to retreat or attack is a message with a single bit of information: a one or a zero.

In general, a solution to an agreement problem must pass three tests: termination, agreement, and validity. As applied to the Byzantine General’s problem, these three tests are:

1. A solution has to guarantee that all correct processes eventually reach a decision regarding the value of the order they have been given.
2. All correct processes have to decide on the same value of the order they have been given.
3. If the source process is a correct process, all processes have to decide on the value that was original given by the source process.

Note that one interesting side effect of this is that if the source process is faulty, all other processes still have to agree on the same value. It doesn’t matter what value they agree on, they simply all have to agree. So if the General is subversive, all lieutenants still have to come to a common, unanimous decision.

Difficulties

This agreement problem doesn’t lend itself to an easy naïve solution. Imagine, for example, that the source process is the only faulty process. It tells half the processes that the value of their order is zero, and the other half that their value is one.

After receiving the order from the source process, the remaining processes have to agree on a value that they will all decide on. The processes could quickly poll one another to see what value they received from the source process.

In this scenario, imagine the decision algorithm of a process which receives an initial message of zero from the source process, but sees that one of the other processes says that the correct value is one. Given the conflict, the process knows that either the source process is faulty, having given different values to two different peers, or the peer is faulty, and is lying about the value it received from the source process.

It’s fine to reach the conclusion that someone is lying, but making a final decision on who is the traitor seems to be an insurmountable problem. And in fact it can be proven that it is impossible to decide in some cases. The classic example used to show this is when there are only three processes: one source process and two peer processes.

In the two configurations shown in Figure 1 and Figure 2, the peer processes attempt to reach consensus by sending each other their proposed value after receiving it from the source process. In Figure 1, the source process (P₁) is faulty, sending two different values to the peers. In Figure 2, P₃ is faulty, sending an incorrect value to the peer.

You can see the difficulty P₂ faces in this situation. Regardless of which configuration it is in, the incoming data is the same. He has no way to distinguish between the two configurations, and no way to know which of the two other processes to trust.

Figure 1
The case in which the source process is faulty

Figure 2
The case in which P₃ is faulty

Under the rules of this approach, it is easy to show that regardless of how many processes are in the system, a subversive source process with one collaborator can cause half the processes to choose to attack, while half the processes elect to retreat, leading to maximum confusion.

The Lamport, Pease and Shostak Algorithm

In 1982, Lamport, Pease, and Shostak published a fairly simple solution to this problem. The algorithm assumes that there are n processes, with m faulty processes, where n > 3m. Thus, for a scenario such as that in Figure 1 and 2 with 1 faulty process, there would have to be a minimum of 4 processes in the system to come to agreement. (For the rest of this article, n will always refer to the count of processes, and m will always refer to the number of faulty processes.)

The definition of the algorithm in the original paper is short and succinct, but at least in my experience, is somewhat confusing for programmers without a lot of experience in distributed algorithms.

Lamport’s algorithm is a recursive definition, with a base case for m=0, and a recursive step for m > 0:

Lamport’s algorithm actually works in two stages. In the first step, the processes iterate through m+1 rounds of sending and receiving messages. In the second stage of the algorithm, each process takes all the information it has been given and uses it to come up with its decision.

I found this to be non-obvious without quite a bit of study, which is the reason for this article.

The First Stage

The first stage of the algorithm is simply one of data gathering. The algorithm defines m+1 rounds of messaging between all the processes.

In round 0, the General sends the order to all of its lieutenants. Having completed his work, the General now retires and stands by idly waiting for the remaining work to complete. Nobody sends any additional messages to the General, and the General won’t send any more messages.

In each of the remaining rounds, each lieutenant composes a batch of messages, each of which is a tuple containing a value and a path. The value is simply a 1 or a 0. The path is a string of process ids, <ID₁, ID₂, …, ID_n>. What the path means in this context is that in Round N, PID_N is saying that was told in round N-1 that P_IDN-1 was told by… P_ID1 that the command value was v. (This is very much like the classic party game in which a message is whispered from ear to ear through a chain of players, becoming slightly mangled along the way.) No path can contain a cycle. In other words, if ID1 is 1, no other ID in the string of process IDs will be a 1.

The message definition is easy in round 1. Each process broadcasts a message to all the other processes, including itself, but excluding the General, with the value it received from the General and its own process ID.

In subsequent rounds, things get more complicated. Each process takes all the messages it received from the previous round, appends its process ID where allowed, and sends those messages to all other processes, including itself. (The "where allowed" just means that the process skips any messages where adding its process ID to the list would create a cycle in the string of process IDs.)

For example, let’s suppose that in Round 0 that P₁, a faulty general told P₂, P₃, and P₄ that the command value was 0, and told P₅, P₆, and P₇ that the command value was 1. In round 1, the following messages would be sent:

In the previous table I showed the messages being sent to all six processes, which is fairly redundant, since the same messages are broadcast to all processes. For round 2, I’ll just show you the set of messages that each process sends to all of its neighbors.

The first message that P₂ is sending in round 2, {0,132}, is equivalent to saying "P₂ is telling you that in round 1 P₃ told it that in round 0 that P₁ (the General) told it that the value was 0". The five messages shown in P₂’s column in the table are sent to all six lieutenant processes, include itself.

It’s easy to see that as the number of processes increases, the number of messages being exchanged starts to go up rapidly. If there are N processes, each process sends N-1 messages in round 1, then (N-1)*(N-2) in round 2, (N-1)*(N-2)*(N-3) in round 3. That can add up to a lot of messages in a big system.

The Second Stage

While sending messages in each round, processes are also accumulating incoming messages. The messages are stored in a tree format, with each round of messages occupying one rank of the tree. Figure 3 shows the layout of the tree for a simple configuration with six processes, one of which can be faulty. Since m=1, there are just two rounds of messaging: the first, in which the general sends a value to each lieutenant process, and a second, in which each process broadcasts its value to all the other processes. Two rounds of messaging are equivalent to two ranks in the tree.

Each node in the tree has three elements: an input value, a path, and an output value. The input value and path are defined in the first stage of the algorithm - they are simply the messages received from the peer processes. The output value is left undetermined until the second stage of the algorithm, which I am defining here. Note that in the figure below, the output values are initially set to '?', indicating that they are presently unknown.

Figure 3
The Tree Layout for 5 processes with 1 faulty process

Once a process has completed building its tree, it is ready to decide on a value. It does this by working its way up from the leaves of the tree, calculating the majority value at each rank and assigning it to the rank above it. The output value at each level is the third item in the data structure attached to each node, and those values are all undefined during the information gathering stage.

Calculating the output values is a three step process:

1. Each leaf node in the tree (all values at rank m) copies its input value to the output value.
2. Starting at rank m-1 and working down to 0, the output value of each internal node is set to be the majority of the output values of all its children. In the event of a tie, an arbitrary tie-breaker is used to assign a default value. The same default value must be used by all processes.
3. When complete, the process has a decision value in the output of the sole node at rank 0.

In Figure 3, step 1 of the process assigns the initial values to the leaf nodes. In the next step, the majority value of { 1, 1, 1, 0, 0 } is evaluated and returns a value of 1, which is assigned to the output value in rank 0. Because that is the top rank, the process is done, and P₁ decides on a value of 1.

Every lieutenant value in a given exercise will have the same paths for all its nodes, and in this case, since only the General is faulty, we know that all lieutenants will have the same input values on all its leaves. As a result, all processes will agree on the same value, 1, which fulfills the agreement property.

A More Complicated Example

Getting a good understanding of the algorithm really requires walking through an example that has at least three ranks. (Examples on the web, mostly extracted from lecture notes, nearly always have the simple two-rank example.) For this example, consider an example with n=7 and m=2. We’ll continue with the convention that the General is P₁, and instead of having a faulty general, we’ll have P₆ and P₇ be faulty processes. After the initial three rounds of information exchange, we have the three-ranked tree shown in Figure 4:

Figure 4
A tree with n=7, m=2, and faulty processes P₆ and P₇

You’ll see that at rank 1, the values from path 17 and 16 are both set to X. In the first round the two faulty processes communicated possibly false values to all other processes, and may have arbitrarily changed the values sent to different processes in order to skew the results.

As a result of those bad values in rank 1, we see their frequent occurrence in rank 2. The incorrect values show up not only in direct messages from the faulty processes, but also in any message from a correct process that includes a faulty process earlier in its path.

All in all, at the leaf nodes, we have 18 deceptive values at the leaf nodes, and only 12 accurate messages that trace their way all the way back to the general through nothing but correct processes. Obviously, if we just voted on the majority of the messages we had received, we would be susceptible to falling for the wrong value.

Fortunately, the layout of the tree guarantees that we will actually get a correct value. In Figure 4, the roll up of the output values hasn’t occurred yet, so every node has a question mark in the output value. In Figure 5, the output values are shown. The leaf rank has the output values set to the input values, with X used to indicate unknown values from faulty processes.

When the leaf rank is rolled up to the second rank, the nodes with paths 12, 13, 14, and 15 all have clear majority values of 0 for their output values, with 16 and 17 set to X, as their values are uncertain.

The final roll up to the top rank successfully sets the output value to 0, as four of the inputs are set to 0 and only 2 are set to X. Mission accomplished. And because of the way this was calculated, we know that the correct result will be achieved regardless of what deceptive values are sent by the two faulty processes.

Figure 5
The tree after calculating the output values

The Sample Code

I’ve included a simple C++ program that implements this algorithm, with extensive internal documentation. It has a Process class that is used to send and receive messages, as well as to roll up the decision tree. A Traits class is used to define the number of processes, the number of faulty processes, the source process, and what values the faulty processes send in various rounds.

To help with visualization, the program will output the tree for a given process in the format used by dot, part of the free Graphviz program. You can then use dot to create a nice picture of the output graph – all the figures in this article were created that way. I find that using the SVG format for the output produces very readable results.

As supplied, the program is set for values of n=7 and m=2. Good exercises to perform while experimenting with it include:

• Attempt to invalidate the program or the algorithm by getting incorrect results with some particular combination of faulty messages.
• Add a third faulty process and show that it is relatively easy to get invalid output when n=7 and m=2.
• Reduce n to 6 and show that it is relatively easy to get invalid output with two faulty processes.
• Move up to m=3 and n=10. Experiment with various combinations of faulty Generals and lieutenants and see if you can create incorrect results.

Note

Most implementations of this algorithm include logic designed to deal with the case in which a faulty process fails to send a message. This is equivalent to simply having the faulty process send an arbitrary value, so I don’t treat it as a separate case.

Source Code

source.zip, which contains:

• main.cpp
• README.TXT
• VS2003/byzantine.sln
• VS2003/byzantine.vcproj
• VS2005/byzantine.sln
• VS2005/byzantine.vcproj

References

[1] The Byzantine Generals Problem (with Marshall Pease and Robert Shostak)
ACM Transactions on Programming Languages and Systems 4, 3 (July 1982), 382-401.
http://research.microsoft.com/users/lamport/pubs/byz.pdf

[2] The Writings of Leslie Lamport: http://research.microsoft.com/users/lamport/pubs/pubs.html#byz

Lynch, Nancy A. Distributed Algorithms. San Francisco, CA: Morgan Kaufmann, 1997. ISBN: 1558603484.

Graphviz – Graph Visualization Software. http://www.graphviz.org/

CS 6378: Advanced Operating Systems Section 081 Notes on Agreement Protocols (lecture notes by Neeraj Mittal, University of Texas at Dallas) http://www.utdallas.edu/~neerajm/cs6378su07/agreement.pdf

Update 01-November-2007
Distributed Computing With Malicious Processors w/o Crypto or Private Channels, YouTube Video of Google TechTalk, Valerie King, Department of Computer Science, University of Victoria, Victoria, BC, Canada

Dr. Dobb's Journal October, 2002 Source code and install program for Louis, Louis

Wouldn't it be nice to have a radio station that always played your favorite music? A JukeBox that could read your mind - mixing in your old favorites with new music that you really like?

Mental telepathy is still a bit out of reach, but in this article I'm going to show you how to build a Windows-based MP3 player that keeps track of your musical tastes and does its best to accommodate them. By giving you the opportunity to rate artists, albums, and individual songs, the MP3 player can apply a bias towards the music you like and away from the music you don't.

So that you can build an MP3 Player like this, I'll first describe the steps you need to embed Microsoft's Windows Media Player (WMP) in a Visual C++ program as an ActiveX control, and what you need to do to control it, and respond to events from it.

With that infrastructure in place, I'll then show you how to keep a music database and use the ratings it contains to select the music you like.

The Demo Program: Louis, Louis

My demo program is named Louis, Louis, in honor of Louis Glass, a man who is credited by many with being the inventor of the modern Juke Box. In 1889 Louis introduced the first coin-operated phonograph at the Palais Royal saloon in San Francisco.

Figure One shows Louis, Louis in action. The main window of the program is divided into three parts. At the top, there is information about the track currently being played, which includes the Artist, Album, and Song name, along with your current ratings for each.

Figure 1
Louis, Louis at Work

The bottom section of the dialog contains buttons that are used for the various functions I've added to the program.

A Few Details

Louis, Louis is an MP3 player that has two different modes of operation. Random Play Mode is enabled by checking the box of the same name in the top section of the Dialog. In this mode, the program randomly selects tracks from the master catalog. As it selects each song, it determines how likely you are to want to hear it. The selection algorithm first checks to see if you've rated the song. If you haven't, it then checks to see if you've rated the album, and finally the artist. Your chances of hearing the song range from 0 to 100 percent, depending on your rating on a score of 0 to 10. If you haven't applied any rating at all, the probability is fixed at 25%.

If you uncheck the Random Play box, Louis, Louis operates more or less like a conventional MP3 player. You can set it to start playing tracks anywhere in your playlist, and it will work its way sequentially through all your music. You can navigate through your playlist by clicking the Playlist button on the main screen. That brings up a tree view of all your music, as shown in Figure 2.

Figure 2
The Tree View of the Master Playlist

Figure 3
Scanning for New Songs

Building an MP3 Player

If you want to build your own MP3 player, there are many avenues you can take. Just as an example, the folks at javazoom.net have developed a free MP3 player component written in pure Java, suitable for developing an app that can play on multiple platforms. There are many other freeware MP3 players that you can modify to suit your needs.

When I set out on the Louis, Louis project I was hoping to avoid the work of completely implementing an MP3 player from scratch. After a little research, I saw that both WinAmp (from Nullsoft at winamp.com) and Microsoft's Windows Media Player had SDKs that would allow me to control the player while taking advantage of the existing user interface, playlist features, and so on.

Both of these programs were good candidates for a personal jukebox, but in the end I chose the Microsoft solution. The fact that I could embed WMP as a control in my program gave me a somewhat cleaner solution than I would have controlling Winamp as an external program. Just as importantly, I found out that WMP has an additional bonus: Microsoft's license to use MP3 technology is passed on to users of the WMP SDK. Thus, if Louis, Louis were ever to be distributed, the headache of acquiring a license to use MP3 patented technology would be avoided.

Embedding WMP in Louis, Louis

Microsoft's SDK for Windows Media Player can be found by navigationg to their developer support site. From the Downloads link you can navigate down through Graphics and Multimedia to Windows Media Technologies and finally download the Windows Media Player 7.1 SDK. When you download and install the SDK you will have to agree to an End User License Agreement which will describe the terms under which you can use WMP. For PC based MP3 player you don't have to worry about royalties or other payments; you can even distribute the WMP components as long as you follow the rules.

Besides the license, the most important thing you get from the SDK is a set of documentation. The help pages describing the WMP object model are essential, including all the functions and events needed to develop an app.

Although the SDK ships with a decent amount of sample code, C++ programmers should prepare for disappointment. There is a grand total of one C++ example, which is basically an API exerciser. It might help you perform a few experiments, but there is no way the sample is going to show you how to write an application like Louis, Louis.

Since the single C++ example was an ATL-based app, I decided to create Louis, Louis as a dialog-based ATL application, embedding the WMP OCX as a control in the dialog. ATL allows you to do this without writing too much code, but you will have to do a substantial amount of work without the benefit of the wizard-driven coding used for MFC development.

The ATL App

Microsoft's AppWizard doesn't have an option for creating a simple ATL Window-based application, so I had to wing it. My initial app was created following a fairly simple recipe from Andrew Whitechapel on the invaluable CodeGuru site. I made a couple of modifications needed to change my main Window class to be Dialog-based, and had an app that was ready to go.

At this point I had a main window class, CLouisDialog, and was ready to start working with the WMP OCX. The first task was to create an instance of the OCX and embed it in the dialog.

ATL has a Window class called CAxWindow that is specifically designed to contain ActiveX controls, so the first thing I did was add a member of this class to CLouisDialog, then create the window as a child in CLouisDialog::OnInitDialog(). Once the window is created, it's a simple step to create an instance of the Windows Medial Player OCX in that window. The code extracted from OnInitDialog() is shown below:

Working With the Control

After instantiating the WMP control, the next step is setting up the program elements needed to call its methods and respond to its events. The first step in this is to use Visual C++'s nifty import facility to pull in a complete definition of the COM objects in the OCX. This is accomplished by adding a single line of code to STDAFX.H:

I then added a set of member variables to class CLouisDialog:

These four COM objects represent the interfaces by which I call methods on the various components of the player. They are initialized in CLouisDialog::OnInitiDialog() right after the WMP OCX is created:

With these objects in place I now have the ability to insert and remove songs into playlist, skip from one song to the next, and so on. All I need now is the ability to receive events.

Receiving Events

To receive events from the OCX, I need to create a separate event sink class. The ATL has a template class called IDispEventImpl that is designed to do just this. Following Microsoft's cookbook for this class, I created an event sink that had support for the only event I need, the CurrentItemChange event. The resulting class definition looks like this:

Adding new events to this map is simply a matter of creating a new entry in the header file sink map, then adding a new member function to handle the event. Note that the dispid that uniquely identifies the event (0x16ae in this case) and the function prototype for the event handler are found in the WMP.TLI file created when processing the #import statement.

To actually connect to the event mechanism, I create a CLouisEventSink member in CLouisDialog, then connect it to the OCX in the InitDialog() function:

A corresponding Unadvise function call has to be made when the program is shutting down.

Database

With the control in place, the other major component needed for full functionality is the database. For this program, I created a simple Access database with three tables: Tracks, Albums, and Artists. Each contains the name and rating for the specific items. The Tracks table also contains a file name that denotes the actual location of the specfic song.

I created a single query that groups all of this data into a single sorted master playlist - this is what the program uses to decide what to play during normal operations. The three primary tables are only accessed directly when new tracks are being added.

Communicating with this database is nearly trivial when using OLE DB access as provided through ATL. For each of the three tables and the single query I simply used the ATL Object Wizard to create a consumer class that gives me full access to that object.

I had some concern about the speed with which the program can access items in the database. At program startup, the single query is executed and all of its information is loaded into memory. On my development system, processing around 8,000 songs seems to take only a second or two, which is acceptable. However, users with significantly slower machines or significantly larger music libraries might have to rethink this strategy.

Adding tracks

Adding tracks to the database is a two-step operation. As Figure 3 shows, the first step is to identify a root folder, then execute a file search for files with the MP3 extension. This is cookie-cutter code and needs no elaboration.

Finding the file is the simple part - but once you have the file, you have to determine the title of the song, the album, and artist. Readers who remember my DDJ article of January 2000, The Ultimate Home Jukebox, know that I keep my personal music organized in a directory structure that automatically gives you the artist, album, title, and track number of any given file.

But it would be unrealistic to expect everyone to follow this convention, particularly when a better solution is available: ID3 tags. The ID3 tag is a conventional way of embedding information about a track directly in the MP3 file itself. A nice explanation of how this works can be found at http://www.id3.org, the ID3v2 web site.

Rather than writing my own ID3 decoder class, I went to one of my favorite programming web sites, The Code Project, and found the CMP3Info class by Roman Nurik. I dropped it into my project and it worked perfectly with no muss, no fuss!

As nice as Roman's class is, I still elected to perform a bit of surgery on it. Roman was getting some detailed information about the encoding of each block in the MP3 file, which took a considerable amount of time. I cut that code out, limiting the class to just the ID3 info I cared about.

The resulting code seems pretty efficient. On my sub-GHz development computer, I can scan my entire library of 8300 tracks, create track, artist, and album records for each, and update the database in under two minutes, or roughly 75 tracks per second. Given that you will do updates of this magnitude infrequently, this seems pretty good to me.

Overall Operations

The actual operation of Louis, Louis is essentially identical whether in random shuffle or normal jukebox mode. In both cases, tracks are selected from the master playlist, which is sorted by artist, then album, then track number. In random play mode the list undergoes a single shuffle operation, and as songs are selected they may be skipped if they fail the internal coin toss.

The WMP component is fed a playlist of five songs, allowing for a current song being played, two previous songs, and two upcoming songs. This insures that that user can use the skip buttons on WMP to immediately go ahead to the next song or back up to listen to one that was previously heard.

Each time a song completes, a CurrentItemChange event is sent to the event sink. Each time we move forward, Louis, Louis adds a new song to the playlist on the upcoming end, and removes one from the already-been-played side. This virtual playlist makes random play mode easy to implement.

Requirements and a Quirk

To build and work with Louis, Louis, you will need to have Visual C++ 6, and it's probably a good idea to bring it up to Service Pack 5. In addition, you'll need to install Windows Media Player 7.1. Odds are you'll want to install the Windows Media Player SDK as well, if only for the help file.

One additional point that you'll need to be aware of before using Louis, Louis. By default, Windows Media Player will not allow external apps to insert tracks into its playlist, which means programs like Louis, Louis are disabled by default. To enable external control of the playlist, you will need to start Windows Media Player, select the Tools|Options menu item, switch to the Media Library tab, and select Full Access for Internet Site rights.

In summary, creating Louis, Louis has been a lot of fun. To Microsoft's credit, Windows Media Player gives me a powerful tool to make this happen, in combination with ATL for the UI and OLE DB for database access. I started this project with little knowledge of all three topics, and was able to fumble my way through to a finished product that I'm pretty happy with. If only all programming assignments could follow this path!

© 2002 Mark Nelson

Dr. Dobb's Journal August, 2002

This article describes an interesting data compression technique called star-encoding. I'll explain what star-encoding is, describe a simple C++ implementation, and show you what sort of results you can expect to see when using it.

Pre-Compression Transformations

One interesting class of data compression techniques consists of transformation algorithms. It is often possible to perform a reversible transformation on a data set that increases its susceptibility to other compression techniques.

A great example of this is the Burrows-Wheeler Transform (BWT), which I described in an article in Dr. Dobb's Journal in 1996. The BWT is a completely reversible sorting algorithm that tends to create long runs of identical characters in the output text. Using a Move To Front algorithm followed by Huffman or arithmetic coding gives compression that outperforms all but the best algorithms.

A similar example is that of the Discrete Cosine Transform (DCT). In the JPEG compression algorithm, 8X8 blocks of pixels are run through the DCT, which transforms the spatial data points to the frequency domain. At that point, lossy compression algorithms can then be effectively applied.

Of course, the transformations I'm describing here generally don't compress data at all; they merely prep the data before compression is applied. Thus, a key factor in using any transform is understanding what compression technique can be effectively be used on the output. With the Burrows-Wheeler transform, the best results come from a three-stage compressor. The output of the BWT transform is usually piped through a move-to-front stage, then a run length encoder stage, and finally an entropy encoder, normally arithmetic or Huffman coding. The actual command line to perform this sequence will look something like this:

BWT < input-file | MTF | RLE | ARI > output-file

Star-Encoding

The Burrows-Wheeler transform is a fairly complicated sorting algorithm that radically changes the nature of an input file. Star-encoding is much simpler to understand, and can be implemented easily with the help of the standard C++ library.

Star-encoding works by creating a large dictionary of commonly used words expected in the input files. The dictionary must be prepared in advance, and must be known to the compressor and decompressor.

Each word in the dictionary has a star-encoded equivalent, in which as many letters a possible are replaced by the '*' character. For example, a commonly used word such as the might be replaced by the string t**. The star-encoding transform simply replaces every occurrence of the word the in the input file with t**.

Ideally, the most common words will have the highest percentage of '*' characters in their encodings. If done properly, this means that transformed file will have a huge number of '*' characters. This ought to make the transformed file more compressible than the original plain text.

As an example, a section of text from Project Gutenberg's version of Romeo and Juliet looks like this in the original text:

But soft, what light through yonder window breaks?
It is the East, and Iuliet is the Sunne,
Arise faire Sun and kill the enuious Moone,
Who is already sicke and pale with griefe,
That thou her Maid art far more faire then she

Running this text through the star-encoder yields the following text:

B** *of*, **a* **g** *****g* ***d*r ***do* b*e***?
It *s *** E**t, **d ***i** *s *** *u**e,
A***e **i** *un **d k*** *** e****** M****,
*ho *s a****** **c*e **d **le ***h ****fe,
***t ***u *e* *ai* *r* f*r **r* **i** ***n s**

You can clearly see that the encoded data has exactly the same number of characters, but is dominated by stars. It certainly looks as though it is more compressible. But before you can test the compressibility of the output data, you need to build the three programs this system requires:

• A program to create the dictionary.
• An encoder which star-encodes a file using a dictionary.
• A decoder which decodes the file using an identical dictionary.

I'll discuss each of these in turn in the remainder of this article.

Creating the Dictionary

The program that creates the dictionary, MakeDictA.cpp, is show in Listing 1. This program is a testament to the power of the C++ standard library classes. It would literally triple in size if it were written in C, and would have been a nightmare to debug.

MakeDictA is invoked with a list of file names as command line arguments. It scans through each file looking for tokens, adds them to the dictionary if needed, and updates the token's frequency count.

After all of the tokens have been counted, the list is sorted by frequency. The program then goes through the list, starting with the most frequent tokens, and assigns them star-codes. As a final step, the dictionary is written to standard output.

Tokenizing and scanning

Unfortunately, the C++ standard library doesn't have a stream tokenizer. If my needs were more complicated, I might use the Boost Tokenizer found at the boost.org Tokenizer page, but in this case I spurned reuse and wrote just a few lines:

This function is called repeatedly from main(). The map object named frequency is updated as each token is encountered. If the string is not found in frequency, it is inserted with a count of 1. Otherwise the count is incremented:

Sorting and Assigning Codes

After all the tokens from all the input files have been scanned, you have a map with an appearance count for every word encountered in the input text. The next step requires that you iterate through all the words starting at the most frequent and going all the way to the least.

Unfortunately, the frequency map is keyed on the word name, not the frequency. So before code assignment can start, you need to reorder the data. You can do this with just a few lines of code by inserting all the string/integer pairs into a map object called counts. Since this map is keyed on the integer value instead of the string, the values are automatically sorted in the form you want.

With the data sorted correctly, the remaining piece of hard work is to iterate over the counts map and assign a code to each one:

A couple of interesting things of note show up in this loop. First, each time a new code is assigned to a token, the code value is stored in a set called used_codes. You have to keep track of the used code values during the assignment process so that no star code is inadvertently used twice.

Second, you can see that the value of each token and its code is being written to standard output. (I write the frequency count as well, although this is technically not needed. It helps me to keep an eye on the process and ensure it is working as desired.)

The Code Assignment Algorithm

In MakeDictA, I do star code assignment in a routine called create_star_code(). This algorithm uses a greedy heuristic to assign codes, attempting to use the highest possible number of '*' characters to each token.:

For a token of length N, the assignment routine first attempts to replace all characters in the token with stars, then all but one, then all but two, and so one. Note that the often-overlooked next_permutation function from the standard library helps out quite a bit with this.

Executing MakeDictA

To test this program, I went to Project Gutenberg and procured the text of eight of Shakespeare's plays. (The text is included in the source zip file for this project.) With allowances for line breaks, the text below shows the command line I used to create the dictionary file:

[c:\star] MakeDictA text\AsYouLikeIt.txt
text\Hamlet.txt
text\JuliusCaesar.txt
text\KingLear.txt
text\Macbeth.txt
text\RomeoAndJuliet.txt
text\TheTamingOfTheShrew.txt
text\TheTempest.txt
> DictA.txt


The program creates 16,927 codes in fairly short order, producing DictA.txt, a file that is shown below in somewhat abridged form:

the *** 4556
I * 4390
and **d 3676
to ** 3218
of *f 2899
you **u 2775
a a 2714
.
.[skipping 8500 lines]
.
vexations ****t**n* 1
vestall **s*a** 1
versall **r**l* 1
verities ***i***s 1
verge v*r** 1
.
. [skipping 8500 lines]
.
'Twentie *T****** 1
'Saue *S*** 1
'Prethee *P****** 1
'Pre *P** 1
'Boue *B***

With this somewhat limited vocabulary, you can see that even words that appear very rarely are replaced with a very high percentage of stars. The most frequent words are replaced completely.

Compressing and Decompressing the Transformed Data

Listing 2 and 3 show StarEncode.cpp and StarDecode.cpp, the short programs used to transform text files to and from star-encoded format. I won't go into the details of their operation - if you understand MakeDictA.cpp, you won't have any trouble with these two programs.

Both programs are invoked with the name of the dictionary file on the command line. The programs then transform standard input to standard output. Internally, they simply read the dictionary file into a map object that holds all the encodings. They then read in text, passing special characters through unchanged, and translating tokens whenever possible.

Figure 1 shows the encoding program in operation, as it encodes Romeo and Juliet using the dictionary created earlier:

Figure 1 - Encoding Romeo and Juliet

The result of this encoding should be a file that is now more compressible than the original. To test this theory, I compressed both the original and the encoded file in WinZip, with the results shown in Figure 2.

Figure 2 - Compressing the encoded file

Variant B

While building the encoder, I started wondering about the structure of my star codes. It seemed kind of arbitrary to insist that every star code have the same number of stars as the word it was encoding. For example, the most frequent word, the, was encoded as "***", while the second most frequent word, I, was encoded with a single star. Shouldn't the most frequent word get the fewest number of stars?

To test this theory, I created a second dictionary builder using a new heuristic for assigning codes. In this heuristic, a star code is simply a single star character concatenated with a plain-text code number, with the most frequent token being assigned code 0.

Using this scheme simplified my encoder quite a bit. The main loop that assigns codes to tokens no longer needed to keep track of used codes, and was simple enough to do code assignment inside the loop, instead of in a separate function:

Comparing the output from Variant B to that describe earlier shows how the two strategies create quite different dictionaries:

the *0 4556
and *1 3676
to *2 3218
of *3 2899
you *4 2775
.
.[skipping 8500 lines]
.
vexations *8480 1
vestall *8481 1
versall *8482 1
verities *8483 1
verge *8484 1
.
. [skipping 8500 lines]
.
'Twentie *16899 1
'Saue *16900 1
'Prethee *16901 1
'Pre *16902 1
'Boue *16903 1

One notable difference in this case is that I don't bother to encode strings of just a single character in length, as their encoded values would always be larger. To test this variant, I compiled MakeDictB.cpp, created DictB.txt, and ran RomeoandJuliet.txt through StarEncode. The five lines shown earlier in this article have now morphed into the following:

*43 *1069, *41 *349 *469 *1492 *2583 *12782?
*114 *7 *0 *2550, *1 *417 *7 *0 *931,
*5359 *193 *902 *1 *521 *0 *1548 *616,
*168 *7 *1283 *731 *1 *695 *13 *1042,
*34 *21 *27 *938 *144 *3817 *49 *193 *37 *67

The text is somewhat harder to read, but also a bit more compact. Note also that the character set is much more limited here, since every word is star-encoded using only the star character and the integers.

Comparing A and B

Figure 3 shows the listing of a Zip file that contains both encoded texts plus the original.

Figure 3 - A Zip file with both A and B variants

In this case, the difference between a standard Zip compressor and variant B runs nearly 15%, quite a nice savings. You can see that variant B clearly comes out ahead, with a delta that seems worth going for.

Recommendations and Caveats

Star-encoding is clearly a handy way to squeeze some additional fluff out of your files. Of course, there is a catch. For star-encoding to be effective, you have to know in advance what the vocabulary of your source files is going to be. There are many times when this will be the case. Typical examples might include:

• Computer-generated reports
• Email or newsgroup archives
• Trace or log files
• Cached web pages

I doubt that star-encoding could be used in a general purpose compressor. A hypothetical archiver called StarZip might attempt to create a good dictionary and tokenizer for various file types, but the task seems difficult.

The code I have written here is typical magazine code: short and simple. It could form the basis for production code, but would need some hardening first. For example, the programs shown here can't cope with something as simple as a star character embedded in the plain text. Caveat emptor.

There is plenty of room for experimentation here. I'd love to hear back from any readers who develop encodings that can outdo the ones shown here. Send me an email if you create something interesting here. Enjoy!

References

Robert Franceschini and Amar Mukherjee. Data Compression Using Encrypted Text. Proceedings of the third Forum on Research and Technology, Advances on Digital Libraries, ADL 96, 1996, pp. 130-138. Robert Franceschini, Holger Kruse, Nan Zhang, Raja Iqbal and Amar Mukherjee, Lossless, Reversible Transformations that Improve Text Compression Ratios.

Mark Nelson, Data Compression with the Burrows-Wheeler Transform, Dr. Dobb's Journal, September 1996.

Listings

Listing 1 - MakeDictA.cpp
Listing 2 - StarEncode.cpp
Listing 3 - StarDecode.cpp
Listing 4 - MakeDictB.cpp
Sample text - Shakespeare.zip

Windows Developer Magazine July, 2002

Back in March 1998, Tom Armstrong and I published an article in Dr. Dobb's Journal showing a technique for implementing transparent regions in bitmapped images. In those days, Windows developers had to roll their own transparency solution, and the path was quite arduous. Today, Windows programmers can render images with transparency using Microsoft's IPicture COM interface, which is considerably easier than the solution Tom and I suggested. It's an easy way to accomplish the same task - if you watch out for its single notable drawback

The IPicture interface is a good way to draw images with transparency today. In the future, you'll probably want to use GDI+, Microsoft's new GDI interface for C++ programmers. I'll show you how to convert the sample program to write cleaner code with GDI+, and discuss the improvements it brings to the table.

Transparent GIFs

When creating a GIF file, you have the option of designating one color in the image as transparent. This transparency feature is normally used to make it appear as if an irregularly shaped image is floating on the background. An example of how this can be useful is shown in Figures 1 and 2.

In Figure 1, two news articles displayed on a Web page have graphical icons that illustrate article topics. The users of this particular web site have the ability to change their personal settings to use a different layout, which changes the background color of the page. In Figure 2, a different theme is used to show the same articles.

In both Figure 1 and Figure 2, the background of the GIF file used to illustrate the article is transparent. Web browsers know that they need to let the background color show through in the transparent regions of the graphic. You can see that the browser deals with the transparency properly in both cases, with nice effect. A beige background in Figure 1, a white background in Figure 2, and in both the topic images float nicely above the background.

Figure 1 - Transparent GIFs on a light brown background

Figure 2 - The same GIFs on a white background

Loading the GIF File

Getting an external file into a format Windows can display can be a chore. Tom and I chose to skirt the issue in our original article by working with files stored in Windows' Device Independent Bitmap format. Although not a popular file format, it was one of the few that Windows was able to read and write without help from third-party libraries.

The situation has improved quite a bit since then. You can now use Microsoft's OleLoadPicture() function to load both JPEG and GIF files. Given the popularity of these two file formats, you might find that OleLoadPicture() covers all your needs in this area.

OleLoadPicture() loads data from an IStream object and stores it in an IPicture object. You'll find that having the data in IPicture format is quite convenient when it comes time to render it. However, the odds are good that you want to be able to load your data from a file, which is not quite the same thing as an IStream interface.

The easiest way I know of to get from a file to an IStream object is to simply read the whole thing into memory and then use CreateStreamOnHGlobal() to create an IStream. A fragment that accomplishes that is shown here:

At this point you have an IStream object ready for use, and all that remains is to make the call to load the IPictureobject:

Although the code isn't particularly elegant looking, it has accomplished the goal of reading a GIF or JPEG file into memory in just a dozen lines or so. All that remains is the task of drawing the code on the appropriate display.

The Drawing Part

In our original article, Tom and I presented a straightforward method for displaying bitmaps with transparency. It required the use of three bitmaps:

1. The background bitmap.
2. The bitmap to be displayed, with a known transparent color.
3. A special mask bitmap. The mask bitmap should have all bits set to zero in areas where the display bitmap is opaque, and all bits set to one in the areas where the display bitmap is transparent.

Displaying the bitmap is simply a matter of performing the following sequence of operations:

Output = Bitmap1 XOR Bitmap2 AND Bitmap3 XOR Bitmap 2

With just a little bit of work, you can see that the output will be a copy of the pixel from Bitmap1 in every position where the mask (Bitmap3) has a value of all ones. The output will be a copy from Bitmap2 in every position where the mask has a value of all zeros.

That's all you really need to understand in order to draw bitmaps with transparency in any sort of graphics environment. What made it into a long article in 1998 was simply all the plumbing you had to do in order to get the background bitmap, create the mask, get the display bitmap, and so on.

The good news is that Microsoft's IPicture interface can take care of all of the messy details here with one call to its Render() function. Once you have loaded the GIF file into an IPicture object, the snippet of code below is all you need to get it onto your display:

Note that the code shown here centers the image in the client window, which complicates things just a bit. It also has to deal with the fact that IPicture uses the HIMETRIC mapping mode, which I have to convert to the more convenient pixel measurements.

A First Pass

Pressing this code into service reveals one striking problem with the IPicture interface. Calling its Render() function with the intention of writing directly to your display window leads to a somewhat unpleasant surprise. It turns out that IPicture uses the standard algorithm I describe earlier to implement transparency. Unfortunately, it performs each of the three steps directly on the display, so you get to see each intermediate bitmap in the process. The four screen shots below show it in all its glory.

Figure 3a - Opening the file

Figure 3b - The background has been XORed with the image

Figure 3c - The resulting bitmap has now been ANDed with the special transparency mask

Figure 3d - After a final XOR, the image is rendered with transparency

This sequence might be interesting to the average DDJ reader, but the end users of your program are hardly going to want to see these three images flashing in quick succession across their screen. The flickering is barely noticeable with small images, but on slow machines, big images go through the painting sequence with painful precision.

The solution to this problem is straightforward. Instead of drawing directly to the screen, you can create an off-screen bitmap, have Render() draw to that bitmap, then copy the completed product to the display. Having done that, you'll find that not only is the flicker eliminated, but the drawing operation actually completes much more quickly.

An MFC routine that accomplishes this in conjunction with the rendering routine above is shown here:

You can see that it only takes a few lines to create the off-screen bitmap. The code above creates a bitmap of the correct size, fills it with the correct background pattern, then calls the rendering routine. Once that is done, all that is left is a BitBlt() call which copies the result to the screen.

The Sample Program

To demonstrate this, I created an MFC program named TransparentGif, forgoing the usual document/view architecture. Instead, I just paint the entire client background of the window with a pattern, the better to view the transparency of the loaded image.

When the program first starts up, by default it draws directly to the screen. If you are blessed with a sub-Gigahertz CPU, you should be able to see the flickering effect quite clearly when loading the sample GIF included with the source. I added code to force a redraw every time you click on the image, which allows you to see the effect in all its beauty.

You can then use the Options|Off Screen Draw menu item to switch to the use of the off-screen bitmap. A little sampling between the two should be enough to convince you of the value of the off-screen drawing technique.

Switcing to GDI+

Looking forward to the future, you might find that this task is going to become a bit easier. Microsoft has developed a new interface to GDI for Windows programmers called GDI+. The class libraries and supporting DLL for GDI+ are shipping as part of Windows XP, and can be installed as an add-on to Windows NT, 2000, 98, and ME.

The header files and import libraries required for GDI+ are part of Microsoft's Platform SDK. To get this SDK, you'll have to have to subscribe to Microsoft's MSDN program at the Operating System Level or better, which has a list price of $699.

You need to perform two installations to develop and test GDI+ programs: the Win32 Platform SDK and the GDI+ runtime files. Both of these can be found on your MSDN CDs or can be downloaded directly from the MSDN site The GDI+ runtime kit can also be distributed with your software, so your end users need no special licensing.

Additions to the project

I made a copy of the TransparentGif program, renaming all the files and classes so that the new project goes by the name TransparentPlus. I then set to work making the changes needed to use GDI+ as the graphics interface.

The first few changes are the ones you typically have to make when using a new library. I added an include of the master header file, gdiplus.h, to stdafx.h. This ensures that I have all the correct prototypes and class definitions. To make sure that the compiler was able to find the header file, I had to add the SDK include directory to the preprocessor settings for the C++ compiler. I used the default installation location for the SDK, so I simply inserted the path:

C:\Program Files\Microsoft SDK\Include

into the list of additional directories for the compiler.

All of the classes and identifiers in GDI+ are placed in the Gdiplus namespace, so I also added a using namespace Gdiplus; declaration to stdafx.h. In a larger project, I probably wouldn't do this; the use of explicit namespaces in the source would add readability to the code.

Likewise, the import library was in the default SDK directory, so all I had to do was drag:

C:\Program Files\Microsoft SDK\lib\GdiPlus.lib

into my workspace pane, thereby adding it to the project.

Startup code

GDI+ programs require a few lines of code for initialization, as well as a cleanup call. I added the following lines to the InitInstance() function in my application class:

This code is lifted cookie-cutter style straight out of the GDI+ documentation. The token initialized by the startup function is passed to the GdiplusShutdown()method when you are done using the library.

File Loading

As you recall, the file loading code in the standard GDI program was a real rat's nest. The hoops I had to jump through in order to load a file into an IPicture object gave a classic demonstration of an ugly Microsoft API.

I'm happy to say that someone working in API design in Redmond has seen the light in this area. Loading images is an order of magnitude easier with GDI+. Instead of loading into an IPicture COM object, GDI+ supports a C++ Image object. This requires that I change the type of CMainFrame::m_pPicture in the appropriate header file. I can then reduce the file loading code from the ugly mess I had before to this picture of simplicity:

This is a huge improvement, and by itself is almost worth the cost of the SDK.

Drawing

The drawing code with GDI+ is a bit simpler as well, mostly because the Image object uses pixels for measurement instead of the HIMETRIC units from IPicture. Other than that a simple replacement of the Render() function with a call to DrawImage() constitutes most of the work. The updated routine is shown here:

GDI+ makes my code look a lot nicer, and informal tests show that GDI+ renders a bit faster as well. Better yet, GDI+ is smart enough to avoid rendering all three steps directly on to the screen, which means you won't see a flash of the XOR'd or masked image.

Despite the speed of GDI+, I think you'll still find that you need to do this image rendering in an off-screen bitmap. Experimenting with the second test program shows that although you don't see intermediate images, you still get an annoying flickering as the image as drawn. Rendering off-screen eliminates that problem.

Conclusion

This article gives you a couple of different ways to effectively use transparent GIF files in your Win32 programs. For today, the IPicture interface does a pretty good job. When GDI+ rolls out as a standard part of Visual Studio, you'll probably want to switch over to it for your imaging needs. Either way, you've got good options that make things look a lot nicer than they did in 1998.

Links

Source for both demo programs

Dr. Dobb's Journal July, 2002

Update: Intel is no longer giving these libraries away for free - be prepared to fork out real money for this code. Does this render the information in this article useless? I don't know, add your comments below!

In this article I take a look at Intel's Win32 JPEG Library, and present a slide-show app I built to test it. I found it quite easy to render JPEG images using this library, and I think the demo code I provide will provide you with a good starting point for your own work. I'll also perform a short comparison of Intel's library against Microsoft's standard IPicture component, both in terms of performance and ease of use.

Intel's Performance Libraries

Not too many people are aware of the fact that Intel dedicates a substantial amount of energy to its software products. As you would expect, most of the software Intel creates is designed to pull through more hardware sales, not to generate revenue of its own. One good example of this is Intel's JPG Library.

Intel added the MMX instruction sets to their CPU chips starting with their 486 Pentium MMX processors. The MMX instructions are designed to speed up certain operations needed in multimedia applications. One of these is the Inverse Discrete Cosine Transform (IDCT), which is used to when decoding JPEG images.

Developers typically can't take advantage of extended instruction sets in products designed for a widespread market, because they can't count on their target machines having the correct extensions. Companies like Intel or AMD can still leverage their proprietary processor features by providing device drivers, browser plugins, or other components, but this doesn't completely satisfy their desire for market penetration.

For example, my sample slide-show program doesn't use operating system components to decode JPEG files, meaning I can't take advantage of the MMX instruction set without writing specific routines to detect the target processor type, then branch to one of two different decoders depending on the result.

Intel correctly figures that they can convince me to offer support for their hardware if they dish it up on a silver platter, and that's exactly what they do with their series of Performance Libraries. At this time, Intel is offering the following roster of libraries:

• Intel Integrated Performance Primitives - an abstraction layer for multimedia processing.
• Intel Math Kernel Library - linear algebra and FFT routines.
• Intel Recognition Primitives Library - code for speech and character recognition software.
• Intel Signal Processing Library - access to Intel CPU functionality similar to that used in Digital Signal Processors.
• Intel Image Processing Library - image manipulation functions, including filtering, thresholding, morphing, FFT, and DCT.
• Intel JPEG Library - encoding and decoding of JPEG images.

All of these libraries have processor specific routines for various families of Intel CPUs, and all have generic code that will also run on any standard Intel Architecture CPU.

What's the Catch

In a clear case of enlightened self-interest, Intel is making these libraries freely available to software developers. You will have to go through a licensing procedure to in order to redistribute the libraries with your products, but as far as I can tell there aren't any circumstances under which Intel will require payments or royalties.

Support is another matter, as you might expect. Intel's web site has FAQ material, known issues, and so on. Talking to a person will require subscription to a premium support service.

Where to go

You can find Intel's JPEG library on their Web site, www.intel.com, by following the links to
Products/Software/Performance Libraries. At the time this article is going to press, the direct link is at:

http://www.intel.com/cd/software/products/asmo-na/eng/perflib/ipp/index.htm.

The library has many of the niceties that you would expect from Intel. Figure 1 shows one of the first screens from the professional installation program, which naturally allows for a full uninstall as well.

Figure 1 - Installing the Library

As you install the library, you'll also see that it includes examples, white papers, documentation, and support for C/C++, Visual Basic, and Delphi.

Using the Library - an Example

I originally stumbled upon Intel's library while searching for a solution to a pressing problem. My friend Darrick Deel and I were intent on supplying some entertainment to a company shindig at Cisco's development offices in Dallas. We were going to act as roving digital photographers, snapping candid shots of our coworkers as they engaged in atypical mirth and merriment.

Our goal was to use a notebook PC and a projector to randomly display shots of partygoers during the event. What we needed was a slideshow
program that would randomly display JPEG photos from a directory tree. We were hoping for a program that would run in unattended mode, but would still be able to periodically scan the photo archive looking for new shots. This would allow Darrick and I to add photos across the network without having to break into the show.

Slideshow software is easy to come by. As Figure 2 shows, a simple search on a popular download site turns up literally hundreds of possibilities. A surprisingly large percentage of this software is either free or quite cheap.

Figure 2- Searching for Slide Show Software

Despite the bountiful selection of software, Darrick and I were suffering from the Goldilocks syndrome - nothing we tried out was just right. Following in an age-old tradition among programmers, I decided to reinvent the wheel. Having recently downloaded Intel's JPEG Library, I knew I had just the tool I needed to make a quick
job of it.

Using Intel's Library

The library ships from Intel with a pretty decent manual in PDF format, and a nicely fleshed out sample program called JPGView. The manual is full of code fragments that do a good job of illustrating the various features of the library. After reading the manual and examining the sample code, the process of displaying a JPEG image on the screen looked pretty straightforward:

1. Initialize the library.
2. Read the JPEG file header to get the basic image information, including size and color space.
3. Set up the various members of the data structure that will hold a Win32 Device Independent Bitmap (DIB.)
4. Read the JPEG image data into the DIB.
5. Display the DIB.

The first four steps here are accomplished in a single routine I wrote called CreateImage. This function takes an input file name as a parameter and returns a pointer to a JPEG_CORE_PROPERTIES structure, which is where the Intel library stores all its information.

The final step is done in the WM_PAINT handler for the program's WndProc. Since the first four steps create a fully functional DIB, all the WM_PAINT handler has to do is set up some parameters and draw the DIB to the output device context.

SlideShow

I put this knowledge together in an imaginatively named program called SlideShow. Although the code is built using Visual C++, it doesn't use MFC, so you should have good luck using the program with Brand X C++ compilers.

SlideShow starts off as a console program that takes a directory name as a command line argument. That directory name should be the root directory of your album of JPEG files. Slideshow then creates a full-screen main window, starts a five second timer, and proceeds to process the message loop.

Other than a little bit of glue code, the message handling routine for this program really only cares about two messages: WM_PAINT and WM_TIMER.

WM_TIMER

The main window of SlideShow gets a timer tick once every five seconds. It has three main jobs to do when this timer tick occurs:

1. If needed, reload the list of pictures to display.
2. Destroy the current picture object.
3. Create a new picture object.

SlideShow keeps a list of all the pictures it finds in the JPEG directory in a C++ vector named WindowData::picture_names. It also keeps an index of the next picture to decode in WindowData::current_picture. As soon as the timer handler is entered, it checks to see if the current_picture value is at or past the end of the picture_names vector. If it is, the current contents of the vector are erased and the picture directories are scanned for a new list of picture names. (Note that this happens naturally the first time the WM_TIMER tick is processed.) The picture names are randomized so the sequences aren't repeated each time the program is executed.

After potential maintenance of the image name list, it's a simple matter to delete the current image and create a new one. CreateImage,
described a bit earlier in this article, does all the processing needed to load the JPEG image into a Device Independent Bitmap. This new image is ready to be painted on the screen, and I force that to happen by calling the Windows API function, InvalidateRect(). This forces a WM_PAINT message to be sent to the main window.

WM_PAINT

The WM_PAINT message means it is time to paint the JPEG image onto the screen. I've characterized this as being an easy task since the image is now in a DIB, and examination of the WM_PAINT handler in Listing 1 bears this out.

The actual painting is done via a call to Windows API function StretchDIBits(). This routine not only copies the image to the screen, but scales it as well. There are a dozen or so lines of code that set up the call to this function. I scale the image so that it fills as much of the screen as possible without clipping any bits, filling any unused space with the desktop background color.

Figure 3 shows what this looks like on the screen of a PC. The program quietly paints a new random image on the screen every five seconds. The machine shown here is a modified I-opener, which runs Windows 98 and has full network access. With no cooling fan to create ambient noise, the I-opener is an ideal desktop photo album.

Figure 3 - The Slideshow in Action

Making Windows into Objects

If you are used to using MFC for all of your C++ projects under Windows, you might find this to-the-API program a bit different. One thing I did to try and make it a little more structured is to encapsulate the Window data in a C struct, which could just as easily be a class.

When the program first starts, I create a WindowData object, which contains all the information regarding the currently selected
picture, the list of file names, and so on. When the window is created, I stuff a pointer to this object into the GWL_USERDATA DWORD owned by the window. From that point on, every time I enter the WndProc I can get a pointer to the structure, eliminating the need for global variables.

Integrating with Visual Studio

The download package for this program contains a project file, but you may need to do a bit of modification after you install Intel's JPG Library.

First of all, the SlideShow.cpp file includes Intel's header file, ijl.h, as shown in this section of code from the source file:

Naturally, this file won't be in your normal include path. I take care of this by going to the Project|Settings dialog in Visual C++, selecting the C/C++ pane, then the Preprocessor category. You should then enter Intel's header directory in the Additional Include Directories edit box. If you take the default installation, your dialog will look like Figure 4.

Figure 4 - Configuring the Include Path

The project file I built for SlideShow includes the static library, ijl5l.lib. If you took the default path for Intel's install, you don't need to make any changes whatsoever. Figure 5 shows the path that is defined in the current project.

If you would like to use Intel's DLL instead of the static library, or if you chose a different install path, you'll need to delete the current library from the project. After deleting, use menu option Project|Add to Project|Files to add the correct library.

Figure 5 - The correct properties for the Intel Library in Visual Studio

Intel vs. Microsoft

There are any number of reasons you might choose to use Intel's library to display JPEG images. But what about Microsoft's solution? Does the standard distribution of Windows come with components that can perform just as well?

As it happens, Microsoft ships with a standard COM object called IPicture that shares much of the functionality with Intel's library. It not only handles JPEG files, but works with GIF, BMP, and other formats as well.

Ease of use

To get a handle on its ease of use, I reworked my SlideShow.cpp to use an all Micrsoft solution. The resulting program, SlideShowMS.cpp (available for download) shares the same structure, and is probably nearly identical in terms of structure and complexity. The major changes are:

• Microsoft's image data is stored in the IPicture component, Intel's is in a JPEG_CORE_PROPERTIES structure.
• Reading the file data requires two steps and a bit of coding for Intel. Microsoft makes reading easier with the simple OleLoadPicture function, but loses points for not reading from standard files. Setting up the IStream object Microsoft needs throws away any advantage they have in this area.
• I used a standard Windows GDI function to draw the Intel object to the screen. IPicture has a Render method that is somewhat easier to use and requires less setup.

After examining these points, I think you'll agree that there's no clear winner here. If all that matters to you is how much work you have to perform to solve your imaging problems, you can stick with your personal preference and not worry about making a big mistake.

Performance

When it comes to decoding JPEG files and rendering them to the screen, Intel's library stole the show. I ran the Slideshow program through a batch of 20 JPEG files that were mostly 1600x1200 pixels in size. My non-scientific test game the following performance results:

Intel's library seems to take Microsoft's time and cut it in half! It appears that Intel's attention to performance paid off. (Interestingly, I would have expected to see a bigger delta between Intel and Microsoft on the Celeron CPU, but was disappointed.)

I'm sure there are ways to tweak my SlideShowMS program to get better performance from Microsoft's implementation. But the point of this article is to look at an off-the-shelf solution for use by programmers who may not be graphics experts. And for a straightforward implementation, Intel appears to really deliver when it comes to CPU cycles.

What about IJG?

The JPEG file format enjoys great acceptance in the software world. This is due in no small part to the availability of the Independent JPEG Group's (IJG) source code. The IJG created a non-commercial source code package that has been used as a reference and/or library by literally hundreds of programs out there. It enjoys a great deal of respect for its flexibility and feature depth.

Intel knows that many developers have already integrated the IJG code into their products, so they have created a 26-page white paper that discusses what you need to perform open-heart library replacement. There are major structural differences between the two sets of code, so this isn't an exercise for the faint of heart. But it can be done.

Summary

I found Intel's library quite easy to work with, which is probably evident given the length of my sample application. While I didn't do too much objective performance testing, I was happy with the speed I achieved when decoding and displaying multi-megapixel JPEG files on various PCs.

My satisfaction with the library is certainly only enhanced by the absence of a price tag. Intel's JPG library is definitely part of my image processing toolbox, you might want to add it to yours as well.

Article Source Code

PC Magazine July, 2002

Remember when PC Magazine used to give away their utilities for free? Ah yes, those where the days. The earliest PC Magazine utilities were distributed as PC-BASIC programs printed up in the Utilities section, which made good typing accuracy an important skill. Eventually this was supplanted by electronic distribution via BBS and then the Internet. A lot of useful programs were thrown our way for free, and PC Magazine presumably bathed in good will and reflected glory.

Back in July, 2002, I wrote one of those free utilities for PC Magazine. CubeShow was a screensaver that featured a tumbling representation of a 3D cube, adorned with photos of your choice. Having never written a line of DirectX code in my life, it was quite an adventure, but I think I managed to produce something that worked and was popular.

Some of the more interesting problems involved with writing this utility were:

• Finding the right rotation parameters to achieve smooth animation
• Figuring out how to change photos on a cube fact only when it was hidden
• Keeping a photo upright as the cube rotated

A lot of fun!

Figure 1 - CubeShow in action

Links

CubeShow on PCMag.com
CubeShow download link

Windows:: Developer January, 2002

This article describes a URL validation program I wrote in Java, using standard Internet and database language features. I'll show you why Java is the clear choice of languages for this job. Finally, I'll show you how to work around one particularly egregious shortcoming in Sun's JDK 1.3.

Introduction

For several years now I've maintained a web site called DataCompression.info. (Note - this site passed out of my hands in 2003). I have one steady advertiser who helps me cover my hosting costs, but when you consider the time and energy required to keep the site up you'll understand that it is a labor of love.

The Data Compression Library is simply a collection of links to sites related to data compression. I have a database of over 2000 URLs, categorized with one or more of thirty-some topics. On a periodic basis I run a program that passes over the database and generates a batch of pages I upload to a server. Figure 1 shows the main page for the site, and Figure 2 shows a typical topic page.

Figure 1
The Front Page of the Data Compression Library

Figure 2
A Typical Topic Page

Wish You Were Here

One of the biggest ongoing problems maintaining this site is dealing with bad links. Over the course of a year, I have to contend with literally hundreds of sites that change URLs, move within an existing domain, or just disappear from the face of the earth. My users tend to become annoyed when every other link ends up with an HTTP 404 error.

When I first started working on the site I was operating under the illusion that dead links could be dealt with through manual labor on my part and faithful error reporting from my users. It turns out that both of these assets are in short supply. After a couple of years I had more bad URLs than Microsoft has lawsuits. It was time for a change.

An Automated Solution

I knew exactly what I needed: an automated scanning program that I can run periodically. The scanner could update my database with its results, and generate a report showing me which sites are repeatedly AWOL.

Despite the clear need for this program, I had been avoiding getting down to business for some time. My tool of choice for utility programs has always been Visual C++, and the thought of writing more database code in Microsoft's universe was far from appealing.

The problem in this case isn't C++, Visual Studio, or even Microsoft. I get along with all three on a daily basis. The problem is the unfriendly nature of Microsoft's C++ interface to ODBC, their database API of choice. At least from my perspective, it seems that every piece of data that is passed back and forth between my C++ code and the database has to go through at least one or two type transformations. The database code is never content to take a C++ string type, or even a simple character array. Instead, every string has to be converted to type _variant_t or _bstr_t, two monstrosities that Microsoft apparently cooked up for Visual Basic programmers. Visual Basic is weakly typed, C++ is strongly typed. It's not fair to make either of them use the other's type system, but the API developers in Redmond decided a Procustean fit was just fine.

Data read in from the database typically comes in as a variant type, which means that before I can even use it, I first have to check the type, then examine the value, leading to lots of code that looks something like this:

To complicate matters more, there was a matter of another familiar complaint I have with Microsoft's COM interfaces: error handling. Working through a database typically involves a fairly long batch of calls to API calls, each of which can return an error. Dealing with deeply nested error conditions leaves you with truly messy code:

I don't like code like this, whether I'm reading it or writing it.

Java to the Rescue

Fortunately, before I started writing my scanner in C++, I found myself knee-deep in Java programming at work. To my great pleasure, I saw that database programming in Java using JDBC worked around all of my annoyances with ODBC in a manner that made my job a lot easier.

Java's standard interface to databases is done through the Java Database Connectivity interface, known by the trademarked name JDBC. JDBC has two key points that make it look good in comparison to C++/ODBC. First, JDBC uses native types to interface to the database - eliminating the need to convert between variant or non-standard string types. Second, error handling in JDBC is managed via exceptions, which cuts out all those conditional tests I had to use with ODBC.

With these conditions, I found that I was able to implement my scanner in 100 lines of straightforward Java code. The Java code was easier to write, easier to debug, and has been easier to maintain.

Details

Figure 3 shows a screenshot from a Microsoft Access view of my URL database. That view shows all of the pertinent data from the URL table. The fields in each row that are updated by the scanner program are the Scan History, the Last Good Scan field, and the Last Scan Result field.

Figure 3 - A View of the URL Table

The job of the scanner program is to go through all the rows in the URL table. The program attempts to access the web page pointed to by the URL. If the page is found, a hyphen character is appended to the Scan History. If the page isn't found, the character F is appended. (The Scan History is always truncated to a maximum of 16 characters.)

If the page is found, the Last Good Scan field is set to the current date. In the event of a failure, the Java exception code is written to the Last Scan Result. This gives me the information I need to quickly scan through the database and determine which sites are going to give my users trouble.

The Java code to accomplish this is basically divided into two pieces: setup and scanning. The first step is done in the constructor of the HttpScan object, and the second is done in its member function scan(). I call both of these from the static main function in the HttpScan class:

Note that both functions are wrapped in a try/catch block. This means I get to handle fatal errors by simply throwing or propagating an exception, knowing that it will be properly displayed when caught.

Connecting to the Database

JDBC ships with an ODBC bridge for Windows, which means the only database setup you need to perform is to set up an ODBC data source. Under Windows 2000, you can do this from the Administrative Tools section. Figure 4 shows part of the process. The text box labeled Data Source Name defines the name that you use to connect to the database with the call to DriverManager.getConnection().

Figure 4 - Setting up the ODBC Data Source

The full code for the connection to the database is in the HttpScan constructor. The steps taken sequentially are:

1. Load the JDBC-ODBC bridge.
2. Connect to the database.
3. Load the recordset containing all of the http links in my database.
4. Ask the database for the count of links in the database.
5. Create an update statement that will be used throughout the program.

After the constructor executes, I have a count of records in member variable recordCount, a copy of all the records in member variable results, and a member variable called updateStatement that will be used to update the records in the database.

Scanning the records

Once the connection has been made and the recordset loaded, the member function scan() is called. Its job is to attempt to connect to every URL in the recordset. In Listing 1, you'll see that I declare that scan() throws an SQLException - which means I don't have to do any fancy recovery when a fatal error is encountered. I rely on the caller, main(), to catch the exception and print out the corresponding information.

At that point, the scanner enters the main loop, which executes as long as I can successfully call results.next(), which moves to the next available database record. In Listing 1, you can see some typical code used to get data from a record with code that looks like this:

You can immediately see why I would prefer this code to the equivalent C++ code under Windows. All of the data is returned in native Java types, I pass in parameters in native types, and I allow error handling to be done via exception. The result? Code that's easy to read, easy to write!

Once I have the URL in hand, the next step is to create and open a URLConnection object, part of the standard java.net package. After creating the response code, I just check the responseCode member of
the URLConnection object. If I received a 200, it means that the URLConnection object was able to
locate the web site and read in the page. Anything else means an error of some sort.

Note that the code that the library calls that loads the URL may well throw an exception, which I catch inside the scan() routine. When that happens, the Exception object will typically hold useful information as well.

Performing the Updates

The final task that the scan() loop performs is to update the database with the results for the given URL. This is done from one of four different places in the scan loop, where three points represent various failures, and one represents success.

The Update() routine takes four parameters: the record key, the 16 character history string, the status flag, (which is '-' on success and 'F' on failure,) and a string which contains any error message from the attempt to connect.

Again, this is where Java shines in comparison to the Visual C++ solution. As you can see from Listing 1, creating the SQL statement that performs the update is all done using native types and straightforward code.

Using the Program

Running the program is straightforward. The program is a simple console app that displays its results as it runs. I invoke the program from the command line with the following command:

Java -classpath . HttpScan

Figure 5 shows the program in action:

Figure 5 - HttpScan in Action

I try to run the scanner program once a week or so. After each run, I generate a report showing which sites have failed several consecutive scans. From that point I have to go through the laborious process of determining whether the site has moved, gone under, or is just plain flaky.

A Catch

When I first developed this program, I assumed that one complete run through the database would take on the order of an hour or two. The program reaches most URLs in a matter of seconds, with some of the more distant sites taking as long as sixty seconds. So I was quite disappointed when I let the program run overnight and found that it was stalled on one particular site after only running through roughly 100 URLs.

A little research into Sun's bug database showed that I was far from the first person to see this problem. For an example, look up bug 4304701, 4143518, or 4283433. Basically the problem is that socket connections used by URLConnection are opened without a timeout value, which means they can hang forever when network problems crop up.

Of course, there are quite a few possible ways to solve this problem, and no shortage of suggestions in the appropriate newsgroups. Some of the proposed solutions seem to be platform dependent, and may not work in all environments. The only simple solution I found that worked properly on Win32 platforms was to put my URLConnection attempts in a separate thread, and then to call Thread.stop() after an unreasonable amount of time with no results. But Thread.stop() is a deprecated function call, and has the ugly possibility of creating resource leaks.

My Solution

The solution I implemented in HttpScan was based on a post to the java.sun.com site by a person named Denis Haskin. Denis's workaround replaces the default SocketFactory used by the java.net classes with a slightly modified version. The modified version creates sockets that have a timeout set upon creation. Since classes such as URLConnection rely on the default socket factory for their sockets, inserting a replacement factory is an elegant workaround to the problem.

My version of the factory class is called TimeoutSocketFactory, and is shown in Listing 2. You can see that the class is incredibly simple. All it does is return a copy of a socket class I created called MySocketImpl. This class is derived from the PlainSocketImpl class defined in java.net, and differs in only one function: connect(). My version of the connect() function calls the base class connect(), then sets the connection to have a specified timeout. Since the base class defines all of the remaining behavior, I'm confident that my derived socket class won't cause any other problems.

Once I created this class and integrated it with my HttpScan program, things worked perfectly. I make a single call at the start of the program that creates this new socket factory using this code:

It now takes a bit over an hour to scan through the entire database, and a fair number sites do actually generate a timeout.

There is one big catch to this technique. I have to derive my special socket class from PlainSocketImpl, the standard socket class used by the java.net package. Unfortunately, PlainSocketImpl is only visible in the java.net package. So my entire socket factory has to be hoisted into the java.net package, which is clearly coloring outside the lines.

In order to add a class to the package like this, I simply create a java/net directory and put the source file there. I then have to add the following switch to the VM when running the program:

-Xbootclasspath/a:.

This tells the VM to look the other way while I inject some additional code into the library space. When running with JBuilder, VM parameters are added to the Project Properties Dialog on the Run tab.

Sun understands that this is a problem, and documents some of the shortcomings of the current architecture in bug ID 4245730. In fact, the bug documentation is pessimistic enough to say "SocketImplFactories are a dead end." Personally, I'm happy with the workaround.

The JDK 1.4 Solution

By the time you read this, Sun may have released version 1.4 of Java. If they have, this program can use a simpler solution to this problem. In the 1.4 JDK, Sun has documented a pair of system properties that allow you to modify the default timeouts used by sockets. These two properties are sun.net.client.defaultConnectTimeout and sun.net.client.defaultReadTimeout, and both specify timeout values in milliseconds.

I was able to modify the static main() method of HttpScan to look likethis with the 1.4 JDK:

I could then run the program successfully with the 1.4 JDK after removing my modified socket factory. Once that JDK is released and working well I will undoubtedly modify my program to use this more conventional method of controlling timeouts.

Summary

I was very pleased with how easy this project was developed in Java. As I said at the start of the article, my issues with C++ all melted away when I saw the light with Sun. The two technologies used here, JDBC and java.net, are both powerful and easy to use. I hope that the code presented here helps you with your next project in this area.

Listing 1 - HttpScan.java

Listing 2 - java/net/TimeoutSocketFactory.java

Download - source.zip

C/C++ Users Journal March, 2002

Note: Thanks to Shawn McGee for pointing out an error in Figure 1. The print edition of this article in C/C++ Users Journal had an unfortunate extra line!

My daughter's math teacher at Hockaday School in Dallas wants his sixth-grade students to enjoy their class. He's fond of sending home interesting problems that are meant to be both entertaining and enriching. As most parents probably know, this can only mean trouble!

Last week Mr. Bourek sent home a worksheet containing a set of variations on the traditional magic square. Students were given various shapes, such as triangles, stars, and so on, and asked to fill in a set of consecutive numbers at the vertices. The goal was to come up with an arrangement of numbers such that various rows, columns, and diagonals all added up to a given sum.

Kaitlin worked her way through most of the problems in fairly quick order. But the shape shown in Figure 1 managed to stump her.

Figure 1 - Sum = 17

The problem was simple enough. All she had to do was place the numbers 1 through 9 in the nine positions of the figure so that the sum of all the straight lines was 17. Although Kate was able to knock the other problems out quickly, this one was still unsolved after fifteen minutes or so; well past the normal sixth-grade attention span.

Even worse, after another 10 minutes of my help we were no closer to a solution. That's when I decided it was time for a brute force approach. I remembered that the standard C++ library had a handy function, next_permutation(), that would let me iterate through all the possible arrangements of the figure with just a couple of lines of code. All I had to do was check the five different sums for each permutation and I'd have the answer in no time.

The resulting program is shown in Listing 1, and its output is given below:


100706 : 3 5 9 8 2 1 6 4 7
114154 : 3 8 6 5 2 4 9 1 7
246489 : 7 1 9 4 2 5 6 8 3
259937 : 7 4 6 1 2 8 9 5 3
362880 permutations were tested


A little quick sketching will show you that the four solutions are simply rotations and mirror images of the one true solution. You can also see that randomly putting down numbers makes the odds almost 100,000:1 against finding a solution. Not quite as bad as the lottery, but it clearly shows that random guessing isn't going to work. (My daughter asked me to give her the center position only, upon which she solved the rest of it in roughly 30 seconds.)

Magic Permutations

From this program you can see that next_permutation() is a handy function to have in the C++ library. In my case it meant the difference between writing an impulse program versus fiddling around with pencil and paper for another hour. What really makes next_permuation() interesting to me is the fact that it can generate permutations without keeping any additional information beyond the sequence being juggled. I can generate a permutation, go off and do whatever I like with it, even write the numbers out to a file and save them for later. Regardless of what I do, next_permuation() will always be happy to generate the next set in the series given only the previous one as input.

Just writing a function to generate permutations isn't particularly hard. One easy way to tackle the problem is with a recursive approach. If the string you want to permute is n characters long, you execute a loop that makes one pass per character in the string. Each time through the loop you remove character i from the string, and keep it as a prefix. You then print out all the permutations of the remaining substring concatenated with the prefix. How do you get the list of permutations of the substring? By recursively calling the permutation function. The only additional piece of logic you need to include is the test to see if a substring is only one character long. If it is, you don't need to call the permutation function, because you already have the only permutation of the string.

For example, to print the permutations of "abc", you will first strip off the "a" character, and then get the permutations of "bc". To get those permutations, you will first strip off the "b" character, and get a resulting permutation list of "c". You then strip off the "c" character, and get a resulting permutation of "b". The results when combined with the prefix character of "a" give strings "abc" and "acb".

You then repeat the process for prefix "b" and substring "ac", then for prefix "c" and substring "ab".

Using the string class in the C++ standard library makes it fairly easy to implement this logic. Listing 2 shows permute.cpp which implements this algorithm relatively faithfully.

This approach to generating permutations is okay, but its recursive nature makes it unattractive for use in a library. To use this in a library we would have to employ a function pointer that would be invoked from deep inside the chain of function calls. That would work, but it's definitely not the nicest way to do it.

next_permutation() manages to avoid this trouble by using a simple algorithm that can sequentially generate all the permutations of a sequence (in the same order as the algorithm I described above) without maintaining any internal state information. The first time I saw this code was in the original STL published by Alexander Stepanov and Ming Lee at Hewlett-Packard. The original code is shown in Listing 3.

Using this function is simple. You call it repetitively, asking it to permute a given sequence. If you start with a sequence in ascending order, next_permutation() will work its way through all possible permutations of the sequence, eventually returning a value of false when there are no more permutations left.

Internals of next_permutation()

You don't need to be an STL expert to understand this code, but if you've never been exposed to this new part of the C++ standard library, there are a few things you need to know.

First, iterators (and the BidirectionalIterator type used here) are an STL abstraction of pointers. When looking at this code you can mentally think of the iterators as pointers. The permutation sequence is defined by iterators first and last. first points to the first element in the sequence, while last points one past the last element.

The code shown in Listing 3 also uses two other STL functions. iter_swap() swaps the values ponted to by its two arguments. And reverse() simply reverses the sequence defined by its two arguments. By convention of course, the first argument points to the start of the sequence to be reversed, and the last argument points one past the end of the sequence.

To help illustrate the workings of this algorithm, I've included a listing of a permutation sequence in Figure 2. It contains all 120 permutations of a five digit sequence. With that output example, plus Listing 3, it is fairly easy to see how this code works.

The function first does a cursory check for sequences of length 0 or 1, and returns false if it finds either. Naturally, sequences of those lengths only have one permutation, so they must always return false.

After passing those tests, the algorithm goes into a search loop. It starts at the end of the sequence and works its way towards the front, looking for two consecutive members of the sequence where member n is less than member n+1. These members are pointed to by iterators i and ii respectively. If it doesn't find two values that pass this test, it means all permutations have been generated. You can see this is the case in Figure 2 for the very last value, '54321'.

Once iterators i and ii have been properly located, there are still a few more steps left. The next step is to again start searching from the end of the sequence for the first member that is greater than or equal to the member pointed to by i. Because of the previous search for i and ii, we know that at worst the search will end at ii, but it might end earlier. Once this member is located, it is pointed to by iterator j.

Once these three iterators are located, there are only two more simple steps. First, a call is made to iter_swap( i, j ). This simply swaps the members pointed to by i and j. Finally, a call is made to reverse( ii, last ). This has the effect of reversing the sequence that starts at ii and ends at the end of the sequence.

The end result is a routine that is short, simple, and runs in linear time. You really can't ask for much more than that.

Walk through an example

For a quick look at the algorithm in action, consider what happens when you call next_permutation( "23541" ). After passing through the initial size tests, the algorithm will search for suitable values for iterators i and ii. (Remember that you are searching from the end of the sequence for the first adjacent pair where the value pointed to by i is less than the value pointed to by ii, and i is one less than ii.) The first pair of values that meet the test are seen when i points to 3 and ii points to 5. After that, a second search starts from the end for the first value of j where j points to a greater value than that pointed to by i. This is seen when j points to 4.

Once the three iterators are set, there are only two tasks left to perform. The first is to call iter_swap( i, j ), which swaps the values pointed to by the iterators i and j. After you do this, you are left with the modified sequence "24531". The last step is to call reverse( ii, last ), which reverses the sequence starting at ii and finishing at the end of the sequence. This yields "24135". Examining Figure 2 shows that the result demonstrated here does agree with the output of the program.

An additional charming attribute

The algorithm shown here has one additional feature that is quite useful. It properly generates permutations when some of the members of the input sequence have identical values. For example, when I generate all the permutations of "ABCDE", I will get 120 unique character sequences. But when I generate all the permutations of "AAABB", I only get 10. This is because there are 6 different identical permutations of "AAA", and 2 identical permutations of "BB".

When I run this input set through a set of calls to next_permutation(), I see the correct output:


AAABB AABAB AABBA ABAAB ABABA ABBAA BAAAB BAABA BABAA BBAAA


This might have you scratching your head a bit. How does the algorithm know that there are 6 identical permutations of "AAA"? The recursive implementation of a permutation generator I showed in Listing 2 treats the permutations of "AAABB" just as it does "ABCDE", obligingly printing out 120 different sequences. It doesn't know or care that there are a huge number of identical permutations in the output sequence.

It's easy to see why the brute force code in Listing 2 doesn't notice the duplicates. It never pays any attention to the contents of the string that it is permuting. It couldn't possibly notice that there were duplicates. It just merrily swaps characters without paying any attention to their value.

The STL algorithm, on the other hand, actually performs comparisons of the elements that it is interchanging, and uses their relative values to determine what interchanging will be done. In the example from the last section, you saw that an input of "24531" will generate a next permutation of "24135". What if the string had a pair of duplicates, as in "24431"? If the algorithm were ignorant of character values, the next permutation would undoubtedly be "24134".

In the early case, iterators i and ii were initially set to offsets of 1 and 2 within the string. But in this case, since the value pointed to by i must be less than the value pointed to by ii, the two iterators have to be decremented to positions 0 and 1. j would again point to position 3.

The subsequent swap operation yields "34421", and the reverse function produces a final result of "31244". Remember that the algorithm works by progressively bubbling the larger values of the string into position 0, you can see that this permutation has already jumped well ahead of the permutation of "24531" on its way to completion. Thus, the algorithm "knows" how to deal with duplicate values.

Conclusion

The addition of the STL to the C++ Standard Library gave us a nice grab bag of functions that automate many routine tasks. next_permuation() turned out to be just what I needed to solve a sixth grade math problem. It might be time for you to look through the declarations in the algorithm header file to see what else standards committee laid on our doorstep.

Windows Developers Journal January, 2002

Thanks to Shaun Wilde for correcting a couple of mistakes in the code.

XML is a great way to package data. With the approval of the XML Schema standard by the World Wide Web Consortium, developers now have a good tool for validating the form and content of their XML data. This article gives an example of how to perform this validation by using version 4 of Microsoft's MSXML parser.

Real World XML

Over the past few months I've been laboring away at a new book for Cisco Press which has the snappy title Developing Cisco IP Phone Productivity Services. In a nutshell, Cisco IP Phone Services are simply XML files served up to the phone over a standard HTTP link. XML objects can be used to display a wide variety of data on the phone, enabling programs running on a web server to use the phone as a flexible I/O device.

This means that anybody who can put together a web page can create nifty programs that interact with Cisco's IP phones. Examples that have been created so far include flight tracking apps, appointment books, and joke-a-day services. A well-loved productivity application is shown in Figure 1.

Figure 1 - An XML Object Rendered on the Phone

When the programmers at Cisco's Dallas engineering center were designing these services, they naturally chose to use XML to wrap up the various objects that were being sent back and forth. Yes, XML was in the air at the time, but there were also practical reasons. First among these was the availability of a compact XML parser that fit in the phone's firmware footprint. Second was the general feel-good vibe that comes with using an accepted standard as an integral part of our product. An additional boon was the support for XML in off-the-shelf tools, such as Microsoft's Internet Explorer or Altova's XML Spy.

Details

The menagerie of XML objects that the phone knows how to render and/or execute is fairly small. There are XML definitions for menus, graphics, directory listings, and so on. As an example, the simplest XML object supported by the phone is the CiscoIPPhoneText element. A typical example of this element might look like this:

Leading to a display on the phone like that shown in Figure 2.

Figure 2 - The CiscoIPPhoneText Object

Unfortunately, there isn't much to do when the phone gets badly-formed XML. In a typical example, a web designer could inadvertently introduce a typo in the text used in the above example, leading to this XML file on the server:

We've all had times when a fat finger on the keyboard accidentally introduced an extra character, as shown above. The extra period in the closing tag for the Text element is enough to turn this into a badly-formed XML file.

When the Cisco IP Phone attempts to render this XML object, its parser naturally rejects the input. With not many good options at this point, the phone simply displays a blank screen, as shown in Figure 3.

Figure 3 - The Cisco IP Phone Responds to Bad XML Input

This is a bad situation for the user of the IP Phone, but it's even worse for the designer of the web page. There are virtually no clues here to identify the problem. In theory we could load up the phone firmware with additional debugging code to identify problems such as this one, but we are up against constraints in memory space, developer time, and parser intelligence.

The good news is that existing tools such as Internet Explorer can help a lot with problems such as this. Looking at the same XML served up by the web server, Internet Explorer produces the display shown in Figure 4.

Figure 4 - Internet Explorer Finds the Problem

This kind of display is great - it takes you directly to the problem and identifies it clearly. Unfortunately, there is an entire class of XML problems that are not detected by Internet Explorer. Specifically, Internet Explorer (as of today) can't help diagnose an XML streamed that is well-formed but invalid.

Valid vs. Well-Formed

The XML standard defines two different standards for XML documents: well-formed and valid. A well-formed document is one that follows the basic rules of XML regarding basic syntax, such as nesting of tags, placement of attributes, and so on. Creating well-formed documents is fairly easy for developers, because you can simply add new elements and attributes as the need arises, confident that the parser reading the document will know what to do.

On the other hand, a valid document is held to a somewhat higher standard. A valid document has to adhere to a well-defined set of rules regarding the type, order, and placement of tags that appear in it. In the early days of XML, these rules were typically defined in what was called a Document Type Definition, or DTD.

For various reasons, the DTD format first used with XML proved to be unpopular, and a revised schema definition system called XML Schema was born. XML Schema does all that the original DTDs did, with some enhancements. In addition to knowing about the placement and types of tags, XML Schema allows you to specify information about the text that is embedded inside the markup. So, for example, XML Schema allows me to specify that a value between the <Price> and </Price> tags should be a floating-point value between 1.0 and 9.99.

The World Wide Web Consortium ratified XML Schema as an official recommendation in 2001, and we're now seeing the first set of tools that support it. I've been using XML Spy 4.0, both to create schema files and to test them against XML data. Writing a parser that validates against XML Schema is considerably more difficult than a plain-vanilla parser, so it may be some time before a wide selection of tools are available.

The Invalid XML Problem

The difference between a well-formed and a valid file is easy to demonstrate with the XML objects used for the Cisco IP Phone. Figure 5 shows Internet Explorer happily parsing a well-formed XML document and displaying it on a PC. A quick look at this document makes it look as if all is well, but oddly enough, the IP Phone fails to parse and display the data.

Figure 5 - Well-formed, but not valid

This is the kind of problem that can be pretty tough to spot. In this particular case, the programmer of the web page used a lower case 'p' in the opening and closing tags for element CiscoIPPhoneText. Since XML is case-sensitive, this error tripped up the phone, which determined that this is not a recognized document. The end user was faced with the unfriendly blank screen.

The Solution

The solution was to write a validation tool that compared the XML document to the schema defined for the Cisco IP Phones. For this particular project, I chose to use the MSXML parser from Microsoft. Version 4 is currently in an open beta, and will validate XML documents against DTDs and XML Schema. (A popular alternative is the Xerces parser, available for both C and Java.)

My simple version of this tool allows the user to input either a file name or a Universal Resource Locator (URL), then press the Validate button. The data is read from the source, parsed, and the result shown in a dialog box. The XML document from Figure 5 results in the error display in Figure 6.

Figure 6 - A Validating Parser Finds the Error

As you can see, MSXML does a pretty good job of identifying the offending location in the XML, saving the programmer from quite a bit of head scratching. The next section of this article will show you how to put together a program that accomplishes the same thing as my Validator.

Using MSXML

MSXML 4.0 is an ActiveX control, which means it can be used by quite a few different programming languages, ranging from JavaScript to Visual Basic to C++. I chose to write my validation program as a simple MFC dialog application using Visual C++ 6.0.

Visual C++ 6.0 has some language extensions that make working with COM objects a bit easier. (For example, using the #include statement to include a DLL directly, creating type libraries on the fly.) I didn't use these in this application, which means the code you see here should be easily portable to other C++ compilers. Perhaps this philosophy will help when porting the code used here to other languages as well.

Installing MSXML 4.0

In October of 2001 Microsoft released MSXML 4.0 as a free download. All versions of the XML parser are found at Microsoft's XML page, http://www.microsoft.com/XML. You will need to download and install the MSXML 4.0 SDK to build programs that use the new parser.

The install procedure for the parser by default copies header and library files to "C:\Program Files\MSXML 4.0", with the header files in the inc subdirectory and the library files in lib. The include file msxml2.h contains all the definitions you will need in your C++ code, and msxml2.lib contains links to the appropriate parser routines.

The Project Skeleton

My Validator project is a simple MFC dialog-based application. A complete listing is available by following the links below, so I won't give you every detail. After creating the project using Microsoft's Wizard, I made two minor modifications to accommodate the use of MSXML.

• First, I added "C:\Program Files\MSXML 4.0\inc" to my include path for both the debug and release versions of the program. This is where the header file msxml2.h is located, and it will be included in the main project.
• Second, I added the file C:\Program Files\MSXML 4.0\lib\msxml2.lib to the project. This library contains the CLSID and other definitions needed to access the functions in MSXML 4.0.

To this skeleton, I added some code to let the user either browse to select an XML file for input, or simply type in a URL. These pieces of infrastructure are shown in action in Figure 7.

Figure 7 - The I/O portions of Validator

The Real Work - Validation

There are two steps to perform in order to actually validate an XML document using MSXML 4.0. After the necessary plumbing needed to instantiate a copy of the MSXML COM Object of type XMLDOMDocument2, I have to create an XMLSchemaCache object, then add a copy of the Cisco IP Phone schema to it. A reference to that schema container is then passed to the DOM Document. The fragment of code that accomplishes this is shown in Listing 1.

As is usually the case, C++ code that uses COM objects is unattractive, and a bit opaque. But in this simple example you should be able to look past that and see what is actually happening. Most importantly, you can see that this program expects to find a copy of CiscoIPPhone.xsd in the current directory. This file is the file that actually contains the schema for all the Cisco IP Phone XML objects.

Once the DOM Document has a reference to the schema, it is ready to do the actual parsing. Microsoft's DOM Document object is flexible enough to load either a reference to a local file, or to a URL which will supply an HTTP stream. The fragment in Listing 2 does this by simply calling the load method of the object. It then pops up a dialog with one of two possible outcomes: the successfully parsed XML content, or a formatted error message.

Figure 6 showed an example of what you see when the parser reports a failure. Figure 8 shows what it does upon success, which is to just present a copy of the parsed data found in the xml property of the DOM object.

Figure 8 - Succesfully parsed XML Data

Wrap-up

MXSML 4.0 is a great step forward for users of Microsoft's parser. Adding validation to the tool provides a great leap forward for developers, and for end users who benefit from better checking of the data. There is one feature I wish Microsoft had implemented differently. As the control works now, when parsing fails, the document only keeps a copy of one line from the bad input. It would be nice to keep the entire document, or at least a little more context, which would make interpretation of the error a little bit easier.

But given the price tag and the capabilities the control brings to my programs, I have to give it a better than passing grade.

Source

Validator source files are here in Source.zip.

You will need to install version 4 of Microsoft's XML parser in order to build and execute the program.