Mark Nelson

C++11: Range-based for and auto

Mark Nelson — Sat, 07 Apr 2012 18:18:02 +0000

Two really handy features in C++11 are the range-based for statement and the auto type specifier. The former allows you iterate over collections using a much more compact form of expression, and the latter takes some of the headache out of the complex type declarations encountered in the standard library. Both of these features have been available in g++ since release 4.6, and are now present in Visual Studio 11, so you can start using them today. (auto typed variables are available in earlier versions of both compilers.) In this post I’ll give you a description of how these new features works, and show you a concrete example of the positive effects they can have on your programs.

The value of containers

It’s hard to overstate the value of the containers in the C++ standard library. With the addition of the hash-based containers in TR1, I rarely if ever find myself tempted to roll my own, or use a third party library. The flexibility and power of the library created by Alexander Stepanov does everything I need.

Despite the technical merit of the container classes, newcomers are often hesitant about completely embracing them. One of the main reasons has to be the conceptual drag imposed by the use of iterators as the primary means of accessing the objects they contain. It’s not that there is anything complicated about the concept, but the syntax can be more than just a little annoying. Let me illustrate it with an example.

Anagramania

The listing below is a C++ program that reads through the Scrabble dictionary and determines which set of letters generates the most anagrams. I’m using C++ circa TR1, in which I have access to the unordered associative containers, but I don’t take any shortcuts to try to simplify the syntax. (The fact that I can write this program in one screen of simple code is a nice testament to the quality of the container library.)

The logic for the program is simple. I use an unordered_multimap called counts to hold the count of all anagram families in the dictionary, with its key being the sorted value of the scrabble word. This means that all words that are anagrams of one another will have the same key. I use an unordered_multimap called words to hold the list of all words that are anagrams of that key. Each time I process a word, I increment a value in counts and I add a new value to words.

After the input processing is done, I can just iterate over counts from top to bottom, looking for the highest count. When I have gone through the entire map, I have the sorted key that generates the most anagrams. Using that key, I query an unordered_multimap for a range of results. It returns two iterators in a pair object, which I then use to iterate over the result set.

Even if you are familiar with the type system used by the containers and don’t make too many mistakes, just the magnitude of how much you have to type to get this to work is a bit of a downer. And the length of those type definitions doesn’t help make the concepts being used any clearer.

#include 
#include 
#include 
#include 
#include 
#include 

int main(int argc, char* argv[])
{
    std::ifstream data( "sowpods.txt" );
    std::unordered_map counts;
    std::unordered_multimap words;

    std::string s;
    while ( data >> s ) {
        std::string temp = s;
        std::sort(temp.begin(), temp.end() );
        counts[temp]++;
        words.insert( std::make_pair(temp,s) );
    }

    int max_count = -1;
    std::string max_string = "";
    for ( std::unordered_map::iterator ii = counts.begin();
          ii != counts.end();
          ii++ )
    {
        if ( ii->second > max_count ) {
            max_count = ii->second;
            max_string = ii->first;
        }
    }
    std::cout << "The maximum anagram family has " << max_count << " members:\n";
    std::pair< std::unordered_multimap::iterator,
	       std::unordered_multimap::iterator> range;
    range = words.equal_range( max_string );
    for ( std::unordered_multimap::iterator ii = range.first;
          ii != range.second;
          ii++ )
        std::cout << ii->second << " ";
    std::cout << std::endl;
    return 0;
}

Anagram finder circa TR1
Now let’s look at the two features that make major improvements to this program in C++11.

The auto Type Specification

The hard working committee members who hammered out the standard last year clearly listened to the millions of C++ programmers out there. While they were charting new waters for the language with things like move semantics and rvalue references, they were also making a lot of small changes that simply make the language a lot easier to work with. Maybe even a little more fun. The two things I find at the top of my list are the the use of auto type specifier and the for-range statement.

The auto keyword can be used in a number of different contexts, but in general it means that you can declare variables without having to enter a complete type. This solves some tricky problems for template programming, and it provides a convenience for awkward variable declarations. Most notably, it allows you to replace these two wordy lines of code:

    std::pair< std::unordered_multimap::iterator,
	       std::unordered_multimap::iterator> range;
    range = words.equal_range( max_string );

with this much simpler single line:

    auto range = words.equal_range( max_string );

In both cases, the type of range is the same – but by using the auto type specifier, we let the compiler replace all that typing with a bit of simple hand waving.

Bjarne Stroustrup has a good, concise explanation of auto on his C++11 FAQ, I recommend you spend the time to read it.

The Range-based for Statement

When working with standard library containers, one of the most common things we do is iterate over some or all of the container. This generally is done using a for or while loop with an iterator loop variable.

C++11 makes this type of iteration easier with new syntax injected into the for statement that has been around since 1969. The range-based for looks like this:

    for ( declaration : expression ) statement

In this new statement, expression can be an initializer list, an array, or an object that implements container semantics. This means that the object returns an iterator-like object from a begin() and end() methods, or via a call to begin() and end() functions in the current or std namespace.

The variable declaration is either a reference or value of the type of variable held in the container, array, or initializer list. The for loop is executed from the beginning of the container to the end, with statement executed once per value returned by the iterator.

Although this is a completely new language feature, I think most C++ programmers will be comfortable with it from the first time they are able – it makes those iterations over containers clean and concise.

Putting it to Use

Although it didn’t really cut down on my code size in a big way, I first made use of the range-based for in the loop that reads in the data from the scrabble dictionary. My new version of the loop is shown here:

for ( const std::string &s : std::istream_iterator( data ) )
{
    std::string temp = s;
    std::sort(temp.begin(), temp.end() );
    counts[temp]++;
    words.insert( std::make_pair(temp,s) );
}

The only big improvement here is that I was able to declare my string variable on first use, which is always my preference.

However, looking at this code, you might be wondering how it compiles. After all, the istream_iterator doesn’t have begin() or end() member functions.

That’s correct, and the reason it works is that I added a couple of convenience functions to my program that enable the use of this iterator type with the range-based for:

template
std::istream_iterator begin(std::istream_iterator &ii_stream)
{
    return ii_stream;
}

template
std::istream_iterator end(std::istream_iterator &ii_stream)
{
    return std::istream_iterator();
}

I made use of a similar set of template functions to enable the use of the new for statement in my final output statement. I now iterate over the discovered members of the anagram family with two easy-to-read lines:

for ( const auto &map_entry : words.equal_range( ii->first ) )
    std::cout << map_entry.second << " ";

Compare this to the TR1 code that does the same thing, and I think you will see the real value of both auto and range-based for.

Iterating over the values returned from a multimap is a common task, enabled it by these convenient template functions:

template
ITERATOR begin( std::pair &range )
{
    return range.first;
}

template
ITERATOR end( std::pair &range )
{
    return range.second;
}

When I first implemented the functions for my C++11 program, I was halfway expecting to find that this functionality had already been added to the standard library – they really make a big improvement for a small investment. But no, I couldn’t find them, so we will be using our own versions for the time being.

The Final Product

My much improved anagram finder is shown below. In addition to the use of range-based for and auto type declarations, I changed the way I find the maximum element in the container. Now that lambdas are part of the language, there is no excuse for not using the standard library algorithms, and this code gives an illustration of how that works as well.

#include 
#include 
#include 
#include 
#include 
#include 

template
ITERATOR begin( std::pair &range )
{
    return range.first;
}

template
ITERATOR end( std::pair &range )
{
    return range.second;
}

template
std::istream_iterator begin(std::istream_iterator &ii_stream)
{
    return ii_stream;
}

template
std::istream_iterator end(std::istream_iterator &ii_stream)
{
    return std::istream_iterator();
}

int main(int argc, char* argv[])
{
    std::ifstream data( "sowpods.txt" );
    std::unordered_map counts;
    std::unordered_multimap words;

    for ( const std::string &s : std::istream_iterator( data ) )
    {
        std::string temp = s;
        std::sort(temp.begin(), temp.end() );
        counts[temp]++;
        words.insert( std::make_pair(temp,s) );
    }
    auto ii = std::max_element( counts.begin(),
                                counts.end(),
                                [](const std::pair &v1,
                                   const std::pair &v2)
                                {
                                    return v1.second < v2.second;
                                }
                              );
    std::cout << "The maximum anagram family has " << ii->second << " members:\n";
    for ( const auto &map_entry : words.equal_range( ii->first ) )
        std::cout << map_entry.second << " ";
    std::cout << std::endl;
    return 0;
}

Anagram finder in C++11
If I move the four convenience functions into a utility header file, I think you’ll agree that the new version of the code implements my algorithm in a very clean and concise way. The new language improvements make a huge difference in readability and convenience.

Of course, these two features are just one small part of a huge new standard, but for right now, they are the ones I turn to the most. How about you? Let me know!

Visual Studio 11 and Modern C++

Mark Nelson — Tue, 13 Mar 2012 13:13:08 +0000

Despite some harsh words about Visual Studio 11, I’m finding that it makes my heart go pitter-pat every time I use it. Why? Because this early release is finally incorporating a decent set of long-awaited C++11 features. In this article I’ll show you how a little thing like a lambda can make a big difference in your coding style.

Microsoft and C++ – We Have History

Microsoft has a cyclic relationship with C++. In the early MFC days, the love was there big time – you had access to most of the system API using C++. However, around the turn of the millennium, Microsoft came under the Rasputin-like influence of Anders Hejlsberg and his beloved offspring, C#. Now it appears that maybe the pendulum is swinging back a bit, and C++ is no longer viewed as an afterthought. Great news.

Although Visual Studio 11 is a developer’s preview, Microsoft is saying that it is production ready – you can use this to create programs that are ready for release. In addition to touting a complete implementation of the C++11 standard library, an impressive list of language features have been turned on as well. (N.B. the path ahead is still long and arduous.)

Modern C++

Before even using Visual Studio C++ 11 to test a single line of code, I really appreciated reading Welcome Back to C++ (Modern C++), a manifesto that includes the following bullet points:

Modern C++ emphasizes:

Stack-based scope instead of heap or static global scope.

Auto type inference instead of explicit type names.

Smart pointers instead of raw pointers.

std::string and std::wstring types instead of raw char[] arrays.

Standard template library (STL) containers—for example, vector, list, and map—instead of raw arrays or custom containers.

STL algorithms instead of manually coded ones.

Exceptions, to report and handle error conditions.

Inline lambda functions instead of small functions implemented separately.

I feel that all of these changes result in safer code that is easier to read and maintain, without giving up the type-safety and efficiency that we love so much. Fully implementing these features either leans heavily on C++11 or requires it outright.

A Simple Example Using Naive C++98

It’s interesting to watch the evolution of code from C++ 98 to C++11 and see how it affects your code. You’ll see that the transformation can make it look like you are literally using a new programming language.

In this simple program, I’m taking a Scrabble rack of tiles and whipping through the Scrabble dictionary to find matches. Since it is a one-time call, I’m not storing the words, just doing a quick online comparison. In C++ 98, my code might have looked like this:

void find_matches( std::string rack, const std::string &filename )
{
    std::sort( rack.begin(), rack.end() );
    std::ifstream sowpods( filename.c_str() );
    std::string word;
    while ( sowpods >> word ) {
        std::string sorted = word;
        std::sort( sorted.begin(), sorted.end() );
        if ( sorted == rack )
            std::cout << word << " ";
    }
}

int main(int argc, char* argv[])
{
    find_matches( "etaionsr", "sowpods.txt" );
    return 0;
}

This works properly and I get what looks like correct output:

anoestri arsonite notaries notarise rosinate senorita

Classes Good, Templates Better

As people started to get more comfortable with templates and iterators, algorithms like this were commonly rewritten to take an range of iterators as input – much as the standard library algorithm functions do. This meant changing the function to a template function, but it did make it a lot more flexible. I could now call the function to operate on data from a file, just as before, but I can also now use any other container, or even an array as input:

template
void find_matches( std::string rack, ITERATOR ii, ITERATOR jj )
{
    std::sort( rack.begin(), rack.end() );
    for ( ; ii != jj ; ii++ ) {
        std::string sorted = *ii;
        std::sort( sorted.begin(), sorted.end() );
        if ( sorted == rack )
            std::cout << *ii << " ";
    }
}

int main(int argc, char* argv[])
{
    std::ifstream sowpods( "sowpods.txt" );
    find_matches( "etaionsr",
                  std::istream_iterator( sowpods ),
                  std::istream_iterator() );
    return 0;
}

More or less the same number of lines of code, but it is now generic.

Of course, just like with OOP, you need to take some care with template programming. Generic programming can’t be beat when it makes sense, but programmers have a particularly strong susceptibility to pro-innovation bias.

Using the Algorithms Library

Again, prodded by changing styles in the C++ world, my next step is to use a standard library algorithm to do the work. We’re told over and over that turning to the algorithms library allows you to use code that has been optimized to the n-th degree by the clever library teams.

In order to make this work, I have to call an algorithm with a predicate functor, seen below as class sorted_not_equal. Note also that I can’t use the logical function for this, which would be copy_if(). Why not? The committee forgot to put it in back in 1998, 2003, and 2005, a mistake that was fortunately remedied in C++11. So I have to use the inverse function, remove_copy_if(), and invert the logical sense of my functor:

class sorted_not_equal {
    std::string str;
public :
    sorted_not_equal( const std::string & test )
    {
        str = test;
        sort( str.begin(), str.end() );
    }
    bool operator()( std::string test )
    {
        sort( test.begin(), test.end() );
        return ( str != test );
    }
};

template
void find_matches( std::string rack,
                   INPUT_ITERATOR ii,
                   INPUT_ITERATOR jj,
                   OUTPUT_ITERATOR kk )
{
    std::sort( rack.begin(), rack.end() );
    std::remove_copy_if( ii, jj, kk, sorted_not_equal( rack ) );
}

int main(int argc, char* argv[])
{
    std::ifstream sowpods( "sowpods.txt" );
    find_matches( "etaionsr",
                  std::istream_iterator( sowpods ),
                  std::istream_iterator(),
                  std::ostream_iterator(std::cout,  "\n" ) );
    return 0;
}

Functors Not So Hot

So this new approach is supposed to soup up my code by taking advantage of the algorithms that come with the standard library. But if you look around at the code people have been writing for the past 10 years, you’ll find that this style is pretty common in textbooks and magazine articles, but no so much in the real world.

Why not? Well, it’s pretty obvious. The generic algorithms in the library need lots of predicate glue to make them useful, and the work to create these predicates is just a pain. My code is almost twice as long, and the functionality that took two lines of code earlier is now bloated into a complete class definition. It pollutes my namespace, takes up a lot of space, and has to be defined somewhere distant from where it is actually used. Not a win.

This is obviously a problem when you look at the history of Linux, C, and C++. An entire family of technologies and infrastructure was developed with the implicit goal of reducing the number of keystrokes programmers had to enter. (I’m kidding, but only somewhat.) Functors are a step in the wrong direction.

Lambda to the Rescue

So it is with much relief that C++11 delivers lambdas, which allow us to write short sweet predicates exactly where we need them, as shown in this C++11 version of the example:

template
void find_matches( std::string rack,
                   INPUT_ITERATOR ii,
                   INPUT_ITERATOR jj,
                   OUTPUT_ITERATOR kk )
{
    std::sort( rack.begin(), rack.end() );
    std::copy_if( ii, jj, kk,
                  [&rack](std::string str) ->bool
                  {
                      std::sort( str.begin(), str.end() );
                      return rack == str;
                  }
                );
}

int main(int argc, char* argv[])
{
    std::ifstream sowpods( "sowpods.txt" );
    find_matches( "etaionsr",
                  std::istream_iterator( sowpods ),
                  std::istream_iterator(),
                  std::ostream_iterator(std::cout,  "\n" ) );
	return 0;
}

Yes, I now have to get used to a new syntax for writing lambda functions – I think that was unavoidable. But my lambda function is short, it is quite easy to see exactly what it is doing, and it replaces a gangly and awkward functor class.

Best of all, I use the lambda exactly where I need it – as the predicate parameter to an algorithm used in the standard library. Locality rules.

Using Lambdas

Visual C++ 11 provides a great framework for experimenting with lambdas, as they are supporting the 1.1 definition that was ratified as part of the standard. If you want the gory details, I believe the working group’s proposal has essentially the same wording that went into the standard. For a detailed tutorial, Herb Sutter’s talk can’t be beat.

C++ – Where’s the Hate?

Mark Nelson — Tue, 28 Feb 2012 05:23:09 +0000

One thing I’ve become accustomed to over the years is that there are a lot of C++ haters. They have their reasons – some good, some bad – but they are never afraid of sharing their opinions. An article on Slashdot this week touting the release of the C++11 standard should have been a hotbed of language trash talk – instead, it was kind of a low key discussion of both the new language features and some retrospection about the language itself. Where have the haters gone?

The Epitome of C++ Hate

There really is no better example of C++ hate than the screed arising from the poison keyboard of Linus Torvalds back in 2007. One paragraph sums it up well:

C++ is a horrible language. It’s made more horrible by the fact that a lot of substandard programmers use it, to the point where it’s much much easier to generate total and utter crap with it. Quite frankly, even if the choice of C were to do *nothing* but keep the C++ programmers out, that in itself would be a huge reason to use C.

Linus asserts that C++ is actually sort of a honey-trap, pulling in the substandard programmers and keeping them out of decent C development.

Reading through the most coherent parts of his rant, it is my opinion that much of his ire is directed against a couple of real problems:

The early adopter’s fondness for Object Oriented Programming (OOP), which leads to objects everywhere – and particularly in places where they aren’t needed.
The more sophisticated user’s fondness for advanced library and language features found in places like boost.
My personal observations are obviously very subjective, but I feel like the excesses of OOP peaked back in the early part of the last decade. It’s been a while since I saw a program were a user did integer math using a type called Integer derived from something called Object.

These days I see OOP being heavily used for large libraries, and used internally in projects where it makes sense. Often “making sense” is as simple as using objects for RAII, which is really not OOP, just good resource management. Things like RAII are really just better C – providing some guarantees on resource usage that can’t be made when programming in straight C.

And best of all, the boost language features that Linus didn’t like in 2007 made their way into TR1, and finally into C++11. No longer will you need to worry about rebuilding boost for your compiler every time there is a new release of either product. From this point on, it should just work.

The Slashdot Crowd

So instead of seeing a bunch of comments in the vein of Linus’s, the recent Slashdot article contained virtually no blanket dismissals of C++ as something a sane programmer would use. The negative comments were much more constrained, maybe aimed at a pet peeve instead of the language as a whole:

printf() isn’t typesafe, but it’s a fuckton more readable than all that cout formatting stuff. Also, the fact that it’s not typesafe isn’t really an issue if you don’t suck – trivial unit testing will pretty much show any problems immediately. Besides, gcc/g++ is nice enough to warn you about egregious ones now.

I don’t think you’ll see a lot of people flaming C++, there just aren’t that many people that care one way or the other anymore.

I think some of the new features look nice but mainstream use has been shifting away from C++ for a while and I’m not sure I see these new features drawing many back…

Are the features useful? Yes, but they’re taking a complex language and slapping on yet more functionality. Some new C++ code syntax doesn’t even *look* like C++ anymore it’s so different. Not everyone is a C++ guru and the language is bad enough supporting so many different paths to the same implementation outcomes. This is just going to make staffing, testing, training, and code review that much worse trying to juggle yet another barrel-full of C++ “improvements”.

Why the Dropoff?

I think there are two reasons for the drop in C++ antagonism.

The first is the increasing diversity of languages in use today. Over the past 10 years, the most popular programming languages like Java, C, C++, and VB have all experienced drops in adoption. In 2002, the top four languages might have claimed almost 70% of the Tiobe Language Index. Today the top four represent less than 50%. Language selection has been democratized as the bottom tier of the top 10 gains representation in large markets where it makes sense.

In other words, it was a lot easier to focus your wrath on C++ when it represented 20% of the projects out there instead of the 8% of today.

The second is inevitable rationalization of the earlier adopters. C++ programmers in 1998 saw objects behind every tree. In 2003 they were building insanely complicated template metaprogramming frameworks.

All this crazy stuff settles down over the course of a few years as the C++ mainstream settles into paradigms for things that actually work sensibly. Templates for container classes? Sure. Templates for mathematical computation? Uh, no thanks.

Final Analysis

There is a lot of nice stuff in C++11, and nothing that is going to take years to figure out. A lot of it will lead to more readable code, and that’s always good.

The pace of change is pretty comfortable for me. These language features will be rolling out over the next few years, and as they drop into our favorite compilers, we will pick them up quickly.

I, for one, welcome our new ISO overlords.

Streambuf Iterators Are a Big Help

Mark Nelson — Mon, 06 Feb 2012 00:34:45 +0000

A few weeks back I was looking at the choice of whether to use iterators or streaming operations for I/O on my data compression code. I was bemoaning the fact that the C++ iterators that perform stream I/O use the insertion and extraction operators, making them unsuitable for binary data compression.

It looks like I spoke too fast.

The Problem

The canonical way to use iterators to go over an iostream is to use istream_iterator or its partner, ostream_iterator. In this short sample, I create a binary file with values 0 to 256, then try to read back the data using this iterator type:

#include 
#include 
#include 
using namespace std;

int main(int argc, char* argv[])
{
    ofstream temp_out("temp.dat", std::ios_base::binary );
    for ( int i = 0 ; i < 256 ; i++ )
        temp_out.put( (char) i );
    temp_out.close();
    ifstream temp_in("temp.dat", std::ios_base::binary );
    char last = -1;
    int count = 0;
    istream_iterator ii( temp_in );
    while ( ii != std::istream_iterator() ) {
        char c = *ii++;
        count++;
        if ( c != char(last+1) )
            cout << "Error on character number: " << (int) last << endl;
        last = c;
    }
    cout << "Count: " << count << endl;
    return 0;
}

Running this program gives the following output:

Error on character number: 8
Error on character number: 31
Count: 250

Due to the fact that the extraction operator uses whitespace as a delimiter, we get errors trying to read tab, line feed, vertical tab, form feed, carriage return, and space - which means we simply don't see six characters in the input file. This is not too bad when parsing ascii text, but when trying to do compression on binary data, it just won't work.

I figured this was the end of it until reader Fred Jardon chimed in with:

Isn’t istreambuf_iterator the iterator you’re looking for ?

http://cplusplus.com/reference/std/iterator/istreambuf_iterator/

It directly reads from the inner streambuf without using the extraction operator.

The opposite iterator exists: ostreambuf_iterator

Well, Fred is quite right, and it only serves to show my lack of depth when it comes to iostreams. The two classes, istreambuf_iterator and ostreambuf_iterator read directly from the underlying buffer, and don't use the extraction operator, which means they don't have the whitespace issues seen above. Changing just two lines fixes the problem:

    istreambuf_iterator jj(temp_in.rdbuf());
    while ( ii != std::istreambuf_iterator() ) {

Running that I get the pristine output I long for:

Count: 256

I'd like to tell you the ramifications of diving down into the object and working on the buffer, but I'm afraid this implementation details still elude me. There's a lot more going on in iostreams than in good old , and half of what I learned about it was wiped out in the standardization process. So be it.

In any case, I think the proper use of istreambuf_iterator tips the scale in favor of iterators for me, which reverses my previous thinking.

And many thanks to Fred for straightening out the mess.

I’m In the Money

Mark Nelson — Wed, 01 Feb 2012 16:36:28 +0000

It looks like all my long years of studying data compression might be ready to pay off:

Hello Good Day,

This is Troop Emonds With regards to your Company i am sending this email Regards to order some( Compression Machine )I will like to know the type and sizes you have in stock and get me the sales price of one so that i will tell you the quantity i will be ordering, and if you accept credit card as a form of payment..

Hope to read from you soon about my order request……
With Kind Regards.
Troop

I just need to put together some compression machines, and then I’m set.

Mark’s Travel Guide to New Zealand

Mark Nelson — Sat, 28 Jan 2012 22:37:16 +0000

I recently spent a little over two weeks touring New Zealand. It was a self-driving trip, which meant we got to cover a lot of ground, although certainly the coverage was very shallow.

Before this trip, I had not set foot on foreign soil more than one mile from the US border, so the experience of going to a foreign country was in itself new. This means I am compelled to share it with you.

Overall New Zealand made me feel very welcome. I would like to move to New Zealand. Barring that, visiting New Zealand as a tourist was a great experience.

I could write a detailed photoblog of our eighteen day journey, but this would be a lot like going to your Uncle’s house and watching his 90-minute DVD compilation of his trip to Norway – a bit tedious.

One thing I noticed as a tourist is that it is kind of hard to notice things that are there, but very easy to notice the things that are missing. So my detailed summary of the trip will give you the list of things that I noticed were not in New Zealand – at least not on my self-driven tour. (I’d give a link to the fine touring company if they would work out some sort of affiliate program, then you could do it yourself.)

Things that don’t appear to exist in New Zealand:

Stop signs
Iced tea
Pennies
Nickels
Dollar bills
Insane airport security (domestic only)
Pickup trucks
The purported monoculture of sheep
Moas
Cosmopolitans, or more generally, grown-up cocktails
Hot weather
Air-Conditioned hotel rooms
Free Wireless
Reasonably priced Wireless
Horrifying public restrooms
4 lane highways
Any sense of realism about building a country in a volcano/earthquake/tsunami free-fire zone

Pictures and Movies

A bunch of photos here.

Some very short videos here.

A Visit With Tim Bell

Mark Nelson — Sun, 22 Jan 2012 02:22:50 +0000

In my early years of learning about data compression, the book Text Compression by Timothy Bell, John Cleary, and Ian Witten was my resource of first resort. I was in Christchurch, New Zealand, recently and had a chance to meet Tim for the first time in person. Tim teaches at the University of Canterbury in Christchurch, and is Deputy Head of the Computer Science and Software Engineering department. I got a chance to ask him about his work in data compression as well as one of his new areas of interest, Computer Science education.

MN: Tim, it seems like there has been a lot of interest in data compression in the Antipodes. Names that come to mind include you, John Cleary, and Peter Fenwick in New Zealand, and Ross Williams in Australia. Is this just coincidence, or is compression in the air down there?

TB: I’ve sometimes wonder about this myself… during the early days of computing and especially personal computers, it took some time for the latest technology to reach us “down under”, so perhaps we were motivated to get more out of what we had rather than wait some months for a larger disk or new memory to arrive from overseas. When the Internet arrived we started with a very small pipe, so a good compression algorithm could do the equivalent to laying a second cable from NZ to the US – who can resist getting something for free?

MN: Since you wrote Text Compression back in the early 90s, I’d say the biggest development in lossless compression has been the Burrows-Wheeler transform. Is lossless text compression basically done? Are we left with just incremental improvements as processor resources increase?

TB: That seems to be the case; the only big improvements we’ve seen have turned out to be frauds — we even had one in NZ recently, where a Nelson man raised NZ$5.3 million for an impressive sounding method; he was convicted of fraud last year. The main indicator we have that we’re running out of steam (apart from a lack of new discoveries) is Shannon’s experiments on predicting text which gave a bound in the order of 1 bit per character for English text, and current methods are approaching this. Of course, there’s plenty of room for dealing with new kinds of data (for example, bioinformatics deals with massive amounts of data that we’re still trying to understand) and for finding better data structures and algorithms for performing the compression and decompression. Lossy compression is a whole different story…

A Change In Focus

MN: It looks like you are now dedicating a large amount of your time to establishing computer science as part of the basic curriculum in high school education, for students in the 15-18 age range. In many ways, this is as much a bureaucratic problem as an academic one. What motivated you to take it on?

TB: It’s been a problem that we’ve complained about for decades, and it’s been getting worse and worse as computing in schools has focussed increasingly on using computers and not preparing students to be developers. A lot of this can be attributed to bureaucracy – it’s hard to explain to government officials that putting word processors in every classroom isn’t the same as building a computationally literate society. As a result of some strategic lobbying done by others, a small window of opportunity opened for me to be on a group to advise our Ministry of Education, just over 3 years ago. The group managed to convince the officials that something useful could be done, and then we had to work very quickly to come up with a concrete proposal before the enthusiasm died down. This has happened rapidly; the advisory group first met in November 2008, and Computer Science started being taught in schools in February 2011.

MN: What have you been able to accomplish in New Zealand so far?

TB: Computer science (including programming, but also topics the involve understanding the importance of things like algorithms, HCI, programming languages and even compression) is currently available as part of computing courses for two of the three final years of our main high school graduation qualification, with all three years being covered from 2013. After that we would expect some of the introductory material to start filtering down to earlier classes, and for wider offerings as teachers become more confident in the subject. One of the biggest challenges has been preparing teachers, few of whom have significant experience in Computer Science. Many have embraced it enthusiastically, and the universities and others have done a lot of work to help them get up to speed. It’s been a wild ride doing it so quickly, but there have been some very pleasing outcomes.

MN: And how do things look in the rest of the world? Are there any obvious winners and losers at this point? Do you have any concise advice for the world?

TB: Computing in schools is a hot topic around the world; the UK have just announced a strong drive to introduce this sort of material to schools, and the US has people working hard to make it available to students. Israel and Korea have had computer science in schools for some time. We’re learning a lot about what is worth teaching, and what the best pedagogy is for the general classroom (most of our experience is for specialist students who have chosen the subject!) The New Zealand path of getting something going quickly with grass-roots support seems to be more effective than waiting for a top-down approach which could take years to develop and prepare teachers for, although it does make for a bumpy ride as problems are ironed out as we go along!

MN: This might be straying out of your area a bit, but do you see CS in a K-12 education setting having an effect on the representation of women in the STEM fields?

TB: Attitudes that affect representation definitely start at school, and to me the biggest goal of teaching CS in high school is not so much to prepare students for further study, but to enable them to find out what the subject is! School students rarely know what CS is, and even worse, it’s common for them to assume that it must be advanced word processing or some other dull area, and hence they avoid it. It’s particularly important for female students to have the opportunity to find out if it’s something that they might be good at, as the stereotypes associated with computing can make them assume that they shouldn’t consider it as a career.

MN: One final question, Tim. The whole world has seen the devastating damage Christchurch has suffered from the earthquakes in the last year. How has the University of Canterbury held up? Have you managed to maintain continuity in your academic calendar?

TB: It’s been quite a year! Thankfully our university has escaped the brunt of the earthquakes (most of the damage is some distance from the university), and we’ve managed to keep a full programme going despite being closed for three weeks for safety checks. Many students joined the “student volunteer army”, who helped with the cleanup in the damaged parts of town, and that was probably one of the most valuable experiences of their career! It hasn’t been without disruption as buildings need to be checked carefully, and some are still under repair, but with a bit of resourcefulness we managed to keep going (for a while I even delivered my classes in a restaurant while lecture theatres were being inspected) The city is now going through a massive program of redevelopment with some pretty creative ideas, and it’s an exciting time to be part of these changes.

Thanks to Dr. Bell for taking the time to share all this with us. My visit to his amazing homeland was a real treat, and the short time I got to spend with Tim in Christchurch was worth the trip all in itself.

Streams or Iterators?

Mark Nelson — Sat, 24 Dec 2011 18:21:11 +0000

When I updated my LZW reference code to use the latest C++ features, I abstracted my input and output functions using templates. Data was read and written using the iostreams paradigm, which requires simple classes that implement just a few functions. Would I have been better off using the iterator paradigm instead? The C++ algorithms library favors that method of processing data, and it can be both elegant and powerful. Which of the two paradigms is the right one for data compression?

The Conflict

General purpose data compression routines tend to be used on binary streams of data, either from files or in-memory objects. So what is the best general paradigm for input and output when compressing data?

You might analyze this problem by imagining that you need to write a binary copy routine.

template
void bcopy( INPUT_ITERATOR input, INPUT_ITERATOR eof, OUTPUT_ITERATOR output )
{
    while ( input != eof )
        *output++ = *input++;
}

This routine is particularly nice when you are performing a simple copy using pointers to memory – the generated code should be really efficient.

However, the iterator paradigm doesn’t work quite as well when you want to perform a binary copy of data in a file. I can make use of iterators that almost do the job:

 std::ifstream in( "input.txt", std::ios_base::binary );
 std::ofstream out("output.txt", std::ios_base::binary );
 bcopy( std::istream_iterator(in),
        std::istream_iterator(),
	std::ostream_iterator(out) );

But the bad news is that both istream_iterator and ostream_iterator use the insertion and extraction operators, which are really meant for whitespace-delimited textual data, not binary data. The copy routine shown here will not make a binary byte-for-byte copy of the input file.

So when using files, the stream approach seems to be the way to go:

template
void bcopy( INPUT_STREAM in, OUTPUT_STREAM out )
{
    char c;
    while ( in.get(c) )
        out.put(c);
}

If my files have been opened using the iostream classes, you can use this binary copy function without having to write any glue code – they already support the get and put methods, so this works right out of the box.

My Choice

If I’ve made up my mind that my data compression routine is going to use one of these two paradigms, it means I am going to have to write some glue code. If I choose the iterator-based approach, I need the equivalent of istream_iterator and ostream_iterator for binary files – and these aren’t in the standard library. If I choose the stream-based approach, I need efficient put() and get() members for blocks of memory. In some cases basic_stringstream might do the job, but not in all cases.

After dithering around with various solutions, I tentatively opted for the stream paradigm. I found the implementation for various sources of data to be fairly simple, and the interface is easy to understand. I don’t know if it’s the perfect choice, and I’ll keep experimenting, but for now it works for me. My abstraction of the LZW code still needs a lot of work, so it’s always possible I could rethink this at a later date.

I’d like to hear your thoughts – is there an obvious right answer to this question?

Automating Putty

Mark Nelson — Sat, 10 Dec 2011 12:11:15 +0000

Windows users who need a command line connection to another system via telnet or SSH are big fans of PuTTY. It's free, it has every feature you need, and it's reliable.

One thing many people would like to do is use PuTTY as a component in their program. Apparently this comes up so often enough that there is a FAQ entry dedicated to the topic. Alas, PuTTY does not have any sort of automation interface, so this goal has always been out of reach.

In this article I will show you how to work around this minor shortcoming. Creating a version of PuTTY that can be driven from a Windows program turns out to be an easy task. I'll demonstrate this with a small C++ program that shows exactly how to get this versatile program to do your bidding. My solution works for C++, but the changes I make should work well with any Windows software that can properly process a few messages.

Putting Together the Project

I'm using Visual Studio 2010 to build both my program and the modified version of Putty. I created the basic outline as follows:

Use the File|New|Project menu item to bring up the list of available project wizards.
Select MFC project, and enter a project name (I used the uninspired name PuttyDriver.)
I don't want the default MFC settings, so in the MFC App Wizard, select the Next button.
On the Application Type page of the wizard, change the Application Type to Dialog Based.
The project is ready to go at this point, you can click the Finish button and then build your initial project.

My driver program is only going to do one thing: direct putty to connect to the host of my choice, then log in using canned credentials. The resulting UI is shown below, and I am going to leave the very minor details of creating it up to the reader.

The driver program - a simple dialog-based MFC app

Adding Putty to the Project

The next step in this process is to add the Putty components to the project. I downloaded version 0.61 of the PuTTY source from the download page and extracted it to a separate folder. I then used Visual Studio's File|Add|Existing Project to add the compatible project file, Putty.dsp, found in /Windows/MSVC/Putty. Visual Studio has to convert this project to a version 10 project file, but it should do so with no problems.

I then right-clicked on the Putty project in Solution Explorer and renamed it to AutoPutty. Since this version of PuTTY will have some slightly different behavior, I don't want to confuse the executable I am creating with the real thing.

From Project|Project Dependencies, I set the PuttyDriver project to depend on AutoPutty - this insures that both projects get built when I build the entire solution.

My final change to the project is to modify the output directory for both Debug and Release versions of AutoPutty. I set the project to build the executable in the root directory of my PuttyDriver project - this will make it easy to find the executable when I need to launch it. I had to make this change in two places: Properties|Configuration Properties|General|Output Directory and Properties|Linker|Outuput File.

When you finally build the project, you'll find that current version of Microsoft's C++ compiler complain quite a bit about the use of functions like strcpy - Microsoft would like you to use safer replacement functions. You may choose to turn those errors off by defining _CRT_SECURE_NO_WARNINGS in the project file. While you are there, you should define SECURITY_WIN32 as well - it is required by Windows header sspi.h.

After a successful build you should find a copy of AutoPutty.exe in the root directory of your project, and it should run on your system and behave just like PuTTY.

Launching AutoPutty

If I'm going to have a PuTTY component in my PuttyDriver program, one of the first things I need is to be able to start and stop AutoPutty. So my first step in this project is to create the code that launches the program from PuttyDriver. The code below is inserted into the handler for the Start button:

UpdateData( true );
char path[MAX_PATH];
GetCurrentDirectory(MAX_PATH, path);
if ( path[ strlen(path) - 1 ] != '\\' )
    strcat_s( path, MAX_PATH, "\\" );
strcat_s( path, MAX_PATH, "AutoPutty.exe -ssh " );
strcat_s( path, MAX_PATH, m_HostName.GetBuffer() );
PROCESS_INFORMATION pi;
ZeroMemory(&pi, sizeof(pi) );
STARTUPINFO si;
ZeroMemory(&si, sizeof(si) );
si.cb = sizeof(si);
if ( CreateProcess( NULL, path, NULL, NULL, NULL, NULL, NULL, NULL, &si, &pi ) )
{
    Sleep( 1000 );
    BringWindowToTop();
}

This code assumes that AutoPutty.exe is in the current directory, and launches it with a command line telling it to connect to the host named in the dialog using ssh. Assuming that you have the project set up properly, pushing the start button should now start an independent copy of AutoPutty, which will behave identically to classic PuTTY.

Taking Ownership of AutoPutty

At this point I can successfully launch AutoPutty, but I can't really start calling this an integrated part of my main program, PuttyDriver. All I have done is set up a launcher for a separate executable.

The next step in the integration process is to establish PuttyDriver as the owner of AutoPutty's main window. Most Windows programmers are familiar with the traditional parent/child relationship between windows. That relationship is well understood, but I can't use it here - it doesn't work for two top level windows.

Setting PuttyDriver to be the owner (as opposed to the parent) of AutoPutty has the following effects, as explained here by Microsoft:

The owned window will always be above its owner in the z-order.
The system automatically destroys the owned window when the owner is destroyed.
The owned window is hidden when the owner is minimized.

The most straightforward way to set ownership of the window is to pass the owner's handle in the call to CreateWindow(), which means I will now make my first modifications to the PuTTY source code.

There are a number of ways to pass the owner handle to AutoPutty for use in the call to CreateWindow(), with the most obvious being to pass it on the command line. In the interest of minimizing changes to the existing PuTTY code base, I elected to pass it by creating an environment variable that holds the owner window handle. Since a child process inherits the parent's environment, this is a no-fuss way to get the data to AutoPutty.

I added the following code to the end of InitDialog() in PuttyDriver:

CString hwnd_text;
hwnd_text.Format( "%d", m_hWnd );
SetEnvironmentVariable("PUTTY_OWNER", hwnd_text );

This sets the environment variable for AutoPutty to find when it gets launched.

Now I come to the point where I am actually making changes to the PuTTY code. Fortunately, all of the changes needed for this program are confined to two files: terminal.c and windows/window.c. My first change is to window.c. This file contains the WndProc for the PuTTY window, and thus most of the rendering and control code for the GUI.

In order to establish the Owner/Owned relationship, I need to modify the code that calls CreateWindow(). I hoisted the function call into a block, added code to get the owner window handle, and inserted the handle into the call to CreateWindow():

{
    HWND owner_hwnd = 0;
    char buffer[ 132 ];
    if ( GetEnvironmentVariable( "PUTTY_OWNER", buffer, 132 ) )
        sscanf( buffer, "%d", &owner_hwnd );
    if ( owner_hwnd == 0 )
        MessageBox( NULL,
                    "AutoPutty did not find the handle for the "
                    "owner window, this is not going to work",
                    "Fail",
                    MB_OK );
    hwnd = CreateWindowEx(exwinmode, appname, appname,
                          winmode, CW_USEDEFAULT, CW_USEDEFAULT,
                          guess_width, guess_height,
                          owner_hwnd, NULL, inst, NULL);
}

At this point I've only modified one small block of code in the PuTTY source, but I'm well on my way to having it behave more like a component of PuttyDriver and less like an independent program. The ownership status means that the two programs only appear once on the taskbar, and will only appear once when you are pressing ALT-TAB to select a new active process. And they only produce a single entry in the Applications Tab of Task Manager.

The Communications Link

In order to achieve the automation that I am seeking, I also need to have two way communications between AutoPutty and the driver program. Since this is Windows, a natural choice for communications is to use native Windows messages. In order to do this, both programs need the Window handle of their opposite number.

I've already solved half of that problem through the ownership relationship established when I created the main window for AutoPutty. Now that it has set PuttyDriver as its owner window, I can get this window handle any place in the program through a simple function call:

HWND parent = GetWindow(hwnd, GW_OWNER);

But the reverse is not true - PuttyDriver does not know have a copy of the window handle for AutoPutty.

To remedy this situation, I added code to window.c that notifies its owner when it s created, and when it is destroyed. First I add this statement immediately after the call to CreateWindow():

if ( owner_hwnd )
   PostMessage( owner_hwnd, WM_APP, 0, (LPARAM) hwnd );

This tells PuttyDriver that the window is created, and gives it the handle to use for communications.

I also need to know when the window is closed, and I have to add that code two places in window.c - because Putty can be shut down two different ways.

Normally AutoPutty will shut down in response to a windows message. When this happens, I can count on a WM_CLOSE message being sent to the Windows Procedure. I add this code the existing handler for WM_CLOSE:

if (!cfg.warn_on_close || session_closed ||
    MessageBox(hwnd,
               "Are you sure you want to close this session?",
               str, MB_ICONWARNING | MB_OKCANCEL | MB_DEFBUTTON1)
    == IDOK) {
    HWND parent = GetWindow(hwnd, GW_OWNER);
    if ( parent )
        SendMessage( parent, WM_APP, 0, 0 );
    DestroyWindow(hwnd);
}

This lets PuttyDriver know that the window has been destroyed.

The original PuTTY code has an alternative method of shutdown. When it receives one of several possible network events, such as a telnet connection being broken, it calls PostQuitMessage(). When a program shuts down this way, it doesn't issue messages to destroys its windows - it relies on the O/S to destroy the windows when the process exists. As a result, I have to make a change in WinMain(), the main window procedure for PuTTY. This procedure extracts the messages sent to it using PeekMessage, and I add some code to handle the processing when a WM_QUIT message is sent:

if (msg.message == WM_QUIT) {
    HWND parent = GetWindow(hwnd, GW_OWNER);
    if ( parent )
        SendMessage( parent, WM_APP, 0, 0 );
    goto finished;	       /* two-level break */
}

Handling the AutoPutty Lifecycle Events

To keep track of the state of AutoPutty, I have to add a handler for WM_APP to PuttyDriver. It does two things when handling the incoming WM_APP event.

First, then handler stores the handle of the AutoPutty window - or sets the value to 0 when the window has been destroyed.

Second, it either enables or disables the button used to start up AutoPutty. Since this program can only manage one window at a time, I don't want to allow any inadvertent button pushes:

afx_msg LRESULT CPuttyDriverDlg::OnWmApp(WPARAM wParam, LPARAM lParam)
{
    m_PuttyWindow = (HWND) lParam;
    m_StartButton.EnableWindow( !m_PuttyWindow );
    return 0;
}

One final piece of bookkeeping is to make sure that the AutoPutty window is shut down when PuttyDriver shuts down. (The Windows documentation claims this happens automatically to owned windows, but it doesn't seem to be the case.)

void CPuttyDriverDlg::OnDestroy()
{
    CDialogEx::OnDestroy();
    if ( m_PuttyWindow )
        ::SendMessage( m_PuttyWindow, WM_CLOSE, 0, 0 );
}

Monitoring Input Traffic

Now that I have control over the lifetime of my AutoPutty window, it's time to take the next step in automation. My driver program needs to watch all the data coming in from the remote end so that it can take action on various types of input.

Depending on how you set up your connection, PuTTY can receive input data from a serial port, a Telnet connection, or an SSH connection. Fortunately the Windows version of PuTTY uses a standard handle-based interface to all three types of connections. The routine term_data() in terminal.c is called as data arrives, regardless of the source.

Since we are using the Windows API to communicate between processes, it makes sense to use the WM_COPYDATA message to send data to the parent program as it arrives. WM_COPYDATA is a good choice, as it takes care of marshalling the data between the two processes, which can add some complication to other solutions. The modified routine is shown below:

int term_data(Terminal *term, int is_stderr, const char *data, int len)
{
    HWND parent = GetWindow(hwnd, GW_OWNER);
    if ( parent ) {
        COPYDATASTRUCT cd;
        cd.dwData = (ULONG_PTR) 0xDEADBEEF;
        cd.cbData = len;
        cd.lpData = (PVOID) data;
        SendMessage( parent, WM_COPYDATA, (WPARAM) hwnd, (LPARAM) &cd );
    }

Receiving the Data

To receive this messages in PuttyDriver, I simply create a handler for WM_COPYDATA and start grabbing the data as it arrives. One important thing to note is that because AutoPutty has to use SendMessage() to send the data to its parent, it has to wait for PuttyDriver to finish processing the data until it can continue. This dictates a certain style of behavior on my part.

There are quite a few ways to skin this cat, and I'm keeping it very simple here. I'm using a deque container to hold the last 64 characters I've received. After each WM_COPYDATA message I received, I check to see if the current output snapshot ends in one of my trigger messages. If it does, I post the message number to myself for later processing, then return so that AutoPutty can continue its work.

The code I'm using here is doing something fairly simple: automating the login process by using the credentials that I've entered into the dialog box. That means the two strings I'm looking for are the login and password prompts. The resulting code is shown here:

BOOL CPuttyDriverDlg::OnCopyData(CWnd* pWnd, COPYDATASTRUCT* pCopyDataStruct)
{
    char *p = (char *) pCopyDataStruct->lpData;
    int len = pCopyDataStruct->cbData;
    if ( len >= 64 ) {
        p += len - 64;
        len = 64;
        m_Snapshot.clear();
    }
    while ( len-- )
        m_Snapshot.push_front(*p++);
    m_Snapshot.resize(64);
    static const char *needles[2] = { "login as: ", "password: " };
    for ( int i = 0 ; i < 2 ; i++ ) {
        int len = strlen( needles[i] );
        int j;
        for ( j = 0 ; j < len ; j++ ) {
            if ( needles[i][j] != m_Snapshot[len-1-j] )
                break;
        }
        if ( j == len )
            PostMessage( WM_APP+1, i, 0 );
    }
    return TRUE;
}

There is plenty of room for improvement in this routine, much of it depending on what type of automation you are going to be using in your program. Some obvious items would include the ability to add and remove triggers as the program progresses, and regular expression matching for triggers.

Driving PuTTY

This login program is now complete save for one detail: I need a way to send my responses back to AutoPutty.

The first part of this is pretty obvious - I just need to read the data from the dialog box and post it to AutoPutty with my useful WM_COPYDATA command. This happens in my WM_APP+1 handler:

afx_msg LRESULT CPuttyDriverDlg::OnWmAppPlusOne(WPARAM wParam, LPARAM lParam)
{
    UpdateData(TRUE);
    CString msg;
    switch ( wParam ) {
    case 0 :
        msg = this->m_UserId + '\r'; break;
    case 1:
        msg = this->m_Password + '\r'; break;
    }
    if ( this->m_PuttyWindow ) {
        COPYDATASTRUCT cd;
        cd.dwData = (ULONG_PTR) 0xF00DFACE;
        cd.cbData = msg.GetLength();
        cd.lpData = (PVOID) (const char *) msg;
        ::SendMessage( this->m_PuttyWindow,
                       WM_COPYDATA,
                       (WPARAM) this->m_hWnd,
                       (LPARAM) &cd );
    }
    return 0;
}

Sending this data to AutoPutty is fine, but right now the program doesn't do anything with that message. The final piece of work is to add a WM_COPYDATA handler to window.c.

Simply grabbing the data is easy enough - the data structure that accompanies the message contains a pointer to the data and a value indicating its length. However, I have two problems I have to solve before the data is actually sent out to the to whatever device AutoPutty is connected to.

First, I have to take into account the fact that PuTTY was written to use wide characters. My driver program was built using MultiByte characters, so we have a mismatch. This means I have to do a conversion of the data from one domain to the other. This is a two step process - I call MultiByteToWideChar() once to determine how much space I need, then I allocate a buffer and call it again.

The second thing I need to do is determine what to do with the data once I've converted it. PuTTY takes all terminal input and eventually passes through a function called luni_send(). Calling this function directly from the Windows procedure seems to work just fine.

The WM_COPYDATA handler I created looks like this:

case WM_COPYDATA :
{
    COPYDATASTRUCT *cd = (COPYDATASTRUCT *) lParam;
    int wsize = MultiByteToWideChar( CP_ACP,
                                     MB_PRECOMPOSED,
                                     (LPCSTR) cd->lpData,
                                     cd->dwData,
                                     NULL,
                                     0 );
    wchar_t *buf = (wchar_t *) calloc( wsize+1, sizeof(wchar_t) );
    MultiByteToWideChar( CP_ACP,
                         MB_PRECOMPOSED,
                         (LPCSTR) cd->lpData,
                         cd->dwData,
                         buf,
                         wsize + 1 );
    if (term->ldisc)
        luni_send(term->ldisc, buf, wsize, 0);
    free( buf );
}

At this point I have a working program - it connects to my designated host, and sends the username and password of my choice to the host, connecting me to the system.

I should add a note of caution here. Automating logins is a tempting time saver, but in general this is a really bad idea. Any time you hard code credentials into a program, you open the door to all sorts of new attacks on your system.

In my demo program, the user has to enter a name and password, so nothing is hardcoded, but even this adds security holes to a system. I encourage you to think of this as a demonstration only.

Demo of the program in action
For a better view, go to full screen and select 720p

Source Code

I've included the complete source code for PuttyDriver, the MFC project that controls AutoPutty. It was built with Visual Studio 2010, so you may have a little work to do if you backport it to earlier versions. My use of language features and classes should be compatible with much earlier versions - this is all very simple code.

Because PuTTY is always changing, I am not redistributing a snapshot of the version I used. Instead, I'm including before and after copies of the two source files I modified: window.c and terminal.c. If you build with Putty 0.61, you should be able to drop these two files right on top of the files included with the distribution and be on your way. With later versions of PuTTY you will have to perform an intelligent merge of the changes, which I hope will be a fairly effortless process.

Downloads

PuttyDriver.zip. The PuttyDriver source and project. You will need to add the PuTTY project to this solution as described in the article.
putty.zip. This contains the two PuTTY source files modified for this project. Both the original 0.61 source and my modified source are supplied. Executables are supplied as well, which may or may not work on your system.

Sendmail on Linux – the Easy Way

Mark Nelson — Fri, 09 Dec 2011 16:11:05 +0000

This summer I'm teaching a graduate class, Principles of UNIX, which is more or less a crash course in the Mother of All Operating Systems. One of our early topics is email on UNIX, in which I try to impart to the class just how transformative email was back in the day. For early Internet users (mostly UNIX users), this was an incredibly big deal.

Unfortunately, setting up email on a Linux or UNIX system is not quite as automatic as it once was. In our class we use mailx and sendmail as tools to send files from background processes or cron jobs - but mailx will typically not work out of the box. In this post I'll discuss how to get it working on an Ubuntu 11 system.

Things Have changed

Back in the day if you wanted to send mail, you simply found a handy dedicated server that was accepting incoming SMTP connections. There were thousands, and they were undiscriminating, so this was no big deal.

The invention of spam ruined that.

Now any SMTP server you find is going to require you to go through a bit of a dance in order to authenticate and prove you are not a spammer. I started off this post with the intention of showing you how to use your gmail account to access Google's SMTP servers. The process was fairly arduous, as it involved creating a certificate authority, your own certificates, and then setting up the mail server to use this authentication.

While working on this, my son Joey recommended that I just set up a free account on one of several email gateway providers, such as SendGrid or MailJet. Both services will let you access their servers and send up to 200 emails a day for free.

I took him up on it and found the process to be much simpler than using gmail, so I'll pass along the setup procedure here.

Getting an Account

Obviously, SendGrid is in business to get you to purchase a commercial account so you can send thousands of emails a day from your web site. Accordingly, the don't go out of their way to advertise their free plan. If you go to their pricing page, you will find a little tiny link to the free plan hidden at the bottom.

Setting up an account is easy, but SendGrid insists that you have a web site. For automatic verification they will need to find your email address on the site. I opted for an alternate provisioning plan in which I created a page on my site with the phrase "Sendgrid".

Once you have an account, you have free access to the SendGrid SMTP servers for up to 200 outbound messages a day. So you are ready to configure your UNIX system to take advantage of it.

Ubuntu Configuration

Configuring Ubuntu 11 to send email is fairly painless. Using the Ubuntu Software Center, you can locate and install two packages: postfix and bsd-mailx. During the install of postfix, you will get dropped into a debconf window asking you some basic configuration questions:

Figure 1 - The initial configuration screen

I entered the following answers to the two questions I got hit up with:

General configuration: Internet
System mail name: dogma.net

That seemed to be all I needed for basic configuration.

Postfix Configuration

To configure postfix to use SendGrid was just a matter of adding a few lines to /etc/postfix/main.cf, using your SendGrid user name and password. Note that the file probably has an existing relayhost line, this one should replace it:

smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = static:username:password
smtp_sasl_security_options = noanonymous
smtp_tls_security_level = may
relayhost = [smtp.sendgrid.net]:587

After making the changes you should restart postfix so it reads the new config options. I also start watching the mail log file so I can see if there are any problems on first use:

sudo /etc/init.d/postfix restart
sudo tail -f /var/log/mail.log

A test message sent to my cell phone arrived as a text message in just one or two seconds, with the following log messages:

Jun 26 17:02:08 ubuntu postfix/pickup[21145]: 51A5E5E1DA6: uid=1000 from=
Jun 26 17:02:08 ubuntu postfix/cleanup[21336]: 51A5E5E1DA6: message-id=<20110627000208.51A5E5E1DA6@ubuntu>
Jun 26 17:02:08 ubuntu postfix/qmgr[21146]: 51A5E5E1DA6: from=, size=273, nrcpt=1 (queue active)
Jun 26 17:02:08 ubuntu postfix/smtp[21338]: 51A5E5E1DA6: to=, relay=smtp.sendgrid.net[174.36.32.204]:587, delay=0.33, delays=0.04/0.02/0.23/0.04, dsn=2.0.0, status=sent (250 Delivery in progress)
Jun 26 17:02:08 ubuntu postfix/qmgr[21146]: 51A5E5E1DA6: removed

Moving On to Better Things

Now that postfix is properly configured, I can really start taking advantage of the mail infrastructure on my system. The next obvious step is to create a .forward in my home directory, and give it my external gmail address. That external address will now be the recipient of output from cron jobs, or from at or batch. It's nice to have the mail set up as in integral part of the O/S, and if you can just make it through a little bit of setup, it's all yours.

With a limit of 200 messages a day you can still make extensive use of outbound email for system monitoring - whether it is via text to your phone or huge messages being sent to an account used for storing log files. Either way, integral email is still a great feature, almost forty years after it first showed up in UNIX.