Microsoft has added a new keyword to C# as part of the 4.0 release earlier this year. Objects that are typed as dynamic bypass normal static type checking, allowing C# to have the flexibility of other scripting languages.

This is all well and good, but the headline writers of the blogosphere have taken a decided wrong turn with their naming of this feature:

C# 4.0: Dynamic Programming

C# 4.0, Dynamic Programming and JSON

Dynamic Programming Using C# 4.0 and Microsoft Visual Studio 2010

C# 4.0: Dynamic Programming

Note the misuse of the term Dynamic Programming. Everyone who takes an introductory algorithms course learns that the term Dynamic Programming has been in use for over fifty years, and refers to a method for solving problems by decomposition. It’s a useful technique that I’ve covered here in the past, and any skilled programmer should be familiar with it.

No, it’s not the end of the world, but people who are writing about Computer Science really ought to know something about Computer Science, don’t you think?

Dr. Dobb's Portal November, 2007 Article on DDJ site

As I've confessed in the past, I'm a sucker for word puzzles. My recent post on a Will Shortz puzzle from NPR Morning Edition ended up provoking a surprising amount of comment, much of it in the vein of Watch me solve it better, faster, and with more style using language XXX.

I certainly enjoyed watching other people solve the problem, and found their solutions instructive. As the XP crowd has figured out, we programmers spend too much time working on our own problems and not enough time watching how other people work. There's a lot to learn, both good and bad, from getting a peek inside another person's head.

Out of the Blue

Which brings me to the puzzle at the center of this article.

Beth Katz

In what at first seemed to be an incident completely unrelated to word play, I had a pleasant email exchange with Beth Katz, who was teaching a Data Structures class at Millersville University. I happened to look at Beth's current homework assignment for her class, and you can imagine my reaction when I saw the problem she had posted for her class:

We define word reduction as removing a single letter from a word while leaving the remaining letters in their original order so that the resulting sequence of characters is also a word. A good word can be reduced step-by-step until all that is left is a or i. Your program will answer the question: what is the biggest word in the given dictionary that can be reduced?

Beth gave a short example of a good word: planets:

planets
plants
pants
pant
ant
an
a

As you can see, you can remove one letter at a time, and each time you are left with a valid word one character shorter.

This makes for an interesting problem indeed. I've read that the average English speaker has a vocabulary of perhaps 15,000 to 20,000 words, but many reasonable word lists have upwards of 100,000 English words. How many of these words qualify as good words?

As I discussed in the previous word puzzle article, the highly evolved pattern matching facility in the human mind is often pretty good at solving these problems, and I think this is the case (in a limited way) for this particular problem. If I give you a word (like planets) above, I think you'd be able to find a possible reduction path quickly, subject to the limitations of your own vocabulary.

But the human mind is not so good at certain variations on the same problems. Asking you for the biggest word that fits this pattern presents you with an almost impossible task. Basically, it requires you to be able to iterate through the words you know, ordered by length, and test each one. Unless you are subject to some pretty incredible flashes of insight, I think you're going to need a computer for this.

Going Bottom Up

Maybe the feeling isn't universal among programmers, but when I look at a problem, my first instinct is usually to try a top-down approach. For this problem, a top-down approach would mean identifying the longest words in the dictionary, then attempting to decompose them into successively shorter words.

This approach will work, but a little mental analysis shows that it might be a little resource heavy. Imagine that you are decomposing a 10 letter word by taking away one letter at a time. In the worst case, you might find all 9 shorter words in the dictionary, and then you could find all 720 8 letter words, and so on. Although in the general case you might only find one or two matches, particularly at the long lengths, even the potential for factorial growth leaves some room for concern.

So I took a shot at a bottom-up approach instead. It isn't usually my first choice, but I think you'll see that in this case it yields a much more satisfactory and efficient solution to the problem.

The Inner Loop

For this program to succeed, it must terminate its processing with a container that holds the longest good words in the dictionary. For this particular problem, my choice of container is the C++ hash_set, which is non-standard but universally implemented.

My bottom-up approach means that I will fill in the hash_set container for words of length 1 first, then words of length 2, and so on. For reasons of efficiency, it works out better if I keep a separate hash_set for each word length, so the hash_set objects are actually stored in a map that is indexed on the word size:

std::map<size_t,hash_set<std::string> > good_words

To fill in the hash for size i, I need a loop that iterates over all words of size i, removing one character at a time and then testing to see if the result is a good word of size i-1. If it is, I add it to good_words[ i ]. When I'm done iterating over all words of that size, I have a hash_set that contains all good words of that size, and I can move up to the next larger size.

So if we're testing words of size i the innermost part of the loop will look like this:

In this inner part of the loop, we repeatedly remove a character from the word, testing to see if the resulting word appears in the list of words saved in the previous size hash set. If a match occurs, the word is inserted into the hash set for the current word size, and the loop breaks.

This bottom-up approach seems like the ideal way to build my lists of good words for a couple of reasons. First, I don't go to the expense of adding words to the hash sets unless they are good words. Second, determining that a given word is a good word only requires a test against the words at level i-1; I don't have to test for a complete path down to 'a' or 'i'.

Building The Input Data

In the previous section I mentioned that the innermost loop was going to be called as I iterated over all the dictionary words of a given size. So how do I get all the words of a given size?

The first thing that might occur to you is that you could read in all the words from the dictionary, then sort them by size. This approach would work, but given that the standard sort routines available to you in the C++ library are all going to work in O(n·lgn) time, it might get kind of expensive as the dictionary grows to hundreds of thousands of words.

The good news is that with this data set we're in position to take advantage of a linear sort. Yes, we can sort data and do substantially better than O(n·lgn) when we know that the data to be sorted is constrained to a small set of values.

In this case, I just create one linked list for each word size, and as I read the input file, I add each input word to the front of the appropriate linked list. This would be a true linear algorithm if I constrained the input size to a fixed number, say 25, but for convenience I actually store the lists in a map that looks like this:

std::map<size_t,std::list<std::string> > words_by_length

As a result the input code runs in close to linear time. The actual loop that reads in the data is nice and simple:

Once all the words are read in, I can access the list of words of a given size with a simple map lookup: words_by_length[ i ].

Word Lists

One final detail before I can compile my library - I need some lists of words! Lists of words are not hard to come by, although coming up with a suitable one for this exercise might take some work.

One of the first places to look is is on your local Linux system. My system has a list of words in /usr/share/dict/words, which is used by the spell checker application, and possibly by the password app. One one of my systems, this dictionary has a whopping 483,524 words, which means it is packed with obscure words. Just as an example, a typical 12-character good word and its derivation found using /usr/share/dict/words yields this head-scratching sequence:

abranchiate
branchiate
branchiae
branchia
branchi
branch
ranch
rach
ach
ch
h

Probably the first thing you want to do with that file is go through and remove all the single letter words except 'a' and 'i', but even so, you're going to be boggled by some of what you see.

Another good alternative are the collection of word lists distributed on Project Gutenberg as the Moby Word List. This includes a wide variety of lists of various sizes.

Beth Katz had several good dictionaries listed along with her homework assignment, including a short one called kids.dict that is nice and short, making it good for debugging runs.

Finally, Google searches for "word lists" will turn up many other good choices.

Wrapping it up

Once I had the bottom-up good word builder working with the file-based word list, I was ready to put it all together. The core of main() now looks like this:

Procedure read_words() simply reads all the words from the file into the map of hash_set containers called word_by_length, as described earlier.

Then build_up_words collects all the good words and stores them, organized by size, in the map of hash_set containers called good_words. The core of that routine was described earlier.

I show the results of the top two levels in print_good_words(), which is simple enough to read up on in the source code.

The Results

By now you are no doubt dying to see some results from the program. But first, to scope the size of the problem, here's the output from the program as the good_words hashes are populated on a run against SINGLE.TXT, a 354,985 word database:

Loading words from: SINGLE.TXT
Loaded 354985 words from SINGLE.TXT
Found 431 eligible words out of 431 total at length 2
Found 2075 eligible words out of 2092 total at length 3
Found 6213 eligible words out of 6758 total at length 4
Found 11322 eligible words out of 15047 total at length 5
Found 12495 eligible words out of 28473 total at length 6
Found 8939 eligible words out of 40094 total at length 7
Found 4295 eligible words out of 49528 total at length 8
Found 1210 eligible words out of 51216 total at length 9
Found 174 eligible words out of 43964 total at length 10
Found 20 eligible words out of 36082 total at length 11
Found 0 eligible words out of 28009 total at length 12

So that means about 13% of the words in this vocabulary were good words. I'm a little surprised that it's that high. To add some sanity to the mix, I removed all the single character words from SINGLE.TXT with the exception of 'a' and 'i', and the ratio went down to a more reasonable 8%.

You can also see, as you would expect, that the proportion of good words goes down at each level. At lengths 2, 3, and 4 nearly all words are good words, but by the time we get to length 11, we're down to less than one-tenth of one percent good.

Even with my modified version of SINGLE.TXT, you're bound to get plenty of esoteric words when working your way through the derivation of an 11 or 10 character good word. Of the 18 words of eleven characters, the derivation that works best with my vocabulary would be the following:

sparklingly
sparkingly
sparingly
springly
springy
spring
sprig
prig
pig
pi
i

With the more manageable scrabble.dict dictionary, containing 79,340 words, some of the first sequences that pop out include:

shopping hopping hoping oping ping pig pi i
breaches beaches baches aches aces ace ae a
marchese marches arches aches aces ace ae a
prawning pawning awning awing wing win in i
stablest stalest stales tales ales als as a
bravoing braving raving ravin rain ain in i
failings filings flings lings lins ins is i
relaters elaters elates elate late ate ae a
roadster roaster raster rater rate ate ae a
semioses semises seises seise seis sis is i
clambers lambers lamber lamer lame lam am a
claviers clavers lavers avers aves ave ae a
shrieves shrives shives hives hies his is i
stalkier talkier talker taker take tae ae a
statutes statues states tates ates ate ae a
swarming warming waring wring ring rin in i
brambled rambled ambled amble able ale ae a
stratous stratus status stats tats tas as a
paddlers paddles padles pales ales als as a
thirling tirling tiring iring ring rin in i
trucking trucing truing ruing ring rin in i
brawlier brawler bawler baler bale ale ae a
frilling filling filing fling ling lin in i
carouses arouses arouse arose arse are ae a

No doubt there are still plenty of obscure words here, but remember, this dictionary is probably composed of at least 50% words that aren't in your working vocabulary.

Efficiency

When run against a word list with 350K+ entries on my anemic notebook computer, it takes almost 10 seconds for the program to terminate, including display time. The vast majority of that time is spent checking words for goodness, which requires removing characters one at a time, then checking to see if their small descendants are in the word list.

Obviously, if you want to optimize this program for better performance, that's the place to do it. My guess would be that the std::string class member to erase characters from a word is probably far from optimal, and could be replaced by a hand-coded routine designed to do the same task with much greater speed.

Issues With Non-Standard Library Functions

Because hashed containers did not make it into the original C++ standard, there is a somewhat higher level of peril when using them. Problems ranging from syntactic inconsistency to lack of performance guarantees definitely make hash_set and hash_map second class citizens compared to the other standard containers. I saw a good example of this when I first started work on this article.

When solving this problem, the first thing that seemed obvious to me was that we were going to be storing references to dictionary words in hash tables. And I thought it might be interesting to see how well the C++ hash classes were going to be able to handle input data with hundreds of thousands of words.

I thought a good test program would be one that simply reads in the text file and adds it to a hash set:

Under Visual C++ .NET 2005 on a fairly slow laptop, I immediately saw a nasty problem with the Microsoft implementation of hash_set. Every time the counter hit a an even power of two, there was a bit of a pause. The pause grew longer and longer as the count grew larger, until by the time I was up to 128K words, it stretched out to many seconds.

Lesson learned. Hash table resizing can be expensive under some implementations of this non-standard class. Just resizing hash tables in any implementation can be difficult, but the additional requirements imposed on C++ library containers adds significantly to the work that must be done at this point.

I hoped that I would find a reserve() method or a constructor option that would let me preallocate a hash_set with perhaps 200K buckets, but this doesn't seem to be possible with Microsoft's implementation. The good news is that the hash_set replacement in TR1, unordered_set, will impose a requirement that conforming libraries allow for a bucket count as part of the container's constructor.

It turned out to not be too important, however. As I worked on the implementation of the algorithm, I drastically reduced the number of strings that were stored in any one hash, making this a moot point.

Source Code

You can download the project here, including project files for Visual Studio 2003 and 2005, and a simple Makefile for gcc. The code has been checked under gcc 3.4, but I make no claims that it will work with all later versions of the library.

References

Ward, Grady. "Moby Word Lists by Grady Ward - Project Gutenberg." Main Page - Gutenberg. 12 Nov. 2007 http://www.gutenberg.org/etext/3201.
"ISO/IEC JTC1/SC22/WG21 - The C++ Standards Committee." Open Standards. 13 Nov. 2007 http://www.open-std.org/jtc1/sc22/wg21/.

Dr. Dobb's Portal September, 2007 Article on DDJ site

Finding the Convex Hull of a set of points is an interesting problem in computational geometry. It is useful as a building block for a diverse set of applications, including thing such as:

• Collision detection in video games, providing a useful replacement for bounding boxes.
• Visual pattern matching/object detection
• Mapping
• Path determination

Just as an example, if one of the Mars rovers has to chart a path around a boulder, the convex hull can be used to provide the shortest path past the obstacle, given a map that shows the points where the boulder abuts the ground.

This article will go over the definition of the 2D convex hull, describe Graham's efficient algorithm for finding the convex hull of a set of points, and present a sample C++ program that can be used to experiment with the algorithm.

The Convex Hull

There are many ways to draw a boundary around a set of points in a two-dimensional plane. One of the easiest to implement is a bounding-box, which is a rectangle that spans the set from its minimum and maximum points in the X and Y planes.

Creating a bounding-box is easy, but it doesn't form as tight a wrapper as we might like around a set of points. Consider the bounding box around the three points shown in Figure 1.

Figure 1 - A standard bounding box around three points

Figure 2 - A convex hull around the three points from Figure 1

An informal definition that has a little more precision but is still easy to understand might say that the convex hull meets the following properties:

1. The hull is a cycle graph whose vertices are composed of a subset of the the points in set S.
2. No points in S lie outside the graph.
3. All interior angles in the graph are less than 180 degrees.

Computing the Convex Hull

So given a set of points, how do we compute the convex hull without benefit of hammer, nails, and rubber bands?

For some problems, a brute force solution is adequate. In the case of a convex hull, a reasonable brute force algorithm might look like this:

After running this algorithm, you've got a list of point pairs that compose the solution set, and you simply have to put them together in the correct order.

This solution will work, but with just a quick look at the code you can see a big problem - a triply nested loop that runs over the magnitude of N, making this an O(n³) algorithm. That's not going to scale up as well as we might like.

Fortunately, a little searching will show you that there are algorithms that calculate a convex hull in a 2D space considerably faster - in O(n·lgn) time, as a matter of fact.

The Graham Scan

The algorithm I'll demonstrate here is referred to as the Andrew's variant of the Graham scan. Ronald Graham's 1972 paper [1] proposed a convex hull construction algorithm that ran in O(n·lgn) time, and Andrews variation is a simplification that requires a bit less computation. First I'll give the terse definition of the algorithm, then explain each step in more detail.

Algorithm ConvexHull( S )
Sort all points in S based on their position on the X axis
Designate point left as the leftmost point
Designate point right as the rightmost point
Remove left and right from S
While there are still points in S
   remove Point from S
   if Point is above the line from left to right
       add Point to the end of array upper
   else
       add Point to the end of array lower
//
// Construct the lower hull
//
Add left to lower_hull
While lower is not empty
  add lower[0] to the end of lower_hull
  remove lower[0] from lower
  while size(lower_hull >= 3 and the last 3 points lower_hull are not convex
    remove the next to last element from lower_hull
//
// Construct the upper hull
//
Add left to upper_hull
While upper is not empty
  add upper[0] to the end of upper_hull
  remove upper[0] from upper
  while size(upper_hull >= 3 and the last 3 points upper_hull are not convex
    remove the next to last element from upper_hull
//
Merge upper_hull and lower_hull to form hull
return hull

The details

The algorithm starts with an unordered set of points defined by cartesian coordinates - each point has a position on the X-axis and Y-axis. To illustrate the algorithm we'll start with the points shown in Figure 3.

Figure 3 - The raw points used as input to the algorithm

Obviously once the data is sorted it's trivial to find the leftmost and rightmost points, and remove them from the set of points - these are just the first and last members of the sorted array.

Sorting the remaining points into the upper and lower sets requires that we have some function that determines whether a point is above or below a line. I accomplish this using the following strategy. Given a set of points on a line, p0, p1, and p2, I first perform a coordinate translation so that p1 is at 0,0. I then take the determinant of p0 and p2. The resulting value will be negative if p2 angled off in the left direction, positive if it has moved to the right, and 0 if it is colinear with the first two points.

The matrix that is used to get this determinant looks like this:

For this 2x2 matrix, the formula for the determinant will be:

Partitioning set S into upper and lower is simply a matter of iterating over each point, calculating the determinant, and moving it into lower for a value ≥ 0, or upper for a value < 0.

Once this is complete, the resulting partitions will look like the ones shown in Figure 4.

Figure 4 - The set of points after partitioning
Green points are in the upper partition, red in the lower

The upper or lower hull is started by simply adding left to the output hull. Points are then added from the correct input source. As each point is added, if the number of points in the working hull is equal to 3 or more, a test is made to see if the last three points have created a convex angle.

Testing for the convex angle is done using the same determinant formula as shown above. If the hull has n points, we simply test to see if p_n-1 is above or below the line formed by p_n-2 and p_n. When constructing the lower hull, if we see that the point is above the line, we have violated convexity, and the middle point is removed from the hull. The opposite test is made when constructing the upper hull. This test-and-remove process is repeated until the last three points are convex, or there are fewer than 3 points in the working hull.

An animation of this process is shown in Figure 5.

Figure 5 - An animation of the hull being built

Figure 6 - The finished convex hull

Test Code

A small demo program called graham.cpp implements this algorithm fairly faithfully. In order to make the experimentation a little more interesting, graham provides two forms of output showing the results of the program:

• Standard text output listing the various data sets used in the program
• A command file (gnuplot.cmd) that can be used with gnuplot to visualize the process

The images shown in this article were collected using gnuplot 4, which is a fully featured 2D and 3D plotting program, with excellent multiplatform support. Seeing the algorithm operate visually in real time is very helpful in gaining a good understanding of how it works.

The C++ program utilizes a class called GrahamScan that takes care of all these details. By creating the object and then calling its methods, creation and display of the convex hull is easy to follow. In my test program, main() executes the entire operation by creating the object and then calling its methods:

If you ignore method calls that start with log_ or plot_, the real work takes place in only three steps:

1. Construct the GrahamScan object. This also creates the random set of points.
2. Partition the points into the upper and lower hull sets.
3. Build the convex hull.

The constructor

When calling the GrahamScan constructor, you will pass in five numbers: the number of points, and the min and max values for the X and Y axis. The min and max values not only bound the range of the randomly generated points, they also determine the scope of the axes that will be displayed when the values are shown in gnuplot.

The bulk of the work in the constructor is in these few lines of code:

The data set is stored as std::pair objects in a vector aptly called raw_points. After calling the constructor, the log_raw_points() method can be called, which will product output like this:

Creating raw points:
(97,90) (27,10) (59,8) (58,19) (85,90) (62,91) (94,42) (84,68)
(16,21) (49,14) (31,84) (40,25) (59,95) (55,89) (81,95) (22,46)
(27,80) (18,90) (59,37) (38,45)

Calling plot_raw_points() will create a gnuplot command file that produces output like that shown in Figure 3.

The Partitioning Code

The algorithm definition tells us that the partition step needs to identify the leftmost point, the rightmost point, and the two sets of points above and below the line between leftmost and rightmost.

This is all accomplished in a straightforward manner:

Note that by storing the points in an std::pair, with x in the first member and y in the second member, we can use the standard library sort() routine to order the array.

After sorting and then moving each point into one of the two partitions, we have the following members of GrahamScan defined and ready to use in building the hull:

• left, the leftmost point in the set of points
• right, the rightmost point in the set of points
• upper_partition_points, the points above the line between left and right.
• lower_partition_points, the points below or on the line between left and right.

If you call the log_partitioned_points() method, you'll get output that looks something like this after partitioning:

Partitioned set:
Left : (16,21)
Right : (97,90)
Lower partition: (27,10)(40,25)(49,14)(58,19)(59,8)(59,37)(84,68)
                 (94,42)
Upper partition: (18,90)(22,46)(27,80)(31,84)(38,45)(55,89)(59,95)
                 (62,91)(81,95) (85,90)

Calling the plot_partitioned_points() creates a gnuplot command sequence that will display a plot like that shown in Figure 4.

Creating the Hull

The actual creation of the hull is done by method build_hull, which calls method build_half_hull twice, once with the points on the lower hull, and once with the points on the upper hull. The output of build_half_hull is sent on the first call to array lower_hull and next to array upper_hull:

So the bulk of the work is actually done in build_half_hull, which looks like this:

The main loop in this routine simply pulls points out of the input array, adds them to the output array, and then performs the check to make sure that convexity has not been violated. If it has, points are removed until it is again correct. Figure 5 gives a nice animated view of the process.

Once this is done, calling the log_hull routine produces output that looks like this:

Lower hull: (16,21)(27,10)(59,8)(94,42)(97,90)
Upper hull: (16,21)(18,90)(59,95)(81,95)(97,90)
Convex hull: (16,21) (27,10) (59,8) (94,42) (97,90) (81,95) (59,95)
             (18,90) (16,21)

The plot_hull() method can then be called to create a gnuplot command file that will display the convex hull, as shown in Figure 6.

There are quite a few variations on the 2-D convex hull building process, and this program ought to be amenable to trying out many of them. If you use the existing data structures and just change the algorithms, you can use the existing gnuplot routines to animate your work and get a good feel for how it is working. Enjoy!

References and Links

Source code: graham.zip, which contains VS 2003 and 2005 solutions, plus a g++ Makefile.

[1] R.L. Graham, An efficient algorithm for determining the convex hull of a finite planar set, Info. Proc. Lett. 1, 132-133 (1972).

[2] A. M. Andrew. Another efficient algorithm for convex hulls in two dimensions. Inform. Process. Lett., 9(5):216-219, 1979. (A note about the this algorithm can be found here.)

Dr. Dobb's Portal October, 2007 Article on DDJ site

This article discusses the use of C++ hash containers to improve storage of subproblem results when using dynamic programming (DP.) Memoization is a key part of dynamic programming, which is conventionally done by storing subproblem results in simple tables or lists. Using hash tables instead of these simpler structures will allow you to use dynamic programming while retaining your algorithm's natural recursive structure, simplifying design and making your code easier to follow. I'll provide a fully-developed example of an algorithm, and show how it can be adapted to use hash table memoization.

Background

Dynamic Programming (DP) is a useful technique for algorithm development that is saddled with an unfortunate name. When we refer to greedy algorithms, or the use of divide and conquer techniques, the name provides excellent semantic clues as to what is going on. With dynamic programming, no such luck, but I'm afraid were stuck with name for historical reasons.

Dynamic programming is frequently used to solve optimization problems - that is, problems where a number of choices can be made to lead to some maximum or minimum value. Optimization problems can often be solved efficiently using straightforward iterative or divide and conquer techniques, but dynamic programming becomes useful when those other techniques lead to overlapping subproblems.

The Wikipedia article on dynamic programming gives a good example of what is meant by overlapping subproblems. Imagine that you are a C programmer who decides to use a recursive function call to calculate the n^th Fibonacci number:

This code is correct, but it leads to a lot of extra work. Figure 1 shows a map of the recursive calls made using this algorithm:

Figure 1 The Call Map for fib(6)

Dynamic programming provides a good way to avoid duplicate work done solving overlapping subproblems, but for the results to be useful, the problem has to adhere to the optimal substructure property.

Saying that a problem has this property is just a way of saying that we can find an optimal solution for problem A by breaking it down into problems B and C, both of which are optimal as well. Chapter 15 of Introduction to Algorithms gives nice examples of cases where this property holds, and where it doesn't.

Defeating Overlapping Subproblems

Dynamic programming aims to defeat this problem of repeatedly solving overlapping subproblems. Conventionally, this is done by converting the algorithm to one that uses a bottoms up approach, combined with some global storage for intermediate results.

As an example, we could solve the Fibonacci problem using a bottoms-up approach with C++ code that looks something like this:

In this case we only calculate the value of each fib(n) once, completely eliminating the duplicate calculation of those duplicate subproblems.

Most examples of DP that you find in textbooks or on the web will use a table or array to hold the intermediate results from problems. Examples of algorithms that use this type of memoization include Sequence Alignment, Matrix Chain Multiplication, and Optimal Binary Search Trees.

Using the tabular approach for storing subproblems often works well with a bottoms up implementation of the algorithm. DP algorithms typically start by solving the smallest subproblems, storing the results, combining some of those, storing the results in a new level of the table, and so on, until the top case is reached. Again, most of the tutorial DP examples you see will combine a bottoms-up approach with tabular subproblem storage.

DP Problems

From my perspective, the traditional approach to DP suffers from a couple of problems.

First, it seems to me that people feel more comfortable attacking problems in a top-down fashion as opposed to bottoms-up. This is certainly a matter of psychology, not computer science, but it is relevant to consider human factors when designing for maintenance, testability, and review. In many cases the expression of the algorithm seems more natural when given in the original top-down mode, and requires fewer mental gymnastics on the part of whoever is studying the algorithm.

The second problem is the traditional use of array structures to hold subproblems. The problem starts to creep into play when you look at algorithms like Matrix Chain Multiplication, in which half of the storage space is not even used. Things get even worse when you use a problem like that described later in this article, in which subproblems just don't fit into a row/column organizational format.

Both of these problems can be addressed effectively by using standard C++ hash tables. The procedure is as follows:

1. Implement your algorithm using the basic, inefficient recursive implementation.
2. Create a C++ hash table with global scope that will hold your subproblem results. The key for the hash table should be a concatenation of the input parameters to your function, and the value will be the return type from your function.
3. At the entry point to your function, check the hash table for the presence of an existing value for your subproblem, using the input variables to form the key. If the value is present, instead of executing your function, return immediately.
4. At the exit for your function, store the value you are about to return in the hash table, using the input parameters as the key.

With these minor changes, which can often be accomplished in change to as few as three lines of code, you can convert your inefficient recursive algorithm to use dynamic programming, without having to refactor to a bottoms-up implementation, and without having to shoehorn your results into tabular format.

A Simple Example

Applying this technique to our Fibonacci example, we get a routine that looks like this:

So the first time we encounter fib(4), we have to make the recursive calls to fib(3) and fib(2). But the second time around, the result has been stored in the map, and it is returned without any additional calculations taking place.

Hash Table Refresher

Hash tables were part of the Standard Template Library when the language standard was ratified in 1998. And while most of the Standard Template Library was incorporated into the standard library at that time, the hash_set and hash_map classes were excluded. According to Bjarne Stroustroup [1]:

they would have been in C++98 had we had the time to do a proper detailed design and specification job. There was no doubt that a hash_map was needed as an alternative to map for large tables where the key was a character string and we could design a good hash function. In 1995, Javier Barreirro, Robert Fraley and David Musser tried to get a proposal ready in time for the standard and their work became the basis for many of the later hash_maps [8]. The committee didn’t have the time, though..

While these containers will be added to the standard library under the awkward names unordered_map and unordered_set when TR1 is adopted, there is no reason we have to wait that long. Despite the fact that they are missing from the standard, virtually every modern C++ compiler has adopted a reasonable variant of hash_map and hash_set.
The hash_map container does a great job of memoization as described in this article, with a caveat. As I said earlier, you'll need the key to your hash table to be some subset of the input parameters. That's all fine if your key is a single value using a built-in type, such as int, a std::string, or a pointer. These key values are supported implicitly by most implementations of hash_map.

Things get a little more complicated if you are using multiple values or your own structures as a key into the map. Owing to the lack of a standard, implementing non-standard keys requires slightly different techniques, depending on your compiler.

I'll give you examples here that work with the two compilers you are most likely to encounter: Visual C++ .NET 2003/2005, or gcc 3.x.

Let's say you're doing a study tracking peoples reading habits by age, and you want to implement the following code:

You won't be able to compile this code as-is, for at least three reasons:

1. You need to include the correct header file, which unfortunately differs depending on which compiler you are using.
2. You need to hoist the hash_map name into your namespace.
3. You need to define the hash function and at least one comparison function for the key class, Sample.

Item 3 is only needed if you are using something other than a basic type for your class, and it's the trickiest. Defining the comparison is necessary so that the hashing library code can verify key equality during lookups or collisions.

In both cases, the function is usually pretty easy to write - you just compose it using existing functions. Examples for Microsoft and gcc compilers circa 2007 are given below.

Sample Code for gcc

gcc puts the hashing header files in the ext folder, and uses the __gnu_cxx namespace, isolating the non-standard library extensions into a ghetto of sorts. Both of these are dealt with in the first two lines of the sample code below.

The gcc implementation of hash_map needs two template parameters to handle keys that aren't defined as built-in types: one class that has an operator which returns a hash index given an input key, and a second that returns a boolean value for equality test. I pack both of these into a single class, SampleTraits. The traits class is then passed to the hash_map type definition as the third and fourth template parameters.

When you are defining the hash function, you need to somehow combine hash values for each of the members of your structure. Creating hash keys is somewhat of an art, so I try to use the ones provided by the library writer as the basis for my hash function. In this case I take the hash values for the two members of class Sample and simply XOR them together, which should provide a decent value.

If you are using g++ 4.x, you may be better off using the tr1 unordered_map class. While it is not quite part of the standard yet, it should be soon, and that will insure its future support and compatibility.

Visual C++ .NET 2003/2005

Microsoft's compilers have similar issues, but deal with them differently. (That's the problem with non-standard features.)

Although the hash_map header file is accessed from the normal C++ header folder, the class itself, as was the case with g++, is defined in a different namespace, stdext. Again, that is dealt with in the first two lines of the sample code.

The remaining two issues are resolved by defining a global hash_value class that has an operator that returns a hash key for a given object of class Sample, and a global comparison operator that is used to test for equality of two objects of class Sample. (It is not unusual to use the less-than operator to test for equality, by testing both a > b and a < b, we determine whether the objects are equal or not.)

Some Notes on Usage

The member functions of most implementations of hash_map will be identical to those of the standard std::map container. Storing an entry in a hash_map can be done using an overloaded operator that makes the container look like an associative array:

Looking up a value from the table is done using a call to hash_map::find(). This method takes a reference to a key as its argument, and returns an iterator which points to the end of the container on failure, or to a key/value pair on success. The pair is stored in an std::pair object, which allows you to access the elements with the first and second members of that object:

A Weightier Example

Demonstrating the value of a technique with a problem like Fibonacci numbers can be less than convincing - it's pretty easy to devise reasonable solutions to the problems in an ad hoc manner.

For a more substantial example, I've slightly revised a problem from the Cormen, Lieserson, Rivest, and Stein book:

A sample version of the org chart is shown below in Figure 2. Keep in mind that this sample is fairly small, but Bill's chart will normally have almost 80,000 employees.

Figure 2 The Excess Weight Org Chart

Because of the constraints on the problem, we know we can't invite all employees. We need to decide who to invite, but still obey the rule that no employees are there with their supervisors.

This problem is very amenable to a conventional recursive solution. To determine the maximum weight that can be achieved at a given node, we can use something like the pseudo code shown here:

The logic is straightforward. At each node we calculate two possible values for the maximum weight. The first value, weight1, defines the maximum weight if the node does not attend. Since the node is not attending, the weight will consist of the sum of all that node's immediate children.

The second value, weight2, is used to calculate the maximum value of the node when the node does attend. Since the node is attending, none of its immediate children can attend, which means the total weight will be the sum of the node's weight, plus the max of all of its children's children.

You can immediately see that this algorithm shares a characteristic with the Fibonacci algorithm. When evaluating a node, we make recursive calls at two different depths in the calling tree, and this results in overlapping subproblem evaluation. For example, when implementing this pseudo code on the organizational chart showing in Figure 2, the list of nodes passed in to GET-MAX-WEIGHT will be:

So naturally, Dynamic Programming comes to the rescue. It turns out that this particular problem fits into the category well. We have overlapping subproblems, we have optimal substructure, so we can employ DP.

The Dynamic Programming Solution

A true bottoms-up approach is probably impractical for this problem.

First, it's a lot of work to traverse a tree one level at a time, starting at the lowest level. Second, there are problems with the tabular subproblem storage scheme usually favored in dynamic programming. It's not entirely clear where in a table we would store the result from GET-MAX-WEIGHT( q ).

An approximation of the bottoms up approach could be achieved by traversing the tree in postorder, visiting all the children of a node first, then visiting the node itself. In this scenario we would visit the nodes from Figure 2 in the following order:

Of course, traversing the tree in postorder is actually what we're already doing already in the recursive implementation. Seeing this, my solution to the algorithm was to use a C++ hash table to perform memoization of the subproblems. This has a few advantages:

• My implementation looks just like the recursive definition of the algorithm. It just has a few additional lines needed to check for stored subproblems.
• I don't have to worry about finding a tabular storage method - I store the results based on a hash of the the node pointer, making lookup and storage nice and simple.
• I don't have to worry about how to do bottoms-up traversal of the org chart. My recursive definition visits the nodes in the order I want without any changes.

I'll start by simply presenting my modified version of GetMax( node ), then present the infrastructure that goes with it. Finally, I'll discuss the last step in any dynamic programming problem: using the stored subproblem results to get a solution.

The Memoized GetMax()

The C++ code to solve this problem is completely defined in a class called Memo, which relies on a tree whose nodes are defined by class Node. The two class definitions are shown here:

To run an instance of this problem, I first construct a tree by having the Node constructor read in a definition file. The resulting root node is then passed in to Memo::GetMax(), which works its way through all the subproblems and calculates the maximum total weight for this problem.

The code in GetMax() should appear nearly identical to that in the pseudocode version of GET-MAX-WEIGHT(), with one critical difference: in my implementation of GetMax(), I check to see if a hashed value of GetMax() has been saved upon entry to the routine, and if it has, I short-circuit execution and return that value immediately.

Likewise, at the end of the routine, I store the newly calculated maximum value in the hash table.

The key to this hash table is a pointer to a node - it could just as easily have been the name of the node, but using the pointer has the advantage of making the program a little more invulnerable to input error - duplicate node names won't break the algorithm.

The use of a pointer as a key into a hash_map is a technique that can be used to work around some of the issues I discussed earlier in the article. Because the hash containers offer built-in support for pointers of any type, use of a pointer as a key allows you to avoid having to define your own hash function and comparison function, which is nice.

This routine is conceptually pretty simple. The first four lines check to see if I've cached a previously seen solution to the problem in the mSolutions hash_map. If I have, the mMaxWeight member of that solution information is returned immediately, avoiding a duplicated subproblem.

If the solution is not already stored off, I have to do the real work. This means evaluating both possible solutions - the weight if this node attends, and thus excludes all its immediate children, and the weight if this node does not attend, and thus includes the children. Those two values are calculated and stored in local variables weight_attending and weight_not_attending.

Given those two values, its a simple matter to decide which of the two is larger, and return that one. However, there are a couple of issues to deal with before returning.

First, I want to make sure that we store the solved subproblem in the solution table. Both return statements do that by writing the value to the mMaxWeight member of the correct entry in mSolutions.

That would be enough to get the maximum value, but I may also want to create a solution for the problem - which means not only knowing the maximum weight achieved, but a list of attendees. In order to get that list of attendees, I need to also keep track of whether the maximum value achieved at this node was done so by attending or not attending. I do that by updating a member called mAttending in the Solution object. (Note that the constructor sets mAttending to false upon creation.)

Getting The Max Value

If all I wanted to know was the maximum value that could be achieved, this single method in class Memo would be enough. Running a program that calls this routine for the tree shown in Figure 2 produces the following output:

GetMax(h)
GetMax(i)
GetMax(d)
GetMax(q)
GetMax(r)
GetMax(j)
GetMax(k)
GetMax(e)
GetMax(b)
GetMax(l)
GetMax(m)
GetMax(f)
GetMax(n)
GetMax(p)
GetMax(g)
GetMax(c)
GetMax(a)
Max = 31

GetMax() prints out each subproblem it's trying to solve, and you can see that each node is only called once. The listing also shows you that the nodes are evaluated in postorder, which is the order we would have needed to simulate with a pure bottoms-up implementation.

Getting the Solution

As I said, the implementation as is returns the maximum value, but to be really useful we need to actually know which of the people on the org chart are going to be invited to attend Bill's exercise class.

To come up with this list, we need to traverse the tree, only looking at nodes that are included as part of the solution. Nodes that are not included are those nodes that are skipped over because their parent is attending.

Just knowing that a node is included as part of the solution is not quite enough. A node could be included, yet not be attending because the maximum value for that node is achieved by not attending.

The code that actually develops the solution is called MarkIncluded(). It is called with the root node as the target, then successively walks down through the tree, marking child nodes that are part of the solution. As each node is marked, I can see if it is going to be attending, and its name is printed out:

The logic is pretty simple. When this method is called for a node, that node is automatically marked as included in the solution. Depending on whether this node has chosen to attend or not, I either mark all of its children as included, or mark all of its children's children.

Adding this code to the program lets me produce this output for the tree shown in Figure 2:

Attendees: d e q r c l m n p

A quick check of the weights of those nodes shows that their sum is 31, which agrees with the output shown above.

Graphical Output

I like being able to see the results of of my algorithm in a graphical format, so I added one additional method, PrintTree() to class Memo. This method takes the final tree and prints it out in a format that can be processed into a nice graph using dot, one of the programs in the Graphviz package, a free set of tools from AT&T.

The final result of running the data set shown in Figure 2 through my test program is shown in Figure 3.

Figure 13 Program Output Using GraphViz

Running the Test Program

The test program I used to create this output is a simple program that reads in a tree definition file, outputs the program data to the console, and creates a GraphViz format output file. It's run with two arguments, an input definition file (one sample is included), and the desired output file name:

memo tree01.def tree01.dot

Project and/or make files are included for Visual Studio 2003/2005 and gcc.

Note that the format of the tree definition file is documented in source code.

The main() routine for the test program (minus some error handling) looks like this:

This program should give you an idea of how to harness the power of Dynamic Programming in order to solve some fairly difficult problems with relatively simple algorithms. Dynamic Programming is the key to this, and memoization with C++ hash maps helps make it possible.

References

[1] Evolving a language in and for the real world: C++ 1991-2006, Bjarne Stroustrup, ACM HOPL-III. June 2007. http://research.att.com/~bs/hopl-almost-final.pdf
[2] Graphviz. The free project from AT&T that supports graph visualization.
[3] Source Code for sample program
[4] Cormen, Thomas. et. al. Introduction to Algorithms. Cambridge: MIT Press, 2001.

Dr. Dobb's Portal April, 2007 Article on DDJ site

I'm an inveterate fan of wordplay of all sorts - puzzles, anagrams, crosswords. I've been known online by my anagrammatic name, SnorkelMan, all the way back to the ancient days of the text mode BBS. My continual hectoring of the staff at the Dallas Morning News over errors in their print version of the New York Times crossword puzzle led them to finally just give me the job of proofreading it. I spend way too much time on the crosswords and other puzzles, both online and in print. In other words, I'm a sucker for a good word puzzle.

Figure 1 - Dallas Morning News Article

My sense of wordplay was naturally piqued this weekend when I heard the latest weekly puzzle challenge from Will Shortz on NPR Weekend Edition. The challenge, from contributor David Edelheit, read as follows:

Take the names of two U.S. States, mix them all together, then rearrange the letters to form the names of two other U.S. States. What states are these?

As sometimes happens, when I heard this puzzle, and the answer didn't click immediately, my first thought was "I could write a program to solve this faster than I can figure it out myself."

That's a treacherous thought for a puzzler, because it immediately diverts that little thread in the back of your mind that is supposed to be solving the puzzle, instead putting it on the task of writing the program.

But it turned out to be an interesting problem in efficiency, and so I'm glad I went down that path.

First Pass

Most of my work these days is in C++, and while C++ doesn't have the world's best string manipulation facilities, I thought it had enough to do the job on this puzzle. Figuring that the problem was small enough to solve via brute force, I decided that the general course of the program would be to work my way through all 50*49/2 combinations of states, and test them against all 48*47/2 remaining combinations. That's just a little more than a million operations, which ought to be child's play. Thus, the basic program loop was going to look like this:

Still thinking brute force, I was looking for the simplest way to store the data so that it would be easy to compare, and turned to std::multiset. I knew that if I stored all the characters from states i and j in one std::multiset object, and all the characters from states m and n in another, I could quickly compare one against the other with a simple equality operator.

So in the above loop, I inserted these lines after the first two for statements:

(Note that I had snagged the names of the states from one of the first sites in a Google search, and inserted them into an array of character pointers called states.)

I inserted a similar definition for label2 inside the second set of two for statements, which means all I had left to do was a simple comparison of label1 against label2:

I did a quick compile, tested the code, and sure enough, the contents of the multisets were indeed sorted concatentations of the letters of each pair of states. Time to run!

My first disappointment was seeing that, while the program was indeed running properly, it was going slow enough that it looked like it was going to take a sizable fraction of an hour to make it through the entire alphabet. I could just wait, but in this case I decided I could optimize faster than it would take to wait for the first results.

Second Pass

It's pretty obvious that calculating the concatenation of state m and n in the innermost loop is full of wasted cycles, since it is repeatedly calculating the same state values. I knew it was inefficient, but I didn't think it was going to matter too much.

Since it turned out that it did matter, I decided to precalculate all the values before entering the four-deep nested comparison loop, with code like this:

Then I didn't have to do any computation in my main loop, I just had to modify the comparison line in the innermost loop:

This modified program did indeed speed things up considerably, bringing the run time down from a fraction of an hour to just a few seconds, even with a bit of progress tracing turned on.

Third and Final Pass

Although I could have stopped here, I thought it might be interesting to see how expensive the use of an associative container like multiset was compared to something simpler. I replaced the setup code so that it stored the data in an std::vector instead of a multiset, on the theory that the comparison operator would run much faster on a vector. I had to add a call to sort the data after inserting it in the vector, which would be executed 50 times, but nowhere near as many times as the comparison operator.

The results were more or less as I expected. When run under Windows with default Release optimization, the vector version of the program ran about twice as fast as the multiset version. When compiled with g++ 3.3.5 with -O2, I saw roughly the same ratio of execution speeds.

No Spoilers

I'm not going to spoil the puzzle for you by giving away the answer. Let's just say that it is a good word puzzle, and if you manage to arrive at the answer you'll see why. If you don't manage to solve it in your head, you can download the source, compile it, and get there by brute force, just like I did.

usa.cpp

But I don't think it will count as a spoiler if I tell you that you don't need a computer program to solve this problem.

Dr. Dobb's Portal August, 2006 Article on DDJ site

Over the years I've spent quite a bit of time and money experimenting with the best ways to implement networked music and video in my home. Back in 2004 I showed you how to build a home video jukebox, albeit with a few compromises. Today you might think the best way to get this done is with a Windows Media Center Edition PC (or perhaps its recent competitor, Front Row). I'd like to offer up an alternative solution that offers a feature set that competes, and in some case beats the PC-based solution for a lot less money: the GalaxyMetalGear TVisto media player.

The TVisto Concept

The TVisto media player is basically nothing but a souped-up hard drive enclosure. GMG was already in the enclosure business when it must have occurred to them that adding a lightweight Linux distribution plus a little more CPU power, a media processor, and a remote-control driven UI would give them a box that could:

• Play video files encoded in MPEG-2 or DivX formats
• Display picture files
• Act as a music jukebox
• Fill in as a removable hard drive

In other words, do everything that a Windows MCE PC can do with the exception of recording live TV.

GMG sells this box without a hard drive, and with a little shopping you can find it for under $130 - I paid $125 for mine. Add the 5.25" hard drive of your choice and you are in business. My choice was a 300G Maxtor drive purchased at Fry's for $80 - making my total investment just a little over a couple hundred dollars. Because this is a simple hard drive enclosure, you also don't have to worry about a lot of the negatives associated with Media Center PC: no fan noise, small footprint, instant boot time - all big pluses in my book. Once I hooked it up and put it through its paces, I was sold on the value of this as a Media Center Edition replacement.

Setup

The hardware setup for the TVisto is identical to that for any hard drive enclosure: take out a few screws, open the case, connect the power and data cables to the drive, close the box up and you are done. I found the enclosure to be a bit cramped, and I was worried about damaging cables as I forced the drive in, but all went smoothly and that part of the setup took just 10 minutes. If you are comfortable with cracking the case on your PC to install drives or memory this will be a cake walk.

Once you have the drive installed, you connect it to your PC via a USB cable, power it up, and it should quickly appear as a removable drive. You can then format it as an NTFS, Mac OS Extended, or FAT32 drive (yes, this means you can use it with virtually any O/S.)

The next part of the setup gives you a hint about the simplistic nature of the software driving the TVisto. The instructions make it very clear that you have to create the folders shown in the figure below with exactly the correct names: Firmware, Movies, Music, and Pictures. As you will see when using the TVisto, there isn't a complicated database for music and video like you have with iTunes or Windows Media Player - the TVisto simply browses through folders, and it expects your media to be stored in the folder with the appropriate name.

Figure 1
The mandatory folder layout

Once you have the folders created, copying media from your PC is simply a drag and drop project, copying media files of the appropriate type to the correct folder. As you'll see later, large collections need to be organized at this point using a system of nested folders, and that is entirely up to you to do.

Figure 2
Copying media files to the TVisto

The TV Connection

After following safe ejection procedures from your PC, you can power down the TVisto and connect it to your TV. The unit ships with all the cables you need for NTSC viewing in stereo, and again, you should have no trouble connecting these standard inputs to your home video system.

One of the really great things about the TVisto was finding out that this little box also supports high quality audio and video. The standard unit ships with support for everything from Composite NTSC video up through 1080i, (although you need to purchase an optional cable for component HDTV connections). It also supports 5.1 audio via an SPDIF connector. Combining these two features means you can watch your stored DVDs without compromising on audio or video quality.

I purchased the add-on component cable so I could get the benefit of higher resolution DVD viewing - am annoying $30 expense, but if you have an HDTV it is well worth it.

Figure 3
The $30 component cable

Once you connect the TVisto to your TV, you may find that the default video settings aren't correct - this can be remedied by simply cycling through the various possibilities by repeatedly pressing the TV Out button on the remote. Eventually you'll see the menu shown in Figure 4. (Please note that these are low-res 640x480 screen captures - your HDTV viewing will be much crisper than what you see here.)

Figure 4
The intro menu for the TVisto 3500

A nice, simple interface. The place where you will spend most of your time is in Movies, Music, and Pictures. The only thing I've done in the Settings menu is pin down my video output settings, but you also have a few other options, such as Languages, ScreenSaver, and so on.

The three media buttons, Movies, Music, and Pictures, all have what amounts to nothing more than a folder browser interface. You plow through the files that you have loaded in the appropriate folder and select what you want to see, hear, or look at. Once you are in playback mode there are naturally quite a few more options, but the selection process is dead simple.

Movies

Even if you never use the TVisto 3500 for anything but a video jukebox, I think you'll find that it is worth the money. It supports multiple video formats, but I chose to go with the simplest and easiest format. I used the banned program DVD Decrypter, and told it to rip my DVD to an ISO image. (By the way, if you want a good answer to the question "Is this legal" you might want to see what the Electronic Freedom Foundation has to say about it.) I then simply copied those ISO images to my TVisto in the Movies folder, and was then able to browse them as shown in Figure 5:

Figure 5
Selecting a Movie

As you can see, selecting a movie is done simply by browsing file names, so you will want to be careful about naming your ISO files before installing them on the hard drive. There is no horizontal scrolling, either, so if your names are too long you will be in real trouble - the end of a title will simply be unreadable.

You can also see at the bottom of the list of files that I have put these six DVDs into a separate folder - I'm using a hierarchical storage system to make the navigation process a little easier to follow.

While all this may be a little crude, once you hit the play button the remote in order to start a movie, you will beging enjoing exactly the same playback experience you get from your DVD player - the full set of menus and features you expect:

Figure 6
The Main Menu for Alias, Season 3, Disc 2

Figure 6 shows you a playback in progress, looking just as you would expect at the given resolution, with the English subtitles being overlaid on the screen. (Looks like Sydney is in the middle of a big operation!)

Figure 7
Full Feature playback

I couldn't be happier with this DVD emulation mode of the TVisto 3500. It does exactly what I want, the way I'm used to doing it. The ripping process is one-button-click simple, and I don't have to give up anything in terms of features or performance.

Music

I have to confess that I actually purchased the TVisto first as a music player. I like to have access to my entire music collection in my media room, and despite the efforts of all the manufacturers out there, have yet to have a satisfactory experience with a networked player. For example, the Linksys Wireless-B Media Adapter seems like solid product from the spec sheet, but it suffers from such a litany of shortcomings that I found it completely unusable. (Disclosure: I work for Linksys - or at least I did before this article was published.) My Windows Media Center Edition experiences were even more frustrating, again using a Linksys product, the Dual-Band Wireless A/G Media Center Extender (which notably seems to have stopped shipping.) In this case I blame most of the woes on Microsoft, not Linksys, for the unfriendly music experience.

So instead of hassling with a remote box that has to talk to a server, (always including some required server software), the TVisto offers an easier to manage remote experience with less hassle on my part. All I have to do is copy files to my Music Folder, and I'm in business.

Figure 8 shows the music navigation system in place, and just as you might have expected, it is simply a folder browser. This is both a strength and a shortcoming of the TVisto. Since it doesn't have to maintain a big database of titles, it is much simpler and more reliable. At the same time, that database of titles that you get with iTunes or Windows Media Player does make it easier to navigate through your songs.

Figure 8
Navigating through the music collection

Note that again, in order to bring some help to the navigation problem of working through several hundred CDs, I've gone hierarchical, with each family member having their own folder, with artists under those, then albums under those. This is okay, but not optimal. If you want to play a specific CD it means you either have to know whose folder it is in or else spend a lot of time browsing - no search function available.

If your needs are restricted to listening to one CD at a time, you will find yourself in hog heaven. When you select a CD, you can either play individual songs of your selection or with one button press instantly play the entire CD.

Figure 9
Music plaback in progress

The one place where the GalaxyMetalGear team could really afford to work on firmware improvements would be here. Feature competition from Microsoft and Apple has really raised the bar on music playback, and a simple file-folder oraganized MP3 playback experience won't cut it for a lot of people. Things I'd like to see in updates to the current product would include:

• Big improvements in navigation, with an eye towards speed - pressing the page button 20 times to get to the end of a list is pretty annoying.
• Support for playlists, ratings, and searching
• Support for ID3V2 tags, including multiple Genres per song
• Shuffle options
• Display of album artwork and lyrics
• Visualization options

Despite these missing features, make no mistake about one thing, this is still an excellent replacement for the CD player in your media room. It performs the same function, but allows you to quickly access your entire collection and frees up the space that was occupied by that ugly CD rack you bought at Target 10 years ago.

Pictures

I consider the picture album feature almost a throwaway - it was easy to add, so it was included, but I don't know anyone who actually uses their TV to browse through their photos (my apologies to those of you out there who do!)

Basically, the photo album feature operates identically to the music feature, meaning you get to view one folder at a time. Figure 10 shows this in play on my TV, and its certainly nice, but the lack of tagging and organization makes it a bit lame compared to a web site like Flickr.com.

Figure 10
Viewing Photos

Conclusions

If you're looking at price performance, I think the TVisto can't be beat. I won't be surprised if an iPod will be able to do all of these things soon (except for playing DVDs you've ripped yourself), but even then, the form factor will always mean you are paying a huge penalty for disk space. My $200 TVisto has 300G of disk space - an iPod with 1/5 that space costs twice as much.

The one thing that would take this product from second base to home would be network access. Given that it is running a standard Linux distribution, I have to believe that it would be fairly easy to support a USB network interface. If I could load new media on the TVisto without having to lug it from one room to another, I think I would then be able to say the TVisto 3500 is feature complete. (Opening this box up for community software development might be one way to get those new features in more quickly.)

A few notes on pros and cons to help you with your buying decision:

Pro:

• No fan
• Small footprint
• Very inexpensive
• Plays ISO-ripped DVDs
• Plays DivX compressed video
• OS independent

Con:

• No protected music
• No network access
• Lots of UI limitations - slow scan through music and photos
• No playlists
• No shuffling across folders
• Doesn't read ID3V2 tags

Look at it this way - in the worst case, you'll still get to use it as a portable hard drive!

Dr. Dobb's Portal August, 2006 Article on DDJ site

International standards are good, we all love them, sure. But if you aren't lucky enough to be employed by a big company or a university, getting your hands on these standards can be a real pain. Typically the standards bodies sell electronic copies of their work for fairly high prices, justifying it as one of the only ways they can subsidize their activities.

Just an example, if you want to buy a hardcopy of the C++ standard, you can end up paying as much as $US 175 for the privilege - which is a killer if you are just doing this for personal use.

This turns into a FAQ in public newsgroups and forums when the uninitiated start operating under the idea that they ought to be able to just find a copy on the web. This thread on comp.compression is a classic example, where Nils Haeck is looking for free copies of some of the JPEG documents.

But oddly enough, later on in the thread Nils sheds a piece of light on the discussion, disclosing a piece of news that I hadn't heard before. It turns out that the ITU has a program in which individuals can get up to three free electronic copies of their standards documents per year. This is exciting!

I thought I'd give it a try by first signing in at the ITU Bookstore. You'll note on this page there is a button that says: I wish to REGISTER in order to download up to three (3) Recommendations free of charge. Yes, I certainly do. I selected the button and went to the registration page. The registration, shown below, has a bit of a problem being rather corporate-oriented, so I tried to steer it into the notion that I am an a company of one:

Figure 1 - The registration form

All went well and the ITU blessed me immediately. My email was almost instantaneous, and I was ready to shop.

Figure 2 - The ITU loves me

Just to test the system, I first downloaded a spec I knew I would find handy, JPEG 2000 image coding system: Core coding system. Sure enough, a few minutes later I had my personal copy in hand. Now I just have to decide where to spend my precious remaining two downloads:

• JPEG - Digital compression and coding of continuous-tone still images
• JPEG 2000 image coding system: Reference software
• ITU-T T.81 (JPEG-1)-based still-image coding using an alternative arithmetic coder
• List of definitions for interchange circuits between data terminal equipment (DTE) and data circuit-terminating equipment (DCE) (aka RS-232)
• H.323 - Packet-based multimedia communications systems
• G.722 : 7 kHz audio-coding within 64 kbit/s
• Coding of speech at 8 kbit/s using conjugate-structure algebraic-code-excited linear prediction (CS-ACELP)
• Software tools for speech and audio coding standardization

It's nice to at least have a choice, isn't it? Now if we could get the ISO and ANSI to adopt similarly enlightened policies the world would be a much better place.

Dr. Dobb's Portal September, 2007 Article on DDJ site

In the new global economy, companies have the opportunity to market their wares to billions of customers who don't speak a word of English. Java was designed from the ground-up to help programmers deploy internationalized software. In this article I'm going to show you how Java makes your woes in the area of character sets and encoding melt like cotton candy on your tongue.

Dipping My Toes in the Global Pool

For the past year I've been spending a steadily increasing amount of time at work dealing with internationalization of our products. My division of Cisco makes an IP-based phone system that interfaces with users and administrators at dozens of different points. Users have alphanumeric displays on their phones, perform personal and system administration on web pages, and listen to voice prompts when collecting messages. From the user interface perspective there is a lot going on.

Like most new products made in a skunk-works atmosphere, Cisco CallManager was developed with little or no thought towards our international customers. The focus was on quickly developing a stable product with as many features as possible. Of course, our success at this strategy led to immediate discontent from our business partners in Europe and Asia. It turns out that telephone users in France really do want to have their instruction manual to be written in French.

The Four Problems of Text Internationalization

On the surface, modifying your product for users of another language seems simple enough: translate everything and distribute the results. Unfortunately it just isn't that easy. Translation is only the first of four big problems. Translating written material might present some logistical problems, but these are usually more budgetary than technical.

The other three problems are more technical in nature, and Java provides tools to deal with all three. In order, these problems are:

Properly encoding text in a given character set so that it can be stored and transmitted in a world of eight-bit bytes.

Java helps you deal effectively with all three of these problems. The first, management of translated content, is handled using Java Resource Bundles. This article is going to talk about the next two: character sets and encodings.

Character Sets

Cisco's business-class IP Phones have a nice LCD screen that presents call status information to users in a fairly friendly way. One of the first problems we ran into when internationalizing the entire phone system was that this display only supported the 7-bit ASCII character set. This had the unfortunate effect of changing the name of Señor Nuñez to "Senor Nunez" when stored as a speed dial. That's because the character set we were using lacked the letters commonly seen in other countries using the Roman alphabet.

A short-sighted solution to this problem might be to expand the character set to a full eight bits, using the upper 128 character positions for the commonly missed characters. And in fact, there is a standard character set called ISO-8859-1 that does just that. Figure 1 shows how ISO-8859-1 populates the upper half of the character space with 112 characters used in Western Europe. (The first 32 positions in the upper half of the space are not used.)

Figure 1 - The Upper 112 Positions of the ISO-8859-1 Character Set

We quickly modified our phone to accept a new font, and soon found that we could properly render names of people in France, Germany, Italy, the Netherlands, and so on. As long as we confined our sales efforts to our friends in NATO all was well. Figure 2 shows our phone happily rendering most of the ISO-8859-1 character set, ready to march onto desktops anywhere the Euro is honored.

Figure 2 - A Pan-European Phone

Even in today's global economy, many manufacturers of software and other products find this state of affairs to be just dandy. But as soon as you try to sell a phone in Greece or the Russian Federation, you're out of luck. Languages such Russian and Greek just don't have enough characters in common with Western Europe to fit into the ISO-8859-1 character set. Not to fear, because ISO-8859-5 and ISO-8859-7 were created just to deal with this problem.

Figure 3 - The Upper 112 Positions of ISO-8859-7, a Greek Character Set

If you take a look at Figure 3, you can see the seeds of a problem beginning. Adding a new character set to the phone is a manageable problem - one that operating systems like Windows already manage quite effectively. We simply have to make it possible for our telephone to download one font in France, and a different one in Greece.

But now the phone finds itself in a situation that can seem a bit baffling to mossback programmers such as yours truly. If a multinational company has a Señor Nuñez listed in its corporate directory, things will work just fine when a user in France or Italy looks up his name. But unfortunately, a user in Greece will see the name rendered as of Seρor Nuρez. We now have a fundamental problem: a given numeric value is rendered differently depending on the character set in use. A lower case 'n' with a tilde over it in ISO-8859-1 transforms itself into a lower case Greek rho when we switch to ISO-8859-7.

This unpleasant situation causes a huge paradigm shift for programmers working on internationalization. There's no longer any such thing as "plain text." When we store a user's name in the database, we now have to also store the name of the character set that properly renders it. The same thing holds true for error messages, speed number labels, soft-keys, you name it.

Or does it?

Java to the rescue - Part 1

Java deals with this problem in an effective way - it coerces you rather firmly into using Unicode for all character strings. C++ developers have a choice between narrow strings and wide strings - which aren't necessarily Unicode. For better or worse, Java eliminates that choice.

The nice part about this is that in Unicode, U+00F1 is always the ñ character, and the lower case rho, ρ, is always U+03F1. Even if I don't have a rho character on my keyboard, I know that I can use Java's escaping mechanism to represent it as "\u03F1", inconvenient as that may be.

This is a nice feature, because it means that at least internally, a Java string is a string is a string. You don't have to worry about what character set it is from - it's Unicode.

It's worse than you think

In a perfect world, Java's insistence on Unicode would spill over to every file system, network packet type, and so on, and everything would be fine. But unfortunately, there are still billions of web browsers in the world configured to read text from an ISO-8859-X character set. And when our attention turns to Asia, things get even worse, for two reasons.

1. China, Japan, and Korea have character sets composed of thousands of ideographs. To compound this problem, there are competing character sets used to create Chinese web pages. Taiwan and the PRC tend to use two different character sets, known as Traditional (or Big5), and Simplified (or GB2312.)
2. These character sets don't fit in a single byte, and accordingly must be encoded in order to be written into byte-oriented files and networks. Unicode is most commonly encoded as UTF-8, in which a single 16 bit character is encoded as one, two, or three bytes. Other 16 bit characters, including the Chinese, Japanese, and Korean character sets use different encoding schemes, usually a row/column value encoded as two bytes.

Naturally, the different character sets that I've mentioned here are incompatible with one another. Needless to say, the encoding schemes are incompatible as well.

Simply storing your data internally as Unicode doesn't solve the problem of incompatible character sets and encodings. But, the good news is, Java has built-in library support for converting to and from these encodings any time you convert to or from bytes during an I/O operation.

Both the OutputStreamWriter and InputStreamReader class have two constructors: one which takes just a reference to a stream object, and a second which requires both a stream object and an encoding parameter.

If you search through the Java docs for "Supported Encodings", you'll see that Java has built-in support for a huge library of character sets and encodings. Converting one of these to or from Unicode is simply a matter of instantiating a class with the correct encoding parameter.

See the Code

Figure 4 shows a sample Chinese language Web page that is written in Unicode and encoded with UTF-8. Users with the latest operating systems and browsers will usually be able to properly render this page properly.

Figure 4 - Welcome to the Classical Music Site

But not everyone has a Unicode capable computer, operating system, and browser. A user who browsed to this page with a browser set to use a Big5 character set would see the screen shown in Figure 5:

Figure 5 - Welcome to the Illegible Music Site

Solving this problem is easy with Java. All I have to do is develop my content in Unicode, then use Java's built-in classes to churn out localized versions suitable for users of whatever encodings are needed.

Listing 1 shows the WebWriter class that I used for this article. This program has a complete copy of a web page's content stored in an internal string. By using three different encodings, it creates web pages for browsers set to Unicode, Big5, and GB2312. As you can see, choosing the correct character set and encoding from Java is trivial.

PLAIN TEXT

JAVA:

1. import java.io.*;
2.  
3. public class WebWriter {
4.   static String eol = System.getProperty( "line.separator" );
5.   static String s =
6.   "<HTML>" + eol +
7.   "<BODY>" + eol +
8.   "<TABLE cellspacing=\"5\">" + eol +
9.   " <TR>" + eol +
10.   "  <TD><img src=\"michael.jpg\"></TD>" + eol +
11.   "  <TD><H2>" +
12.   "\u6b61\u8fce\u5149\u81e8\u53e4\u5178\u97f3\u6a02\u7a7a\u9593" +
13.   "</H2></TD>" + eol +
14.   "  <TD><img src=\"violin.jpg\"></TD>" + eol +
15.   " </TR>" + eol +
16.   "</BODY>" + eol +
17.   "</HTML>" + eol;
18.  
19.  public static void main(String[] args)
20.  {
21.   try {
22.   FileOutputStream fos =
23.       new FileOutputStream("c:/temp/page_utf8.htm");
24.   Writer out = new OutputStreamWriter( fos, "UTF8" );
25.   out.write( s );
26.   out.close();
27.   fos = new FileOutputStream( "c:/temp/page_gb.htm" );
28.   out = new OutputStreamWriter( fos, "GBK" );
29.   out.write( s );
30.   out.close();
31.   fos = new FileOutputStream( "c:/temp/page_big5.htm" );
32.   out = new OutputStreamWriter( fos, "BIG5" );
33.   out.write( s );
34.   out.close();
35.   }
36.   catch ( Exception e )
37.   {
38.     System.out.println( "Exception " + e );
39.   }
40.  }
41. }

List 1 - WebWriter.java

Details

You can see the actual HTML files created for this article using the links below. Note that well written web pages use meta tags or HTTP headers to help a browser figure out what encoding and character set to use. (See the section 5.2.2 of the HTML Spec for details.) These web pages don't; they provide no information intentionally, making experimentation a bit easier.

To view the web pages in their correct encoding, you will need to change your browser encoding setting, unless it guesses right based on content. In Internet Explorer, you select this from the View|Encoding portion of the menu. Firefox is nearly the same: View|Character Encoding. If you are an English-speaking computer user, you will undoubtedly have to install Chinese or Unicode fonts as well. If you're lucky, this will be a semi-automatic process.

The web pages can be found here:

GB2312 page

If you look at the source code, each Chinese ideograph will be two or more characters, which will look to your ANSI text-editor something like this:

­¹âÅR¹ÅµäÒô˜·¿Õég

The text shown above contains ten GB2312 characters encoded using twenty bytes. For non-Chinese speakers, the following table shows the translation of the individual characters, as well as the more meaningful translation of short phrases consisting of multiple characters.

Ideograph	Unicode	GB2312	Big5	Character Meaning	Phrase Meaning
	U+6B61	2722	C577	happy, pleased, glad; joy; enjoy	Welcome
	U+8FCE	5113	AAEF	receive, welcome, greet
	U+5149	2566	A5FA	light, brilliant, shine; only	to
	U+81E8	3357	C17B	draw near, approach; descend
	U+53E4	2537	A56A	old, classic, ancient	classic
	U+5178	2168	A8E5	law, canon; documentation; class
	U+97F3	5084	ADB5	sound, tone, pitch, pronunciation	music
	U+6A02	3254	BCD6	happy, glad; enjoyable; music
	U+7A7A	3153	AAC5	empty, hollow, bare, deserted	space
	U+9593	2868	B6A1	interval, space; place, between

Table 1 - Translation of the ideographs in the sample web page

Conclusion

Successful products today need to support customers all over the world. Using Unicode for your core content makes this much easier, and Java is ready to help you on this path. More importantly, Java makes it simple to continue talking to devices on the edges of your network that are still using old-school character sets and encodings.

Unfortunately we still have to use human beings to do the difficult work of translating our content from one language to another, but outside of that Java does everything we need.

]]> http://marknelson.us/2006/07/13/java-good-world-citizen/feed/ 0 http://marknelson.us/2005/10/03/ip-proposal/ http://marknelson.us/2005/10/03/ip-proposal/#comments Mon, 03 Oct 2005 09:14:28 +0000 Mark Nelson http://marknelson.us/2005/10/03/ip-proposal/ My modest proposal could make the U.S. the first country to make a sensible adjustment of Intellectual Property laws to deal with realities in the 21st century.]]>

Dr. Dobb's Portal October, 2005 Article on DDJ site

How important is music in your life? Chances are good that the answer ranges from somewhat to very. People like music. I don't think Science can tell us why yet, but the relationship between people and their music goes way beyond normal affection.

How many hours a day does the average cube dweller spend listening to music? If you are a knowledge worker of any ilk, and you don't have to use your telephone, odds are you listen to music eight hours a day or more. And odds are this is very important to your ability to make it through the day.

How many times have you or a friend of yours latched on to a catchy new song, only to have it playing over and over in your head for days on end? Depending on whether you actually like the song, this either sours your mood or puts a little extra spring in your step. Judging by what I see in my humble domicile, I'd have to say the net effect on total happiness in the world is very, very, positive.

A Serious Problem

Given the importance that music plays in our lives, I think we have a serious problem. The industry that controls the flow of music into our brains is having trouble adjusting to the new world we live in. Instead of making it easier and cheaper to listen to music, they're letting new technologies make it more expensive and difficult.

As an example, if I want to purchase digital music for the much-loved iPod, I pay a purchase price close to that I would pay for CD versions of the same music. (Assuming a $12 price at your favorite big box store.) But for that price, I am stuck with a reduced fidelity image that is locked down to a limited number of computers and other devices. And I am limited to playing these songs on a single vendor's portable player. And worse yet, should the vagaries of hard drive reliability play out the wrong way, a careless consumer can lose their entire digital investment via a single misfortune.

This is all wrong. Being a good American, I always want to see market forces solve our problems, but in this case market forces are not working. Instead of making things better for us, they are making things worse.

As a result, I'm calling on our nation's leaders to take a drastic step--nationalize the music industry.

My Proposal - The Details

My proposal is simple, economical, workable, and benefits both listeners and musicians. Let me spell it out for you and I think you will join me in my wish to see this become reality.

Step 1 is for the government to set up a music archive. By law, all copyrighted music must be published to this archive. The Library of Congress is well suited to management of this task, and really, all they need is a giant server farm, a high-speed Internet connection, and a copy of MySQL.

An important point to note about this publishing system is that publishing data to the Library of Congress music library would be a simple task. Any artist who can master the use of a web browser will now be able to publish their music by simply pushing a few buttons. There will no longer be a daunting distribution system standing between artists and listeners--and the majority of the revenue will no longer be siphoned off to what amounts to a parasitical business.

Step 2 is the development of a payment system. This is simplicity itself. Artists will be paid based on bandwidth, and nothing more. I can download 128Kbps MP3 versions of the latest Brittany Spears CD to my iPod, and the pop idol will get a small payment. If I download the 5.1 CD quality image to burn to a DVD, she'll get a bit more. If I stream it to my PC over and over, she'll get even more. The accounting is simple, and doesn't require any technological advances to implement.

Where does the money come from? That's where Step 3 comes in. Every paycheck issued in the US from now until the end of time is going to have a $5 monthly deduction to fund music. Everyone pays, and everyone benefits. With no record companies in the way, that $5 is going to go straight into the artists pockets, and I predict that the vast majority of performers will see an increase in their cash flow.

The final step: We must compensate the poor record companies who are going to lose this source of revenue. Basically, we buy them out using a discounted cash-flow analysis that properly reimburses them for the business they are giving up. For a company that does nothing but sell records, this should approximate their current stock price. And since the record companies tell us that business sucks, they ought to be happy to get out of it. And don't worry about the money, this is going to cost a lot less than the war in Iraq or Katrina rebuilding, and it's going to make everyone a lot happier.

The Results

The upshot? You now get to listen to any music you want. Promotion and distribution of music changes radically. And the U.S. becomes the first country to make a sensible adjustment of Intellectual Property laws to deal with realities in the 21st century.

Incidentally, this may sound like satire in the tradition of Jonathan Swift's Modest Proposal, but I assure you, this is a plan worth considering. Yes, it is a big change in the way we do things, but I believe it is the appropriate response to dealing with a system that is too clumsy and dumb to deal with new realities. We didn't ask the market to build our Interstate Highway system, we don't ask the market to defend our borders, and I don't really want to the market to control my music. It's time for a change.

]]> http://marknelson.us/2005/10/03/ip-proposal/feed/ 5 http://marknelson.us/2004/02/09/video-jukebox/ http://marknelson.us/2004/02/09/video-jukebox/#comments Mon, 09 Feb 2004 12:41:09 +0000 Mark Nelson http://marknelson.us/2004/02/09/video-jukebox/ Byte.comFebruary 9, 2004 Article on Byte.com I've been an ardent proponent of server-based digital media for a long time. Way back in 1999, technology finally reached the point where I could migrate my music collection from CDs to a server, with the result chronicled here. Once that was accomplished, I naturally begin work on moving [...]]]>

Byte.com February 9, 2004 Article on Byte.com

I've been an ardent proponent of server-based digital media for a long time. Way back in 1999, technology finally reached the point where I could migrate my music collection from CDs to a server, with the result chronicled here. Once that was accomplished, I naturally begin work on moving my video collection to a home server as well.

In the millenial year 2000, I found that creating a video server was impossible on nearly every front. My home network couldn't handle the bandwidth. I didn't have decent client devices to play stored video. And my server didn't have the storage capacity to hold my entire video catalog.

I'm happy to report that at the outset of 2004, all of these problems have vanished, and it's now completely feasible to store your entire video collection on a home server. This article is going to tell you how to do this, step by step, in a simple and easy fashion.

In order, you'll learn how to rip your DVD content to your hard drive, convert that content to the super-efficient DivX format, and how to play those videos using off-the-shelf players, over either a wired or wireless network.

The methods I'm describing here aren't necessarily for everyone. This article intends to provide a simple and straightforward solution for folks who want the benefits of a jukebox, but don't want to invest days and weeks learning how to use complex tools. You might want more. Check the disclaimer to see if you should take the path less traveled.

Grabbing the Content

Those DVDs stacked up in your entertainment center have digitized video stored in an encrypted MPEG-2 format. The actual size of the movie varies depending on length, screen size, and compression settings, but most feature films clock in somewhere between 3 and 5 Gigabytes.

No doubt your Windows PC is able to see those files without any trouble. If you insert a standard title into your PC's DVD drive, you'll see something like the listing shown here.

Directory listing of Erin Brockovich

By convention, the MPEG-2 video is stored in VOB files in the VIDEO_TS folder. Everything else on this particular DVD is software and other fluff for PC users. For the purposes of the home server, we're only interested in the video content.

Grabbing that data off of a DVD is a bit problematic. Yes, your PC can easily read those files, but unfortunately the MPEG-2 content has been encrypted so that it can only be played through a licensed piece of hardware.

Fortunately, there are a few excellent programs that are able to bypass this protection, allowing you to decrypt the data and copy it to your PC simultaneously. The program I recommend for this purpose is called DVD Decrypter. It's a completely free piece of software for Win32 PCs, and you can download it here.

A couple of notes about DVD Decrypter. First, if your PC is in the Win9X train, you will also need to download ASPI drivers for your DVD-ROM. (You can get these from the Adaptec ASPI download site.) Second, it is illegal to distribute this program in the United States! Yes, that's right, it is not legal to sell software that accesses the DVD content that you paid good money for. A more detailed discussion of this can be found in the DMCA sidebar.

Ripping your DVD

Installing DVD Decrypter is a straight-ahead process, and I won't go into any of the details here. Once you have the program installed, start it up, and select the Mode menu option. Set the mode to the IFO setting. This is the most straightforward way to pull the correct VOB files from the DVD.

Selecting IFO mode in DVD Decrypter

The real excitement comes when you are ready to rip the content from your first CD. After you start DVD Decrypter, place your DVD in your DVD drive. Depending on the packaging of your DVD, you may be asked to install software that ships with the DVD - just say no! This software won't help you in the ripping process.


DVD software installer

DVD Decrypter should automatically detect the presence of the DVD in your drive. If it doesn't, you can select the appropriate drive using the drop-down box labeled Source. After it finds the DVD content, DVD Decrypter looks through the various programs on the DVD, and selects the longest one. There are rare occasions when this not the actual movie, but 99 times out of 100 this is what you should select. (I'll tell you how to verify that you got the correct program later in this section.)


DVD Decrypter selects the longest program

At this point, I usually click on the Destination icon and change the destination directory to the location I prefer. All that's left after that is to click the bottom icon showing the DVD to Hard Drive picture, and the ripping process starts.

Ripping the disk is really not much more than just copying the files from your DVD drive to your hard drive, so the process will go pretty quickly, especially if you have a nice high-speed DVD-ROM. The figure below shows what things look like while the extraction is in process. If you love details, be sure to turn on the log window so you can get the works.

The DVD Rip in progress

The optional log window

Verifying Your Rip

Once DVD Decrypter finishes its work, you will have a nice collection of files in your destination directory. You should have one IFO file, a collection of VOB files that are each up to 1 GByte in size, and a text file. A sample of that is shown here.

The results of the rip process

DVD Decrypter created the text file to describe what's in the various streams you've captured. The IFO file and the VOB files are decrypted copies of what was on the DVD. Before we move on to the encoding phase, you'll want to verify that these VOB files do actually contain the contents of the movie as you expect to see it.

To do this, I like to use another great piece of free software, the VLC media player from Videolan.org. Follow the links from Videolan.org home page, download and execute the install package, and you're in business. Once you've done that, just start the player, and one by one drag the VOB files from their target directory right onto the VLC control panel. I usually do a quick check to make sure that I've got a series of VOB files that start at the beginning of the movie, end with the credits, and appear to have some reasonable content in the middle. A typical view of the process is shown below.

The VLC Media Player in use

]]> http://marknelson.us/2004/02/09/video-jukebox/feed/ 5