When I finally wrapped up the mathematical modeling of the k-consecutive heads coin toss problem I had a concise formula:

To get the exact result I needed, I used the BigInteger and BigDecimal Java classes to handle those big numbers, and had to wait about an hour to get the result. As it turns out, there is a much faster and easier way to do things

Simplifying the Computation

To solve this problem in a more reasonable matter, take a look at the part of the equation where all the real work is done:

So at first blush I seem to be stuck with calculating the top and bottom parts of the equation independently. But a little inspection of the equation shows that I can actually calculate p in an iterative fashion, starting with n=1, and working my way up to the desired result.

Calculating the Fibonacci Number

The first step in the calculation is writing the code to calculate k-step Fibonacci numbers. Recall that the k-step Fibonacci number n is found by summing the values of the previous k numbers – a recursive definition. The base cases for all values of k are: Fib_k(n) is 1 when n is 1, and 0 when n is less than 1.

To calculate the value of Fib_k in an iterative fashion, I make use of the the handy C++ container deque<T>. To calculate a k-step Fibonacci number, I make use of a deque<double> of size k that contains the previous k Fibonacci numbers. With the data structure set up like that, calculating the next Fibonacci number is simple:

  q.push_front( accumulate( q.begin(), q.end(), 0 ) );
  q.pop_back();

Note that after this calculation, q[ 0 ] contains the next new Fibonacci number, and the previous k-1 numbers are still stored in sequence in the deque. The call to pop_back() discards Fib_k(n-k), which is not going to be needed for the next calculation.

Add the code to initialize the deque, and a loop, and we can print out Fib_k(n) for values 1 through n:

	deque<double> q(k, 0);
	q[0] = 1.0;
	for ( int i = 1 ; i < n ; i++ ) {
            cout << q[ 0 ] << " ";
	    q.push_front( accumulate( q.begin(), q.end(), 0.0 ) );
            q.pop_back();
	}
	cout << q[ 0 ] << endl;

Using the familiar value of k=2, and n=20, I get the following output:

1 1 2 3 5 8 13 21 34 55 89 144 233 377 610 987 1597 2584 4181 6765

The deque is perfect for this class - it lets us simulate a rolling window through the Fibonacci sequence by pushing new values on one end and popping aged ones off the other end. All this while providing constant time access to indexed elements. (Admittedly this code could be written just as efficiently with a list<double>.)

Divide By Two

This is a nice simple calculation, but it only gets me halfway to the goal. I'm calculating the top part of the fraction that computes p, but I need to divide by the bottom value as well. As it turns out, I can accomplish this in a way that allows me to iterate to the next value: before I add the values of the previous Fibonacci numbers to q[0], I just divide all of them by two. The resulting loop ends up looking like this:

	deque<double> q(k, 0);
	q[0] = 1.0;
	q[1] = 1.0;
	for ( int i = 1 ; i < n ; i++ ) {
            q.push_front(0.0);
            for ( int j = 1 ; j <= k ; j++ ) {
                q[ j ] /= 2.0;
                q[ 0 ] += q[ j ];
            }
            q.pop_back();
	}
	cout << q[ 0 ] << endl;

Note that I've slightly changed the initialization - the deque<double> is initialized with Fib_k(2) - this is because the top half of the fraction uses Fib_k(n+2), and the bottom uses 2ⁿ - the value of n for the Fibonacci calculation always has to be two ahead of the exponent of 2.

This calculation allows me to calculate the probabilities for large numbers of tosses in a matter of seconds, instead of the hour-plus it took using BigInteger. Using this code I was able to get the final word on just when we could be certain that a certain number of coin tosses would produce a run of 20 heads. If we want to be modestly certain, with 90% probability I can say that 4,828,842 coin tosses will produce a run of 20 heads.

If my life depended on it, and I wanted to get out to five nines, I can say that 24,144,137 coin tosses will produce a run of 20 heads with 99.999% probability. The output of kfib.cpp is shown here:

There is a 90% probability of seeing 20 consecutive heads in 4828842 tosses of a fair coin
There is a 99% probability of seeing 20 consecutive heads in 9657666 tosses of a fair coin
There is a 99.9% probability of seeing 20 consecutive heads in 14486490 tosses of a fair coin
There is a 99.99% probability of seeing 20 consecutive heads in 19315313 tosses of a fair coin
There is a 99.999% probability of seeing 20 consecutive heads in 24144137 tosses of a fair coin

And that's the last thing I am going to say about it.

A couple of weeks back I took issue with a statement in the New York Times Magazine:

If you have a million coin flips, it’s almost certain that somewhere in those coin flips there will be 20 heads in a row.

My blog post took the Times to task for printing what seemed to be an obviously bogus statement. Since a run of 20 heads is roughly a one-in-a-million occurence, a basic feel for probability should tell you that trying to do this a million times is not going to be a certainty – fairly far from it.

Naturally, I thought to back up my argument with some hard facts, and I came up with a calculation that showed that the chances of this happening were actually around 60%. Unfortunately, I made a pretty basic mistake in calculating the probability, as was demonstrated to me by correspondent Andy Langowitz. My intuition about the likelihood of success was on the money, but my estimate was a bit high.

The story of how I corrected my error in order to get the correct number, around 37.9%, is a good lesson in careful examination of probability rules, with a small detour into the land of the bignum.

Independent Probabilities

When trying to develop a formula like this one, it is often easier to work from the inverse. Instead of calculating how likely it is for 20 heads to occur at any point in the sequence of a million tosses, I thought to instead calculate the probability that it wouldn’t happen. We can then take that number and subtract it from one to get the desired result.

The chance of n heads in a row occurring is 1/2ⁿ, so the inverse probability is (2ⁿ-1)/2ⁿ. If we multiply that probability once for all 999,981 possible occurences of a streak of 20 heads, it seemed to me that I would be in business. Doing this is a simple enough calculation, and the result was the 60% figure. That figure felt like it was in the ballpark to me, and I left it that.

Mr. Langowitz, however, was smart enough to actually test the theory on a smaller set of numbers. Let’s apply this theory to find out how likely we are to throw two heads in a row in four tries.

The chance of two heads in a row is 1/4, so my formula would give a result of 1 – (3/4)³, or a result of 37/64 – a little better than 50% chance of it occuring. But probability is nothing but the art of enumerating and counting, and we can do just that to check the result. The 16 equally possible outcome of four tosses are:

HHHH HHHT HHTH HHTT
HTHH HTHT HTTH HTTT
THHH THHT THTH THTT
TTHH TTHT TTTH TTTT

Look through those outcomes and see how many have two consecutive heads. It turns out to be exactly 8, meaning that my calculation of 37/64 is just flat wrong.

Locating the Mistake

My mistake is a pretty common one among probability neophytes. I assumed that probability at each step in the sequence was identical, because the sequences were completely independent. It’s easy enough to think that if you don’t examine problem carefully.

What actually happens is that when we examine the possibility of an unsuccessful run of heads at toss i, we slightly bias the outcome at toss i+1. A demonstration will show exactly how this happens.

For the sample problem of a sequence of two heads out of four tosses, we can first examine the chance of a negative outcome starting at toss 1. There are just four possible outcomes that have two tosses starting at position 1:

HH HT TH TT

And only one of these tosses yielded two heads in a row, so the probability of not seeing two heads after two tosses is 3/4.

But now when we look at the sequence of tosses starting at position two, we have to throw out the outcomes where we had two heads at toss one – we’ve already seen two heads, so we can’t continue flipping coins in those outcomes. So our universe of possible outcomes is now a bit different:

HTH HTT
THH THT
TTH TTT

Instead of eight outcomes, we have six. And if we look at the first toss seen in position two, instead of having an even distribution of heads and tails, you can see that sample is biased: only two have a head in position two, while four have tails. So the chances of not seeing two heads starting at position two increases to 5/6. Note that this change in probability occurs because we have selected only those outcomes without a streak of two heads at position one.

Likewise, when we look at the possible outcomes for streaks starting at position three, we get a different probability again. Because we have to throw out one sequence in the previous test, the universe of possible outcomes is now limited to:

HTHH HTHT
HTTH HTTT
THTH THTT
TTHH TTHT
TTTH TTTT

So now we have just ten possible outcomes, and two of those will produce the desired outcome, meaning the probability has changed to 4/5.

So what is the probability of all three possible positions not containing a streak? That would be (3/4)*(5/6)*(4/5) which reduces nicely to 1/2, the correct answer.

Finding the General Solution

So let’s generalize the question at hand: what is the probability of seeing k consecutive heads when a fair coin is tossed n times? The previous section showed that we can work it out by hand for small numbers of tosses, but it should be clear that if are going to toss a coin a million times, the universe of possible outcomes is going to get unwieldy. We need a general description of the problem in order to solve it for any values of k and n.

For many probability problems, finding a solution is simply a way of figuring out how to count things, and coin tosses indeed appear to be just such a problem. Let’s try to see if we can count the number of times a sequence of k heads will appear at a given toss.

To start to work out the solution to the problem, I will set k to a value of three – in other words, we will be trying to see what is the probability of seeing three consecutive heads is at toss i, given that there have not been three heads at an earlier toss. To calculate the probability, we need to know two things. First, we need to know all the possible outcomes in our universe of samples at toss i. In the previous section, with a value of k=2, we saw the the number of outcomes for tosses 1, 2, 3, and 4, was 2, 4, 6, and 10.

After determining the number of outcomes, we then need to determine how many of those outcomes were successes. If we defined success as being the number of outcomes in which a k heads in a row appear at position i, the values from the previous section would have been 0, 1, 1, 2.

Counting the Successes

I’ll start with the more difficult problem: counting the number of times k heads appear at toss i. To start with, we have the degenerate cases where i is less than k. In all of those tosses, we know that the number of successful outcomes is going to be zero, because there have not been enough tosses to achieve success yet.

If we work our way forwards with the example of k=3, our first four tosses end up giving us three sets of outcomes:

H T

HHTH HHTT
HTHH HTHT
HTTH HTTT
THHH THHT
THTH THTT
TTHH TTHT
TTTH TTTT

Note that when we get to toss 3, there is just one successful outcome. Likewise, in toss 4, there is just one successful outcome.

It may not be immediately obvious, but we can in fact always tell how many successes we will achieve at toss i+k after we have enumerated all the possible outcomes at toss i. The number is defined as the number of outcomes at toss i that end in a tail.

The logic behind this is straightforward: in order to have a success at position i+k, we need to generate a sequence of k heads, starting at toss i+1. If we have an outcome that currently ends in a tail, it will generate 2^k outcomes in the next k tosses, and precisely one and only one of these will have k consecutive heads. None of these outcomes will result in an sequence of k heads before toss i+k, because they currently terminate in a tail, so all of the outcomes generated from that outcome at position i will be included in the outcomes seen at toss i+k.

Likewise, none of the outcomes at position i that currently end in a head are going to be able to contribute to a success at toss i+k. Any streak of k heads that follows a terminating head at toss i will result in a run of k heads before we reach toss i+k.

Looking at our outcomes for k=3, we can see that at toss 1 we have one outcome ending in a tail, so at toss 4 we will have one success. At toss 2 we have two outcomes ending in a tail, so at toss five we will have two successes. And we have the special case of toss 0 – we have one sequence starting at toss 0 that generates a sequence of k heads at toss k. Although there were no tails tossed at position 0, any sequence that starts there doesn’t have any preceding heads tosses either, so it is as if there was a single outcome at toss 0 with a value of tails.

So each toss that ends in a tail acts as the root of a successful outcome at a future position. This is good information, but in order to turn this in to a formula we need to be able to compute the number of outcomes ending in tails at toss i – we don’t want to have to enumerate all the outcomes in order to get there. I’ll refer to these special outcomes as anchors, as they form the anchor of a future outcome.

Counting the Anchors

The number of anchor outcomes at each position starts out as a nice number while i is less than or equal to k: 2^i. But after toss k, successful outcomes start being removed from the sample set and the formula no longer holds. For k=3, the anchor count starting at toss1 is: 1, 2, 4, 7, 13, 24.

It turns out that the anchor at position i does more than just generate a success at toss i+k. It is also responsible for generating new anchor outcomes at tosses i+1, i+2, …, i+k-1.

Looking at an example for k=3 should clarify this. Our lone anchor outcome at toss 1 is the sequence T. We know that this anchor will create a new successful outcome at toss 4: THHH. But it also creates new anchors at all intermediate tosses: TT, THT, and THHT.

This observation holds true for the generation of all new anchors, and with a little work we can turn this into a usable recurrence. If each anchor at toss i is going to create a new anchor at tosses i+1 through i+k-1, we can calculate the number of anchors at toss i using this formula:

anchors(i)=anchors(i-1) + anchors(i-2).

Yes, that is the Fibonacci sequence. For values of k higher than 2, the formula is the n-step Fibonacci sequence. The well-known Fibonacci sequence recursively adds the previous two values to get the current value. The n-step Fibonacci sequence adds the previous n values in order to get the current value. (For the rest of this article, the n-step Fibonacci number will be referred to as fib_n(i).)

In the standard Fibonacci recurrence definition, we define a base value of fib(1) = 1, and fib(x) = 0 for all x less than 1. Our anchor count is skewed by one, since our base value at toss 0 is 1. As a result, the the anchor count at toss i is equal to fib_k(i+1). And from our observation of the link between anchors and successful outcomes, we can then observe that the number of successful outcomes at toss i is equal to fib_k(i+1-k).

Counting the Outcomes

Knowing the number of successful outcomes at toss i only gets us halfway to knowing the actual probability of seeing a sequence of k heads at that point. To get the full probability, we need to know the number of outcomes as well.

Fortunately, this calculation is nearly trivial. In the previous section we saw that the number of anchors at toss i is equal to fib_k(i+1). Each anchor at toss 1 and greater is simply a sequence of tosses that ends in a tail. And for every sequence ending in a tail, there is a corresponding sequence that is identical except for one key change: it ends in a head instead of a tail. So the number of outcomes at toss i is twice the number of anchors, or 2*fib_k(i+1).

The Absence of a Streak

Now that we can compute the probability of seeing a streak of k heads at toss i, we need to do just a bit more work to see the what the odds are of seeing that streak at any time in a sequence of n tosses. To get there, we need one more piece of information: the probability of not seeing a streak of k heads at toss i – in other words, the chances of a negative outcome.

It’s pretty easy to calculate that number – simply subtract the number of successful outcomes from the total number of outcomes. The sample sequences for values of k corresponding to 2, 3, and 4 are shown here:

In the third line of the figure I simply break out the value first term of the equation into its two component parts. Now, in the next line, I expand the value of fib_k(i+1) into its component parts, using the definition of the Fibonacci sequence. I keep that grouped in parentheses to make it clear that the new terms are the result of the expansion. Just as an example, if we were doing this expansion for a value of k equal to 3, we would expand fib₃(i+1) into the sum of fib₃(i), fib₃(i-1), and fib₃(i-2), using the identity of the n-step Fibonacci sequence.

Note that in line 3, the last term of the expanded Fibonacci sequence in parentheses is canceled by the last term overall, which was the number of successful outcomes. When we remove those two elements from the equation, line 5 has simplified into the identity of fib_k(i+2). So we now have a reasonable number we can use to determine the count of failures at a given toss.

Put it All Together

With all these formulas in hand, we have the tools to determine the final probability we’ve been working towards: the probability that a sequence of k heads will appear in a sequence of n tosses. To do this, we calculate the cumulative probability that the event does not occur, and subtract that value from 1. The result will be the answer to the question.

The first equation below shows the general approach we take to this problem – multiplying the probability of failure at each toss. Once we plug in the actual formula for the failure count at each point, and the formula for the total number of outcomes, we see that most of the terms cancel out. We are left with fib_k(n+2) on top, and 2ⁿ on the bottom. And that is the final formula that provides an answer to the question.

Unfortunately, my calculator is not really up to this. Even a desktop calculator that could handle arbitrary precision math won’t normally have a fib_k button.

If I had a copy of Mathematica, and knew how to use it, I think I could solve this with just a few lines of input. But I don’t, so I coded up a short solution in Java.

Java has two classes that enable me to solve this problem in relatively easy fashion: java.math.BigInteger and java.math.BigDecimal. BigInteger performs simple integer calculations with arbitrary precision, and BigDecimal supports floating point math.

My simple app, contained in Flipper.java, uses BigInteger to calculate fib_k(n+2) and 2ⁿ, then uses BigDecimal to divide the two numbers and subtract the result from 1. Despite fact that the two intermediate results are over 300,000 digits each, the program ran in a very reasonable amount of time, less than an hour. (Optimization of this program would be a very interesting exercise.)

The output of the program is shown here:

fib(20,1000002) = 614579313398524367786474463596 (301000 digits elided)
2^1000000 = 99006562292958982506979236163 (301000 digits elided)
Div = 0.379253961388950068663971868 (999980 digits elided)

So at last, we know the correct result. If you flip a coin a million times, you have a 38% chance of seeing 20 heads in a row. A long way from the certainty claimed by the New York Times, and a bit off from my initial 60% value.

Postscript

Working out the details of this problem was a very enjoyable piece of math. When I first started in on the problem, I hoped I would be able to find a reference that simply told me how to calculate the number, but I had no luck. As I worked through the problem, I ran into the existence of the n-step Fibonacci numbers, which I had never heard of. Once I found the reference to them on the Wolfram Alpha page (linked above), I saw that the page had a terse note describing this problem, but with no details. With luck the next person trying to understand this problem will be able to make sense of it by reading this page.

Newcomers to the world of data compression often stumble on this old idea in hopes of creating a novel and powerful algorithm. In a nutshell, the idea is to create an enumerative coding system that uses combinatorial numbering to identify a message, in hopes of providing a more compact representation . Unfortunately, these schemes always fail, for reasons that I’ll lay out in this article.

Combinations

To design a combinatorial algorithm that will compress files, you can think of the file as a series of integers. Since most of the files that you use are streams of bytes, consider each file to be a sequence of integers with values from 0 to 255.

If you go back to your first classes on probability and statistics, you might remember the definition of a combination. A combination is simply a way of selecting a number of things from a larger set. When you are trying to compress a file of bytes, the natural size of this set is 256.

Probability theory tells us that we can count the number of combinations of a given size using a pretty simple formula. If a set has n elements and we are choosing k at a time, the number of possible combinations is given by the formula n!/k!*(n-k)!. This number is also known as the binomial coefficient.

Just for a simple example, the number of different ways you can select three bytes out of a set of 256 is 2,763,520. In general, with a large set, most combinations are going to generate very large numbers. The exceptions will be for values of k that are either very small or very close to the size of the set.

Combinations are well ordered, so any instance of the three bytes has a specific number between 0 and 2,763,519. We can call this the combinatorial rank. This means I can identify any three byte sequence by a combination number.

Assuming all combinations are equal, we can use an optimal arithmetic coder to encode this number in lg(2,763,510) bits, roughly 21.4. That’s interesting, because the three bytes actually take up 24 bits, so maybe there is some savings to be found here.

The First Problem

Knowing the combinatorial rank is good, but it won’t let you reconstruct a compressed file on its own. The combinatorial rank gives you the set of bytes in the file, but doesn’t tell you the order of those bytes. If there are k bytes, they can be ordered in k! different permutations. So to fully describe the file, you need to encode the combination rank and the permutation number.

Encoding the permutation for your 3 byte file is going to take 2.58 bits, calculated as lg(3!). This makes the total needed to encode your three byte file 23.98 bits. Admittedly not a lot of savings, but it’s also non-zero.

Let’s look at the number of bits needed to encode a 20 byte message. The number of combinations of this length are roughly 2.8*10^29, which will take 97.8 bits to encode. 20! is roughly 2.43*10^18, which will take 61.1 bits to encode. The total comes out to 158.9 bits. Since we’re encoding 160 bits of information, there is clearly a greater savings.

As the message size increases, the savings start to grow. At a message length of 50 we save 7.5 bytes, at 75 bytes we save 16 bytes per message. The trend looks good. By the time you get to a message length of 100 bytes, you’re saving 32 bits per message – a compression of 4% for doing nothing but recoding!

The Second Problem

The second problem you encounter in the combinatorial system is that, by definition, a combination is composed of unique elements. So if you are compressing a three byte file, you can’t have any duplicate bytes. Is this a problem?

Your inclination is to hope not. You know that every compression scheme only works on a subset of files, so perhaps the combinatorial scheme can be developed to work on segments of files with no duplicates.

How likely are you to find a duplicate in a file of three bytes? You can start by enumerating the total number of files of that length: 256^3. And you know how many files there are with no duplicates: the combinatorial number times the number of permutations. So it’s a simple matter to calculate the probability that a message of length k has no duplicated bytes. The value will be n!/(n-k)!*256^k.

For a value of 3, we see that the probability of no duplicate bytes is .988 – this means you can compress almost every file by a fraction of a bit.

You’d like to think that you can look at pretty long stretches of data and expect a low probability of duplicates, but unfortunately you run into the Birthday Paradox. In the birthday paradox, you’re asked a question something like this: in a room of 23 people, what are the chances that two people share a common birthday? For most people, the answer, 50% or so, is non-intuitive.

Likewise, it means that a file with 100 bytes and no duplicates is such a rarity that it might as well never appear – the chances are less than one in a billion.

Facing the Music

You can see the problem here. We can compress very short sequences using a combinatorial system, but the savings are very small. Even so, we can compress most files. We can compress longer files for greater savings, but very few sequences will prove to be eligible.

It’s actually worse than that. Let’s work out the number on a hypothetical compressor. This compressor will use a combinatorial scheme to compress all files of 10 bytes. The compressor will look at the file, and if it has no duplicates, it will set a flag symbol in the output stream to be true, followed by the combinatorial number, followed by the permutation.

If the 10 byte file has duplicates, the compressor will generate a flag symbol of false, followed by the uncompressed data.

If this scheme gives us some savings, we can scale it up to operate on files of any size – we’ll just compress them in 10-byte chunks.

So let’s analyze the result. First, the number of files that will make it past the first test is pretty impressive: 83.695%. Each of these files will be compressed down to 79.743 bits. The remaining 16.305 percent will take exactly 80 bits in the output stream. So the overall size of our output file thus far is going to add up to 79.78519 bytes. Our algorithm is still in the black!

Unfortunately, we also need to account for the cost of the flag message. Using optimal coding, when the flag is true we are going to require .25679 bits. When it is false, optimal coding of the much rarer message will require 2.6 bits. Add in the cost of the flag, and the average output size goes up to a smidgen over 80 bits.

In other words, you lose.

Conclusion

The problem is a familiar one in data compression. Every time you come up with a way to encode a subset of files that saves some space, you find that all your savings are lost when you try to encode the files that aren’t part of the subset. Even using a single bit to flag special files as being incompressible is enough to wipe out your savings. It is the definition of Whac-A-Mole.

With combinatorial coding you will find that same rule to hold true as for all forms of data compression: It isn’t going to be a universal compressor that can reduce every file in size. The only reason it will be useful is if you have an input set of files that all have a common characteristic: a preponderance of streams where duplicates are rare. And odds are, this set of files will probably be compressible using some more reasonable algorithm.

The New York Times has an interesting article today examining the curious fact that certain types of terrorist organizations have an unusually high ratio of engineers among their members. An interesting point to study, no doubt, but what caught my eye was this little blunder:

William A. Wulf, a former president of the National Academy of Engineering, is, no surprise, no fan of the Gambetta-Hertog theory. “If you have a million coin flips,” he says, “it’s almost certain that somewhere in those coin flips there will be 20 heads in a row.”

This numerical gaffe is a prime example of innumeracy, a favorite topic of mine, and it is doubly bad. First, the New York Times with its old-school print-format hubris regarding fact checking should not have let this slip by unnoticed. Second, the fact that the speaker is not just an engineer, but president of our National Academy, adds insult to injury.

Probability 101

The Wikipedia says that Numeracy is the ability to reason with numbers and other mathematical concepts. In today’s world, it should be considered as important as literacy. So let’s try doing some thinking about this problem.

What should first catch your eye in this is the meaning behind “20 heads in a row.” As a programmer, you are instinctively aware that 2 to the 20th power is roughly one million. This means that the chances of flipping a true coin and having it land heads up 20 times in a row is inded roughly one in a million. Does this mean that flipping a coin a million times renders such a streak “almost certain?” Of course not.

If the chance of flipping a single head is one in two, and I flip a coin two times, am I almost certain to see one head? No. If the chances of two heads in a row is one in four, am I almost certain to see a streak of two if I flip four times? Still we intuitively answer no. It seems likely, but nowhere near a certainty. So the task in front of us is to scale this equation up and see if it changes in character as we near one million.

Pinning it Down

Determining how likely this streak is requires a frequent ruse we employ in probability. Instead of calculating the probability directly, we determine out how likely it is not to occur, then subtract that value from one.

We know that the chance of the coin flip happening in the first 20 flips is 1/2^20. We’ll call this number p. Now let’s imagine a sequence of a million coin flips. The chance of a streak of 20 heads not starting at position one is 1-p. The chance of it not happening in the sequence of coins starting at position 2 is likewise 1-p. The same probability is true for every sequence of flips from position 1 to position 999,981, the last possible start of a streak of twenty.

The chances of not seeing a coin flip in every one of those positions is found by multiplying each of their values, leading to the rather unwieldy formula (1-p)^999,981. Unwieldy, perhaps, but your scientific calculator will quickly tell you it resolves to roughly 0.39. So the chances of seeing 20 heads in a row after a million coin flips is more or less 61%. Hardly “almost certain”.

Finding Almost Certain

I’d like to think that “almost certain” is somewhere in the neighborhood of 99%. I’ll leave the calculation as an exercise for the reader, but if your calculator has a log button you will be able to determine that you will need almost five million coin tosses to achieve near certainty. And when you think about it (using your beloved numeracy) that number seems a lot more realistic. Something that has a one in a million chance of occuring would seem to be only somewhat likely to occur in a million tries. Give me five million and it’s a sure thing.

Ironically, the Gray Lady just ran an ode to fact checking a few weeks ago. Apparently that department is short on people with any sort of mathematical fluency. Perhaps they should think about hiring an engineer or two?

You may have never heard of Vinay Deolalikar, but there is a chance that he may become next year’s Turing Award winner, not to mention an overnight millionaire. It seems that Vinay dropped the news at the start of this week that he had proven that P does not equal NP. In short, this proof means that many problems we suspect are hard to solve are in fact provably hard to solve. Whether his proof succeeds or not, the Interwebs are abuzz with the news.

P versus NP is arguably the biggest unanswered problem in Computer Science, and proving it one way or another will cascade into new results and conclusions in many areas of computing and mathematics. The problem is at least 40 years old, and despite people literally devoting their entire careers to the problem, has yet to be decided. Some researchers suspect it can’t be proved, although this has not been proven either.

When a famous problem like this has been stewing for so long, it is not unusual to see sporadic cries of proof, but these need to be taken with a grain of salt. The first test is to see if the person is a flake with a proof that won’t survive a casual scan. Passing that test, the proof then has to endure the normal peer review process. Proofs that plumb difficult new ground can take years to be accepted.

It appears that Deolalikar has passed the first test – people are seriously studying his proof. However, given its complexity and length, you can expect that general acceptance will not happen overnight. More likely, specific problems will arise, challenges will be published, and Deolalikar will either respond, modify his proof, or retreat for more study. If he indeed has the golden ticket, he will be cashing a million dollar prize from the Clay Mathematics Institute sometime soon.

It’s remarkable news indeed, and perhaps the only way it would have been more remarkable would have been with a proof that P equals NP.

Feel like weighing in? Check out the proof yourself and see what you think.

Back in 1981, fresh out of school, I was awestruck by Steve Wozniak’s program that calculated over 100,000 digits of e on an Apple II. (Anyone who has a scan of his article in the June 1981 issue of Byte, please email me a copy!) Shortly after reading the article, I ported his program to the PDP-11 at my office and duplicated his results, down to the last digit.

These days the stakes are much higher when it comes to calculating the values of constants. Alexander Yee and Shigeru Kondo have just announced the calculation of pi to 5 trillion digits. And oddly enough, this was accomplished on a desktop machine running Windows server, not the Linux cluster I would have expected.

Here are some of key stats:

Operating System

Windows Server 2008 R2 Enterprise x64

Software

y-cruncher

Processor

2 Intel Xeon X5680 @3.33 GHz offering 24 hyperthreaded processors

Disk Space

The computation required roughly 22TB of disk space, and the compressed result takes another 3.8TB

Time

The task took 90 days to complete, and 66 hours to verify

The y-cruncher software package that broke this record also holds records for several other constants, including one trillion digits of e. So in 30 years, more or less, the desktop PC has gone from constant calculations under a million digits to over a trillion digits. That growth rate of 1.7x per year maps pretty well to Moore’s Law, suggesting that we can expect these numbers to continue climbing for at least a few more years.

Kudos to Alexander Yee and Shigeru Kondo on a smashing accomplishment!

This isn’t the first time I’ve complained about innumeracy, and I’m sure it won’t be the last. Just to get off on the right foot, let me give the definition of the word from thesite innumeracy.com:

A term meant to convey a person’s inability to make sense of the numbers that run their lives. Innumeracy was coined by cognitive scientist Douglas R Hofstadter in one of his Metamagical Thema columns for Scientific American in the early nineteen eighties. Later that decade mathematician John Allen Paulos published the book Innumeracy. In it he includes the notion of chance as well to that of numbers.

The example of innumeracy found in this post is somewhat more interesting than most, because it comes from a source that really should know better: Discover Magazine.

In the July 2008 of Discover Magazine, I was reading an article titled Ocean Acidification: A Global Case of Osteoporosis, and saw this quote:

One such event occurred 55 million years ago at the so-called Paleocene-Eocene Thermal Maximum (PETM), when 4.5 million tons of greenhouse gases were released into the atmosphere.

Now, we’re supposedly talking about an event in which so much greenhouse gas was emitted that extraordinary climate change occurred. Is 4.5 million tons really a lot? Checking the Wikipedia article on greenhouse gas emissions gave this interesting quote:

According to a preliminary estimate by the Netherlands Environmental Assessment Agency, the largest national producer of CO2 emissions since 2006 has been China with an estimated annual production of about 6200 megatonnes. China is followed by the United States with about 5,800 megatonnes.

So the US and China produce 12 million billion tons of CO2 in a year, while the PETM produced 4.5 million tons. And the PETM was a major event that we are blowing away every year, year after year? Something is not right here.

Check the Source

Fortunately, Discover references the source of their information right on the web page, and quick check of the paper shows what the actual number is supposed to be:

Atmospheric temperatures inferred from surface ocean (references in Zachos et al. [2005]) and terrestrial (e.g., Wing et al. [2005]) proxies warmed by 5 – 9° C globally during the PETM. Warming was closely associated with the release of between ~1500 and 4500 Gt of carbon to the ocean and atmosphere, resulting in large but poorly quantified increases in atmospheric CO2 levels [Zachos et al., 2005].

Okay, so in my book, Gt means Gigatonne, and regardless of whether we are using English or Metric units, that’s going to be measured in billions, not millions of tons.

How did Discover take this number from the paper and mangle it by three orders of magnitude? We’ll never know. But avoid innumeracy, try to be aware of just how big the earth is, and realize that what seem like big numbers don’t always do it justice.

Probably the scariest thing I have to do at work each day is to walk past Marc Leger’s office. Since Marc’s office is on the way to the bathroom, I have a tough time avoiding it. So several times a day, I have to face the possibility that Marc will accost me at a time when I am really in a hurry to be elsewhere.

Today was not going to be one of those lucky days when I got off the hook. I slipped down the hallway as quietly as I could, but I wasn’t quiet enough. Just as I reached Marc’s office, I heard the familiar demand. “Mark! Mark! Come here, I need help!”

Into the breach

With a sigh of sympathy for my bladder, I pulled up a chair behind Marc’s desk. He was running Turbo Debugger, and had a program similar to that shown in fp1.cpp.

Marc was working on a customer problem. A Greenleaf Database Library user was attempting to insert floating point data into a database, and our library was returning an error code indicating a loss of precision. Marc was in his usual state of consternation for two reasons. First, he hadn’t seen this problem occur before. Second, he couldn’t reproduce the problem when using doubles, just floats.

The smoking gun

The offending code is fairly simple. It converts a C++ double to ASCII representation for storage in a database. After converting and storing the result, the code wants to see if any loss of precision was caused during the conversion process. For example, if I’m using 2 digits after the decimal point, I should get an error if I try to store 3.141. The trailing 1 would be lopped off by the binary to ASCII conversion done by sprintf(). Marc’s test program from Listing 1 produced the following output:

    add_double( 1.12 ) returns success
    add_double( 1.125 ) returns failure
    add_double( d ) returns success
    add_double( f ) returns failure

As expected, when using two digits of precision, 1.12 is added to the database without error. 1.125 fails, because it will be stored in ASCII as 1.12. Using a double that has been assigned the value of 1.12 works properly. But using a float with a value of 1.12 fails!

Seeing double

The thing that was really bothering Marc was the display on his debugger. He was at the end of the add_double() routine, and had all three local variables in his watch window. They all looked perfect. The two floating point numbers had values of 1.12, and the character buffer contained a “1.12″. Even so, the comparison “test == d” had just failed, causing the routine to return an error. Was the compiler somehow at fault? Marc was convinced it was a conspiracy on Borland’s part to make his life hell.

Mark’s problem was fairly simple, once you looked at it from the right perspective. Numbers such as 1.12 can’t be represented exactly in a float, or even in a double. Thus, there are rounding and truncation errors when the conversion takes place. The specific problem in fp1.cpp is the loss of precision found when placing 1.12 in a float. Assigning that value to a double later on doesn’t fix the problem, it only perpetuates it.

In the add_double() routine, the actual value passed using a float is just a little bit off from the desired value of 1.12. But printing the value out to a buffer using just two digits of precision manages to throw out the error. Thus, when atof() converts the buffer back to a double, the error is gone. This means that the input value and the test value are going to differ by a small version.

fp2.cpp is a program that demonstrates how this works. d1 is a double that has been initialized with the value of 1.12. d2 has the same value after being initialized using a float. The output of the program is shown below:

f  = 1.12  1.120000005  dump: 29 5c 8f 3f
d1 = 1.12  1.12         dump: ec 51 b8 1e 85 eb f1 3f
d2 = 1.12  1.120000005  dump: 00 00 00 20 85 eb f1 3f

As you can see, looking at all three values with just 3 digits of precision makes them look identical. However, when you use a greater precision, the representation error of a float rears its ugly head. The binary dump of the second double shows exactly where the problem lies. The lower half of the double contains all zeros, a result of having the smaller float value copied into it. Clearly, when comparing (test == d ) in fp1.cpp, the compiler will notice the difference.

A Happy Camper

Marc wasn’t particularly happy with my explanation. First, it didn’t offer an immediate solution to his problem. And second, he resented my glib advice: “Never use floating point numbers if you expect your program to work.”

The best answer for our customer would have been to keep all floating point data in doubles at all times. Unfortunately, this solution was impractical. Instead, we created a short routine for her that simply rounded off doubles to 2 significant digits, throwing out anything beyond that. Life would have been much easier if we could have convinced her to simply ignore the error return!

My advice to you is the same: avoid floating point anytime it isn’t really necessary. If you do find you need to use it, try to avoid mixing doubles and floats. Stick to the preferred C++ floating format of double for all your data and calculations, and maybe you’ll avoid those moments of hysteria that Marc has to constantly deal with.