Photo by Bob Brown

One of the oldest rules in print journalism is to get the reader’s attention with the lead, and to make your whole point in the first paragraph. Here goes: The storage of plaintext or encrypted passwords by any company that does business with the public is an act of stupidity. An act of stupidity so dangerous that it needs to be made illegal. Yes, we need federal laws banning the storage of passwords on more or less all IT systems in the world. The recent break-in of the Gawker user database makes this point more clearly than anything I can say, but that won’t stop me from trying.

Best Practices

Long ago, in the pre-PC days, I was a neophyte learning my way around the UNIX system my employer used for software development. It didn’t take long before I bumped into one of the more interesting files on the system: /etc/passwd. To my young eyes, it appeared that this file contained the encrypted passwords for all the users in the company. All I had to do was get my hands on the system software that managed logins, and I could quickly print out a complete list of credentials. Awesome!

Soon enough I learned my lesson. One of those bearded UNIX gurus was kind enough to take me aside and point out the obvious: /etc/passwd doesn’t contain encrypted passwords. It contains hashed passwords. Hashed and salted, in fact. Because of this, the original designers of UNIX were able to do what seemed like hubris to me: leave the password file with a mode value of 644 – available for anyone on the system to read.

Even though this was a best practice almost forty years ago, the Gawker debacle shows us that some people just don’t learn. Media empire Gakwer, host of dozens of popular web sites, had their internal database hacked. After much of the data was posted to public web sites, the truth came out. Gawker had over 1.25 million passwords stored in their database. Encrypted passwords, which they felt were quite safe. Safe, that is, until they showed up on the Pirate Bay’s lists of torrents.

Gawker was stupid – that much is obvious. Anything that is encrypted can be decrypted. That’s the nature of the algorithm, and that is why the passwords were stored that way. And if a password can be decrypted, you are just one security breach away from a bad actor having a plaintext password for every user on your system. If your passwords are hashed and salted, the bad actor can still get that list – but it should take at least a few decades.

But this kind of stupidity has the power to do much more damage than letting a script kiddy post comments on Lifehacker using my name. In today’s web, the average citizen who creates an account on a Gawker property such as Lifehacker, Gizmodo, or Fleshbot uses a password that is identical to the one they use on Amazon, eBay, PayPal, and their bank. So it’s pretty obvious what a black hat can do with that list of email address/password combinations – the economic damage can be stupendous.

That’s why we need to make the storage of encrypted passwords illegal.

Yes, Illegal

If it was just a matter of education, we could attack this problem in a sensible way. But the truth is, when it comes to security, the average IT person is an idiot, and the average user is an idiot. Just for example, my hosting service, Dreamhost.com, stores user passwords in their database. I’ve argued with them to no end about how stupid this is. But they’re content – the passwords are encrypted, and hardly anyone has access to them. They’ll happily go on in this mode until a disgruntled employee mails out the list, or the inevitable security breach leads to a mini-Gawker episode. And this is a company hosting close to a million domains. Just imagine how many startups out there have your credentials stored with no more protection than an XORed string in a database, and nothing but some caffeine-fueled PHP code protecting you from a SQL injection hack.

No, I think it’s time to acknowledge that my credentials belong to me, and they need the legal protection that my other properties enjoy. We need to to develop a new FIPS regulation regarding the storage of passwords, and then enact it as law – with hefty penalties. A Gawker-size breach should result in an instant fine on the order of tens of millions of dollars.

It’s Clear This is Needed

Like millions of people, I received the Gawker email shortly after passwords were posted:

Gawker Media to markn
12/13/10
This weekend we discovered that Gawker Media’s servers were compromised, resulting in a security breach at Lifehacker, Gizmodo, Gawker, Jezebel, io9, Jalopnik, Kotaku, Deadspin, and Fleshbot. As a result, the user name and password associated with your comment account were released on the internet. If you’re a commenter on any of our sites, you probably have several questions.
…

The site had a link to a FAQ which answered questions about the breach. The most important question thing to note about this FAQ is that months after the breakin, we still see a very bad entry:

11) What are you doing to ensure this doesn’t happen in the future?
We’re bringing in an independent security firm to improve security across our entire infrastructure. Additionally, we will continue to work with independent auditors to ensure we maintain a reliable level of security, as well as the processes necessary to ensure we maintain a safe environment for our commenters.

There is really only one good answer to this question: it should have read “We will never store passwords on our systems again – either in clear text, or encrypted. As this episode demonstrates, storing passwords is a worst practice. We were stupid once, but we are going to show the world that we have the capacity to learn. From now on, our passwords will be hashed and salted using the strongest possible algorithms available on our back end.”

What You Can Do

The obvious thing you need to do is to not use the same password on every site – this is a no-brainer. If you currently use fluffy99 as your password on Amazon, PayPal, eBay, and Poker Stars, you are vulnerable. Just make a small change in your algorithm. Instead of using fluffly99, simply append the first letter of the site to the password: Amazon gets fluffy99a, PayPal gets fluffy99p, etc. Use some easy-to-remember variation on this theme and you are immediately protected against the first wave of automatic attacks following a breach.

The second thing is to start shaming the morons. From time to time, go through a password recovery exercise on your favorite sites. If they offer to send you an email with a copy of your password, or to show it to you after you answer some security questions, you are dealing with Gawker-league stupidity. Call it out, publicly, loudly, and make sure you file a support case on it.

And finally, if you work in an organization that holds sway over Internet policy, take this call for legal action seriously. If you’re part of the EFF, or CERT, or the FCC, start pushing for this legislation – it’s the right thing.

Afterward

Calling for federal legislation in a blog post is clearly troll bait. For starters, the Internet user base has a fairly large number of people who lust for a global adoption of Anarcho-capitalism. In their view, any new government institution is a step in in the wrong direction. They will weigh in on this idea with some vigor, arguing that the problem can easily be handled with existing tort law. Additionally, they will point out that smart people (like them) are already immune to this problem, because they carry USB keys with a list of randomly generated 32 character passwords for every web site they use.

There will also be a large number of people who claim structural obstacles with the exclusive use of hashed passwords – the example being my Dreamhost captors.

Countering all of the objections they will raise in this post runs the risk of turning it into a manifesto, and that’s not my goal – I don’t get paid by the word. I will address the arguments as they come in. The structural and procedural arguments will all be wrong, and easily addressed. The philosophical arguments are more difficult, because winners and losers in those arguments are generally selected based on personal opinion, only slightly tempered with the facts. (And much of political argument starts off with name-calling and deteriorates from there.) But it’s safe to say I don’t agree with the idea that there are no good laws.

Maybe Gawker will be the last million-user password breach. I hope so. But somehow I doubt it. It’s kind of fun to think of the same type of breach happening at, say, Mint.com or Intuit. Now that will be a call to action!

Update

Since writing this, I received some useful clarification from a source who has worked with Gawker. Like most people, I was basing my information on Gawker’s security practices from their own statement:

Passwords in our database are encrypted (i.e., not stored in plain text),

As it turns out, this statement false. According to a post from Gawker, their user passwords were hashed using crypt(3), the same algorithm used by the bearded UNIX gurus I talked to over 30 years ago.

This means that Gawker was using a hashed, salted password. Unfortunately, they were using an algorithm that was considered to weak for any reasonable security as long as 20 years ago. Dictionary-based cracking programs on modern computers can break crypt(3) passwords with astonishing speed. That is apparently what happened to Gawker.

This doesn’t change the gist of the problem much – Gawker was still horribly negligent with user passwords. For whatever reasons they chose to use a security scheme that simply doesn’t pass muster. This could clearly have been prevented with a FIPS standard based on modern technology.

A Visit From Mr. Language Policeman

As an aside, is it reasonable to claim that Gawker misspoke when they said their passwords were encyrypted? It’s perhaps a fine point, but the Wikipedia definition of encryption says:

In cryptography, encryption is the process of transforming information (referred to as plaintext) using an algorithm (called cipher) to make it unreadable to anyone except those possessing special knowledge, usually referred to as a key.

Clearly, the process of hashing passwords using crypt(3) does not meet this definition. You will sometimes hear people refer to this as one way encryption, but this usage doesn’t turn hashing into encryption.

Email-based reminder services are a handy tool for keeping track of pending activities. I recently gave a try to a service called FollowUpThen. It works by simply sending an email to a specific address in the followupthen.com domain, which then creates a reminder. Examples on their web page include:

• 3hours@followupthen.com
• tomorrow@followupthen.com
• 11am@followupthen.com

Last night I sent an email to the address noontomorrow@followupthen.com. The format seemed in line with the other examples given. Fortunately, I took the time to read the fine print in the response, shown below:

One would think that the FollowUpThen programming team would have a unit test in place that flags reminders scheduled for forty years in the past.

Any theories on how this date was arrived at without an error message?

Microsoft has added a new keyword to C# as part of the 4.0 release earlier this year. Objects that are typed as dynamic bypass normal static type checking, allowing C# to have the flexibility of other scripting languages.

This is all well and good, but the headline writers of the blogosphere have taken a decided wrong turn with their naming of this feature:

C# 4.0: Dynamic Programming

C# 4.0, Dynamic Programming and JSON

Dynamic Programming Using C# 4.0 and Microsoft Visual Studio 2010

C# 4.0: Dynamic Programming

Note the misuse of the term Dynamic Programming. Everyone who takes an introductory algorithms course learns that the term Dynamic Programming has been in use for over fifty years, and refers to a method for solving problems by decomposition. It’s a useful technique that I’ve covered here in the past, and any skilled programmer should be familiar with it.

No, it’s not the end of the world, but people who are writing about Computer Science really ought to know something about Computer Science, don’t you think?

The New York Times has an interesting article today examining the curious fact that certain types of terrorist organizations have an unusually high ratio of engineers among their members. An interesting point to study, no doubt, but what caught my eye was this little blunder:

William A. Wulf, a former president of the National Academy of Engineering, is, no surprise, no fan of the Gambetta-Hertog theory. “If you have a million coin flips,” he says, “it’s almost certain that somewhere in those coin flips there will be 20 heads in a row.”

This numerical gaffe is a prime example of innumeracy, a favorite topic of mine, and it is doubly bad. First, the New York Times with its old-school print-format hubris regarding fact checking should not have let this slip by unnoticed. Second, the fact that the speaker is not just an engineer, but president of our National Academy, adds insult to injury.

Probability 101

The Wikipedia says that Numeracy is the ability to reason with numbers and other mathematical concepts. In today’s world, it should be considered as important as literacy. So let’s try doing some thinking about this problem.

What should first catch your eye in this is the meaning behind “20 heads in a row.” As a programmer, you are instinctively aware that 2 to the 20th power is roughly one million. This means that the chances of flipping a true coin and having it land heads up 20 times in a row is inded roughly one in a million. Does this mean that flipping a coin a million times renders such a streak “almost certain?” Of course not.

If the chance of flipping a single head is one in two, and I flip a coin two times, am I almost certain to see one head? No. If the chances of two heads in a row is one in four, am I almost certain to see a streak of two if I flip four times? Still we intuitively answer no. It seems likely, but nowhere near a certainty. So the task in front of us is to scale this equation up and see if it changes in character as we near one million.

Pinning it Down

Determining how likely this streak is requires a frequent ruse we employ in probability. Instead of calculating the probability directly, we determine out how likely it is not to occur, then subtract that value from one.

We know that the chance of the coin flip happening in the first 20 flips is 1/2^20. We’ll call this number p. Now let’s imagine a sequence of a million coin flips. The chance of a streak of 20 heads not starting at position one is 1-p. The chance of it not happening in the sequence of coins starting at position 2 is likewise 1-p. The same probability is true for every sequence of flips from position 1 to position 999,981, the last possible start of a streak of twenty.

The chances of not seeing a coin flip in every one of those positions is found by multiplying each of their values, leading to the rather unwieldy formula (1-p)^999,981. Unwieldy, perhaps, but your scientific calculator will quickly tell you it resolves to roughly 0.39. So the chances of seeing 20 heads in a row after a million coin flips is more or less 61%. Hardly “almost certain”.

Finding Almost Certain

I’d like to think that “almost certain” is somewhere in the neighborhood of 99%. I’ll leave the calculation as an exercise for the reader, but if your calculator has a log button you will be able to determine that you will need almost five million coin tosses to achieve near certainty. And when you think about it (using your beloved numeracy) that number seems a lot more realistic. Something that has a one in a million chance of occuring would seem to be only somewhat likely to occur in a million tries. Give me five million and it’s a sure thing.

Ironically, the Gray Lady just ran an ode to fact checking a few weeks ago. Apparently that department is short on people with any sort of mathematical fluency. Perhaps they should think about hiring an engineer or two?

The Pigeonhole Principle, also referred to as the Counting Theorem, is a handy tool for mathematicians, and naturally, computer programmers.

The loose version of this principle says “After placing n pigeons into m compartments, if n is greater than m, you will find that some compartment must contain more than one pigeon.”

Seems obvious, and perhaps it is, but at least in the world of data compression it must be trotted out from time to time in order to bludgeon dreams back to reality.

Impossible Compression

A common dream for the novice is the creation of a compressor that will reduce the size of all files. (Often touted as the ability to compress “random” data.) For example, Dr. Constant Wong of Recursiveware has been polishing his technique for compressing random data since 2003. And the USENET newsgroup comp.compression always has at least one thread dedicated to thrashing a new and eager theorist with a (flawed) idea.

The Pigeonhole Principle quickly puts this idea to rest. We know that if a file is of length n bits, there are 2ⁿ possible input files. If a compressor can reduce the size of every file, the number of possible output files is 2ⁿ-1. The Pigeonhole Principle tells us that the output of at least two file compressions have to be identical. And since they are identical, the decompressor cannot create two different output files.

And More

The Wikipedia has another nice example of the principle in use. Imagine that you have a party with n people attending. At random, people shake hands with one another as they mill about. At the end of the night, we check the number of unique individuals each person has shaken with. What are the odds that two people will have shaken hands with the same number of people?

The answer is of course that there will always be two people who have shaken the same number of hands. There are n-1 pigeonholes, and n pigeons, QED.

Don’t Go There

If you ever find yourself spiraling down the rabbit hole of impossible data compression, I urge you to grab the life jacket of the Pigeon Principle before you are lost. It will save you a lot of pointless effort, plus clue you in to the fact that there are two IBM employees with the same number of hairs on their head.

This isn’t the first time I’ve complained about innumeracy, and I’m sure it won’t be the last. Just to get off on the right foot, let me give the definition of the word from thesite innumeracy.com:

A term meant to convey a person’s inability to make sense of the numbers that run their lives. Innumeracy was coined by cognitive scientist Douglas R Hofstadter in one of his Metamagical Thema columns for Scientific American in the early nineteen eighties. Later that decade mathematician John Allen Paulos published the book Innumeracy. In it he includes the notion of chance as well to that of numbers.

The example of innumeracy found in this post is somewhat more interesting than most, because it comes from a source that really should know better: Discover Magazine.

In the July 2008 of Discover Magazine, I was reading an article titled Ocean Acidification: A Global Case of Osteoporosis, and saw this quote:

One such event occurred 55 million years ago at the so-called Paleocene-Eocene Thermal Maximum (PETM), when 4.5 million tons of greenhouse gases were released into the atmosphere.

Now, we’re supposedly talking about an event in which so much greenhouse gas was emitted that extraordinary climate change occurred. Is 4.5 million tons really a lot? Checking the Wikipedia article on greenhouse gas emissions gave this interesting quote:

According to a preliminary estimate by the Netherlands Environmental Assessment Agency, the largest national producer of CO2 emissions since 2006 has been China with an estimated annual production of about 6200 megatonnes. China is followed by the United States with about 5,800 megatonnes.

So the US and China produce 12 million billion tons of CO2 in a year, while the PETM produced 4.5 million tons. And the PETM was a major event that we are blowing away every year, year after year? Something is not right here.

Check the Source

Fortunately, Discover references the source of their information right on the web page, and quick check of the paper shows what the actual number is supposed to be:

Atmospheric temperatures inferred from surface ocean (references in Zachos et al. [2005]) and terrestrial (e.g., Wing et al. [2005]) proxies warmed by 5 – 9° C globally during the PETM. Warming was closely associated with the release of between ~1500 and 4500 Gt of carbon to the ocean and atmosphere, resulting in large but poorly quantified increases in atmospheric CO2 levels [Zachos et al., 2005].

Okay, so in my book, Gt means Gigatonne, and regardless of whether we are using English or Metric units, that’s going to be measured in billions, not millions of tons.

How did Discover take this number from the paper and mangle it by three orders of magnitude? We’ll never know. But avoid innumeracy, try to be aware of just how big the earth is, and realize that what seem like big numbers don’t always do it justice.

American Express is so excited about having me as a customer that they were willing to pay me $5 to take part in a survey:

American Express Needs YOUR Feedback!

Dear American Express Blue Cardmember:

American Express would like your feedback. We would like you to participate in a survey about your Blue card from American Express. Your participation will provide us with valuable feedback and help us tailor card benefits to better meet your needs.

As a token of our appreciation, you will receive $5 from American Express*. Please note that this survey will be running for a limited period of time. To increase your chances of receiving the honorarium, please complete the survey at your earliest convenience.

The web address for the survey is shown below. To begin the survey, simply double-click on the address to go directly to the questionnaire. However, if you are unable to double-click on the address, please copy and paste the text below into your browser’s address bar.

Well, I need five bucks, and a little bit of checking gave me moderate insurance that this was really from American Express, not a phisher. The survey was being outsourced to Confirmit, a real company, and they don’t seem to be on any malware site lists. Furthermore, as things went on, there was an enormous amount of content in the survey and seemingly no payoff for phishers, so it seems unlikely that this is a scam.

However, right off the bat there was some cause for concern. Double-click on the address? How many browsers do you use that need a double-click before they follow a link? If Confirmit is a real company, they clearly didn’t assign their top copywriter to this project.

The real fun started when I actually started the survey. The classic “make a good first impression” rule that your mother taught you is just as true for web pages as it is for anything else. And the first page of the American Express survey was the equivalent of showing up for your first date with a big gravy blotch on your tie:

Yes, that’s right, the first thing I see is some inner workings they’ve inadvertently exposed. No doubt the URL I clicked was supposed to preload a survey question and zip me right past this. This question was undoubtedly in use to stage their testing of the survey, and was supposed to be removed in the published version. There’s a lesson in that.

From Bad to Worse

The rest of the survey only served to further tarnish my impressions of Amex’s IT outsourcing choice. After making it through a few innocuous questions, another page I wasn’t suppose to see popped up:

Apparently I was going to be be seeing ths particular error quite a few times:

Survey Hell

Finally I seem to have made my way through the setup questions. The progress bar showed 50%, and I detected that I was entering the first ring of survey hell. This is where the survey designer starts trying to milk you for a ton of information by repeatedly varying some scenario, then asking you a bunch of detailed questions about it. (This is sheer idiocy on their part. Once they start trying to find the correct adjectives for how a particular ad makes me feel about the product/company, it’s over.)

In this case AmEx had a list of perhaps 15 benefits that their card offers. They started tossing them up in various combinations on the screen, and in each case asking me on a scale of 1-10 how that made me feel about the card. They then tried to quantify the results by asking me how much more I would put on my card each month given that benefit. (Again, totally moronic. No data retrieved this way could possibly have any value.)

This would all be great if their page actually worked. In this case, I’ve told them that I put a tidy $5,000 on the card each month, and they want to see how much I’m going to bump that up if someone comes to my house to give me a back rub. (No joke!) Just to see what would happen, I put down a lower number, and got this nifty error message:

It turns out the only way I could get past this page was by putting in 1, yes $1. But that’s okay, because at this point there’s obviously no reason to cooperate with the tragically broken survey.

Unending Hell

AmEx then applied the coup de grâce by inserting a fiendish logic error in the survey. I started looping through various permutations of three possible card benefits, over and over, for each one picking a number from 1-10 and then assigning it a cash value. I realized that the progress bar at the top of the page was actually moving back to 50% after each answer! In other words, I was stuck in the survey forever. They had a bug that prevented them from exiting the loop.

Nicely done, Amex!

So I bailed. In theory I can come back in a day or two and finish. Perhaps they will have fixed things up by then. More likely they will have fired Confirmit. Will I get my $5? It doesn’t seem likely. AmEx put the fix in here as well, by breaking the payment page. As you can see here, I’ve entered data in all the fields, but the survey software refuses to accept it, somehow thinking I’ve left something blank. This is a big PR win for Amex, putting people through a 20 minute broken survey, then refusing to pay!

Moral

The moral of the story is obvious. If you’re going to send an email out to thousands of customers, all of whom are most likely uninformed civilians, it would be a good idea to thoroughly test your product first. The fact that Confirmit clearly didn’t do that is a nail in their coffin, considering this is their livelihood. The fact that AmEx hired this joke of a company doesn’t say much for them, either.

Priceless Web Design:

• Designing a survey for your customer base? $10,000
• Deploying the survey and passing out rewards? $25,000
• Making yourself look like an incompetent pack of rubes for the whole Web to see? Priceless.

Just before boarding a flight out of Anchorage last week I saw a familiar warning in an unfamiliar place:

From this we know two things: The Anchorage airport runs their flight listings on Windows PCs, and they aren’t using any anti-virus software. Homeland Security, what do you think?

The world is full of people making miraculous claims for video compression. For example, the hucksters at Euclid Discoveries have been leading their investors on with tales of incredible video compression for years now. No doubt that somebody, someday, might make a quantum leap in video compression, but for the past 25 years it’s been just hard work with slow and steady progress.

A lot of the people who have made these claims in the past are either crackpots or criminals, so I found myself kind of annoyed when Robert X. Cringely popped up with a miracle compression claim in his June 24, 2004 column. Robert isn’t crazy, and he’s not a criminal. He spends a lot of time talking to smart people and trying to synthesize their ideas into frameworks that he can deal with, and he’s pretty good at it.

In this case, he asserted (without references) that the bandwidth of the optic nerve was in the 100Kbps range – thereby implying that we ought to be able to cook up a compression scheme that uses a model of the human eye to provide full bandwidth video at 100 Kbps. It’s not the first time Bob has wandered down this alluring path, he got hyped up in 2002 about Foveating Codecs. (Seen any deployed recently?)

Anyway, the point of all this Bob background is just to point to this paper in which some Penn researchers assert that the bandwidth of the human optic nerve is around 10 Mbps – a bit different from the Cringely numbers. What’s interesting is this means that present coding schemes that provide decent representation of video are operating in the ballpark of that number, which might mean the whole thing kind of adds up.

It’s easy for people to get caught up in a whirl of excitement about impossible video compression, and it happens all the time – just ask the hundreds of innocents who have busted open their piggy banks to buy shares in Euclid. I’ll close with a quote from Bob showing how compelling the idea can be – compelling enough to make people overlook the basic facts:

Oh, and there’s another little side benefit — the end of blindness.

And it is doable, the algorithms have already been worked out and are running today in Matlab.

That was two years ago, so I guess the algorithms are still stuck in their Matlab jail.