Comments on: The Hutter Prize http://marknelson.us/2006/08/24/the-hutter-prize/ Programming, mostly. Tue, 07 Feb 2012 16:05:51 +0000 hourly 1 http://wordpress.org/?v=3.2.1 By: Mark Nelson http://marknelson.us/2006/08/24/the-hutter-prize/comment-page-1/#comment-325169 Mark Nelson Wed, 14 Apr 2010 14:01:27 +0000 /2006/08/24/the-hutter-prize/#comment-325169 @jiahb: I don't know how much this has been studied, but I have a strong feeling that using an order-0 or order-1 model to compress equivalent English and and Chinese texts will yield very similar results. I'd be suprised if there is much to gain by what amounts to tokenizing English text. - Mark @jiahb:

I don’t know how much this has been studied, but I have a strong feeling that using an order-0 or order-1 model to compress equivalent English and and Chinese texts will yield very similar results. I’d be suprised if there is much to gain by what amounts to tokenizing English text.

- Mark

]]> By: jiahb http://marknelson.us/2006/08/24/the-hutter-prize/comment-page-1/#comment-325167 jiahb Wed, 14 Apr 2010 13:56:56 +0000 /2006/08/24/the-hutter-prize/#comment-325167 Translate into Chinese and you guys get probably beyond the Shannon limit. Perhaps Google will be able to do that. 1 Character = 2 Bytes. 1 Latin word = 7 Bytes = 1 Chinese word = 2 Characters = 4 Bytes Use whatever conventional compression routine, then you get it. Of course, this compression may lose information, or may gain more information, depending on how good is your translator. And it's not 100% revertable. Because you are using sample from Wiki, not random stuff, so there are already some internal information, or to say, quantum entanglement. Good luck on y'all! Translate into Chinese and you guys get probably beyond the Shannon limit. Perhaps Google will be able to do that.
1 Character = 2 Bytes.
1 Latin word = 7 Bytes = 1 Chinese word = 2 Characters = 4 Bytes
Use whatever conventional compression routine, then you get it.
Of course, this compression may lose information, or may gain more information, depending on how good is your translator. And it’s not 100% revertable.
Because you are using sample from Wiki, not random stuff, so there are already some internal information, or to say, quantum entanglement. Good luck on y’all!

]]> By: Mark http://marknelson.us/2006/08/24/the-hutter-prize/comment-page-1/#comment-10464 Mark Mon, 19 Mar 2007 15:37:06 +0000 /2006/08/24/the-hutter-prize/#comment-10464 era, i never know whether to bless them or curse them. it bothers me that academic papers are locked up and inaccessible to the general public behind pay-to-read journal sites like ieee eexplore, especially when you consider that we, the taxpayers, pay for 90% of the stuff that is published there. on the other hand, it is copyrighted material, and under current law the authors do have the right to keep it private. meanwhile i suppose i ought to snag a copy... era, i never know whether to bless them or curse them. it bothers me that academic papers are locked up and inaccessible to the general public behind pay-to-read journal sites like ieee eexplore, especially when you consider that we, the taxpayers, pay for 90% of the stuff that is published there.

on the other hand, it is copyrighted material, and under current law the authors do have the right to keep it private.

meanwhile i suppose i ought to snag a copy…

]]> By: era http://marknelson.us/2006/08/24/the-hutter-prize/comment-page-1/#comment-10431 era Mon, 19 Mar 2007 09:15:51 +0000 /2006/08/24/the-hutter-prize/#comment-10431 Looks like somebody at Brown scanned the BSTJ article and put it on-line as a PDF: http://www.cs.brown.edu/courses/cs195-5/extras/shannon-1951.pdf The quality is, erm, shall we say, a good test of human ability to reconstruct information from context. Looks like somebody at Brown scanned the BSTJ article and put it on-line as a PDF: http://www.cs.brown.edu/courses/cs195-5/extras/shannon-1951.pdf

The quality is, erm, shall we say, a good test of human ability to reconstruct information from context.

]]> By: Mark http://marknelson.us/2006/08/24/the-hutter-prize/comment-page-1/#comment-356 Mark Wed, 06 Sep 2006 09:53:08 +0000 /2006/08/24/the-hutter-prize/#comment-356 Looks like the existence of the prize, and perhaps the prize money, are having the desired effect of accelerating the rate of improvement. Looks like the existence of the prize, and perhaps the prize money, are having the desired effect of accelerating the rate of improvement.

]]> By: James Bowery http://marknelson.us/2006/08/24/the-hutter-prize/comment-page-1/#comment-355 James Bowery Wed, 06 Sep 2006 05:02:17 +0000 /2006/08/24/the-hutter-prize/#comment-355 Alexander Ratushnyak's latest (Aug 29) entry , paq8hp3, if confirmed, compresses enwik8 to 1.394 bits per character. This represents nearly a 5% improvement since the contest began whereas historically the improvement has been approximately 3% per year. When Mr. Ratushnyak achieved 4% improvement with paq8hp2, some speculated that all "low hanging fruit" was "gone, gone, gone". For Marcus Hutter's sake I hope there is a reprieve for at least a few months so some other donors can feel comfortable placing their money "at risk" in this purse. However, there is still the potential that someone will soon use a serious language model and push the compression ratio to the top end of Shannon's estimate of human intelligence of 1.3 bits per character. Alexander Ratushnyak’s latest (Aug 29) entry , paq8hp3, if confirmed, compresses enwik8 to 1.394 bits per character. This represents nearly a 5% improvement since the contest began whereas historically the improvement has been approximately 3% per year.

When Mr. Ratushnyak achieved 4% improvement with paq8hp2, some speculated that all “low hanging fruit” was “gone, gone, gone”.

For Marcus Hutter’s sake I hope there is a reprieve for at least a few months so some other donors can feel comfortable placing their money “at risk” in this purse.

However, there is still the potential that someone will soon use a serious language model and push the compression ratio to the top end of Shannon’s estimate of human intelligence of 1.3 bits per character.

]]> By: Mark http://marknelson.us/2006/08/24/the-hutter-prize/comment-page-1/#comment-294 Mark Fri, 25 Aug 2006 21:51:06 +0000 /2006/08/24/the-hutter-prize/#comment-294 Thanks for the correction on the Shannon estimation, Matt. Lacking an online copy of the paper, I checked my copy of <a href="http://www.amazon.com/gp/product/0139119914/102-2559624-1104943?v=glance&n=283155" rel="nofollow">Text Compression</a> by Cleary, Witten, and Bell, and they cite the same upper and lower bounds as you, 0.6 to 1.3. Which is good, because it means that there is still a lot of room to go for Hutter prizewinners. If the 0.6 lower bound were to hold, we could see the 100MB corpus reduced to 7.5 MB, roughly half the size of the best efforts to date. And I am a believer, to get to that point is going to require something that starts to look like intelligence. (And as Turing proposes, if it looks like intelligence...) Thanks for the correction on the Shannon estimation, Matt. Lacking an online copy of the paper, I checked my copy of Text Compression by Cleary, Witten, and Bell, and they cite the same upper and lower bounds as you, 0.6 to 1.3.

Which is good, because it means that there is still a lot of room to go for Hutter prizewinners. If the 0.6 lower bound were to hold, we could see the 100MB corpus reduced to 7.5 MB, roughly half the size of the best efforts to date. And I am a believer, to get to that point is going to require something that starts to look like intelligence. (And as Turing proposes, if it looks like intelligence…)

]]> By: Matt Mahoney http://marknelson.us/2006/08/24/the-hutter-prize/comment-page-1/#comment-293 Matt Mahoney Fri, 25 Aug 2006 21:23:31 +0000 /2006/08/24/the-hutter-prize/#comment-293 It is true today that data compression programs are not very smart, but I think that will have to change to make progress in the Hutter competition. English has a lot of redundancy that is difficult to exploit. For example, it is hard for a program to know that "roses are red" is more likely than "roses are green" or even "roses red are". A compressor that can learn syntax and semantics from text and apply that knowledge to predict future text will do better than one that can't. Also, I believe Shannon estimated the entropy of written English to be about 0.6 to 1.3 bits per character. I got those numbers by eyeballing a graph in Shannon's original paper, "Prediction and Entropy of Printed English", Bell Sys. Tech. J (3) p. 50-64, 1950. The paper is not online as far as I know. I don't know where Wikipedia got the numbers 1.1 to 1.6 bpc. Cover and King (1978) used a text prediction gambling game to estimate the upper bound at 1.3 bpc. It is true today that data compression programs are not very smart, but I think that will have to change to make progress in the Hutter competition. English has a lot of redundancy that is difficult to exploit. For example, it is hard for a program to know that “roses are red” is more likely than “roses are green” or even “roses red are”. A compressor that can learn syntax and semantics from text and apply that knowledge to predict future text will do better than one that can’t.

Also, I believe Shannon estimated the entropy of written English to be about 0.6 to 1.3 bits per character. I got those numbers by eyeballing a graph in Shannon’s original paper, “Prediction and Entropy of Printed English”, Bell Sys. Tech. J (3) p. 50-64, 1950. The paper is not online as far as I know. I don’t know where Wikipedia got the numbers 1.1 to 1.6 bpc. Cover and King (1978) used a text prediction gambling game to estimate the upper bound at 1.3 bpc.

]]>