Oddly, every time I offer this to someone who claims to have solved the problem, they decline!

- Mark

it is done :) tested and it works great, unfortunately commercialization is the only practical option here. I’m am not posting this to gloat rather to tell you guys that the answer is out there staring you in the face, you just need to apply more imagination than math and think out of the box.

Thanks and Good Luck to all

As far as I know it started as a way to silence those claiming fantastic algorithms that do the “Impossible.”

In it’s long life it has been a point of humor for some. A source of ridicule for others.

I may be the only one still working on it. I seem to be the only one updating the blog somewhat regularly.

Speaking of that.

UPDATE:

I have been successful in generating a 16-gigabyte dataset from a 10kbyte program.
I studied the dataset for a few weeks and have decided on the initial dimensions and layout of this Matrix.

I am currently indexing the Million Digit file. That will take some time the way I am doing it so I have time to ready other aspects of the encoder.
In a few posts back I wrote about learning and designing hash functions. I also wrote about being sidetracked with new maths for me to check out.
Well it has come full circle and that effort with the hash functions is again the focus. Good thing I did all that work already. This hashing effort will be more fun than work I believe.
If I can hash some of the indexing then compression is possible.
Wishful thinking suggests storing a byte in 6.5 bits.
What is really true is until I have the indexed data I cannot test any hashing.
It won’t be long now but still it will take a while.

I thought to stop in and add to the blog.

Good luck Challenge people!

Ernst, here are some useful comments for your work on challenge

>> 617 and 673 is 415241

617 and 673 is 1290, not 415241

>> … but since we are all interested in prime pairs …

log(a * b) = log(a) + log(b)

Let million-digit-file = x * y, then
log(x) + log(y) = log(x * y) = log(mdf) =~ 3320000

>> Hint: Well try recursive modulus.

My hint would be: Well try math!

Pardon Tobbi those are the factors of 415241 as in 617 times 673 = 415241.

I can understand your confusion and I promise to not try and trick you again.

Be Well!

I see you found the challenge interesting!

cool!

But hiding data is not in the spirit of the challenge for sure!

Give it some time and maybe you will find a unique solution!

Update :

I’m working on bit-pattern matching using a constructed data-set. Also I am learning about OpenMP and parallel processing along with this effort.
So win or lose there are benefits for trying.

Ernst

It makes sense, the rules of the game are to actually compress the data, not shuffle it around to unseen places. It took me all of one hour to write the code of what I posted above, and that’s just the simplest way to use the filesystem to hide data.

It’s amazing how recent some of these claims for incredible compression are, and that there’s still people who buy into it. Oh well, they may be wasting their time, but it’s at least slightly amusing to read about it.

One of the reasons that I have the rather vague comment at the end is because I don’t want to get in a never-ending contest with people who find clever ways to hide their extra data. I can’t anticipate every possible way to do this in advance, and I don’t really have any interest in doing so.

For example, let’s say I limit things to one file. Well, someone will come along and create a directory structure with 400,000 entries, and just one file.

Then I say no, it all has to go in a single directory. Then they create a file name with 400,000 characters in the name.

Really, who cares? This is not interesting and I don’t want to deal with it.

And so far, this has worked okay, I haven’t really had anyone suggest that they have met the challenge while staying within the rules.

- Mark

From what I read, storing data in the filenames is not allowed. I read before that someone “beat” a $5000 compression challenge in this way before, so I can see what’s trying to be avoided. (notice “beat” in quotes)

The only loophole I see is this: would enumerating files be considered storing data in them? I mean something like: 001.file, 002.file, 003.file, 004.file etc.

Because just the sheer presence of these files provides information that otherwise does not exist. For example if a certain bit pattern that repeats in the original data is found, then you could take that bit pattern out. Normally, you’d have to store at which locations you took the pattern out of, so it can be re-inserted, and this would quickly add up to more info than you saved.

But, if instead you split the file each time you encountered this bit pattern and stored these pieces of the original file in enumerated sequential files, you wouldn’t have to store the information of where to re-insert the bit pattern. The concatination would simply be 001.file + bit pattern + 002.file + bit pattern + … etc.

The question is only to find such a bit pattern. This isn’t difficult:
The file is 415,241 bytes. A single byte can have 256 separate states (meaning there are 256 unique bytes)

By the pigeonhole principle then, just under 415,000 of those bytes have to be a repeat of a previous byte that’s already been seen.
In one extreme, each one of the 256 bytes would be repeated about 1,600 times, and in the other extreme, 1 of these bytes would be repeated the whole 415,241 times.

Since if only one byte repeated, this data wouldn’t be very random, the case here is probably closer to the first one, where the most any single byte is repeated is ~1,600 times.

So, we find this most-often repeating byte, and split the data each time we encounter it, storing each new chunk in an enumerated file.

The “decompressor” should simply store the value of this byte, and then concatenate these files together, each time inserting this byte between two files.

So, basically, just by having enumerated files, we should save about 1.6kb (in the worst case scenario). The source code for the concatination program could potentially be written in under 1.6kb – it is rather simple.

This process can also be repeated with nibbles (4bits) and 2bit patterns that repeat – there should be even more of those repeating.

But, even if this works, this is NOT quite in the spirit of the challenge, and isn’t compressing the data, so much as just hiding the information elsewhere.

To be clear, I am indeed a programmer by profession, but I have NOT programmed this. I found this article today at work and the idea occurred to me when I read some of the comments later.

I am NOT CLAIMING to be able to compress this (or any other) random data – only that there’s a potential loophole in the rules (if Mark allows enumerated files). I am also well aware of the quite solid proof of the counting argument of why no method can compress all files.

If I do have a free time, I’ll try to write a program to run this sort of data-hiding scheme, to see if it would really end up being smaller.

>> 617 and 673 is 415241

617 and 673 is 1290, not 415241

>> … but since we are all interested in prime pairs …

log(a * b) = log(a) + log(b)

Let million-digit-file = x * y, then
log(x) + log(y) = log(x * y) = log(mdf) =~ 3320000

>> Hint: Well try recursive modulus.

My hint would be: Well try math!