Comments on: Data Compression with the Burrows-Wheeler Transform

By: wsu123

wsu123 — Thu, 24 Nov 2011 19:07:12 +0000

Can anyone help me find the links for BWT Compression Algorithm code in MATLAB??

By: speedy

speedy — Tue, 06 Sep 2011 12:35:10 +0000

I AM TESTING BWT WITH MTF WHICH USES THE LOCALITY OF THE ALPH-BETICAL SET OF THE LAST COLUMN. THEREFORE bwt-mtf -> bwt-mtf MAY PRODUCE BETTER RESULTS. THEN lzw OR ARITHMETIC OR HUFFMAN IN THE FINAL!

By: Ericke

Ericke — Mon, 08 Aug 2011 04:32:25 +0000

Damn…..

Where the hell is my post about the link that i suggest…

Okay, it doesn’t matter mark…!!!

lol….

By: SPEEDY

SPEEDY — Mon, 18 Jul 2011 17:33:55 +0000

After reading the posts here, I think BWT can be understood in a much simpler way as follows:

1. the first sorted column is embedded in all columns
2. the adjacent relationships in rows are preserved in any permutations of the rows

3 two adjacent columns and an index to the original row are enough to decode. The last and first column are adjacent in the cyclic view.

4. sorting speed is important in the encoding

By: Mark Nelson

Mark Nelson — Mon, 17 Jan 2011 15:18:05 +0000

@Shoaib:

Google is your friend.

By: Shoaib

Shoaib — Sun, 16 Jan 2011 16:17:24 +0000

Or anyone who has doen it please let me know… would be of great help.

By: Shoaib

Shoaib — Sun, 16 Jan 2011 16:16:43 +0000

HI mark could you put up a java implementation?

By: Philips Telaumbanua

Philips Telaumbanua — Mon, 22 Nov 2010 04:09:45 +0000

Thank you Mark for response again!

Yeah, I just realized that the EOF character should be less than every characters in the block. So, it would reduce the comparison. In other words it wouldn’t continue the comparison if x or y = N (x or y = EOF), since EOF character always < other characters so the comparison can be terminated if one of them reached EOF. A good example is when we'd like to transform string "aaaaa$".

I think I've got the point.

Thank you very much Mark!

Ooo,,,I just want to tell you that your articles is very great, good written, neat, and clear. (article + source code = perfect). It has been a huge help.

Thank again Mark…

- Philips Tel

By: Mark Nelson

Mark Nelson — Sun, 21 Nov 2010 20:49:29 +0000

There is a big advantage to placing an EOF character at the end of the string. If the character is not found anywhere else in the input text, it means that you can use normal string compares to sort the strings – so instead of checking to see if the index wraps around at N, you can be guaranteed that the two strings won’t match when one index reaches N.

That speeds things up a lot, and it means you can use a native sort, which might even be faster.

The only disadvantage is that you have to take the EOF characters out of the block when you decompress. No problem.

I’d suggest continued study of the Burrows Wheeler suggestions for improving the sort speed, there is a lot of good in there.
- Mark

By: Philips Telaumbanua

Philips Telaumbanua — Sun, 21 Nov 2010 08:14:42 +0000

Hi Mark,
Thank for fast response!

I'm using Delphi. And my comparePresorted function look like this:

PLAIN TEXT

function comparePresorted(myString, x, y) {
n = length(myString); /* the length of string */
i = 0;
/** continue compare string a byte at a time until
* there's the difference or length of string is reached
*/
while (myString[x] = buffer[myString]) && ( i != n) {
x++;
y++;
if( x>= n ) {
x = 0;
} if( y>= n ) {
y = 0;
}
i++;
} // end while
if( myString[x] [y]){
return TRUE;
} else {
return FALSE;
}
}

That function works by comparing the string a byte at a time based on the starting point that passed in x and y which means "if character in position[x] < character in position[y] would return TRUE, otherwise FALSE";

example : string "BANANA" (characters position from 0,1,3..5)

if we call comparePresorted("BANANA", 2, 4), it would compare strings from position 2,3,4,5,0,1 and 4,5,0,1,2,3.
In this case the comparePresorted will stop the comparison when x = 4 and y = 0 (since position 4 is 'N' and 0 is 'B' and they are difference)

In the original MergeSort algorithm, it compares two numbers to sort the sequence of numbers. It something like this:

PLAIN TEXT

if number1 <number 2 {
// some MergeSort's codes go here
} else {
// some MergeSort';s codes go here
}

Then I modified it by passing the comparePresorted() function in it. So, it would something like this:

PLAIN TEXT

if comparePresorted("BANANA", 2, 4) {
// some MergeSort's codes go here
} else {
// some MergeSort';s codes go here
}

After it, we would get a sorted indicies.

But, as I said it just works for file 100 kb (very slow)

Is it good statistic Mark for BWT?
I don't know how improve it, but based on Burrows' paper, he suggest by using an EOF ($). So, our previous "BANANA" would "BANANA$". But I don't know what is the benefit of EOF '$' in sorting the rotated strings.

Would you tell me mark what's actually the using of EOF $ ???
It's really confused.

Thank in Advance
- Philips Telaumbanua