Comments on: LZW Data Compression http://marknelson.us/1989/10/01/lzw-data-compression/ Programming, mostly. Fri, 12 Mar 2010 02:30:07 +0000 http://wordpress.org/?v=2.2.3 By: funman http://marknelson.us/1989/10/01/lzw-data-compression/#comment-323507 funman Wed, 03 Mar 2010 15:23:34 +0000 http://marknelson.us/1989/10/01/lzw-data-compression/#comment-323507 Hello, I noticed your code crashes on 64 bits architecture: In input_code() left-shifting the bit buffer doesn't clear the top 32 bits, so I just clear non significative bits in the return code [c] --- lzw.c.orig 2010-03-03 16:08:05.000000000 +0100 +++ lzw.c 2010-03-03 16:17:13.000000000 +0100 @@ -13,6 +13,7 @@ #include #include #include +#include #define BITS 12 /* Setting the number of bits to 12, 13*/ #define HASHING_SHIFT (BITS-8) /* or 14 affects several constants. */ @@ -317,6 +318,9 @@ input_bit_count += 8; } return_value=input_bit_buffer >> (32-BITS); +#if ULONG_MAX > (1 [/c] Hello,

I noticed your code crashes on 64 bits architecture:
In input_code() left-shifting the bit buffer doesn't clear the top 32 bits, so I just clear non significative bits in the return code

PLAIN TEXT
C:
  1. --- lzw.c.orig  2010-03-03 16:08:05.000000000 +0100
  2. +++ lzw.c   2010-03-03 16:17:13.000000000 +0100
  3. @@ -13,6 +13,7 @@
  4.  #include
  5.  #include
  6.  #include
  7. +#include
  8.  
  9.  #define BITS 12                   /* Setting the number of bits to 12, 13*/
  10.  #define HASHING_SHIFT (BITS-8)    /* or 14 affects several constants.    */
  11. @@ -317,6 +318,9 @@
  12.      input_bit_count += 8;
  13.    }
  14.    return_value=input_bit_buffer>> (32-BITS);
  15. +#if ULONG_MAX> (1

]]> By: nikos http://marknelson.us/1989/10/01/lzw-data-compression/#comment-321790 nikos Sun, 24 Jan 2010 08:43:30 +0000 http://marknelson.us/1989/10/01/lzw-data-compression/#comment-321790 I wrote a small blog article about this code, including a C++ wrapper to it, that extends it for variable bit length encoding which should help improving the compression ratio regardless of source file size: http://zabkat.com/blog/24Jan10-lzw-compression-code.htm I wrote a small blog article about this code, including a C++ wrapper to it, that extends it for variable bit length encoding which should help improving the compression ratio regardless of source file size:
http://zabkat.com/blog/24Jan10-lzw-compression-code.htm

]]> By: Mark Nelson http://marknelson.us/1989/10/01/lzw-data-compression/#comment-321692 Mark Nelson Thu, 21 Jan 2010 12:47:45 +0000 http://marknelson.us/1989/10/01/lzw-data-compression/#comment-321692 @YH: I don't really think LZW is suitable for a static dictionary. There would be a lot of problems creating it, and what you would end up with would be something a bit different from LZW. - Mark @YH:

I don't really think LZW is suitable for a static dictionary. There would be a lot of problems creating it, and what you would end up with would be something a bit different from LZW.

- Mark

]]> By: YH http://marknelson.us/1989/10/01/lzw-data-compression/#comment-321689 YH Thu, 21 Jan 2010 11:13:21 +0000 http://marknelson.us/1989/10/01/lzw-data-compression/#comment-321689 Dear Mark, I am glad to use your code to compress binary data in PC-based program and it results in great compression power. However I'm having a problem of not enough RAM to decompress it in a small footprint device. Therefore, I'm thinking of hard-coding the tables. From my experiment, it seems that compression and decompression make use of different tables, i.e. prefix_code, append_character. Do you think is it possible to implement fix pattern tables for both compression and decompression? Thank you Regards, YH Dear Mark,

I am glad to use your code to compress binary data in PC-based program and it results in great compression power.
However I'm having a problem of not enough RAM to decompress it in a small footprint device. Therefore, I'm thinking of hard-coding the tables. From my experiment, it seems that compression and decompression make use of different tables, i.e. prefix_code, append_character. Do you think is it possible to implement fix pattern tables for both compression and decompression?
Thank you

Regards,
YH

]]> By: nikos http://marknelson.us/1989/10/01/lzw-data-compression/#comment-321513 nikos Mon, 18 Jan 2010 14:13:33 +0000 http://marknelson.us/1989/10/01/lzw-data-compression/#comment-321513 thanks for this minimal and clear-cut implementation of LZW i've been trying to adapt your code to use variable bit encoding so as to behave good for both small and large files I've hit a but in your code which has been mentioned earlier by "jj" when you define BITS (constant) 10 or 9 to reproduce, take your code and add these 2 lines _after_ the definition of the hash table size [c] #define BITS 10 #define MAX_VALUE (1 << BITS) - 1 [/c] this way the hash table will be defined large enough for BITS=14 (any value is ok) then you redefine BITS to a lower value to demonstrate the bug for certain kinds of input files, the final file test.out is DIFFERENT from the input file by a couple of bytes near the end I am certain that the problem is your output_code() implementation that for 10 bits and lower isn't guaranteed to flush the final integer to the file (despite you writing MAX_VALUE and a zero at the end of your compress function). It is easy to reproduce this bug if you try your code in a few files in your %TEMP% folder, I'm sure one of them will fail the test. Or if you can't find one I can send you a small test file. thanks for this minimal and clear-cut implementation of LZW

i've been trying to adapt your code to use variable bit encoding so as to behave good for both small and large files

I've hit a but in your code which has been mentioned earlier by "jj" when you define BITS (constant) 10 or 9

to reproduce, take your code and add these 2 lines _after_ the definition of the hash table size

PLAIN TEXT
C:
  1. #define BITS 10
  2. #define MAX_VALUE (1 <<BITS) - 1

this way the hash table will be defined large enough for BITS=14 (any value is ok) then you redefine BITS to a lower value to demonstrate the bug

for certain kinds of input files, the final file test.out is DIFFERENT from the input file by a couple of bytes near the end

I am certain that the problem is your output_code() implementation that for 10 bits and lower isn't guaranteed to flush the final integer to the file (despite you writing MAX_VALUE and a zero at the end of your compress function).

It is easy to reproduce this bug if you try your code in a few files in your %TEMP% folder, I'm sure one of them will fail the test. Or if you can't find one I can send you a small test file.

]]> By: Mark Nelson http://marknelson.us/1989/10/01/lzw-data-compression/#comment-320050 Mark Nelson Tue, 15 Dec 2009 16:34:55 +0000 http://marknelson.us/1989/10/01/lzw-data-compression/#comment-320050 Mohan - Sorry the program does not meet your needs. Looks like you have some work to do, better get busy. - Mark Mohan -

Sorry the program does not meet your needs. Looks like you have some work to do, better get busy.

- Mark

]]> By: Mohan http://marknelson.us/1989/10/01/lzw-data-compression/#comment-320049 Mohan Tue, 15 Dec 2009 16:26:45 +0000 http://marknelson.us/1989/10/01/lzw-data-compression/#comment-320049 Thank you Mark. I got the output. But the test.lzw is not in a readable form. Also I kindly request you for the codes of other compression algorithms like the one you have posted for lzw as its more easy to compile and execute. Possibly if I could get lzss, bwt or others it will be very helpful for my dissertation. Thanks for your immediate response. Regards, Mohan Thank you Mark. I got the output. But the test.lzw is not in a readable form. Also I kindly request you for the codes of other compression algorithms like the one you have posted for lzw as its more easy to compile and execute. Possibly if I could get lzss, bwt or others it will be very helpful for my dissertation. Thanks for your immediate response.

Regards,
Mohan

]]> By: Mark Nelson http://marknelson.us/1989/10/01/lzw-data-compression/#comment-320046 Mark Nelson Tue, 15 Dec 2009 16:07:24 +0000 http://marknelson.us/1989/10/01/lzw-data-compression/#comment-320046 Mohan, the executable will either take input and output file names from the command line, or will ask you for them if you leave the command line empty. That is the basic procedure. - Mark Mohan, the executable will either take input and output file names from the command line, or will ask you for them if you leave the command line empty.

That is the basic procedure.

- Mark

]]> By: Mohan http://marknelson.us/1989/10/01/lzw-data-compression/#comment-320044 Mohan Tue, 15 Dec 2009 15:58:38 +0000 http://marknelson.us/1989/10/01/lzw-data-compression/#comment-320044 Dear Mark, I'm doing a dissertation in evaluating the compression algorithms. When I compiled the code in dev C++ its fine. But when I execute it it returns me a file which has got nothing in it. Since I'm not very good at code development I'm not able to understand the problem. I wanted to know whether this code has any data base connectivity and what shoudl I need to do in order to compress a text file. Should I need to change anything in the code. Please let me know some basic procedures behind this execution. Thank you. Regards, Mohan Dear Mark,

I'm doing a dissertation in evaluating the compression algorithms. When I compiled the code in dev C++ its fine. But when I execute it it returns me a file which has got nothing in it. Since I'm not very good at code development I'm not able to understand the problem. I wanted to know whether this code has any data base connectivity and what shoudl I need to do in order to compress a text file. Should I need to change anything in the code. Please let me know some basic procedures behind this execution. Thank you.

Regards,
Mohan

]]> By: Mark Nelson http://marknelson.us/1989/10/01/lzw-data-compression/#comment-318204 Mark Nelson Sun, 08 Nov 2009 01:04:44 +0000 http://marknelson.us/1989/10/01/lzw-data-compression/#comment-318204 @ICodeLikeAGirl: First, your coment about wanting to compress binary and ASCII doesn't make any sense to me. The code as written doesn't distinguish between the type of data - as long as it comes in a stream of bytes it's going to compress properly. If you want to write 12 bit chunks you can certainly just use the code in the article, which is bit-oriented. Or you can save up two 12-bit tokens and write them out as a three-byte sequence. This would be very efficient but you would have to possibly pad the file with one additional token at the end. - Mark @ICodeLikeAGirl:

First, your coment about wanting to compress binary and ASCII doesn't make any sense to me. The code as written doesn't distinguish between the type of data - as long as it comes in a stream of bytes it's going to compress properly.

If you want to write 12 bit chunks you can certainly just use the code in the article, which is bit-oriented. Or you can save up two 12-bit tokens and write them out as a three-byte sequence. This would be very efficient but you would have to possibly pad the file with one additional token at the end.

- Mark

]]>