A Closer Look at Huffman Encoding Using C#
+ +06 Apr 2016 +
+ +
+ Algorithms, C#
+ ++ +
On this page:
+-
+
- How it Works +
- Constructing the Binary Tree +
- Encoding the Data +
- Unpacking the Data +
- Final Thoughts +
- The Code +
Huffman encoding is a compression technique developed by David Huffman and published in his 1952 paper 'A + Method for the Construction of Minimum-Redundancy Codes'. The technique centres on encoding characters as a + list of code words, which are then used to generate a coded version of the original data. Huffman encoding + can be very efficient, and should generally yield compression as low as 20% of the original data size.
+ + +How it Works
+ +The input data is scanned, and a list of characters and their corresponding weight (the number of instances + the character appears in the data). The weights of each character are used to compute the length of the code + words that represent that character: larger weights should yield smaller code words, a key aspect of + compressing the data. Characters are then arranged into a binary tree, where the resulting layout is used to + determine the encoding table. The last step is using the encoding table to create the new file.
+ + +Constructing the Binary Tree
+ +Consider the following text:
++abccccddddddefghijjjjj ++ +
Represented as a list of characters with corresponding weights:
+ ++ +Index Char Weight +0 a 1 +1 b 1 +2 c 4 +3 d 6 +4 e 1 +5 f 1 +6 g 1 +7 h 1 +8 i 1 +9 j 5 ++ + + + +
This list is now turned into a binary tree. A binary tree is a tree where each node may have a maximum of two + child nodes. The left child node is consider the 0 child node, whereas the right child node is considered + the 1 child node. The first two nodes are removed from the list, and a new parent node is constructed. This + parent node's weight is the combined weight of both children. In our example, after the first iteration our + list would look like this:
+ ++ +Index Char Weight +00 null null // used to be a +01 null null // used to be b +02 e 1 +03 f 1 +04 g 1 +05 h 1 +06 i 1 +07 c 4 +08 j 5 +09 d 6 +10 a + b 2 // new parent node ++ + + + +
The next two lowest weight nodes are added to a parent:
+ ++ +Index Char Weight +00 null null // used to be a +01 null null // used to be b +02 null null // used to be e +03 null null // used to be f +04 g 1 +05 h 1 +06 i 1 +07 c 4 +08 j 5 +09 d 6 +10 a + b 2 // new parent node +11 e + f 2 // new parent node ++ + + + +
And so on. At the end of this round of processing, the list looks like this:
+ ++ +Index Char Weight +00 null null // used to be a +01 null null // used to be b +02 null null // used to be e +03 null null // used to be f +04 null null // used to be g +05 null null // used to be h +06 null null // used to be i +07 null null // used to be c +08 null null // used to be j +09 null null // used to be d +10 a + b 2 // new parent node +11 e + f 2 // new parent node +12 g + h 2 // new parent node +13 i + c 5 // new parent node +14 j + d 11 // new parent node ++ + + + +
Null nodes are removed from the list:
++Index Char Weight +0 a + b 2 +1 e + f 2 +2 g + h 2 +3 i + c 5 +4 j + d 11 ++ +
And the next round of processing begins, always taking the smallest weighted nodes and combining them under a + new parent. At the end of the next round of processing, the list looks like this:
+ ++ +Index Char Weight +0 null null // a + b +1 null null // e + f +2 null null // g + h +3 null null // i + c +4 j + d 11 +5 a + b (2), e + f (2) 4 // new parent node +6 g + h (2), i + c (5) 7 // new parent node ++ + + + +
Null nodes are removed from the list:
++Index Char Weight +0 a + b (2), e + f (2) 4 +1 g + h (2), i + c (5) 7 +2 j + d 11 ++ +
At the end of the next round of processing:
+ ++ +Index Char Weight +0 null 4 // a + b (2), e + f (2) +1 null 7 // g + h (2), i + c (5) +2 j + d 11 // new parent node +3 (a + b, e + f) - (4), (g + h, i + c) - (7) 11 // new parent node ++ + + + +
After removing null nodes from the list, we are ready for the final round of processing:
+ ++ +Index Char Weight +0 j + d 11 // new parent node +1 (a + b, e + f) - (4), (g + h, i + c) - (7) 11 // new parent node ++ + + + +
And finally, after all processing, there is only one node left:
+ +
+
+Index Char Weight
+0 (j + d) - 11, (a + b, e + f, g + h, i + c) - (11) 22 // new parent node
+
+
+
+
+
+ This final node forms the root of the tree, and is used to generate a list of binary words that represent + each character. Starting at the root of the tree, following a left child node inserts a 0 into the binary + word, following a right child node inserts a 1 into the binary word. Using this method, the following list + of binary words is created:
+ +
+
+Char Weight BinaryWord Int32Word
+j 5 00 0 // This column is for sorting purposes only
+d 6 01 1
+a 1 1000 8
+b 1 1001 9
+e 1 1010 10
+f 1 1011 11
+g 1 1100 12
+h 1 1101 13
+I 1 1110 14
+c 4 1111 15
+
+
+
+
+
+ Note that nodes without a character value are not present in this list, but are able to be inferred from the + list sorting operations. For the moment, I'll leave you to construct the tree using the information + above.
+ + +Encoding the Data
+ +The next step is to encode the original data using the binary words created from the previous steps. Moving + forward through the data one byte at a time, each character is used to perform a lookup for its + corresponding binary word. This binary word is added into a binary word buffer. When the binary word buffer + reaches sufficient length, 8 values are read off and removed to construct a new byte. Consider the + following:
+ ++ +Index Char BinaryWord ByteBuffer +00 a 1000 // Only 4 bits here, not enough for a new byte +01 b 1001 // Combined with the previous entry, there are enough bits here for a new byte: 10001001 => 137, remove 8 entries from the buffer +02 c 1111 // Start a new byte using these 4 bits +03 c 1111 // 11111111 => 255 +04 c 1111 // Start a new byte +05 c 1111 // 11111111 => 255 +06 d 01 // Start a new byte +07 d 01 // 0101 - the buffer is only half full +08 d 01 // 010101 +09 d 01 // 01010101 => 85 - note that this one byte actually represents 4 chars! +10 d 01 // Start a new byte +11 d 01 // 0101 +12 e 1010 // 01011010 => 90 - this one byte represents 3 chars +13 f 1011 // Start a new byte +14 g 1100 // 10111100 => 188 +15 h 1101 // New byte +16 i 1110 // 11011110 => 222 +17 j 00 // New byte +18 j 00 // 0000 +19 j 00 // 000000 +20 j 00 // 00000000 => 0 +21 j 00 // Start a new byte ++ + + + +
With the processing of the final character, the byte buffer only contains 2 values, which is not enough to + construct a byte, however as the binary word is 00 it is simply converted to 0. Therefore, the final byte to + be constructed is 0. Processing of these 22 characters has yielded a list containing only 9 bytes, about 41% + of the original data size.
+ + +Unpacking the Data
+ +Decompression (in this example at least) is relatively straight forward. The byte list is read, and each byte + converted into a binary word, which is added into a buffer. If the buffer is less than 8 values, then it is + 'fixed' (more on that later). Reading each bit from the binary word indicates which path to take through the + binary tree (constructed above): starting at the root node, follow the left child for a 0 value, and follow + the right for a 1 value. The entire decompression process for the above data:
+ +
+
+Index Byte Characters
+0 137 => 10001001 a, b
+1 255 => 11111111 c, c
+2 255 => 11111111 c, c
+3 85 => 01010101 d, d, d, d
+4 90 => 01011010 d, d, e
+5 188 => 10111100 f, g
+6 222 => 11011110 h, i
+7 0 => 00000000 j, j, j, j
+8 0 => 00 j // This is the last value, so rather than pad it out, we can just go directly to the char value
+
+
+
+
+
+ The output data now contains 22 characters, the data has been successfully unpacked.
+ + +Final Thoughts
+ +Conveniently, this example ignores completely the size of the encoding table, as well as how storage and + retrieval mechanisms, so theoretically 9 bytes is probably as small as we can get. If this compressed data + were to be used outside this example, then the binary tree and code words would also need to be stored along + with the byte list in order to unpack the data at some other time.
+ + +The Code
+ +I've included my somewhat rambling encoding and decoding process, in C# below.
+ + ++ +public class HuffmanCompressor +{ + private HuffmanNode oRootHuffmanNode; + private List<HuffmanNode> oValueHuffmanNodes; + + private List<HuffmanNode> BuildBinaryTree(string Value) + { + List<HuffmanNode> oHuffmanNodes = GetInitialNodeList(); + + // Update node weights + Value.ToList().ForEach(m => oHuffmanNodes[m].Weight++); + + // Select only nodes that have a weight + oHuffmanNodes = oHuffmanNodes + .Where(m => (m.Weight > 0)) + .OrderBy(m => (m.Weight)) + .ThenBy(m => (m.Value)) + .ToList(); + + // Assign parent nodes + oHuffmanNodes = UpdateNodeParents(oHuffmanNodes); + + oRootHuffmanNode = oHuffmanNodes[0]; + oHuffmanNodes.Clear(); + + // Sort nodes into a tree + SortNodes(oRootHuffmanNode, oHuffmanNodes); + + return oHuffmanNodes; + } + + public void Compress(string FileName) + { + FileInfo oFileInfo = new FileInfo(FileName); + + if (oFileInfo.Exists == true) + { + string sFileContents = ""; + + using (StreamReader oStreamReader = new StreamReader(File.OpenRead(FileName))) + { + sFileContents = oStreamReader.ReadToEnd(); + } + + List<HuffmanNode> oHuffmanNodes = BuildBinaryTree(sFileContents); + + // Filter to nodes we care about + oValueHuffmanNodes = oHuffmanNodes + .Where(m => (m.Value.HasValue == true)) + .OrderBy(m => (m.BinaryWord)) // Not really required, for presentation purposes + .ToList(); + + // Construct char to binary word dictionary for quick value to binary word resolution + Dictionary<char, string> oCharToBinaryWordDictionary = new Dictionary<char, string>(); + foreach (HuffmanNode oHuffmanNode in oValueHuffmanNodes) + { + oCharToBinaryWordDictionary.Add(oHuffmanNode.Value.Value, oHuffmanNode.BinaryWord); + } + + StringBuilder oStringBuilder = new StringBuilder(); + List<byte> oByteList = new List<byte>(); + for (int i = 0; i < sFileContents.Length; i++) + { + string sWord = ""; + + // Append the binary word value using the char located at the current file position + oStringBuilder.Append(oCharToBinaryWordDictionary[sFileContents[i]]); + + // Once we have at least 8 chars, we can construct a byte + while (oStringBuilder.Length >= 8) + { + sWord = oStringBuilder.ToString().Substring(0, 8); + + // Remove the word to be constructed from the buffer + oStringBuilder.Remove(0, sWord.Length); + } + + // Convert the word and add it onto the list + if (String.IsNullOrEmpty(sWord) == false) + { + oByteList.Add(Convert.ToByte(sWord, 2)); + } + } + + // If there is anything in the buffer, make sure it is retrieved + if (oStringBuilder.Length > 0) + { + string sWord = oStringBuilder.ToString(); + + // Convert the word and add it onto the list + if (String.IsNullOrEmpty(sWord) == false) + { + oByteList.Add(Convert.ToByte(sWord, 2)); + } + } + + // Write compressed file + string sCompressedFileName = Path.Combine(oFileInfo.Directory.FullName, String.Format("{0}.compressed", oFileInfo.Name.Replace(oFileInfo.Extension, ""))); + if (File.Exists(sCompressedFileName) == true) + { + File.Delete(sCompressedFileName); + } + + using (FileStream oFileStream = File.OpenWrite(sCompressedFileName)) + { + oFileStream.Write(oByteList.ToArray(), 0, oByteList.Count); + } + } + } + + public void Decompress(string FileName) + { + FileInfo oFileInfo = new FileInfo(FileName); + + if (oFileInfo.Exists == true) + { + string sCompressedFileName = String.Format("{0}.compressed", oFileInfo.FullName.Replace(oFileInfo.Extension, "")); + + byte[] oBuffer = null; + using (FileStream oFileStream = File.OpenRead(sCompressedFileName)) + { + oBuffer = new byte[oFileStream.Length]; + oFileStream.Read(oBuffer, 0, oBuffer.Length); + } + + // Find the zero node + HuffmanNode oZeroHuffmanNode = oRootHuffmanNode; + while (oZeroHuffmanNode.Left != null) + { + oZeroHuffmanNode = oZeroHuffmanNode.Left; + } + + // Unpack the file contents + HuffmanNode oCurrentHuffmanNode = null; + StringBuilder oStringBuilder = new StringBuilder(); + + for (int i = 0; i < oBuffer.Length; i++) + { + string sBinaryWord = ""; + byte oByte = oBuffer[i]; + + if (oByte == 0) + { + sBinaryWord = oZeroHuffmanNode.BinaryWord; + } + else + { + sBinaryWord = Convert.ToString(oByte, 2); + } + + if ((sBinaryWord.Length < 8) && (i < (oBuffer.Length - 1))) + { + // Pad binary word out to 8 places + StringBuilder oBinaryStringBuilder = new StringBuilder(sBinaryWord); + while (oBinaryStringBuilder.Length < 8) + { + oBinaryStringBuilder.Insert(0, "0"); + } + + sBinaryWord = oBinaryStringBuilder.ToString(); + } + + // Use the binary word to navigate the tree looking for the value + for (int j = 0; j < sBinaryWord.Length; j++) + { + char cValue = sBinaryWord[j]; + + if (oCurrentHuffmanNode == null) + { + oCurrentHuffmanNode = oRootHuffmanNode; + } + + if (cValue == '0') + { + oCurrentHuffmanNode = oCurrentHuffmanNode.Left; + } + else + { + oCurrentHuffmanNode = oCurrentHuffmanNode.Right; + } + + if ((oCurrentHuffmanNode.Left == null) && (oCurrentHuffmanNode.Right == null)) + { + // No more child nodes to choose from, so this must be a value node + oStringBuilder.Append(oCurrentHuffmanNode.Value.Value); + oCurrentHuffmanNode = null; + } + } + } + + // Write out file + string sUncompressedFileName = Path.Combine(oFileInfo.Directory.FullName, String.Format("{0}.uncompressed", oFileInfo.Name.Replace(oFileInfo.Extension, ""))); + + if (File.Exists(sUncompressedFileName) == true) + { + File.Delete(sUncompressedFileName); + } + + using (StreamWriter oStreamWriter = new StreamWriter(File.OpenWrite(sUncompressedFileName))) + { + oStreamWriter.Write(oStringBuilder.ToString()); + } + } + } + + private static List<HuffmanNode> GetInitialNodeList() + { + List<HuffmanNode> oGetInitialNodeList = new List<HuffmanNode>(); + + for (int i = Char.MinValue; i < Char.MaxValue; i++) + { + oGetInitialNodeList.Add(new HuffmanNode((char)(i))); + } + + return oGetInitialNodeList; + } + + private static void SortNodes(HuffmanNode Node, List<HuffmanNode> Nodes) + { + if (Nodes.Contains(Node) == false) + { + Nodes.Add(Node); + } + + if (Node.Left != null) + { + SortNodes(Node.Left, Nodes); + } + + if (Node.Right != null) + { + SortNodes(Node.Right, Nodes); + } + } + + private static List<HuffmanNode> UpdateNodeParents(List<HuffmanNode> Nodes) + { + // Assign parent nodes + while (Nodes.Count > 1) + { + int iOperations = (Nodes.Count / 2); + for (int iOperation = 0, i = 0, j = 1; iOperation < iOperations; iOperation++, i += 2, j += 2) + { + if (j < Nodes.Count) + { + HuffmanNode oParentHuffmanNode = new HuffmanNode(Nodes[i], Nodes[j]); + Nodes.Add(oParentHuffmanNode); + + Nodes[i] = null; + Nodes[j] = null; + } + } + + // Remove null nodes + Nodes = Nodes + .Where(m => (m != null)) + .OrderBy(m => (m.Weight)) // Choose the lowest weightings first + .ToList(); + } + + return Nodes; + } ++ + + + +
Here's the code for the HuffmanNode type itself:
+ + ++ +public class HuffmanNode +{ + public HuffmanNode() + { + } + + public HuffmanNode(char Value) + { + cValue = Value; + } + + public HuffmanNode(HuffmanNode Left, HuffmanNode Right) + { + oLeft = Left; + oLeft.oParent = this; + oLeft.bIsLeftNode = true; + + oRight = Right; + oRight.oParent = this; + oRight.bIsRightNode = true; + + iWeight = (oLeft.Weight + oRight.Weight); + } + + private string sBinaryWord; + private bool bIsLeftNode; + private bool bIsRightNode; + private HuffmanNode oLeft; + private HuffmanNode oParent; + private HuffmanNode oRight; + private char? cValue; + private int iWeight; + + public string BinaryWord + { + get + { + string sReturnValue = ""; + + if (String.IsNullOrEmpty(sBinaryWord) == true) + { + StringBuilder oStringBuilder = new StringBuilder(); + + HuffmanNode oHuffmanNode = this; + + while (oHuffmanNode != null) + { + if (oHuffmanNode.bIsLeftNode == true) + { + oStringBuilder.Insert(0, "0"); + } + + if (oHuffmanNode.bIsRightNode == true) + { + oStringBuilder.Insert(0, "1"); + } + + oHuffmanNode = oHuffmanNode.oParent; + } + + sReturnValue = oStringBuilder.ToString(); + sBinaryWord = sReturnValue; + } + else + { + sReturnValue = sBinaryWord; + } + + return sReturnValue; + } + } + + public HuffmanNode Left + { + get + { + return oLeft; + } + } + + public HuffmanNode Parent + { + get + { + return oParent; + } + } + + public HuffmanNode Right + { + get + { + return oRight; + } + } + + public char? Value + { + get + { + return cValue; + } + } + + public int Weight + { + get + { + return iWeight; + } + set + { + iWeight = value; + } + } + + public override string ToString() + { + StringBuilder oStringBuilder = new StringBuilder(); + + if (cValue.HasValue == true) + { + oStringBuilder.AppendFormat("'{0}' ({1}) - {2} ({3})", cValue.Value, iWeight, BinaryWord, BinaryWord.BinaryStringToInt32()); + } + else + { + if ((oLeft != null) && (oRight != null)) + { + if ((oLeft.Value.HasValue == true) && (oRight.Value.HasValue == true)) + { + oStringBuilder.AppendFormat("{0} + {1} ({2})", oLeft.Value, oRight.Value, iWeight); + } + else + { + oStringBuilder.AppendFormat("{0}, {1} - ({2})", oLeft, oRight, iWeight); + } + } + else + { + oStringBuilder.Append(iWeight); + } + } + + return oStringBuilder.ToString(); + } +} ++ + + + +
And finally, an extension method that is also required for the code to compile:
+ + ++ +public static int BinaryStringToInt32(this string Value) +{ + int iBinaryStringToInt32 = 0; + + for (int i = (Value.Length - 1), j = 0; i >= 0; i--, j++) + { + iBinaryStringToInt32 += ((Value[j] == '0' ? 0 : 1) * (int)(Math.Pow(2, i))); + } + + return iBinaryStringToInt32; +} ++ + + + +
To get this code running, it will need to be stitched together, perhaps in a console application:
+ + ++ +public static void Main(string[] Arguments) +{ + HuffmanCompressor oHuffmanCompressor = new HuffmanCompressor(); + oHuffmanCompressor.Compress("Huffman.txt"); + Console.WriteLine(); + oHuffmanCompressor.Decompress("Huffman.txt"); + Console.WriteLine(); +} ++ + + + +
Note that for this example, decompression is only possible if the binary tree has been constructed, i,e., the + data has already been compressed.
+ ++ +
+ +
Copyright © 2024 carlbelle.com
+ + + +|
+
+ *The author is the sysop of the Science SIG of PV-VAN. His address is: 12-2-404 Green Heights, 580 + Nagasawa, Yokosuka 239 Japan + + Introduction+In the spring of 1988, I wrote a very simple data compression program named LZSS in C language, and + uploaded it to the Science SIG (forum) of PC-VAN, Japan's biggest personal computer network. + That program was based on Storer and Szymanski's slightly modified version of one of Lempel and Ziv's + algorithms. Despite its simplicity, for most files its compression outperformed the archivers then + widely used. + Kazuhiko Miki rewrote my LZSS in Turbo Pascal and assembly language, and soon made it evolve into a + complete archiver, which he named LARC. + The first versions of LZSS and LARC were rather slow. So I rewrote my LZSS using a binary tree, and so + did Miki. Although LARC's encoding was slower than the fastest archiver available, its decoding was + quite fast, and its algorithm was so simple that even self-extracting files (compressed files plus + decoder) it created were usually smaller than non-self-extracting files from other archivers. + Soon many hobby programmers joined the archiver project at the forum. Very many suggestions were made, + and LARC was revised again and again. By the summer of 1988, LARC's speed and compression have improved + so much that LARC-compressed programs were beginning to be uploaded in many forums of PC-VAN and other + networks. + In that summer I wrote another program, LZARI, which combined the LZSS algorithm with adaptive arithmetic + compression. Although it was slower than LZSS, its compression performance was amazing. + Miki, the author of LARC, uploaded LZARI to NIFTY-Serve, another big information network in Japan. In + NIFTY-Serve, Haruyasu Yoshizaki replaced LZARI's adaptive arithmetic coding with a version of adaptive + Huffman coding to increase speed. Based on this algorithm, which he called LZHUF, he developed yet + another archiver, LHarc. + In what follows, I will review several of these algorithms and supply simplified codes in C language. + + Simple coding methods+Replacing several (usually 8 or 4) "space" characters by one "tab" character is a very primitive method + for data compression. Another simple method is run-length coding, which encodes the message + "AAABBBBAACCCC" into "3A4B2A4C", for example. + + LZSS coding+This scheme is initiated by Ziv and Lempel [1]. A slightly modified version is + described by Storer and Szymanski [2]. An implementation using a binary tree is + proposed by Bell [3]. The algorithm is quite simple: Keep a ring buffer, which + initially contains "space" characters only. Read several letters from the file to the buffer. Then + search the buffer for the longest string that matches the letters just read, and send its length and + position in the buffer. + If the buffer size is 4096 bytes, the position can be encoded in 12 bits. If we represent the match + length in four bits, the [position, length] pair is two bytes long. If the longest match is no more than + two characters, then we send just one character without encoding, and restart the process with the next + letter. We must send one extra bit each time to tell the decoder whether we are sending a [position, + length] pair or an unencoded character. + The accompanying file LZSS.C is a version of this algorithm. This implementation uses multiple binary + trees to speed up the search for the longest match. All the programs in this article are written in + draft-proposed ANSI C. I tested them with Turbo C 2.0. + + LZW coding+This scheme was devised by Ziv and Lempel [4], and modified by Welch [5]. + The LZW coding has been adopted by most of the existing archivers, such as ARC and PKZIP. The algorithm + can be made relatively fast, and is suitable for hardware implementation as well. + The algorithm can be outlined as follows: Prepare a table that can contain several thousand items. + Initially register in its 0th through 255th positions the usual 256 characters. Read several letters + from the file to be encoded, and search the table for the longest match. Suppose the longest match is + given by the string "ABC". Send the position of "ABC" in the table. Read the next character from the + file. If it is "D", then register a new string "ABCD" in the table, and restart the process with the + letter "D". If the table becomes full, discard the oldest item or, preferably, the least used. + A Pascal program for this algorithm is given in Storer's book [6]. + + Huffman coding+Classical Huffman coding is invented by Huffman [7]. A fairly readable account is + given in Sedgewick [8]. + Suppose the text to be encoded is "ABABACA", with four A's, two B's, and a C. We represent this situation + as follows: + + +
Combine the least frequent two characters into one, resulting in the new frequency 2 + 1 = 3: + + +
Repeat the above step until the whole characters combine into a tree: + + +
Start at the top ("root") of this encoding tree, and travel to the character you want to encode. If you + go left, send a "0"; otherwise send a "1". Thus, "A" is encoded by "0", "B" by "10", "C" by "11". + Algotether, "ABABACA" will be encoded into ten bits, "0100100110". + To decode this code, the decoder must know the encoding tree, which must be sent separately. + A modification to this classical Huffman coding is the adaptive, or dynamic, Huffman coding. See, e.g., + Gallager [9]. In this method, the encoder and the decoder processes the first letter + of the text as if the frequency of each character in the file were one, say. After the first letter has + been processed, both parties increment the frequency of that character by one. For example, if the first + letter is 'C', then freq['C'] becomes two, whereas every other frequencies are still one. Then the both + parties modify the encoding tree accordingly. Then the second letter will be encoded and decoded, and so + on. + + Arithmetic coding+The original concept of arithmetic coding is proposed by P. Elias. An implementation in C language is + described by Witten and others [10]. + Although the Huffman coding is optimal if each character must be encoded into a fixed (integer) number of + bits, arithmetic coding wins if no such restriction is made. + As an example we shall encode "AABA" using arithmetic coding. For simplicity suppose we know beforehand + that the probabilities for "A" and "B" to appear in the text are 3/4 and 1/4, respectively. + Initially, consider an interval: + + + + Since the first character is "A" whose probability is 3/4, we shrink the interval to the lower 3/4: + + + + The next character is "A" again, so we take the lower 3/4: + + + + Next comes "B" whose probability is 1/4, so we take the upper 1/4: + + + + because "B" is the second element in our alphabet, {A, B}. The last character is "A" and the interval is + + + + which can be written in binary notation + + + + Choose from this interval any number that can be represented in fewest bits, say 0.1, and send the bits + to the right of "0."; in this case we send only one bit, "1". Thus we have encoded four letters into one + bit! With the Huffman coding, four letters could not be encoded into less than four bits. + To decode the code "1", we just reverse the process: First, we supply the "0." to the right of the + received code "1", resulting in "0.1" in binary notation, or 1/2. Since this number is in the first 3/4 + of the initial interval 0 <= x < 1, the first character must be "A". Shrink the interval into the lower + 3/4. In this new interval, the number 1/2 lies in the lower 3/4 part, so the second character is again + "A", and so on. The number of letters in the original file must be sent separately (or a special 'EOF' + character must be appended at the end of the file). + The algorithm described above requires that both the sender and receiver know the probability + distribution for the characters. The adaptive version of the algorithm removes this restriction by first + supposing uniform or any agreed-upon distribution of characters that approximates the true distribution, + and then updating the distribution after each character is sent and received. + + LZARI+In each step the LZSS algorithm sends either a character or a [position, length] pair. Among these, + perhaps character "e" appears more frequently than "x", and a [position, length] pair of length 3 might + be commoner than one of length 18, say. Thus, if we encode the more frequent in fewer bits and the less + frequent in more bits, the total length of the encoded text will be diminished. This consideration + suggests that we use Huffman or arithmetic coding, preferably of adaptive kind, along with LZSS. + This is easier said than done, because there are many possible [position, length] combinations. Adaptive + compression must keep running statistics of frequency distribution. Too many items make statistics + unreliable. + What follows is not even an approximate solution to the problem posed above, but anyway this was what I + did in the summer of 1988. + I extended the character set from 256 to three-hundred or so in size, and let characters 0 through 255 be + the usual 8-bit characters, whereas characters 253 + n represent that what follows is a position of + string of length n, where n = 3, 4 , .... These extended set of characters will be encoded with adaptive + arithmetic compression. + I also observed that longest-match strings tend to be the ones that were read relatively recently. + Therefore, recent positions should be encoded into fewer bits. Since 4096 positions are too many to + encode adaptively, I fixed the probability distribution of the positions "by hand." The distribution + function given in the accompanying LZARI.C is rather tentative; it is not based on thorough + experimentation. In retrospect, I could encode adaptively the most significant 6 bits, say, or perhaps + by some more ingenious method adapt the parameters of the distribution function to the running + statistics. + At any rate, the present version of LZARI treats the positions rather separately, so that the overall + compression is by no means optimal. Furthermore, the string length threshold above which strings are + coded into [position, length] pairs is fixed, but logically its value must change according to the + length of the [position, length] pair we would get. + + LZHUF+LZHUF, the algorithm of Haruyasu Yoshizaki's archiver LHarc, replaces LZARI's adaptive arithmetic coding + with adaptive Huffman. LZHUF encodes the most significant 6 bits of the position in its 4096-byte buffer + by table lookup. More recent, and hence more probable, positions are coded in less bits. On the other + hand, the remaining 6 bits are sent verbatim. Because Huffman coding encodes each letter into a fixed + number of bits, table lookup can be easily implemented. + Though theoretically Huffman cannot exceed arithmetic compression, the difference is very slight, and + LZHUF is fairly fast. + The accompanying file LZHUF.C was written by Yoshizaki. I translated the comments into English and made a
+ few trivial changes to make it conform to the ANSI C standard.
+
+
|
+
This document was + last updated on Sunday, + December 13, 1998
+ +Introduction
+ +Every
+effort has been made to document LZH header formats as well as changes made for
+features not yet implemented. Corrections, additions and suggestions are
+always welcomed. Header fields in Italics are currently under development
+for Huffman Compression Engine only, and should be ignored (skipped) if not
+supported, as should any extended header.
+If you are a developer of compression utilities in .lzh file formats,
+please feel free to jump in and help.
Table
+of Contents
Introduction. 1
LZH format. 3
level-0..... 3
level-1, level-2..... 3
Level 0 header structure.. 4
Level 1 header structure.. 5
Level 2 header structure.. 6
Extended headers.... 7
Handling of extended headers.... 9
Method ID............. 10
Variances 11
Huffman Compression Engine II....... 11
Generic time stamp....
+ 11
OS ID... 12
Links to other LHA utilities and compression references:.
+ 13
0x39 Multi-disk field.... 14
SPAN_COMPLETE. 14
SPAN_MORE......... 14
SPAN_LAST........... 15
size_of_run............. 15
span_mode:.............
+ 15
Termination.............
+ 18
+ + +
LZH + format
+ +Byte Order:
+Little-endian
There are 3
+types of LZH headers, level-0, level-1, and level-2.
All .lzh and
+.lha files are null terminated. The
+last byte of the file should be a 0, but if it's not, it will be interpreted as
+null terminated. The reason for this
+termination is that it implies that the next header-size is zero. Huffman Compression Engine adds the null but
+doesn't need it.
level-0
+ +LZH header
Compressed
+file
LZH header
Compressed
+file
LZH header of
+size 0 (1 byte null)
level-1, + level-2
+ +LZH header
Extension
+header(s)
Compressed
+file
In all cases,
+read the first 21 bytes of the header.
+After determining the header type, you will then have to handle the
+header as needed or suggested.
+ + +
Level 0 header structure
+ +
|
+ Offset |
+
+ Size in bytes |
+
+ Description |
+
|
+ 0 |
+
+ 1 |
+
+ Size of
+ archived file header (h) |
+
|
+ 1 |
+
+ 1 |
+
+ 1 byte Header checksum |
+
|
+ 2 |
+
+ 5 |
+
+ Method
+ ID |
+
|
+ 7 |
+
+ 4 |
+
+ Compressed
+ file size Refered to as C in subsequent fields. As there are no extended headers in level 0 format archive
+ headers, this value represents the size of the file data only. |
+
|
+ 11 |
+
+ 4 |
+
+ Uncompressed
+ file size |
+
|
+ 15 |
+
+ 4 |
+
+ Original
+ file date/time (Generic time stamp
+ ) |
+
|
+ 19 |
+
+ 1 |
+
+ File or
+ directory attribute |
+
|
+ 20 |
+
+ 1 |
+
+ Level
+ identifier (0x00) Programmers should
+ only read the first 21 bytes of the header before taking further action |
+
|
+ 21 |
+
+ 1 |
+
+ Length of
+ filename in bytes (Refered to as N in subsequent fields |
+
|
+ 22 |
+
+ N |
+
+ Path and
+ Filename |
+
|
+ 22+N |
+
+ 2 |
+
+ 16 bit CRC
+ of the uncompressed file |
+
|
+ 24+N |
+
+ C |
+
+ Compressed
+ file data |
+
+ + +
Level 1 header structure
+ +
|
+ Offset |
+
+ Size in bytes |
+
+ Description |
+
|
+ 0 |
+
+ 1 |
+
+ Size of
+ archived file header (h) |
+
|
+ 1 |
+
+ 1 |
+
+ 1 byte Header checksum |
+
|
+ 2 |
+
+ 5 |
+
+ Method
+ ID |
+
|
+ 7 |
+
+ 4 |
+
+ Compressed
+ file size Refered to as C in subsequent fields |
+
|
+ 11 |
+
+ 4 |
+
+ Uncompressed
+ file size |
+
|
+ 15 |
+
+ 4 |
+
+ Original
+ file date/time (Generic time stamp
+ ) |
+
|
+ 19 |
+
+ 1 |
+
+ File or
+ directory attribute |
+
|
+ 20 |
+
+ 1 |
+
+ Level
+ identifier (0x00) Programmers should
+ only read the first 21 bytes of the header before taking further action |
+
|
+ 21 |
+
+ 1 |
+
+ Length of
+ filename in bytes (Refered to as N in subsequent fields |
+
|
+ 22 |
+
+ N |
+
+ Path and
+ Filename |
+
|
+ 22+N |
+
+ 2 |
+
+ 16 bit CRC
+ of the uncompressed file |
+
|
+ 24+N |
+
+ 1 |
+
+ Operating
+ System identifier. See OS ID chart |
+
|
+ 25+N |
+
+ 2 |
+
+ Next Header
+ size |
+
|
+ 27+N |
+
+ 3 or more
+ bytes |
+
+ Extended
+ headers ( Note:
+ Extended headers are optional, and have no preset maximum. The first byte of an extended header
+ identifies the type of header, and the last 2 bytes of the header identify
+ whether or not more headers are defined.
+ Huffman Compression Engine will use the extended header filename if
+ both exist in level 1 archives. |
+
|
+ |
+
+ C |
+
+ Compressed
+ file data |
+
+ + +
Level 2 header structure
+ +|
+ Offset |
+
+ Size in bytes |
+
+ Description |
+
|
+ 1 |
+
+ 2 |
+
+ Total size
+ of headers, including Extended headers for this entry. |
+
|
+ 2 |
+
+ 5 |
+
+ Method
+ ID |
+
|
+ 7 |
+
+ 4 |
+
+ Compressed
+ file size Referred to as C in subsequent fields This value
+ excludes the size of all Extended headers and only refers to the actual compressed data. This is an improvement, and can be
+ problematic if not handled properly. |
+
|
+ 11 |
+
+ 4 |
+
+ Uncompressed
+ file size |
+
|
+ 15 |
+
+ 4 |
+
+ Original
+ file date/time (Generic time stamp
+ ) |
+
|
+ 19 |
+
+ 1 |
+
+ File or
+ directory attribute. Not supported in all compression utilities. |
+
|
+ 20 |
+
+ 1 |
+
+ Level
+ identifier (0x00) Programmers should
+ only read the first 21 bytes of the header before taking further action |
+
|
+ 21 |
+
+ 2 |
+
+ 16 bit CRC
+ of the uncompressed file |
+
|
+ 23 |
+
+ 1 |
+
+ Operating
+ System identifier. See OS ID chart |
+
|
+ 24 |
+
+ 2 |
+
+ Next Header
+ size |
+
|
+ 26 |
+
+ 3 or more
+ bytes |
+
+ Extended
+ headers ( Note:
+ Extended headers are not optional, and have no preset maximum. Minimum compatibility should include a
+ type1 extended header. The first
+ byte of an extended header identifies the type of header, and the last 2
+ bytes of the header identify whether or not more headers are defined. Huffman Compression Engine will use the
+ extended header filename if both exist in level 1 archives. Your loop for reading of these headers
+ should include offset 24 for the first assignment and loop until extended
+ header size = 0. |
+
|
+ |
+
+ C |
+
+ Compressed
+ file data |
+
+ + +
Extended headers
+ +Unspecified
+size fields are intentionally left blank.
|
+ ID |
+
+ Size |
+
+ Description |
+
|
+ 0x00 |
+
+ 2 |
+
+ CRC-16 of
+ header and an optional information byte. |
+
|
+ 0x01 |
+
+ |
+
+ Filename |
+
|
+ 0x02 |
+
+ |
+
+ Directory
+ name |
+
|
+ 0x39 |
+
+ |
+
+ 0x39
+ Multi-disk field |
+
|
+ 0x3f |
+
+ |
+
+ Uncompressed
+ file comment. |
+
|
+ 0x48 0x4f |
+
+ |
+
+ Reserved for Authenticity verification |
+
|
+ |
+
+ |
+
+ |
+
|
+ |
+
+ |
+
+ |
+
|
+ |
+
+ |
+
+ |
+
|
+ |
+
+ |
+
+ |
+
|
+ 0x5? |
+
+ 2 |
+
+ UNIX related
+ information. Optional
+ information byte |
+
|
+ 0x50 |
+
+ 2 |
+
+ UNIX file
+ permission |
+
|
+ 0x51 |
+
+ 2 |
+
+ Group ID |
+
|
+ 0x52 |
+
+ |
+
+ Group Name |
+
|
+ 0x53 |
+
+ |
+
+ User Name |
+
|
+ 0x54 |
+
+ |
+
+ Last
+ modified time in UNIX time |
+
|
+ |
+
+ |
+
+ |
+
|
+ 0xcn |
+
+ |
+
+ Under development: Compressed file comment.
+ Method -lhn- is assumed Compressed comment cannot exceed 64K in size. Applicable range for Huffman Compression Engine (4..8) |
+
|
+ 0xdx |
+
+ |
+
+ Under
+ development: Operating system
+ specific header info. These fields
+ may have different meanings for different platforms. If the file was not created on the same
+ platform as your own these signatures should be ignored. |
+
|
+ 0xd1 |
+
+ |
+
+ Under
+ development Autodelete
+ after autorun |
+
+ + +
Handling of extended headers.
+ +
Proper
+procedure for handling of extended headers can be summed up in virtual code:
1
+Read the first 21 bytes of the real header to determine the
+size of the first extended header. If
+the
2
+Use the first extended header in the loop that reads
+subsequent headers
3
+Assign this value to a variable which is used inside the
+extended header loop
4
+Repeat
5
+Allocate enough memory for the header
6
+Read the header into the array
7
+Assign headersize to a word variable at the last 2 bytes of
+the array
8
+Goto 3
9
+Handle compressed data if it exists.
Code
+has not been supplied intentionally. It
+is expected that the programmer reading this document has enough knowledge of
+programming to perform the task of writing real code.
+ + +
Method ID
+ +|
+ Signature |
+
+ Description |
+
|
+ -lh0- |
+
+ No
+ compression |
+
|
+ -lzs- |
+
+ 2k
+ sliding dictionary(max 17 bytes) |
+
|
+ -lz4- |
+
+ No
+ compression |
+
|
+ -lh1- |
+
+ 4k
+ sliding dictionary(max 60 bytes) + dynamic Huffman + fixed encoding of
+ position |
+
|
+ -lh2- |
+
+ 8k
+ sliding dictionary(max 256 bytes) + dynamic Huffman (Obsoleted) |
+
|
+ -lh3- |
+
+ 8k
+ sliding dictionary(max 256 bytes) + static Huffman (Obsoleted) This method is
+ not supported by Huffman Compression Engine |
+
|
+ -lh4- |
+
+ 4k
+ sliding dictionary(max 256 bytes) + static Huffman + improved encoding of
+ position and trees |
+
|
+ -lh5- |
+
+ 8k
+ sliding dictionary + static Huffman |
+
|
+ -lh6- |
+
+ 32k
+ sliding dictionary + static Huffman |
+
|
+ -lh7- -lh8- |
+
+ 64k
+ sliding dictionary + static Huffman Lh8
+ has yet to be discovered except in my own utility. It existed from 0.21d to 0.21M and was actually -lh7- per
+ Harihiko's review of Yoshi's notes. |
+
|
+ "-lhd-" |
+
+ Directory
+ (no compressed data). This signature
+ may not contain any information at all.
+ In some cases it only is used to signify that there are extended
+ headers with important information.
+ In level 0 archives it most likely contains the directory for the next
+ header's file, but in level 1 and 2 headers, it most likely contains nothing
+ except for possibly the size of the first extended header. |
+
Huffman
+Compression Engine II dynamically sets the method for compression based on the
+file size. Reasoning: It makes little sense
+to use a 64KB buffer if the file is 1K.
+The next section covers variances, which basically define what to do in
+known cases where the file signature does not match the dictionary size.
+ + +
Variances
+ +Huffman Compression Engine II
+ +Versions after
+0.21M now dynamically size the dictionary buffer according to the size of the
+file to be compressed. It is not
+uncommon to have signatures of -lh4- thru -lh7-. A future implementation
+will include -lh9- through -lhc- and -lhe-, which represent dictionary sizes of
+128KB through 2MB. -lhd- is skipped due to the fact that it is a reserved
+signature. During 0.21, a
+misunderstanding about the method identifier lead me to think that a 64KB
+buffer was -lh8-. Although logic
+dictated that it should be, 16KB dictionaries were bypassed entirely, which
+lead to this confusion. In order to
+simplify the problem, decode should use 64KB for -lh6- through -lh8-.
Generic time stamp
+
+ 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17
+16
|<--------
+year ------->|<- month ->|<-- day -->|
15 14 13 12 11 10 9 8 7 6 5
+4 3 2 1 0
|<--- hour
+--->|<---- minute --->|<- second/2 ->|
Offset Length
+Contents
0
+8 bits year years since 1980
8
+4 bits month [1..12]
12 4 bits day [1..31]
16 5 bits hour [0..23]
21 6 bits minite [0..59]
27 5 bits second/2 [0..29]
+ + +
OS ID
+ +|
+ 'ID |
+
+ Platform |
+
|
+ M |
+
+ MS-Dos |
+
|
+ '2' |
+
+ OS/2 |
+
|
+ '9' |
+
+ OS9 |
+
|
+ 'K' |
+
+ OS/68K |
+
|
+ '3' |
+
+ OS/386 |
+
|
+ 'H' |
+
+ HUMAN |
+
|
+ 'U' |
+
+ UNIX |
+
|
+ 'C' |
+
+ CP/M |
+
|
+ 'F' |
+
+ FLEX |
+
|
+ 'm' |
+
+ Mac |
+
|
+ 'w' |
+
+ Windows 95,
+ 98 |
+
|
+ 'W' |
+
+ Windows NT |
+
|
+ 'R' |
+
+ Runser |
+
+ + +
Links to other LHA utilities and compression + references:
+ +The following
+hyperlinks are visible in word97. If
+you cannot see these links, you may also jump on the Internet to http://fidonews.webworldinc.com/lzhformat.htm to
+see them in your favorite browser.
LHA
+World by Dr. Haruyasu Yoshizaki
Dolphin's Home Page The author, Tsugio Okamoto,
+maintains "Lha for Unix."
If you are
+interested in how LHA works, its source code is a very good place to start.
This site
+includes useful info like LHA header specs, in Japanese.
Lha for UNIX patch by Yoshioka
+Tsuneo.
MacLHA
+for Macintosh systems.
Network
+Mahjong International (LHA
+in Java)
Micco's HomePage (UNLHA32.DLL UNARJ32.DLL LHMelt)
Haruhiko Okumura's Compression Pointers.
+Haruhiko Okumura is the original designer of the lha compression algorithm.
Compatibility
+with the above links can only be guessed at this point, as few lha style
+compressors support anything above -lh5-.
+However, if your interest is in maintaining compatibility with these
+other platforms, -K5 added to the command line during compression will force
+compatibility with these compression utilities.
+ + +
0x39 Multi-disk field
+ +
+
The following is a
+planned implementation. The original
+can be found at:
http://www.creative.net/~aeco/jp/lzhspc01.html
struct MDF {
byte span_mode;
long beginning_offset;
long size_of_run;
}
span_mode: This identifies the mode of
+this segment of file. The values are:
#define SPAN_COMPLETE 0
#define SPAN_MORE 1
#define SPAN_LAST 2
SPAN_COMPLETE.
+ +This specifies that
+the information following this header contains a complete (optionally
+compressed) file. This is often unused because MDF is not needed in these
+cases. In an unsplit file, the header information and format should follow the
+standard LZH format.
SPAN_MORE.
+ +This specifies that the
+information following this header is incomplete. The uncompressor needs to
+concatenate this segment with
information from the following volume.
+It should continue to do that until it sees a volume with a header information
+that contains span_mode
SPAN_LAST
+ +This specifies that
+the information following this header is the last segment of the (optionally
+compressed) file.
beginning_offset:
This value specifies
+the location in bytes of where this segment (run) of information will fit into.
size_of_run
+ +This is the size of
+this segment of information.
span_mode:
+ +This identifies the
+mode of this segment of file. The values are:
#define SPAN_COMPLETE 0
#define SPAN_MORE 1
#define SPAN_LAST 2
SPAN_COMPLETE. This
+specifies that the information following this header contains a complete
+(optionally compressed) file. This is often unused
because MDF is not needed in these
+cases. In an unsplit file, the header information and format should follow the
+standard LZH format.
SPAN_MORE. This specifies that the
+information following this header is incomplete. The uncompressor needs to
+concatenate this segment with
information from the following volume.
+It should continue to do that until it sees a volume with a header information
+that contains span_mode
SPAN_LAST.
SPAN_LAST. This specifies that the
+information following this header is the last segment of the (optionally
+compressed) file.
beginning_offset: This value specifies
+the location in bytes of where this segment (run) of information will fit into.
size_of_run: This is the size of this
+segment of information.
+ + +
The illustration below contain two
+volumes with two compressed files, one of them split between the two volumes.
+"File 1" is compressed and fits
within the first volume. "File
+2" is a file 100 bytes long compressed to 90 bytes. The first 50 bytes of
+which resides on the first volume and the
last 40 bytes on the next.
Volume 1
+ +++--------------+
+ +| +----------+ + |
+ +| |LZH + header| | <- MDF not needed
+ +| + +----------+ | <- header unchanged from non-spanned versions of LZH
+ +| | File + 1 | |
+ +| | | |
+ +| | | |
+ +| + +----------+ |
+ +| |
+ +| + +----------+ | <- span_mode = SPAN_MORE
+ +| |LZH + header| | <- beginning_offset = 0
+ +| + +----------+ | <- size_of_run = 50
+ +| | File + 2 | |
+ +| | + split | |
+ +| | | |
+ +++--------------+
+ +
+
Volume 2
+ +++--------------+
+ +| + +----------+ | <- span_mode = SPAN_LAST
+ +| |LZH + header| | <- beginning_offset = 50
+ +| + +----------+ | <- size_of_run = 40
+ +| | File + 2 | |
+ +| | | |
+ +| | | |
+ +| | | |
+ +| + +----------+ |
+ +| +----------+ + |
+ +| | [0] | | <- end of volumes, a byte with + value zero (0)
+ +| + +----------+ |
+ +++--------------+
+ ++ + +
+
Termination
+ + -----------
In addition to the
+above changes, the compressor must before closing the file after writing the
+last volume, write a null byte at the end of the file. This byte serves to
+inform the decompressor that this is the last volume and no other comes after
+it.
This end of volume
+byte is needed to tell the decompressor when to stop prompting for the next volume. This termination byte is optional, as the
+decompressor may also stop when it has completed a file, i.e., see SPAN_LAST or
+just a regular file with
MDF. However if the end of this
+completed file coincides with an end of volume, there would be not way for the
+compressor detect that the following
volumes and prompt for them.
This termination byte is a way around
+this potential bug.
Note that this null byte coincides with
+header_size in the LZH header.
Pseudo code for a sample implementation
---------------------------------------
boolean spanned = false;
while(file_available()) {
compress file();
if(size_of_compressed_file >
+available_space()) {
while(size_of_compressed_file) {
size_to_write = available_space();
+construct_header_with_MDF(size_to_write);
write_header();
write_data(size_to_write);
size_of_compressed_file -=
+size_to_write;
prompt_for_next_volume();
spanned = true;
}
}
else {
+construct_header_without_MDF(size_of_compressed_file);
write_header();
write_data(size_of_compressed_file);
}
}
if(spanned)
write_null_byte();
Author’s address:
Aeco Systems
826 28th Avenue
San Francisco, CA 94121
Phone: (415) 221-7806
EMail: aeco@creative.net