2024-07-23 16:52:00
LZ4 refers to both a particularly fast compression algorithm and a corresponding reference implementation. The latter consists of the library Liblz4 and the command-line program “lz4”. The complete system is now available in version v1.10.0, whose codename “Multicores Edition” already points to one of the biggest innovations.
Advertisement
Multithreading
LZ4 now distributes its work across multiple threads. On processors with many cores, this leads to a significant speed boost. The performance leap is particularly noticeable at very high compression levels, which shrink files that require particularly high levels of effort (High-Compression modes). According to the benchmark results from the developers, the new LZ4 version works up to eight times faster than its predecessor. The team observed the largest speed gains on a test system with an AMD Ryzen 7 7840HS processor and Windows 11.
Furthermore, I/O operations and decompression processes overlap. Together with this optimization, LZ4 required only three seconds instead of five to unpack a 5 GByte text file on an unspecified test system. The speed increases not only benefit the High-Compression modes but are also useful when using fast storage systems, such as NVMe drives.
Speaking of compression levels: LZ4 offers a new Compression Level 2. In this level, the packing program operates somewhat slower than in the default Level 1 (Fast Compression), but compresses more strongly. However, the generated files are not as small as in the higher Level 3. The developers appropriately recommend Level 2 whenever smaller archives than those produced in Level 1 are needed but where waiting as long as with Level 3 is not an option.
Dictionary Compression
The previous version already supported a so-called Dictionary Compression. In this compression method, the packing program replaces recurring file contents with a shorter symbol. The corresponding mappings are stored in a file referred to as the Dictionary. If several files with similar content share a Dictionary, this can drastically reduce the storage requirements of the compressed versions. Thus, using a Dictionary is particularly worthwhile for many smaller files that do not compress well on their own.
While Dictionary Compression was still considered experimental in the previous version, LZ4 v1.10.0 now officially supports the method. In particular, programmers can finally use Dictionaries safely in their applications. The Liblz4 library offers several new functions for this purpose. For example, “LZ4F_decompress_usingDict()” decompresses multiple data blocks (in the form of an LZ4 frame) using a Dictionary. LZ4 can also work with existing Dictionaries created by the Dictionary Builder of the compression tool Zstd.
Finally, LZ4 supports the processor architectures loongArch, RISC-V, m68k, MIPS, and Sparc, and the developers have also made slight adjustments to the build system. All changes are summarized in the official announcement.
(vbr)