State of open source compression in 2018

Date: Sat, 20 Oct 2018 22:54:55 +0200
From: Adam Borowski <kilobyte@angband.pl>
To: dng@lists.dyne.org
Subject: Re: [DNG] How to unarchive an .xz

On Sat, Oct 20, 2018 at 12:39:39PM -0400, Steve Litt wrote:

> I want to make a VM from devuan_ascii_2.0.0_amd64_qemu.qcow2.xz
>
> After 15 minutes googling and reading Devuan docs, I have nothing that
> works. How do I unarchive the file? Could you please put the method in
> the README file?

Uhm, you do know that xz-utils/liblzma has been Essential since March 2010?[1]

But, if an experienced person like you lacks this part of knowledge, let me provide some information, for the sake of readers of this list:

Among mainstream compression algorithms, only three have a reason to be used (other than for compatibility purposes):

• xz: slow memory-hungry strong compression, for most compress-once / transmit-multiple purposes (such as release tarballs, etc.)
• zstd: balanced speed-vs-ratio, for most local purposes
• lz4: memory-bandwidth-speed, for stuff that doesn't even hit the disk

There are three things to look at, when comparing compression algorithms: strength, speed, and memory use. The latter rarely matters, so it's mostly about the strength-to-speed envelope. Most algorithms take a "level" parameter: xz is faster-(but better)-than-gzip at its lowest setting; zstd beats the stuffing out of competition, for a big range from lzop-like at the lowest setting to mid-xz at the highest, etc.

Anything else has been obsoleted, and should be deprecated:

• gzip is venerable, and had seen so much use in the past that it won't ever go away. But you shouldn't use it: its speed is same as xz at level 2.5, but compression ratio is drastically worse.
• bzip2 is drastically slower (esp. at decompression) than xz and zstd; it needs to die.
• lzop had a good run, but zstd is much stronger at the same speed, while lz4 is much faster.
• lzip is a worse alternative to xz, peddled by a couple of trolls who send untruthful propaganda to projects, hoping people won't do research before accepting its use. Just say no and use xz.
• ncompress is completely dead by now.
• zip is used by Windows folks only, similar to gzip.
• rar is unfree.
• 7z is similar but incompatible to xz; also is a container rather than a pure compressor.

The above info has been greatly simplified, but should give an idea.

[1]RM footnote: Adam means marked as an Essential package in the Debian (and thus now also Devuan) Linux distribution, the point being that the "xz" compression tool can now be assumed standard in Linux installations.