The history of Zstandard

Project history

Zstandard was developed at Facebook by Yann Collet and collaborators as a modern successor in the design space occupied by Deflate, gzip, zlib, lz4, and xz. Meta announced Zstandard 1.0 on 2016-08-31, describing it as a compression algorithm and implementation designed for modern CPUs, high decompression speed, and a broad range of lossless-compression scenarios.

The reference implementation lives in the facebook/zstd repository as a C library and CLI. The README describes a stable format, the libzstd API, the zstd command-line utility, support for producing and decoding .zst plus compatibility with .gz, .xz, and .lz4 files, and a list of ports and bindings for other languages.

The format moved from a popular open-source implementation into a documented Internet format. RFC 8878 describes Zstandard compression and registers the application/zstd media type, content encoding, and structured syntax suffix for transporting zstd-compressed content.

Adoption history

zstd spread because it hit a package-maintainer sweet spot: faster decompression than gzip-class tools, better ratio than many fast compressors, a stable file format, and a normal Unix CLI. That combination made it attractive for source archives, package payloads, container layers, log compression, backup streams, and embedded compression inside larger systems.

Filesystem adoption gave zstd particular visibility among system users. Btrfs documentation records ZSTD support since Linux 4.14 alongside ZLIB and LZO, and Meta engineering later wrote about transparent zstd compression in Btrfs as a way to improve storage efficiency without a noticeable performance drop in its environment.

The project stayed active rather than freezing at the format layer. The changelog shows continuing work through the 1.5.x series and a 1.6.0 entry in Dec 2025, including CLI, API, build-system, portability, and performance improvements.

How it is used

The package gives users commands such as zstd, unzstd, zstdcat, zstdgrep, zstdless, pzstd, and zstdmt. Common shell usage mirrors gzip-style workflows: compress a file or stream to .zst, decompress with unzstd or zstd -d, pipe through zstdcat, and tune level or --fast modes for speed/ratio trade-offs.

For developers, libzstd exposes one-shot compression/decompression, reusable contexts, streaming APIs, dictionaries, frame inspection, and advanced experimental APIs. The README also documents dictionary training with zstd --train for small repeated records, a major reason zstd appears in data stores and network protocols rather than only in file archivers.

Why package nerds care

zstd is one of the rare compression packages that matters at every packaging layer: as a user-facing CLI, a shared library dependency, an archive format, a filesystem compressor, and a performance knob inside build and distribution infrastructure.

For package nerds it is also a case study in successful infrastructure standardization: a vendor-originated tool became a stable RFC-described format with independent implementations, many bindings, and broad distribution support while retaining the boring Unix ergonomics expected from gzip-era tools.

Timeline

• 2016-08-31: Facebook announced Zstandard 1.0.
• 2017: ZSTD support entered Btrfs documentation as available since Linux 4.14.
• 2018-12-19: Meta engineering published notes on large-scale storage improvements with Zstandard.
• 2021: RFC 8878 documented the Zstandard format and application/zstd media type.
• 2025-02: zstd 1.5.7 changelog entry records CLI, API, performance, and portability work.
• 2025-12: zstd 1.6.0 changelog entry records legacy-format support disabled by default.

Related projects

• Related projects include zlib, gzip, Deflate, lz4, xz, Brotli, Finite State Entropy, Huff0, Btrfs, SquashFS, libarchive-style tooling, and language bindings listed from the Zstandard homepage.