facebook/zstd: Fast lossless compression and decompression for real-time workloads
A C library and CLI for fast lossless compression of real-time data, with dictionaries for small records.
GitHub facebook/zstd Updated 2026-09-09 Branch dev Stars 27.8K Forks 2.6K
C/C++ Lossless compression libzstd Linux/Unix

🧭 Decision Guide

Try it if you

  • You need real-time lossless compression in Linux data streams using the zstd CLI or libzstd
    The README introduction describes zstd as targeting real-time compression scenarios and provides both a C library and command-line utility
  • Your service processes roughly 1KB, correlated records of the same data type
    The README's The case for Small Data compression section uses roughly 10K github-users records of about 1KB to explain training mode and dictionaries
  • Your C/C++ project wants to build libzstd with make, CMake, or Buck
    The README's Build instructions section gives build methods for make, CMake, and Buck

Skip it if you

  • You want one dictionary shared across every data type without maintaining per-type dictionaries
    The README explicitly says there is no universal dictionary and that one dictionary per data type provides the greatest benefits
  • Your primary goal is the highest benchmark decompression throughput rather than zstd's speed-and-ratio trade-off
    In the README Benchmarks, lz4 1.10.0 reaches 3850 MB/s decompression, above zstd 1.5.7 --fast=4 at 2050 MB/s

Requirements

  • The system must support standard make or gmake; the README calls make the project's main build system.
  • With CMake, use `cmake -S . -B build-cmake` and `cmake --build build-cmake`.
  • When running `src/tests/playTest.sh`, `$ZSTD_BIN` and `$DATAGEN_BIN` must point to the zstd and datagen binaries.
  • When using a dictionary, the README requires training it from samples and loading it before compression and decompression.

First step (verbatim from README)

make check

Watch out

  • The negative compression level --fast=# increases speed, but the README explicitly says it reduces compression ratio.
    README Benchmarks: The negative compression levels ... offer faster compression and decompression speed at the cost of compression ratio
  • Dictionary gains are mostly effective in the first few KB; later content uses previously decoded content for compression.
    README's Dictionary compression How To section and its accompanying explanation
  • CMake defaults CMAKE_BUILD_TYPE to Release, so the default build is not a Debug build.
    The README's cmake section explicitly states: By default, CMAKE_BUILD_TYPE is set to Release

Alternatives

  • lz4:When decompression throughput is the priority, the README benchmark reports 3850 MB/s for lz4 1.10.0, versus 2050 MB/s for zstd 1.5.7 --fast=4.
    README: Benchmarks
  • brotli:When the benchmark's compression ratio is slightly more important than zstd 1.5.7 -1's result, brotli 1.1.0 -1 reports 2.883 versus zstd's 2.896; the speed data must be considered alongside it.
    README: Benchmarks
  • zlib:When an existing system explicitly depends on zlib 1.3.1 or its compatibility ecosystem, zlib may be preferable; the README lists it as a comparison algorithm.
    README: Benchmarks

Not stated in the README

  • The README does not provide concrete build steps for Windows Visual Studio, Meson, VCPKG, Conan, or Bazel; those sections were omitted from the supplied material.
  • The material does not describe compatibility changes, upgrade migration requirements, or ABI details between zstd 1.5.7 and older versions.
  • GitHub metadata lists the license as Other, while the README states BSD OR GPLv2; the material does not explain this presentation difference.
  • The material does not provide benchmark results for the target hardware, actual input data, or business throughput.
  • The material does not list specific language bindings or confirm whether a usable port exists for a target language.

💡 Deep Analysis

6
No My Linux distribution service currently uses lz4 on hardware close to the README's Core i7-9700K setup, and maximum decompression throughput is the top metric; should I switch to zstd even if it saves storage and bandwidth?
For: An infrastructure engineer maintaining an lz4-based Linux distribution service on Core i7-9700K-class hardware with an extreme priority on decompression throughput

No, not as a direct replacement, because the README benchmark shows lz4 with substantially higher decompression throughput; reconsider only if ratio or tunability outweighs that loss.

  • On the same Core i7-9700K, Ubuntu 24.04, GCC 14.2.0, and Silesia benchmark, lz4 reaches 3850 MB/s decompression, versus 1550 MB/s for zstd -1 and 2050 MB/s for zstd –fast=4.
  • Zstd’s advantage is ratio and tunable speed: zstd -1 reaches 2.896 versus lz4’s 2.101, but that does not remove lz4’s decompression advantage.
  • The README states that --fast=# sacrifices ratio for speed; even fast=4 remains below lz4 in this benchmark.
  • If your constraint is explicitly maximum decompression throughput, the project data does not support switching; further comparison makes sense only if network or storage capacity becomes the bottleneck.
  • Benchmarks: lz4 1.10.0, ratio 2.101, compression 675 MB/s, decompression 3850 MB/s
  • Benchmarks: zstd 1.5.7 -1, ratio 2.896, compression 510 MB/s, decompression 1550 MB/s
  • Benchmarks: zstd 1.5.7 --fast=4, ratio 2.146, compression 665 MB/s, decompression 2050 MB/s
  • Quote: The negative compression levels ... offer faster compression and decompression speed at the cost of compression ratio.
Not stated in the README:The README does not identify whether network, disk, or CPU is the actual bottleneck in your distribution service.;The README benchmark does not cover your file types, concurrency, or end-to-end cache behavior.
Yes My Linux service currently uses zlib -1 on servers close to the Core i7-9700K setup in the README; if I need lower compression latency, better ratio, and still-fast decompression, is zstd a suitable replacement?
For: A backend engineer maintaining a zlib-based Linux service on Core i7-9700K-class servers, constrained by both compression latency and network bandwidth

Yes, because the README benchmark shows zstd -1 outperforming zlib -1 in compression speed, decompression speed, and ratio on the same setup.

  • zstd -1 reaches a 2.896 ratio, 510 MB/s compression, and 1550 MB/s decompression; zlib -1 reaches 2.743, 105 MB/s, and 390 MB/s.
  • The project targets “fast lossless compression” and real-time compression scenarios, with small configurable steps between speed and ratio.
  • For an online path that prioritizes latency, negative levels through --fast=# trade compression ratio for faster compression and decompression.
  • The format is documented by RFC8878 and has multiple independent implementations, which supports long-term interoperability planning.

However, the README benchmark uses a specific CPU, Ubuntu, GCC, and the Silesia corpus, so it does not directly establish end-to-end gains for your service.

  • Benchmarks: zstd 1.5.7 -1: ratio 2.896, compression 510 MB/s, decompression 1550 MB/s
  • Benchmarks: zlib 1.3.1 -1: ratio 2.743, compression 105 MB/s, decompression 390 MB/s
  • Quote: Speed vs Compression trade-off is configurable by small increments.
  • Quote: The negative compression levels, specified with `--fast=#`, offer faster compression and decompression speed at the cost of compression ratio.
  • Quote: Zstandard's format is stable and documented in RFC8878.
make
Not stated in the README:The README does not provide results for your actual dataset, CPU, I/O path, or end-to-end latency.;The README does not specify a migration or rollback strategy for existing zlib data.
It depends I maintain roughly 10K GitHub API user records of about 1KB each, like the README example; if individual objects are too small for ordinary compression, should I use a zstd dictionary?
For: A server-side engineer handling 10K roughly 1KB GitHub API user records and needing to reduce API response size

It depends: a zstd dictionary is worthwhile only when the 1KB records have stable structural or content correlation.

  • The README explains that smaller inputs are harder to compress because a new dataset has insufficient historical context; this is the problem dictionaries target.
  • Its example uses roughly 10K GitHub user records of about 1KB each, trains a dictionary from samples, and reports dramatically better small-data ratios while also improving compression and decompression speed.
  • The dictionary must be loaded before compression and decompression, and the README explicitly says there is “no universal dictionary”; a data-specific dictionary is more effective.
  • If records are weakly correlated, or dictionary distribution and update costs are high, the gain may not justify the integration complexity.

The deciding factor is therefore not record count alone, but sample representativeness and consistent dictionary management on both sides.

  • The case for Small Data compression: the example contains roughly 10K records weighing about 1KB each
  • Quote: The smaller the amount of data to compress, the more difficult it is to compress.
  • Quote: Using this dictionary, the compression ratio achievable on small data improves dramatically.
  • Quote: The result of this training is stored in a file called "dictionary", which must be loaded before compression and decompression.
  • Quote: there is no universal dictionary
Not stated in the README:The README does not provide a correlation measurement between your production records and training samples.;The README does not specify dictionary size, update frequency, distribution protocol, or old-dictionary compatibility policy.
Yes I need to embed compression into a C/C++ database or messaging system and produce a CMake Universal2 build supporting Apple Silicon M1/M2 and Intel; does zstd fit these integration constraints?
For: A systems engineer embedding compression into a C/C++ database or messaging system while using CMake and Apple Universal2 builds

Yes, because the project provides an embeddable C reference library, a CMake path, and explicit Universal2 build and installation commands in the README.

  • The README describes the repository as an open-source dual-licensed C library and also provides a command-line utility, making direct database or messaging-system integration possible.
  • It supports streaming compression, decompression, and large-file processing, which fits incremental embedded workloads, although API lifecycle details must follow the library documentation.
  • The README calls make the reference build system and lists CMake, Meson, Buck, Bazel, VCPKG, Conan, and Visual Studio support; CMake fits your build system.
  • The Universal2 section explicitly includes x86_64, x86_64h, and arm64, with a Ninja build and installation flow.

The license is not a single BSD-only term: it is BSD or GPLv2, so the distribution model still requires internal review.

  • README opening: open-source dual BSD OR GPLv2 licensed C library
  • Quote: Zstandard supports streaming compression, decompression and large files
  • Build instructions: `make` is the main build system of this project
  • Support for Fat (Universal2) Output: Apple Silicon (M1/M2) as well as Intel
  • README command: cmake -S . -B build-cmake-debug -G Ninja -DCMAKE_OSX_ARCHITECTURES="x86_64;x86_64h;arm64"
cmake -S . -B build-cmake-debug -G Ninja -DCMAKE_OSX_ARCHITECTURES="x86_64;x86_64h;arm64"
Not stated in the README:The README does not state the exact compatibility or ABI policy for the C/C++ APIs you need.;The README does not describe option differences between your build system and CMake.;The README only states the dual-license form and does not determine the legal outcome for your distribution model.
Yes I need to batch-process .zst, .gz, .xz, and .lz4 files on Linux and want one command-line tool for compression and decoding; is the zstd CLI suitable rather than using it only as a library?
For: A Linux operations engineer batch-processing .zst, .gz, .xz, and .lz4 files and seeking one CLI instead of multiple format tools

Yes, because the README explicitly says the repository provides a command-line utility that produces and decodes .zst, .gz, .xz, and .lz4 files.

  • This meets the requirement for one CLI to handle several existing formats rather than exposing only a C library API.
  • The Build instructions identify make as the project’s main build system, and the CLI can be built from source; the basic compression, decompression, and installation path is relatively direct.
  • The project also supports configurable compression levels and --fast=#, allowing batch jobs to choose different speed-versus-ratio points.
  • However, handling related formats does not prove identical behavior for every format; the README does not enumerate all parameters, metadata-preservation rules, or exit-code semantics your scripts may require.

The fit is strong for a unified entry point and common conversions, but the available README evidence is insufficient for format-specific advanced features.

  • README opening: a command line utility producing and decoding `.zst`, `.gz`, `.xz` and `.lz4` files
  • Build instructions: `make` is the main build system of this project
  • Quote: Speed vs Compression trade-off is configurable by small increments.
  • Quote: The negative compression levels, specified with `--fast=#`...
make
Not stated in the README:The README does not list the complete CLI parameters or metadata-preservation behavior for each format conversion.;The README does not specify all exit codes and error-output formats that batch scripts may depend on.
It depends I maintain a cross-language messaging protocol where senders will use zstd's C library and receivers are not in C/C++; if I need a stable format and interoperability across independent implementations, is zstd suitable as the compression layer?
For: A distributed-systems engineer maintaining a cross-language messaging protocol and seeking interoperability between the C reference implementation and other languages

It depends: the format-level interoperability foundation is strong, but the README does not confirm a mature compatible implementation for your receiver language.

  • The README says the zstd format is stable, documented by RFC8878, and already has multiple independent implementations, all positive signals for a cross-language messaging protocol.
  • The repository itself provides a C reference library; for other languages, the README points to a ports-and-bindings list on the Zstandard homepage rather than promising complete in-repository bindings.
  • For streaming messages, the project supports streaming compression and decompression, but receivers still need correct handling of frame boundaries, partial input, and errors; the selected README text does not define your protocol wrapper.
  • The license is BSD or GPLv2. Format availability and enterprise distribution licensing are separate decisions, so an open format alone does not remove license review.

The format is worth considering, but binding quality, version synchronization, and protocol error semantics remain decisive unknowns.

  • Quote: Zstandard's format is stable and documented in RFC8878.
  • Quote: Multiple independent implementations are already available.
  • README opening: a reference implementation ... C library
  • Quote: a list of known ports and bindings is provided on Zstandard homepage
  • Quote: open-source dual BSD OR GPLv2 licensed C library
Not stated in the README:The README does not state the maintenance status, version synchronization, or error handling of the binding for your target language.;The README does not define how your messaging protocol should wrap zstd frames, handle truncation, or negotiate dictionaries.;The README does not reach a licensing conclusion for your enterprise distribution model.

✨ Highlights

  • zstd 1.5.7 -1 compresses at 510 MB/s and decompresses at 1550 MB/s
  • The format follows RFC8878 and provides both a C library and the zstd CLI
  • Training mode uses a dictionary to improve compression for roughly 1KB records
  • Make, CMake, and Buck can build both libzstd and zstd

🔧 Engineering

  • The zstd CLI can produce and decode .zst, .gz, .xz, and .lz4 files
  • libzstd targets real-time compression with fine-grained speed-versus-ratio control
  • zstd --train generates a dictionary for both small-data compression and decompression

⚠️ Risks

  • The README benchmark depends on a Core i7-9700K, Ubuntu 24.04, and the Silesia corpus
  • A dictionary must be trained first and loaded during both compression and decompression
  • The README explicitly states that there is no universal dictionary for all data types

👥 For who?

  • Linux teams needing a C library or zstd CLI for real-time files and streams
  • Services processing many small records that can maintain dictionaries per data type
  • C/C++ projects integrating libzstd through make, CMake, or Buck