Cloudflare transcodes its cache to Zstandard and saves petabytes of storage
On September 1, 2026, Cloudflare detailed Cache Transcoding, a prototype that compresses eligible assets with Zstandard directly inside Pingora. Encoding shrinks on-disk size to roughly a third, for a few points of CPU, and cuts bandwidth between data centers.
September 1, 2026. Cloudflare published an engineering post about a prototype called Cache Transcoding, which encodes eligible responses with Zstandard directly inside Pingora. 2016. The year Yann Collet open-sourced Zstandard at Facebook. 67.3%. The share of requests carrying compressible text in the traffic sample studied. Why it matters: memory and disk keep getting more expensive, and compressing the cache itself becomes a lever for savings at the scale of a global CDN.
The problem Cache Transcoding attacks
The post states the premise up front: memory costs are climbing sharply, and the price of both RAM and hard drives has exploded over the past year. Cloudflare runs several massively distributed storage products, including its famous CDN, that depend on using the memory already deployed efficiently enough to keep serving every customer.
The answer it prototyped is to compress objects at the moment they enter the cache, not just on the way out to the browser. When an eligible response is written to disk, Cache Transcoding encodes it with Zstandard before the write. The compressed form is kept for the whole time the object lives in cache and while it moves between data centers through Tiered Cache, then decoded just before it is served to the client.
In early testing, this encoding shrinks eligible assets to about a third of their original on-disk size. The estimated extra CPU cost on the origin-facing proxy is small. A modest bump in compute gives Cloudflare petabytes of effective cache capacity and cuts the data transferred between its data centers. The encoding is paid once when an asset enters the cache; the storage and bandwidth savings repeat every single time that asset is reused.
Why Zstandard over Brotli or gzip
Zstandard, or zstd, is a lossless compression algorithm developed by Yann Collet at Facebook and open-sourced in 2016. Lossless means that once compressed data is decoded, every byte is identical to the original. You can change how an asset is represented on disk without changing the asset itself.
zstd is designed to balance compression ratio against speed. In Cloudflare’s earlier browser compression testing, it compressed data 42% faster than Brotli while producing nearly the same file size, and produced files 11.3% smaller than gzip at a comparable speed. That balance is decisive: Cache Transcoding touches an enormous volume of traffic, so both encoding and decoding have to stay fast.
The prototype uses zstd level 3, which captures most of the compression benefit without turning cache fills into a CPU bottleneck. The post puts numbers on it: 4.31 ns per byte, roughly 232 MB/s, for encoding paid once per fill, and 1.56 ns per byte, roughly 641 MB/s, for decoding paid on every serve. Encoding costs more per byte, but an asset is served far more often than it is filled.
What is eligible, and what is not
Transcoding does not mean compressing everything. Images, video, and fonts are usually already compressed. In Cloudflare’s traffic sample, that media slice was 21.4% of requests but 63.3% of bytes. Recompressing it would burn CPU for nothing.
Compressible text is another story. HTML, JSON, CSS, and JavaScript made up 67.3% of requests and 22.3% of bytes. Within that text slice, about 71% arrived uncompressed, with no Content-Encoding, and compresses well. That is where Cache Transcoding pays off, with a controlled test corpus that compressed eligible assets by roughly 2.8 times.
The prototype only encodes a 200 OK response when Content-Encoding is unset, the Content-Type is compressible text, and the response has a known Content-Length of at least 4 KiB. Slice subrequests, responses using active upstream compression, range requests, precompressed responses, unknown-length bodies, and binary content stay unchanged. The 4 KiB threshold removes a large number of tiny requests while leaving out only about 1% of the otherwise eligible bytes.
The path of an object through the tiers
The architecture detail is worth understanding, because it determines where decoding happens:
- On a cache miss, the Pingora-based proxy encodes the body with zstd before writing it to disk. Cache metadata records that the stored representation is compressed and preserves the original content length. Before the response leaves the proxy, the body is decoded back to its original identity representation.
- On a cache hit, the stored zstd object is read and decoded. With Tiered Cache, the compressed representation moves from the upper tier to the lower tier in compressed form. Decoding only happens on the client-facing hop.
- On a full miss, the upper tier fetches identity bytes from the origin, encodes them once, stores them as zstd, and transfers them to the lower tier compressed. The lower tier also stores the zstd representation, then decodes it for the request.
A storage encoding marker prevents an object from being encoded more than once. A cache layer that receives an object from another tier sees it is already stored as zstd and keeps it in that form.
The chosen policy is also a lesson in restraint: limiting transcoding to popular content alone did not help, because decoding happens on every serve. Transcoding all eligible compressible text at or above 4 KiB captures nearly all of the measured benefit while staying within the CPU budget.
How the prototype was validated
The prototype was exercised against a controlled test zone, with every request correlated across request logs, Prometheus metrics, and Jaeger traces. The correctness campaign covered cache misses, hits, single-hop fills, and Tiered Cache fills, varying cache keys to force each path.
A performance campaign sent more than a million requests across 10 cache servers, half with Tiered Cache disabled and half enabled, to measure local cache behavior separately from transfers between tiers. The two test assets were about 195 KiB and 272 KiB, both compressing by roughly 2.8 times. That was deliberately a compressible corpus, which validates the architecture without representing every text object on the Internet: a broader corpus is still needed before treating that ratio as a fleet-wide constant.
Why this matters right now
The timing is not accidental. When RAM and disk prices spike, the economics of a CDN tilt toward density: squeezing more objects onto the hardware you already own beats buying more of it. Cache Transcoding is one of those rare optimizations whose cost, a few percent of CPU, is already the cheapest resource in a data center, while its benefit, petabytes of capacity and less backbone traffic, compounds on every cache fill. It is a prototype today, but the direction is unmistakable, and the engineering is documented closely enough for any large cache operator to reproduce the experiment on its own corpus.
Verdict
If you operate your own cache or CDN, the lesson transfers directly: encoding eligible text assets in zstd at cache entry, with a size threshold and a whitelist of content types, converts cheap CPU into storage capacity and cross-data-center bandwidth. Start by measuring how much of your traffic arrives without Content-Encoding.
If you are simply a CDN customer, there is nothing to configure: this is the kind of optimization that happens inside the provider’s infrastructure and shows up as better cache density and less frequent eviction of your content. But it is a reminder that level-3 compression on text remains one of the best cost-to-benefit ratios in cache engineering.