FR
live

Google's HTTP/2 codec costs Envoy 20% throughput — the performance regression nobody profiled

Apoxy's engineering team measured a 20% HTTP/2 throughput loss per core after Envoy switched to Google's oghttp2 codec. The investigation shows the bottleneck is in the bookkeeping layer above the Huffman decoder, not the decoder itself.

A rack server with one blade glowing slightly more amber than the others, suggesting abnormal CPU load

August 4, 2026. Dmitry Ilyevsky, co-founder and CTO of Apoxy, published a performance investigation that will be remembered in the Envoy ecosystem. A routine version bump cost a customer 20% of their HTTP/2 throughput. The cause was neither traffic shape nor config drift — it was the default HTTP/2 codec that Envoy changed in v1.34.

Envoy v1.34 switched from nghttp2 to oghttp2, Google’s C++ codec from the QUICHE family. The regression was confirmed by multiple users, and Envoy v1.37.0 reverted the default back to nghttp2. But Apoxy didn’t stop there. They profiled both codecs across four microarchitectures and published the flamegraphs. What the investigation reveals changes how we think about HTTP/2 proxy performance.

It’s not the Huffman decoder

The initial intuition blamed oghttp2’s Huffman decoder. HPACK (RFC 7541) compresses HTTP/2 headers with a canonical Huffman code frozen into the RFC a decade ago. Every request crossing a proxy gets decoded and re-encoded — hundreds of thousands of operations per second per core on an edge proxy. If the decoder is slow, the proxy falls over.

nghttp2 has used a precomputed finite-state machine for Huffman decoding since 2014. With 4-bit chunks, the state table is 16 KiB — it fits in L1 cache. Two lookups per input byte, zero conditional branches. It’s quiet software engineering at its best, and it profiles dead flat.

So the team expected to find the problem in QUICHE’s Huffman kernel. The flamegraphs said no. oghttp2’s decoder is slower, yes, but the bulk of the difference sits above it: the HttpHeaderBlock class and the layers of representation and bookkeeping each decoded header passes through. That overhead is a consistent 20–30 µs per request, on every host measured.

Four architectures, one verdict

Benchmarks ran on Intel Sapphire Rapids, AMD Zen 4, AWS Graviton4, and Google Axion. The protocol is reproducible: h2load → Envoy → Go h2c backend over same-host loopback, every process pinned to disjoint physical cores, Envoy at --concurrency 1. RPS/core measures pure CPU efficiency.

nghttp2 beats oghttp2 by 15–25% RPS/core on header-heavy proxied traffic, and by 7–19% under heavy connection churn. The regression is real, portable, and silicon-independent.

A side note worth internalizing: if you build Envoy yourself, check your optimization flags. Envoy’s .bazelrc does not default to -c opt. A plain bazel build produces a fastbuild (debug) binary that benchmarks 15–32× slower than the same source at -c opt. Apoxy found this out the hard way.

What this means for proxy operators

The immediate fix is simple: stick with nghttp2. Envoy v1.37.0 made it the default again, and a comment in the source promises to try again “once performance aligns with nghttp2.” But Apoxy’s investigation points to a deeper problem.

oghttp2 shares code with Envoy’s HTTP/3 stack. That’s the unification argument: one codec for HTTP/2 and HTTP/3, reduced maintenance surface, fewer divergence bugs. The measured regression shows that unification has a cost, and that cost isn’t in the algorithm — it’s in the software architecture wrapping the algorithm.

For proxy operators handling tens of thousands of requests per second per core, 20% less throughput means 20% more servers. The codec choice isn’t an implementation detail — it’s an infrastructure cost line item.

Verdict

If you run Envoy in production today, stay on nghttp2. The switch to oghttp2 delivers no measurable benefit and costs you throughput. If you’re on ≥ v1.34 and didn’t explicitly choose the codec, check your configuration: the regression is silent — your CPU graphs just go up, that’s all.

Apoxy’s investigation ends with a promise: a follow-up post where they make the fast codec even faster. Until then, the verdict is clear. nghttp2 is Envoy’s production codec.

References

The cyber brief, every Tuesday

The flaws that matter and the patches to apply, in a ten-minute read.

No spam. One-click unsubscribe.
read next

On the same topic

← Back to the feed

Type at least two characters.

navigate open esc dismiss