AWS Lambda rewrites its network flow logging in eBPF and Rust to survive microVM density
Lambda engineers detail how they replaced an iptables-based packet counter, unable to scale to IPv6, with an eBPF and Rust pipeline that logs every flow across thousands of Firecracker microVMs per host. The lesson outlives AWS: at multi-tenant density, network bookkeeping must be nearly free and provably correct.
September 11, 2026. Prashant Kumar Singh and Kshitij Gupta, two AWS Lambda engineers, publish a technical retrospective on how the platform logs every network flow that crosses its hosts. The framing number: a single Lambda worker is a bare-metal EC2 instance packed with thousands of microVMs, each an isolated Firecracker guest, all talking to S3, other AWS services, the public internet, and the customer’s VPC. Something has to keep an honest record of that traffic. The post explains why the old system ran out of road and how a rewrite in eBPF and Rust held the scale. The lesson is for anyone operating observability on a dense platform: at this density, every megabyte of RAM and every microsecond of CPU counts twice.
A record you are not allowed to falsify
A network flow log is the system of record for what happened to a packet: investigation, incident response, audit, reconstructing what a workload did. The same records feed billing and metering services that demand accuracy above all else. Everything must be persisted for audit and compliance.
Two properties dominate the rest. The first is correct attribution: every packet, every flow, must be tied to the microVM where it landed and the tenant that produced it. The second is completeness: no missed packets. A lost or misattributed record causes billing errors and observability gaps — and that, for every microVM, when Lambda serves millions of requests per second. Overhead matters for a reason invisible to the customer but central to the operator: at Lambda’s density, every extra megabyte of RAM and microsecond of CPU eats into utilization, operating margin, and the ability to serve under load.
Why the old system ran out of road
While building the new multi-tenant, Firecracker microVM-powered Lambda, the team first reused a system inherited from the older, looser single-tenant EC2 era. Two parts: a kernel-side extension that counted packets and matched them to tenants via rules, and a userspace daemon that read the counters, aggregated them into records, serialized them, and uploaded the files. The system worked for a small number of VMs. It broke at Lambda’s density, for two mutually exclusive reasons.
The first is rule explosion. iptables walks its rules more or less linearly for every packet, and each new microVM piles more rules onto the chain. A worker hosting two thousand microVMs needed more than a hundred thousand iptables rules just to keep the record. Every packet paid a tax proportional to how crowded the host was — exactly the wrong direction, since the whole point was to pack more microVMs onto a host.
The second reason is harder: the borrowed kernel module did not speak IPv6. A record that cannot see half the address space is not a record you can trust, and the moment dual-stack IPv6 support was proposed for Lambda, the old approach was finished, whatever the performance tuning. As the post puts it: a record blind to half the network is not a record at all.
Three pieces, one chain
The rewrite is an assembly of three cooperating components, each with a sharp role.
At the bottom, the kernel capture layer: a set of small eBPF programs attached to the traffic-control (tc) hook of each network’s virtual devices. They intercept packets and emit one compact event per packet into a ring buffer. They only watch — no code path lets them copy, block, drop, or rewrite a packet.
In the middle, the tagger: an unprivileged userspace process written in Rust, one per network. It drains its own ring buffer, rolls raw events into per-flow records, and writes them to disk in the inherited Amazon Ion format. On top, the orchestrator: one privileged process per host, owner of everything requiring elevated permissions — loading the eBPF programs, wiring up traffic control, supervising the fleet of taggers. It exposes a small lifecycle API over a Unix socket so the control plane can create, reassign, and tear down tagging as microVMs come and go.
The decoupling does the work: capture in the kernel, aggregation in userspace, and one process per host to orchestrate it all. The produced records flow to the downstream consumers unchanged.
Capturing in the kernel without getting in the way
The capture programs attach to the clsact qdisc in traffic control, on the ingress and egress side of each device. A network spans two devices, so that is four attach points per network. Each program reads the packet and returns the “keep going” action: no customer packet is ever modified or dropped, and nothing adds meaningful latency.
For each packet, the program walks the headers — Ethernet, then IPv4 or IPv6, then TCP, UDP, or ICMP — and writes one fixed-size event, about 24 bytes for IPv4:
/* one event per packet, ~24 bytes for IPv4 */
struct flow_event {
u8 ip_version; /* 4 or 6 */
u8 protocol; /* TCP / UDP / ICMP */
u8 direction; /* ingress or egress */
u8 device_id; /* which of the network's devices */
u16 local_port; /* "local" is always the sandbox side */
u16 remote_port;
u32 flags_and_bytes; /* TCP flags in bits [31:24], bytes in [23:0] */
u32 local_addr; /* 16 bytes for IPv6 */
u32 remote_addr;
u32 received_time_ms;
};One packet, one event, one byte count. The TCP flags and the byte count share a single 32-bit word: eight bits of flags on top, a 24-bit byte count beneath. Doing less work per packet in the kernel is the entire point — aggregation is somebody else’s job.
Getting this logic past the eBPF verifier took work. The header parser became a shared subroutine, proven once instead of re-inlined at every attach point. The IPv6 extension-header walk is bounded to a fixed number of hops so the verifier can guarantee termination. Fragments past the first report zero ports and flags rather than vend garbage. And super-packets coalesced by GSO/GRO have their byte counts handled correctly — a wrong port or an over-counted byte would create a false log entry.
The verifier is not the only gate. This code produces a record critical to billing, compliance, and audit, so each eBPF program is written in C and run through a formal model checker (CBMC) on every build. Its harnesses assert, among other things, that the event struct’s byte layout stays compatible with what every attached program expects — a silent one-byte shift is the kind of bug that corrupts every record without anyone noticing until they need the log.
Sizing the ring buffer by calculation
The ring buffer is the one thing the kernel producer and the userspace consumer share, and its size is a real tradeoff. Too small, and you drop events under a burst — a hole in the log at the worst moment. Too large, and you waste memory, paid on every ring buffer on the host.
The team did not guess: they derived a floor from each microVM’s maximum packet rate. For a ceiling of 100,000 packets per second per direction and a drain every 100 milliseconds:
ring ≈ 62,500 pps × 0.1 s × ~24 bytes × 2 directions ≈ 300 KBThe ring buffer API requires a power of two, so the default is 512 KiB. That is the smallest buffer that cannot overflow between drains at the guest’s own maximum rate — in other words, the floor guarantees a guest cannot outrun the recorder, even while trying. The current deployment provisions more generously, on the order of a few megabytes, while per-workload tuning continues. The number to defend is the floor, and it comes from a hard system limit, not a guess.
The drain cadence has one more property: waking a process is not free, and a fleet of thousands of processes all waking constantly would thrash the CPU. So the kernel decides when to bother. It only forces a wakeup once the ring crosses about 1% full; below that it stays quiet. On the userspace side, the tagger does not read more than once every 100 milliseconds. A quiet flow just sits there until the next drain, basically free. A busy one trips the threshold and is read almost immediately. Nothing is on a fixed timer, so neither case gets the timing wrong.
Aggregating in Rust
The choice of Rust comes down to boring, practical reasons. At this density, thousands of taggers run per host, each holding a small amount of state that has to be right. A garbage-collected runtime would impose pauses and memory that balloons under load, and a pause in the wrong place would create a gap in the record. Rust gives predictable memory, no collector, and a compiler that flat-out refuses to build whole classes of misattribution bugs. Each tagger runs in a few hundred kilobytes of RAM, against a budget of about one megabyte. That is what makes thousands of them per host affordable.
Inside, it is a small set of cooperating tasks on a single-threaded async runtime: one task reads the ring, another owns the flow state, a third writes parcels. The only work fenced off onto a blocking pool is the couple of operations that genuinely block — receiving the ring descriptor and serializing Ion. The tagger reads the ring via epoll: it sleeps when there is nothing, and wakes when there is.
As events arrive, the tagger drops them into a flow map keyed by device, the five-tuple, and an attribution handle handed over by the control plane. Matching events accumulate bytes, packet counts, and OR’d TCP flags. Aggregation happens at read time, so the hot path stays a lookup and an add.
What this changes for you
Lambda’s retrospective is portable to anyone operating telemetry at scale.
- Replace rule-based counting with passive capture. iptables did not hold the density; an “observe-only” eBPF hook removes the per-packet tax and the risk of touching customer traffic.
- Treat correctness as a build requirement. Running CBMC on eBPF C code shows that a billing record deserves a proof, not a hope.
- Size your buffers by calculation, not by feel. The 512 KiB floor follows from a maximum rate and a cadence, not intuition — the only way to guarantee no customer can outrun the recorder.
- Choose Rust for high-density state. The absence of a collector and predictable memory are what make thousands of processes per host economically viable.
The post’s conclusion is not a slogan, it is an engineering discipline. Observability at a cloud provider’s level runs on records whose cost nobody sees while they are correct, and everyone discovers the day they are not.
Verdict
Lambda’s flow-logging rewrite is a case study in replacing a legacy system with an eBPF + Rust pipeline — not for novelty, but because iptables-based counting was structurally incapable of moving to IPv6 and surviving microVM density.
If you operate network observability on a dense fleet, take three things: passive kernel-side capture instead of exploding rules, a buffer size derived from a system limit, and a memory-predictable language for per-process state. If you run a multi-tenant platform, add formal proof on the code that produces your billing records. The cost of a wrong record never shows up when you write it — it shows up the day you need it.