AI crawlers burn 20% of git.kernel.org CPU scraping commits one by one
On August 29, 2026, Konstantin Ryabitsev published hard numbers on AI crawler traffic to the Linux kernel repositories: six million requests a day, a fifth of CPU capacity, and barely 2% legitimate traffic. For any public self-hosted service, it is proof that a proof-of-work challenge only moves the problem.
August 29, 2026. Konstantin Ryabitsev, one of the operators of git.kernel.org, publishes “Creepy crawlies,” a post that finally puts numbers on something he has been complaining about for a year. 6 million daily requests demand random commits. 66% are batted away by the Anubis challenge, but 33% now solve it. Of 90 cores spread across 5 nodes, 14 to 16 do nothing but render commits as HTML for scrapers. The Linux kernel — the largest software project on Earth — spends a fifth of its hosting capacity feeding AI models that will not even bother to clone.
A post that measures what everyone is feeling
Ryabitsev’s conclusion fits in one line: we spend more CPU cycles rendering commits as HTML for scrapers than on all other legitimate access combined, including git clones. The post ships graphs to prove it — crawler-versus-clone traffic share, the Anubis difficulty curve, the request funnel. This is no longer a tired sysadmin’s impression; it is a measurement.
The most telling figure is the legitimate-traffic share: under “a bunch of generous assumptions,” about 2% of git.kernel.org requests come from real developers. The remaining 98% are scrapers. And those scrapers use, in his words, “the stupidest possible way of doing it”: instead of cloning the repositories and walking the history locally, they render every commit as HTML through cgit and then parse it.
The rational method exists and is trivial:
# The efficient way: clone the whole history in one shot
git clone https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.gitA single git clone gives you the entire history. But the scraper prefers to generate billions of valid URLs — cgit can produce “1.2 metric bajillion” addresses for a single fork of linux.git, all distinct, all expensive to render.
Why the kernel is a goldmine for AI
The reason for this obsession is subtler than it looks. linux.git holds about 1.48 million commits, and git.kernel.org hosts 922 forks of the repository. For an LLM, this history has a particular value: it is guaranteed pre-AI content, unpolluted by the models’ own output.
Ryabitsev makes the risk concrete with a precise image: training a model on model-produced content is like inoculating it with “the equivalent of a digital prion disease.” A source you can prove is LLM-free — like the entire kernel commit history — is worth its weight in gold as training data. Hence the bots’ persistence, and their willingness to pay the Anubis toll.
The arms race, step by step
The defense followed a familiar arc. First, fail2ban on obvious user-agents. The bots responded by dressing up as ordinary browsers. Then IP bans, then whole-ASN bans — justified when a Google Compute address pretends to be a Firefox user. The bots then migrated to millions of residential and mobile IPs, each sending 4–5 requests before vanishing from the logs.
That is the economics of proxy SDKs: apps — including, the post notes, some smart TVs — monetize their network access by reselling residential traffic. Banning those IPs is pointless: they never come back. “They descend like swarms of locusts, hit hard until the system falls over, then move on to the next target,” Ryabitsev writes.
Against this, the team deployed Anubis, a proof-of-work challenge: compute a string that, combined with the visitor’s IP and a secret, yields a sha256 with leading zeroes. Difficulty 4 worked at first. The bots solved it. Raised to 5. A few months later the bots are solving 5, even as real users’ phones warm up crunching the math. That is arms-race logic: every step costs both sides, and the side that can absorb the cost wins.
What this means for self-hosters
The lesson reaches far beyond kernel.org. Any public service exposing enumerable URLs — a cgit mirror, a forge, a mailing-list archive — becomes a scraping target sooner or later. The answer is not to close the service but to shrink the surface: disable expensive renders, gate costly anonymous endpoints, and accept that part of your capacity now serves as an invisible tax on AI training.
Ryabitsev is blunt about the consequence: the team will “turn off features” to cut the number of crawlable URLs and gate expensive actions for anonymous access. Users will lose some convenience, regretfully. The data will stay available for download — “you just may have to jump through more hoops to get it.” It is a forced choice: between access convenience and service survival, the operator no longer has the luxury of hesitation.
What self-hosters can do
Ryabitsev is blunt: there is no simple fix. But a few measures reduce the pressure, roughly in order of effectiveness.
- Shrink the URL surface. Turn off expensive renders and costly anonymous views — cgit, mirrors, arbitrary diffs — to close the tap instead of bailing water.
- Charge a cost. A proof-of-work challenge in front filters most bots at the price of some human friction; it is a truce, not a solution.
- Serve static content. Where possible, serve snapshots or pre-rendered archives instead of generating pages on the fly.
- Accept the asymmetry. robots.txt is useless against bots impersonating browsers; bet on the marginal cost you impose on the scraper, not on politeness.
The real lesson is budgetary: 20% of a public infrastructure’s capacity now serves as an invisible tax on model training. Budgeting for that load — CPU, bandwidth and defensive complexity — belongs in the sizing of any exposed service, right alongside peak traffic. The tax is not optional: an operator who ignores it will simply absorb it silently, watching legitimate latency climb while the graphs look normal.
The deeper cost is harder to see. Every cycle spent rendering commits for scrapers is a cycle not spent serving the human developers who keep the kernel alive. The 2% figure is the clearest measure of that inversion: infrastructure built for an open community now mostly feeds machines that contribute nothing back. When a mirror or an archive closes its anonymous access, it is not punishing anyone — it is protecting the minority of real users from a majority of silent consumers.
There is no end state where the operator wins. The scrapers have an economic incentive to keep up with whatever cost is imposed, and residential proxy networks give them a near-unlimited supply of IPs. The realistic goal is to raise the cost high enough that the swarm targets someone else — not a solution to the problem, only a migration of it. Every public archive that hardens itself is, in effect, asking the swarm to find a softer target.
Verdict
If you run a public code or archive service, measure your crawler traffic share first — you will likely find a comparable “background radiation.” Put a proof-of-work challenge in front, but treat it as a truce, not a solution: it shifts the load, it does not remove it.
If you consume data for training, the lesson is the reverse: clone, do not scrape. A git clone is faster, more reliable and far cheaper for the source. The “stupid” commit-by-commit scrape is also the most toxic — it is exactly what pushes operators to close what used to be open.