FR
live

GitHub was down for 7 hours 47 minutes on August 17, and the outage was a capacity failure, not a bug

On August 20, 2026, GitHub’s CTO published the post-mortem for the August 17 outage: 7 hours and 47 minutes of downtime caused by an infrastructure component that failed to scale with load, as monthly commits climbed from 1.4 to 2.9 billion since April. The lesson for SRE teams fits in two words: retry budgets.

A dark server rack in a data center, a single amber warning light lit on one server’s front panel.

August 17, 2026. GitHub goes down for 7 hours and 47 minutes, disrupting github.com, authentication, Actions, APIs, pull requests, issues, and Copilot. August 20, 2026. Vlad Fedorov, the CTO, publishes the official post-mortem. Between April and August 2026. Monthly commits climbed from 1.4 billion to 2.9 billion.

The post-mortem’s conclusion fits in one sentence, and it is uncomfortable: neither August outage was caused by a code or configuration change. Both were capacity failures.

A capacity failure, not a software incident

GitHub’s account is unusually blunt. The investigation found the outage began when traffic reached a new peak, and a critical infrastructure component in the Central US data center failed to scale with it. The resulting capacity pressure spread through the systems, causing authentication failures and disrupting multiple services.

It was the second significant incident of the month, after an Actions failure on August 6. And it was not the first time GitHub has spoken about this: in March and April 2026, the company detailed its reliability workstreams. The August 20 message is that they must be accelerated.

The diagnosis is owned: GitHub did not scale critical components before demand exceeded their capacity. Monthly commits have doubled since April. That growth explains the pressure — but it does not excuse the outage. The distinction matters, because it shifts responsibility from an “unpredictable bug” to “insufficient sizing,” which is an engineering decision.

What the recovery phase reveals

Recovery required several coordinated actions: rerouting traffic, isolating affected infrastructure, and restoring services in stages. Most services recovered that day, but some Copilot services took longer.

That is where the post-mortem offers its most concrete lesson. Errors in those services triggered a client-side retry loop that increased traffic during recovery. The team had to neutralize that behavior before it could safely restore traffic.

This is the classic retry storm: under load, a service fails; clients retry en masse; the retries add load; the service fails harder. The outage feeds itself. GitHub’s immediate response was to apply consistent retry limits, retry budgets, and variable timeouts across service-to-service interactions, to prevent retry storms and cascading load.

The strategy: capacity, Azure, and isolation

The post-mortem lays out three priorities in flight since the start of the year: add capacity, improve efficiency, and remove architectural bottlenecks. The numbers are large: more than 3 million CPU cores added, 120 petabytes of high-speed storage, and significant network capacity. GitHub installed as much hardware as available power allowed in its data centers, while accelerating its migration to Azure.

The most striking shift is infrastructural: Azure now serves roughly 58% of the platform’s load and half of all Git operations, up from 12% of platform load in May. That expanded footprint has also absorbed the growth in Actions job runs.

Two more workstreams round out the picture. First, isolating critical systems and removing shared dependencies between them, to reduce the likelihood of an outage and limit its blast radius. Second, a promise of an architecture that scales read capacity linearly with the number of readers, starting with the largest monorepos — an admission that the current model plateaus on giant repositories.

Finally, the team is reviewing lower-priority CPU and memory alerts to identify components that could fail during a sudden spike. An ignored weak signal is an incident waiting to happen.

Verdict

If you build on GitHub, treat this outage as a supply-chain argument, not an anecdote: a platform that hosts your CI/CD is a critical component whose availability conditions your own. Anyone who depends on it heavily should have a fallback — repository mirrors, regular exports, and a deploy chain that does not stop dead when central authentication goes down.

If you operate services, the lesson is actionable this week. The root cause — a component that failed to scale with demand — and the amplifier — the retry storm — are both avoidable failures. Instrument your retry budgets and variable timeouts, and treat low-priority capacity alerts as pre-incident signals: when a service nears its ceiling, the time to act is before the spike, never during it.

If you want a single indicator: GitHub doubled monthly commits in four months, and its outage lasted 7 hours 47 minutes. The next time your team defers a capacity project to “later,” remember that growth does not warn you — it saturates you.

References

The cyber brief, every Tuesday

The flaws that matter and the patches to apply, in a ten-minute read.

No spam. One-click unsubscribe.
read next

On the same topic

GitLab patches a critical unauthenticated GraphQL code injection flaw (CVSS 9.4)

On August 18, 2026, GitLab released fixes for two vulnerabilities, including a critical code injection via a GraphQL directive (CVE-2026-19478, CVSS 9.4) exploitable remotely without authentication or user interaction, allowing attackers to modify or delete public projects. Every self-managed installation must upgrade immediately — GitLab.com and GitLab Dedicated are already patched.

← Back to the feed

Type at least two characters.

navigate open esc dismiss