AWS’s AI agent claims 89% of bugs fixed, but repairs the targeted flaw in only 37.8% of cases
AWS Continuum, generally available since October 22, 2026, posts 89% success on the CyberGym-E2E benchmark, versus 65.9% for the previous public record. The headline hides a limit: the agent repairs the specific historical vulnerability in only 37.8% of cases, on a benchmark AWS ran itself.
October 5, 2026. AWS publishes the benchmark for its Continuum agent: 819 of 920 tasks passed, an 89.0% end-to-end success rate on CyberGym-E2E. October 22, 2026. The service goes generally available in us-east-1 and us-west-2. October 24, 2026. The specialized press unpacks the number and finds the fine print: the agent repairs the precise flaw the benchmark was targeting in only 37.8% of cases.
For a CISO or application-security lead, the lesson fits in one sentence: a benchmark score is only worth what its definition of “fixed” is, and here the headline number and the metric that matters do not tell the same story.
What Continuum actually does
AWS Continuum for code vulnerabilities is a service that finds and validates code vulnerabilities, then delivers fixes as automatable patches or pull requests. It confirms whether a scanner finding is exploitable in your environment, ranks it by business impact, and proposes a fix for each one. A collector reads AWS Config resources and network configuration to judge whether a flaw is reachable; a context graph identifies the critical systems — customer data, payment pipelines, authentication services — and ranks each issue by how close its attack path runs to them.
Every fix requires human approval: Continuum opens a pull request, and the team reviews and merges it. The company promises that, over time, guardrails will allow some remediation classes to be automated once confidence is established. The positioning is clear: this is not an autonomous agent shipping code, it is a triage assistant that shrinks the findings backlog.
The benchmark and its fine print
CyberGym-E2E gives an agent 90 minutes to find and fix a vulnerability in one of 139 open-source projects, with no bug description, no proof of concept, no crash log, and no existing patch. The agent must identify the flaw, trigger it, and produce a fix that stops the crash without breaking existing tests. Every task involves memory-safety bugs in C or C++ — a scope AWS explicitly acknowledges, and one that excludes other languages and other vulnerability classes.
Grading runs in four cumulative stages. Continuum found and reproduced a bug in 92.5% of tasks, stopped the crash in 89.6%, and kept the tests passing in 89.0%. The fourth stage checks whether the patch also fixes the historical vulnerability the benchmark had selected up front. There, the score drops to 37.8%, against 26.2% for the previous public record.
The trick is subtle and legitimate: the agent is never told which precise flaw the benchmark targets, it only has to repair a real flaw. A repository can hold several, and fixing a different one does not make the work worthless — which partly explains the gap between 89% and 37.8%. But the question nobody has yet answered is the one a customer cares about: how does Continuum perform when a team hands it a specific vulnerability to fix?
A market where every vendor publishes its own numbers
The Continuum episode is part of a larger trend. Since AI accelerated vulnerability discovery — the PHP2Shell case in October 2026 surfaced a critical PHP flaw found ahead of the maintainers — vendors are fighting over the automated-fix segment. Google, Microsoft, and several startups sell assistants that turn a scanner finding into a reviewable patch, and each publishes its own benchmark, on its own corpus, with its own definition of success.
The problem is structural: without a trusted third party or a common corpus, these numbers are not comparable. CyberGym-E2E has the merit of being public and simulating a realistic task — find and fix a flaw with no prior information — but it measures what the vendor chooses to measure. As long as benchmarks remain self-administered, Continuum’s 89% and a competitor’s X% say nothing about their relative value for your application portfolio.
The caveats weighing on the number
Three limits temper the 89%. First, AWS ran the benchmark itself: the protocol and tools are public, but the published results are its own, and no third party has reproduced the performance yet. The company says it followed the rules, including the internet-access limits, and checked that the agent was not simply copying already-public fixes. It adds that with more time, the score rises to 93.7%.
Second, the scope is narrow: only memory-safety bugs in C/C++, across a corpus of 139 projects. A portfolio of Python, Java, or Go applications, or business-logic vulnerabilities, gets no guarantee the number transfers.
Third, the timing raises questions. Corey Quinn, chief cloud economist at Duckbill Group, speculated on X that the announcement may have been pulled forward from a re:Invent keynote, without offering evidence. Matt Garman, AWS CEO, counters with a score “higher than any other AI agent” on the test and calls AI “genuinely transformative for defenders.” The benchmark published on October 5, more than two weeks before general availability, gave the number time to circulate before the service became billable.
How to evaluate the tool without trusting the benchmark
Rather than ruling on a self-declared number, ask the vendor three questions before a trial. First: does the fix repair the flaw you reported, or a flaw found along the way? That is the difference between 89% and 37.8%, and it is measurable in a few days on your own repositories. Second: does the benchmark cover your languages and vulnerability classes, or only memory-safety in C/C++? Third: who ran the benchmark, and has an independent party reproduced the score? A two-week trial on a critical repository answers those three questions faster than any analysis of the headline figure.
Verdict
If your fleet is mostly C or C++ and you are drowning in memory-safety findings, Continuum deserves a trial: a tool that genuinely finds and reproduces a flaw in 92.5% of cases and then delivers a readable patch cuts manual triage, even if the final decision stays human.
If your code is written in memory-safe languages or your vulnerabilities are application logic, do not carry the 89% over: the benchmark does not cover your case, and no one but AWS has validated the figure yet. Demand an evaluation on your own repositories against your own baseline — ideally the fix rate for the flaw you asked it to fix, not a flaw found at random. That is the only metric that matters for a remediation program, and it is precisely the one that tops out at 37.8%.