Tailscale unearths a sixteen-year-old SQLite bug that corrupted its databases silently
Over six months, Tailscale hit nineteen corruption incidents caused by a sixteen-year-old data race in SQLite’s WAL-reset path. If you self-host SQLite-backed apps, check the bundled version and stay on the beaten path.
August 12, 2026. Tailscale publishes the account of a six-month investigation. Nineteen database corruption incidents, one bug at least sixteen years old, and a fix shipped twice — the first attempt pulled almost immediately. The culprit was not in Tailscale’s code but in SQLite, the “boring” database that underpins much of the web and most of your homelab.
The story matters because it changes what “reliable” means. SQLite is everywhere — phones, browsers, and especially the apps you self-host: Immich, Paperless-ngx, Gitea, Vaultwarden, Nextcloud. A bug of this magnitude, sleeping for sixteen years, is a reminder that even the most battle-tested software can hide a fault that only an unusual workload will ever wake up.
An outage nobody could explain
Tailscale’s control plane is split into shards — independent coordination servers. Each shard holds a single SQLite database, accessed by a single Go process, which is exactly the single-writer model SQLite was built for. Since 2022, the team has backed up each database by snapshotting the whole file every few minutes and shipping it to an S3 bucket.
In August 2025, a pipeline that reads those backups reported an error. A PRAGMA integrity_check confirmed it: the database was corrupted. Corruption is “possible” in theory, but vanishingly rare in practice. The team repaired the database, investigated, found nothing — and then it happened again. And again. And again. Nineteen times in six months.
What made the bug so slippery was its total lack of regularity. No common factor between incidents: not the shard, not the customer, not the feature, not the time of day, not the load. Sometimes hours between failures, sometimes six weeks of deceptive calm — including a lull from October to December, before it returned as a Christmas present. The bug could not be reproduced synthetically, so the team had to deploy passive forensic telemetry in production and wait to catch it red-handed.
The mechanism: a race between checkpoint and write
To understand the bug, you need to know how SQLite writes. A database is made of pages. In WAL (Write-Ahead Logging) mode, new pages are not written directly to the main file: they go first to the WAL, a journal. Periodically those pages are copied from the WAL back into the main file — a checkpoint.
Tailscale was doing something unusual: it drove checkpoints manually, at an aggressive pace, to get fast and consistent backups. That choice — documented and supported — still pushed them off the beaten path, and that is exactly what made them vulnerable.
The SQLite maintainers built a custom debugging tool, the tmstmpvfs shim, which wraps the virtual-filesystem layer to trace every write. Tailscale funded it and ran it in production. At the next incident, the logs spoke: a rare data race between a checkpoint and a write transaction.
Specifically, if a write lands at a particular moment during a checkpoint, the checkpointing process convinces itself that some pages have already been copied from the WAL into the database — when they have not. Those pages are never written, and their data is permanently lost, with no error raised. Worse, other pages that reference the missing ones — an index, say — do get written, leaving the database inconsistent. The maintainers named it the “WAL-Reset bug” and estimate it had been in the code for at least sixteen years.
The fix, then a false alarm
The fix shipped in SQLite 3.52.0: one extra check in the checkpointing function that detects when the WAL has been reset by another thread. Tailscale rolled it out carefully, starting with canary shards.
The backup monitor promptly turned red: thirteen databases flagged as corrupted. False alarm. Those databases were fine — they had hit a second bug, the stale expression index problem. Tailscale stored high-precision timestamps as text, converted to floats in a generated column; the 3.52.0 release had, in passing, subtly changed the rounding of text-to-float conversions. The result: indexes whose values no longer matched their column, reported as corruption by mistake.
The lesson is double. SQLite withdrew version 3.52.0 and published 3.51.3, containing only the WAL-Reset fix. Then 3.53.0 added a self-healing index mechanism. On its side, Tailscale dropped its timestamp precision to whole seconds — a text-to-integer conversion with no ambiguity. Final proof came two months later: an alert confirmed the exact conditions of the bug were occurring in production, without corrupting anything. Since then, four months without an incident.
The recovery pipeline that surfaced the bug
While hunting the root cause, Tailscale still had a live platform to run. It built a transaction-logging pipeline: every SQL statement that modified a database was streamed to a separate log file. Because SQLite is single-writer with serializable transactions, that history was linear and deterministic — something impossible in a multi-writer database like PostgreSQL. Replaying those transactions against the last known-good backup could restore the database to its latest state, safely skipping the corruption.
The pipeline did more than recover data: it produced the decisive clue. In two incidents, the logs failed to replay cleanly, and data written by one transaction became invisible to later ones. A write had vanished without raising an error — the anomaly that eventually pointed to the checkpoint race.
Why your homelab should care
You probably don’t drive checkpoints by hand, and you’re not doing millions of writes a minute. The odds that you trigger the WAL-Reset bug are tiny: it’s a rare race that SQLite had to provoke artificially in tests to observe at all. But Tailscale’s post holds three lessons that apply directly to you.
First, the version. Most self-hosted apps bundle SQLite statically inside their Docker image. You don’t “upgrade SQLite”; you upgrade the app. Checking the bundled version is not a luxury:
sqlite3 --version
# 3.51.3 minimum for the WAL-Reset fix; 3.53.0 adds index self-healingSecond, default configuration. The investigation’s core message fits in one line: running boring technology in a non-standard way is a risk. WAL mode, automatic checkpointing, and the single-writer model are massively tested; stepping away from them, even with documented options, puts you on a path few people walk.
Third, integrity checking. Tailscale would never have seen the corruption without a monitor running PRAGMA integrity_check over its backups continuously:
sqlite3 /path/to/my.db "PRAGMA integrity_check;"
# expected: "ok"A backup that is corrupted but never restored is false security. Test your restores, not just your backups.
Verdict
If you self-host SQLite-backed apps, two steps are enough: keep your images current so they bundle SQLite ≥ 3.51.3, and don’t touch checkpoint or WAL settings without a documented reason. Add a periodic integrity_check on your backups — it costs seconds per database.
If you run SQLite in production with manual checkpoint control — Tailscale’s exact profile — move to 3.51.3 or later immediately, and question that design decision: the speed gain is paid for in silent-corruption risk.
The underlying signal: SQLite’s reliability is not undermined by this hunt; it is reinforced. But that reliability rests on a contract — standard configuration, single writer, integrity checks. Step outside the contract, and you become the tester.