A Kubernetes backup is not disaster recovery
On September 10, 2026 two CNCF ambassadors published three reproducible failure scenarios that separate having backups from being able to actually recover. Test full restore into a cluster that has never run the workload, validate data against an expected result, and time the whole thing.
September 10, 2026. CNCF ambassadors Saiyam Pathak and Saloni Narang publish a Kubernetes disaster-recovery guide built on three failure scenarios reproducible on a laptop. Their thesis fits in one sentence: having backups is not being able to recover. Why it matters: most teams watch the Completed status of their backups, and almost none actually rehearse a restore.
The four layers of a recovery
For a recovery to count, four layers must come back: data, workload, cluster, and traffic. Each layer has mature tooling, and each usually recovers fine in isolation. Recovery fails at the joins between the layers: a restored cluster with no data, restored data with no traffic path, an application definition that provisions an empty volume. The guide’s three scenarios each break one join.
The lab is deliberately modest: two clusters (a production one and a recovery one that exists before anything goes wrong), an S3-compatible object store outside both clusters so losing one cannot take the recovery points with it, and Git with a GitOps controller pointed at it. The workload is a PostgreSQL database with known contents — four rows — so every restore is validated against an expected result rather than a green dashboard.
Scenario 1: verifying that a backup contains data
A Kubernetes backup has two distinct parts: the resource definitions (the YAML) and the persistent-volume bytes. Tools protect volumes through CSI snapshots, filesystem backup, or snapshot data movement to an external store. The lab uses the last of these, with Velero and its data mover.
Most verification stops at the backup’s Completed status. The guide goes one step further and confirms the bytes actually moved:
kubectl -n velero get datauploads -l velero.io/backup-name=$BACKUP \
-o custom-columns='NAME:.metadata.name,PHASE:.status.phase,BYTES:.status.progress.bytesDone'
# NAME PHASE BYTES
# guestbook-rehearsal-20260727001126-q2j9m Completed 47989888The data mover confirms 47,989,888 bytes left the cluster and landed in the external store. A tool that cannot produce this number for a given backup deserves scrutiny. Deleting the namespace — PVC included — and restoring from the backup returned the four rows in about two minutes. That is the happy path, and it hides three things no tool does automatically: protecting volume data does not make a database application-consistent (flush or quiesce hooks are required), restoring onto different infrastructure may need storage-class mappings, and a Completed status does not prove the application will start with the expected data.
The boundary is sharp: backup tools restore into a cluster that already exists. They do not create the cluster, the nodes, the network, the load balancers, or DNS. That is the job of infrastructure as code or Cluster API. A recovery plan that begins with “restore the backup” must state what the backup is restored into.
Scenario 2: declared state is not stored state
The production cluster is powered off. The recovery cluster, which existed before the disaster, has a GitOps controller pointed at Git and a backup tool pointed at the shared store. It has never run the application.
Syncing from Git succeeds: the sync reports Synced, the StatefulSet rolls out, the database pod is Running and Ready, every dashboard is green. Querying the database then returns:
ERROR: relation "attendees" does not existThe database is running and it is empty. Nothing malfunctioned. Git only ever contained the declarations, so Kubernetes did exactly what the YAML says: create a StatefulSet, create a Service, and provision a brand-new, empty volume for the PVC. GitOps reconstructed the declared state perfectly and restored none of the stored state. Both tools are required because there are two different things to bring back, and each tool carries exactly one of them: Git stores intent, backups store state.
The recovery that produced validated data followed three steps: remove the empty application the sync created, restore the application volumes and all from the backup store, then validate the data against the expected contents. The measurement is instructive: powering off production to validated data took four minutes live, and just under two minutes in a rehearsed rerun. The moment the dashboards turned green was not the recovery; the moment the data came back and was checked was.
Scenario 3: multi-volume consistency
Real stateful applications span multiple volumes: database data plus WAL, message-broker partitions, replica sets. The lab stand-in writes matched pairs — order n to one PVC, payment n to another — five times a second, with one invariant: every payment must have its order.
Snapshotting the two volumes individually, five seconds apart, produced two snapshots that were each ReadyToUse and individually perfect. Restoring both and comparing the last committed sequence numbers gives:
last order committed : 108352 last payment committed : 108377
[FAIL] 25 payments have NO matching order.
[FAIL] Each snapshot succeeded. The restore is still wrong.Twenty-five payments reference orders that do not exist. No component failed, every operation reported success, and the combined recovery point describes a moment in time that never existed. In production, that five-second gap is a backup tool walking a list of a hundred PVCs one by one.
The API answer is VolumeGroupSnapshot, promoted to GA in Kubernetes 1.36. A single object selects PVCs by label, and the CSI driver receives one request for a coordinated, crash-consistent recovery point across all of them:
apiVersion: groupsnapshot.storage.k8s.io/v1
kind: VolumeGroupSnapshot
metadata:
name: ledger-group-snap
spec:
volumeGroupSnapshotClassName: csi-hostpath-groupsnapclass
source:
selector:
matchLabels:
group: ledgerRestoring the group’s member snapshots and rerunning the verifier this time yields 109169 on both sides, with the verdict “[OK] Every payment has a matching order.” The caveats hold for the real world: support is driver-specific (most major cloud drivers did not implement it as of mid-2026), setup is explicit (the snapshot controller’s CRDs and feature gates), and crash-consistent is not application-consistent: the API removes cross-volume timing skew but does not flush the database.
What a real recovery test looks like
A recovery test is not deleting a pod and watching it return; that tests workload reconciliation. A recovery test restores a complete stateful application into a clean target that has never run it, validates the data and the user path against expected contents (not resource statuses), and measures the whole thing with a clock.
The guide points at three open gaps no single tool closes today: there is no common cross-cluster failover contract, no standard recovery unit for an application (each tool draws the boundary differently), and backup success is treated as recovery proof while restore-rehearsal results are almost never monitored. The Cloud Native Business Continuity initiative under the CNCF TAG Operational Resilience is seeking contributors on exactly these topics.
Verdict
The Pathak and Narang guide restates a truth most teams avoid: the cost of a backup is not creating it, it is rehearsing its restore. If your only metric is Velero’s Completed status, you know the backups run, not that they are good for anything. If you operate stateful applications, institute a recovery rehearsal that restores into a fresh cluster, validates data against an expected result, and times the whole thing — and treat VolumeGroupSnapshot as the answer to multi-volume consistency, provided your CSI driver implements it.