Case study
Scope: a self-hosted multi-machine homelab: dev server, production server, NAS backup target, and a decommissioned laptop used as the crash-test dummy.
The lab had a mature backup pipeline: nightly database dumps, nightly config archives, Git-server dumps, and weekly whole-system images from every machine to the NAS, all dead-man-monitored. But none of it had ever been restored. A backup that has never been restored is a hope, not a backup. Two proofs were owed: a first-ever quarterly restore drill (does a nightly dump actually restore, with the data it claims?), and a bare-metal drill (can a weekly system image resurrect a dead machine, onto different hardware?). A retired laptop, already scheduled for a wipe, became the drill target, with a standing rule: drill first, then wipe, and salvage the old disk's scripts before anything destroys them.
The runbook said to restore the production database dump into a generic version of the database engine. Done exactly as written, the restore exited 0, reporting success, while it had silently thrown away most of the data. The generic engine was close enough to look right and different enough to quietly fail partway through. The mechanism was a bulk-load cascade: a data load that hits a table which never got created executes those data lines as commands instead of erroring, and silently stops the moment one of them reads as a meta-command. The failure mode didn't surface as an error; it surfaced as a false "done."
The dump itself was fine: the testing apparatus was the bug. Restoring into the same setup production actually runs (plus a couple of authentication quirks found live) produced a second run where every row count matched the source exactly, and data freshness matched the dump to the minute. The archive check passed too, valid and same-morning fresh. The runbook was rewritten the same session so future drills inherit the fix, not the trap.
Before wiping the old laptop, its disk was mounted read-only and the original automation scripts salvaged for comparison. Most reconciled clean; one didn't. When the dev server had been rebuilt on new hardware a month earlier, its nightly config-backup script was recreated from memory, and the recreation had quietly dropped the lines that mattered most. For weeks those files were covered only by the weekly system image, a far wider exposure window than the daily one that was intended. Fixed same day, with a self-check added: the backup now fails loudly if anything it is supposed to capture goes missing from the archive.
The dev server's latest weekly snapshot was restored onto the wiped laptop: different vendor, different decade, different disk. 1.4 million files / 27.5 GiB in about seven minutes, bootloader reinstalled, filesystem IDs re-stamped, all networking disabled for isolation. It booted to a working login with the complete home directory intact: system, accounts, and data proven on foreign hardware. One caveat recorded rather than rounded up: the graphical desktop didn't start on the 2014-era GPU, so a real recovery may need a short driver fix before the remote-desktop path returns. Console and SSH work regardless.