Turned a self-hosted homelab from a fleet with no real backups and a single-point-of-failure credential store into one with monitored, restore-tested nightly backups on a checksum-protected NAS, a permission model designed so a leaked automation credential can't reach personal data, and a self-hosted password manager, closing three debugging mysteries along the way.
The challenge
The development server had never been backed up, not once. The rest of the fleet's backups were fragile and local-only: the production node wrote to a single local USB drive via cron, with no offsite copy anywhere, and RAID was being mistaken for backup. The homelab's credentials lived solely in one machine's password app: a genuine single point of failure. A real power outage had just exposed how fragile recovery was, surfacing boot-ordering races that would recur as a theme.
The goal: stand up a NAS as the fleet's backup destination and personal-file store; give every machine real, monitored, restore-tested backups over the network; design a permission model so a leaked automation credential can't reach personal data; eliminate the credential single point of failure with a self-hosted password manager; and keep the whole thing reachable only over a private VPN, with no vendor cloud or telemetry.
Approach
Design for the leaked-credential case; verify the mechanism, not the checkbox.
- A checksum-protected, self-healing store. The NAS runs a redundant array on a checksumming filesystem, with data-integrity protection enabled at share-creation time (it can't be turned on later) so bit-rot is caught and the good copy rewritten, plus a monthly scrub. All vendor cloud, telemetry, and remote-access features declined; reachable only over a private mesh VPN.
- An asymmetric user model: the core security idea. A non-admin service account has write access to the backup share and explicit no-access to personal files; a separate personal account has access to personal files and explicit no-access to the backup share, with credentials never embedded in any script. So a leaked cron credential's blast radius is exactly the backup share, never personal data, and a compromised script can't expose the interactive account. The explicit no-access grants are deliberate defense against future group-grant creep.
- Rejected a tempting-but-wrong control. A third "write-only-append" backup account (so ransomware on a source couldn't overwrite prior snapshots) was considered and rejected: the NAS genuinely can't express append-only semantics, and the backup tooling rewrites files atomically anyway. The real control (immutable snapshots + an offsite copy) lives at a different layer; a split account would have been a weaker, more complicated version of the same protection.
- Cut over with overlap, and restore-test everything first.
Results
- The development server's first backups ever, plus the production node migrated off its lone USB drive: nightly database dumps, config tarballs, and app-data backups landing on the checksum-protected NAS over the private VPN, each with its own uptime monitor, all green.
- A transport pivot. The original plan relied on key-based authentication over SSH, but the NAS rejected the keys despite a textbook setup, diagnosed as a platform-wide issue rather than an account-specific one. Rather than chase an opaque platform bug, the transport pivoted to a simpler alternative that was already working: fewer moving parts per machine to operate, and the advantage of the original approach was irrelevant for small write-once nightly dumps. Least-privilege was then tightened so the service account could use only that one path.
- Restores verified before the old system was cut: database dumps validated for a good archive and valid content, config tarballs verified to contain the real files. A 7-day dual-write overlap on the production node made the cutover reversible rather than a leap; the legacy USB backup subsystem was later fully decommissioned.
- A boot-time mount race, fixed the robust way. On a cold reboot, the NAS mounts failed because they tried to resolve the VPN hostname before the VPN daemon had rejoined and its DNS was resolving. An ordering directive wasn't enough: the daemon reporting "active" doesn't mean its DNS is answering yet. The durable fix deferred the mount to first access (automount), sidestepping boot-ordering entirely.
- A silent security-patching failure, found and fixed. Automatic security updates looked enabled across the fleet, but a distro variable resolved to the wrong thing, matching nothing in the security repositories, so patches were silently never applying. Fixed by correcting the value and verifying with a dry-run, then formalizing a daily-auto/monthly-manual cadence.
- The credential single point of failure closed with a self-hosted, VPN-only password manager, backed up from day one, so the fix for the SPOF didn't become a new one.
What this demonstrates
- Design the permission model for the leaked-credential case, not the happy path: an asymmetric no-access boundary makes a compromised script's blast radius exactly the backup share, never personal files.
- Know your platform's limits before designing a control: the append-only account was rejected because the platform genuinely can't express it; the correct control lives at a different layer.
- "Enabled" is not "working," and "service active" is not "service ready": auto-patching applied nothing because a distro variable resolved wrong, and both boot races came from depending on the VPN daemon being up when what mattered was its DNS resolving. Verify the mechanism, not the checkbox.
- A backup you haven't restored isn't a backup, and cut over with overlap, not a leap: integrity-test every target and run a dual-write window so the migration stays reversible.