Relocated a production node's monitoring and backup stack onto a dedicated hardware hub (migrating each self-hosted tool's data verbatim so dashboards, history, logins, and 2FA all survived intact), while deliberately narrowing the plan to avoid a costly change that added no value.
The challenge
The production node did double duty: it ran the application workloads and the "infrastructure of infrastructure" that watches and supports them: a self-hosted uptime monitor, a self-hosted metrics/monitoring agent, a self-hosted dashboard, plus utility services and the backup orchestration. That coupling meant a reboot or maintenance window on the production box also blinded the monitoring and took the support tooling down with it. A single machine held both the apps and the tools meant to observe them.
A newly provisioned mini-PC was standing by to become a dedicated infrastructure hub. The task: relocate the support stack onto it without losing a single monitor, metric, saved login, or two-factor secret.
Approach
Three decisions shaped the work: the first was the most important, and it was a decision to do less.
- Scope discipline: move what delivers the value, not everything on the list. The original roadmap also moved a self-hosted Git server and its CI runner to the new box. On analysis that was the wrong call: the Git server was lightweight and already ran on more reliable hardware, its CI runner lived on a third machine already, and relocating it would have forced a fleet-wide rewrite of a hard-coded address embedded in dozens of repositories and documents: real risk and churn for no real gain. The genuine benefit came entirely from moving the monitoring stack. So the Git server stayed put, and instead of relocation it got layered redundancy: an existing nightly dump plus a planned warm-standby copy on the new box. Migrating the objective beat executing the plan verbatim.
- Data-directory-verbatim migration. Rather than stand up fresh tools and re-create their contents, I carried each stateful service's entire data directory across byte-for-byte. This preserved not just settings but all history, saved logins, and 2FA, and critically, each tool's own identity, so its agents and clients kept trusting the relocated instance with no re-enrollment.
- Parallel build, verified cutover. New instances came up alongside the live originals; the originals were only stopped (never deleted) to serve as a live rollback. Nothing was destroyed until the new box was verified reachable and correct.
Results
- Five services relocated in one session: utilities, the metrics hub, the uptime monitor, and the dashboard, each verified reachable before its original was retired.
- Zero data loss. The uptime monitor came back up with all its checks, their full history, and my login plus two-factor intact (a login, not a fresh setup). The metrics hub kept its identity, so every reporting agent across the fleet kept trusting it with no re-enrollment.
- A large, unnecessary change avoided. Keeping the Git server on its reliable home sidestepped a fleet-wide rewrite of a hard-coded address across dozens of files.
- Host-relative configuration repointed. Several monitors referenced services via a "this host" gateway address: an address whose meaning changes when the monitoring server moves machines. Each was rewritten to an explicit target, and backup heartbeat pings were re-aimed at the new server and confirmed end-to-end with a live test rather than assumed.
- A wrong assumption caught mid-flight. I initially believed relocating the metrics hub required no changes to the reporting agents; the first logs proved the opposite: the agents dial the hub, so each needed its target repointed. Owned immediately, fixed, moved on.
- Rollback preserved and security tightened. The originals were left stopped-but-intact, so a revert is a restart, not a rebuild; the broad automation used to build the box was narrowed at close-out to a least-privilege grant scoped to exactly what ongoing management needs.
What this demonstrates
- Scope is a design decision, not a given. "Move X, Y, Z" deserves the question which of these actually delivers the goal? Here, moving one of the three would have added disproportionate risk for zero benefit, so it didn't move.
- For stateful self-hosted tools, migrate the whole data directory. It carries settings, history, credentials, and the identity that keeps dependent clients trusting the instance, turning a re-enrollment project into a copy.
- Verify the direction of a connection before assuming it. Who-dials-whom determines what has to change when you relocate one end; check the logs, don't guess.
- Close out access you opened. Broad automation to build a box gets narrowed to least-privilege to run it, as the final step of the same work.