Turning app retirement from an ad-hoc, error-prone act into a repeatable nine-phase runbook applied across a dozen-plus apps. The standout moment was querying a database's own foreign keys to prove the docs had mislabeled a table's owner, preventing a live app's data from being deleted.
The challenge
An app fleet grows faster than it retires, and many apps were deployed onto a shared database schema rather than their own namespace, so one app's tables sit table-by-table alongside its siblings'. Generic table names, the kind every app creates, collide across apps, so a naive "drop this app's tables" can destroy a different live app's data. Compounding the risk, the development server had no backups: no safety net if a destructive operation went wrong.
The core problem: retire apps completely and safely, without breaking siblings and without leaving orphaned ports, firewall rules, secrets, monitors, or routes behind.
Approach: a nine-phase runbook, written from real teardowns
I formalized the procedure from an actual teardown rather than in the abstract, and every later execution that hit a new failure mode fed a new rule back into it. The nine phases:
- Pre-flight: confirm the last backup ran, confirm no live session touches the schema, dependency-check, optional dump.
- Service teardown: stop and remove the systemd unit.
- Network teardown: firewall rules deleted highest-number-first, both IP families; mesh-VPN routes revoked.
- Schema teardown: drop, update the API schema list, recreate the containers (a restart doesn't reload the env), verify.
- Monitoring cleanup.
- Sudoers cleanup.
- File archive: move, never delete.
- Source archive: repo archived read-only, never deleted.
- Docs + references update.
Plus a cross-cutting credential rotation: retirement is the trigger to revoke every external key the app used.
Two hard sub-rules were born from real incidents. For apps on the shared schema, run three sanity queries before any drop (list the shared tables, read the app's own schema for its exact table list, and grep every app's schema for those table names to catch collisions). And, since the development server had no backups, every destructive op there required an explicit "this data is disposable" confirmation in place of a backup check.
Consolidation, not just deletion. Many teardowns fired because a successor superseded the app: two apps folded into a single successor, a purpose-built service replacing an adopted tool, a docs system replacing a dashboard. That makes the runbook a deliberate architectural-upgrade tool, not attrition.
Results
- The standout save: evidence beat documentation. In a four-app teardown sweep, the docs listed one retiring app as owning a generically-named table. Before dropping it, the shared-schema rule triggered a foreign-key attribution query (grouping the schema's foreign keys by referenced table), and it proved the table appeared to belong to the retiring app but its foreign keys pointed to a different, live sibling app's tables. The table was left intact; only the foreign-key-confirmed set was dropped. Had the drop trusted the docs' table list by name-match, it would have destroyed a live app's data. This became a standing rule: attribute table ownership by foreign-key clustering before any shared-schema drop, never by name-matching the docs.
- A formal, reusable runbook: nine phases, written from real teardown experience and operationalized as a one-command procedure, applied identically across a dozen-plus app retirements.
- Consolidation, not just deletion: multiple multi-app collapses into better successors, reducing the fleet's surface area while preserving every capability.
- Concrete reclamation: dozens of ports freed, a shared schema reduced to a single app's tables, the API schema list trimmed, and stale runner/sudoers/registry references removed.
- A documentation-integrity pass: tearing apps down forced a reconcile of every documented fact against reality, surfacing five separate doc-drift errors (a mislabeled table, a secret documented on the wrong machine, a nonexistent cleanup step, a phantom registry entry, a stale authoring path).
What this demonstrates
- Evidence over documentation for destructive ops. When being wrong means irreversible data loss, derive ground truth from the system itself, the database's own foreign keys, and treat the docs as a hypothesis to verify, not a fact to trust.
- Codify the runbook from a real execution, then feed every new failure mode back into it, so the procedure compounds instead of decaying.
- Teardown is a full-stack, many-layered act: services, firewall, VPN routes, schema, monitors, sudoers, credentials, source, and docs. A checklist is the only way to avoid an orphaned tail, and the stale-reference cleanup matters as much as stopping the service.
- Consolidation is the highest form of retirement: the best teardowns fold two apps into one better successor, making retirement an upgrade rather than a loss.