I replaced per-app manual SSH-and-restart deploys with a self-hosted CI runner and a small set of canonical per-architecture workflow templates, so a push to the dev branch auto-syncs and restarts the app, and rolled it across a heterogeneous app fleet with narrowly-scoped runner privileges and a fail-loud secrets pattern.
The challenge
I run a self-hosted homelab. Deploys to the development server were fully manual: for every app, I opened a terminal, SSH'd in, cd'd to the app directory, pulled the branch, and restarted the service, a per-app runbook that scaled linearly with the app count. Two earlier automation attempts (webhooks and cron SSH polling) had failed and been abandoned. And the fleet was growing, so the manual toil kept getting worse.
The box also has an unusual constraint that shapes everything: it runs the dev branch as its live environment (automated pushes only ever target dev; the main branch is reserved for promotion to the real production machines).
The goal: stand up push-to-deploy across the whole fleet. A push to dev should automatically sync the code and restart the service, with a repeatable, documented per-app bootstrap so every new and existing app conforms to one canonical pattern rather than a hand-improvised workflow.
Approach: "template the pipeline per architecture; scope the runner narrowly"
- A self-hosted CI runner, run with a deliberately narrow footprint. A single runner handles jobs for the whole fleet, one at a time, which keeps the model simple and makes any hung job a visible fleet-wide signal rather than a silent one.
- Every workflow triggers only on a push to the dev branch, never main.
- A small set of canonical per-architecture templates, not per-app workflows: a handful of standard shapes covering static apps, build-and-serve frontends, and backend variants. The standing rule, "never hand-improvise a workflow; copy the matching template and substitute the app and service names," eliminated a whole class of one-off deploy bugs.
- A repeatable runner-permissions bootstrap run once per app, scoped to exactly what that app's restart needs and nothing more, to contain blast radius if the runner is ever compromised.
- Secrets out of the repo, fetched at deploy time, fail-loud. Backend apps pull their secrets from the centralized secrets service at deploy time, failing hard on any missing key. And because the deploy step ends in a service restart, secrets are re-hydrated fresh on every deploy.
Results
- Auto-deploy live across the entire fleet: a push to dev triggers fetch + reset + restart on the box, replacing the manual SSH-and-restart runbook for every app.
- Every active app on the single canonical workflow after the rollout, up from the smaller starting fleet, spanning a heterogeneous stack with just a handful of templates.
- Narrowly-scoped runner privileges: restart-only permissions per service, no wildcards, and a fail-loud secrets pattern re-hydrated on every deploy.
- Hardened the sync step from fragile to idempotent. It originally used
git pull, which aborts on a dirty or divergent tree and silently froze a clone at a stale commit for a long stretch. I moved the whole fleet to a force-sync approach, which is idempotent (with the caveat baked into the rule that the reset targets the current branch, so the clone must be pinned to the deploy branch). - A fleet conformance audit corrected the record. Standardizing the workflow filename fleet-wide, I found my own assumption about which apps had drifted was simply wrong. A live audit turned up only a couple off-pattern (not the ones I'd assumed), both fixed and re-verified green.
What this demonstrates
- Template the pipeline per app architecture, not per app: a handful of canonical templates covered a heterogeneous fleet and killed a recurring class of one-off bugs.
- Least privilege by construction: restart-only permissions per service, no wildcards, so a compromised runner can't do more than restart known services; secrets kept out of the repo and re-hydrated on every deploy.
- A hard force-sync beats a plain pull for unattended CI deploys. A plain pull silently freezes on a dirty tree; a force-sync is idempotent (just pin the clone's branch first). And service names must match the service-manager unit filename exactly, hyphens and all: a silent, easy-to-miss break.
- Verify live state before "fixing" from memory. The conformance audit found reality didn't match my assumptions about which apps had drifted; a single-slot runner also makes any per-job hang a visible fleet-wide stall, surfacing problems instead of hiding them.