Hardening a Fleet's Deploy Path After a Silent, Months-Long Freeze

A routine maintenance sweep uncovered one app whose auto-deploy had been silently frozen for months (its records claimed deployments that never happened), so I root-caused the failure class and hardened every app's deploy path so a dirty working tree could never silently freeze a clone again.

The challenge

While verifying a small fix on the development server, I found the deployed copy of one app frozen at a months-old commit, running stale code. Every push since had failed to auto-deploy with no error surfaced anywhere.

The blast radius was bigger than one stale app. A large slice of intervening work had never reached the box, including a frontend fix that had sat "pending verify" for a long time because the code genuinely wasn't there. Worse, it exposed a documentation-integrity problem: every "deployed" claim for that intervening work was simply false. And this wasn't the first freeze of this clone: the underlying permission-drift class had recurred across many earlier sessions.

Approach

I treated it as a class of bug, not a one-app fluke: diagnose the real cause, apply a durable structural fix, then generalize across the whole fleet and update the source-of-truth templates so the fix would be inherited rather than re-discovered.

The root cause was two stacked failure modes. First, cross-user git-object permission drift: the deploy user and the file owner shared no common group, so each wrote git objects the other couldn't touch, and a fetch failed with a permission error. Second, and the real silent-killer: the deploy step used git pull, which aborts on any dirty or divergent tree: a single stale local edit was enough to freeze the clone indefinitely. Because the deploy ran on a self-hosted CI runner with a single slot, it just failed silently on every push.

For the frozen app I applied a durable fix (group-share plus setgid, so new git objects inherit group-write), then force-synced, rebuilt, and restarted, confirming runner self-heal with a no-op push. Then I surveyed every app's deploy workflow: a handful already used the safe git fetch + reset --hard pattern; most used the fragile git pull. I switched the fragile ones to git fetch + reset --hard, which is idempotent and kills the dirty-tree abort mode regardless of permissions. That sweep surfaced a couple more silently-frozen clones. They were caught only because the fix was applied fleet-wide.

I left deliberate tradeoffs alone: the manual deploy fallback stays on git pull, because a human wants a loud dirty-tree failure, not silent destruction of local work. reset --hard is right for an unattended machine; git pull is right for a human.

Results

• A months-long silent freeze found and cleared, and the affected work finally deployed.
• The large majority of app workflows hardened (a handful were already safe): the whole fleet now on the resilient, idempotent deploy pattern; a couple of additional silently-frozen clones surfaced and fixed in the process.
• Every canonical template updated so new apps inherit the safe pattern, with the load-bearing caveat baked in (the reset targets the current branch, so the clone must be on the deploy branch).
• Honest residual: the deeper group+setgid durable fix was applied to only the one app; the others still relied on the runner being the consistent writer. That's why the complete structural fix (a fleet-wide POSIX-ACL permissions migration) followed, and is written up as its own companion piece. This work was the discovery and workflow hardening, and it explicitly marked its own single-app fix as interim rather than papering over it.

What this demonstrates

• Silent CI failure is the dangerous kind. A single-slot runner plus a git pull that aborts on a dirty tree yields an indefinitely frozen clone that reports success at the surface. I prefer idempotent, unconditional deploy steps for machine-driven deploys.
• "Deployed" is a claim to verify, not assert. Months of records said the work shipped; it hadn't. That's exactly what motivates a running-commit freshness check instead of trusting an annotation.
• Fix the class, not the instance, and update the source-of-truth templates in the same pass, or the next app re-inherits the bug.
• Different tradeoffs for human vs. machine commands: a loud-failing pull for a human fallback, a silently-destructive reset for an unattended runner. Don't blanket-apply one pattern.