Killing a Year-Long CI-Hang Bug Class with POSIX ACLs Across the Fleet

Replaced a fragile world-writable permissions hack (which had silently broken automated deploys across a fleet for over a year) with a self-healing POSIX-ACL model, piloting on one app to prove the mechanism before migrating the fleet, and shipping a daily detector for the exact failure the existing monitors were blind to.

The challenge

A development server runs a couple dozen web apps, each auto-deployed by a self-hosted CI runner (a dedicated deploy/runner user) that does git fetch && git reset --hard inside each app directory. The apps were owned by a regular user, but the runner needs to write .git during deploy. The historical fix was a recursive chmod granting world-write (o+w) across the whole app tree.

That fix was structurally unstable: new files inherit the creator's umask, which strips the world-write bit, so any file added after the chmod silently drifted back to non-writable, and the next deploy failed or hung. This bug class recurred for over a year across many apps. Its worst incarnation: one app sat silently frozen at an old commit for weeks (its CI job hung for minutes on the un-writable .git, blocking the single-slot runner) and no monitor caught it, because the uptime monitor sees liveness, not deploy freshness. An early audit found several more apps in the same latent state; their next push would have hung identically.

The real fix had been tracked but repeatedly deferred as "too big to do at once."

Approach

The core insight: reach for the right OS primitive. chmod's "other" class can't be scoped to one user and doesn't propagate to new files, while chown transfers ownership away from an app's own service user. POSIX ACLs solve both problems. They are additive (the owner keeps full access; the runner gets an explicit grant) and support a default ACL that new files inherit automatically:

`` setfacl -R -m u:runner:rwX <app> # immediate grant, scoped to the runner setfacl -R -d -m u:runner:rwX <app> # default ACL → new files inherit find … -exec chmod o-w {} + # drop the world-write crutch ``

The capital X is deliberate: it sets the execute bit only on directories, never flagging regular files executable, which would break git's mode tracking, a documented failure of naive chmod 777.

• Pilot before fleet. Validate on one fresh, low-risk app and prove the access comes from the ACL, not residual world-write, by removing o+w and confirming a real deploy still works.
• Additive by design, to dodge the chown footgun. Any app whose backend runs as the owner and reads/writes runtime files in its own directory (.env, heartbeats, caches) must use ACLs, never chown -R runner.
• Ship a standing detector. A daily readiness sweep that checks, per app, whether the runner can write the exact git paths a fetch + reset --hard touches (made ACL-aware), plus a deploy-freshness check comparing each app's running health-endpoint commit SHA against its checked-out SHA, catching the "pulled but didn't restart" class the uptime monitor can't see.

Results

• The entire CI-hang / umask-drift bug class structurally eliminated: closing a year of deferred work at once.
• Every CI-deployed app now on a durable permission model: most migrated to ACLs, the rest already robust (runner-owned or already on ACLs). No world-writable app trees remain.
• Pilot: a clean deploy in ~8 seconds with world-write removed, empirically proving the ACL, not o+w, was granting access. A second app verified live, deploying through the ACL. This is the difference between proving the mechanism and merely proving the outcome; with two overlapping grants you otherwise can't tell which one is load-bearing.
• The chown footgun, proven empirically. Applying chown -R runner to one app broke its background service twice: first it couldn't read its .env (mode 600, owner-only), then, after chowning that back, it couldn't write its heartbeat file. Root cause: chown transferred ownership away from the service user. Fixed properly with restore-ownership + setfacl: direct empirical proof of exactly the failure ACLs are designed to avoid.
• Several latent CI-hang landmines defused before they fired; the frozen app unfrozen after being stranded for weeks.
• New self-healing property: new files inherit the runner grant via the default ACL. The old human-remembered "re-apply triggers" are gone entirely.
• Standing detector in place, running daily, filling the exact blind spot that let the frozen app rot.
• Caught its own tooling bugs before trusting them. The readiness sweep shipped with two bugs, both found by running it before relying on it: it over-flagged benign immutable git objects, and its ACL-aware check read the write bit one position too far right. Running your own tooling before trusting it is the only reason those shipped correct.

What this demonstrates

• Reach for the right OS primitive. POSIX ACLs are the correct tool whenever "user A owns it but user B also needs to write it": scoped to a specific user, additive, and, with a default ACL, self-healing against umask drift. A year-long recurring bug class disappeared once the underlying model was right.
• Pilot then batch: prove the mechanism, not just the outcome. Removing the redundant grant and still deploying proved the ACL was load-bearing; scripting the batch then turned a "too big" task into one session's work.
• Design out drift instead of re-patching it. Self-healing beats a human-remembered list of re-apply triggers, and observability should target the failure your existing monitors are blind to: here, deploy freshness and permission state, not liveness.
• Run your own tooling before trusting it, and never chown -R a directory whose service writes runtime files in it: the double-broken service is the textbook proof.

---

A companion piece covers the earlier interim fix and the discovery of the silent deploy freeze that surfaced this work.