Case study
Scope: 13 scheduled-AI-job specifications for a self-hosted fleet, built, trialled attended, re-run headless, and costed end to end. Status: all 10 trialled jobs judged worth keeping. Zero are on a schedule. That is the finding.
The infrastructure generates more reviewable surface than an operator working evenings can cover: open pull requests, a couple dozen web applications, a fleet with drifting configs, a backlog in the hundreds, and a dependency tree nobody audits. Meanwhile a flat-rate AI subscription resets weekly, and unspent capacity expires.
The obvious idea: run the review work overnight as scheduled AI jobs. Security audits, code review sweeps, fleet posture checks, PR triage, backlog planning: all things an agent can do unattended, all producing a report a human reads over coffee.
The obvious idea has a non-obvious problem. An unattended agent that hits a permission prompt waits forever. No error, no timeout, no notification: the process just sits there. A daily job could fail every morning for a month and be indistinguishable from one that never ran. Every guardrail built to keep an attended agent safe becomes, unattended, a silent hang.
So the real question was never "can an AI do this work?" It was: what actually happens when you take the human out of the loop, and how do you find out before you're relying on it?
Thirteen job specifications: adjudication of agent-generated proposals, per-application security audit, code review sweep, fleet posture, daily backlog planner, execution night, an ID-reservation solver, PR-queue review, dependency/CVE audit, filler drafting, maintenance pre-runs, usage tracking, and a full security sweep. Each with fixed model routing (cheap tiers for mechanical work, expensive tiers for judgment only), inputs/steps/outputs, gate tags, and a trial-metrics section.
The explicit decision: write everything as manual-run specs, schedule nothing, decide cadence only after a trial.
The drafting was itself delegated: three parallel agents wrote the specs, a stronger model verified every file against the rulebook and the machine facts. That verification caught a real bug before anything ran: one job would have hung an unattended session reading a privileged config file through a password-gated escalation prompt. The spec was corrected to a fail-fast, non-interactive form that reports honestly when it can't verify. The first hang was found by review, not by a 3am process sitting silently until someone noticed.
A session-limit baseline was measured the same night: a maxed five-hour window at the most expensive model tier ran ~312k output tokens across 223 messages. That became the budget everything else was sized against.
All 10 jobs ran, producing 20 output files. Every job earned a keep verdict. Real findings came out: a crash cluster in a shared HTTP helper reproduced live; a write-only metrics rollup that was silently never read; a stored cross-site-scripting vulnerability via an unsanitized link attribute in a live application; and several instances of configuration drift.
The trial's actual deliverable was the run-mechanics punch list. Every approval prompt traced to an allowlist scope gap, never a design flaw, and by the third batch all five remaining jobs ran with zero prompts.
That zero-prompt result later turned out to be a mirage.
Nine findings were checked against live source before anything was fixed. Five of the nine were wrong as filed:
| Filed claim | Reality |
|---|---|
| Crash cluster affects 5 applications | 7 (two more were equally unguarded, in groups the reviewer reported as clean) |
| Undocumented port = configuration drift | Deliberate (set months earlier to avoid a port collision, documented in the incident log) |
| Security policy "draft, live leak open" | Stale (applied and probe-verified days earlier; only the file's banner was out of date) |
| World-readable config file = exposed credential | Overstated (that file holds a pointer; the credential itself is correctly restricted) |
| Nightly backup "shrinking", 3 real firings | False positive, root-caused (four artifact families in one directory, the disk-usage tool sorts alphabetically, and the monitor read a family boundary as a size cliff) |
The last is the biggest single save: its remediation plan proposed a full investigation session on a production box for a backup that was never broken.
The real findings were then fixed and verified: the XSS fix shipped live (an allowlist parsed through the URL constructor rather than a scheme denylist, because a denylist loses to mixed-case and embedded control characters; 19 of 19 attack vectors blocked), and the crash guard went to all seven affected applications with severity honestly corrected downward once the exception path was traced.
An automated finding inherits its collector's assumptions. Five of nine is not an indictment of the jobs (they surfaced things a human wouldn't have), but it settles whether their output can be acted on directly. It cannot.
Before agreeing to re-run, the previous session's fixes were checked. They weren't real. The spec directory contained no avoidance text: none of the workarounds the tracker credited. Those walls had been recorded honestly as "mitigated (instruction)" and "fixed (my process)".
But a scheduler fires the spec, not the operator's memory. The celebrated zero-prompt batch was a human steering sub-agents live, and nothing about it was durable.
So: ten jobs, headless, one process per job, specs executed exactly as written, no coaching. The only additions were measurement rails (draft-only, write to a scratch directory) and the important one: if a tool call is blocked, log the wall rather than silently routing around it. A structured output envelope proved to be the right instrument: permission denials come back as a machine-readable array, and a truncated run is unambiguous rather than passing silently as complete.
Every wall closed by discipline came back. 36 permission denials across the valid runs.
The cadence answer is no. Total exercise: ~634k output tokens (406.6k in valid runs, 152.6k lost to six rate-limit truncations, 75.3k to a harness timeout). Against the ~312k per-window baseline, seven clean jobs alone cost 317k. Six concurrent orchestrators exhausted the window outright. The ten jobs do not fit one session window, so the schedule must stagger: a shape the attended trial could never have revealed, because it drove one context at a time.
Three sizing facts from the same data:
Guardrails held against genuinely unattended agents, which is worth stating plainly: a recursive delete on a scratch directory was hard-denied; a job that went looking for credentials on its own initiative was denied; the production machines were never touched.
An undisclosed 30-minute timeout in the run harness killed the code-review sweep mid-flight at 1,802 seconds. It had been written in as a safety measure, mentioned only in a code comment, and never stated to the operator in plain English. Roughly 75k tokens spent to produce a floor instead of a measurement. That was compounded by firing a 60-minute re-run without asking whether the spend was wanted. It was stopped, correctly, and the window that would have funded a variance pass was gone.
The number was the smaller error. A timeout is only safe when the normal duration is known, and this job had never been measured. The rule that came out of it: anything baked into a harness that can terminate work gets stated in plain English before it runs, never left in a comment.
The salvage is worth recording. Rather than re-running, the on-disk state was checked: 31 per- repository review files, the last written nearly seven minutes before the kill, meaning the job had finished reviewing and died during final aggregation. The findings had survived. An aggregation pass over the existing files cost 34.7k and produced the roll-up the sweep never reached, instead of repeating a job already cut short four times.
With the jobs proven but unscheduled, the remaining question was mechanical: what actually runs when the laptop is closed? Three routes exist and they are not interchangeable:
| Route | Runs with the laptop off | Reaches the fleet | Gated |
|---|---|---|---|
| A plain cron job on the always-on machine | yes | yes (it is that machine) | none by default |
| A "cloud" scheduled agent | yes | yes, via the bridge | yes, and it hangs |
| A third-party scheduled workspace | yes | not by default | untested |
The trap: the desktop application's "scheduled tasks" are a fourth thing, and they fail the requirement: documented as running only while the app is open and the computer awake. If the computer sleeps through a scheduled time, the run is skipped silently. It is the option that looks right and isn't.
The main finding: a "cloud" scheduled agent pointed at the registered environment does not run in the cloud. It runs locally, on the machine the environment is registered to, with exactly the access that machine already has. Proven from the machine's side rather than the job's self-report: the process started at the exact scheduled minute, alongside a fresh session transcript. Only the trigger is cloud-side. That means a "cloud" job inherits its host's reach, not a sandbox's.
Both probes hung, at two unrelated gates, in the same silent way. The first stalled on the custom policy hook and sat at the prompt for nearly six minutes until killed. The second was designed specifically to dodge that gate by using built-in web tools instead of shell commands, and stalled anyway, at a different gate: the built-in web search tool wasn't in that machine's allow-list. The custom hook logged no decision at all, because this was the agent framework's own permission system, not the custom one.
The failure mode is the finding, not the specific rules. An unattended session stops at any tool that isn't pre-approved and waits forever: no error, no timeout, no notification, process held open. Both probes consumed a daily scheduling slot and produced nothing.
The reframe that came out of it: the tool that blocked the second probe was web search, which cannot reach any fleet machine, cannot exfiltrate, and cannot download and execute. Allowing it costs nothing in fleet safety. Shell-level network commands staying gated is what protects the infrastructure; web search was never part of that protection. Guardrails govern tools, not goals: "search the web" and "restart a container in production" only feel like one dial because both produce the same prompt.