About Professional
How I Build How I Build Meet the Team
Technology Homelab App Showcase Case Studies
Maverick & Luke Say Hello

Case study

Configuring the Agent Itself: Skills, Model Routing, Memory, and the Git Repos That Keep the Fleet Identical

Scope: the configuration layer that turns a general-purpose AI coding agent into a fleet-specific operator: a library of skills, a set of memory files, a permission rule set, a policy hook, and a sync daemon, all held in git repositories. Companion: the permission model itself is covered separately; this study is everything around it.

Problem

A coding agent out of the box is an amnesiac generalist. Every session starts from zero: it doesn't know that machines in the fleet sit on different approval tiers, that deployments follow a numbered checklist, that a particular command is spelled a particular way on every box, that pushing to the release branch is forbidden, or that it was corrected about exactly this two weeks ago.

For most developers that's a mild tax: you re-explain, or you don't bother, because you'd catch a bad suggestion by reading it.

This operator can't read the suggestion. He builds and runs the entire fleet through AI agents and judges outcomes in plain English. That inverts the economics of configuration:

So the question was never "how do I write a good prompt?" It was: how do you make an agent's behaviour a versioned, reviewable, portable artifact, the way you'd treat any other production config?

Approach: six layers, each solving a different failure

Layer 1: layered instruction files, with the rules in the tool-agnostic one

Three files stack at session start: a thin global rulebook (who the operator is, communication style, verify-before-asserting, model tiering, the spend gate), a thin project file, and the substantial one: a shared operating rulebook read by every agent tool in use, containing the approval gates, the machine table, command rules, git discipline, and secrets conventions.

The non-obvious decision is the split. The project file is deliberately thin and says so in its own text: the shared rulebook is where every coding agent reads its operating rules. Edit them there, not here. The global file carries the same discipline in the other direction: keep this thin, only things true everywhere, target well under 200 lines.

Rationale: most of the agent tools in use have limited mechanical guardrails of their own. For those, the written rule carries much more of the weight. Putting the operating rules in a file named after one vendor's tool would have produced a fleet where the rules bound only the agent that also had enforcement, exactly backwards.

Layer 2: skills as executable runbooks

A set of project skills load only inside the infrastructure tree; a smaller set of user-level skills load in every session on the machine. They cover deployment, promotion, decommissioning, session wrap-up, documentation updates, backlog operations, security audits, secret rotation, restore drills, maintenance passes, incident response, container updates, and mobile-app build and release.

Most wrap a numbered procedure that already existed in the documentation repo: the deployment checklist (phases 0–8), promotion (phase 9), decommissioning (nine phases). The skill isn't a second copy of the runbook; it's a loader that makes the runbook unskippable, because the agent reads the phases into context before acting rather than working from an approximate recollection.

A single index page lists every skill, its command, and its backing runbook, under an explicit maintenance rule: creating, renaming, or retiring a skill updates this page in the same session. A stale quick reference is worse than none.

Layer 3: per-skill model routing

Every skill pins its own model tier in frontmatter, and the reasoning for each is recorded in one revertible table:

TierWorkWhy
Most capableMulti-agent backlog batches, whole-repo documentation audits, security auditsLong-horizon reasoning; dispatches executors and verifies their work
HighDeployment, incident response, restore drills, maintenance, container updates, promotion, decommissioning, secret rotationConsequential, multi-machine, judgment at every gate
MidDocumentation fact edits, session summaries, teaching docs, backlog closureWell-scoped single concern; writing quality matters, deep reasoning doesn't
Cheapest (with reduced reasoning effort)Appending a backlog row, appending an incident rowAppend a row to a file

Three skills (promotion, decommissioning, and secret rotation) additionally carry a flag meaning the agent cannot load them on its own initiative; only the operator typing the command fires them. All three are production-facing and hard to undo. The stated tradeoff: matching the approval gates matters more here than the convenience of auto-invocation. Verified live after the change: all three dropped out of the auto-invocable listing while remaining typable.

A harness limitation, worked around visibly rather than papered over. The model-override frontmatter key applies for the current turn only; the session model resumes on the next prompt. The three most-capable-tier skills all run across many turns, so the frontmatter alone would give them the right model for turn one and the session default afterwards. Each therefore carries a Step 0 in its body: check the active model, and if it's wrong, surface a "your step: switch models" instruction and wait. The assignments document states plainly that removing Step 0 doesn't break the skill: it just silently degrades to the session model after the first turn. The limitation is named, its consequence is named, and the workaround is visible rather than magic.

Layer 4: memory as single-fact files

Each memory is one file holding one fact, with typed frontmatter (who the user is / feedback on how to work / ongoing project context / external references), a one-line description used for recall, and a body that cross-links related memories. A one-line-per-memory index loads at session start.

The content is the part worth noting: these are overwhelmingly corrections, not facts. One exists because completed work was re-proposed twice. One exists because a verification check was trusted that would have returned the same answer whether or not the thing worked. One exists because a hardware state was asserted from a software reading five times while the operator sat looking at the actual hardware. The memory store is an error log with the errors turned into rules.

Layer 5: guardrails

Covered in depth in the companion case study: an allow/ask/deny permission rule set, a pre-execution policy hook with an extensive self-test suite, and file-tool scoping with private keys denied by exact name.

Layer 6: two git repos and a sync daemon that refuses to be clever

The entire configuration is two git repositories: one for global rules, user skills, hooks and the shared memory store; one for project skills and the permission config. Both are working-branch-only with self-hosted remotes and read-only mirrors on a second machine.

A small sync script keeps them identical across machines. It runs on session start, session end, and an hourly backstop, and its design posture is stated in its own docstring:

a deliberately CONSERVATIVE posture: it never merges divergent history, never force-pushes, never resets, never deletes. It only fast-forwards, auto-commits local config edits, and pushes them. Anything it can't do cleanly it logs LOUDLY and leaves untouched for a human.

SCOPE: only the two config repos. Real project repos are never touched.

It ALWAYS exits 0 so it can never block a session.

Two details make it work. It always exits zero. A config sync must never be able to prevent a session from starting, so problems surface in a log and a printed summary instead. And the memory store is shared across machines through a symlink bridging the two machines' differing project identifiers, so a correction learned in a phone session is present in the next laptop session.

Key decisions and the why

What's weak, honestly

A documented mechanism that wasn't in the file it named: found by this write-up, fixed the same day. The global rulebook stated that sessions default to a particular model tier, and cited a specific settings key in a specific file. Checked across all five settings files plus the tool's own state file: that key did not exist anywhere. A neighbouring key it also cited is genuinely present; this one wasn't (the default comes from the account or tool default, not the file named). The policy was being followed correctly; the sentence describing how it was enforced had gone stale. Same failure class as the machine-tier lists found disagreeing silently: a written rule and the mechanism it names were allowed to drift apart, and only a direct check found it. The reference was dropped rather than corrected, on the reasoning that a rule stating policy doesn't need to name its enforcement point, and a pointer that isn't there can't go stale a second time.

Changes take effect at session start. Both the permission config and skill frontmatter load once. With multiple windows open in parallel, a fix is live in the window that made it only after a restart, while other windows keep running the old rules. Documented in three places precisely because it has caused confusion.

Outcome

Lessons

  1. Agent configuration is a codebase: treat it like one. Version it, branch it, mirror it, and write the rationale in the file, not in a chat log. Inline comments in the config file have prevented more re-derivation than any amount of session history.
  2. Put the rules where every tool reads them, not where your favourite tool reads them.
  3. Document the rollback before you make the change. Every tuning pass here opens with the prior state and the revert command. That is what makes an experiment cheap enough to actually run.
  4. A skill is a runbook the agent can't skim. The value isn't a shortcut command. It's that the numbered phases enter context before the work starts, instead of being recalled approximately.
  5. Name the harness limitation and work around it visibly. A one-turn override that silently degrades is far more dangerous documented nowhere than documented with an ugly extra step.
  6. Route model cost per task, not per session. One day of measurement beat months of intuition about which tier a session "should" run on.
  7. Memory should mostly be corrections. Facts belong in a documentation repo, which can be searched. What can't be searched is "this class of reasoning went wrong last time," the highest-value thing a persistent memory can hold.
  8. A sync tool should refuse to be clever. Never merge, never force, never reset, never delete, always exit zero, and shout when stuck. A config synchroniser that resolves a conflict on its own is one that can silently revert a guardrail.