Building a Source of Truth an AI Agent Can Be Held To

Scope: a markdown spec repository serving as the single factual substrate for a self-hosted infrastructure, maintained entirely by AI coding agents. Scale after three months: hundreds of markdown files, hundreds of thousands of words, over a thousand commits, hundreds of session logs, several dozen application directories (most of them decommissioned), and a backlog with hundreds of items tracked to closure

Problem

The infrastructure ran its first stretch of working sessions on a small set of Word documents. It worked, and it accumulated four frictions that a growing fleet made untenable:

• Edit fragility. Every update meant unpacking the document XML, splicing into byte ranges, and repacking. Multi-table documents silently dropped rows when a conversion shortcut was used. Header cells grew into 90KB blobs no editor handled gracefully.
• No version diff. Reviewing what changed between two revisions meant diffing zipped XML.
• No same-session updates. Because a one-line change cost twenty minutes of manual repack, updates batched to end-of-session, and then didn't happen. Systems got reconfigured mid-week with the documentation left behind.
• No structural search. Searching unpacked XML returns matches wrapped in formatting tokens. Finding "where is this documented" across 14 files was an XML traversal problem.

But the friction was the surface problem. The real one is the reason this case study exists:

The operator works in outcomes, not code. The entire fleet is built and run through AI coding agents, and decisions are judged in plain English, so the written record has to be authoritative on its own, not something you'd cross-check against a diff. That makes the agent's factual accuracy load-bearing for the whole system. A language model asked "what port does this service run on?" will produce a confident, plausible, wrong number. There is no natural error signal. A fabricated port becomes a fabricated firewall rule becomes a broken deployment, and the human in the loop cannot catch it by reading the diff.

The requirement was therefore not "better documentation." It was a machine-greppable factual substrate that an agent can be forbidden from answering around.

Approach: eight decisions locked before a line was written

One session produced eight architecture decisions that still hold three months later:

1. Markdown in a git repository, rendered as a static documentation site. The previous format was declared final.
2. One directory per application: an index page plus optional architecture, deployment, operations, changelog, and handoff pages. Decommissioned applications move to an archive directory rather than being deleted.
3. Same-session update discipline: any infrastructure change updates its page in the session where the change is made. Not deferred, not added to a todo list. This is the single most important rule in the system.
4. Retirement of a legacy dashboard deferred until after one session of real use.
5. Narrow write scope for the agent: markdown files and git operations on the working branch only; remote access, privilege escalation, force-push, and secret files all denied.
6. A mitigation pattern for any future privilege expansion: pre-execution hooks requiring typed confirmation.
7. A repository split deferred pending experience.
8. Third-party agent configurations are reference-only: never copy hooks, skills, or instruction files without line-by-line review and rewrite. Prompt-injection attack surface.

Three of the eight founding decisions were about constraining the agent that would maintain the documentation.

What got built

The content layer. Five trees: the infrastructure itself (machines, networking, firewall, database stack, backups, diagrams); one directory per application; numbered procedures (deploy checklist, promotion, decommission, git workflow, port registry, security principles); a running log (session summaries, backlog, incident log); and metadata about the documentation system itself.

The publish pipeline. Push to the release branch → CI runner → strict site build → served internally on the private network. The strict build is the last line of defense: it fails on broken internal links, missing navigation entries, malformed metadata, and pages without a heading. If it fails, the previous site stays live. It is never bypassed.

Branch discipline. All work lands on a working branch. Promoting to the release branch is a manual step the operator performs in the web UI; agents never touch it. That merge is the human review gate: the one place the operator can meaningfully approve, because by then the change is rendered prose, not a diff.

The binding rules, which live in a shared rulebook at the repository root that every agent reads at startup:

• The spec repo is the single source of truth for infrastructure facts.
• Search it before claiming anything about a service's ports, status, schema, secrets, or deployment state. Do not answer from memory.
• Say "the docs say X," not "X is true." The repo can carry fabricated facts from past sessions, so treat it as a strong source, not an infallible one. If two places disagree, surface the contradiction; don't silently pick one.
• Never call something a "convention" or "standing rule" unless it was said this session or verified by search. Inferring conventions is fabrication.

The third rule is the design's most unusual move. The source of truth is explicitly declared fallible, and the agent is required to attribute rather than assert. That converts a confident-hallucination failure into a citation a non-technical reader can challenge.

Key decisions and the why

• Markdown over a database or wiki. The system is deliberately not a CMS and not a wiki: no write API, no admin UI, no public read access. The entire value proposition is that it is a git repository of plain text: diffable, searchable, forkable, and readable by any agent with filesystem access and zero integration work.
• Session logs are part of the source of truth, not a changelog. More than half the files are session logs. They carry the why: the rejected alternative, the correction, the thing that turned out to be wrong. Current-state pages go stale; the log explains how the state got there, which is what you need to judge whether a fact is still load-bearing.
• Decommissioned systems are archived, not deleted. Two-thirds of the application directories describe things that no longer run. Deleting them would make "was there ever something on this port?" unanswerable.
• Store the corrections, not just the conclusions. Wrong diagnoses are preserved explicitly. When one backlog entry was compressed during a cleanup pass, its embedded "this diagnosis was unfounded" note was deliberately carried into the compressed line, because a wrong-diagnosis record must not compress into something that reads as clean.
• Three independent copies. The canonical git remote, a local working clone, and a cloud-synced folder. The rendered site is a reproducible output, not a primary copy.

What went wrong, and what it taught

The stale-artifact incident. Early on, the pipeline auto-generated a single concatenated file on every release, which was then manually uploaded into an AI assistant's knowledge base. It was eventually discovered that this artifact had been stale for multiple sessions: the local clone had drifted onto the wrong branch, so pulling fetched without updating the working tree, and nothing checked. The uploaded file was weeks old, silently. The procedure was hardened into a single atomic command with verification at each failure point. The deeper fix came later: once the agent could read the repository directly, the entire upload ritual was retired rather than hardened further. A manual sync step you keep hardening is a step you should be deleting.

The docs carry fabricated facts: measured, not assumed. A truthfulness audit checked hard claims across every session log then existing: the overwhelming majority verified clean, a small number were unverifiable (predating the git era), and a small number were confirmed false, plus one accusation refuted by a skeptic re-verification stage. The failure pattern was razor-sharp: every claim citing a commit hash checked out (often to exact insertion counts), and every falsehood lived in prose with no commit cited. The resulting rule: hard claims cite a commit or get marked unverified.

Drift is the steady-state condition. The first full re-derivation audit read a large set of current-state pages with parallel extraction agents, cross-diffed every checkable claim against a canonical baseline, and corrected a meaningful share of files. The dominant drift came from a single machine rename: a batch of documents still described a public, internet-facing tier, with tunnel and reverse-proxy infrastructure presented as live, that had not existed for two weeks. Second theme: status headers reading "plan, not applied" for work completed and verified days earlier. This audit is now a quarterly cadence item, not a one-off.

The rename that didn't happen. A backlog item proposed renaming the repository, since its name now collides with a well-known product. Asked for a straight answer on whether it was worth doing, the honest one was that it would touch hundreds of cross-references, break every historical log's paths, and buy nothing that day. So it was held, not executed. Worth recording because the discipline cuts both ways: the same verify-before-asserting rule that stops an agent inventing a port also has to stop it executing a tidy-sounding refactor with no payoff.

Outcome

• Hundreds of files, hundreds of thousands of words, over a thousand commits: three months from a small Word document set.
• The same-session update rule holds well enough that sessions carry their own log, and current-state pages are re-derived quarterly rather than trusted indefinitely.
• The repository is what every other project in this portfolio cites: the restore-drill runbook, the decommission procedure, the port registry, the machine table, the incident log.
• Agents across four different tools read one rulebook pointing at one substrate. Adding a new agent is a config file, not a knowledge-transfer project.
• Measured honesty: the overwhelming majority of hard claims verified clean, with a named failure mode and a rule that closes it.

Lessons

1. A source of truth for an AI agent is a different artifact than documentation for a human. Humans tolerate ambiguity and infer from context; the same inference in a language model is fabrication. The rules that matter (search before claiming, cite don't assert, surface contradictions rather than resolving them silently) are about removing the agent's license to interpolate.
2. Declare your source fallible in the rulebook. "Say the docs say X, not X is true" costs one line and converts a class of confident errors into visible, challengeable citations. A source of truth trusted absolutely is just a bigger hallucination.
3. Cheap updates or no updates. The old format's real defect wasn't the format, it was that a one-line change cost twenty minutes, so it didn't happen. Same-session discipline was only enforceable once the format made it nearly free.
4. Store the wrong answers. The retracted diagnoses, the refuted accusations, the "this was applied but the banner still says draft" corrections: those are the entries that stop the next pass from re-litigating a closed question.
5. Drift is measured, not prevented. A meaningful share of files went stale from one rename. No discipline stops that; a scheduled re-derivation audit catches it. Budget for the audit instead of promising harder.
6. The manual step you keep hardening is the step to delete. Three rounds of hardening an artifact-sync ritual were all obsoleted the moment the agent read the repository directly.
7. Constrain the maintainer on day one. Three of eight founding decisions limited the agent that would write the docs: narrow write scope, typed-confirmation hooks for privilege expansion, and no copying third-party agent configs without review. Retrofitting those onto a running system would have been far harder.