Case study
Scope: an end-to-end usage and cost pipeline for AI coding work across a self-hosted fleet: collector, pricing layer, API, dashboard, and a native mobile client. Headline numbers: 93 per-day rows · 74 activity days of history recovered · 100% priced, zero unpriced · lifetime ≈ $17,560 of subscription-covered usage, of which the obvious data source showed only ~28%.
The AI coding work runs on a flat-rate subscription. The tooling never bills per token, which sounds like the absence of a problem and is actually the source of one: there is no natural cost signal at all. No invoice arrives, no meter ticks, and nothing pushes back when a session burns half a million tokens on routine read-and-check turns.
Three concrete decisions were blocked on numbers nobody had:
And a fourth, softer one: knowing what the equivalent metered API cost would have been is the only way to say whether a flat rate is a good deal. That number turned out to be far larger than anyone guessed.
The machinery had been built weeks earlier and left unwired. Checking it produced the first finding, and it's a good one:
Every dashboard file on the server hash-matched the authoring copies, so the build genuinely was deployed and current, including later fixes. And the usage table held zero rows. Nothing ran the collector, anywhere.
The collector had been copied to the wrong machine: one with no meaningful transcripts on it at all. The real data lives elsewhere in the fleet. The pipeline was correctly built, correctly deployed, and pointed at an empty disk.
A second dead component surfaced alongside it: a manual-logging table with zero rows, never used once. Worth separating carefully, because the assumption that the import would retire it was half right: it doesn't replace that feature (which tracks browser sessions with no API and no local files, genuinely unimportable). It was simply dead independently.
The build carried a blocking question: was the ~312k output-token session baseline line-summed, and therefore up to ~3× too high in the unsafe direction? Everything about the overnight schedule was sized off that figure.
Re-measured over the same window: naive line-sum 573,035 tokens against deduplicated 187,295. The recorded ~312k sits far below the naive figure, so it demonstrably did not multi-count. An independent corroboration came from a completely different method that never touches the transcript files at all: a live headless run reaching ~317k across seven jobs before exhausting the window.
The 3× risk is real, just not what happened. Measured duplication was 3.00× on the laptop's archive and 3.98× on the always-on box's, the latter higher because a backup folder duplicated 22 transcripts, which a global message-ID dedup absorbed exactly as designed.
One honest wrinkle: exact reconciliation was impossible because the documentation recorded that window in local time while the collector bucketed UTC. That produced a genuinely confusing mid-session comparison, and is why the collector grew an explicit UTC flag for forensic re-derivation.
The pricing module listed the current flagship model in its "unpriced" set, and that model was 100% of every day's usage for the preceding week, roughly 20% of all volume ever. The dashboard's headline cost figure was excluding most of the usage it exists to measure.
Pricing three models closed it. The scale of what had been invisible: one older model alone accounted for 1.85 billion tokens ≈ $1,806 that the cost display had been silently reporting as nothing.
The deliberate choice was to keep the mechanism with the set now empty, because an empty set still routes an unrecognised model to a visible "unknown model" label rather than to a $0 that reads as free, which is the entire point of having the mechanism.
The tooling prunes its transcripts at roughly 30 days. Having verified that transcripts stopped six weeks back, the conclusion offered was that the earlier months were "unrecoverable."
The operator pushed back (his own usage view showed him statistics going back to May), and he was right. A cumulative statistics cache survives the pruning entirely, and nothing in the pipeline was reading it.
What it holds, at three different resolutions:
| Layer | Range | Recoverable? |
|---|---|---|
| Lifetime totals per model | all of it, no dates | yes |
| Daily activity (messages, sessions, tools) | ~3 months | yes, but no tokens |
| Per-day tokens by model | ~1 month | same window as transcripts |
So per-day token detail before the pruning horizon is genuinely gone (the narrow part of the original claim survives), but the totals are not, and they are 3.5× larger than the transcripts showed. Proof that the file accumulates rather than recomputing from disk: it still reports two models with no surviving transcripts anywhere on the machine.
Transcripts covered ~28% of true lifetime usage. Lifetime lands at ≈ $17,560 of subscription-covered work against ≈ $4,763 for the trailing 30 days, and the two must never be summed, because lifetime already contains that window.
One machine has no statistics cache at all: only interactive clients write one, and that machine never runs an interactive client. Its early history is gone with no backstop, and the rest erodes unless another session collects it.
Collection is a session-wrap step, not a scheduled job. The machine holding most of the data is a laptop that is asleep at 3 AM, where a scheduled job doesn't merely fail: it fails silently. A wrap step fires when the machine is demonstrably awake. And because every run re-reports all days rather than an increment, a single run backfills every one that was missed. Idempotency turns "did it run?" into a question that doesn't matter.
The laptop collects the other machine over SSH, leaving nothing behind: the collector is copied over, run, and removed within the same step, so that box keeps no tooling and no credential.
A scheduled job on that box was considered and explicitly rejected: a net-new timed service plus a credential on a machine that holds none would trip the provisioning gate, for no gain. The rejection and its reasoning were written into the wrap procedure itself, so a future session doesn't helpfully re-propose it.
Idempotency was proven, not asserted. Re-ingesting an identical file returned "inserted 0, updated 66" with every total byte-identical: double-capture is structurally impossible, not merely avoided. The failure path was tested too: pushed at an unreachable server, it reports which legs failed and exits rather than hanging.
Standing traps were written where they'll be found. That the date and the machine name are both part of the record key (so changing the timezone or renaming a box re-buckets history under new keys and leaves the old rows inflating every total) lives in the collector's own docstring. That lifetime and per-day totals must never be summed is recorded next to both. And the error-swallowing operator on the wrap call is annotated as load-bearing: a failed usage push must never fail a session wrap.
Four issues surfaced on the first walkthrough, two of them design decisions rather than bugs. Four of the six substantive corrections across the whole project were his, not the builder's.
| Version | Approach | Why it failed |
|---|---|---|
| v1 | Four stat cards | Not understandable at a glance |
| v2 | Rolling five-hour sum, explained | Wrong concept: the limit is a fixed block, not a rolling window |
| v3 | Blocks anchored on long gaps | Right concept, wrong answer: claimed a reset time nearly three hours off |
He caught the third version by checking it against what the tool itself reported.
The cause is structural, not arithmetic. The limit counts all usage on the account (browser, desktop, mobile) while this data is coding-tool transcripts only. A browser conversation starts a block and leaves nothing on disk, so the anchor is simply unknowable from this data source. A fourth version reduced it to an honest lower bound with a disclaimer, and his verdict was that the number still didn't make sense. He was right: the top of the page was occupied by a tile whose real message was "this isn't the number you want." Removed, with the reasoning written above the function in the source so it survives the next person's good idea.
A native client followed: no third-party dependencies, four tabs, exactly one write action (a kill switch) behind a confirmation and deliberately built as a button rather than a toggle, because a toggle fires the write before the confirmation is answered.
Three bugs, two found by him on first use. The client initially targeted a route guessed from a database table name rather than the real one. A debug preset filled in a placeholder host, fixed by making presets carry the address only, never a credential, so no code path exists by which a token reaches any build. And costs over $1,000 printed a literal format string, because the formatting call is C-style printf and the thousands-separator specifier isn't valid there: values under 1000 took the other branch and looked fine, which is exactly why it read as one odd tile rather than a broken app.
His question, "can you preload the token... unless that is a bad idea," drove the credential work. It was a bad idea, but the concern underneath was right, and the first answer to it was wrong: the credential was assumed reachable from his phone, and it wasn't, because the infrastructure credentials live under a separate account from the one his phone carries. The proposed fix would have collapsed a separation he had built on purpose. Proposed, then withdrawn.
The real fix was to make the credential smaller: hashed scoped tokens mirroring the existing secrets pattern rather than inventing a scheme, with an absent file preserving previous behaviour so the deploy was behaviour-neutral and verified as such before minting anything. The phone keeps the kill switch by explicit choice, which is safe because of the asymmetry: a leaked kill switch is reversible from any browser, while a leaked delete route destroys data.
Enforcement checks the request body, not just the route: the update endpoint accepts a dozen fields, so a route-level allow would have handed the phone the entire routing policy. 18 unit cases including path-confusion probes, plus 12 live checks with a real token.
A pre-existing bug surfaced along the way: unmatched API requests hung open rather than returning 404, the result of a bare return in the fallback handler that neither responded nor passed control on. Found only because a verification script was sloppy enough to hang on it, where a tidier one would have timed out and moved on.