I turned a passive usage logger into the mandatory gateway every app's AI traffic flows through (with per-function routing, a central kill switch, privacy-by-architecture provider locks, pre-flight cost caps, automatic cloud-to-local failover, and a recommendations engine), then migrated most of the fleet's AI-calling apps onto it (a couple explicitly deferred) and made "all AI routes through here" an enforced architectural rule rather than a convention.
Context: this runs a self-hosted homelab I operate, a fleet of apps across several machines connected by a private VPN.
The challenge
The earlier version of the AI gateway was a local-model-only usage logger: a proxy fronting the local model runtime that recorded token counts, latency, and electricity cost, tagging traffic by app via a URL path prefix. It never touched hosted-LLM traffic at all.
That left three real gaps. Apps called AI providers directly: each hit the hosted API with its own key, or the local runtime directly, so there was no central chokepoint. There was no cloud cost or usage visibility, because those calls bypassed the gateway entirely. And there was no governance. Model choice was per-app via individual settings dropdowns (changing one meant editing and restarting the app), with no way to lock a sensitive function to local-only inference, cap a per-request cost, or kill a runaway app's spend centrally. The hardware made this matter: the production node has no usable GPU, so real work had to go to paid hosted APIs, which is exactly when cost governance stops being optional.
My objective: elevate it from a logger into the AI operations control plane for the whole fleet, one mandatory gateway proxying both local and hosted traffic, with unified logging, per-function model routing, policy enforcement (including privacy locks), per-request cost capture, and a recommendations engine analyzing real usage. A business goal rode on it too: routing app inference through per-token hosted rates was what made it safe to drop the AI subscription a tier, an estimated $80–180/month saving.
Approach: route on the function, make the routing table the enforcement boundary
- Dual proxy with per-app key injection. A hosted-provider proxy sits beside the existing local one; it resolves each app's key server-side, so keys never live in app code. Per-app keys (not one shared key) mean spend can be killed for one app without touching others. Missing key → fail-loud 403 with a diagnostic.
- Register functions, not just apps. Apps register AI functions (
app/function), auto-populated on first request, so consumers need zero provisioning code. This granularity is what makes per-function model overrides, cost attribution, and privacy locks possible. - The routing table IS the policy boundary. Provider locks, cost caps, kill switches, and model overrides live centrally in the registry, so a consumer can't bypass policy by shipping a bad config. Governance is architectural, not cooperative.
- Encode identity in the request itself. Every call is self-documenting, and development vs production spend separate cleanly with independent kill switches.
- Establish the rule, then migrate against a working reference. "All AI traffic flows through the gateway: no app calls a provider directly" became an enforced standing rule the moment the proxy, registry, and policy table existed.
Results
- A full set of routed endpoints at close: proxy (local + hosted), admin/stats/registry/recommendations, model management, alerts, and reports.
- A unified request log carrying provider, function, exact cost, machine, and retry/timeout observability columns: one place to see every AI call the fleet makes.
- Most of the fleet's AI-calling apps live on the control plane, a couple explicitly deferred with documented reasons (one pending a prerequisite secrets migration, one whose AI feature was still a stub). All direct-to-provider calls eliminated for the migrated set.
- Governance is real and verified live: a central kill switch, per-function model override without app restarts, privacy locks enforced at the gateway, pre-flight cost caps that reject before spending, and automatic cloud→local failover, each demonstrated with a concrete before/after (a provider-lock 403, a cost-cap pre-flight rejection, a timeout that failed over to the local model and succeeded).
- A recommendations engine on a 6-hour schedule turning real usage into four classes of deduplicated advice: overspend (cost over threshold), underpowered (high retry rate → upgrade model or raise timeout), model overkill (expensive model on tiny prompts), and idle (no requests in 14 days → removal candidate).
- Enabled the subscription downgrade the effort was partly justified by. The $80–180/month figure is the planning estimate, stated honestly as such, not a measured post-migration number.
Verifying the first migrated consumer end-to-end surfaced a stale auto-deploy, and revealed that every other app still called providers directly, which is what turned a verification step into the trigger for the whole migration batch.
What this demonstrates
- Make governance architectural, not cooperative. Putting locks, caps, and kill switches in a central routing table, not app config, means a consumer physically cannot bypass policy.
- Route on the function, not the app. Per-function registration made model reassignment, cost attribution, and privacy locks granular; auto-registration-on-first-request meant consumers needed zero provisioning code.
- Fail loud on missing keys. A 403-with-diagnostic turned silent misconfiguration into an immediately debuggable error: a later consumer's chat outage was diagnosed in minutes because of it.
- Idempotent schema migrations from day one. Every alter is safe to re-run at restart: essential for a service every other app depends on.
A companion piece covers hardening this same system's own attack surface.