Replaced an opaque, poorly-performing off-the-shelf RAG stack with a purpose-built retrieval service: incremental content-hash ingestion, a fast similarity index, a least-privilege database role, and a clean authenticated query contract that a second app consumed without rework.
The challenge
An earlier attempt at a "knowledge base" retrieval layer had adopted an off-the-shelf RAG tool with an embedded vector store. It technically worked, but retrieval quality was poor: broad questions returned generic model answers instead of answers grounded in the ingested source. The stack was also opaque: there was no way to tune how documents were chunked, how retrieval ranked, or where the similarity threshold sat. It had also been wired to call a local embedding model directly, bypassing the mandatory AI gateway, on an untested assumption.
Two premises had calcified around the failure: that a real RAG layer "needed a dedicated AI machine," and that the off-the-shelf tool was good enough to keep. Both turned out to be wrong. The real problem was the ingestion approach (embedding one giant concatenated file) and the tool's opacity, not a hardware limit. A lean, embedding-only service would run fine on the existing hardware, against the Postgres database already there.
Approach
I built a self-hosted RAG service that serves as the retrieval substrate: not a chat UI, but the thing any app can call to get back relevant, cited chunks of a knowledge source. Three decisions drove the build-vs-adopt call:
- Transparency and tunability. Owning the chunking, retrieval, reranking, and similarity threshold was the decisive factor: exactly what the off-the-shelf tool couldn't do, and exactly what had made its retrieval poor.
- Reuse the database already running. A vector-search extension lives inside the existing database, so there was no separate vector database to operate, back up, or monitor: a whole operational surface eliminated.
- Compliant by construction. Routing embeddings through the mandatory AI gateway was satisfied natively by the design, rather than bypassed as the earlier tool had done.
Architecture: a small backend service (one process serving both an admin UI and a JSON API), a dedicated schema tracking collections (a multi-collection model; v1 seeds one), documents (each carrying a content hash that drives incremental ingest), and chunks (the embedding plus provenance columns, including heading, character offsets, and chunk index, so retrievals can cite location, not just the file). The embedding column carries a fast similarity index. The service connects to the database as a dedicated least-privilege login role scoped to only its own schema, through the connection pooler, never with admin credentials. The public contract is a single authenticated query endpoint returning ranked, cited chunks, with tunable top-K and similarity threshold, deliberately shaped to match the consumer app's existing config object so the two round-trip without translation.
Some execution highlights:
- Incremental, content-hash ingestion. The ingest job chunks each document by heading, computes a content hash per document, and upserts: unchanged documents skipped, changed ones re-embedded, removed ones reconciled out. No more re-embedding the whole corpus on every refresh.
- Token hygiene: never on disk, never in argv. The private-repo clone injects a read-only access token URI-encoded, then scrubs it from the cached repo's stored remote so the token never sits in
.git/config on disk. - Corrected a load-bearing wrong assumption. The earlier tool had bypassed the AI gateway on the belief that "the proxy may not handle embedding requests." The build proved it does, end-to-end.
- "Compliant by construction" still needed an observability check. After launch, ingest traffic was correctly routed through the gateway but silently not logged: a usage-logging predicate matched a stale endpoint name while the service used a newer one, so the telemetry write was skipped even though the data path worked. I fixed the predicate (preserving legacy callers) and verified logging with model + latency.
Results
- Live v1 ingest: a full document set chunked and embedded in a dedicated schema, behind a fast similarity index, embeddings confirmed routing and logging through the AI gateway.
- Query contract verified end-to-end: bearer auth, tunable top-K and similarity threshold, cosine search returning cited chunks with source path, heading, and similarity score.
- Adopted by a second app (an agentic coding IDE) without rework. Its config-gated, never-throwing, timeout-bounded retrieval call injected cited context into its chat flow and surfaced a "Sources" list in the UI. A smoke test returned a documentation-grounded answer citing five chunks at 0.75–0.78 similarity.
- Least-privilege by design: a dedicated database login role scoped to a single schema, via the pooler, off all shared/admin credentials.
- Retired the black box. The off-the-shelf tool it replaced was demoted on launch and later decommissioned.
What this demonstrates
- Build-vs-adopt hinges on transparency and fit, not just features. The deciding factor was that the tool's vector store was opaque and its chunking/retrieval/threshold couldn't be tuned. Reusing existing infrastructure also erased a whole operate/back-up/monitor surface.
- Store provenance alongside the vector. Heading and character offsets are what make retrieval citable, and citability is what turned a generic answer into a grounded one in the consuming app.
- Design the schema for later without migration. A collections table plus collection-tagged chunks let v1 ship one source while keeping every future source a pure data addition.
- "Compliant by construction" still needs a telemetry check. Verify the observability, not just the data path: routed correctly is not the same as logged correctly.