Knowledge Graph RAG
Tenant-scoped retrieval over a million-plus entity knowledge graph, with a five-layer failure taxonomy so a bad answer can be traced to its cause.
- entities in production across four content sources
- 1M+entities in production across four content sources
- retrieval modes benchmarked into a documented mode-per-tool policy
- 5retrieval modes benchmarked into a documented mode-per-tool policy
- independently measured failure layers, from ingestion to operations
- 5independently measured failure layers, from ingestion to operations
- golden questions drawn from real production traces, not invented
- 75–100golden questions drawn from real production traces, not invented
Agents that answer from product knowledge need retrieval that is both grounded and tenant-isolated. I took the knowledge graph from proof of concept to production over a million-plus entities, wired four content sources in with incremental update paths, and built the evaluation that says which of five possible failures caused any given bad answer.
The problem
Office OS agents were answering product questions without grounding, and plain vector retrieval could not follow relationships across root-cause analyses, test cases, release notes, and documentation. Multi-tenancy raised the stakes: retrieval had to be scoped so no query could surface another customer's data. And when a retrieval answer is wrong, the cause could be in any of five independent places — scoring only the final answer tells you something is broken but never what.
Constraints
- Tenant scoping is not optional. Retrieval crosses root-cause analyses and support content, so isolation is a correctness requirement, not a feature.
- Four content sources with different shapes and different update cadences, each needing an incremental path rather than a full rebuild.
- Graph extraction runs an LLM over source documents, so a cheap model swap there silently degrades the entire graph.
- Five query modes existed and tools were choosing between them by convention rather than evidence.
Architecture
Decisions
What I chose, why, and what I turned down to get there.
A graph layer over plain vector retrieval
The questions that mattered were relational — which connectors changed in releases that also touched authentication — and vector similarity cannot follow that. Entities and relations extracted into a graph, alongside vector search over the same corpus, made multi-hop questions answerable.
Considered and rejected
- Vector-only retrieval — fine for lookups, blind to relationships between documents
Evaluate retrieval per mode, per question type
Specific lookups, thematic questions, and multi-hop questions win under different modes, which is exactly why mode choice should not be folklore. Running all five modes over a golden set of real questions pulled from production traces — scoring recall@k, MRR, and context precision — produced a documented mode-per-tool policy with numbers behind it.
Considered and rejected
- One default mode for all tools — simpler, and measurably worse on two of three question types
Ingestion integrity as scheduled assertions, not evaluations
A dedup bug based on titles left dozens of stale documents in the knowledge base, found by a one-off manual scan. Source parity counts, duplicate and stale scans, an idempotency probe that re-ingests a known document, and a freshness lag check now run nightly with alerts. These are assertions with a right answer, not metrics with a threshold — treating them as evaluations would have been a category error.
Gate the graph extraction step on human annotation
Twenty-five source documents stratified across content types, with humans listing the entities and relations that should be extracted, scored for entity and relation precision and recall plus duplication and orphan rates. It is manual and rare, and it runs before any change to the extraction prompt, model, or chunking — because everything downstream is built on the graph being right.
Five layers, because it fails five ways
The system is not one thing: it is an ingestion pipeline, an LLM extracting a graph, a retriever with five modes, a generator, and a service. When an answer is bad, exactly one of these happened — the document never made it in, the graph extracted the wrong entities, the retriever pulled the wrong context, the model ignored good context, or the call timed out. Separate evaluation per layer turns a vague quality complaint into a pointed diagnosis.
- Ingestion integrity — source parity, duplicate and stale counts, idempotency, freshness lag. Nightly assertions with alerts.
- Graph extraction quality — entity and relation precision and recall against human annotation, plus duplication and orphan-entity rates. The evaluation almost everyone skips.
- Retrieval quality — recall@k, MRR, and context precision, measured per query mode and per question type.
- Answer quality — faithfulness and answer relevancy, reference-free so they also run on live traffic; correctness against golden facts offline.
- Operations — p95 latency, timeout rate, empty-result rate, cost per query, and publish-to-queryable lag.
Four pipelines, each incremental
Test cases, support documentation, root-cause analysis items, and release notes each got an initial load and a separate incremental update path, so the corpus stays current without rebuilding the graph. A dedicated release-notes retrieval tool followed, and the test-case generation flow was rewired to query the graph so generation starts grounded in the accumulated corpus rather than cold.
Exposed as a tool, not an endpoint
Retrieval reaches agents through the governed MCP registry, which means tenant scoping, authorization, and audit attribution come from the same layer that governs every other tool. Retrieval spans flow into the same OpenTelemetry pipeline as everything else, so its extraction and query calls are metered and scored alongside the rest of the estate.
Stack
- Neo4j
- pgvector
- PostgreSQL
- Python
- FastAPI
- Ragas
- Langfuse
- OpenTelemetry
- Model Context Protocol
- AWS
What was mine
The retrieval architecture, the four ETL pipelines, the evaluation design, and the mode policy are mine. Server provisioning and deployment for the retrieval service were handled with platform DevOps.