LLM Evaluation Platform
An OpenTelemetry and Langfuse platform that gates every prompt, model, agent, and tool change on measured accuracy, latency, and cost.
- AI surfaces scored against golden datasets
- 9AI surfaces scored against golden datasets
- Cohen's kappa required before a judge may gate a release
- ≥ 0.7Cohen's kappa required before a judge may gate a release
- scorer tiers, escalating only when the cheaper one cannot catch the failure
- 4scorer tiers, escalating only when the cheaper one cannot catch the failure
- of live traffic scored continuously for drift
- 5–10%of live traffic scored continuously for drift
Nine AI surfaces were shipping on assertion. I built the evaluation platform that made all of them measurable before release: golden datasets harvested from work reviewers were already doing, a scorer ladder that spends nothing until it has to, and judge models that must prove themselves against human labels before they are allowed to gate anything.
The problem
A senior engineer asked whether we had an evaluation framework to justify swapping a production model. We did not. Prompts, models, agents, skills, MCP tools, and retrieval were all changing on reasoning rather than measurement — which meant nobody could say whether a change had improved anything, and a vendor retiring a model turned into a fire drill instead of a scored diff. Rather than defend the gap I filed it as the ticket, then built well past its original scope.
Constraints
- No new vendors. LiteLLM, Langfuse, LightRAG, and n8n were already self-hosted; the design had to wire together what existed plus one small container.
- Two very different audiences: workflow authors who need a UI, and statistical rigour that only a code harness can provide.
- Community-tier n8n, so custom span attributes were unavailable and trace tagging had to route through gateway request metadata instead.
- Judges are models too. Any LLM-as-judge score had to be proven trustworthy before it could block a release.
Architecture
Decisions
What I chose, why, and what I turned down to get there.
Instrument at the LiteLLM gateway, not in each workflow
Because no workflow or agent has direct model access, a single callback line on the proxy instruments every model call in the organization at once — n8n workflows, retrieval extraction and query calls, agent tool use. The chokepoint that existed for governance turned out to be the highest-leverage line of config in the whole design.
Considered and rejected
- Per-workflow instrumentation — N places to change, N places to drift
- Application-level SDK calls in each producer — couples every service to one backend
OpenTelemetry collector as the single ingest point
PII redaction, sampling, and batching happen once, in one place, and backends can be swapped or added without touching a single producer. Pointing callbacks straight at the tracing backend works on day one, but bakes in a vendor at the wire.
Considered and rejected
- Direct-to-backend OTLP as the permanent design — fine as a shortcut, a lock-in as an architecture
Langfuse for traces, datasets, prompt registry, and scores
It gave versioned prompts, datasets of record, annotation queues, and observation-level judges in one surface, on a stack already hosted internally. Evaluated against the alternatives on whether results from a separate code harness could be written back into the same place engineers already look.
Considered and rejected
- promptfoo — strong for prompt sweeps, weaker as a system of record
- DeepEval — good metric library, not a trace and dataset backend
- n8n native Evaluations alone — kept, but as a second tier, not the whole answer
Two evaluation tiers instead of one
Workflow authors get n8n's built-in Evaluations with Data Table datasets so they can iterate without writing Python. Everything statistical — repeated runs with variance, confusion matrices, sweeps, CI gating — lives in a pytest harness in git. One tier would have either blocked non-engineers or given up rigour.
Error analysis before metric selection
The rule above all others: no metric before reading 50 real traces. Doing this first repeatedly showed the top failure was something a free deterministic check catches — malformed CSV, empty tool response, wrong repository — and no judge was needed at all. Picking metrics from a list first is the most expensive way to start.
The scorer ladder, cheapest first
Every surface is scored by the least expensive method that can actually catch its failures, and only escalates when that is genuinely insufficient.
- Deterministic code checks — schema validity, label in the allowed set, file paths that exist, symbols that resolve, completeness as a set comparison. Free, and always right.
- Statistical metrics — scikit-learn per-class precision, recall, F1, and a confusion matrix. Overall accuracy hides the class that matters, so the minority class is always reported separately with a pre-committed recall floor.
- Ragas — faithfulness and answer relevancy for retrieval. Both are reference-free, so they double as online monitors on live traffic.
- LLM-as-judge — last resort, for prose where no single right string exists.
Validating the judges
A judge is a model making predictions, and it can be wrong. Until a judge agrees with a held-out, human-labeled set at Cohen's kappa of at least 0.7, its scores are not allowed to gate anything. Kappa is rechecked quarterly. This is the step that separates an evaluation system from a second unmeasured model.
Golden datasets that cost nothing to build
Ground truth arrives free from work people were already doing. Every time a reviewer corrects a generated test-case CSV or a human overrides a suggested severity, that is one labeled example. The system's job was to capture those corrections rather than commission a labeling effort.
Offline gates, online drift
Offline evaluation runs before release on a fixed dataset as a pass-or-fail gate in Jenkins CI. Online evaluation scores 5–10% of live production traffic continuously, because offline sets never cover everything and quality drifts under a prompt that never changed. Both are needed: one gates changes, the other catches the world moving.
The layer nobody else measures
Tool descriptions and output formats shape the behavior of every agent that connects to them, so the MCP layer gets its own evaluation. A fixed set of real user intents measures whether an agent picks the right tool on the first call. Running the same intents with 3, 7, 10, and 14 servers enabled shows where selection accuracy bends — which is where registry policy should cap defaults. The same intents across JSON, Markdown, XML, and prose answer 'which output format suits our models' with a number instead of a preference.
Stack
- OpenTelemetry
- Langfuse
- LiteLLM
- Ragas
- scikit-learn
- pytest
- Python
- Jenkins
- n8n
- AWS Bedrock
- Grafana
- Prometheus
What was mine
The evaluation framework, metric catalog, scorer harness, and CI gates are mine. The self-hosted Langfuse deployment underneath it is owned by platform DevOps — a deliberate split between infrastructure ownership and application ownership.