Tracing and Scoring a RAG Pipeline Without Slowing It Down

The question that decides whether evaluation survives contact with production is not "which metrics?" It is "when does the judge run?"

Run it inline and every answer waits for a second model to grade the first. Run it in a nightly batch and you learn about a bad week on Monday. The pipeline from the first post does neither. Score after you answer. The judge starts the moment the response stream closes, on the same event loop, and it attaches its scores to a trace that was recording the whole time.

I wrote about which metrics matter in Evaluating Generative AI in Production. This post is the wiring.

Trace first: one callback per request

Langfuse is opt-in. It runs as a Docker Compose profile (--profile observability) with its own Postgres, and it is off unless three environment variables are set. When it is on, the chat endpoint builds one callback handler per request and passes it into the graph's config:

handler = make_chat_handler(thread_id, skill, payload.web_search)
if handler is not None:
    config["callbacks"] = [handler]

LangGraph propagates that handler to every node and every nested LLM call. The result is a single trace per request: contextualize → retrieve_vector → rerank_vector → grade_documents (one generation span per document) → generate → reflect (two spans) → finalize, each with its prompt, its tokens and its latency. The trace is grouped by session_id = thread_id, so a whole conversation reads as one session, and tagged with the skill and whether web search was on.

The same callback works whether the LLM is a local Ollama model or a hosted one. That was a requirement, not an accident: the graph LLM is pluggable and the tracing must not care.

Every node also logs node=<name> elapsed_ms=<n> and emits a timing event on the SSE stream. When something is slow, you do not guess. The reranker fix from post one came straight out of those numbers.

The stream, and the moment it closes

The SSE stream has a fixed order: thread (the id, so a client that omitted one can capture it), then interleaved status, sources and token events, then a terminal done. Tokens are filtered to the generate node only, so the grading and reflection calls never leak into the answer.

When the stream ends, the endpoint reads the final persisted state for the thread with graph.aget_state(config) rather than trusting what it accumulated. That gives it the answer, the source list, and which [[n]] ids the answer actually cited. It writes the citation map onto the trace, yields done, and only then does anything about scoring happen.

Fire and forget, with three safety nets

task = asyncio.create_task(evaluate_and_score(trace_id, question, answer, chunks))
_eval_tasks.add(task)
task.add_done_callback(_eval_tasks.discard)

Three things in those lines are load-bearing.

It is a task, not an await. The response has already gone out. The judge runs on the event loop after done, and because judge calls are I/O-bound awaits, it does not starve other requests.

The task is held in a module-level set. A bare create_task with no reference can be garbage-collected mid-flight. The set keeps it alive; the done-callback removes it. This is the standard fire-and-forget pattern in asyncio and it is easy to get wrong.

It cannot fail the request. The scheduling call is wrapped in try/except. The body of evaluate_and_score is wrapped in try/except and logs. Each metric inside it is wrapped in try/except, so one metric failing does not drop the others. DeepEval itself is imported lazily inside the function, so if the package is missing or broken the chat path never notices. I looked hard for a way scoring could affect the response and did not find one.

The judge is not the graph model

The graph runs on whatever LLM_PROVIDER says, often a small local model. Judging with the same small model produces scores that are indicative at best. So the judge has its own settings, EVAL_PROVIDER and EVAL_MODEL, falling back to the graph LLM only if unset. Score on something stronger, or on something local to protect a metered quota; the app does not care.

Two implementation details worth stealing:

• Retry inside the judge. Hosted providers return transient 5xx errors. The judge wrapper retries twice with linear backoff before giving up on a metric, so one blip does not fail a whole score.
• Refuse the schema kwarg on purpose. DeepEval will ask the judge for structured output when it can. Small and hosted models are unreliable at that. The judge's generate deliberately does not accept the schema argument, which raises a TypeError that DeepEval catches, falling back to plain text it parses itself. It is a documented hack, and it works.

Three live metrics, no reference answer

On live traffic there is no ground truth, so the metrics are referenceless. DeepEval's GEval takes a natural-language rubric and returns a 0 to 1 score with a reason:

The third one is the tuning signal. A low relevancy score with a high retrieval score means the right chunks are in there but so is a lot of noise: chunk size or top-K is too generous. It can be switched off (EVAL_CONTEXTUAL_RELEVANCY=false) to save judge calls on a metered provider.

Scores land on the request's trace by id, so in Langfuse you open a run and see the nested spans, the prompts, the latency, the citation map and the three scores in one place. When a practitioner says "that answer was wrong", you have everything.

Two metrics live traffic cannot compute

Recall and precision need a reference answer. "Did retrieval capture everything the correct answer needs?" is unanswerable without knowing the correct answer. So those two run offline, against a golden set per skill:

python -m app.eval.offline --skill regulatory-research
python -m app.eval.offline --limit 5          # quick subset

Each golden item is a question plus an expected_output. The harness runs each one through the real compiled graph, scores contextual_recall (did retrieval get everything the reference needs, which points at embeddings, chunking and top-K) and contextual_precision (are relevant chunks ranked above irrelevant ones, which is the reranker's report card), prints a table, and records a Langfuse dataset run so you can diff two commits.

The harness and the API share one build_runtime(). Whatever the app wires at startup, the offline evaluator wires identically. Evaluating a different graph than the one you ship is a classic way to be confidently wrong.

What to watch

• Judge cost. Every live request adds judge calls. That is why it is off by default and why the judge is decoupled. Turn it on for a sample, not for everything, once you trust the numbers.
• Small judges flatter. On a tiny local judge the scores are directional. Point EVAL_MODEL at something strong before you put a number in a report.
• A yes/no gate that defaults to yes. The graph's own relevance and groundedness gates parse a one-word answer and default to "yes" when the model rambles. That bias is deliberate, it favours progress over spinning, but it means the gates get softer as the model gets smaller. The GEval scores are how you notice.

The takeaway

• Score after you answer. The judge starts when the stream closes. Zero added latency, and the client never waits on a grader.
• One callback, one trace. Attach Langfuse per request and let LangGraph propagate it; every node and LLM call lands in a nested trace grouped by thread.
• Fire and forget correctly. create_task, hold the reference, wrap every layer. Scoring must not be able to fail the request.
• Decouple the judge. Separate provider and model settings, retries for transient errors, plain-text fallback for models that cannot do structured output.
• Live for referenceless, offline for the rest. GEval and relevancy on real traffic; recall and precision against a golden set, through the same runtime the app uses.

Previous: Skills as Sub-Agents. That closes the three-part series on the adaptive-rag system: the human-chosen route and the capped self-reflection loop, the skill layer over one shared graph, and the observability that runs beside it without slowing it down.