Hey friends, today is the day!
Error analysis, a six-mode failure taxonomy, deterministic checks, binary LLM judges calibrated with TPR and TNR, durable evals on Inngest… it's all in here.
But looking back, every lesson so far has also been quietly leaving pieces of an evaluation system lying around.
In Lesson 1, the CostCaptureMiddleware started recording every token spent inside an Inngest step. In Lesson 2, POST /ask began saving a context snapshot with every candidate the retriever surfaced, left its confidence column empty, and sent a kb/answer.produced event that nobody was listening to yet. In Lesson 3, handle-function-failed started collecting real failures in the failures table. And in Lesson 4, every claim got a type (fact, mechanism, opinion, prediction or recommendation) and our first deterministic guardrail.
👉 Today, all of those threads come together.
Most AI tutorials treat evaluation as an afterthought: import a bloated eval library, ask an LLM to score your answers from 1 to 5 on "helpfulness", "coherence", and "conciseness", and plot a green average of 4.2 / 5.0 on a dashboard.
👉 In production, those dashboards are comforting lies.
They never tell you why a citation pointed to a superseded chunk or why your synthesizer ignored a conflicting opinion from another newsletter.
In Lesson 5, we combine Hamel Husain's bottom-up Error Analysis methodology with Inngest's durable evaluation architecture to build an evaluation suite grounded in our system's actual failures, keeping the prose focused on the core architecture and saving the live terminal, Postgres, and MCP walkthrough for the companion video.
☝️ What do you think about these new video format? Helpful? Let me know in the comments!
Enjoy!
Why generic "vibe evals" fail?
As Hamel Husain and Shreya Shankar emphasize in their research on production LLM evaluations, the #1 anti-pattern in AI engineering is jumping straight to automated LLM-as-a-judge metrics before looking at your own data.
Instead of guessing what to test, effective evaluation follows a five-stage loop adapted from qualitative grounded theory:
Open Coding Over snapshots and failures
Before writing a single judge prompt, we pull real rows from our snapshots and failures tables and perform Open Coding: a human engineer reads through traces and writes raw, unconstrained notes describing the first thing that went wrong.
“Answer cited
[4]at the end of the sentence, but only 2 passages were retrieved.”“Retrieved passages had Miguel (The Neural Maze) arguing 3-way RRF avoids reranker latency and Paul (Decoding AI) arguing cross-encoder rerankers are essential, but the answer presented
[1]as unanimous consensus.”“Passage
[1]was tagged as a forward-lookingprediction, but the answer wrote ‘Benchmarks prove...’.”
Axial Coding Into a 6-Model Fail Taxonomy
Next, we group those raw open-coding notes into mutually clear axial categories. In The Substack Brain, every observed defect falls into six failure modes across retrieval and synthesis:
Notice why storing all candidate passages in snapshots.retrieved back in Week 2 was so important: when an answer fails, snapshots.retrieved immediately tells us whether the right passage was never retrieved (FM-6, a retrieval bug) or retrieved and mishandled (FM-1–FM-5, a synthesis bug).
Level 1: code before judges
Here is a non-negotiable rule of Harness Engineering: never pay an LLM to judge what five lines of Python can verify deterministically.
Before evaluate-answer invokes any LLM judge, run_deterministic_checks executes four zero-cost, sub-millisecond assertions over the snapshot:
def run_deterministic_checks(
*,
answer_text: str,
citations: list[dict[str, Any]],
retrieved: list[dict[str, Any]],
prompt_limit: int = 5,
) -> DeterministicEvalResult:
inline_indices = extract_inline_citation_indices(answer_text)
max_valid_idx = min(len(retrieved), prompt_limit)
citation_idx_map = {
int(c["citation_idx"]): c
for c in citations
if isinstance(c, dict) and "citation_idx" in c
}
# FM-1: Every inline [k] is within 1..max_valid_idx and present in citations
bounds_ok = bool(retrieved or not inline_indices) and all(
1 <= idx <= max_valid_idx and idx in citation_idx_map for idx in inline_indices
)
# FM-1: Every cited passage_id actually exists in snapshots.retrieved
retrieved_by_pid = {str(r["passage_id"]): r for r in retrieved if "passage_id" in r}
subset_ok = all(str(c.get("passage_id", "")) in retrieved_by_pid for c in citations)
# FM-2: No cited passage or claim has valid_to set (temporal supersession check)
freshness_ok = all(
c.get("valid_to") is None
and retrieved_by_pid.get(str(c.get("passage_id", "")), {}).get("valid_to") is None
for c in citations
)
# FM-3: Abstain when retrieved is empty; require >=1 citation when answering
abstain = is_abstain_answer(answer_text)
discipline_ok = (
(abstain and not inline_indices and not citations)
if not retrieved
else (True if abstain else bool(inline_indices and citations))
)
...
If an answer hallucinates [6] on a 5-passage prompt (FM-1) or cites a superseded revision (FM-2), Python catches it in 0.2ms for $0.00.
Level 2: narrow binary judges, calibrated
For FM-4 (faithfulness) and FM-5 (disagreement), Python isn't enough. We need an LLM as a judge. But we follow three rules from Hamel Husain's approach.
PASS or FAIL, never 1 to 5. Nobody knows what a faithfulness score of 3.5 out of 5 means in practice. A binary verdict forces a clear line: either every claim in the answer is supported by its cited passage, or at least one isn't.
Critique first, verdict second. Our output schema puts the critique before the verdict, so the judge has to compare each sentence with its passage and the claim’s type before it decides:
class BinaryVerdict(BaseModel):
critique: str = Field(
min_length=10,
description=(
"Step-by-step verification comparing each sentence and citation in the "
"answer against the retrieved passages and claim types."
),
)
verdict: Literal["PASS", "FAIL"]
failure_code: Literal["NONE", "FM-3", "FM-4", "FM-5"] = "NONE"One narrow judge per failure mode. A single judge that checks everything at once mixes criteria up. So we have two separate judges, built with the same hand-written agent layer from Lesson 4.
faithfulness_judge_agent checks that every sentence is grounded in its passage and that an opinion or a prediction is never upgraded into a measured fact. disagreement_judge_agent only runs when the retrieved passages come from at least two authors or publications. When they all come from one author, there's nothing to disagree about, so it skips the call entirely and records NOT_APPLICABLE.
Grading the Grader
An LLM judge is a classifier, and accuracy is the wrong way to measure it. If 95% of your answers are fine, a broken judge that always says PASS scores 95% accuracy while catching none of your failures!
So we measure two separate rates against human labels. The true positive rate (TPR) is the share of truly bad answers that the judge marks as FAIL: how many failures it catches. The true negative rate (TNR) is the share of truly good answers it marks as PASS .
scripts/calibrate_judges.py computes both on a human-labelled set of traces in evals/labeled_traces.jsonl, and a judge is only trusted to score real snapshots when both rates are at least 0.90.
Durable evals with Inngest
Why run evaluations in Inngest, instead of a cron script or right inside POST /ask? Because evaluation is a durable workflow problem, for three reasons.
Users never wait for it. Running two LLM judges inside POST /ask would add several seconds to every question. Instead, /ask saves the snapshot with confidence empty, sends kb/answer.produced and returns. The event we started sending in Lesson 2 finally has a listener: evaluate-answer picks it up in the background.
Retries never pay twice. Each stage of evaluate-answer (load-snapshot, score-deterministic, judge-faithfulness, judge-disagreement, persist-eval-scores) is its own step. Inngest saves the result of every step, so if judge-disagreement hits Anthropic's rate limit, only that step is retried. The faithfulness judge never runs, or gets billed, a second time.
Experiments run side by side. POST /evals/run sends a kb/eval.run event, and run-eval-suite benchmarks four retrieval variants on the same golden set of questions: full-text only (sparse_bm25), vector only (dense_pgvector), the two-way fusion from Lesson 2 (hybrid_2way_rrf_k60) and the three-way fusion from Lesson 4 (hybrid_3way_rrf_k60). Each variant runs in its own step, alongside a step that replays the failures collected since Lesson 3.
When the suite finishes, GET /evals/latest and the tenth MCP tool, kb_eval_report, show the results. On our golden benchmark, they make a clear case for the graph we built in Lesson 4:
Full-text only reached a
Recall@5of0.67. It nails exact identifiers likecreate_functionoruq_passages_article_chunk, but misses paraphrases.Two-way fusion reached
0.83, with a cross-publication coverage of0.50. It fixes single-article recall, but misses disagreements between newsletters when authors use different words.Three-way fusion reached
1.00on both. Following topics to claims in Memgraph connects authors across The Neural Maze, Decoding AI and Ahead of AI.
The golden set is small, so read these as a strong signal rather than a final verdict. The point is that we now have a repeatable way to compare retrievers on real questions, instead of picking one on vibes.
Running the Week 5 Evaluation Suite
Let's walk through the exact commands to run the Week 5 evaluation suite from your terminal and what happens under the hood after each step.
Set up and load the newsletters
cp .env.example .env # set ANTHROPIC_API_KEY, OPENAI_API_KEY and PRIORITY_BACKFILL_S=0
uv sync --all-extras
make start # Postgres, Memgraph, the Inngest Dev Server, migrations and FastAPI on :8000In a second terminal, backfill the three newsletters:
curl -X POST http://localhost:8000/publications -H "Content-Type: application/json" -d '{"feed_url": "https://theneuralmaze.com/feed"}'
curl -X POST http://localhost:8000/publications -H "Content-Type: application/json" -d '{"feed_url": "https://www.decodingai.com/feed"}'
curl -X POST http://localhost:8000/publications -H "Content-Type: application/json" -d '{"feed_url": "https://magazine.sebastianraschka.com/feed"}'Calibrate the judges
# Offline mode (runs without API keys using deterministic + heuristic evaluators)
uv run python scripts/calibrate_judges.py --mode offline
# Live mode (calls Claude against the human-labeled dataset)
uv run python scripts/calibrate_judges.py --mode liveThe script loads the 20 labelled traces (10 good answers and 10 bad ones covering FM-1 to FM-5), runs the deterministic checks and then the judges on each one, and compares the verdicts with the human labels. It prints TPR and TNR for each evaluator and for the whole pipeline. For any trace it gets wrong, it shows the judge's critique next to the human's note, so you can see exactly where they disagree. It exits with code 0 when both rates are at least 0.90, and 1 otherwise, so you can use it as a gate in CI.
Twenty traces is a starting point, not a finish line: with ten bad answers, one miss is the difference between passing and failing the gate. The fixture is meant to grow every time error analysis finds a new kind of failure.
Ask a question and watch it get scored
curl -X POST http://localhost:8000/ask \
-H "Content-Type: application/json" \
-d '{"question": "How do durable execution engines handle failure recovery compared to traditional queues?"}'The answer comes back exactly as before. Almost all of the wait is Claude writing it, and none of it is evaluation. Right after saving the snapshot, /ask sends kb/answer.produced, and in the Inngest dashboard at http://localhost:8288 you'll see evaluate-answer run its five steps.
The last one writes a row to eval_results and fills in snapshots.confidence: 1.0 if everything passed, 0.5 if only the disagreement check failed, and 0.0 if a deterministic check or the faithfulness judge failed.
Then look at the result:
docker compose -f infra/docker-compose.yml exec -T postgres \
psql -U substack_brain -d substack_brain -P pager=off -x -c \
"SELECT s.snapshot_id, s.confidence, e.deterministic_passed, e.faithfulness_verdict, e.disagreement_verdict, e.failure_codes, LEFT(e.faithfulness_critique, 120) AS critique FROM snapshots s JOIN eval_results e ON e.snapshot_id = s.snapshot_id ORDER BY s.created_at DESC LIMIT 5;"For your five most recent answers, you'll see the confidence score, whether the deterministic checks passed, both judges' verdicts (PASS, FAIL or NOT_APPLICABLE), the failure codes they triggered, such as ["FM-4"], and the start of the judge's critique.
Compare the retrievers
curl -X POST http://localhost:8000/evals/run \
-H "Content-Type: application/json" \
-d '{"suite_name": "golden_v1", "include_failures_table": true}'This returns straight away with an event_id and a trace_url. In Inngest, run-eval-suite runs seven steps: it loads the questions from evals/golden_benchmark.yaml and the failures table, evaluates each of the four retrieval variants in its own step, groups the recorded failures by function, and saves the side-by-side results to eval_runs.
When it's done, read the report:
curl http://localhost:8000/evals/latestYou get the totals (how many answers were scored and how many passed each check), the count of each failure mode, the answers with the lowest confidence along with their critiques, and the latest comparison of the four retrievers. Any coding agent connected over MCP can read the same report with kb_eval_report and tell you which failure mode, or which retriever, needs attention.
What's next
At the start of the week, we had no idea how good our answers were. Now every answer is scored in the background without slowing anyone down, every failure is labelled with a mode we discovered by reading real traces, the cheap checks run before the expensive ones, and the judges have to prove they agree with humans before we trust them. And we can compare retrievers on real questions instead of guessing.
That's the foundation for the last lesson. In Lesson 6, we put a research agent on top of everything we've built, and take the whole system to production.
All the code from this lesson is in the course repository.
The Substack Brain, lesson by lesson
Lesson 1: A durable ingestion pipeline (article · video)
Lesson 2: Flow control and hybrid retrieval (article · video)
Lesson 3: A freshness poller and an MCP server (article · video)
Lesson 4: A knowledge graph in Memgraph (article)
→ Lesson 5: Error Analysis for Agent Products
Lesson 6: A research agent in production
Our sponsor
The Substack Brain is made possible by Inngest, the durable execution engine behind this course. Every step's result is recorded, so a crashed run resumes instead of restarting, and the flow control we used this week (concurrency keys, throttling and priority) is configured right on the function, with no queues or workers of your own to run.
If you're building AI pipelines or agents that need to survive the real world, give Inngest a try. Thanks to the Inngest team for supporting this course 💙
See you next week!
Error analysis rocks!