Hey friends! 👋
Welcome back to The Substack Brain! This is Lesson 4 of our free six-week course, where we build a living knowledge base from the AI engineering newsletters you actually want to read, one production problem at a time.
Where we left off
In Lesson 3 we turned our pipeline into a polite, self-updating subscriber and exposed the entire knowledge base to AI coding assistants via MCP.
Across the first three lessons, we built a durable ingestion foundation on Inngest (Lesson 1), scaled it across multiple newsletters with per-origin concurrency, global throttling, hybrid search, and context snapshots (Lesson 2), and added HTTP conditional polling (ETag / Last-Modified), non-destructive temporal versioning (valid_to), a production failures dataset, and an eight-tool MCP server (Lesson 3).
What breaks this week
Everything we have built so far still shares one blind spot: to Postgres, every newsletter is just a flat bag of passages.
Hybrid search (BM25 + pgvector) is great when the answer lives inside a single paragraph. It falls apart the moment you ask a question that crosses articles or authors:
"Which newsletters have written about Reciprocal Rank Fusion or pgvector, and what does each author actually claim?"
"Where do two authors disagree on the same topic?"
"How has an author’s position changed after editing an article?"
Vector search can find paragraphs with similar embeddings, and BM25 can match keywords, but neither knows who asserted what about which canonical topic.
The tempting fix is the standard "Graph RAG" tutorial: hand each article to an LLM and ask it to emit (subject, predicate, object) triples. That looks magical on three articles and collapses on thirty. Left unconstrained, the model invents edges like :DISCUSSES, :MENTIONS, and :TALKS_ABOUT, splits one concept into "pgvector", pg-vector, and PGVector, and paraphrases quotes the author never wrote. Within a week, no Cypher query works twice.
The system for Week 4
To build a knowledge graph that survives production, we add four pieces to last week’s stack:
🧭 A locked 5-node, 5-edge ontology (Publication, Author, Article, Claim, Topic) with deterministic topic canonicalization in Python, so the LLM never invents schema or splits a topic in two.
🤖 A 60-line hand-written agent (kb/agents/) orchestrated by a new 6-step Inngest function, `extract-claims`, triggered asynchronously whenever an article is ingested or updated.
🛡️ Our first free guardrail (kb/sources/quote_validator.py), which drops any extracted claim whose quote_span isn’t a verbatim substring of the article before it ever touches Postgres or Memgraph.
🕸️ Three-way Reciprocal Rank Fusion (kb/retrieval/hybrid.py) combining BM25, pgvector, and 2-hop Memgraph Cypher traversal—plus a ninth MCP tool, kb_sources, to trace any claim or passage back to its exact verbatim quote.
Locking the ontology before touching an LLM
Before writing a single prompt, we freeze the graph schema in kb/graph/ontology.py. Every node in Memgraph must belong to one of five labels, and every relationship to one of five edge types:
When the extractor LLM reads an article, it is not allowed to emit Cypher or invent node types. It only returns a typed Pydantic batch (ClaimExtractionBatch) containing claims, their claim_type (fact | mechanism | opinion | prediction | recommendation), a verbatim quote_span, and raw topic strings.
Python then canonicalizes every topic through a controlled alias dictionary (TOPIC_ALIASES) and NFKC slug normalization before any write happens:
def normalize_topic(raw: str) -> ExtractedTopic | None:
"""Deterministically map a raw entity/topic string to its canonical `(slug, name)`."""
cleaned = unicodedata.normalize("NFKC", raw).strip()
cleaned = re.sub(r"\s+", " ", cleaned)
lowered = cleaned.lower()
if lowered in TOPIC_ALIASES:
slug, name = TOPIC_ALIASES[lowered]
return ExtractedTopic(slug=slug, name=name)
slug = slugify(cleaned)
if not slug or slug in STOP_TOPICS:
return None
if slug in TOPIC_ALIASES:
canonical_slug, canonical_name = TOPIC_ALIASES[slug]
return ExtractedTopic(slug=canonical_slug, name=canonical_name)
return ExtractedTopic(slug=slug, name=display_name(cleaned))See it on GitHub: kb/graph/ontology.py
Whether one author writes ”RAG”, another writes ”Retrieval-Augmented Generation”, and a third writes ”retrieval augmented generation”, all three collapse into the single :Topic {slug: “rag”} hub in Memgraph.
A 60-line agent and a 6-step enrichment function
Why didn't we add claim extraction as Step 8 of ingest-article?
Because of the Asymmetry Principle from Lesson 2!
Chunking and embedding an article takes a couple of seconds; extracting structured claims with an LLM takes longer and has tighter rate limits. By triggering extract-claims off kb/article.ingested and kb/article.updated, new passages become searchable in Postgres immediately while graph enrichment catches up asynchronously in the background.
And because Inngest already gives us step checkpointing, retries, and state replay, we don't need LangChain or CrewAI. Our entire agent execution layer in kb/agents/core.py is a thin, typed wrapper around step.run:
async def run_agent[T: BaseModel](
step: inngest.Step | Any,
agent: Agent[T],
user_input: str,
step_id: str,
article_id: UUID | None = None,
) -> T | str:
"""Execute one LLM call for `agent` inside `step.run(step_id, ...)` so its
output is durably memoized by Inngest and its token cost is captured.
"""
async def _call() -> dict[str, Any]:
return await call_llm(
system_prompt=agent.system_prompt,
user_input=user_input,
model=agent.model,
max_tokens=agent.max_tokens,
output_schema=agent.output_schema,
article_id=article_id,
)
step_result: Any = await step.run(step_id, _call)See it on GitHub: kb/agents/core.py
If Memgraph is temporarily unreachable on Step 5 (sync-claims-memgraph), Inngest retries from Step 5 and replays the memoized LLM extraction from Step 2 (extract-claims-llm) without burning a single extra Anthropic token.
Our first free-guardrail: verbatim quote spans
LLMs love to helpfully summarize or stitch together two sentences when asked for supporting evidence. If a fabricated quote enters your knowledge graph, every downstream answer that cites it is compromised.
Before Step 4 writes to Postgres or Step 5 touches Memgraph, Step 3 (validate-quote-spans) runs a deterministic Python guardrail that costs zero tokens and under a millisecond:
def is_verbatim_quote(
article_text: str,
quote_span: str,
*,
min_chars: int = MIN_QUOTE_SPAN_CHARS,
) -> bool:
"""Return True iff `quote_span` is at least `min_chars` long and appears
verbatim inside `article_text` after normalizing consecutive whitespace.
"""
if not article_text or not quote_span:
return False
norm_quote = normalize_whitespace(quote_span)
if len(norm_quote) < min_chars:
return False
norm_article = normalize_whitespace(article_text)
return norm_quote in norm_articleSee it on GitHub: kb/sources/quote_validator.py
If quote_span isn’t an exact substring of the article text, the claim is dropped on the spot and counted in dropped_spans. Surviving claims get a deterministic claim_id = uuid5(NAMESPACE_URL, f”{article_id}:{content_hash}:{norm_quote}:{norm_claim}”) so retries and Cypher MERGE statements are 100% idempotent.
When an article is edited (kb/article.updated), we apply the same temporal supersession pattern we introduced for passages in Lesson 3: stale claims receive a valid_to timestamp in both Postgres and Memgraph, and updated claims on the same topic link to the old ones via (:Claim)-[:SUPERSEDES]->(:Claim).
See it on GitHub: kb/graph/cypher.py
Three-way Reciprocal Rank Fusion and kb_sources
How do you combine a BM25 keyword score, a pgvector cosine distance, and a 2-hop Cypher graph traversal that has no similarity score at all?
Because we chose Reciprocal Rank Fusion (RRF) back in Lesson 2, the upgrade is effortless. RRF ignores raw scores and fuses candidates purely by their ordinal rank in each retrieval engine:
When a query arrives at /search or /ask, search_graph_claims extracts canonical topic slugs from the question, traverses active claims in Memgraph (WHERE c.valid_to IS NULL), maps their verbatim quote_span back to the parent passage in Postgres, and hands the ranked list to compute_rrf:
sparse_results, dense_results, graph_results = await asyncio.gather(
search_passages_bm25(session, query, limit=candidate_pool),
search_passages_dense(session, query_embedding, limit=candidate_pool),
search_graph_claims(session, query, limit=candidate_pool),
)
fused = compute_rrf(sparse_results, dense_results, graph_results, k=settings.rrf_k)See it on GitHub: kb/retrieval/hybrid.py
A passage backed by all three engines gets tagged with source: “all” and rises straight to the top of the context snapshot, carrying its structured claim_id, claim_text, and claim_type into the synthesis prompt so Claude can explicitly compare what different authors claim.
Finally, we add a ninth tool to our MCP server (kb/mcp/tools.py) and HTTP API (GET /sources/{id}): kb_sources. Pass it any claim_id or passage_id, and it returns the full provenance chain—the article metadata, the exact quote_span, the canonical topics in Memgraph, and any superseded claim history.
Running it end to end
Boot the upgraded stack (which now includes Memgraph on Bolt port 7687 and Memgraph Lab on port 3000) and run the migrations:
make startIn a second terminal, verify all four subsystems—including Memgraph—are healthy and run the Week 4 test suite:
curl -s http://localhost:8000/health | jq .
uv run pytestRather than walking through twenty pages of terminal output here, watch the full workflow in action—or follow the step-by-step commands in docs/week-4.md:
The Substack Brain, lesson by lesson
Lesson 1: A durable ingestion pipeline (article · video)
Lesson 2: Flow control and hybrid retrieval (article · video)
Lesson 3: A freshness poller and an MCP server (article)
→ Lesson 4: A knowledge graph in Memgraph (you are here)
Lesson 5: Evals built from real failures
Lesson 6: A research agent in production
Our sponsor
The Substack Brain is made possible by Inngest, the durable execution engine behind this course. Every step’s result is recorded, so a crashed run resumes instead of restarting, and the flow control we used this week (concurrency keys, throttling and priority) is configured right on the function, with no queues or workers of your own to run.
If you’re building AI pipelines or agents that need to survive the real world, give Inngest a try. Thanks to the Inngest team for supporting this course 💙
See you next week!
A verbatim span can still be someone else's words. Newsletter authors quote people they're arguing against all the time, so the claim lands on the wrong Author node with a perfect provenance trail. Flagging spans that sit inside quotation marks or a blockquote for a second check would catch most of those.