Durable Research Agents in Production

Bounded loops, self-evaluating reports and a deploy to Inngest Cloud

Hey friends! 👋

Welcome to the final lesson of The Substack Brain, our free six-week course where we build a living knowledge base from the AI engineering newsletters you actually want to read, one production problem at a time. Every piece of it runs on Inngest, the durable execution engine that keeps our pipelines, evals and, from today, our agent from losing work when something fails.

🙋 If you're just joining us, every lesson builds on the ones before it. You can check every lesson and office hours here!

---

---

What breaks this week

Ask the Substack Brain a focused question, like "how does conditional polling work?", and it does great: one search, a cited answer, a few seconds.

Now ask it this:

"Compare how The Neural Maze, Decoding AI and Ahead of AI approach durable agent orchestration versus test-time compute scaling, including real latency and cost numbers, and where the authors disagree."

That's really five questions in one: three newsletters, two topics, hard numbers and disagreements. A single search can only bring back the handful of passages that match best overall, and they’ll mostly be about whichever part of the question has the most matching words. You might get great passages on durable orchestration from The Neural Maze and nothing at all on test-time compute from Ahead of AI. The answer still sounds confident. It's just missing half the question.

A human researcher would work differently. They'd break the question into smaller searches, read what comes back, notice what’s missing, search again for the gaps, and only then write. That's what a research agent does.

The easy way to build one is a loop: plan, search, check, repeat until it's done. And that loop is exactly where things break in production, in three ways.

It doesn't know when to stop. If the newsletters never covered the cost of test-time compute, the agent keeps trying anyway: rephrase, search, nothing, rephrase again. Every round costs tokens. Eventually it hits a rate limit or a timeout and crashes, and every round it already paid for is lost with it.

It drowns in its own results. Each round adds its results to the prompt. The best passage about durable execution shows up in round one, again in round two and again in round three, now cited as [1], [6] and [11]. The prompt fills up with the same evidence three times, there's less room for anything new, and the citations stop making sense.

Nobody checks the final report. After all that work, the agent hands over a long report. Does citation [9] exist? Did it quote an old version of an article? Did it turn an author's prediction into a proven fact? Without checks, you'd never know. Every failure mode from Lesson 5 can now hide inside a 1,500-word document.

This lesson fixes all three, using only the building blocks from the last five weeks.

The system for Week 6

To fix those three problems, and to take everything to production, we add four pieces:

🔁 A bounded research agent, run-research-agent, where every planning, retrieval and reflection step is a durable Inngest step, and the loop has three hard stopping conditions: a maximum number of rounds, a USD budget, and stopping as soon as a round finds nothing new.

🧪 Self-evaluating reports. Before a report is saved, it goes through the same deterministic checks and calibrated judges we built in Lesson 5.

📰 An automated weekly digest, generate-weekly-digest, that turns the week’s articles and verified claims into a cited briefing every Monday.

🚀 A production deployment with a Dockerfile, a production Compose file and a direct connection to Inngest Cloud, plus three new MCP tools for a total of 13.

The Asymmetry Principle from Lesson 2 still decides where each piece runs. POST /ask stays a fast, single-pass read. A research run can take tens of seconds and make several LLM calls, so it's expensive work we'd hate to lose, which is exactly what belongs in Inngest.

---

Why agent loops belong inside durable steps

When teams build a multi-hop research agent, the usual reflex is to pull in a big agent framework and run a while loop in memory, inside a request handler or a background thread.

Now imagine that loop hits an "overloaded" error from the LLM provider on the third round. The whole run crashes. On retry, it re-runs the planner, the first round of searches, the first reflection and the second round, paying for all of them again. And because LLM outputs aren't deterministic, the retry can take a completely different path on the first round, which makes the failure almost impossible to debug.

run-research-agent avoids this by turning every phase into its own Inngest step, which records its result the moment it finishes. Here's the heart of the loop:

for iteration in range(1, max_iterations + 1):
    if total_cost_usd >= max_usd:
        stop_reason = "budget_exceeded"
        break

    if not current_queries:
        stop_reason = "no_followup_queries"
        break

    iterations_used = iteration
    hop_queries = [
        q for q in current_queries[: settings.research_max_subqueries_per_hop]
        if q not in executed_queries
    ]
    if not hop_queries:
        stop_reason = "queries_exhausted"
        break

    hop_retrieval = await ctx.step.run(
        f"hop-{iteration}-retrieve",
        _make_retrieve_step(list(hop_queries)),
    )
    raw_candidates = cast("list[dict[str, Any]]", hop_retrieval["candidates"])
    executed_queries.extend(hop_queries)

    evidence_pool, newly_added = merge_hop_candidates(evidence_pool, raw_candidates)

    # Convergence & budget checks before paying for a reflection LLM step
    if newly_added == 0 and iteration > 1:
        hops_trace.append(
            {
                "hop": iteration,
                "queries": hop_queries,
                "newly_added_passages": 0,
                "cumulative_passages": len(evidence_pool),
                "decision": "stop_converged_zero_new_passages",
                "rationale": "Hop produced zero new unique passages; halting loop.",
                "step_cost_usd": 0.0,
            }
        )
        stop_reason = "converged"

Three things happen in those lines. Each round's searches run inside a step named after the round (hop-1-retrieve, hop-2-retrieve, and so on), so if a later step fails, Inngest retries only that step and replays everything before it from its saved results, at no token cost. The loop checks the budget before starting a round, and the accumulated cost comes from the est_usd that every LLM step already reports, the same usage data CostCaptureMiddleware has been recording since Lesson 1. And if a round brings back nothing new, the loop stops right there, before paying for another reflection.

The function also has its own concurrency limit, so no more than two research runs happen at the same time, however many are requested:

@client.create_function(
    fn_id="run-research-agent",
    name="Run Bounded Research Agent",
    trigger=inngest.TriggerEvent(event=RESEARCH_REQUESTED),
    concurrency=[
        inngest.Concurrency(limit=settings.concurrency_research_runs),
    ],
    retries=2,
)

---

Three roles, three stopping conditions

Instead of one big prompt, the agent is split into three roles, each with one job, built on the same hand-written Agent[T] layer from Lesson 4.

The query planner splits the user's question into one to three targeted searches and up to five canonical topics. Following the rule from Lesson 5, its output schema puts the reasoning before the searches:

class ResearchPlan(BaseModel):
    """Structured output of `query_planner_agent` (rationale FIRST, subqueries SECOND)."""

    rationale: str = Field(
        description=(
            "Concise explanation of how the question is decomposed across lexical, "
            "semantic, and cross-publication topic dimensions."
        )
    )
    subqueries: list[str] = Field(
        min_length=1,
        max_length=3,
        description="1 to 3 targeted search sub-queries for the initial retrieval hop.",
    )
    ...

The coverage critic looks at everything gathered so far after each round, decides whether it's enough, lists what's missing, and suggests up to two new searches. Python then throws away any search that has already been run, so the agent can't keep asking the same thing.

The research synthesizer writes the final report from the combined evidence, with inline citations, keeping each claim's type (fact, mechanism, opinion, prediction or recommendation) and pointing out where authors disagree.

Deduplicating across rounds

Across several searches and rounds, the same good passage often comes back more than once, for example from full-text search in the first round and from the graph in the second. So between rounds, merge_hop_candidates merges everything into a single pool, one entry per passage. It keeps the highest score each passage got, fills in any claim details that were missing, and counts how many passages are genuinely new:

def merge_hop_candidates(
    existing_pool: list[dict[str, Any]],
    new_candidates: list[dict[str, Any]],
) -> tuple[list[dict[str, Any]], int]:
    """Deduplicate retrieved passages across hops by `passage_id`, preserving the
    highest `rrf_score` and enriching any missing `claim_id`/`claim_text`/`claim_type`.

    Returns `(sorted_merged_pool, newly_added_count)`.
    """
    by_id: dict[str, dict[str, Any]] = {
        str(item["passage_id"]): dict(item)
        for item in existing_pool
        if item.get("passage_id")
    }
    newly_added = 0

    for cand in new_candidates:
        pid = str(cand.get("passage_id") or "")
        if not pid:
            continue
        score = float(cand.get("rrf_score", 0.0) or 0.0)
        if pid not in by_id:
            copied = dict(cand)
            copied["rrf_score"] = score
            by_id[pid] = copied
            newly_added += 1
        else:
            current = by_id[pid]
            current_score = float(current.get("rrf_score", 0.0) or 0.0)
            if score > current_score:
                current["rrf_score"] = score
            if not current.get("claim_id") and cand.get("claim_id"):
                current["claim_id"] = cand["claim_id"]
                current["claim_text"] = cand.get("claim_text")
                current["claim_type"] = cand.get("claim_type")

    merged = sorted(
        by_id.values(),
        key=lambda item: (-float(item.get("rrf_score", 0.0) or 0.0), str(item["passage_id"])),
    )
    return merged, newly_added

That count is what powers the third stopping condition. Together, the loop stops when it:

• reaches the maximum number of rounds (3 by default, and never more than 5),

• reaches the USD budget ($0.25 by default), in which case it writes the report from the evidence it already has, or

• finds nothing new: if any round after the first adds zero new passages, the loop stops immediately, without paying for a reflection.

It also stops early when the critic decides the evidence is already enough, or has no new searches left to suggest.

One honest detail about the budget: it caps the research loop. Once the loop stops, the report is still written and evaluated, so the final bill can go slightly over the limit. If you need a strict ceiling on the whole run, set the budget a little below it.

---

Reports that check themselves

When the loop is done, run-research-agent doesn't simply trust the report. In the evaluate-report step, it runs the same two-level evaluation we built in Lesson 5.

First come the deterministic checks, which take less than a millisecond and cost nothing: every inline citation has to exist and point to a passage in the evidence pool (FM-1), no cited source can be an old version of an article (FM-2), and the report has to either cite its sources or clearly say it doesn't know (FM-3). Then the calibrated judges check that no claim goes beyond its source or turns an opinion into a measured fact (FM-4), and, when the evidence comes from at least two authors or publications, that disagreements are actually reported (FM-5).

Finally, persist-research-run saves the full trail in the new research_runs table: the initial plan, every round with the critic’s reasoning and cost, the report, its citations, the evaluation results and an overall confidence (high, medium or low). Then it sends a kb/research.completed event.

---

A weekly digest that writes itself

A living knowledge base shouldn't only speak when spoken to. generate-weekly-digest turns the week's articles and verified claims into a weekly briefing. It runs on an Inngest cron trigger every Monday at 08:00 UTC, or on demand through POST /digests/run:

@client.create_function(
    fn_id="generate-weekly-digest",
    name="Generate Weekly Knowledge Base Digest",
    trigger=[
        inngest.TriggerCron(cron=settings.digest_cron),
        inngest.TriggerEvent(event=DIGEST_REQUESTED),
    ],
    retries=2,
)
async def generate_weekly_digest(ctx: inngest.Context) -> dict[str, object]:

It has four steps:

1. collect-weekly-window gathers the articles and current claims (valid_to IS NULL) from the last seven days, grouped by topic, and flags topics that at least two publications wrote about.

2. synthesize-weekly-digest writes a cited briefing with three sections: key technical claims and measurements, threads and disagreements across publications, and a watchlist of predictions.

3. verify-and-persist-digest checks the citations with the same deterministic checks and saves the digest with an ID derived from the week’s dates, so running it twice on the same day updates the same row instead of creating a duplicate.

4. emit-digest-published sends a kb/digest.published event with the number of articles, claims and cross-publication topics.

---

Thirteen MCP tools

This week's three new tools follow the Asymmetry Principle from Lesson 3, which is why research is split across two of them. A three-round investigation can run up to nine searches and six LLM calls and take tens of seconds, which is long enough to time out a tool call in Claude Code or Cursor. So kb_research only starts the run and returns a research_id straight away, and kb_research_result fetches the finished report from Postgres whenever the agent asks for it. The third tool, kb_weekly_digest, reads the latest digest.

That brings the server to 13 tools: the eight from Lesson 3, kb_sources from Lesson 4, kb_eval_report from Lesson 5, and these three.

---

Taking it to production

Until now, make start ran Postgres, Memgraph and the Inngest Dev Server in Docker, with the FastAPI app on your machine for quick iteration. This week ships what you need to run it for real.

The Dockerfile builds the app on python:3.12-slim with a pinned version of uv, installs the locked dependencies without dev tools, and runs the database migrations before starting the server.

The production Compose file, infra/docker-compose.prod.yml, runs Postgres, Memgraph and the app with automatic restarts and health checks. It switches the app to production mode and refuses to start without your Inngest keys:

DATABASE_URL: postgresql+asyncpg://${POSTGRES_USER:-substack_brain}:${POSTGRES_PASSWORD}@postgres:5432/${POSTGRES_DB:-substack_brain}
MEMGRAPH_URI: bolt://memgraph:7687
INNGEST_DEV: "false"
INNGEST_BASE_URL: ${INNGEST_BASE_URL:-https://api.inngest.com}
INNGEST_EVENT_KEY: ${INNGEST_EVENT_KEY:?Set INNGEST_EVENT_KEY for production Inngest}
INNGEST_SIGNING_KEY: ${INNGEST_SIGNING_KEY:?Set INNGEST_SIGNING_KEY for production Inngest}
OPENAI_API_KEY: ${OPENAI_API_KEY:?Set OPENAI_API_KEY in .env}
ANTHROPIC_API_KEY: ${ANTHROPIC_API_KEY:-}
LAB_PORT: "8000"

With INNGEST_DEV set to false, the Inngest client runs in production mode, which means it verifies that every request really comes from Inngest.

The part we like most is how the app talks to Inngest Cloud. Normally, Inngest Cloud has to reach your app over the internet, which on a laptop or a private network means setting up a tunnel like ngrok or Cloudflare Tunnel. Instead, when the app starts in production mode, it opens an outbound WebSocket connection to Inngest Cloud using Inngest Connect:

@contextlib.asynccontextmanager
async def _lifespan(_app: FastAPI) -> AsyncIterator[None]:
    """When running against Inngest Cloud (`INNGEST_DEV=0`), establish an outbound
    WebSocket worker connection (`inngest.connect`) so Inngest Cloud can sync and
    invoke all 9 functions without needing ngrok, cloudflared, or public ingress.
    """
    conn: Any = None
    connect_task: asyncio.Task[None] | None = None
    if not settings.inngest_dev and settings.inngest_signing_key:
        from inngest.connect import ConnectionState, connect

        logger = logging.getLogger("uvicorn.error")
        conn = connect([(client, list(FUNCTIONS))], shutdown_signals=[])
        connect_task = asyncio.create_task(conn.start())

        async def _log_when_active() -> None:
            await conn.wait_for_state(ConnectionState.ACTIVE)
            logger.info(
                "Inngest Cloud Connect active (app=%s, functions=%d)",
                client.app_id,
                len(FUNCTIONS),
            )

        asyncio.create_task(_log_when_active())
    try:
        yield
    finally:
        if conn is not None:
            await conn.close(wait=True)
        if connect_task is not None and not connect_task.done():
            connect_task.cancel()7. Quick operational walkthrough

Inngest Cloud then sends work down that connection. No tunnel, no public URL to keep in sync: your app and all nine functions simply show up in the Inngest Cloud dashboard.

To deploy, you only need three things:

1. Create a free account on Inngest Cloud.

2. Copy your INNGEST_EVENT_KEY and INNGEST_SIGNING_KEY from the Keys section, at the bottom left of the dashboard.

3. Put them in your .env, starting from .env.example as in previous weeks, and set INNGEST_DEV=0.

---

Running it end to end

Start the stack. It applies this week’s migration, which creates the research_runs and digests tables:

make start

Because INNGEST_DEV=0, make start skips the local Inngest Dev Server, starts Postgres, Adminer, Memgraph and Memgraph Lab, and launches FastAPI. As soon as FastAPI starts, it connects to Inngest Cloud, and the substack-brain app appears in the dashboard with all nine functions.

Run a research investigation

curl -X POST http://localhost:8000/research -H "Content-Type: application/json" \
  -d '{"question": "How do durable execution engines, hybrid RRF retrieval, and knowledge graphs work together in production AI systems?", "max_iterations": 3, "max_usd": 0.25}'

curl http://localhost:8000/research/<research_id>

You might wonder why we still call localhost if the functions run on Inngest Cloud.

It's because only the orchestration lives in the cloud. POST /research runs on your machine: it sends a kb/research.requested event to Inngest Cloud and returns straight away with a research_id.

Inngest Cloud then runs run-research-agent through the WebSocket connection, step by step, from record-research-start and plan-subqueries through each round to synthesize-report, evaluate-report and persist-research-run. You can follow it live in the Runs tab. Finally, GET /research/<research_id> reads the finished report, its rounds and its citations from your own Postgres.

Generate the weekly digest

curl -X POST http://localhost:8000/digests/run -H "Content-Type: application/json" -d '{"lookback_days": 7}'
curl http://localhost:8000/digests/latest

It works the same way: POST /digests/run sends kb/digest.requested to Inngest Cloud, generate-weekly-digest runs its four steps and writes the digest to your Postgres, and GET /digests/latest returns it.

Run every check from the course

make check

That runs the linters, type checks and tests for all six weeks.

---

That's a wrap

Six weeks ago, we started with a simple system that could ingest five articles from one newsletter. Today we have a system that ingests three newsletters without hammering anyone's servers, keeps every version of every article, builds a knowledge graph of who claimed what, answers with verified citations, scores its own answers against failure modes we found by hand, runs multi-step research on a budget, writes its own weekly briefing, and runs in production on Inngest Cloud.

None of it needed an agent framework. Every piece was a small, testable building block on top of durable steps, and that's the real lesson of harness engineering: the reliability of an AI system comes from the harness around the model, not from the model alone.

Thank you for following along, for every comment and question, and for your patience when a lesson came out a day late. All the code is open source in the course repository, so fork it, break it and build something of your own with it.

---

Our Sponsor

None of this would have been possible without Inngest.

A huge thank you to the Inngest team for backing this project. Supporting a free, fully open-source course, where every line of code is public and every lesson is available to everyone, is a real bet on the developer community, and I’m genuinely grateful for it.

Thanks for making this happen! 🙌