Every long conversation with an AI agent eventually hits the same wall. Context windows keep growing, but agent sessions grow faster, and sooner or later the model runs out of room.
What happens at that point is called compaction.
And honestly, it's one of those topics that rarely gets attention, yet it quietly decides whether an agent keeps doing good work or forgets the one detail that mattered.
That's why I'm really happy to introduce today's guest author, Jason Rametta! Jason and I are working together at Zapier, building durable agents, and I've learned a lot just watching how he approaches problems like this one. He's not satisfied with something that only works in a demo. He wants to understand why it works, when it breaks and what it costs. That comes from years of building close to the metal: he wrote The TypeScript Compiler API book, and before working on AI agents he built SDKs for a Rust game engine!
When he published his post on model compaction, I asked him to write an extended version for The Neural Maze. In it he explains when agent harnesses decide to compact, which strategies they use, and how to make all that forgetting a bit less lossy.
Over to you, Jason! 👇
If you're working with ai or building an ai harness then you're probably aware of the context limits that models have. You can't chat with a model infinitely and have it remember everything perfectly. but all popular chat apps seemingly do let you chat infinitely, so how do we get around the technological limits of model context windows?
What is compaction?
Let's say you're building a harness around one specific model: claude fable 5.2.
Anthropic says that fable52 has a context window of 1 million tokens, which means you can provide up to 1M tokens in every request before anthropic starts rejecting your request with 'context length exceeded' errors. But when you're running a lot of turns with an agent, executing a lot of tool calls with large outputs and consuming large files, you can quickly reach that 1M limit.
So how do harnesses deal with this issue?
Compaction is the process of reducing the size of your session so that you fit within the context window again and continue working.
Third party harnesses like OpenCode, Pi, Claude Code, etc, will have their own implementation of compaction built-in so you don't have to worry about it. Each harness implementation of compaction differs slightly and will impact your results in large sessions.
When does compaction happen?
It's up to the harness to decide when to compact, so if you're building your own harness then it's up to you, but a lot of harnesses choose to compact when the total tokens in the sessions are within some threshold of the limit.
So for example, you can trigger a compaction when your tokens reach 75% of your context window for your model, which means when you reach 750k tokens out of 1m. Usually a buffer is provided because output is counted against the window limit.
This check we do for tokens can be done at the beginning of a new turn, mid-turn if you happen to surpass the threshold because of large tool call outputs, end of turn, manually by the user, or even after being idle for some time.
Claude code, for example, can compact a large session after about an hour idle, right before the 1-hour prompt cache would expire, so your next message doesn't have to reprocess the whole history uncached.
Compaction strategies
This is the most interesting part to me.
How do harnesses compact? What exactly is happening during this process?
Well, it varies by harness, but the vast majority of harnesses will compact the session by doing another LLM call to summarize the session. During summarization, the model takes a customized prompt that explains that it needs to compact and it takes the history of the session and it tries to output a much shorter version of what happened and tries to keep the important bits.
There are a lot of things you can do here before calling the summarization model call. Before we get into them though, let's recap what's part of a typical session.
In a normal session you typically have system prompts, assistant messages, user messages and tool calls. Tool calls can contain things like skills that are loaded, or they can just contain unnecessary things that are no longer needed.
Before calling the LLM to summarize you can choose to delete all tool calls for example, and that will result in a cheaper and more accurate summarization step. It depends on what your harness is built for though, if you don't want your model repeating expensive tool calls again then you can persist those post compaction so future requests still see them.
Summarization in the form of an LLM call at a trigger point is a popular strategy for compaction, but it's not the only strategy.
Some systems decide to keep a running notebook of important items so when it's time to compact, instead of doing a summary call to the LLM, it just persists the notes from the notebook and continues on (credit to Robin Salimans, Zapier). This could work well for your use case but of course it depends on the results of your own evals.
Below is an example of what compaction can look like on a real session:
🙋 Go to Jason's original article to play with his interactive application!
Recall and history
Compaction is inherently lossy, meaning you'll lose some information during the summarization step.
Losing some non-critical information may be fine for your use case, but usually we want to provide future model calls the ability to make tool call requests to recall data from before the compaction took place. This can be optimized by giving future calls some sort of table of contents to the pre-compacted conversation with references to jump to specific topics.
This recall ability provides the model a way to lazy-load more context on demand instead of needing to keep it all in the context live. If your harness is running in an environment where you can provide a file system or virtual file system, then another good strategy for recall is dumping important tool call results and past conversation directly to files on disk and providing your harness the ability to search through and parse these files for context.
Conclusion
Context compaction is a really fascinating topic. For further reading i recommend checking out these papers that dive deeper into it:




Plain-English version for anyone who hasn't hit this yet: when a session gets too long, most agent tools compact it with another model call that summarizes it, and that summary is where details get lost. The fix we'd copy first is the recall one. Dump the important tool results to files and keep a table of contents to the old conversation, so a detail that missed the summary can still be looked up. Has anyone compared the running-notebook approach against the summary call on their own evals?