At work, I built an AI assistant that uses tools: it searches records, looks up details and drafts letters. Users liked it and kept one conversation going for a long time.

Then a long chat stopped working. Every message failed with the prompt is too long, even short questions and retries.

The chat was not broken, it was too big for the model to read. Here is how I solved it.

The idea in one picture: old messages become a short summary, and recent messages stay as they are.

1.Why a chat becomes full

An AI model remembers nothing between messages. On every message, your app sends the complete conversation again: instructions, old messages, tool results and the new message.

A model can read only a limited amount at once, called the context window, measured in tokens (small pieces of words).

So the conversation grows every turn, fastest with tool results, since one document can be thousands of tokens. For example:

It never shrinks by itself. Turn 22 sends the same history plus one message, so the user is stuck.

2.The idea: minutes of a meeting

When a colleague joins a meeting late, nobody replays the recording. He gets the minutes and follows the current discussion.

Compaction works the same way:

The model reads much less but keeps what matters. This runs before every message:

Each step has one rule, explained below.

Rule 1:Use the real size, do not guess

I first tried counting tokens myself. It is harder than it looks: each model family splits text differently, languages other than English count very differently, and some inputs, like tool descriptions, are not in your message list at all.

The model already reports the exact number in every response: tokens in and tokens out. Save it after each turn and check it before the next:

after each turn:
    save  last_size = tokens_in + tokens_out     # reported by the model

before the next turn:
    if last_size > 80% of the context window:
        compact the conversation

Two points:

•  

  Count input and output. The answer becomes the next turn’s input, so counting only input gives a number that is too low.

•  

  Why 80%? The number is from the previous turn. The remaining 20% leaves space for the new message and any large tool results.

Rule 2:Never separate a tool call from its result

A tool use is stored as a pair, the call and its result:

User:       "Please create a draft letter."
Assistant:  [calls tool: create_draft]
Tool:       [result: letter #48213, created]
Assistant:  "Letter #48213 has been created."

Models expect each result right after its call. If your cut keeps the result but drops the call, the model rejects the conversation, and since that state is saved, every later message fails too.

The fix: start the recent part with a user message. A user message never sits between a call and its result, so cutting before one is always safe.

cut = number of messages minus recent messages to keep

while the message at the cut is not a user message:
    move the cut one step earlier          # earlier, never later

old part    = everything before the cut
recent part = everything from the cut on

Also keep out of the old part: system instructions (rebuilt every turn) and old summaries (see Rule 3).

Rule 3:Write a useful summary

Three things helped most.

Use a separate, plain model call

Do not ask the main assistant to summarise inside the chat: it may call a tool instead, and its reply becomes an odd extra message. Use a separate call with no tools and temperature 0 for stable output.

Ask for notes with fixed sections, not a paragraph

A paragraph summary slowly loses details. Ask for fixed sections:

Update the old notes instead of summarising them again

On later compactions, send the previous summary plus the newly old messages and ask for one updated set of notes. Summarising summaries loses detail each time, like a photocopy of a photocopy.

Rule 4:Check the summary before you delete anything

Deletion cannot be undone, so I run two checks first:

if summary is empty:                    reject
if summary is not at least 50% smaller: reject

If a check fails, the chat stays as it is for one more turn. Nothing is lost.

Rule 5:Swap correctly (the LangGraph part)

This part needs real code. In LangGraph, the saved message list only grows: anything you write is appended. Saving a shorter list changes nothing, and saving only the summary makes the chat bigger.

To delete, you send LangGraph a special RemoveMessage for each old message, together with the new summary, in one update:

from langchain_core.messages import RemoveMessage, SystemMessage

removals = [RemoveMessage(id=m.id) for m in old_part + old_summaries]
summary  = SystemMessage(content=summary_text,
                         additional_kwargs={"is_summary": True})

await graph.aupdate_state(config, {"messages": removals + [summary]})

Two points:

•  

  Tag the summary (the is_summary flag above) so you can find it later. Then remove every old summary each time, not just the last one. If a past run failed halfway, the next one cleans it up.

•  

  Do not write “the messages below” in the summary. LangGraph appends it at the end, so avoid referring to its position.

Rule 6:Keep a backup plan

Rule 1 has a weakness: when a turn is too big, the model returns an error with no size number, so the check never runs. One huge tool result can also cross the limit in a single turn.

So catch the “too long” error and recover once:

try:
    answer the user
except "prompt is too long" error:
    compact hard        # keep only the latest user message and everything after it
    try once more       # continue the same turn; don't add the message twice

•  

  Compact hard, but follow Rule 2. Keep only the latest user message and everything after it. Never keep just the newest message: in a failing turn it is often a tool result, and without its call the chat breaks permanently.

•  

  Detect the error by its text. Providers word it differently (“maximum context length”, “prompt is too long”), so match a few specific phrases. A broad word like “context” can match unrelated errors.

•  

  Retry only once. If hard compaction fails, the cause is something it cannot shrink, like very long instructions. Tell the user clearly instead of letting them retry forever.

The most important point: never break the user’s chat

Compaction is a background task, and the user came for an answer. If anything fails (summary timeout, slow database, a bug in the cut), log it and continue with the chat unchanged:

try:
    compact if needed
except any error:
    log it          # the chat continues uncompacted; next turn tries again
answer the user

Related habits:

•  

  Check first, delete last.

•  

  Put compaction behind an on and off switch to enable it gradually and disable it quickly.

•  

  Log every attempt with its result and the size before and after. Many rejections mean weak summaries, not broken compaction.

•  

  Keep the full original messages in your database. Compaction only changes what the model reads.

Checklist

Final thoughts