Context Engineering: History, Tool Output, and What to Measure

Part one argued that a context window is a budget rather than a container, spent against three limits at once — space, money and latency, and attention — and that the last one is the reason adding merely-related material makes answers worse instead of leaving them unchanged. It then spent most of its length on retrieval, because retrieval is the part of the context you deliberately choose: you decide the chunking, the k, the reranker, the floor beneath which you return nothing at all.

This part is about the two sources you don’t choose. Conversation history grows on its own, every turn, whether or not anyone is watching it. Tool output arrives at whatever size the tool felt like producing and then stays in the transcript, being re-sent and re-read for the rest of the session. In most production apps these two are the majority of the context by volume, and in almost all of them they are the part nobody has tuned.

History is a liability that grows on its own

The full transcript is the naive approach and it is wrong past about ten turns: cost grows quadratically over a session — each turn re-sends everything before it — and turn 3’s clarification about a date format is still occupying attention at turn 30. But the opposite, a sliding window of the last N turns, is also wrong, because the constraint the user gave you in turn 1 (“I’m on Postgres 14, not 16”) falls off the end and the model starts confidently giving advice for the wrong version.

Both failures are the same mistake in different directions: treating recency as a proxy for relevance. What actually works is a hybrid with three parts.

• A pinned, structured facts block. Not prose. A small set of extracted key-values — user, environment, stated constraints, decisions already made — that you update as the conversation reveals them and re-send verbatim every turn. This is cheap and it is what stops the version-drift failure.
• A rolling summary of the middle. When the transcript passes a threshold, summarise the oldest turns into a paragraph and drop them. Summarise forward from the previous summary rather than re-summarising from scratch, or you pay for the whole history again every time you compact.
• The last few turns verbatim. Recency matters for coherence, and pronouns (“do that again, but for staging”) only resolve against the literal recent text.

Compare the three columns on the one measure that matters, which is not token count: does the fact from turn 1 survive to turn 30? The full transcript keeps it and pays for twenty-nine turns of everything else to do so. The sliding window lost it long ago and has no way of knowing. The hybrid keeps it for a few dozen tokens a turn, forever, because somebody decided once that it was a constraint rather than a message. That decision — which parts of a conversation are facts and which are just text — is the whole of history management, and no amount of context window makes it go away.

Populating the facts block without another model call per turn

The obvious implementation — ask a model to re-extract the facts on every turn — reinvents the cost you were trying to remove. Two cheaper habits cover most of it.

First, capture what you already know structurally. Tenant, plan, locale, the environment selected in the UI, the document currently open: your app knows these without asking an LLM, and they are exactly the facts that stop the worst class of wrong answer.

Second, extract only on turns that could plausibly have changed something — user messages, not your own assistant’s — and ask the model for a patch rather than a rewritten block: keys to add, keys to replace. Patching keeps the block stable, which matters for caching, and it keeps it small, because a rewrite slowly turns a key-value list back into prose. Cap the block, and evict from it by supersession rather than by age.

One thing belongs in this block verbatim and must never be summarised: exact strings. Error codes, identifiers, version numbers, the snippet the user pasted. Summarisation is lossy by design, and the tokens it is most likely to smooth away are the ones a correct answer depends on character-for-character.

Compaction and the cached prefix

Trigger compaction on a budget share rather than a turn count — say, when history passes 40% of your target context — because turns vary wildly in size and “every 10 turns” fires at the wrong moment in both directions. Then run it out of band, after a response has finished streaming, so the user never waits on a summarisation call that could have happened while they were reading.

The subtler constraint is prompt caching. A cache hit requires an exact prefix match, so rewriting the middle of the prompt invalidates every cached block from the edit point onward; the next turn pays full-price prefill for everything after it. Two rules fall out. Order your context stable-to-volatile — system prompt, tool schemas, pinned facts, rolling summary, then recent turns and retrieved passages at the end. And compact decisively and rarely rather than trimming one turn every turn, which keeps the cache permanently cold in exchange for saving a few hundred tokens.

Tool output is context you didn’t write

The last source is the one that surprises teams, because it isn’t in the prompt template at all: whatever your tools return goes straight into the model’s context, at whatever size the tool felt like producing.

A SELECT * that returns 200 rows of JSON is perhaps 15,000 tokens the model has to read to answer “how many are overdue?” — a question whose correct answer is two characters long. An HTTP tool that returns a raw response body inlines an entire HTML page, navigation and all. And the damage is not confined to the turn it happened on: once a bad tool result is in the transcript it stays there, re-sent and re-charged on every subsequent turn, so a single careless call at turn 4 can be the largest line in your budget at turn 20.

Treat the tool boundary as an interface you design, not a pipe you connect.

• Return the answer, not the data. If the model asked a counting question, return the count. Aggregate, filter and project on your side, where it costs microseconds instead of tokens.
• Cap and say you capped. "showing 10 of 847 matches" is far better than 847 rows, and immeasurably better than 10 rows that pretend to be all of them — the model needs to know it’s looking at a sample or it will reason as though it isn’t.
• Errors should be actionable and short. "no user with id 42; ids are UUIDs, not integers" gets a corrected retry. A 40-line stack trace gets an apology and a hallucinated user.
• Describe the tool in terms of decisions. The description is context too, spent on every request. It should say when to reach for the tool and when not to, which is what the model is actually deciding.

The figure’s second number is the one to take away. A wasteful result is not a spike, it is a slope: the tokens it added are charged again on every later turn of the session, so the right comparison is never “15,000 tokens versus 40” but the two lines those numbers draw over the next twenty turns. This is also the reason a retry loop with fat error output is so expensive — three failed calls with 40-line stack traces don’t just waste three calls, they permanently install 120 lines of noise in the transcript that every subsequent turn pays to re-read.

That last point has a corollary teams discover late: tool descriptions are a fixed cost paid on every single request, and they scale with how many tools you expose. Thirty tools at 150 tokens each is 4,500 tokens before the user has typed anything — permanently, on every turn, in the region of the context that competes with your system prompt. If a surface genuinely needs thirty tools, retrieve the tool schemas the way you retrieve documents, and send the handful the current request could plausibly need.

What to do about it

Four practices, in the order they pay off.

Measure the composition, not just the total. Log, per request, how many tokens went to each section: system, tool schemas, tool results, retrieval, history, the question itself — plus the turn index, so you can see the curve rather than a point. You cannot budget what you don’t itemise, and the answer is very often “62% history” for an app whose team spent a quarter tuning retrieval. Watch p95 rather than the median; the expensive conversations are not the typical ones.

Build the eval set before you tune. Thirty real queries with known-correct answers, harvested from actual traffic rather than invented at a desk, and including the ones whose correct answer is “I don’t know” — otherwise every change that makes the model more willing to guess will look like an improvement. Every knob here — chunk size, k, reranking, compaction threshold — is a trade, and without a fixed set you are choosing by vibes on whichever example you last looked at.

Put the important things at the edges. Instructions early, the retrieved passage the question actually turns on late, filler nowhere. This is free, and it follows directly from how attention degrades in the middle of a long context.

Cache the stable prefix. Prompt caching pays for exactly the structure these two articles argue for: a fixed system block and fixed tool schemas at the front, pinned facts and summary next, volatile retrieval and recent turns at the back. Order your context stably and a large share of your input cost stops being charged at full price.

The short version

The prompt is a small part of what the model reads. The rest — retrieval results, conversation history, tool output, tool descriptions — is assembled by your code on every request, and that assembly is where reliability is won or lost.

Budget the window against space, cost and attention, and remember the third one is real: adding merely-related material makes answers worse, not neutral. Retrieve wide and rerank narrow, run hybrid search so identifiers don’t vanish, and return nothing when nothing clears the bar. Compact history into pinned facts plus a rolling summary plus recent turns verbatim, because both the full transcript and the sliding window fail, in opposite directions. Design what your tools return as carefully as what your prompt says, and remember that what they returned is still being paid for twenty turns later.

Then itemise every request by section and look at the numbers, because the part of the context eating your budget is almost never the part you have been tuning.