Context Engineering: The Window Is a Budget, Not a Container

Your demo works because the prompt is short, the context is clean, and the conversation is short. Production doesn’t look like that. By the time an AI app starts using retrieval, tool calls, conversation history and system instructions all competing for the same (context) window, the thing that determines whether it’s reliable or not is no longer the prompt, it’s everything else going into model alongside it.

That discipline is context engineering, and it’s a different skill from writing a good prompt. Prompt engineering asks “what should I say?”. Context engineering asks “of everything I could put in front of the model right now, what earns its place?” — which is a budgeting problem, and budgeting problems have engineering answers.

This is part one of two. It covers the budget itself, why attention is a cost line rather than a metaphor, and how to make retrieval earn its share of the window. Part two takes the two sources you control least, conversation history and whatever your tools hand back and finishes with what to measure once both are under control.

The window is a budget, not a container

The first mistake is treating a large context window as somewhere to put things. It is something you spend, and you spend against three separate limits at once.

1. Space. Obvious, and the least interesting of the three. You have a hard token ceiling and everything competes for it. It is least interesting because it’s the limit you notice: you hit it, something raises an error or silently drops the oldest half of the transcript, and you go and fix it. The other two limits never announce themselves. They just make the app expensive and mediocre while every request keeps returning a 200.

   Worth saying plainly, since it’s the usual response to this section: a bigger window is not a fix. Moving from 128k to 1M multiplies the ceiling and changes neither of the limits below, because you are still charged for every token you send, on every turn, and the model still reads the middle of a long context worse than the ends. A larger window buys you room to make the same mistake at greater expense.

2. Money and latency. Input tokens are cheap per unit and ruinous in aggregate, because you pay for the whole context on every turn of a conversation, not once.

   Work it through with numbers that aren’t extreme. A system prompt plus tool schemas that come to 1,800 tokens. Five retrieved passages at roughly 4,200. Around 700 tokens of user-plus-assistant text per exchange, and a couple of tool calls early on whose results are 900 tokens each and stay in the transcript afterwards. Turn one is about 6,000 tokens and looks completely fine.

   Turn ten is those same 6,000, plus 6,300 of accumulated history, plus 5,400 of tool results nobody has looked at since — roughly 18,000 tokens. And the bill for the session isn’t 18,000, it’s the sum over every turn, because each one re-sent its own entire context to get an answer. Ten turns lands near 110,000 input tokens for what the user experienced as ten short questions. Thirty turns is over half a million. Nothing in that arithmetic involved a long document or a badly written prompt. It is just the shape of the default.

   Latency moves with the same number. Prefill — the pass over the prompt before the first output token appears — scales with prompt length, and that is precisely the part a user feels as the app being slow to start. Streaming hides decode time. It does not hide prefill.

3. Attention. The one people miss, and the one that decides whether the other two were worth spending. It gets its own section below.

Drag through the presets and watch the system prompt stay put. That is the shape of the real problem: the fixed costs are fixed, so every token of growth comes out of the same squeezed middle where retrieval and history are fighting, and the first thing to get evicted is usually the thing you most needed.

Attention is the limit nobody budgets

A model does not use all of its context equally. The lost-in-the-middle result — that retrieval accuracy drops sharply when the relevant passage sits in the middle of a long context rather than near either end — has held up across model generations and across context lengths. Plotted against position, accuracy is a U: strong at the start, strong at the end, sagging in between.

The reason this keeps surprising teams is the benchmark they saw instead. “Needle in a haystack” plants one unrelated sentence in a long filler document and asks the model to repeat it; models score near-perfectly, and the screenshot of a solid green grid gets read as “long context is solved”. But that task has a single fact, no competing candidates, and an exact-match lookup as the success criterion. A real request has five plausible passages, three of which say something adjacent to the answer, and needs synthesis rather than recall. A near-perfect needle score tells you the tokens were ingested. It does not tell you they are being reasoned over.

Two things follow, and they are the whole reason context engineering exists as a practice.

The first: adding material that is merely related rather than relevant does not leave accuracy unchanged. It pushes the useful passage toward the middle and adds plausible distractors beside it, so the answer degrades in a particular way — it stops being wrong and starts being vague. Filling the window is not a neutral act.

The second is cheerier: position is a control you already own and probably aren’t using. Instructions and task framing at the front. The passage the answer actually turns on at the back, immediately before the question. Nothing in the middle that you couldn’t defend individually. This costs nothing to implement, needs no new infrastructure, and is the highest return-per-hour change in this article.

Relevant is not the same as related

Naive RAG does one thing: embed the query, take the top k by cosine similarity, paste them in. It works in the demo and degrades in a specific, recognizable way — the answers stop being wrong and start being vague, because the model is averaging over five documents that are all about the right topic and only one of which contains the answer.

Similarity is topical. Relevance is about whether the passage answers this question. Four things close the gap, roughly in order of return on effort.

Chunk on structure, not on character count. A 512-character window that splits a table from its header, or a function from its signature, produces chunks that embed plausibly and inform nobody. Enterprise | 5,000 | 90 days is a perfectly good row and a useless chunk once the header naming those columns landed in the previous one: it will embed, it will retrieve for the right query, and it will answer nothing. Split on headings, on function boundaries, on paragraph breaks — and prepend each chunk its ancestry (“Billing › Refunds › Partial refunds”), which costs about ten tokens and repairs both the embedding and the model’s reading of what it was handed.

Retrieve more, then rerank. Take 30 candidates by embedding similarity — cheap, fast, recall-oriented — and pass them through a cross-encoder reranker that scores each one against the query jointly rather than comparing two independently computed vectors. Then keep the top 3. This is the single highest-leverage change in most pipelines, because embedding recall is usually fine and embedding precision is what’s failing: the answer is in your top 30 far more often than it’s in your top 3.

The price is one extra model call on the critical path — tens of milliseconds for a small local cross-encoder, more for a hosted one — paid to delete 27 passages you were otherwise about to send. That trade is nearly always positive, and it gets more positive as the conversation goes on, because rerank latency is paid once per query while the tokens it removed would have been re-sent on every turn that followed.

Run hybrid search. Dense vectors are hopeless at exact tokens — error codes, SKUs, surnames, ERR_CONN_REFUSED. BM25 is excellent at exactly that and hopeless at paraphrase. Run both and fuse the rankings rather than the scores: reciprocal rank fusion gives each document 1 / (60 + rank) from each list and adds them, which needs no calibration between the two systems. That matters more than it sounds. Cosine similarities and BM25 scores are not on the same scale, and making them comparable is a tuning project with no natural end. Ranks are comparable for free.

Put a floor on the score. If nothing clears the bar, retrieve nothing and let the model say it doesn’t know. Set the floor on the reranker’s score rather than on cosine similarity — cosine values bunch into a narrow band where the difference between excellent and irrelevant is a couple of hundredths, while a cross-encoder score behaves much more like an actual relevance judgement.

And the empty result has to be stated, not merely left blank. That branch should put a line in the context saying no supporting document was found and the answer must say so. Padding to k guarantees that a query with no good answer gets three bad ones, which is how confident hallucinations get manufactured.

The failure mode worth internalizing is the last preset. Filling the window is not neutral: it dilutes the signal you did retrieve and pushes it into the region the model attends to least.

What to evict first

Sooner or later a request doesn’t fit, and something has to go. Decide the order now, deliberately, rather than discovering it later in whatever order your string concatenation happens to run.

1. Padding results. Anything retrieved only because k demanded a number. These cost tokens, cost accuracy, and were never load-bearing.
2. Stale tool output. A tool result from fifteen turns ago that has already been summarized into an answer is the cheapest large thing in the window to drop. Replace it with a one-line note that the call happened and what it returned.
3. The oldest conversation turns. Compacted rather than deleted — which is most of part two.
4. Never the pinned constraints. The user’s stated environment, their explicit instructions, the decisions already made. These are a few dozen tokens and they are the difference between a correct answer and a confident answer for the wrong system.

Underneath the order is a rule: evict by relevance to the question currently being asked, not by age alone. A turn from an hour ago that established the deployment target outranks three exchanges of pleasantries from thirty seconds ago, and a policy that only knows about timestamps cannot tell the difference.

Where this goes next

Retrieval is the part of the context you choose. Get chunking, reranking, hybrid search and the score floor right and you have made the part you control as good as it reasonably gets — while the two parts you don’t control, conversation history and tool output, keep growing on their own schedule.

That is part two: compacting history so it stops eating the majority of your budget, designing what tools return as carefully as you design what the prompt says, and then itemizing every request by section — because the part of the context eating your budget is almost never the part you have been tuning.