Context Window

Think of the context window as working memory: only what fits inside it is "visible" to the model during the current request. Anything that falls outside — such as the beginning of a very long conversation — ceases to exist for the model unless you explicitly include it again.

Large context windows have made many applications practical for the first time: reviewing complete contracts, summarizing long email threads, comparing multiple documents simultaneously. That said, feeding in relevant excerpts strategically (RAG) is usually more precise and cost-effective than dumping everything in.

For persistent AI agents, you need additional long-term memory outside the context window: a database or knowledge base from which the agent retrieves relevant information for each task. This keeps company knowledge permanently available without having to send it along with every request.

Context windows have grown dramatically in just a few years — from a few thousand tokens to hundreds of thousands, even over a million in current models. This has opened up new application classes: reviewing entire contract sets in a single pass, analysing large codebases, evaluating extensive project histories.

There's also a known practical limit: with very long contexts, the model's attention to detail drops in the middle of the text — information at the beginning and end gets processed more reliably. Professional systems therefore structure long inputs deliberately, place critical information at the edges, or break the task into steps rather than relying on sheer window size.

Economics also favour discipline: every token in the context window is charged for with each request. If you routinely send hundreds of thousands of tokens when three relevant passages would suffice, you're multiplying ongoing costs with no quality gain. Techniques like prompt caching (recurring instructions are cached and charged at a lower rate) help, but they don't replace the discipline of selecting relevant content carefully.

In extended conversations — like a support chat spanning many messages or an agent handling a complex task across multiple steps — the history grows and risks filling the context window. Naively sending the complete history with every request is expensive and eventually dilutes precision because what actually matters gets lost among the noise.

Professional systems solve this through compression: older parts of the conversation are condensed by the model into a compact summary, while recent messages remain in full text. This preserves the thread without overflowing the window. Additionally, facts that matter permanently — customer ID, decisions made, open items — are stored in external memory and retrieved only when needed.

For clients, this is a quality marker worth asking about: how does the system handle long-running processes? An assistant that 'forgets' the beginning during extended conversations or suddenly becomes slow and expensive has exactly this gap. Clean context management is invisible when it works — and painfully obvious when it doesn't.