Context window
The maximum number of tokens a model can process in a single request, encompassing both the input prompt and the generated output.
What it is
The context window defines the limits of the attention mechanism, which calculates relationships between all input tokens simultaneously. Because the underlying hardware stores the active state of these relationships in memory, larger windows require exponentially more VRAM. Current models commonly support windows from 32k to 2M tokens. When the total token count exceeds this limit, the model effectively loses visibility of the earliest parts of the conversation.
Why it matters
Exceeding the context window triggers truncation errors or requires you to implement aggressive data eviction strategies like sliding windows or summarization pipelines. If you rely on long-term memory, simply stuffing documents into the prompt will eventually hit latency and cost bottlenecks. You must design your system to prioritize only the most relevant context to maintain performance and keep inference costs manageable.
In practice
You manage this by monitoring the total token count of your input payloads and configuring the max_tokens parameter in your API calls to prevent overflow. In production, you will observe latency spikes as you approach the window limit due to the overhead of the KV cache. If you hit the limit, you must implement manual RAG or document chunking to fit the necessary information.
The tradeoff
Larger context windows provide more background information but increase the cost per request and slow down time-to-first-token. A common mistake is assuming that a large window guarantees perfect recall of all input data, as models often perform worse on information buried in the middle of a massive context.