Context Window Explained: Why the AI Forgets

You are two hours into a chat. You set ground rules at the start: tone, format, what to leave out. Now the assistant ignores them, and it asks a question you answered an hour ago. It didn't get tired. It hit the limits of its context window.

A context window is the amount of text a model can work with at one time. Everything it needs to answer you has to fit inside it, and what doesn't fit, or what it can't keep track of, effectively isn't there.

Short version:

• It's working memory, not storage. Anthropic's docs call the context window a "working memory" for the model. Google's Gemini docs use short term memory as the analogy.
• It's measured in tokens, not words, and it holds more than your messages.
• Bigger isn't automatically better. Anthropic says outright that more context isn't automatically better.
• Fixes exist at three levels: shorter chats, summaries of old turns (compaction), and keeping information outside the window (memory and retrieval).

What fills a context window

The window isn't just the chat you can see. Anthropic's context window guide spells out what counts: the system prompt, every message in the conversation (including tool results, images, and documents), and the tool definitions. The reply counts too, including any extended thinking the model does before it answers.

In practice, the pieces look like this:

Two details surprise people.

The history is re-sent every turn. OpenAI's API docs say each text generation request is independent and stateless. A multi-turn conversation works because earlier messages are passed back in as part of each new request. Anthropic describes the same loop: each turn's input holds all previous history plus your new message, and the reply becomes part of the next turn's input. A chat app does this for you, which is why a conversation feels continuous. The model isn't remembering anything between turns. It's rereading the transcript.

Tool definitions cost space before you type a word. Each connected tool describes itself to the model, and that description sits in the window. Add a dozen tools and you've spent room on instructions instead of your task. That's one reason an agent can feel worse after you plug in more integrations, and the Claude Code 1M-context guide notes the same startup load from every connected server.

Window sizes change with every model release, so we won't list them. As of October 2026, Anthropic's page describes its window as "up to 1M tokens, depending on the model", and Google's page gives a sense of scale for 1M tokens: about 50,000 lines of code or eight average-length English novels. Check your model's own documentation for the number that applies to you.

Tokens are not words

Models don't read words. They read tokens, which are chunks of text. OpenAI's docs describe tokens as commonly occurring sequences of characters and give a rough rule of thumb for English: one token is about four characters, or three quarters of a word.

That's an estimate for English prose, and it moves around:

• It's a rule for English text. For code, other languages, or anything unusual, count instead of assuming.
• Tokenizers differ between models. Anthropic's token counting guide says newer Claude models use a newer tokenizer that produces roughly 30 percent more tokens for the same text than earlier models, and it tells you to recount your prompts against the model you plan to use. The same document is a different size on different models.
• Counts are estimates. The same guide says the count it returns is an estimate that can differ by a small amount from the real usage.
• Images and PDF pages count too. Anthropic's context guide lists them among the things that count, so attachments take room alongside your typed text.

If you work in the API, both vendors offer a way to check: OpenAI has a tokenizer tool, and Anthropic has a token counting endpoint. In a chat app, the practical version is simpler: a long pasted document is a lot more tokens than it looks.

Why long chats go wrong

Four things happen, and they're easy to confuse.

1. The window fills up. On the API, Anthropic says a request whose input alone exceeds the window is rejected with a "prompt is too long" error. OpenAI's docs say tokens generated beyond the limit may be truncated. Chat apps handle the limit in their own way. Anthropic's docs note that chat interfaces such as claude.ai can manage the window on a rolling, first-in-first-out basis, which means the oldest messages go first. If an app trims that way, the ground rules you wrote at the start are the first thing to disappear.

2. Quality drops before the window is full. Anthropic's context window guide says that as token count grows, accuracy and recall degrade, and it names the effect context rot. Its context engineering article describes the model as having an "attention budget" that large amounts of text draw down. Anthropic frames this as a reason to curate what's in the window, not just to buy more room. You can feel it as the assistant drifting from your instructions in a very long thread, even though nothing has technically fallen out.

Google's docs make a related point from the other direction. Its long-context page says single-fact lookup tests are the simplest setup and that when you need several pieces of information found at once, the model does not perform with the same accuracy, and performance can vary a lot depending on the context. Google also says that, for Gemini, putting everything into the window is now the default place to start. Vendors differ in emphasis, but none of them says the window is free to fill.

3. A new chat starts empty. Because the model reads only what's sent to it, a fresh conversation contains none of your last one. People call that forgetting. It's closer to a different notebook.

4. Stale material stays in the window. A wrong draft, an abandoned plan, and a long error log all keep taking space and attention until something removes them. Anthropic's context editing docs put it bluntly: context is a finite resource with diminishing returns, and irrelevant content degrades focus.

How compaction and summarising work

When a conversation gets long, the standard fix is to replace the old turns with a summary. Anthropic's compaction overview describes it as replacing the older turns of a conversation with a summary the model writes, which keeps the conversation inside the window and keeps the active context small. OpenAI's compaction guide describes the same idea: reduce context size while preserving the state needed for later turns, which it can trigger automatically once the token count crosses a threshold you set.

At a concept level, the steps are:

1. The conversation grows toward a limit or a chosen threshold.
2. The model, or the app, writes a condensed version of everything up to that point.
3. The summary replaces the old turns. Recent turns may be kept word for word.
4. The conversation continues from the summary.

Two caveats from the vendors' own docs are worth knowing:

• A summary can lose things. Anthropic offers a way to write your own summarization prompt for the case where the default summary drops something a later turn needs. Details you care about may need to be called out explicitly.
• You may not be able to read it. OpenAI says its compaction output is opaque and not intended to be human-interpretable, so don't assume you can audit what was kept.

Compaction isn't the only option. Anthropic's context editing docs describe clearing old tool results by rule instead of summarizing them, which suits agent runs that read lots of files and web pages. If you use a coding agent, the same idea shows up as its own feature: how Claude Code auto-compact works covers what survives and what doesn't.

Context window vs memory vs retrieval

These three get blended together, and mixing them up is how people end up debugging the wrong thing.

Anthropic's memory tool docs describe memory as letting Claude store and retrieve information across conversations in files, without keeping everything in the context window. Notice what that means: memory isn't a bigger window. It's a place to put things so they can be loaded back when they matter. Our guide to AI agent memory goes through the types.

Retrieval is the other half of the same idea. Google's long-context page lists RAG with semantic search among the older strategies for moving data out of the context window and into a vector database, and notes that large windows opened up new options. Anthropic's context engineering article describes a "just-in-time" style, where an agent keeps lightweight pointers such as file paths and links and loads data into the window only when it needs it. The explainer on what RAG is covers how that lookup works.

The short rule: the window is what the model is reading right now. Memory and retrieval decide what gets put in front of it.

Habits that keep a long session useful

None of these is magic. They all come from the same logic: a window is finite, and what's in it matters more than how much.

• One task per conversation. When you move to something unrelated, start a new chat. It costs nothing, and the old material stops competing for attention.
• Put the rules in the first message, then restate them when it matters. If the thread is long and the model drifts, paste the key rules again near the end.
• Ask for a summary before you leave. Ask the assistant for a short brief of decisions made, open questions, and constraints. Paste it into a fresh chat. This is compaction you control.
• Keep a notes file for anything long-lived. A project brief you maintain does the job of memory across sessions, and you decide what's in it.
• Attach what's needed, not everything. If you need one section of a long document, paste that section. A big PDF takes space even when only a page matters.
• Don't stack large tool sets. Every connected tool adds its description to the window. Connect what the task needs.
• Watch for the symptoms. Ignoring early instructions, repeating questions, or contradicting earlier decisions all mean the thread has outgrown its usefulness.
• Remember the cost side. In API products, every turn sends the whole history again. Anthropic's docs note that prompt caching changes what you pay for those tokens, not whether they count toward the window.

Where this leaves a non-developer

Most of this is setup knowledge that someone works out for themselves: how to split a task, what to save, how to restore it. Taku is built for the case where you'd rather not rebuild that setup from scratch. You pick an AI setup someone has already got working, Taku mirrors it into a desktop workspace, and you run it, remix it, and keep it. It doesn't change how context windows work, and it doesn't make the limits go away. It spares you from starting from a blank page. Taku is in Beta, and the Mac app is available now.

FAQ

What is a context window?

It's the amount of text a model can use at one time. It includes your messages, the system prompt, attached files, tool results, and the model's own reply. Anthropic calls it the model's working memory, and Google compares it to short-term memory.

Is a context window the same as memory?

No. The window is what the model reads in the current request. Memory is information saved outside the window so it can be loaded into a later one. Anthropic's memory tool docs describe exactly that: storing and retrieving information across conversations without keeping everything in the context window.

Why does the AI forget what I said earlier?

There are a few common causes: the conversation outgrew the window and older messages dropped out, the thread got so long that recall and accuracy suffered (Anthropic calls this context rot), or you started a new chat that doesn't contain the old one. Which one it is depends on the app.

How many words fit in a context window?

A rough estimate for English is that one token is about three quarters of a word, per OpenAI's docs. Code and other languages usually take more tokens per word, and different models count tokens differently, so use the vendor's token counter if the number matters. Window sizes also change with each model, so look up your model's current figure.

Is a bigger context window always better?

No. A bigger window lets you fit more, but Anthropic says more context isn't automatically better, and Google notes performance can vary when you need many pieces of information found at once. Cost and focus matter too. A tidy conversation inside a big window beats a cluttered one.

What is compaction?

It replaces older parts of a conversation with a summary so the work can continue inside the window. Anthropic and OpenAI both offer it in their APIs. The tradeoff is that a summary can drop details, so check what was kept.