AI Agent Memory: How It Works and What It Costs You

You told your assistant on Monday that you write in British English and hate bullet points. On Thursday it hands you a bulleted list in American English. Did it forget? No. It never remembered in the first place.

AI agent memory is everything an application builds around a model so that yesterday's information shows up in today's prompt. The model itself holds nothing between calls. Every apparent memory is text that some piece of software saved and then put back in front of the model.

That has a practical consequence. Each memory layer has a different owner, a different place where your data sits, and a different way to fix or delete it. Here is the quick map.

Why a model has no memory of its own

A language model is a function. Text goes in, text comes out, and nothing is kept. OpenAI's developer docs say it directly: each text generation request is independent and stateless. A chat feels continuous because the app resends the earlier messages with every new one.

Anthropic's docs describe the same mechanism. On each turn the input is all previous conversation history plus the current message, and the response becomes part of the input for the next turn. The context window is the model's "working memory", and it is separate from the large body of data the model was trained on. Our explainer on the context window covers sizes and limits in detail.

Two things follow:

• You pay for memory on every call. OpenAI's docs note that even when you chain responses with previous_response_id, all previous input tokens in the chain are billed as input tokens.
• More is not better. Anthropic's docs state that as token count grows, accuracy and recall degrade, a phenomenon they call context rot. Stuffing the whole history into every prompt is the naive version of memory, and it gets worse the longer it runs.

An LLM agent makes this sharper. An agent loops through many model calls in one task, so its history grows fast, and a long run can fill the window before the job is done.

The four layers of AI agent memory

Layer 1: the context window (short-term memory)

This is the transcript of the current session: your messages, the model's replies, tool results. It is what people mean by short-term memory.

When it fills up, apps have two choices. They can drop old turns, or they can summarize them. Anthropic's docs describe server-side compaction as automatically summarizing earlier parts of the conversation so it can continue past the window limit. They also note that chat interfaces such as claude.ai can manage the window on a rolling first-in, first-out basis. Either way, detail gets lost. Whatever the summary leaves out is gone for the rest of the session.

What it costs you: tokens, and a ceiling on how much the agent can hold in mind. For privacy it is the simplest layer, since nothing outlasts the session unless another layer saves it.

Layer 2: a saved file the app re-reads

The least glamorous layer is often the most reliable. You write instructions into a file, and the app loads that file at the start of every session.

Claude Code's memory docs show this layer clearly. Each Claude Code session begins with a fresh context window. CLAUDE.md files are instructions you write; Claude reads them at the start of every session. They can live at several scopes: personal preferences for all projects, team-shared instructions kept in the project and shared through source control, and a local file for your own project-specific preferences.

The docs add a caveat worth remembering for any tool like this: Claude treats these files as context, not enforced configuration. A line in a file is a strong suggestion, not a lock. To block an action regardless of what Claude decides, the docs point to a hook instead.

What it costs you: your own time to write and prune it, and context space every session. Anything in the file is sent to the model, so keep secrets out of it. The control is excellent, because it is a text file you can open, read, and edit.

Layer 3: a store the agent reads and writes

Here the agent decides what is worth remembering, saves it, and reads it back later. There are three versions of this worth knowing.

Notes in files. Claude Code also has auto memory, where Claude writes notes for itself from your corrections and preferences. Per the docs, it is on by default in local sessions, each project gets its own memory directory, and the notes are plain markdown you can edit or delete at any time. Only the first 200 lines or 25KB of the MEMORY.md index load at the start of a session; the topic files are read on demand. The docs also say it is machine-local, so the notes are not shared across machines or cloud environments.

A memory tool you wire in. Anthropic's memory tool lets Claude create, read, update, and delete files in a memory directory that persist between sessions. It is client-side: Claude requests file operations and your application executes them, so memory lives entirely in your application, on storage you control. The docs also hand security work to you. Claude usually refuses to write sensitive information to memory files, but for stronger guarantees you add validation, cap file sizes, and delete stale files.

A store in an agent framework. LangGraph's persistence docs split memory into two systems. Checkpointers persist a thread's state and are for short-term, thread-scoped memory. Stores persist application-defined key-value data across threads and are for long-term memory such as user preferences and facts. The same page notes that the in-memory checkpointers keep data in RAM, so every checkpoint is lost when the process restarts, and it points to a persistent checkpointer for production.

What it costs you: this is where privacy gets decided. The agent chooses what to write, so you need a way to read it. And where the store lives matters. Notes on your machine, a database your team runs, and a vendor's servers are three very different answers to "who can see this?".

Layer 4: retrieval over past material

The last layer does not save a summary at all. It keeps old material, such as past chats or documents, and searches it when it looks relevant. The matching pieces get pasted into the context window for that call. That is retrieval-augmented generation, pointed at your own history.

Claude's chat product shows a mild version. Its memory help page describes memory saved as individual topics you can read, edit, and delete under Settings, then Memory, and a separate ability to search previous conversations when you ask what you discussed about a topic. You can pause memory (existing memories stay, no new ones are created) or reset it, which permanently deletes all memories. With incognito chats, Claude won't remember them. Each project has its own separate memory space.

What it costs you: retrieval pulls up what matches, which may be something you forgot you said, or something out of date. And when material is indexed as well as stored, it is worth asking whether deleting the original also removes it from the index. That is one of the questions below.

Short-term vs long-term memory for agents

The split LangGraph's memory concepts page uses is the one most teams adopt:

The same page also names three kinds of long-term content: semantic memory (facts), episodic memory (past events or actions), and procedural memory (the rules for doing a task, which the page describes as the model weights, agent code, and prompt together). The words are borrowed from psychology, but the engineering is ordinary: files, key-value records, and search.

What to ask any AI product about its memory

Whether it is a chat app, a coding agent, or a workflow tool, six questions cover most of the risk:

1. What does it save, and does it save it automatically? Some products save only when you say "remember this". Others decide on their own.
2. Can you read all of it? If the memory is a pile of topics or a plain file, you can audit it. If it is hidden, you cannot.
3. Can you edit one item and delete everything? A reset that removes all memories is not the same as deleting the one wrong fact.
4. Where is it stored? On your machine, in your own infrastructure, or on the vendor's servers. The Claude Code docs and the memory tool docs give different answers, which is the point.
5. Does it carry across projects and people? Per-project memory keeps work contexts apart. Shared memory can leak one context into another.
6. Is there an off switch, and a private mode? You want pause, off, and a way to have a chat that is not remembered.

Notice what the checklist does not ask: how smart the memory is. A visible, editable, deletable memory beats a clever hidden one.

Memory you keep yourself

Everything above is about an assistant remembering you. A related problem is keeping the setup that works. You spend an afternoon getting an AI workflow right, and a week later you are re-explaining it from scratch.

Taku comes at that from the other side. It is an AI-native desktop workspace where you mirror an AI setup someone already got working, run it, remix it for your own needs, and keep it, so the useful part is saved as a workflow rather than as one more chat. It is not a model memory system, and it does not replace the memory features of the assistants and agent tools above. Taku is in Beta, and the Mac app is available now.

FAQ

What is AI agent memory?

It is the set of mechanisms an application uses to carry information from earlier calls, or earlier sessions, into the current prompt. The model has no memory of its own, so memory is text that gets saved somewhere and then placed back into the context window.

How does AI memory work?

Software around the model saves information, such as the conversation, a notes file, or a database entry, and retrieves it later. It inserts the relevant parts into the next request. The model reads that text as if it were part of your message.

What is the difference between short-term and long-term memory for agents?

Short-term memory is the state of the current session or thread, which lives in the context window. Long-term memory survives across sessions and lives in a file, a store, or a searchable index. LangGraph models them as checkpointers and stores respectively.

How does Claude memory work?

It depends on the product. In Claude Code, you write CLAUDE.md files that load every session, and Claude can also write auto memory notes that you can edit. In Claude's chat product, memory is saved as topics you can view, edit, and delete in settings. Through the API, the memory tool stores files in your own application.

Can I turn off or delete a chat app's memory?

It depends on the product, so check its own settings. In Claude, Anthropic's help article says you manage memory under Settings, then Memory. You can edit or delete saved topics, pause memory (Claude keeps what it has but stops using or adding to it), or reset it, which permanently deletes all memories including project memories. Incognito chats aren't saved to memory. Whatever app you use, ask what it saves, whether you can see it, and how to delete it.

Can an agent remember everything forever?

No, and you would not want it to. A bigger context window raises the ceiling but does not remove the degradation Anthropic describes as context rot. Good memory is selective, and it needs pruning as facts change.