# What Does an AI Agent Need to Remember?

> Context, memory and state do different jobs for an AI agent. Walking through them with a refund agent and a python plushy.

- Author: Nyghtowl (Melanie Warrick)
- URL: https://nyghtowl.com/posts/2026/09/what-does-an-agent-need-to-remember/
- Published: 2026-09-03
- Tags: AI agents, memory, context engineering, state, Temporal
- Site index for agents: https://nyghtowl.com/llms.txt




**TL;DR:** We use “memory” as a catch-all, but an AI agent needs different information for different jobs. Context gives the model what it needs for the current decision. Memory brings useful information forward. State tells the application what is true and where the work stands. Understanding those roles matters more as the agent capabilities grow and work stretches across model calls, tool calls, waits on humans and failures. The separation is a tool for clarity. You could call all of it memory; the point is that each piece needs to be handled differently, and handling them well is what makes the agent more capable.

---

I’ve been digging into how an AI agent “thinks” over the last few months to better understand agents, especially long-running agents. Context is a term we all know and love now because it’s top of mind whenever we work with LLMs or agents. We’ve been working on how to manage and optimize it and keep that conversation going for more complex work and thus improve what our agents can do. And I’ve been in that situation where context was lost or everything went into compaction right when I was in the flow or a big change was in progress. 

Pulling in outside information, and keeping useful things beyond the current call, is what brings in memory: RAG, vector databases, Skill files, markdown files and all the other ways we try to help an agent keep track of what matters. State is different again. It is the record of what is actually true and where the work stands, owned by the systems that did the work. Context, memory and state often get grouped together in these conversations, even though they can solve different problems.

You may already know plenty about agents and memory, or nothing at all. I’m going to walk through some fundamentals around context, memory and state because I like doing this kind of thing. This is a broad subject and this post does not cover all of it, but hopefully it gives you a useful jumping-off point. You should dig into it more and/or have your agent do it.

## An Example to Work With

Let’s use an autonomous AI support agent with the authority to issue a refund to give something to anchor on while we step through the concepts. I built a sample demo, [Agent Memory and State repository](https://github.com/temporal-community/agent-memory-and-state), with versions to help give clarity on context, memory and state. The repo has an agent that works on a plain loop and a loop supported by tracking execution state, so you can crash it in the same place and watch how context, memory, effect state and execution state each behave.

In the example, the agent can:

1. Ask the customer what happened.

2. Look up the order.

3. Check the customer’s history and the refund policy.

4. Ask whatever follow-up questions it needs.

5. Decide whether the refund is valid.

6. Call the payment API.

Let’s say you were trying to get a refund for something like a python plushy. 

{{< figure src="python-plushy.jpg" alt="A green python plushy, coiled and smiling" width="65%" >}}

You answer the questions, the agent checks the order and decides the refund is valid but there is a disruption in the system. When the agent comes back up does it know your answers? Does it know which tools already ran? Does it know that issuing the refund was next? What if the payment call happened but the response never made it back? Would it start over? Would it issue the refund twice?

| | What it answers | Result when data is missing |
| --- | --- | --- |
| **Context** | What does the model see for this decision? | The agent loses the thread or makes a worse decision. |
| **Memory** | What retained information can the agent use again? | It repeats questions, forgets useful history or fails to learn from experience. |
| **State** | What has actually happened, and what is safe to do next? | It loses work, repeats an effect or acts when it should not. |

These are not three storage buckets and there is some overlap here. The same order detail could exist in a database, get copied into memory and then be loaded into context. The difference is the role the information is playing and which system owns it.

## Hello Agent

At its simplest, an agent is a model call with a result. And for an autonomous agent it’s a loop with a model. The model receives a set of instructions and data, decides what to do, takes some type of action, gets a result and goes around the loop again. The action could be a tool call, a database lookup, a message or whatever effect it leaves behind. The result gets added to what the model sees on the next call. Around again until the agent reaches a goal, hits a stop condition or fails. 

![The agent loop: the model reasons and observes, takes an action, and the results feed back into context for the next call](agent-loop.jpg)

This becomes a basic “for” or “while” loop that can be represented by something as simple as this Python code below (MAX_TURNS is acting as our arbitrary stop condition in this case).

```python
def run(request):
    messages = []

    for _turn in range(MAX_TURNS):
        step = agent_step(request, messages)

        if step.action == "decide":
            return finish(request, step)

        result = run_tool(step.tool, request)
        messages.append({
            "tool": step.tool,
            "result": result,
        })
```

A lot of scaffolding tends to grow around this bare bones version to help it function on real scenarios and in production. In the example, the agent can look up the order, look up customer history, check the refund policy and issue the refund (the tool that actually moves money). Each of those needs code and systems behind it: something to fetch the information, something to take the action, and something to carry the results back into the next call.

We need to figure out what data do we put into the model and how do we load it? How much can the model hold? What survives after the current call? What happens if the process dies? What does a new session know about what the agent has been doing as well as the world around it? These questions get grouped together as “memory” a lot, but pulling them apart is what lets you handle each one well, and that is what makes the agent more capable.

A way to define that scaffolding is that AI agents get more capable as we find ways to give them more data and better data. Think of it as a ladder, where each rung adds model complexity and capability.

- Rung 1: one model call. Everything it knows is in the prompt. That prompt is context, and it is all there is.

- Rung 2: the loop. Tool results pile up, the window fills, and you start deciding what to keep. The moment you carry something forward on purpose, you have memory.

- Rung 3: long-running. The work now spans many model calls, waits on humans and crashes. A long-running agent does not mean one model call running for hours. It means the model is not running most of the time, so something else has to know where the work stands. That is state, and unlike memory, it has owners.

- Rung 4: self-improving. The agent starts curating its own memory: reflecting on runs, extracting lessons, updating its playbook or Skill files. Now memory can be wrong on purpose, because the agent wrote it, and state is what keeps it honest.

![The ladder from one model call to a self-improving agent: each rung adds another job the model cannot reliably handle on its own](agent-ladder.jpg)

The point of the ladder is not the rungs. It is that each level changes what the agent needs fed to it, and how. One call needs a good prompt. A loop needs choices about what to carry. Long-running work needs a record that outlives the model. A self-improving agent needs all of that plus a way to tell its own notes from the truth. Context, memory and state are the three ways we serve that information, and the rest of the post takes them one at a time.

## What Is Context for an Agent?

Context is everything the model receives for the current call and it can include what is input in real time to the agent. We’ve known it as that chat window with ChatGPT or Claude or your agent of choice. The input can include system instructions, recent messages, tool definitions, database results and it can come from outside sources like a human, .md files, outside vendor input like Stripe or many other sources that we are hooking up every day. Context is also increasingly multimodal. We all know the text input but models are starting to be able to receive images, audio, video and other structured information gathered from tools and sensors.

The interface to submit context to an agent doesn’t have to be a chat window. A human has five senses, but an agent can have as many input channels as we can figure out how to connect. At the lowest level, it all boils down to binary 1s and 0s anyway. The important part is the layer in between: something has to translate a message, image, sound, GPS coordinate, wind sensor reading or even the viscosity of some mysterious goo into a form the model can use as context. 

The context window is the limited space available to hold information that is entered real time and from memory and state. A surrounding agent application (usually code and lately included as part of the harness conversation) can help assemble context every time it calls (invokes) the model. The model can only use earlier messages if the application (scaffolding) or model service makes them available again. In our refund example, context might hold the customer’s last message, order 1234, the refund policy and the result of the latest lookup. That is what the model can use to choose its next action.

## Managing What the Model Sees

Approaches on how to better manage context have been evolving and include:

- **Put everything in the window.** Simple until data no longer fits or useful details get buried.

- **Truncate it.** Drop the oldest turns and hope they are no longer important.

- **Compact it.** Summarize older turns so the main story survives in fewer tokens and hope it has enough details.

- **Retrieve it.** Use RAG, memory tools, files or databases to bring in what seems relevant and reduce context footprint until needed.

- **Use bigger windows.** Fit more history, documents and results into a single call. More capacity doesn’t guarantee the model will find or use every important detail.

- **Structure and curate it.** Give the agent Skill files, scoped instructions or a playbook so it can learn over time what context to retain and how to retain it.

These approaches solve different problems and can be combined. A large context window still needs useful information. Retrieval still needs to find the right information. Compaction is popular as of late but challenging. You are asking a model to decide what mattered about the last forty turns. It may keep the narrative and drop the specifics. Fine for a conversation. Less fine when the specific it dropped was the order number.

More context is not always better context. Extra information adds cost and noise, and old information may no longer be true. The real goal is to give the model the relevant, current and trustworthy information it needs for its goal. This is part of why we’ve expanded from prompt engineering to context engineering. The work went from prompt writing for better results to building the machinery that decides what to include, what to leave out, when to retrieve something, how to structure it and how to update it.

This gets sharper the longer the work runs. A single session might never fill the window. An agent working a case over three days across dozens of calls will hit every one of these limits.

## What Is Memory for an Agent?

Memory is information an agent retains or actively maintains so it can use it across decisions. Memory is not one thing. To affect a decision, it has to be brought into context as part of active working memory while other kinds of memory can sit outside the current window to be retrieved as needed. A taxonomy borrowed from cognitive science, and used in work such as [Cognitive Architectures for Language Agents](https://arxiv.org/abs/2309.02427), helps break it down:

![Four kinds of memory an agent can use: working, episodic, semantic and procedural, plus parametric memory in the weights](four-kinds-of-memory.jpg)

On the refund agent, all four show up. Working memory is the running case notes for this refund: what the customer said, which lookups are done, what comes next. Episodic memory is what happened the last three times a crushed plushy came in. Semantic memory is the refund policy and the fact that this customer has returned two items before. Procedural memory is the refund Skill file, the steps the agent follows to decide.

The borders (despite the image) are fuzzy. A Skill file may contain both facts and instructions, so parts of it can act like semantic memory while other parts are procedural. A vector database is one mechanism for storing and retrieving memory. Depending on what you put into it, it can hold experiences, facts or procedures. These categories describe the role information plays, not the file or store where it lives.

The fuzzy borders are especially felt with working memory and context. The context window is basically a buffer where we dump instructions, messages, tool results, files and whatever else we give the model for the current call. Not everything in it is in use. Tool definitions, a stale lookup and a summary of turns the agent has moved past are all present, but the model is not reasoning from them. Working memory is the information the agent is actively using to make its current decision, and it can even live outside the window, like the running case notes it re-reads each turn. Once we or the agent start deciding what stays, what gets retrieved, what gets summarized and what gets dropped, we are trying to make the window hold working memory. That is what context engineering is. The window is the space; working memory is the active information inside it. When the engineering is good they look identical, which is why they get confused.

There is also parametric memory, which is the information baked into a model’s weights (the model’s core structure and internal knowledge). It is why a model may know something nobody added to the prompt. For whichever model you are on during a run (e.g. Sol, Fable, etc.), that memory is not something your agent can directly inspect or edit, so we are treating it differently from memory we can manage. It contributes to the model’s general knowledge and behavior (what can give it “personality”), but it does not contain current or private information out of the box such as the customer’s plushy order unless that information was part of its training (which would change its weights). That’s why context, memory and state play key roles to make the agent functional beyond its “basic” training.

It’s also important to understand that the memory type is not determined by where it is stored. The categories are useful, but the storage label matters less than knowing what you are retrieving and why. 

## How Memory Is Changing

For a while, agent memory mostly meant saving the chat history or putting documents into a vector database and retrieving the closest matches. Those are still useful. RAG is not dead. It is one part of a larger memory system. Agent memory is retained information plus the mechanisms that make that information available and useful to later decisions.

The work is expanding from storing and retrieving information to managing a memory lifecycle:

- What is worth remembering?

- Who or what is allowed to write it?

- Is it specific to this customer, this task or every future run?

- Where did it come from?

- How and when should it be retrieved?

- When should it be updated or forgotten?

- What happens when two memories conflict?

- Did the memory help produce a better outcome?

Agents are also doing more of the curation themselves. They can reflect on what happened, extract a lesson and update a playbook or Skill file. If there were three crushed-plushy cases where the carrier’s damage record settled the refund before any customer questions did, the agent could add a step to its refund Skill file: check the carrier record first. Details from a failed run may be kept as episodic memory, then distilled into a semantic fact or a procedural instruction for future runs. 

[Agentic Context Engineering](https://arxiv.org/abs/2510.04618), for example, treats context as an evolving playbook and uses generation, reflection and curation to keep useful lessons without repeatedly rewriting everything into a smaller summary. The result is memory the agent wrote for itself. That can make an agent improve without changing the model’s weights, but it introduces another set of problems. A memory can be stale, incomplete or just wrong. If an agent writes down that a customer was eligible for a refund last month, that does not mean the customer is eligible now. If it remembers that it already issued the refund, that does not prove the money moved.

At the end of the day, agent memory is less a place and more a capability. It is the information an agent retains, along with the mechanisms that make that information useful to a later decision. The information still has to be represented somewhere, in model weights, a message history, a database or another service, but we know storage alone is not a memory system. A database full of information the agent never retrieves is stored data, but it is not functioning as memory for that agent until it is retrieved and used. While memory is useful information for making a decision, it is not automatically the authority for that data.

## What Is State for an Agent?

State is operational information a system owns and can authoritatively answer. There are many different types of state and in the refund example there are at least four with different owners:

| State | Owner in the example | What it knows |
| --- | --- | --- |
| **Execution state** | Application or workflow system like Temporal | Which work completed, the identity of the attempt and where the operation is blocked. |
| **Authorization state** | Auth system | Whether the agent may act, what it may touch and whether that permission has been revoked. |
| **Effect state** | Payment system like Stripe | Whether the payment or refund actually committed. |
| **Domain state** | Application's databases | The order, customer plan, refund row and ticket status. |

The test is which system can authoritatively answer the question right now. The agent or its memory system owns the memories it creates and maintains. Those memories may contain copies of facts owned by other systems, but storing a copy does not make the agent authoritative about the underlying fact.

The same storage technology can support both memory and state. The difference is the role:

- Memory helps the agent reason based on retained knowledge, experience or instructions.

- State authoritatively records what is true, what happened or where work currently stands.

It’s important to know that there is no single technology for any of these kinds of state. For execution state, you can build an application state machine using a database, queues and scheduled jobs. In that setup, you own the transitions, timeouts, retries and idempotency. Agent frameworks such as LangGraph provide persistence and checkpoints so a graph can resume from saved state, though you still need to design task boundaries and make external effects safe to repeat. Durable execution systems such as Temporal, DBOS, Restate and Azure Durable Functions take on the preserving and resuming for you. What separates them is not what they store but how much of the hard part you still own, and the same is true for the other kinds of state.

For the other kinds of state, authorization may live in an application’s permissions database, an IAM service, a policy engine or an approval system. Effect state lives with the system performing the action, whether that is Stripe, an email provider or another API. Many of us already think this way about domain state, which is usually spread across several databases and services rather than owned by one technology.

Even an agent that starts and finishes in one call leans on state. It still needs the order to exist and the permission to refund, which are domain and authorization state owned by other systems. What it does not need is execution state, because nothing has to survive between calls. Its plan and its reasoning live in context and vanish with the call. Once the work spans calls, waits on humans and failures, execution state becomes the thing that helps ensure the work continues.

## Where It Gets Muddy, and How to Keep It Straight

Depending on the framework, paper or person, memory may be described as state, state may include memory, or both may be treated as persisted context. If state means everything an agent carries from one step to the next, then memory can be considered state. If memory means retained knowledge and experience while state means the current condition of the work, separating them becomes useful. 

{{< figure src="plushy-in-the-mud.jpg" alt="The python plushy, now sitting in a mud puddle" width="65%" >}}

This gets more noticeable with long-running agents. The model is not continuously running the whole time. The workflow around it carries the work across model calls, tool results, human waits, pauses and failures. Persisted workflow state can keep messages, plans and other working memory available during that time. Later, the history of the run can also become raw material for longer-term memory. The same information may help resume the current run and teach the agent something for future runs, but those are different jobs.

I’m not trying to settle whether memory is state or state is memory. For this post, I’m separating context, memory and state by the jobs they are doing to help clarify the different parts of information that are needed to make these AI agents successful. The same information may appear in more than one place and play more than one role, which is exactly why this gets muddy.

For order 1234, it lives in the orders database, where it is domain state and the database is the owner. The agent may keep a note that this customer ordered a python plushy last month, and that note is memory. When the customer writes in, the order details get loaded into the prompt, and now they are context. This one fact shows up in all three, and only the database is allowed to settle an argument about whether the order exists. This is why it’s important to think about what role the information is playing and which system answers when two copies disagree. 

The table below shows what each one holds during a normal refund, and what happens in two failure cases: the process dies after the agent approves the refund but before the payment system receives it, and the process dies after Stripe processes the refund but before its response gets back to the agent.

| | Normal refund process | Process dies before the refund | Process dies after payment is accepted |
| --- | --- | --- | --- |
| **Context** | The plushy complaint, the customer's form answers and the order 1234 lookup | Gone with the process | Gone with the process |
| **Memory** | The refund policy and this customer's past returns | Can restore remembered answers, but cannot say where the work stopped | Says a refund was pending, but does not prove the money moved |
| **State** | Order and history checked, refund not yet issued and agent still authorized | Stripe: no refund. Temporal: lookups are done; issuing the refund is next | Stripe: refund succeeded. Temporal: refund in progress, waiting for the response |

When it comes to state, for “did the money move?” Stripe is the owner. For “has this workflow recorded the Activity as completed?” Temporal is the owner. For “may the agent still issue a refund?” the authorization system answers. There is no single record that wins every disagreement. The owner wins for the question it owns. You can check out the repo shared at the top if you want to try it out yourself and watch the difference.

## Closing Time

So, back to the original question: what does an agent need to remember? Not everything an agent needs belongs in memory, and memory itself is not one thing. An agent needs different information to do different jobs.

Context gives the model what it needs for the current decision, like whether to refund the python plushy. Memory brings useful information forward from earlier runs, like the fact that this customer has never asked for a refund before. State tells the application what is true and where the work stands: the order exists, the refund has not been issued yet and the agent is still allowed to issue it.

The same information may appear in all three, but they are not interchangeable. When you are designing an agent, ask what the model needs now, what will be useful to remember later and which facts need to be checked with the system that owns them. Each one fails differently too. Context gets crowded and loses the specifics, memory goes stale and starts sounding like fact, and state gets skipped when a memory says the work already happened.

The higher an agent climbs on the autonomy ladder, the more help it needs curating what it carries: context so it knows what to focus on now, memory so it keeps getting better, and state so what it believes stays true, checked against the systems that own the facts. Be aware of the role each one plays in the agents you build and work with, and use what fits the problem you are actually trying to solve. If all you need is one model call, or losing the work is cheap and starting over is fine, keep it simple. And go get your python plushy.

## If You Want a Place to Dig In More

- Sumers, Yao, Narasimhan and Griffiths, [Cognitive Architectures for Language Agents (CoALA)](https://arxiv.org/abs/2309.02427)

- Zhang, Hu, Upasani and others, [Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models](https://arxiv.org/abs/2510.04618)

- [Agent Memory and State demo](https://github.com/temporal-community/agent-memory-and-state)
- [Temporal Workflow Execution and replay](https://docs.temporal.io/workflow-execution)
- [Stripe idempotent requests](https://docs.stripe.com/api/idempotent_requests)
- [LangGraph persistence](https://docs.langchain.com/oss/python/langgraph/persistence)