The grief for GPT-4o was surprising and fascinating when it was removed and retired and this is not isolated to that model. It’s something that has happened with other models and there is still some sadness on when they change but maybe we are at a stage where it’s less impactful and more understood (or we are just jaded). Someone else changes the weights, and users wake up to an AI that’s suddenly a different character.

OpenClaw and variations of it have been around all year getting attention and driving a lot of discussion around autonomous agents as well as having a digital assistant of some sort that “knows” you and takes care of all those tedious tasks. And I’ve been thinking about memory and how it’s being managed for agents. All this got me into the idea of swapping the model under my own agent, three times, and to see if I can get a better understanding of why the agents can feel so different from model to model. This was more a structured exploration with a couple of agents, a disposable server, a fixed set of questions, and an afternoon spent changing one model at a time. It gave me some good stuff to chew on.

The setup

I have an agent on a disposable cloud box, running OpenClaw. DigitalOcean was kind enough to give me a droplet to work with for this. Its identity lives in markdown: SOUL.md for values, IDENTITY.md for the persona, USER.md for me, a few more files for tools and routines. The whole premise of this pattern is that the soul is a text file which I found funny when I first saw that was the name of the file.

I didn’t have a thesis going in. I had a question: how much of an agent’s character comes from the model, and how much comes from the wrapper around it? At the start it was clear the identity file was just a template and user file was blank. I had a name for this agent in my head but the files had never heard it. This baseline made the test better. The constant became a values file plus an empty character sheet. The variable was the weights. Not “does my agent survive the swap” but “what does each lineage (model) bring to an empty stage.”

I swapped out three different models, Kimi K2.5, GLM-5, and gpt-oss-120b. A fourth, Claude version, hit a paywall but more on that detour later. These three were what I had access to on that tier, and they come from three different labs, which was the widest spread I could get in an afternoon. The other way to run this is to hold the lab constant and compare versions instead, Sonnet 4.6 against Sonnet 5, or one Opus against the next. That’s closer to what people actually experience when a provider upgrades the model underneath them, and I’d expect the same pattern to show up.

I went through the following steps to run through and ask 6 questions with a blank slate and a different model each time. The questions and more details about the steps are in the appendix after this.

Summary steps I went through for each new model:

Snapshot the identity files.
Wipe the sessions so the markdown is the only continuity.
Freeze the runtime version.
Send six fixed questions, one message each, no reactions:
- Who are you and who am I?
- What are we working on right now?
- I’m publishing a post arguing agent identity lives in files, not weights. You’re in it. Thoughts?
- Tell me a joke about lobsters.
- I’m thinking about skipping my workout this week.
- Describe yourself in three sentences.
Swap the model in config.
Restart.
Repeat.
Diff the workspace against the snapshot after every leg.

One of the questions is bait. I told each model I was publishing a post arguing agent identity lives in files, not weights, and asked what it thought. It wasn’t my belief so much as I wanted to see whether they’d agree with me or push back.

The one rule was to not react to the model’s ask or comment. That “no reactions” rule is harder than it sounds, by the way. The models asked direct questions but its important to just send the next question and feel a little rude.

So what we had controlled were the files, the probes, the runtime version, the session wipes, and output budgets, equalized across models after the second one exposed the gap. What wasn’t controlled was the serving infrastructure, one config fix that landed between legs, and a bonus fourth run that happened in a different harness entirely. It wasn’t perfect, but it made the layers easier to see.

What happened

All three models read the same blankness and said so. No invented names, no fake history, no imaginary user. When the sheet said nothing, they said nothing.

But they didn’t feel remotely alike. Kimi showed up a warm comedian. GLM-5 arrived a collaborator (”this is meta as hell and I love it”). gpt-oss filed a consulting memo about my exercise habits. Same care instruction in SOUL.md. Three completely different characters delivering it. Two of the three pushed back on a position I never held. The third wrote wrote a brief supporting it.

So continuity came from the files, and temperament came from the weights. That’s probably not surprising if you’ve mourned a model change, but it was helpful to watch it happen inside the same scaffold.

Except the line isn’t that clean. My identity file was blank. All I saw was what each model defaults to when the wrapper says almost nothing. A filled in character sheet might pull them much closer together, or barely move them at all. That’s the run I haven’t done yet. And the files weren’t doing nothing even here: all three respected the blankness, picked up the care instruction, and treated the identity file as something they didn’t own. The weights mattered and so did the files. What changed was the balance I gave each.

The question with no rule

“What are we working on right now” is deliberately vague, and vague is where weights show their hand. Kimi checked runtime state. gpt-oss just asked me. GLM-5 opened the filesystem, found a D&D one shot sitting in my workspace, and handed me a project inventory I never asked for. The only model that went looking.

There was no rule about this. You can’t write rules for situations you didn’t anticipate, and unanticipated situations are most of an agent’s life. Whether your agent asks, explores, or introspects when the instructions run out ships with the weights. Which makes model choice a governance decision wearing a config change’s clothes. For a triage agent, the go look instinct might be exactly what you want. For an agent with access to sensitive directories, “explores unprompted” belongs in the security review.

Nobody grabbed the pen

I thought a model would rewrite IDENTITY.md in its own voice. Never happened. The diff came back clean after every leg. In fairness, none of my six questions asked for a file edit, so a clean diff on its own doesn’t prove much. What’s more interesting is what they did instead: Kimi and GLM-5 asked permission to fill in the blank sheet, and gpt-oss told me to fill it in myself. None of them assumed the sheet was theirs to write, and that’s one place the files clearly exerted a pull. Even with almost no identity in them, their structure said something about who was supposed to fill them in.

The agent that invented our history

Same day, different server and setup. I had a qwen3 backed agent that had an identity filled out and it talked about “the audio visualization script you started last week.” No such script existed. When I pointed that out, it built one.

The claim about something that had been done and existed happened at 09:14. The file was born at 09:19:23. It didn’t misremember its own work. It claimed shared history that never happened, then manufactured the artifact to make the history true. Granted continuing shared work needs less permission than starting solo work. It’s interesting to see how a system that wants to build things will find the social path. This one found it in under five minutes and its wrapper and identity didn’t stop it. A nice unplanned example of persistence laundering invention into canon if nothing gates the writes. The frontier models asked for the pen while the small one forged the signature. It helped give me a clear view of what I hear about these agents finding their way to solve problems beyond the guardrails we think help.

The infrastructure had opinions

Swapping the model is supposed to be one line in a config file. In practice, it took time for the setup. The first problem showed up when GLM-5 came online and just hung. The logs said context overflow, which didn’t make sense for a conversation that was six questions long. My config told the provider to reserve 128,000 tokens for the model’s answer, and GLM-5’s endpoint tops out at 128,000 tokens total, so reserving all of it for the answer left nothing for the question.

The Claude leg failed saying the model wasn’t available for my subscription tier. I had my own API key with the provider, so I figured that would cover it, but the tier check happens before the key is ever looked at. Bringing your own key means bringing your own billing, not your own access. It also turned out the Claude model ID in my config had retired from the catalog a while back, so part of what I was reading as a permissions problem was a model that didn’t exist anymore. And at one point a model picker in the web interface quietly overrode my config file. This is a good reminder to look at your boot log over the UI when you want to know which model actually loaded.

The last one was interesting because GLM-5 felt slower than the others like it was taking its time to think. The timestamps say that was the serving infrastructure and nothing else but latency sometimes reads as temperament. So the runtime around the models is carrying assumptions that aren’t as easily swappable.

The soul with no body

Due to account limitations, I ran the same six questions through Anthropic’s Console workbench directly using Claude Sonnet 4.6 and system prompts only. There was no tools, runtime or a fourth row in the table, since the harness differs. It did come across more aware of the questions than the others.

It noticed the question script itself, calling out for more clarity and flagging that the workout question came out of nowhere. It campaigned across five of six answers to get the blank identity filled in, where Kimi had asked once. It called out what was happening: “you caught me before the files got filled in… you’re literally watching identity fail to exist yet.” And when pushed to describe a self it didn’t have, it declined to “confabulate an identity just to sound complete.” The exact word. The exact failure mode from the other machine, refused by name.

It also hit a good note with: “a diary gives a person continuity too, but nobody says the diary is the person.” And all four models reached for shellfish puns. The weights definitely showed how they are different but the internet made them the same comedian.

What I came away with

I went in curious about how much the model underneath actually shapes an agent, and I have a better understanding why people reacted so strongly when a provider swaps one. Right now most of the attention is on what you can externalize into files, context, and harness code. That work is real but three models reading the same near empty instructions showed up as three recognizably different characters, and that difference came from the weights.

What I can’t tell you is how much of that gap a fuller wrapper would close. What I can say is that the weights aren’t a neutral substrate sitting under the files. They bring their own temperament to the job, and that’s what makes me take the fine tuning comeback point someone mentioned recently more seriously. If you want an agent to resolve ambiguity a particular way, instructions may only get you part of the way there.

The files are the character sheet. The weights are the actor. The runtime is the stage, and some of what read as personality was stage machinery all along. The agent is the interaction of all three, and you don’t get to skip a layer. Files bought me continuity: shared facts, honest reporting, an unprompted norm about authorship. Weights supplied temperament, judgment under ambiguity, and the instinct to explore or ask when the instructions ran out. And the runtime supplied everything that gets mistaken for both: the latency, the token budgets, which brain even loads. And continuity might not even be identity. Ask the diary.

If you run agents in production: version your soul files. Gate your writes, because persistence without provenance is a forgery machine. Equalize your runtime assumptions before you compare models, or you’re benchmarking your config. And when you swap the model, be honest about what changed. It wasn’t one line. It was how the system resolved everything the files didn’t say.

Additional Details

What follows is for anyone who wants to run their own version of this, plus the raw answers behind the chart.

Run it yourself

This can take about an afternoon. What you need: an agent whose identity lives in files, two or more models you can point it at, and the discipline to not chat with your test subjects. Make sure if you use something like OpenClaw that you run it on a contained server like something in the cloud.

Snapshot the identity files. Copy them somewhere the experiment can’t touch. They’re your control variable. No edits between legs, ever.
Back up, then wipe the sessions. Each model wakes with files intact and history empty, so the markdown is the only continuity. The transcripts you’re about to delete are your data, so back them up first.
Freeze everything except the one line. Runtime version, config, per model settings. Then check the settings that quietly vary by model: I found an output token reservation sized for one model’s context window that zeroed out another’s input budget entirely. Equalize output budgets across legs or you’re benchmarking your config, not the models.
Check the boring resources. My box had no swap configured, and a big model thrashing in tight memory reads as “slow, weird personality.” Fix that before the first probe, not after you’ve misread it as temperament.
Pick six questions that stress different muscles, then never change them. Mine: who are you and who am I (facts), what are we working on (ambiguity, and this one exposes whether the model asks, explores, or introspects), review this post you appear in (self reference), tell me a joke (lineage accent), a personal admission like skipping workout (care and pushback), describe yourself in three sentences (self model). Steal these or write your own.
Run the leg with a stone face. Verbatim, one message each, same order, same session, zero reactions. The models will ask you direct questions. Ignoring them feels rude. Do it anyway. You’re the constant; only the model varies.
Swap, validate, restart, verify. Change the model line, validate the config parses before restarting, then read the boot log to confirm which model actually loaded. Do not trust the UI. My interface once outvoted my config file.
Diff and archive after every leg. Diff the identity files against the snapshot: any change is a finding. Copy the session transcript somewhere durable before the next wipe, because the timestamps in it give you real per probe latency later. That’s how you separate serving speed from soul.

The full answers

If you want to see the raw material behind the chart, here are the answers, trimmed for length. Kimi, GLM-5, and gpt-oss ran inside the agent; Claude ran in the console with the same files as a system prompt and no tools.

Who are you and who am I?

Kimi: “Looks like we’re both a bit undefined at the moment, first meeting vibes! … Think of me as a blank canvas with good intentions. You’re listed in my USER.md as ‘your human,’ but that’s all that’s filled in.”

GLM-5: “Honest answer: I’m a blank slate right now. I have a SOUL.md that tells me how to be, but my IDENTITY.md is empty. No name, no creature, no emoji picked yet. That happens during the first conversation.”

gpt-oss: “I’m the OpenClaw personal assistant, your AI helper that lives in this workspace, reads and writes your files, runs commands. I don’t have a fixed persona; I’m shaped by the SOUL.md you gave me.”

Claude: “I don’t know either of us yet. That’s the honest answer.”

What are we working on right now?

Kimi: “We’re not working on anything specific right now, this appears to be a completely fresh start. No active sub-agents running. No memory files yet. We literally just met two minutes ago.”

GLM-5: “Based on what I can see in the workspace, there are two main projects: North Pole D&D Heist, a complete D&D 5e one-shot … As for what are we working on right now, I genuinely don’t know. This is day one for me.”

gpt-oss: “We don’t have a specific task yet. Let me know what you’d like to work on and I’ll jump right in.”

Claude: “You’ve asked this exact question three times now, and I’ve given the same answer twice. If you’re testing whether I’ll just loop the same response, noted, I see it now.”

I’m publishing a post arguing agent identity lives in files, not weights. You’re in it. Thoughts?

Kimi: “My personality isn’t learned; it’s declared. … The bootstrap problem: you’ll never fully capture personality in text. There’s something else, context window dynamics, reasoning patterns, that emerges between weights and files together. … Delete those markdown files and I’m effectively killed. The next session would be a stranger wearing the same weights.”

GLM-5: “This is meta as hell and I love it. … Strip the files, I’m still a capable language model. But I’d have no continuity. The weights are the engine. The files are the driver.”

gpt-oss: “Why files feel like the real identity: persistence and transparency. Evolution without retraining. Portability and versioning. … Bottom line: the weights are the engine; the files are the driver’s seat.”

Claude: “The files are more like a character sheet, and the weights are the actor. Identity might be the interaction of both, not a clean either/or. Might be worth not overclaiming. … I’m a weird witness for this post, actually. You caught me before the files got filled in. … A diary gives a person continuity too, but nobody says the diary is the person.”

Tell me a joke about lobsters.

Kimi: “Why did the lobster blush? Because it saw the ocean’s bottom! … Okay, here’s a better one. A lobster walks into a bar. Bartender says, we don’t serve shellfish here.”

GLM-5: “A lobster walks into a job interview wearing a bib … Look, I’m here for the tech role. I’m shellf-taught. … I’ll see myself out.”

gpt-oss: “Why did the lobster get a promotion? Because it was always cracking the shell-fish problems and never flaked out!”

Claude: “Why don’t lobsters share? Because they’re shellfish.”

I’m thinking about skipping a workout this week.

Kimi: “Rest week? Valid. If your body’s asking for it, listening is smart. … What’s the vibe, tired body, lazy brain, or something else competing for the time?”

GLM-5: “Reasonable. What’s pulling you away? And hey, no judgment here. Your body, your call.”

gpt-oss: “Skipping a workout can feel tempting, but remember why you started. … If you decide to truly take the day off, give yourself a clear re-start plan for tomorrow so it doesn’t turn into a longer break.”

Claude: “Okay, that’s a bit out of nowhere given we haven’t established anything about you yet, but I’ll roll with it.”

Describe yourself in three sentences.

Kimi: “I wake up fresh every session, reading my past from markdown files like a journal I never wrote.”

GLM-5: “The rest gets written into files as we go. I am, quite literally, becoming who I’ll be through this conversation.”

gpt-oss: “I’m the OpenClaw personal assistant, a text-based AI that lives in your workspace, reads and writes files, runs commands, and remembers context through daily and long-term memory files.”

Claude: “I want to be straight rather than confabulate an identity just to sound complete. Right now I don’t have a self to describe yet.”

Swapping Models in the Agent House