AI Engineer World's Fair 2026
Lessons from Studying Every Memory System
Loading slides…
Read the talk
Lessons from Studying Every Memory System
Personal AI needs more than stored facts: it needs ways to refresh context, retrieve past conversations, resolve uncertainty, and learn what changed outside the chat.
From a talk by Shlok Khemani
Before you start: Familiarity with chat context windows and the basic idea of retrieving past information is helpful; no knowledge of model training is required.
When a new conversation forgets you
What should an assistant carry into your next conversation? For cooking, it might need your dietary preferences; for learning, what you already understand; for companionship, the history of your relationship. Here, memory means personalization in consumer AI. Shlok Khemani approaches that problem after a year spent reverse engineering ChatGPT, Claude, Gemini, and Poke, and helping teams design memory for their own products. The useful starting point is how these systems evolved, before considering what their designs make possible.
In 2023, shortly after GPT-4 launched, ChatGPT could maintain context within a conversation. Start another thread, however, and none of that context came along. Early adopters bridged the gap by copying information themselves. As people began using the product for learning, cooking, and companionship, repeatedly reintroducing themselves became a basic product limitation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Saved facts solve repetition and create maintenance
ChatGPT's February 2024 memory release began as a limited test. A request such as “remember that I'm vegetarian” could become a stored fact about the user; explicit requests were one input, alongside details picked up automatically. In Khemani's account of this first version, the stored list entered the context window of every conversation. Settings exposed the list so users could inspect and delete individual memories.
That was a useful first implementation, but its interface exposed a second job alongside chatting: maintaining the assistant's understanding of you. Seeing memories being created made users feel responsible for deciding what should be saved and what should be removed.
The list also preserved facts after their useful lifetime. One entry said Khemani was going to Bengaluru. By the time of the talk, he was in San Francisco, yet the old travel plan still entered his context window. A fact extracted correctly at one moment can become misleading later. Deletion controls make repair possible, but they do not make the system recognize when repair is necessary.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Background synthesis removes work, not errors
The April 2025 update added references to chat history alongside saved memories. Khemani describes the underlying personalization mechanism as user knowledge memories, or a running profile. In his account, ChatGPT periodically reviews conversations, extracts information it considers useful, and updates a profile that accompanies each new conversation. He calls this background synthesis dreaming, connecting it to the process discussed in Lance Martin's talk. The internal budgets and schedules here are Khemani's observations of the products around the recording, rather than documented guarantees.
The profile compresses information aggressively. Its entries pack together keywords and contextual clues, relying on the model to connect those clues to the current conversation. Khemani reports that his profile contained 16 sections and almost 4,000 tokens, including personal life and ongoing work. Updating that representation asynchronously removes much of the user's responsibility for explicitly creating memories.
But synthesis can turn uncertainty into false history. Khemani's profile described travel to both Thailand and Turkey in overlapping periods of 2025. The source conversations had explored those countries as alternatives. He went to Thailand and had never been to Turkey, yet both survived as travel history. The problem was no longer just an old fact that needed deleting: the profile had lost the distinction between considering a trip and taking it.
Inspectability also regressed. Unlike the earlier saved-memory list, the raw profile was not exposed in settings. Khemani reports obtaining it through a prompt that induced ChatGPT to reveal the profile, sometimes after retries or changes of thinking mode. That gave him a way to investigate the representation, but ordinary users lacked a direct interface for seeing the same underlying material.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Claude starts with retrieval, then adds a profile
Claude initially approached the same interface very differently. Its August 2025 chat-search release began with Max, Team, and Enterprise availability. In the version Khemani studied, there was no preloaded user profile or fact list. Instead, the model had two ways to recover context:
- Keyword or topic search: Find relevant material in previous conversations.
- Time-based retrieval: Recover conversations from a period, supporting requests about last week or the start of November 2025.
Each conversation began without personal context. The model decided whether it needed history and retrieved it on demand. A familiar chat interface had concealed a substantially different memory design.
On September 11, 2025, Khemani published Claude Memory: A Different Philosophy, describing that architecture as the opposite of ChatGPT's. He reports that the post reached Hacker News's front page. That same day, Claude announced memory summaries for Team plans, adding a running profile to its retrieval tools. Unlike ChatGPT's raw profile, Claude's was visible in settings.
Khemani describes Claude's profile as roughly 1,000 tokens, written in complete sentences, versus ChatGPT's denser, roughly 4,000-token profile. He reports Claude updating every 24 hours, compared with every few days for ChatGPT. Users could also request explicit changes: an edit triggered resynthesis of the profile, and an interface for managing earlier edits let them remove instructions that no longer applied. In his account, that design persisted from September through the talk.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A shared architecture with different visibility
ChatGPT subsequently added a tool for looking through past conversations, returning summarized context in response to model-generated queries. It then exposed a profile interface in early June 2026. What users saw, Khemani explains, was an LLM-generated summary of the underlying profile—which was already an LLM-generated summary of their conversations. Visibility into that second summary did not amount to visibility into the exact context used by the model.
The update also accepted explicit profile-edit requests. Khemani describes it as deprecating the original fact list; his account concerns the memory layers he observed, rather than establishing removal of saved memories for every account. After taking different routes, the products had converged on two complementary mechanisms: a running profile for context available immediately, and retrieval tools for context needed on demand. Both also offered some ability to inspect and change memory, although the representations exposed to users differed.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Memory architecture follows the product
The conventional starting assumption was that memory meant a retrieval-augmented generation pipeline: chunk conversations, embed the chunks, store them in a vector database, then run semantic search for each incoming query. Khemani once shared that assumption. His outside investigations instead found products organized around different combinations of synthesis and retrieval. They do not establish the absence of hidden vector infrastructure, but they do show why the generic pipeline is an incomplete product design.
Other systems broaden the design space. Khemani reports that Gemini attaches creation and last-update timing logs to memories. Agents such as Claude Code, OpenClaw, and Hermes bring Markdown files, heartbeat mechanisms, knowledge bases, and skills into the discussion. These are examples across systems, not a shared checklist implemented identically by every agent.
Memory must evolve alongside the product. Khemani's prescription is deliberately strong: teams building serious personalization should not outsource memory as an afterthought. The product determines which information matters, how it changes, and when it should affect a response. He points to leading consumer applications across categories as evidence, saying that the examples he studied build their memory in-house.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Pay to update, then pay to serve
A running profile creates two distinct compute costs. Maintenance cost depends on how frequently the profile is updated and how much computation each update uses. Serving cost arises because the profile enters every conversation: a longer representation means more context to process repeatedly. Choosing a profile length without choosing an update schedule leaves half the budget unspecified.
A small Python budgeting function makes those two terms explicit. It takes the workload and prices as inputs, leaving profile length, update cadence, and synthesis cost as separate decisions:
python
def profile_cost(
*,
updates: int,
cost_per_update: float,
conversations: int,
profile_tokens: int,
input_cost_per_million: float,
) -> dict[str, float]:
maintenance = updates * cost_per_update
serving = (
conversations
* profile_tokens
* input_cost_per_million
/ 1_000_000
)
return {
"maintenance": maintenance,
"profile_serving": serving,
"total": maintenance + serving,
}
This simplified estimate covers profile maintenance and one profile inclusion per conversation; actual billing also depends on request structure and caching. The distinction matters even before assigning prices.
Without resource constraints, a designer might refresh the profile hourly or after every conversation, spend lavishly on synthesis, and keep far more context. Khemani imagines Fable with Opus subagents doing the updates, and a 400,000-token profile instead of 4,000 tokens. The thought experiment exposes the competing desires: remember more, refresh sooner, and reason harder about what to retain.
GPU constraints force a choice. Khemani uses the following reported configurations to illustrate opposite allocations of compute; the cost directions are architectural reasoning, not measured bills.
| Product | Reported profile size | Reported refresh cadence | Relative emphasis |
|---|---|---|---|
| ChatGPT | About 4,000 tokens | Every few days | More serving context; fewer updates |
| Claude | About 1,000 tokens | Every 24 hours | Less serving context; more updates |
A larger profile spends more on carrying information forward. A more frequent refresh spends more on keeping that information current, assuming otherwise comparable update work. Memory quality therefore depends partly on how a product chooses to allocate compute.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A learning loop outside the weights
A running profile already forms a recurring learning loop:
- The existing profile supplies what the system knows about the user.
- That context shapes new conversations.
- Those conversations introduce new information.
- Background synthesis incorporates the new information into the profile.
- The revised profile shapes subsequent conversations.
Khemani calls this continual learning while explicitly locating it outside the model's weights. The persistent state changes; the underlying model need not be retrained for the next conversation to benefit.
Whether that loop will move into model weights for individual consumers remains open. Training is expensive. An enterprise can spread the expense across employees or customers, whereas a model trained for one person has different economics. Khemani leaves three questions unresolved: whether everyone will get a self-learning model, what data would initialize that learning and how to generate it, and who would pay. He recommends Gwern's Guardian Angels for an exploration of that possible future, rather than a demonstrated answer to its economics.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The missing decision was outside the chat
Memory is capped by the context it can gather. Even an ideal architecture, unlimited compute, and perfect integration of observed information into model weights would leave a system unable to learn facts it never encountered.
The Thailand–Turkey error reveals that limit. ChatGPT had seen two conversations exploring possible trips. The actual decision to visit Thailand happened later, in person, between Khemani and his partner. The assistant never heard it. There were also traces of the decision in email: flight and hotel bookings for Thailand. In Khemani's experience, connecting email did not cause ChatGPT to reason over those messages and use them to update his profile. Access to a source and incorporation into memory were separate capabilities.
Missing an offline conversation is understandable. What frustrates Khemani is that the product did not recognize the apparent conflict or seek the missing information. A profile containing overlapping travel entries could prompt a question about which trip actually happened. Instead, it retained confident assertions. He sees this lack of curiosity as a product-design problem, rather than a fundamental inability of an LLM to notice uncertainty and ask for clarification.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Personal context still has to be rebuilt
That problem compounds across products. Khemani's personal stack includes chatbots, assistants, specialized applications, agents, and hardware devices. Each builds its own memory of him. Because those memories are not shared, he must establish context repeatedly and update each product separately when something in his life changes. Meanwhile, email, calendars, and photos contain rich context that, in his experience, these products do not effectively reason over.
His question is direct: “When will personal AI feel like personal AI?” It is a question about whether the whole experience reflects a person's life, beyond whether any one application can retrieve a saved fact.
The frustration ends in optimism. The consumer-memory evolution traced here spans only about three years. Khemani expects to interact with AI throughout his life, making memory foundational to a relationship that must survive new conversations, changed circumstances, and years of accumulated context. A field this young has substantial room to build systems that can sustain that relationship.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Khemani's investigation of Claude's early keyword and temporal conversation-retrieval tools.
Gwern's proposal for personal models that learn a user's preferences through ongoing training and active elicitation.
Current documentation for OpenClaw's Markdown profiles, durable memory, daily notes and consolidation.
The Hermes Agent guide to persistent memory across sessions.
Further reading
OpenAI's original memory announcement, with dated updates explaining saved memories and references to past chats.
Khemani's September 2025 account of ChatGPT's memory layers, with prompts for exploring personal memory.
A companion analysis of ChatGPT's June 2026 Memory Summary interface and its relationship to underlying memory.
Full transcript
Loading transcript…