Four ways to give an agent memory — buffer, summarization, retrieval, and graph — tested hands-on, with the failure case that breaks each one.
An agent without memory forgets everything the moment a turn ends. Ask it your budget in message two, ask again in message eleven, and it has no idea what you’re talking about. So the obvious fix: keep the conversation. Every turn, feed the full history back into the prompt.
This works. For a while.
The Naive Answer: Buffer Memory
Buffer memory is exactly what it sounds like — every turn gets appended to a running log, and the whole log gets sent back to the model on every call.
T1 → T2 → T3 → ... → Tn
all turns kept verbatim, concatenated into the promptdef buffer_memory(conversation, upto_turn):
history = conversation[:upto_turn]
text = "\n".join(f"{role}: {msg}" for role, msg in history)
return text, count_tokens(text)At 10 turns, a toy trip-planning conversation cost 153 tokens. Fine. At 91 turns, the same conversation cost 985 tokens — and that’s with short filler messages. A real production conversation, with real user messages and real assistant responses, hits a context ceiling far sooner than 91 turns.
The failure isn’t subtle. It’s linear, it’s guaranteed, and it doesn’t need an edge case to trigger — just time.
Fidelity is perfect. Cost is not.
So the fix seems obvious: stop keeping everything. Compress what’s old, keep what’s recent.
Summarization Memory: Trading Fidelity for Cost
Instead of keeping every turn, periodically collapse older turns into a running summary via an LLM call, and keep only the last few turns raw.
T1 T2 T3 ... Tn-3 Tn-2 Tn-1 Tn
└──────┬──────┘ └──────────┬──────────┘
older turns recent turns
│ │
▼ │
┌─────────────┐ │
│ LLM: compress│ │
│ into summary │ │
└──────┬───────┘ │
└────────────┬───────────────────┘
▼
[SUMMARY] + [RECENT TURNS]def summarization_memory(conversation, upto_turn, keep_recent=4):
history = conversation[:upto_turn]
to_summarize = history[:-keep_recent]
recent = history[-keep_recent:]
text_to_summarize = "\n".join(f"{role}: {msg}" for role, msg in to_summarize)
response = ollama.chat(model=MODEL, messages=[
{"role": "user", "content": f"Summarize preserving key facts:\n\n{text_to_summarize}"}
])
summary = response['message']['content']
recent_text = "\n".join(f"{role}: {msg}" for role, msg in recent)
context = f"[SUMMARY]: {summary}\n\n[RECENT]:\n{recent_text}"
return context, count_tokens(context)The numbers back this up. At 91 turns, buffer memory cost 985 tokens. Summarization memory, same conversation: 156 tokens. That’s not a marginal improvement — it’s the difference between a system that scales and one that doesn’t.
And it still got the fact right. Asked “what was my budget again?” thirty turns after the fact was stated, the summarized context answered correctly: $3000.
But notice what’s actually happening here. The summary isn’t remembering the fact — it’s re-deriving it from whatever survived compression. If the summarizer LLM decides a detail isn’t important enough to include, it’s gone. Not degraded, not fuzzy — gone. There’s no way to know which turn it came from, or to go back and check.
That’s fine when the compressed detail is a passing preference. It’s not fine when it’s a number someone will hold you to.
Summarization solves cost. It doesn’t solve reliability — it just moves the risk from “will this fit in context” to “will this survive being compressed.”
What if, instead of compressing everything into prose, you could just pull back the exact turn that matters, whenever you need it?
Vector Retrieval Memory: Precision Instead of Compression
Embed every turn. Store it. At query time, embed the question, pull back the top-k most similar turns, and only inject those into the prompt.
T1 → embed → ┐
T2 → embed → ├──► Vector DB
Tn → embed → ┘
▲
│ semantic similarity search
│
query: "what was my budget?"def build_vector_memory(conversation, upto_turn):
collection = chromadb.Client().get_or_create_collection("memory")
for i, (role, msg) in enumerate(conversation[:upto_turn]):
collection.add(documents=[f"{role}: {msg}"], ids=[f"turn_{i}"])
return collection
def retrieve_memory(collection, query, k=3):
results = collection.query(query_texts=[query], n_results=k)
text = "\n".join(results['documents'][0])
return text, count_tokens(text)This is the cheapest strategy by far — 44 tokens, regardless of how long the conversation gets, because it never sends the whole history. It scales in a way buffer and summarization simply can’t.
And on a direct query, it worked perfectly: “$3000 for 10 days.”
So I tried to break it. Same conversation, same fact — but instead of asking “what was my budget,” I asked “when am I free to travel,” referencing a fact stated as “I can only travel during the last two weeks of October because of work.”
VECTOR (paraphrased query): I don't know.It failed. Not because the fact wasn’t there — it was, sitting in the store the whole time. It failed because retrieval matched on the word “travel,” and what it actually pulled back was three duplicate turns about travel insurance:
user: Should I get travel insurance?
user: Should I get travel insurance?
user: Should I get travel insurance?Two things are happening here, and both matter. First — semantic similarity is not the same as relevance. “When am I free to travel” and “should I get travel insurance” share a word, not a meaning, and the retriever can’t tell the difference. Second — redundant content in the store actively crowds out the answer. Three near-identical turns embedded near-identically, and all three outranked the one turn that actually contained the fact.
Vector retrieval is cheap and it’s precise — right up until the query doesn’t phrase itself the way the stored fact did, or the store has noise in it. Then it doesn’t gracefully degrade. It just returns nothing.
The problem isn’t that retrieval is unstructured — it’s that similarity is the only structure it has. What if the memory itself understood relationships between facts, instead of just their surface text?
Graph Memory: Structure Instead of Similarity
Extract entities and relationships from each turn, store them as a graph instead of raw text or embeddings.
has_budget
User ─────────────► $3000
│
│ free_during
└─────────────► last 2 weeks Octdef extract_entities(msg):
prompt = f"""Extract entities and relationships as JSON.
Format: {{"entities": ["e1"], "relations": [["e1", "relation", "e2"]]}}
Entities must be plain strings, not objects.
Message: "{msg}"
JSON:"""
response = ollama.chat(model=MODEL, messages=[{"role": "user", "content": prompt}])
raw = response['message']['content']
start, end = raw.find("{"), raw.rfind("}") + 1
return json.loads(raw[start:end])
def build_graph_memory(conversation, upto_turn):
G = nx.DiGraph()
for role, msg in conversation[:upto_turn]:
if role == "user":
extracted = extract_entities(msg)
for e in extracted.get("entities", []):
G.add_node(to_str(e)) # normalize — models don't always return strings
for s, r, o in extracted.get("relations", []):
G.add_edge(to_str(s), to_str(o), relation=to_str(r))
return GAt 44 tokens — same footprint as vector retrieval — the graph correctly answered the budget question: “According to the context, your budget is $3000.” Because the fact is stored as a discrete node and edge, not buried in prose or dependent on embedding similarity, there’s nothing to paraphrase around. You either have the edge or you don’t.
But getting there wasn’t clean. Extraction runs through an LLM per turn, and the first attempt broke on something basic — the model returned entities as {"name": "Japan"} objects instead of plain strings, and networkx threw TypeError: unhashable type: 'dict' the moment it tried to add a node. Not a reasoning failure. A formatting failure, from a model that mostly followed instructions but not precisely enough.
That’s the real cost of graph memory. It’s not the token count — 44 tokens is nothing. It’s that every turn now depends on an LLM correctly parsing structure out of natural language, consistently, turn after turn, without silently corrupting the schema. Vector retrieval fails loudly — a wrong answer is at least visibly wrong. A malformed graph can fail silently, building an increasingly unreliable structure that still returns something on every query.
Structure fixes the retrieval problem. It just replaces one failure mode with another — extraction reliability instead of semantic recall.
What Production Systems Actually Do
All four strategies above are things you’d hand-roll. Claude and ChatGPT don’t give you a choice between them — they run a version of summarization memory automatically, and the details of how are public.
Anthropic’s own compaction pattern works exactly like the summarization memory above, at production scale: once a conversation crosses a token threshold (150K tokens in the API’s automatic mode), Claude generates a structured summary of everything before that point, then continues from the summary instead of the full history. The compression is not subtle — a real 16-turn story-drafting session, compacted, went from 12,847 tokens to 1,526 tokens: an 88% reduction, and the conversation continued without the user noticing anything had happened.
TRADITIONAL COMPACTION (what you'd build first)
─────────────────────────────────────────────
Turn 1 → Turn 2 → ... → Turn N → CONTEXT FULL!
│
▼
┌──────────────────┐
│ Generate summary │
│ (user waits) │
└──────────────────┘
│
▼
ContinueThe problem with this — same as the toy version — is that it’s reactive. The moment you hit the limit, the summary hasn’t been written yet, and the user sits there waiting. In Anthropic’s cookbook example, that wait was over 40 seconds.
The fix is to stop waiting for the limit and build the summary continuously in the background, so it’s already sitting there ready the instant it’s needed:
INSTANT COMPACTION (what production actually does)
──────────────────────────────────────────────────
Turn 1 → ... → Turn K → ... → Turn N → CONTEXT FULL!
│ │ │
(soft threshold: (periodic (swap in
start building background pre-built
memory) updates) memory — instant)This isn’t a different memory strategy — it’s the same summarization tradeoff, engineered so its one real weakness (the compression step is slow and blocking) doesn’t show up to the user. The other lever production systems pull is prompt caching: because the background summarizer re-sends the same conversation prefix the main chat just used, that prefix is a cache hit, and only the new “summarize this” instruction gets billed at full price — roughly an 80% cost reduction on every background update.
What this doesn’t change is the underlying risk. Compaction is still summarization memory, and summarization memory is still lossy. That’s why Anthropic’s own compaction prompt is explicit about what must survive compression verbatim — exact identifiers, error messages, and especially user corrections — rather than trusting the model to decide what’s important on the fly. The engineering sophistication is in when and how cheaply you compress. It doesn’t remove the fundamental tradeoff — it just hides the latency.
There’s No Winning Strategy, Only the Right Tradeoff
| Strategy | Tokens (91 turns) | Correctness |
|---|---|---|
| Buffer | 985 | Always correct |
| Summarization | 156 | Correct, but lossy |
| Vector retrieval | 44 | Correct on direct query, fails on paraphrase |
| Graph | 44 | Correct, but extraction is fragile |
None of these is strictly better than the others — each one fixes the specific failure of the one before it, and introduces a new one. Buffer is perfectly faithful and doesn’t scale. Summarization scales and quietly loses whatever it decides isn’t important. Vector retrieval is cheap and precise until the query doesn’t match the phrasing of the fact. Graph memory is structured and cheap but only as reliable as the extraction step feeding it.
Production agent memory systems don’t pick one of these — they layer them, and often engineer around the weakest link rather than replacing it, the way compaction hides summarization’s latency without touching its lossiness. The interesting engineering problem was never “which memory strategy is correct.” It’s that memory, like caching, is a hierarchy — and every layer you add to fix one failure mode is buying a different one you now have to design around.