If you only have a minute, here's what you need to know.
- Most enterprise AI agents forget everything when a session ends. Every conversation starts at zero, no matter how much context was built up before.
- Throwing a bigger context window at this problem makes it worse, not better. Research shows performance degrades as context grows through a phenomenon called "context rot."
- Agent memory has four distinct types: working, episodic, semantic, and procedural. Most organizations only address the first one.
- RAG is not memory. Confusing them is one of the most common and costly architectural mistakes in enterprise AI right now.
- Memory systems are where compliance risk hides. GDPR's right to erasure is structurally at odds with how most vector stores work, and most teams don't discover this until it's too late.
- The agents that accumulate knowledge over time compound their value. Stateless agents do not. That gap becomes a competitive moat.
Think about the last time you hired someone exceptional. In week one, they were sharp but still learning. By week twelve, they knew your clients, your quirks, your processes. They remembered what you tried last quarter and why it didn't work. That accumulated knowledge is what made them genuinely valuable.
Now imagine if they forgot everything every night.
That is exactly what most enterprise AI agents do today. Every session begins from scratch. Users re-explain context they've already covered. Agents make recommendations that contradict decisions from two weeks ago. Support interactions restart at zero even when the customer called yesterday with the same problem.
We've spent enormous energy making AI agents smarter. We've spent almost none making them remember.
That is starting to change, and the organizations that get this right in the next twelve to eighteen months will have built something competitors cannot easily replicate.
What "memory" actually means for an agent
When most people talk about AI memory, they mean the context window — the working space where the current conversation lives. That is one type of memory. It is not the only one, and it is the least interesting one for enterprise use cases.
There are four distinct layers:
Working Memory
The active context window. Everything in the current session: the conversation, tool outputs, instructions, reasoning. Fast, but volatile. When the session closes, it's gone.
Episodic Memory
The record of what happened before. What did this user ask last week? What approach worked for that type of request? This is the layer that turns a stateless tool into something that learns.
Semantic Memory
Structured facts. Customer preferences, account attributes, domain knowledge, organizational context. This is the layer that makes an agent feel like it actually knows your business rather than just knowing things in general.
Procedural Memory
Knowing how to act, not just what to know. Workflows, escalation rules, resolution patterns. The most advanced frameworks now let agents rewrite their own operating instructions based on experience.
The four types of agent memory. Most enterprise deployments only address the first.
Most enterprise AI deployments today operate exclusively in working memory. The other three layers are either absent or cobbled together with workarounds. That is the root cause of the gap between what agents promise and what they deliver.
The context window is not the answer
When teams run into the memory problem, the instinct is to make the context window bigger. Use a model with a million tokens. Dump everything in. Problem solved.
It isn't.
Chroma's 2025 "Context Rot" research found that LLM performance degrades non-uniformly as input length grows, even well within the stated limits. The specific failure mode is called the "Lost in the Middle" effect: models attend strongly to the beginning and end of a context window and systematically underweight what's in the middle. Put a critical fact in the center of a long context and the model will behave as though it doesn't exist.
Anthropic's own guidance on agent context engineering puts this directly: the goal is "the smallest possible set of high-signal tokens that maximize the likelihood of the desired outcome." Not the largest context. The smallest right one.
This inverts the intuition most people have. Bigger is not better. More is not more. The discipline is curation, not accumulation. And organizations optimizing for cost by running smaller models with larger contexts are compounding both problems at once — smaller models are especially sensitive to irrelevant content flooding their context.
Context engineering is the new data engineering. The teams that understand this distinction will build agents that consistently outperform competitors running technically superior models.
RAG is not memory
This is the single most important distinction to get right, and the one most often blurred in enterprise AI conversations.
RAG retrieves documents. It is very good at finding relevant content from a corpus you've assembled. It does not know that a particular user prefers brief summaries over detailed analysis. It does not know that the last three support tickets from this account escalated because of a billing dispute. It does not know that the technical approach agreed upon in Tuesday's session was rejected by the security team on Thursday.
RAG retrieves what you already knew. Memory records what the agent has learned.
The practical failure looks like this: a team builds a well-governed RAG corpus, deploys an agent on top of it, and calls it intelligent. The agent retrieves documents accurately. It has no record of what happened with this customer in this context over this history. Every conversation starts at zero regardless of how much has come before.
This isn't a flaw in RAG — it is simply not what RAG is for. The mistake is architectural, not technical.
The four ways to implement memory
There is no single "memory solution." Enterprise memory architecture is a stack of choices, each suited to a different part of the problem.
In-Context Compression
Before the context window fills, compress the conversation history into a structured summary and carry that forward. No persistence across sessions, some loss of precision, but it costs almost nothing to implement. If your agents are doing nothing about memory today, start here.
Vector Database Retrieval
The current default for semantic and episodic memory. Embed past interactions and facts into dense vectors, retrieve relevant ones at query time. Fast at scale, straightforward to integrate — but weak at temporal queries and carries GDPR exposure most teams discover late.
Temporal Knowledge Graphs
Timestamps on every stored fact let the agent reason about what was true at a given point in time. The right architecture for financial services, legal workflows, or anything where "when was this true?" matters as much as "what is true now?" Higher complexity, but for those use cases nothing else comes close.
Managed Memory Services
Mem0 (AWS's default Agent SDK memory provider) and Amazon Bedrock AgentCore Memory treat memory as managed infrastructure. You get production-grade compression, retrieval, and cross-session continuity without spending months building the plumbing yourself.
Four memory implementation patterns compared across cost, complexity, temporal awareness, compliance fit, and use case.
Most mature production systems end up combining these layers. RAG for document retrieval. Vector storage for learned preferences and history. A managed service for compression and session continuity. Knowledge graphs when temporal accuracy or compliance auditing demands it. These aren't competing options — they're complementary tools for different parts of the same problem.
The compliance issue hiding in plain sight
47% of agentic AI systems fail compliance audits due to missing consent mechanisms. Another 39% lack any defined retention policy.
— IAPP Agentic AI Audit Survey, 2025
Memory systems accumulate personal data by design. That is the entire point. But most teams build the memory layer without thinking through the compliance architecture until something forces the conversation.
GDPR's right to erasure creates a structural problem for vector stores that most developers don't anticipate. When you delete a source record, the embedding generated from it may persist in your index. The personal data is technically still there, encoded in high-dimensional space. Purging it requires explicit engineering — it doesn't happen automatically when you delete the source.
The EU AI Act adds another layer for high-risk systems: HR tools, credit decisions, critical infrastructure. These require durable, searchable audit trails of what the agent retrieved, what it decided, and what data it touched. That is not the same requirement as "log the outputs." It requires co-designing your memory architecture and your audit architecture from the start.
Fine-tuning is not a memory strategy. Baking interaction history into model weights creates write-only memory with no access control, no retrieval precision, and no ability to delete specific records. It cannot satisfy GDPR's right to erasure. If someone in your organization is proposing fine-tuning as a way to make agents "remember" things, flag this before it becomes a legal problem.
What to do this week
Map your agents against the amnesia problem. For every agent in production or active pilot, answer one question: what does this agent actually remember across sessions? If the answer is "nothing," that is your baseline. Estimate the real cost — how often do users re-explain context, how many interactions fail because the agent has no history.
Separate your RAG and memory budgets. They solve different problems and should be treated as separate investments. Your RAG corpus is about what you already know. Your memory layer is about what your agents learn. If you have one and not the other, you now know which gap matters more right now.
Design retention policy before you deploy. Define what gets stored, for how long, and how it gets purged. Verify that your chosen vector store actually supports the deletion semantics GDPR requires. Do this before legal asks, not after.
Start with a managed service, not a custom build. The instinct is to build. Resist it. Managed memory services give you production-grade infrastructure in days. Use the time you save to focus on what the agent should actually learn, not on building storage plumbing.
The agents that remember compound their value with every interaction. The agents that don't start over every day. That difference, multiplied across every customer conversation, every support ticket, every internal workflow, is not a feature gap. It is a strategic one.
Build the memory layer now, while it is still a competitive advantage and not a catch-up requirement.
Matthew Kruczek is Managing Director at EY, leading Microsoft domain initiatives within Digital Engineering. Connect with Matthew on LinkedIn to discuss agentic memory architecture for your enterprise AI program.
References
- Mem0. "State of AI Agent Memory 2026." mem0.ai
- Chroma Research. "Context Rot." 2025. research.trychroma.com
- Anthropic Engineering. "Effective Context Engineering for AI Agents." anthropic.com
- arXiv. "Memory in the Age of AI Agents." December 2024. arxiv.org
- arXiv. "Mem0: Building Production-Ready AI Agents with Long-Term Memory." 2025. arxiv.org
- AWS Machine Learning Blog. "Amazon Bedrock AgentCore Memory." 2025. aws.amazon.com
- IAPP. "Engineering GDPR Compliance in the Age of Agentic AI." iapp.org
- Gartner. Agentic AI Production Deployment Analysis. 2025.
