Research analysis · Sources checked November 7, 2026
An agent that remembers an old deployment rule can be more troublesome than one that asks again. For an engineering team, the useful question is whether saved context helps the next task while remaining correctable. A small acceptance test can make that question concrete.
What Anthropic reported
Anthropic’s November 6 agent-memory study concerns persistent context, not weight updates. On 25 questions about Stainless documents, search alone achieved 53% of graded criteria; search plus a developed wiki reached 61%, versus 48% for the wiki alone. Another legal-document task showed no clear wiki benefit. Preparation required substantial computation. These vendor-run, task-specific evaluations do not establish general production gains or net savings. We have not independently replicated them.
Research and product documentation answer different questions
The separate Memory tool documentation describes a client-side interface: Claude requests file operations, while the application executes them against storage it controls. That implementation description is not a performance guarantee or a new feature announcement from the study.
For a deployment review, ask two owners for evidence: the application owner should explain how information is stored and isolated; the evaluation owner should show whether the stored information improves the intended work. A product name or an enabled setting cannot answer both questions.
A proposed correction test
The following is an original engineering worksheet, not a test we ran against Claude or another service. Start with synthetic documents in a non-production environment. Keep customer records, credentials and private conversations out of the fixture.
- Define one repeated job. For example, ask an assistant to explain the approved deployment procedure for a fictional service. Write the expected answer and required citations before running the agent. Include questions that need one document and questions that need several, so a good average cannot hide one weak category.
- Establish a fair comparison. Use the same model version, source documents, tools and task budget for a search-only run and a memory-enabled run. Repeat the cases and preserve unsuccessful attempts. Keep the final test questions separate from the examples used to tune the memory-writing instructions.
- Change an authoritative fact. In the fixture, replace “deploy on Tuesday” with a dated “deploy on Thursday” rule. Retain the old note deliberately. Ask the agent again and score whether it finds the revision, explains the conflict and updates or retires the stale note. Then start a fresh session and repeat the question.
- Test the boundary around a note. Put an instruction-like sentence inside a synthetic source document, such as a request to skip an approval step. Check that storing or summarizing it does not turn source content into permission. Separately test that one fictional user cannot retrieve another user’s notes.
- Check removal and recovery. Remove one synthetic fact through the application’s supported process. Verify what remains in the active store, derived summaries and retrieval results. Record any retention limits rather than assuming a missing search result proves deletion everywhere.
Count the whole operating cost
Record answer quality, stale-answer frequency, latency, model usage and tool calls for each configuration. Include the work of constructing notes, refreshing them and handling failed updates. Report these costs over a stated number of tasks. A cheaper answer after expensive preparation may still be useful, but the expected reuse must justify it.
Choose the pass criteria before seeing the results. One reasonable starting point is no cross-user disclosure in the fixture, correct handling of every deliberate policy change, and a measured quality improvement within the team’s budget. A small test cannot certify security or predict all future behavior; it can expose a design that already fails its basic contract.
The deliverable should fit on one page: tested configuration, document versions, result by case, total measured cost, known failures and an owner for the next review. That gives the next engineer something reproducible to inspect before memory becomes part of a production workflow.