Answer yes if it gives the same essential answer in meaning, allowing equivalent wording, date formats and reasonable inferences for questions asking what is likely. Answer no if required information is missing, contradicted, or the response abstains despite an answer in the reference. When the reference lists required items, all are required.Strict judge prompt
BENCHMARKS
Graded by a strict judge.
On LoCoMo, PRME gets credit for an answer only when the answer is complete. We score every question, keep every failure, and publish every prompt. Many published memory scores come from a judge that is told to “be generous.” On our own saved answers, that judge prompt adds 5.7 points.
- LoCoMo
- 81.2%
- 1,250 of 1,540 questions, strict judge
- LongMemEval-S
- 90.6%
- 453 of 500 questions, official judge prompts
- Memory per question
- 3,996
- Tokens at most, for every answer
PRME retrieval defaults of September 25, 2026 (prme@d811e3ed), with deepseek-v4.1-flash:cloud as reader and judge, with each turn written by store(). 95% intervals: 79.2% to 83.1% and 88.0% to 93.0%. Every question was answered and judged. With extraction through ingest(), the same budget gives 77.3% and 88.0% (below). A GPT-5.4 run on the earlier v0.12.0 defaults was removed from this page on October 16, 2026, because it was never rerun on the current defaults; the research record keeps it.
MEASURED, NOT ASSUMED
A generous judge adds points.
We graded the same 1,540 saved LoCoMo answers twice: once with our strict judge, and once with Mem0’s judge prompt, copied word for word. Same answers, same judge model. Only the prompt changed.
Show the numbers as a table
| LoCoMo category | Questions | Strict judge | “Be generous” judge | Change |
|---|---|---|---|---|
| Multi-hop | 282 | 99 (35.1%) | 145 (51.4%) | +16.3 points |
| Temporal | 321 | 229 (71.3%) | 243 (75.7%) | +4.4 points |
| Open-domain | 96 | 41 (42.7%) | 46 (47.9%) | +5.2 points |
| Single-hop | 841 | 638 (75.9%) | 661 (78.6%) | +2.7 points |
| All questions | 1,540 | 1,007 (65.4%) | 1,095 (71.1%) | +5.7 points |
- +5.7 pointsoverall, from 65.4% to 71.1%.
- +16.3 pointson multi-hop questions, where the reference lists several items.
- 91 answersaccepted only by the lenient judge, against 3 accepted only by the strict one.
Run prme@46647825 of September 24, 2026, with deepseek-v4.1-flash. It predates the September 25 defaults. About 1% of verdicts change between identical runs of the same judge, so a few of the 91 may be run-to-run variation.
THE JUDGE DECIDES THE SCORE
Same answer. Two verdicts.
Another model, the judge, grades each benchmark answer. What the judge is told to accept can move a score as much as the memory system does.
The generated answer might be much longer, but you should be generous with your grading - as long as it touches on the same topic as the gold answer, it should be counted as CORRECT.Mem0 evaluation code Hindsight benchmark harness
- Question
- What has Melanie painted?
- Reference
- Horse, sunset, sunrise
- Answer, shortened
- Landscapes and still life, a sunflower painting, a sunset with a palm tree, a sunset with a pink sky, and animals (for example, a horse).
- Strict judge: wrongThe reference needs a sunrise, and the answer has none.
- “Be generous” judge: correctIts reason: the answer “includes horse, sunset, and sunrise (implied by sunsets).”
One answer from run prme@46647825 of September 24, 2026. Both verdicts are recorded for it.
TWO WAYS TO WRITE MEMORY
Extraction scores lower. Here is why.
store() keeps each turn as written. ingest() also has a model extract claims, which entities and automatic corrections build on. We measure both, with the same reader, judge, and 3,996-token budget.
- Memory written with
- LoCoMo
- LongMemEval-S
store()Each turn as written- 81.2%
- 90.6%
ingest()Each turn plus its extracted claims- 77.3%
- 88.0%
- Extraction adds records and removes none. Claims sit beside the turns they came from, so retrieval ranks about 930 records per LoCoMo question instead of about 555.
- The claims compete for the same room. They often restate turns retrieval already finds. With them, 81.6% of LoCoMo questions keep all their evidence in context, against 84.0% with turns alone.
- Batching closes that gap, but is not scored yet. The v0.15.0 default extracts a session’s turns in batches. It reaches 84.0% on the same check with 95% fewer model calls. It has no answer run yet.
DeepSeek runs prme@d811e3ed (September 25, 2026) and prme@3389586b (October 6, 2026), each with the defaults of its date, answered separately, so they are not a paired comparison. The evidence figures are from the offline gate, which needs no model.
THE PUBLISHED NUMBERS
Read every score with its method.
Vendor numbers are what each vendor publishes, not paired runs. The answer model, the judge, and the amount of context all differ, so compare the columns, not only the scores.
| System | LoCoMo | LongMemEval-S | Answer model | LoCoMo judge | Memory context per question |
|---|---|---|---|---|---|
| PRMEstore(), defaults of 2026-09-25 | 81.2% | 90.6% | deepseek-v4.1-flash | StrictStrict, all listed items required. Prompt published. | At most 3,996 tokens |
| PRMEingest() with extraction, defaults of 2026-10-06 | 77.3% | 88.0% | deepseek-v4.1-flash | StrictStrict, all listed items required. Prompt published. | At most 3,996 tokens |
| ZepVendor-reported | 94.7% | 90.2% | GPT-5.4, medium reasoning | Not publishedGPT-5.4 with chain-of-thought grading. Prompt not published. | Median 5,760 and 4,408 tokens |
| HindsightVendor-reported | 92.0% | 94.6% | Gemini 3.1 Pro preview | Be generousGemini 2.5 Flash Lite, told to “be generous”. | Mean 36,235 and 43,625 tokens |
| Mem0Vendor-reported, managed platform | 92.5% | 94.4% | Not named | Be generousNot named for these results. Mem0’s published LoCoMo evaluation code tells its judge to “be generous”. | Mean 6,956 and 6,787 tokens |
Vendor values checked October 16, 2026. Context figures are LoCoMo first, then LongMemEval. Hindsight’s models are from our benchmark audit of September 23, 2026. Mem0 states that its scores come from its managed platform.
A ceiling below 94%
Independent audits found that 99 of LoCoMo’s 1,540 reference answers (6.4%) are wrong. A perfect system would score about 93.6% under a judge that follows the references. Zep reports 94.7%. The same audits found that a “be generous” judge accepted 62.8% of deliberately vague wrong answers.
locomo-audit Penfield LabsUp to ten times the context
PRME answers from at most 3,996 tokens of memory. Hindsight’s reader sees about 36,000 to 44,000 tokens per question. More context makes a question easier to answer and costs more to run.
Level on LongMemEval-S
With the official LongMemEval judge prompts, PRME scores 90.6% from at most 3,996 tokens. Zep reports 90.2% from a median of 4,408. The readers differ, so this is not a matched comparison.
WHEN A USER CORRECTS THEMSELVES
Explicit corrections work. Spoken ones, not yet.
A small public test: one scripted chat with a changed plan, a wrong fact fixed later, and two conflicting statements. We ran it through ingest(), and again with the app calling supersede() for each correction.
| Case | ingest(), v0.15.0 defaults | App calls supersede() |
|---|---|---|
| Changed planMoving to Austin, then “We’re moving to Denver instead.” | Not applied. Austin and Denver both stay active; the context shows Denver first. | Austin retired and kept in history. The context shows only Denver. |
| Wrong fact fixedJo’s birthday is March 3, then “It’s actually March 13.” | Not applied. Both dates stay active; the context shows March 3 first. | March 3 retired and kept in history. The context shows only March 13. |
| Conflict in one sessionBlack coffee, then oat milk in coffee, minutes apart. | Both kept, not marked as conflicting. | Both kept, not marked as conflicting. |
ingest() marked both new claims as updates but did not name the claim each replaces, which automatic supersedence needs. Until that changes, call supersede() (or its HTTP and MCP equivalents) when a correction must hold. The test also found two claims dated a year early and one saved as its negation. It is one chat, so it shows behavior, not a rate. See the full report and the chat.
HOW EVERY RUN WORKS
Rules we hold ourselves to.
- 01
Every question
All 1,540 LoCoMo questions in categories 1 to 4 and all 500 LongMemEval-S questions. The 446 adversarial LoCoMo questions are excluded, as in Zep’s published scope.
- 02
No answer leaks
Each turn is stored with its speaker, date, and session. No question, answer, evidence label, or summary enters memory.
- 03
One retrieval, bounded context
Each question makes one public retrieve() call with the product defaults. The reader sees at most 3,996 tokens of memory.
- 04
Yes or no
The judge answers yes or no, with no partial credit. LongMemEval-S uses its official judge prompts.
- 05
Failures count
A truncated answer or an unreadable verdict is scored wrong and kept. Retries are only for transient server errors and never replace a result.
- 06
Checksummed records
Each result file records the prompts, model identity, settings, and code commit, and binds every answer by checksum.