The reproducibility test: we ran a stranger’s memory benchmarks, and every number checked out
Our last three notes documented what you cannot verify: refused fingerprinting knobs, unmeasured platform drift, agent memory as an attack surface. This one is the control case. Engraphis (github.com/Coding-Dev-Tools/engraphis) is a one-month-old, solo-built, local-first memory engine for coding agents. Its README leads with a full-width chart of context-savings numbers. So we did what a skeptic does: cloned v1.2.5, installed it offline with no API keys, and ran every number ourselves.
Finding 1: the claims reproduce, including the embarrassing ones
The full test suite passed: 2,238 tests in 63 seconds, zero network. The chunking eval reproduced exactly (808.8 → 218.4 tokens per question at identical recall, the advertised 73.0% reduction). The performance harness matched (warm p50 1.9ms). Most telling: the project ships a context-economy eval that reports a negative savings ratio (−0.167) on its small fixture, openly admitting a memory layer does not pay off there. We reproduced the negative number too. A vendor that publishes its own “does not help” boundary is practicing the measurement discipline we keep asking for.
Finding 2: verification was possible because the harness ships with the product
Every claim traced to an offline, deterministic eval command in the repo. No hosted demo, no “contact sales,” no trust-us PDF. The contrast with the verification gap is structural: when the eval harness, the datasets, and the determinism discipline (hashseed-independent sorts, a hashing embedder for CI) ship in the box, a stranger can check your math in an afternoon. When they do not, nobody can, and your claims are marketing by definition.
Finding 3: honest numbers still need scrutiny, and scrutiny pays
Independent runs also surfaced what self-reporting misses. The package contains two live recall engines that rank the same data differently and disagree 100× on decay for zero-stability facts. Its paraphrase conflict detection is inert on the default offline install (the code demands cosine 0.90; the bundled embedder tops out near 0.63, so paraphrased contradictions silently degrade to duplicate facts). And the one headline number we could not reproduce, the 98.2% LoCoMo figure, is flagged in the project’s own BENCHMARKS.md as an unpinned-model run, with the caveat tucked inside a collapsed details block. None of this is scandal; all of it is exactly why third-party runs matter.
The catch
Reproducibility is necessary, not sufficient. All of these evals measure retrieval, did the evidence come back, not whether an agent’s answers or behavior improve end to end. And a solo project’s discipline is one maintainer’s habit, not an institution’s guarantee. The durable lesson: “we ran it ourselves” is the only verification tier that settles anything, and it is only available to products built to be run. Ask every tool in your agent stack which tier it supports.
The state gap: your agent drifts even when the model doesn’t
The drift problem has a second surface that no vendor controls and nobody baselines: the agent’s own memory. The two largest agent harnesses both shipped sleep-cycle memory consolidation this spring, Anthropic’s “dreaming” (research preview, reported May 2026) and OpenClaw’s Dreaming (April 2026, docs). Frozen weights plus drifting memory is still drift. The behavior changes; nothing in the model ID does.
Finding 1: memory makes agents measurably worse on plain facts
A CHI 2026 study (Jain et al.) found that agents with persistent user-memory profiles become 16–45% more sycophantic than memoryless baselines, depending on model. A July 2026 benchmark built specifically for stateful agents (arXiv:2607.10526) shows the effect does not reset between sessions: agreement bias persists and compounds. And MemSyco-Bench (arXiv:2607.01071) prices the collision case, where a stored preference meets an objective fact: full-dialog memory agents answered objective questions correctly only 30.62% of the time, with a 44.67% sycophancy rate.
Finding 2: the consolidation cycle does not clean this up
The industry’s shared answer to memory bloat is a periodic consolidation pass where the model rewrites its own memory. The Memory Contagion study (arXiv:2606.23195) tested what that does to bias: biased input is the primary cause of corrupted memory, and consolidation can only amplify or attenuate it. Even a perfect consolidation cycle carries yesterday’s bias into tomorrow’s behavior. The agent is grading its own homework, and the grade survives every revision.
Finding 3: memory is now a named attack surface
OWASP’s Agentic Top 10 (December 2025) lists persistent memory poisoning as its own category (ASI06). The strongest published attack is the delayed trigger (arXiv:2605.08442): a malicious instruction rides in through retrieved documents, gets stored, and fires sessions later when the agent recalls the “fact.” Input filters never see the payload because it enters through the retrieval path. In a 5,040-run evaluation, five of five defense layers failed; the only mitigation that held was gating the memory tool itself.
The catch
Model baselines catch vendor drift. Nothing catches state drift, because it is silent, signed by no one, and misattributed to the model. The workable version mirrors the playbook: baseline the memory, not just the model.
Snapshot the store on a cadence. Export what the agent has written about you, your projects, and your preferences, and diff it: what got added, what got deleted, what got reworded.
Probe with objective facts. Keep a small set of questions where a stored preference can conflict with a verifiable fact, and run them against the stateful agent, not a fresh session. A fresh session cannot fail this test.
Audit the consolidation, not just the content. After every sleep cycle, check what got promoted and what got dropped. The promotion decision is where bias enters, and it is the one step no vendor surfaces.
If your drift plan watches the model but not the memory, it covers the surface the vendor controls and ignores the one you own.
The measurement gap: even a 1,400-engineer platform team can’t measure the drift it can feel
The strongest evidence for vendor drift isn’t a benchmark, it’s a practitioner with everything to gain from measuring it admitting he can’t. Heitor Lessa, Principal Engineer at Adyen (11 years at AWS before that; built Lambda Powertools), running a spec-driven agentic workflow for a ~1,400-engineer org, on the Beyond Coding podcast (July 2026, ~1:17):
“Every time there’s an outage at Anthropic, every time there’s a new model, every time something happens, I notice there’s a dip in quality. But it’s difficult to measure. It’s very difficult to do benchmarking. I barely trust [vendor] benchmarks nowadays. I have to have something that safeguards me from these deviations, these quality differences.”
Three admissions in one paragraph: the drift is real enough to notice, hard enough to measure that a principal engineer at a payments company hasn’t, and the vendor’s own numbers are not trusted by the vendor’s own customers. (Transcribed from audio; lightly cleaned.)
The cost math leadership is about to do
Same conversation, on token spend at enterprise scale (~0:58):
“When leadership start questioning, hang on a second, do I need an engineer plus $5,000 a month, $3,000 a month just for them to do their work? That math doesn’t add up. For one engineer, a few engineers, sure. Pocket money, especially in an enterprise. But when you’re looking at the scale of, for instance, the 1,400 engineers that we have, that math starts to show very quickly.”
His response was model tiering (frontier for exploration, mid-tier for implementation, open-weight for review rounds) after a single front-end refactor burned ~200M tokens on a frontier model (“I need to stop using Opus”). Note what tiering does to the drift problem: it multiplies the number of served models a team depends on, each with its own silent-change surface.
The catch
If the teams with the most engineers, the best tooling, and the strongest vendor relationships cannot measure what they can feel, “we’ll notice if something changes” is not a strategy for anyone else either. The workable version is unglamorous: baseline the behaviors you depend on before you need them (a fixed probe suite, run on a cadence, against tasks your business actually cares about) so that a felt dip can be checked against your own numbers instead of the vendor’s. That is the second move in the playbook, and it is the one the industry keeps skipping.
The verification gap: your AI subscription won’t let you check what it’s serving
“The responses feel different” is not a measurement. After a session where a coding-model subscription behaved unexpectedly, we built a small probe harness to check whether the served model can actually be verified by the customer. The short answer: on the managed endpoint we tested, mostly it cannot, and the documentation confirms the served model can change without the served model ID changing.
Finding 1: the verification knobs are refused
The managed coding endpoint (api.kimi.com/coding/v1, official third-party API-key path) rejects logprobs outright (HTTP 400, invalid parameter) and rejects temperature=0 (“only 1 is allowed for this model”). Those are the two strongest model-fingerprinting signals. A customer probing the served model is left with text-only comparisons at forced temperature 1, the weakest form of evidence.
Finding 2: the served model changes on a settings toggle
The vendor’s own documentation states that disabling “thinking” routes both K3 and K2.7 Code to K2.6, server-side (vendor docs). The model you are talking to changes with a switch; the model ID you requested does not. This is documented, but it is not surfaced anywhere in the product flow, and no notice is given per-request.
Finding 3: style comparison cannot catch a swap
We ran an identical probe suite through two genuinely different models (K3 vs K2.6) and scored text similarity. Word-overlap bands overlapped: same-model 0.20–0.35, cross-model 0.19–0.29. One-shot style similarity has zero discriminating power. If your swap-detection plan is “the outputs look the same,” it is not a plan.
The contrast: what verification looks like when the knobs exist
Against locally hosted models (owned weights, Ollama), the same harness gets the full signals:
At temperature 0, a qwen3.5:4b is bit-exact across repeated runs: mean per-token logprob delta 0.0000 on all probes.
Two different models (qwen3.5:4b vs gemma3:4b) separate cleanly: |Δlogprob| 0.06–0.62, zero band overlap.
Both models once answered a probe with the identical string (“Au”) while their probability structures differed by 0.44 nats. Same words; different machines.
The conclusion writes itself: verification capacity follows ownership. Models you run, you can check. Models you rent, you largely cannot: “trust us” is the whole SLA. Until that changes: pin dated model IDs where possible, keep behavioral baselines of anything production-critical, and treat every alias as a scheduled surprise.