Evaluation

RememberStack is design-first and gate-heavy. Evaluation is not a marketing scoreboard first — it is how regressions in truth, retrieval honesty, and agent curriculum are caught.

TL;DR for agents and operators

Public docs state methodology posture, not a live SLA score. Published numbers require pinned dataset, protocol version, and revision.

What “good” means here

1. Correctness over recall theater — projections may lag; they must not serve dead facts as current.
2. Grain honesty — claims never answer “is it true now?”
3. Agent curriculum — a cold harness with only SKILL.md plans correctly.
4. Replayability — decisions, costs, and traces are ledgered where the product requires.
5. Comparable benchmarks — only under a fixed protocol fingerprint.

Design-conformance evals

Acceptance checks in the repository judge the code against the architecture the product documents — not against ad-hoc prompts.

Use these when changing spine, retrieval, or lifecycle behavior: failing a design eval is a product regression even if a demo still “looks fine.”

Cold-agent skill check

Input: rendered deployment SKILL.md + a task.
No: design corpus, repository context, or hidden cheats.

Must demonstrate:

See Mounts for the skill surface.

Golden sets and regression canaries

Examples that ship with the system (not exhaustive):

Treat measured spike numbers as capacity evidence, not promises.

LoCoMo (full-system protocol)

Repository adapter: benchmarks/locomo/.

Posture

• Unshipped as a vendored dataset — you supply pinned locomo10.json after confirming CC BY-NC 4.0 terms.
• Operator-invoked real provider runs; not silent CI spend.
• Current protocol is versioned and fingerprinted in the adapter README and design notes.
• Answer agent can use the complete shipped read plane: assured operations, direct primitives, open SQL/live graph helpers and saved queries, P1, and the ordinary published P3 mount. It has no private database/object-store shortcut.
• Readiness is machine-verified before answer; live graph health and P3 publication are separate capabilities.
• Costs, tool calls, envelopes, and failures are checkpointed.
• Primary metric: judge accuracy; secondary: official LoCoMo F1; diagnostic: session-grain recall from envelope evidence only (not turn Recall@k).

Safe first command (no API spend)

uv sync --extra benchmark
uv run --extra benchmark python -m benchmarks.locomo prepare \
  --dataset /absolute/path/locomo10.json \
  --tier smoke \
  --protocol full-v16 \
  --output .benchmark-runs/locomo-smoke

Do not run remote stages until you have read the adapter README and accepted cost/isolation gates.

V16 also prevents an identity-only lookup from becoming a terminal Unknown: the answer harness requires one ordinary content-bearing testimony, fact, context, primitive, row-returning query, or P3 search/read attempt first. It uses the existing call and cost budgets and records unknown_guard_retries.

Honesty about scores

• Protocol version, model IDs, surface hashes, and dataset pin must match to compare runs.
• A weak reader/harness can tank accuracy with a healthy store — diagnose retrieval vs reader separately.
• Optional Langfuse tracing for answer/judge requires the observability extra and all three Langfuse env vars (Configuration).
• Sharding across hosts is supported for large publication runs; merge via documented summarize flags.

How agents should use this page

What evaluation is not

• Not a substitute for honest product claims on the managed cloud site
• Not automatic continuous LoCoMo spend on every PR
• Not permission to game judges with protocol-specific hacks
• Not a finished public scoreboard until methodology and results are published together

Progressive disclosure

Next

• Troubleshooting
• Configuration
• Project status