Troubleshooting

Operator runbook for a single self-host deployment. Prefer machine-verifiable endpoints and CLI over guessing from logs alone.

Decision tree (start here)

Health and readiness

curl --fail http://localhost:8000/healthz
curl --fail http://localhost:8000/operations

GET /operations must list exactly four assured ops: resolve_entity, testimony_context, fact_context, and answer_context.

curl --fail -X POST http://localhost:8000/readiness \
  -H 'content-type: application/json' \
  -d '{"version_ids":["VERSION_UUID"],"require":{"pipeline":true,"p1":true,"live_graph":true,"p3":false}}'

The four capability keys are exhaustive. Use p3=false for ordinary live recall and p3=true only when the published CorpusFS tree is required.

Readiness also echoes non-secret model bindings for the API process (configuration review, not per-document provenance).

Human fallback:

docker compose exec postgres psql -U rememberstack -d rememberstack -c \
  "SELECT stage, status FROM processing_state ORDER BY enqueued_at DESC LIMIT 40"

Ingestion stuck or failed

1. Unroutable MIME

Smoke profile routes Markdown. PDF/HTML/audio without a registered converter → non-retryable failure for that version.

Fix: ingest text/markdown, or compose additional conversion routes.

2. Missing / bad provider key

Extraction and embeddings need a live provider. Bad credentials surface as stage failures / DLQ after retries.

Fix: set REMEMBERSTACK_OPENROUTER_API_KEY; re-run or remember ops replay for dead-lettered rows only.

3. Stage in retry / dead_letter

remember ops inspect --deployment <deployment-uuid>

Needs rememberstack[server] + REMEMBERSTACK_DATABASE_URL.

Replay one DLQ row:

remember ops replay <processing-uuid> \
  --deployment <deployment-uuid> \
  --attempts 1

Does not wipe prior tracebacks; only grants additional attempts. Non-DLQ replay fails closed.

4. Budget parked (not failed)

If REMEMBERSTACK_WORK_BUDGETS ceilings are set, exhaustion parks work.

remember budget inspect --deployment <deployment-uuid>

Parked work resumes when the window rolls — it is not a poison DLQ.

Empty or weak retrieval

Wrong SQL pattern (claim windows as truth) is documented in Open query — start from facts_current.

Connection refused / wrong port

If you change REMEMBERSTACK_SELFHOST_API_PORT, also set REMEMBERSTACK_API_URL. Clients default to http://127.0.0.1:8000 independently of the listen port.

P3 publication and live graph health

docker compose --profile operations run --rm projections

Then request readiness with p3=true. This job builds P3 only; graph reads do not wait for it.

Drill rebuilds (server extra):

remember ops rebuild --deployment <uuid> --snapshot-root ./snapshots --version drill-…

Same production builders and validation path — not a second recovery stack.

If only the live-graph catalog/readiness check fails after a logical restore, inspect and replay metadata without touching tenant rows:

remember ops graph-catalog ensure

Observability not appearing

Empty env = exporters stay off. Ledger state still lives in Postgres.

Agent grain mistakes (look like “bugs”)

Destructive reset (dev only)

docker compose down --volumes

Erases PostgreSQL, MinIO, and the forget-manifest volume for the disposable stack. Not a production restore path.

Progressive disclosure

Next

• Configuration
• Deployment
• Ingestion pipeline