Ingestion
Ingestion is the write path: raw inputs become durable evidence, then adjudicated facts, then projection updates. Everything an agent later retrieves was produced here under ledgered, versioned stages.
Why the write path is the product
If extraction is sloppy, retrieval only surfaces garbage faster. RememberStack spends design budget on:
- Immutable testimony — claims never rewrite.
- Grounding — every claim points at a source span / locator.
- Identity — entities resolve to a canonical registry.
- Adjudication — facts supersede and contradict out loud.
- Lifecycle honesty — edits and re-extractions do not inflate confidence.
- Cost control — reprocess proportional to the edit, not the whole corpus.
Read this page top-to-bottom for the map. Stage detail and lifecycle rules live on child pages.
Pipeline map
Diagram shows the full designed chain. The Compose smoke profile starts with Markdown conversion; other MIME routes are registered per deployment.
| Stage | Input | Output | LLM? |
|---|---|---|---|
| E0 ingest | Bytes + metadata | Stored content object, lineage/version rows | No |
| E0 convert | Raw bytes | Markdown representation + source map (+ media sidecars when those routes are registered) | Route-dependent (smoke profile: Markdown passthrough) |
| E0 structure | Markdown | Section tree, roles, summaries, placement | Mostly deterministic; bounded model helpers |
| E1 chunk | Blocks from Markdown | Retrieval chunks + optional context prefixes | No (deterministic) |
| E1 embed | Chunk text (policy-controlled) | P1 vectors / FTS rows | Embedder only |
| E2 claims | Chunks + structure | Immutable claims + grounding | Yes (extraction) |
| E3 normalize | Claims | Entity mentions → canonical entities; candidate facts | Resolution cascade (mostly non-LLM; one binary T4 call for the residue) |
| E3 adjudicate | Candidate facts + evidence | Relations/observations, supersession, contradictions | Cascade + ledgers |
| P1 / graph / P3 | Spine | Search indexes, live PostgreSQL graph, corpus FS | No completion on query; P3 build only |
Deep dive: Pipeline stages.
Entry points (how data gets in)
| Entry | Surface | Notes |
|---|---|---|
| Push upload | POST /ingest (and client SDK/CLI) | Explicit client write; always enters E0 |
| Watched directory | Connector extra | Polls filesystem; debounce + revision no-ops |
| Other connectors | Provider ports | Connector-native source_ref defines lineage identity |
One deployment is one trust domain. Auth (when configured) gates the API perimeter before any write.
Guarantees you can rely on
1. Idempotent by content
Identical bytes share a content object (content_hash). You do not pay conversion twice for the same file appearing in two folders.
2. Lineage ≠ version ≠ bytes
| Concept | Stable when… |
|---|---|
Lineage (doc_id) | Same connector source_ref (Drive file id, message id, …) |
| Version | New observed bytes (or explicit new snapshot) on that lineage |
| Representation | One conversion run’s outputs (ASR upgrade = new representation, same version) |
3. Claims are append-only
Re-extraction creates new claim ids. Old claims remain history. Testimony currency decides what counts as current support — it does not rewrite claim text.
4. Work is ledgered
Each stage runs as tracked work: retries, dead-letter, poison inspection, optional spend ceilings. Reads never enqueue writes.
5. Raw is always reachable
Derived Markdown/transcripts feed the pipeline. Original bytes stay stored; media locators point to page/bbox/time interval for audit.
Lifecycle in one paragraph
Edited watched sources create new versions. Unchanged chunks reuse prior claims and vectors (measured spike hit rates are high). Reconciliation updates current testimony, recounts evidence by distinct lineages, and closes facts only under explicit rules (especially living sources that remove content). Full rules: Lifecycle & versions.
After the spine settles
- P1 updates as claim/chunk/fact-label embeddings land.
- Graph relations are immediately visible after commit; P3 is the only whole-corpus publication (Compose: operations profile).
- K recompiles on debounced evidence change for routed pages.
Agent checklist after ingest
- Wait for stage terminal state (or readiness for target version ids).
- Prefer
fact_context/facts_currentfor “what do we know now?” - Use
testimony_context/ claims for “what sources said.” - Hydrate to sources before high-stakes action.
- If mounts exist, open P3 stubs → artifacts → raw only when needed.
Progressive disclosure
| Depth | Page |
|---|---|
| Map (this page) | Overview + guarantees |
| Stage mechanics | Pipeline stages |
| Versions, currency, counting, deletion | Lifecycle & versions |
| Run it | Getting started · Deployment |
Next
- Pipeline stages
- Lifecycle & versions
- Retrieval — how the write products are consumed