Pipeline stages

Exhaustive stage reference for the per-document evidence chain. Parent overview: Ingestion.

Stage graph

ingest → convert → structure → chunk → embed(chunks)
                                      ↓
                              extract claims → embed(claims)
                                      ↓
                              normalize / resolve entities
                                      ↓
                              adjudicate facts (supersession, contradictions)
                                      ↓
                              reconcile (currency, evidence counts)
                                      ↓
                         P1 updates · live graph · (scheduled) P3 · K triggers

All stages are idempotent with respect to their versioned inputs. Workers use the deployment work ledger: retry, then dead-letter for systemic failures (provider outage, database errors, unrecoverable handler bugs). That is not the same as a re-derivable soft drop inside a successful stage: a predicate outside the governed vocabulary is preserved through the other: escape when possible, otherwise it is dropped with structured telemetry and does not dead-letter the document version. Entity types and domain/range signature gates are not part of E3.

E0 — Files

Ingest

Job: accept bytes, compute content_hash, attach connector identity, create or advance lineage/version.

Rules:

• Content-identical payloads reuse the content object.
• Lineage identity is connector-native (source_kind, source_ref) — not title, path, or hash.
• source_modified_at feeds claim assertion time; cursor-only polls must not rewrite it casually.

Outputs: raw object URI, document/version rows, enqueue convert.

Convert

Job: produce an immutable representation: clean Markdown + source map (+ media sidecars when composed).

Shipped smoke profile: registers text/markdown → Markdown passthrough. Unregistered MIME types fail as non-retryable unroutable input.

Designed / composable routes (register on the deployment; do not assume the Compose smoke stack has them):

Rules:

• Conversion is not regenerated in place when tools improve: a new run = new representation_id beside the old.
• Media claims inherit derivation kind and evidence mode (source_expression | model_observation | model_interpretation) for honest read-side disclosure when media routes are used.
• Raw originals remain the byte authority; Markdown is the navigation-promoted form.

Structure

Job: per-document section tree (PageIndex-class): headings, roles, spans, summaries, placement hints.

Rules:

• Prefer deterministic heading/skeleton construction.
• Bounded model seats for role/summary/sanity/fallback only — not unbounded freeform rewrite of the whole doc.
• Structure feeds selection for claim extraction and P3 navigation stubs.

E1 — Chunks

Blockizer

Job: deterministic sequence of paragraph-grain blocks from document.md.

Why blocks exist: section trees snap to a stable grid; chunk packing becomes editable without reshuffling the world.

Chunk packer

Job: pack whole blocks into non-overlapping, section-bounded, anchor-stabilized chunks.

Properties:

• Re-ingest after a small edit reuses unchanged chunk work (with lifecycle cache keys).
• Context prefixes (headers, titles) are generated for embedding/readability but stored separately from source body under the embedding-input policy.
• Chunks are evidence grain source text — never atomic claims, never adjudicated facts.

Chunk embedding (P1)

Job: write vector + FTS rows for chunks under the active embedder generation.

Policy notes (shipped):

• P1 text column holds normalized body (not silent header+body mash that poisons audit).
• Location headers return separately when present.
• FTS index bootstrap and write-path optimization are automatic; reads must not full-scan the corpus.

E2 — Claims

Extraction

Job: decompose chunks into atomic, verifiable natural-language assertions.

Each claim carries:

• Immutable text identity (assertion-by-a-source)
• Grounding to character spans (and through them to page/region/time locators)
• Source-asserted validity interval when the source states one (asserted validity — testimony, not system verdict)
• Extractor / representation / structure generation coordinates (basis)

Gates:

• Deterministic grounding checks reject unanchored inventions.
• Decision ledgers record extraction outcomes for replay and eval.

Claim embedding (P1)

Job: index claim text for semantic + lexical nomination. Default evidence search uses both channels (not double semantic).

Currency (bookkeeping, not validity)

When a new extraction basis completes, prior claims for that basis become non-current testimony via an append-only transitions ledger. Claim rows themselves stay immutable. See Lifecycle.

E3 — Facts

Entity resolution

Job: map surface forms to canonical entity ids (write path).

Write-time cascade tiers:

Query-time resolve_entity is T0 exact today — see Concepts and Primitives. Merges are reversible with transcripts. Only canonical ids flow downstream.

Write-time T0 never accepts an exact name by itself. T4 prefers a compatible existing entity: missing overlap and different topics do not establish a new referent. new requires positive distinction from every supplied candidate; it may mint a second active entity with the same normalized lemma and stores exclusions only for those supplied candidates. Candidate truncation remains visible in the audit but does not create a third decision state. Lemmas that span multiple entity ids are marked promiscuous and rank below unguarded fuzzy/phonetic matches without being dropped from the candidate set. The shipped cascade has no exact-name auto-accept flag.

After supported open-ended observations or relations change, a deterministic refresher rebuilds the entity’s short summary and entity-profile-v2 vector. Unchanged input hashes skip embedding. Supersession, merge/un-merge, terminal human review, normal deletion/reconciliation, and hard-forget use the same refresh seam; if no supported fact remains, every profile cache field clears together. A survivor profile includes evidence from its complete redirect closure. A merged member keeps a separately attested member-local profile for joint neighborhood re-decision, but remains excluded from public entity resolution and search. During the profile-policy cut, the self-host image's setup entrypoint clears name-only vectors, rebuilds active and merged entity profiles in bounded resumable pages, and publishes entity semantic search only after that pass completes.

Current profile publication invokes bounded local clustering over the touched entity's alias lemmas. Identical merge-review proposals deduplicate by their live roots and cluster configuration. The default remains fail-closed: auto_merge_enabled=false, so unattended convergence produces reviewable diagnostics rather than an uncalibrated silent merge.

Bare head nouns (game, app, system, card, photo, module, the system) are not minted as entities. Qualified names (FIFA 23, James's Unity strategy game) may mint. When the claim spelling differs from the canonical name (App vs Application), both surfaces are stored as aliases on the same id. Entities have no type class: extract emits names only. works_for between two people is allowed.

Predicate governance

Relations use a governed predicate vocabulary with escape values and promotion — not free-text edge soup.

Relations vs observations

Both are bi-temporal facts with evidence links and supersession.

Evidence links

Many-to-many fact_claim_evidence with stance supports | contradicts. Counts aggregate by distinct current-testimony lineages.

Supersession & contradiction

• Supersession closes valid_until / invalidates belief per shape rules.
• Contradictions group live disagreements; both sides remain.
• Every adjudication is a logged, replayable decision.

Observation shapes (retract semantics)

Reconciliation

Runs when basis or version changes:

1. Flip testimony currency for affected claims.
2. Recount evidence / contradict counts.
3. Close sole-supported facts under living removal rules.
4. Flag support-withdrawn cases from toolchain re-derivation for review.
5. Emit change signals for K / projections consumers.

Projection touchpoints

Failure modes & operations

Operator surfaces: deployment CLI, ledger queries, rebuild drills. See Deployment.

Next

• Lifecycle & versions
• Retrieval
• Architecture