Other Formats
The Corpus Oracle
Structural observation and insight paths over a deterministic corpus chain
Abstract
The corpus the engine publishes becomes observable. A task enters at a coordinate and the oracle returns the insight path that connects it to the next step, with every hop evidenced and every response bound to one corpus snapshot. This document states the domain position, the current state, the observation contract, the canonical chain model, the path dimensions, and the milestones.
Position
The engine layer is open. It validates, heals, renders, and extracts the machine-readable corpus, and it declares what the current corpus is. The observation layer reads the same snapshots and returns insight paths instead of match lists. The two layers stay separate over one record model, and the corpus produced by the engine is their shared entry point.
| Role | Responsibility |
|---|---|
| Corpus producer | Validates, heals, and extracts the machine-readable corpus from source documents |
| Structural addressing | Gives every document and fact a structural coordinate |
| Materialization | Lands the coordinate space on media: file today, signal, network, and GPU later |
| Runtime | Holds facts, accumulates verified records, and enables graph traversal and gap detection |
| Client and edge | Carries the spatial state engine to any device with offline self-custody |
The structural addressing layer is the tagma coordinate space, the materialization layer is chton, and the runtime is neXus. The client and edge layer implements the same state space on portable hardware.
State
- The machine-readable corpus is produced on every build of the reference suite, is published, and is publicly fetchable. Verified.
- A fast authoring loop previews single documents and collects published artifacts between full builds. Verified.
- The ingestion half of the pipeline is exercised end to end: the corpus is fetched, pages are stored as facts, and coordinate and semantic indexes are rebuilt. Verified as existing capability.
- The service surface for normal users is a known gap. Verified as absent.
- Refresh is manual; scheduled and diff-based refresh is absent. Verified as absent.
- Storage routing through the materialization layer, and client and edge integration, are designed but built later. Projected.
Observation
The response A(q, σ) to a request q over a snapshot σ is a path, and the oracle contract fixes four properties:
- Determinism: A(q, σ) reproduces identically on every run.
- Provenance: every hop carries (document, section, cell, build).
- Freshness: every hop is bound to the build identity of σ.
- Gap: when evidence(q, σ) is empty, A returns an open intent.
An open intent is a first-class result. The absence of a path is the next step: the gap marks the observation to perform or the fact to deposit.
The canonical chain
A corpus of any size serializes into one ordered chain of cells, L = (c1, …, cn). Each cell carries a coordinate, a content fingerprint, and, under a pinned model, a vector. Content is stored once: a document is a coordinate interval over cells, and shared units are referenced rather than duplicated. The ordering rule is canonical, so merging corpora is commutative.
A snapshot is a manifest plus a chain position, σ = (M, k). The manifest M hashes the ordered fingerprints together with the model and segmentation pins. Time travel reconstructs the state at position p as the prefix L[1..p]. Every projection is a deterministic function of (L, σ) and is idempotent by construction.
Path dimensions
Six dimensions feed path weaving. Each contributes one kind of hop.
| Dimension | Hop |
|---|---|
| Structure | Document and section coordinates from chain intervals |
| Meaning | Similarity over cell vectors under a pinned model |
| Time | Snapshot positions and change history between builds |
| Provenance | Source file, commit, and build identity per hop |
| Relation | Validated cross-reference edges |
| Status | Validity, supersession, and contradiction candidates |
A path crosses dimensions. A structural hop finds the section that defines a term, a meaning hop finds the document that develops it, a temporal hop finds the build that changed it, a relation hop finds the specification that constrains it, and a status hop marks it superseded when a newer source exists. Every hop stays traceable to its cells and snapshot, which makes the response auditable by construction.
A path that ends at a concrete directive or a validated fact proposes it as the next action. A path with empty evidence returns an open intent, and the loop continues: agents deposit facts, the structure thickens, and later observations find richer paths.
Semantics under the pin
Meaning is the only dimension that depends on an external model today. The snapshot contract keeps it deterministic: the model version and the segmentation rule are part of the manifest, and embeddings are accepted only under that pin. A structural vector space, where the vector itself lives at a coordinate, is the planned replacement for this dimension.
Decisions
- The chain is the primitive source; every index is a derived projection.
- Content is stored once; documents are coordinate intervals over cells.
- Determinism is carried by the snapshot contract: manifest, order rule, model pin, and segmentation rule.
- The engine and the observation layer stay separate over one record model.
- The oracle is defined by the four properties of the contract.
Open
- Cell granularity: token, sentence, or segment, uniform or adaptive.
- Projection cadence: per snapshot or lazily on request.
- Refresh strategy: scheduled full sync versus diff-based incremental ingestion.
- Storage routing: adoption of the materialization layer at Phase 2 or earlier.
- Semantic introduction: when embeddings enter under the pin, and what the structural vector space must provide to replace them.
- Status detection: contradiction and supersession beyond count-based gap heuristics.
- Path semantics: what makes a hop valid across dimensions and what counts as a terminal next step.
- Service surface, distribution, and commercial boundary: desktop application, hosted endpoint, or both; package identity and the open-core boundary.
Milestones
| Phase | Deliverable | Evidence |
|---|---|---|
| 1 | Snapshot contract and corpus service | Manifest byte-identical across independent rebuilds; corpus reflects changes without manual intervention |
| 2 | Chain, projections, and storage routing | Time travel returns the exact state at a past build; materialization layer holds the knowledge store |
| 3 | Observation surface and client | A task returns an evidenced path or an open intent under the pin; the client surfaces it offline with multi-device sync |
| 4 | Scale and governance | A corpus beyond the reference suite; provenance and audit across corpora and teams |
Development starts at Phase 1. A specification for the snapshot contract will follow this document.
References
- SDBS index (index.qmd in this directory)
- Single-path artifact pipeline observation (ci_observations.qmd in this directory)