Catalog and generations

A committed segment becomes searchable when a catalog generation includes it. Each generation identifies the active segment set for queries that pin it.

A catalog generation is an immutable protobuf naming its parent generation and the complete active segment set — or a bounded delta chain plus a periodic full checkpoint — with digests and pruning statistics.

One active publisher creates a linear lineage. It uploads the immutable generation object first and updates catalogs/latest second, using an object-store conditional write where the backend exposes one, and Kubernetes leader election plus parent validation otherwise.

latest is a discovery hint, not the authority. Readers validate the target and can fall back to the highest valid descendant they already know.

  gen-A ──> gen-B ──> gen-C ──> gen-D
                                  ^
                          catalogs/latest  (a hint; validated on read)

If a control-plane partition produces conflicting children of the same parent, they resolve deterministically: the lowest valid generation ID among those children wins. The losing lineage's committed segments are republished on the winner and a degraded condition is recorded. No document is deleted by this repair — the segments were already immutable and already committed.

Catalog publication serializes the choice of immutable segments in the next generation. Losing the publisher delays visibility; it cannot invalidate a prior generation, and query workers need no catalog quorum to serve.

push ──> spool ──> seal ──> build ──> upload checkpoint ──> segment.commit
                                                                 │
                                                                 v
                                                        publisher announces
                                                                 │
                                          immutable generation object written
                                                                 │
                                              catalogs/latest updated (CAS)
                                                                 │
                                                   follower polls / is nudged
                                                                 │
                                                      engine built and swapped
                                                                 │
                                                       next search sees it

A spool seals when it reaches ingest.seal_bytes, or ingest.seal_documents (100,000), or when its first document is ingest.seal_age (30 s) old, or on an explicit flush, or when a batch arrives under a newer schema version — a build is built under exactly one schema — or on restart, which seals whatever was open and queues everything already sealed.

The default push carries a flush on its last batch, so it returns only once the segment is published and served by that process. --no-flush hands the documents to the age sealer and returns at once. Neither mode is read-your-write across nodes.

A crash between publish and announce is replayed at startup: a commit marker found is re-announced, partial objects are abandoned to the GC sweep, and the build is re-minted under a fresh ID.

A query pins exactly one generation for its lifetime. This is what makes a long-running result stream coherent: the segment set it planned against cannot change underneath it, and the segments themselves are immutable.

serve follows the catalog and swaps engines as generations publish; a query already running keeps the generation it started on, and the superseded engine is retired follower.retired_generation_grace (30 s) after its last reference is released. node instead resolves and pins its generation before the listener reports serving.

Generation engines are reference-counted. Each ranked execution and each stored-projection call holds its engine while it uses the reader.

Workers also keep an age-based generation window. If a coordinator names a generation the worker has not yet observed through latest, the worker performs one single-flighted resolve by generation ID, and builds that exact engine only when created_unix_millis + follower.generation_overlap is still in the future. That engine does not replace the follower's current engine and does not reconcile the cache inventory; it stays addressable until the overlap expires and its reference count reaches zero. Old unseen generations are refused.

A coordinator may refresh and replan once when such a refusal arrives before its merge frontier has emitted anything. It never replans after emission.

A document's identity is its logical key, hashed into the 128-bit public ID. Pushing the same key twice produces the same public ID. Both immutable records coexist until compaction.

Each catalog generation carries a winner-only liveness object. Query selection excludes the superseded ordinal before top-K, so results contain one live version per key and still return a full requested prefix.

Every document also carries a mutation version: a caller-supplied string stored verbatim — versions compare as bytes, so zero-pad if you want numeric order — or one the server mints when the document has none. A minted version is 26 characters of Crockford base32 encoding a millisecond timestamp, the node ID, and random bits, monotonic within a process. The ingest path always mints, so a served index never has an absent version.

Ordering between copies of a key is: greatest mutation version wins, ties broken by commit time, then by segment ID.

yolosearch delete writes a tombstone: a document with deleted = true, carrying no fields and no payload, sent over the same Ingest stream push uses. It competes for the key exactly as an ordinary version would. A winning tombstone additionally clears its own ordinal — which an ordinary winning version never does — so the key has no live candidate rather than resolving to an empty document.

Two properties follow, and both matter:

A winning tombstone excludes the key from queries using the new generation's liveness metadata. Existing queries keep their pinned generation. Compaction can omit deleted records from its output, but retired segment objects remain until garbage collection can reclaim them under the retention rules.

A higher-version write can restore a deleted key. A later write with a greater version supersedes the tombstone, exactly as it would supersede an ordinary earlier version. There is no separate permanently-deleted state and no setting that changes this. A caller that needs a key to stay gone must not re-ingest it with a version that would win.

Deleting a key that does not exist is not an error: a tombstone that never had anything to supersede never becomes a candidate.

Every applied schema is canonicalized (fields sorted by ID, analyzer unicode-simple-v1), hashed over its canonical encoding, and stored immutably as indexes/<index>/schema/<n>.binpb with a schema/latest pointer. An index exists exactly when it has a schema/latest.

Changes are additive and auto-versioned. Adding a field, adjusting weight or b, or turning stored on mints the next version. Removing or renaming a field, reusing an ID, changing a type, changing indexed, filterable, repeated, the key, the message, or the analyzer, or turning stored off, is refused with the diff.

The reason is structural: every segment keeps the schema it was built under, and the engine resolves names per segment. A generation may therefore mix schema-1 and schema-2 segments, and a stored field a segment predates is reported unavailable from that segment rather than failing the search — without changing the reported ranking exactness.

Compaction writes a new segment and a new generation and retires its inputs. It never modifies a visible segment.

A pass leaves a recent delta tier alone, groups similarly sized segments, writes replacement objects copy-on-write, and publishes a new catalog generation. The dedicated compactor role follows latest, runs a size-tiered pass immediately, and schedules another bounded pass immediately whenever it completed real work; a no-op or a failure returns it to compaction.interval.

The all-in-one scheduler inside serve is different: it is off unless compaction.enabled is set, waits for two consecutive idle-builder observations, and never lets fan-out pressure override foreground ingest. It also stays idle while the build queue is non-empty, because compaction's input hydration, merge scratch, and output upload contend with all three foreground resources.

Compaction reduces the number of segments planned and scored per query. The engine repository's docs/query-performance-audit-2026-08-27.md records these historical examples:

These measurements apply to their recorded workloads. Schedule compaction to control active segment count, and measure the effect on your query mix.

Reclamation is two-phase and quarantined, and it is off by default.

1. Write an immutable dry-run proposal record naming the unreferenced objects. This removes nothing.
2. Wait at least gc.quarantine_age.
3. Re-resolve latest and re-mark reachability from scratch.
4. Intersect the two exact orphan sets.
5. Write an immutable delete intent.
6. Issue per-key idempotent deletes, then write an immutable completion record.

gc.sweep_enabled defaults to false; enabling it is an explicit authorization to delete. The retention, stream-lifetime, grace, and minimum-upload-age horizons are re-read for every pass, and the sweep uses the more conservative value from the proposal and the current configuration.

Retention keeps everything referenced by the current generation, by retained historical generations, by generations newer than the maximum stream lifetime plus grace, and by active compaction or restore records. The practical effect: a 24-hour result stream cannot have its inputs deleted underneath it.

• Storage modes and caching — how a generation's segments are read
• The query lifecycle — what pinning a generation means for a request
• Ingest — the spool, seal, and flush semantics in operational detail