Segments and objects

The segment is the unit of immutability, publication, caching, placement, and compaction. Everything else in YoloSearch is arranged around it.

Physically, a segment is one WavesDB checkpoint: an LSM-derived immutable store with an object-native tier source. It is not one large interleaved data object. WavesDB keeps each column family in its own table and blob files, so a query can address only the physical files it needs.

There are nine families:

An ID-only lexical search does not read the stored-document family. The full document and the lexical index share a segment identity and a commit, but reads address their physical files independently.

For an object root ROOT, index articles, segment SEGMENT, and generation GENERATION, publication uses this shape:

ROOT/indexes/articles/
├── schema/
│   ├── latest
│   └── <version>.binpb
├── segments/SEGMENT/
│   ├── segment.pb                 immutable metadata envelope
│   ├── segment.commit             visibility marker, published last
│   ├── ids.bloom                  public-ID negative-proof sidecar
│   ├── ids.winners                exact ordinal/ID/version sidecar
│   ├── ids.ordinals               format-v5 dense ordinal→public-ID table
│   ├── terms.bloom                per-field term-presence sidecar
│   ├── filters.postings           low-cardinality exact-filter sidecar
│   └── wavesdb/
│       ├── MANIFEST
│       └── ...                    WavesDB table/blob checkpoint objects
├── catalogs/
│   ├── latest                     current generation hint
│   ├── generations/GENERATION.pb
│   └── liveness/GENERATION.bin
├── compactions/<job>.pb
├── retired/<segment>
└── gc/...

Local disk beneath cache.dir is disposable. Losing it may make the next request cold, but it cannot lose an indexed document or make an unpublished segment visible.

A builder constructs a segment on ephemeral disk, uploads a consistent WavesDB checkpoint plus its routing and identity sidecars into a unique object prefix, verifies them, and writes segment.commit last.

The marker carries segment and index IDs, a build nonce, document count and key range, the object prefix, SHA-256 digests and byte sizes for the metadata envelope and the WavesDB manifest, aggregate checkpoint size and object count, and every relevant version: format, schema, analyzer, vector codebook, scorer, and score bound.

Object keys are never overwritten during normal publication, so a retried build either verifies the same bytes or fails as a collision. This is what makes WavesDB's per-family atomic bulk ingestion sufficient: cross-family atomicity is not needed, because catalog visibility is the cross-family transaction.

A segment never changes after commit. Compaction writes a new segment and a new generation and retires its inputs; it does not modify a visible segment in place.

A document has three different names, and confusing them is the most common source of surprise about query cost.

The public ID is a deterministic 128-bit identity used in result frames and tie ordering. It is what --ids prints, as 32 lowercase hex characters. Postings themselves use compact segment-local ordinals; format-v5 readers bind those ordinals to the ids.ordinals sidecar, legacy readers to document-value blocks.

The logical key is the schema field marked key. It defines upsert identity and lives in each stored record's header, occupying no field ID — which is why a schema's fields may start at 1. Its value is hashed with xxh3-128 into the 128-bit document ID the engine uses. It is sent in a document frame only when a projection is requested.

Stored fields are the source values eligible for projection. The whole containing stored block must be fetched and decoded before the requested record and fields can be selected.

This is why --keys costs more than --ids: the logical key is not derivable from the public ID, so key projection additionally reads stored-record and mutation-version blocks.

For --ids output, the wire result is essentially 16 bytes per hit plus small framing — no scores and no document frames. The server still has to identify and rank those hits:

1. term metadata is read to resolve the query;
2. matching posting blocks are read and scored;
3. position blocks are read only for phrase or proximity verification;
4. format-v5 postings supply exact indexed-field lengths, and the reader's in-memory ordinal table supplies public IDs;
5. no stored block and no mutation-version lane is requested.

Exact dictionary resolution is batched per segment. The planner sorts every unique (field_id, term) key the selection requires — including exclusions — and submits the set through one WavesDB multi-get. That uses one immutable snapshot, shares a data-block read when several terms land together, and overlaps independent remote block reads. A single-term query keeps the ordinary point-read path.

Format v5 moved exact indexed-field lengths into the posting row and introduced a dense ids.ordinals sidecar of exactly:

16 + document_count × 16 bytes

The fixed 16 bytes are magic, version, count, and checksum framing; every document contributes one 128-bit public ID. The sidecar is fetched and CRC-validated once when a remote segment reader enters a serving generation, decoded into a dense in-memory slice, and shared by every query using that reader. For a 9,850,954-row article corpus that is about 150.3 MiB of persistent ID data across the whole generation — targeted metadata residency, not admission into the general block cache.

The trade is explicit: four extra bytes per posting row, repeated when a document contains several queryable terms, in exchange for removing a corpus-wide document-value lane from the scoring path. Cold query work becomes proportional to terms and matching posting blocks rather than to every ordinal block preceding a candidate.

Formats v1 through v4 remain readable and use their legacy ordinal document-value path. Older formats have real limits: against a format-1 segment, projection returns no values (and the header's projection_unavailable_segments names the segment), filters are refused with FAILED_PRECONDITION, and a phrase is refused naming the segment, because there is no positions lane.

Format v5 retains the format-v4 vector layout: an independent manifest and centroid/codebook directory per vector field. IVF-PQ reads only the probed list blocks, then reranks survivors from exact float32 rows stored beside each IVF list. Those rerank rows carry their own public IDs, so an approximate-nearest- neighbor search reads neither the ordinal docvalue lane nor the stored-document lane.

That placement is the point. The earlier v3 layout stored exact vectors ordinal-aligned, so scattered candidates touched many unrelated blocks. In v4 and v5, rerank fan-out is proportional to the selected list blocks.

Multiple vector fields stay independent in dimension, embedding fingerprint, block geometry, and query fan-out while sharing one segment commit. Fields backed by different models do not share a vector space, and YoloSearch never silently falls back to a different one.

Four sidecars are eagerly loaded before a generation is exposed:

• terms.bloom conservatively proves a segment cannot contain a term;
• filters.postings supplies exact postings for useful low-cardinality filters;
• generation liveness bitmaps remove superseded document versions during selection;
• ids.bloom and ids.winners mainly accelerate ingest publication and liveness construction rather than ordinary query projection.

Every Bloom filter here is one-sided: a negative can skip work, a positive must retain the exact path. None is a correctness authority.

• Catalog and generations — how a committed segment becomes visible
• Storage modes and caching — where a reader gets these bytes from
• Schemas — how field options decide which lanes a document lands in