Storage modes and caching

The storage mode controls where an immutable WavesDB reader gets checkpoint bytes. It does not change query semantics or the segment format. The same query over the same generation returns byte-identical packed frames in every mode — a correctness test asserts exactly that, for absent, rare, selective, common, and Boolean query shapes.

Requires a complete, verified local copy of every selected segment before executing against it.

• A verified full-cache hit is pinned and used immediately.
• A miss downloads every object named by the segment checkpoint, verifies it against the commit, and atomically admits the completed directory.
• The generation keeps one already-open remote-capable WavesDB reader. A pinned local overlay redirects that reader's object accesses to the hydrated files; the reader is not reopened once per query.
• The persistent decoded-block cache is disabled on this path. The source is already a complete local copy, and WavesDB's bounded in-memory block cache remains active; writing a second decoded copy to disk would add filesystem work and consume additional disk space.

Appropriate when the complete routed working set fits local disk and the caller accepts hydration latency before the first query. Forcing it can be very expensive for a cold multi-gigabyte segment, because it downloads unrelated families and blocks as well as the ones the query needs.

Opens stable generation-held WavesDB readers directly over immutable object sources. Opening reads table metadata; data blocks are fetched with bounded range requests on demand.

• WavesDB's in-memory table, file, and decoded-block caches retain hot read state for open readers.
• A persistent decoded-block cache beneath cache.dir survives reader and process churn, subject to its byte and entry budget.
• Scan and readahead reads may consult existing entries but bypass admission, so a one-off scan does not evict a selective hot set.
• A missing range capability, or another retryable remote-open or remote-read failure, may fall back to full hydration. Corruption and cancellation do not.

This minimizes cold-read bytes and is normally best for selective queries, especially when the full corpus does not fit local disk.

Decides per segment and reports both the selected and the effective mode in the receipt.

1. A verified full-cache hit executes locally.
2. Otherwise the prepared query supplies predicted request count, byte count, and scan fraction to the policy, together with measured remote latency and remote/hydration throughput.
3. If the policy prefers full hydration but the segment is cold, the interactive request still runs through remote blocks. Full admission is queued on a bounded two-worker, 64-entry background queue.
4. A successful remote-cost decision may also enqueue admission when the whole routed working set is admissible, so a likely duplicate can become local.
5. Hysteresis plus the full-cache byte and entry budgets prevent a small estimate change from oscillating modes.

The per-segment result cache is keyed on segment commit, liveness digest, schema, mode, limit, and request digest — which is why it survives a generation swap and lets a query recompute only the segments that changed. The query embedding cache is keyed by embedding-space fingerprint plus a SHA-256 of the text; raw query text never enters a cache key.

Two block budgets name different tiers, and the names are close enough to confuse:

• cache.block_bytes and cache.block_entries bound the persistent decoded-block cache on local disk. It is consulted only for remote-block execution and survives a process restart.
• cache.read_block_bytes bounds decoded blocks retained in process RAM by WavesDB. cache.read_open_files, cache.read_open_readers, and cache.read_reader_bytes bound the associated file handles plus decoded table indexes and Bloom filters. These serve hydrated and remote readers alike and disappear on restart.

The model cache has its own directory and budget. server.data_dir derives segment cache state under cache/ and model packages under models/, so a large model download cannot evict hot postings, and segment churn cannot unload a model.

For an identical request against an unchanged generation, the final result cache should make ranking a memory hit.

If only some segments and their liveness are unchanged after a catalog swap, the cross-generation per-segment result cache can reuse those ranked prefixes and recompute only the changed segments.

If both result caches miss, the lower layers are still warm: stable readers, WavesDB table metadata, decoded blocks, and optionally full-segment residency.

Result shape changes the request. --ids, --keys, and a --fields query are three distinct requests, because their projection and score requirements differ. Run the same mode on both sides when measuring duplicate latency.

In one measured remote-block fixture, a format-v4 cold top-10 query used 30 range reads and 1,195,784 bytes, while an immediate duplicate reached object storage zero times. Those are fixture numbers — diagnostic, not a corpus-scale latency claim.

Before a full-cache entry is admitted after a process restart, YoloSearch reads its receipt, validates the content-addressed identity, inventories its local tree without following links, and hashes every named immutable checkpoint file. Unexpected, missing, or digest-mismatched files quarantine the entry.

This protects correctness even if the previous process or the local filesystem did not shut down cleanly — but it means a cold restart reads every retained full-cache byte before serving.

Entries are recovered one at a time, so inventory, quarantine, budget, and LRU state stay deterministic. Within an entry, four bounded verification workers hash independent files concurrently with 128 KiB buffers.

The server narrates it:

cache recovery starting index=articles expected=16
cache recovery running index=articles verified=10/16 quarantined=0 read=13038MiB elapsed=33s
cache recovery running index=articles verified=15/16 quarantined=0 read=26073MiB elapsed=1m6s
cache recovery complete index=articles verified=16/16 quarantined=0 read=28565MiB wall=1m13s

The expected count is the active generation's digest inventory. A smaller recovered count at completion means some active segments were not present in the persistent full cache and will use remote blocks or hydrate later. These reads are local filesystem traffic and do not increment object-store counters, so read startup logs separately from object-read totals.

• Keep active physical segment fanout bounded with scheduled compaction. No amount of cache tuning makes an unbounded number of independently planned and scored segments free.
• Size cache.full_bytes for the complete hot routed working set, not for an average segment, and check that cache.full_entries is not the tighter limit.
• Leave local capacity for the persistent block cache and for ingest and compaction scratch, in addition to the full cache.
• Size cache.read_block_bytes for the recurring posting working set, and keep cache.read_open_readers above the number of table readers a normal query touches.
• Sustained misses with nearly one eviction per miss mean the RAM working set is churning. Table evictions, or a resident reader count pinned to its limit, mean index and Bloom metadata is reopening.
• Prefer --ids for large top-K throughput tests. Use --keys only when key projection is part of the workload being measured.

Cache and object traffic are exposed as bounded-cardinality metrics:

yolosearch_storage_wavesdb_read_cache_events_total{cache="block|table",outcome="hit|miss|eviction"}
yolosearch_storage_wavesdb_read_cache_bytes{cache="block|table",state="resident|limit"}
yolosearch_storage_wavesdb_read_cache_entries{cache="block|table|file",state="resident|limit"}
yolosearch_storage_process_object_read_bytes
yolosearch_storage_process_object_write_bytes

The block-entry family carries only state="resident": the decoded-block cache is bounded by bytes, not by an independent entry count.

The two object-byte histograms have a wider scope than query traffic. Their sample count is the request total and their sample sum is transferred bytes, and the counting wrapper encloses the complete serve object store — so totals include ingest publication, catalog access, compaction, and cache fills as well as query reads.

Request-scoped object accounting labels vector traffic as manifest, index, list, rerank, locator, legacy exact, or public_id. A format-v4 ANN query should report zero exact, locator, and public_id object bytes; an immediate repeat with warm caches should report zero bytes in every class.

• Segments and objects — what these caches hold copies of
• The query lifecycle — where the mode decision happens in a request
• Observability — the full metric surface
• Configuration reference — every cache.* key with its scope and default