Sizing

Size each role for its workload. Cache budgets determine local disk and RAM use; query mix, segment count, ingest volume, and object-store latency also affect CPU, network traffic, and request rates.

Three numbers have to be consistent with each other, and the operator refuses the pod if they are not:

roleOverrides:
  worker:
    resources:
      limits: {ephemeral-storage: 48Gi}
    cacheVolume: {sizeLimit: 32Gi}

defaultConfig:
  cache.full_bytes: 8GiB
  cache.block_bytes: 2GiB

• cache.full_bytes + cache.block_bytes must fit inside cacheVolume.sizeLimit.
• cacheVolume.sizeLimit must not exceed resources.limits.ephemeral-storage.

A cache volume larger than the pod's ephemeral-storage limit can let cache growth trigger eviction before the volume reaches its own limit. A medium: Memory cache volume is charged against the memory limit instead.

Leave real headroom between the cache budgets and the volume. The ingest spool, compaction scratch, and /tmp all live on the same node's storage.

Four budgets bound four different things. Two are on disk and survive a restart; two are in RAM and do not.

The two block-cache budgets govern different tiers:

• cache.block_bytes bounds the persistent decoded-block cache on local disk. It is consulted only for remote-block execution and survives a process restart.
• cache.read_block_bytes bounds decoded blocks retained in process RAM by WavesDB. It serves hydrated and remote readers alike and disappears on restart.

All of these are startup settings. Changing one needs a restart.

Include these caches when accounting for process memory. The query embedding cache is configurable; the other budgets listed here are fixed in code:

The per-segment result cache is process-lived and keyed by segment commit, so it survives generation swaps; the final result and stored projection caches are generation-local and are discarded with their generation.

embedding.model_cache_bytes defaults to 10 GiB with model_cache_packages: 8, in its own directory (embedding.model_cache_dir, or models/ under server.data_dir). Its separate budget prevents model downloads from evicting segment-cache entries.

If you use internal embedding, that 10 GiB is 10 GiB of disk on top of the cache budgets. If you do not, the directory stays empty.

The guidance the storage documentation gives, in order of impact:

1. Size cache.full_bytes for the complete hot routed working set, not for an average segment, and make sure cache.full_entries is not the tighter limit. 1,024 entries at 8 GiB is 8 MiB per entry on average; a fleet with large compacted segments hits the byte budget first, a fleet with many small segments hits the entry budget first.
2. Leave capacity for the block cache and for ingest and compaction scratch in addition to the full cache. They are not carved out of it.
3. Keep active physical segment fanout bounded with scheduled compaction. Cache tuning cannot make an unbounded number of independently planned and scored segments free.

cache.read_block_bytes should cover the recurring posting working set, and cache.read_open_readers should stay above the number of table readers a normal query touches. Two signals say you got it wrong:

Both come out of the same family:

yolosearch_storage_wavesdb_read_cache_events_total{cache="block",outcome="eviction"}
/
yolosearch_storage_wavesdb_read_cache_events_total{cache="block",outcome="miss"}

yolosearch_storage_wavesdb_read_cache_bytes{cache,state="resident|limit"} and ..._entries{cache="block|table|file",state} give occupancy against the configured limits directly. The block-entry family has only state="resident": the decoded-block cache is bounded by bytes, not by an independent entry count.

Roles have different resource profiles. The operator sample includes worker resource requests; size the other roles for their expected work as well.

A dedicated compactor opens no query engine and no full-segment cache, so it does not need the worker's cache budgets at all.

builder memory ≈ builder.build_concurrency × builder.sort_memory_bytes

builder.sort_memory_bytes defaults to 256 MiB and is the total accounted budget for one build — one ledger every part of a build charges against. It divides into the public-ID sorter (an eighth, capped at 64 MiB), the docvalue and lane spools, the analysis share (builder.analysis_memory_bytes), and the term partitions, which take what is left.

builder.build_concurrency defaults to one per CPU, bounded at twelve. On a 16-core builder that is 12 × 256 MiB = 3 GiB of builder budget before anything else. A build that cannot hold its term partitions in the remainder spills to disk instead, which is the difference between one file per build and hundreds.

builder.max_queued_builds defaults to four times build_concurrency. A queued build is a spool on disk, not memory, so it costs ingest.seal_bytes of disk per queued build and buys ingest room across a build.

server.max_receive_bytes and server.max_send_bytes both default to 4 MiB, which is why one record may not exceed an ingest batch: a batch carrying it could not be delivered.

Watch saturation as a ratio, which is exactly what the shipped YolosearchRouterSaturated rule does:

yolosearch_ingest_queued_bytes{role=~".*router.*"}
/
yolosearch_ingest_queue_capacity_bytes{role=~".*router.*"}

The storage mode decides where an immutable reader gets checkpoint bytes. It never changes query semantics or the segment format.

AUTO is the default. It can hydrate cold segments in the background:

1. A verified full-cache hit executes locally.
2. Otherwise the prepared query supplies predicted request count, byte count, and scan fraction, together with measured remote latency and throughput.
3. If the policy prefers full hydration but the segment is cold, the interactive request still runs through remote blocks. Full admission is queued on a bounded two-worker, 64-entry background queue.
4. Hysteresis and the full-cache budgets prevent a small estimate change from oscillating modes.

Five settings feed the decision, and every one of them is runtime-changeable with config set:

Watch the decision, not the outcome:

sum by (requested_mode, policy_reason) (
  rate(yolosearch_storage_decisions_total[5m])
)

Inspect policy_reason alongside the ratio of hydrated to remote requests to understand why the storage policy changed its choices.

Segment fanout is a capacity input, not only a tidiness concern. Every active segment is planned and scored independently.

Run compaction on a dedicated compactor role. It follows latest and performs single-flight copy-on-write physical merges under these budgets, and it opens no query engine or full-segment cache, so it does not compete with workers for cache.

GC is off by default and has two modes.

yolosearch catalog gc-dry-run proposes unreferenced objects against explicit horizons and removes nothing. Run it and read the proposal before enabling sweeps.

Retention horizons protect uploads and readers: gc.minimum_upload_age of 24 hours exists because a multipart upload in flight looks exactly like an unreferenced object, and gc.maximum_stream_lifetime exists because a long relevance-ordered stream may still be reading a generation that latest has moved past.

For a fleet serving a corpus that does not fit local disk, with moderate ingest:

spec:
  defaultConfig:
    storage.mode: AUTO
    cache.full_bytes: 8GiB
    cache.block_bytes: 4GiB
    cache.read_block_bytes: 1GiB
    compaction.enabled: "true"
  roleOverrides:
    worker:
      resources:
        requests: {cpu: "4", memory: 12Gi}
        limits: {memory: 24Gi, ephemeral-storage: 64Gi}
      cacheVolume: {sizeLimit: 40Gi}
    builder:
      resources:
        requests: {cpu: "4", memory: 8Gi}
        limits: {memory: 16Gi, ephemeral-storage: 32Gi}
    compactor:
      resources:
        requests: {cpu: "2", memory: 8Gi}
        limits: {memory: 16Gi, ephemeral-storage: 32Gi}

Then measure, in this order: the storage-decision reasons, the WavesDB block-cache miss and eviction rates, the full-cache occupancy against its budget, and publication lag. Move one budget at a time.

• Observability — the series these recommendations read
• Configuration reference — every setting, with scope and default
• Storage modes and caching — the mechanism behind the budgets