Architecture

The yolosearch binary runs the engine roles in supported combinations. yolosearch-local combines the single-process server and admin console for local use. Distributed deployments use separate processes for query, ingest, and maintenance roles.

Published data is durable in object storage. Query caches and placement observations can be reconstructed.

  AUTHORITATIVE
  ┌──────────────────────────────────────────────────────────┐
  │  Object storage                                          │
  │    immutable segment prefixes + commit markers           │
  │    immutable catalog generations + `latest` CAS hint     │
  │    immutable schema versions                             │
  └──────────────────────────────────────────────────────────┘
  ───────────────────────────────────────────────────────────── the line
  ADVISORY / DISPOSABLE
     local full-segment caches      worker capacity reports
     decoded-block caches           cache-membership Bloom filters
     WavesDB read caches            hotness sketches
     query-plan and result caches   placement decisions

Cached segment bytes are verified against immutable metadata. Placement observations expire and are checked through worker admission; they are not content-digest-verified data. Stale observations can affect availability or placement quality without changing the pinned segment contents.

yolosearch node --roles accepts the eight engine roles below as a comma-separated list. Unknown or repeated roles and missing prerequisites are rejected at startup.

The admin console runs separately as yolosearch-admin, or within yolosearch-local. It connects to an upstream API target through its bridge.

The router, builder, and publisher roles that split the write path are implemented — routers health-poll and rendezvous-route across fleet.builders, external builders publish immutable commit markers, and the lease-fenced publisher owns catalog mutation — but their comparable mixed-load qualification is still open.

Two combinations matter in practice:

• yolosearch serve is the single-process server: it hosts every index under an object root, accepts schemas and documents over the wire, and follows the catalog as new generations are published. yolosearch-local is that same data plane with the admin UI attached and no flags to type.
• node --roles coordinator,worker,merger,aggregator,compactor is the single-process form of the distributed query path, useful for exercising the real planning and merge code without a fleet.

The dedicated compactor role opens no query engine and no full-segment cache. That is the isolation mechanism for maintenance under sustained query load: run it on separate resources and its input hydration, merge scratch, and output uploads stop contending with foreground ingest and queries.

  push / schema apply
        │  gRPC Ingest (bidi stream)
        v
  ┌───────────┐   spool    ┌───────────┐   immutable segment    ┌──────────────┐
  │  router   │──────────> │  builder  │──────────────────────> │              │
  └───────────┘            └───────────┘   commit marker last   │    Object    │
                                 │                              │   storage    │
                                 │ announce                     │              │
                                 v                              │  (authority) │
                           ┌───────────┐  generation + latest   │              │
                           │ publisher │──────────────────────> │              │
                           └───────────┘                        └──────┬───────┘
                                                                       │
  search ──> coordinator ──┬─> worker ──> local cache ───────────────────┘
        (pins a generation)├─> worker ──> local cache        reads immutable
                           └─> merger tree                   segments + catalog
                                 ^
              aggregator ────────┘  placement hints only, never correctness

Published segments and catalog generations connect the write and read paths. Query workers load the segments named by the pinned generation.

Each index hosted by serve runs a follower. Every follower.poll_interval (2 s by default) it reads the catalog's latest pointer and, on a new generation, builds the engine and swaps it in place. A query that was already running keeps the generation it started on until it finishes; the superseded engine is retired follower.retired_generation_grace (30 s) after its last query releases it.

In the single-process server, the builder nudges the follower after announcing a segment. A flushed push waits for local visibility without waiting for the next scheduled poll.

node resolves its generation before the listener reports serving, so a process that cannot read its catalog fails at startup rather than accepting requests it cannot answer. Recent generations published between follower polls remain servable for a bounded overlap window; old unseen generations are refused.

All application and inter-node communication is gRPC with Protocol Buffers. There is no REST gateway. The only exceptions are operational endpoints — health checks and the Prometheus scrape — and the browser edge of the admin console, which speaks ConnectRPC to a stateless Go bridge that holds a native gRPC upstream.

Search is unary-request / server-streaming. Ingest is bidirectional streaming. Every node also serves an AdminService whose StreamNodeStats answers from that node's own metrics registry, stamped with its identity, role mix, and boot incarnation; StreamFleetStats fans in across statically configured nodes and names an unreachable node in the frame's completeness rather than zero-filling it.

Consensus is reserved for four transitions, all infrequent:

1. electing or fencing the catalog publisher;
2. selecting the next generation where the object store exposes no conditional write;
3. serializing an incompatible schema or physical-format change; and
4. authorizing the final phase of destructive GC or index deletion.

It must never hold worker load, cache membership, hotness, query assignments, result cursors, or per-document operations. Losing the publisher delays visibility; it cannot invalidate a prior generation, and query workers require no catalog quorum to serve.

Placement uses two information paths with different freshness requirements.

Immediate capacity flows on long-lived Advertise streams from each worker to two zone-local aggregators, once per second and on material change. A frame carries admission headroom, per-lane free slots, queue depth, send-blocked time, NVMe bytes, restore bandwidth, supported versions, boot incarnation, a monotonic sequence, and a five-second expiry. Aggregators keep only the newest sequence per incarnation and never merge free-slot counts across observations.

Cache membership and hotness flow as slower expiring summaries: an epoch-rotated Bloom filter over resident segments plus an exact bounded hottest list, and one-minute hotness windows combining a heavy-hitter list with a Count-Min sketch, each attributed to an origin so relaying never double-counts.

Selection takes the union of both aggregators' candidates, orders warm claimants by rendezvous hash for stability, compares the top two by free lane slots, and advances on RESOURCE_EXHAUSTED. The chosen worker performs the authoritative admission check, which closes the observe-then-consume race without consensus.

With no aggregator reachable, selection uses power-of-two choices over a static fallback list. That list is also the dispatch universe: aggregators alone cannot be dialed for work.

A first-party Kubernetes operator owns cluster lifecycle, schema bootstrap, role workloads, scaling inputs, upgrades, and retention-safe deletion. It includes a reader-before-writer compatibility state machine that stages format reader capability before enabling a newer writer format, and a finalizer that fails closed while the object store is unreachable.

Instrumentation uses bounded metric labels. No metric label may contain query text, document IDs, segment IDs, index names, endpoints, object keys, or error strings. Tests populate the registries and check the series count against per-role budgets.

• Segments and objects — what the builder writes
• Catalog and generations — how it becomes visible
• The query lifecycle — what the coordinator and workers do with it
• Kubernetes — running these roles as workloads