Documentation
Search over immutable objects
YoloSearch combines lexical and vector search with stateless query workers, object storage, and streamed results.
Immutable storage
Published segments live in object storage. Query workers read them on demand and retain local caches that can be rebuilt from the stored objects.
Segments and objectsRanked result streams
Lexical queries produce an exact ranked prefix. Vector queries report whether retrieval was exhaustive or limited to candidates. Results stream as packed document IDs with optional scores and fields.
Ranking and exactnessWorker placement
Aggregators collect worker capacity and cache observations. Coordinators use those hints to assign work, and fall back to configured workers when aggregators are unavailable.
ArchitectureYoloSearch stores documents in immutable segments and uses a catalog to select the segments visible to each query. Query workers read those segments from object storage and cache data locally. Workers can serve any assigned segment; they have no permanent shard ownership.
Published data lives in object storage. Each segment is a WavesDB checkpoint published with a commit marker written last. A catalog generation records the active segments. Query workers can reconstruct their caches from these objects. Unpublished ingest data has separate durability limits; see ingest acknowledgments.
Queries use a fixed generation. A coordinator plans segment work and sends it to available workers. Placement uses capacity and cache observations to choose workers, with a configured fallback list when aggregators are unavailable.
Results report their exactness. Lexical search uses score bounds to produce an exact ranked prefix. Approximate vector retrieval reports exact ordering within its candidate set. An optional lexical tail can use approximate ordering with a reported error bound. See ranking and exactness.
Results stream over gRPC. Search returns packed document IDs and optional
scores and stored fields. The current server supports up to 100,000 results
per request, subject to its configured limit. HTTP/2 flow control propagates
backpressure from the client through the merge tree.
The fastest way to see it working is against the public FineWeb corpora, with
no checkout, no build, and no credentials. Two are published: fineweb-200,
a ~200 GB build holding 72 million documents, and a smaller ~100 GB build
holding 42 million. The demo container serves fineweb-200, and that is what
these commands open:
brew tap yannick/tap
brew install ysearch-demo
ysearch-demoTry it has the exact sizes, what to query, and how to serve the smaller corpus beside it.
To run your own corpus instead, build and start the local server:
just build
./bin/yolosearch-localIn a second terminal, apply the sample schema, load documents, and search:
./bin/yolosearch schema apply deploy/swarm/articles.proto
./bin/yolosearch push articles deploy/swarm/articles.jsonl
./bin/yolosearch search articles 'title:search' --fields title,urlThe quick start covers prerequisites, expected output, and cleanup. Local mode stores its objects on the filesystem and includes the admin console.
The engine is written in Go and uses gRPC and Protocol Buffers for client and inter-node APIs. A Kubernetes operator manages role workloads, upgrades, and retention-aware deletion. The admin console supports queries, schema management, document uploads, and supported index lifecycle operations through a Go bridge.
Lucene-style and CQP queries lower through a shared abstract syntax tree. Equivalent canonical trees share a digest; each grammar supports its own set of syntax forms.
- Try it — query the public FineWeb corpora, nothing to build
- Quick start — build, load, and search locally
- Why YoloSearch — intended workloads and current limits
- Architecture — components, storage, and coordination