Ingest

Ingest is a bidirectional gRPC stream. The client converts and validates documents against the index schema, streams them in batches, and the server acknowledges each batch. Behind the stream, a builder spools documents to disk, seals a build, builds a segment, publishes its objects, and announces the new generation to the catalog. This page follows that path from the command line down.

yolosearch push <index> <files...|-> [--batch 500] [--batch-bytes 2097152] \
  [--format auto|jsonl|json] [--no-flush] [--require-ack]

yolosearch push articles docs.jsonl
# pushed 1204 documents in 10 batches · sealed segment 9f3a1c2b… · generation 7d1e0a4c… · PUBLISHED in 1.8s

push fetches the index schema first — an index with no schema fails here — then reads every file, converts and validates each document client-side, and streams batches of --batch documents or --batch-bytes bytes. A batch never spans files. Validation errors name <file>:<line>. Validation happens during streaming, so earlier batches may already have reached the server when a later record fails.

Input is JSONL, or a JSON array when the first byte is [, or a single pretty-printed object; - reads standard input. --format overrides the auto-detection.

A document is a JSON object whose members are the schema's field names, plus the reserved key (when there is no key field, or to check the one there is), id, and mutation_version.

{"url":"https://example.test/new","title":"A new search document","body":"This document arrived over standard input.","site":"example.test","category":"demo","published":"2026-09-03T12:00:00Z"}

Values are typed by the schema:

Streaming one document from a shell is the shortest way to check a schema:

printf '%s\n' '{"url":"https://example.test/new","title":"A new search document","body":"Arrived over standard input.","site":"example.test","category":"demo","published":"2026-09-03T12:00:00Z"}' \
  | yolosearch push articles -

The last batch carries flush, so by default push returns once the segment is published and served by this process: the next search against this server sees the documents.

Each batch comes back with one of four states:

A rejected batch is reported on stderr and makes the exit code 1; an invocation error exits 2. --json streams every acknowledgment as one object per line and prints no summary — that is the form to pipe into a loader's own logging.

Server-side limits are ingest.max_batch_bytes (4 MiB) and ingest.max_document_bytes (16 MiB). A batch naming an older schema version than the current one is rejected with the current version named. A push to an index with no schema reads index "x" has no schema; apply one first.

A document's identity is its key: the key field's value, hashed with xxh3-128 into the 128-bit document ID the engine uses. Pushing the same key twice produces the same public ID. Both immutable records coexist until compaction, but each catalog generation carries a winner-only liveness object, and query selection excludes the superseded ordinal before top-K. JSON, table, and raw gRPC results therefore contain one live version per key and still return a full requested prefix.

Every document carries a mutation version: a caller-supplied string stored verbatim, or one the server mints when the document has none — 26 characters of Crockford base32 encoding a millisecond timestamp, the node ID, and random bits, monotonic within a process. Versions compare as bytes, so zero-pad if you want numeric ordering. The ingest path always mints, so a served index never has an absent version.

yolosearch delete <index> <keys...|-> [--batch 500] [--batch-bytes 2097152] \
  [--no-flush] [--require-ack]

yolosearch delete articles \
  https://example.test/a https://example.test/b https://example.test/c
# deleted 3 keys in 1 batches · sealed segment 9f3a1c2b… · generation 7d1e0a4c… · PUBLISHED in 0.4s

delete tombstones keys: a document with deleted = true, carrying no fields and no encoded payload, sent over the same stream push uses with the same acknowledgment machinery. Keys are literal arguments, or read one per line from standard input (blank lines skipped) when the sole argument is -. Deleting a nonexistent key is accepted. Repeating a delete sends another tombstone through the ingest pipeline and consumes ingest and storage work.

Visibility changes when the generation is loaded. A winning tombstone excludes the key from lexical and vector search through the generation's liveness metadata. Existing queries keep their pinned generation. Compaction can omit deleted records from new segments; retired input objects remain until garbage collection can reclaim them.

A higher-version write can restore a deleted key. A tombstone takes part in the ordinary version comparison — greatest mutation version wins, ties broken by commit time then segment ID — so a later write with a greater version supersedes the delete and becomes live again. There is no separate "permanently deleted" state and no setting that creates one. A caller that needs a key to stay gone must not re-ingest it with a version that would win.

Under ingest.dir/<index>/ the builder keeps three directories:

A build seals when any of these happens:

• the spool reaches ingest.seal_bytes (a zero value derives one quarter of builder.sort_memory_bytes);
• it reaches ingest.seal_documents (100000);
• its first document is ingest.seal_age (30s) old;
• a flush arrives;
• a batch arrives under a newer schema version — a build is built under exactly one schema; or
• the process restarts, which seals whatever was open and queues everything sealed.

Up to builder.build_concurrency sealed builds run at once, and further builds wait in the bounded builder.max_queued_builds disk queue. New batches are THROTTLED only when active plus queued sealed builds reach that bound.

The build ID is the segment ID — a time-prefixed 128-bit ID — so sealed directories list in open order and a PUBLISHED acknowledgment can name the segment before it exists. The pipeline builds the segment under the builder's budget, publishes it commit-marker-last, announces it to the catalog, and removes the sealed directory. A failure backs off (1s doubling to 5m) and retries with the spool intact. A crash between publish and announce is replayed at startup: a commit marker found is re-announced, partial objects are abandoned to the GC sweep, and the build is re-minted under a fresh ID.

Compaction on the same node stays idle while that queue is nonempty, because its input hydration, merge scratch, and output upload contend with all three foreground resources.

Watch the queue drain:

yolosearch stats articles

yolosearch segment build --schema accepts the same schema forms schema apply does, takes the same typed document shape, and writes a segment without a running server. --message and --index cover a file that declares neither, --node-id makes it mint mutation versions, and --max-line-bytes (16 MiB) bounds the JSONL scanner. Its receipt carries format_version, schema_version, and minted_versions; without --node-id, "minted_versions": false.

segment build is the one verb that creates its object directory. Every other verb expects one that exists. yolosearch segment publish then publishes the built segment commit-last and announces it.

Use push for command-line bulk loading. For a smaller corpus, or the first documents in a new index, the admin console's Ingest section takes .jsonl, .json, .txt, .gz, and .tar.gz files straight from the browser and streams them through the same IngestService. See the admin console.

• Schemas — the schema every push is validated against.
• Filters and projection — reading the documents back.
• Catalog and generations — what a published generation is.
• The CLI reference — ingest commands and flags.