Skip to content

Start

Try it

Query 72 million FineWeb documents from your terminal in three commands, with no account, no credentials and no build.


Three commands and you are querying 72 million documents.

zsh
brew tap yannick/tap
brew install ysearch-demo
ysearch-demo

It needs an Apple silicon Mac with macOS 26 (Tahoe) or newer, because the server runs on Apple's container runtime. Homebrew installs that runtime along with ysearch-demo; start it once, before the first run:

zsh
container system start

Nothing to sign up for and nothing to configure. The corpora are public and read-only, so no credentials are involved at any point.

What you are querying

Two indexes over FineWeb (HuggingFaceFW/fineweb), Common Crawl text filtered for quality. Both are built from shards of its sample-100BT subset and published to the same public bucket, s3://fineweb-public/, served straight out of object storage.

Index What it is Documents Index data in S3
fineweb-200 The ~200 GB build. The demo container serves it, and ysearch-demo opens it. 71.6 million 175 GB
fineweb-100-qnode-0f98f61 The ~100 GB build, written in the current segment format. Served only when you ask for it. 42.1 million 96 GB

Each index is named for the size it was built to: ingestion stopped once its published segment data passed 200 GB or 100 GB. Compaction has run on both since, merging segments and retiring superseded copies of documents, and it is still running — so the stored size sits below the name and drifts as it continues. The figures above are what each catalog reported on 23 September 2026, and count live segment data only, not objects compaction has replaced but not yet deleted.

ysearch-demo with no arguments needs nothing more:

ysearch-demo                       # opens fineweb-200
ysearch-demo 'text:kubernetes'     # one query against fineweb-200

Querying the smaller corpus below adds fineweb-100-qnode-0f98f61 beside it.

How it fits together

ysearch-demo is a client and nothing else — no engine, no index code, no query grammar. Your query goes to the server as a string and is compiled against the index schema there.

The server is a container that runs the real engine:

ysearch-demo ──gRPC──▶ yolosearch-demo container ──▶ S3
  (client)               (engine + console)          (public corpora)

The first run offers to start that container for you. It runs on Apple's container runtime, which Homebrew installed with ysearch-demo — on an Apple silicon Mac with macOS 26 or newer, the only place that runtime runs. Without Homebrew, install it from the signed package on its releases page, then start it once with container system start.

On an Intel Mac or an older macOS, ysearch-demo still installs and queries a YoloSearch server you name with --address host:port.

The container also serves the read-only admin console; ysearch-demo prints its URL on startup.

The cache matters

The container keeps corpus metadata in ~/.ysearch. The first start against an empty cache reads catalog metadata from S3 for every segment and verifies each one, which takes about a minute; later starts reuse what that directory holds instead of reading it all again. Deleting it is safe; the next start re-reads what it needs.

Worth trying

In the REPL, \fields lists every field the corpus actually holds, with its type and what it can be used for — which is the quickest way to stop guessing field names:

ys> \fields
  fineweb-200 declares 8 field(s):

  * id              string     key, filterable
    text            string     searchable text
  * url             string     filter only
  * language        string     filter only
  * token_count     int64      filter only

Then combine a full-text match with filters:

text:kubernetes AND language:en
text:"machine learning" AND token_count:[1000 TO *]

? explains the whole syntax. \top 20 changes how many hits come back, and \timing reports what your queries have been costing.

Other commands

zsh
ysearch-demo 'text:kubernetes'   # one query, no REPL
ysearch-demo stats               # describe the corpus
ysearch-demo serverlog -f        # follow the container's output
ysearch-demo --help

Without Homebrew

The binary is a universal (Apple silicon + Intel) Mach-O, Developer ID signed and Apple-notarized. Two downloads, both always the newest build:

zsh
# Disk image — use this if you are clicking a link in a browser.
curl -fsSLO https://git.teixos.net/api/packages/yannick/generic/ysearch-demo-release/latest/ysearch-demo-macos-universal.dmg

# Tarball — smaller, for scripts.
curl -fsSLO https://git.teixos.net/api/packages/yannick/generic/ysearch-demo-release/latest/ysearch-demo-macos-universal.tar.gz

Each has a .sha256 beside it, and VERSION names the release:

zsh
curl -fsSL https://git.teixos.net/api/packages/yannick/generic/ysearch-demo-release/latest/VERSION

Why there are two

macOS attaches a quarantine flag to anything a browser saves, and a standalone command-line binary cannot carry a stapled notarization ticket the way an app bundle can — so Gatekeeper refuses the bare binary even though it is signed and notarized. The disk image can be stapled, and is, so it opens with no warning and no xattr incantation.

The tarball is fine for curl, which sets no quarantine flag, and Homebrew removes the flag regardless. Use the disk image when a browser is involved.

Running the container yourself

ysearch-demo starts it for you, but there is nothing hidden about it:

zsh
container run -d --name ysearch --memory 12g --cpus 4 \
  -v ~/.ysearch:/cache tcr.teixos.net/yannick/yolosearch-demo:latest

--memory is not optional. Apple's runtime gives a container 1 GiB when nobody asks for more, and a demo started that way does not fail — it starts and never begins serving. The image sizes its caches for 12 GiB.

Apple's runtime gives each container its own IP rather than publishing ports onto localhost, so ysearch-demo finds it by name. To point at a server you run yourself, pass --address host:port.

Querying the smaller corpus

The container serves fineweb-200 alone unless told otherwise. A second corpus is a second live generation, with its own cold start and its own resident metadata, so serving both is a choice, and it needs more memory. Replace the running container with one that names both:

zsh
container stop ysearch && container rm ysearch

container run -d --name ysearch --memory 16g --cpus 4 -v ~/.ysearch:/cache \
  -e YOLOSEARCH_DEMO_SOURCE="s3://fineweb-public/corpus/fineweb-200/indexes/indexes/fineweb-200/,s3://fineweb-public/corpus/fineweb-100-qnode-0f98f61/indexes/indexes/fineweb-100-qnode-0f98f61/" \
  tcr.teixos.net/yannick/yolosearch-demo:latest

ysearch-demo still opens fineweb-200 first. Name the other one to start there instead:

zsh
ysearch-demo --index fineweb-100-qnode-0f98f61

or move between them inside the REPL without restarting:

ys> \indexes
ys> \index fineweb-100-qnode-0f98f61

The open index and its server stay on a line above the prompt, so a result is never read against the wrong corpus.

Ready to run your own corpus instead? The quick start builds an index from your own documents.