Skip to content

Reference

Metrics reference

Every Prometheus family YoloSearch exports, with its type, labels, and meaning — plus the closed label vocabularies and the cardinality policy that keeps them bounded.


Engine server processes expose /metrics on observability.metrics_listen (:9550 by default) when metrics are enabled. The admin console exposes its own metrics on its HTTP listener. The tables below describe the application families; engine registries also include Go and process collectors.

Labels use closed enumerations; index names, query text, document identities, object keys, and node addresses are never labels. A registry that would break that rule is rejected at construction.

The cardinality policy

Two rules are enforced in code before a process serves anything.

Every label must be on the allowlist. The complete set:

role  lane  service  method  code  outcome  cache_tier  object_operation
exactness  cache  state  storage_mode  segment_format  strategy  event
source  operation  publication_operation  catalog_operation  gc_reason
lease_operation  cache_state  requested_mode  policy_reason  phase  reason
tier  ack_status  seal_trigger  mode  similarity  candidate_mode  tail_ordering

Certain substrings are forbidden outright, so that a refusal says why:

query  document  segment_id  tenant  generation  object_key  error_message
url  model  fingerprint  text_digest  field  index  node  address  endpoint
peer  worker  aggregator  unit

Each of those would publish one series per thing that exists rather than per value in an enum.

The role label

Every series carries role as a constant label. Its value is the role mix the process serves, not a single role: node --roles builder,publisher reports role="builder-publisher". Under the Kubernetes operator each Deployment runs one role, so values are plain role names there.

instance and pod are not emitted by the process and must come from the scrape target. See observability for the consequences.

Label vocabularies

These are the closed enumerations the tables below refer to.

Label Values
outcome success, canceled, error, rejected
lane unspecified, interactive, streaming, background
exactness GLOBAL_EXACT, EXACT_WITHIN_CANDIDATES, APPROXIMATE
phase (query) parse, resolve, plan, place, execute, merge, encode, send
phase (seal) build, publish, announce, visible
ack_status accepted_ephemeral, rejected, throttled, published
seal_trigger age, bytes, documents, flush, schema, replay
cache_tier full, block
cache_state total, used, pinned, partial, quarantined, evictable
cache (WavesDB) block, table, file
state (WavesDB) resident, limit
storage_mode, requested_mode hydrate_full, remote_blocks, auto, unspecified
policy_reason full_cache_hit, absent, match_all, unknown_statistics, scan, hydrate_cost, remote_cost, capacity_rejection, availability_fallback, explicit, unspecified
lease_operation acquire, renew, loss
catalog_operation (publisher) reconcile, announce

The shared outcome label uses four values with a common mapping. A canceled context or a CANCELED/DEADLINE_EXCEEDED status is canceled; an explicit server rejection — INVALID_ARGUMENT, NOT_FOUND, FAILED_PRECONDITION, RESOURCE_EXHAUSTED, OUT_OF_RANGE, PERMISSION_DENIED, UNAUTHENTICATED, UNIMPLEMENTED — is rejected; anything else is error. What an operator branches on is whether the caller gave up, whether the server refused, or whether something broke. The error's own text says the rest, and is exactly what may not become a label.

Cache operations use their own outcome vocabulary rather than the shared one: hit, miss, admitted, evicted, quarantined, error, and — for the block tier — bypassed. The byte counters add restored for the full tier.

Every family

Query

Metric Type Labels Meaning
yolosearch_query_candidates_total counter lane, exactness, strategy Total candidates scored by bounded execution strategy.
yolosearch_query_completed_total counter lane, exactness, outcome Total search queries completed by outcome.
yolosearch_query_duration_seconds histogram lane, exactness Search query execution duration in seconds.
yolosearch_query_posting_frames_total counter storage_mode, outcome Total posting frames processed by storage mode and bounded outcome.
yolosearch_query_result_bytes_total counter lane, exactness Total packed result payload bytes delivered to clients.
yolosearch_query_result_cache_total counter outcome Generation-local result cache lookups by bounded outcome.
yolosearch_query_result_frames_total counter lane, exactness Total packed result frames delivered to clients.
yolosearch_query_send_blocked_seconds histogram lane, exactness Time spent in synchronous result stream sends, including flow-control blocking.
yolosearch_query_started_total counter lane, exactness Total search queries admitted for execution.
yolosearch_query_streamed_ids_total counter lane, exactness Total result document IDs delivered to clients.

Distributed execution

Metric Type Labels Meaning
yolosearch_distributed_phase_duration_seconds histogram phase, outcome Distributed query phase duration by bounded phase and outcome.
yolosearch_distributed_work_unit_failures_total counter reason Work streams that failed their query outright, by bounded reason.
yolosearch_distributed_work_unit_retries_total counter reason Work streams reassigned to another target, by bounded reason.
yolosearch_distributed_work_units_total counter lane, outcome Work units a worker served, by lane and outcome.

Merge tier

Metric Type Labels Meaning
yolosearch_merge_bound_updates_total counter outcome Upward frames stamped with a bound, by whether it tightened a held frame or closed the run.
yolosearch_merge_buffered_frames gauge — Decoded work pages this node holds at once across every live merge.
yolosearch_merge_children_total counter outcome Direct children merged, by whether the child's run was observed complete. Divided by runs_total this is the fan-in.
yolosearch_merge_parent_send_blocked_seconds histogram — Time one upward frame spent in the parent's send, including flow-control blocking.
yolosearch_merge_runs_total counter outcome Merge runs this node served, by outcome.

Placement

Metric Type Labels Meaning
yolosearch_placement_advances_total counter lane, reason Dispatches a candidate refused, advancing to the next, by lane and bounded reason.
yolosearch_placement_advertisement_age_seconds histogram — Age of an advertisement when an aggregator folded it in, measured against the worker's own observation stamp.
yolosearch_placement_advertisements_total counter outcome Advertisement observations, by what became of one: published, dropped, folded in, or expired.
yolosearch_placement_candidates_total counter tier Candidates a lookup answered, by whether the node claimed the segment.
yolosearch_placement_lookups_total counter outcome Aggregator lookups this node asked, by outcome.
yolosearch_placement_selections_total counter lane, tier Dispatch targets chosen, by lane and the regime that produced the choice.

Storage and object I/O

Metric Type Labels Meaning
yolosearch_storage_cache_requests_total counter cache_tier, outcome Total cache lookups by bounded tier and outcome.
yolosearch_storage_decisions_total counter requested_mode, policy_reason Storage decisions by requested mode and bounded policy reason.
yolosearch_storage_fallbacks_total counter outcome Remote reads answered by verified hydration instead, by outcome.
yolosearch_storage_object_bytes_total counter object_operation, outcome Total object-store bytes transferred by operation and outcome.
yolosearch_storage_object_requests_total counter object_operation, outcome Total object-store operations by operation and outcome.
yolosearch_storage_open_duration_seconds histogram storage_mode, outcome Segment open duration by explicit storage mode and outcome.
yolosearch_storage_restore_duration_seconds histogram outcome Hydrated checkpoint restore duration by outcome.

Segment readers

Metric Type Labels Meaning
yolosearch_segment_read_operations_total counter source, operation Segment object reads by byte source and operation.
yolosearch_segment_reader_events_total counter event Immutable segment reader lifecycle events.

Disposable cache

Metric Type Labels Meaning
yolosearch_cache_bytes gauge cache_tier, cache_state Cache occupancy in bytes by bounded tier and state.
yolosearch_cache_entries gauge cache_tier, cache_state Cache occupancy in entries by bounded tier and state.
yolosearch_cache_operation_bytes_total counter cache_tier, outcome Bytes moved by cache operations by bounded tier and outcome.
yolosearch_cache_operations_total counter cache_tier, outcome Cache operations by bounded tier and outcome.
yolosearch_cache_restore_duration_seconds histogram outcome Full-segment hydration (restore) duration by outcome.

Publication

Metric Type Labels Meaning
yolosearch_publication_duration_seconds histogram publication_operation, outcome Segment publication phase duration by bounded operation and outcome.
yolosearch_publication_object_bytes_total counter publication_operation, outcome Bytes published or verified by bounded operation and outcome.
yolosearch_publication_operations_total counter publication_operation, outcome Segment publication phases by bounded operation and outcome.

Catalog

Metric Type Labels Meaning
yolosearch_catalog_duration_seconds histogram catalog_operation, outcome Catalog operation duration by bounded operation and outcome.
yolosearch_catalog_gc_proposals_total counter gc_reason Objects a dry run proposed, by bounded reason. A dry run never deletes.
yolosearch_catalog_lineage_capacity_ratio gauge — Larger of generation-count and encoded-byte utilization observed by the most recent publisher lineage scan.
yolosearch_catalog_lineage_repairs_total counter reason Degraded lineage conditions a reconciliation observed, by bounded reason: a repaired sibling conflict, or a reconciliation that never converged and repaired nothing.
yolosearch_catalog_operations_total counter catalog_operation, outcome Catalog operations by bounded operation and outcome.

Catalog publisher

Metric Type Labels Meaning
yolosearch_publisher_catalog_operations_total counter catalog_operation, outcome Publisher catalog cycles by bounded operation and outcome.
yolosearch_publisher_leader gauge — One while this process holds and is renewing the catalog publisher lease.
yolosearch_publisher_lease_operations_total counter lease_operation, outcome Publisher lease operations by bounded operation and outcome.
yolosearch_publisher_publication_lag_seconds histogram — Commit-marker durability to catalog-generation publication latency.

Catalog follower

Metric Type Labels Meaning
yolosearch_follower_held_engine_references gauge — Live query and segment-execution references preventing engine reclamation.
yolosearch_follower_lag_seconds gauge — Seconds a newer latest has been observed without being served; zero when current.
yolosearch_follower_polls_total counter outcome Catalog latest polls by outcome.
yolosearch_follower_retired_engines gauge — Superseded engines still held for in-flight queries or the retirement grace.
yolosearch_follower_swaps_total counter — Generations installed by the follower.
yolosearch_follower_window_engines gauge — Exact-generation engines held until their catalog-overlap expiry.
yolosearch_follower_window_resolutions_total counter outcome Exact-generation window resolutions by bounded outcome.

Ingest router

Metric Type Labels Meaning
yolosearch_ingest_admission_refusals_total counter reason Router admission refusals by bounded reason: queue or quota.
yolosearch_ingest_batches_total counter ack_status Ingest batches by the acknowledgement they received.
yolosearch_ingest_builder_selections_total counter outcome Builder routing decisions by bounded outcome.
yolosearch_ingest_bytes_total counter ack_status Ingest request bytes accepted or rejected.
yolosearch_ingest_documents_total counter ack_status Ingest documents accepted or rejected.
yolosearch_ingest_eligible_builders gauge — Builders eligible in the router's most recent asynchronous health snapshot.
yolosearch_ingest_pending_acks_dropped_total counter — Tracked batches dropped from a session's bounded PUBLISHED tracking.
yolosearch_ingest_publish_failures_total counter — Seal pipelines that failed to build, publish, or announce; the sealed spool is kept and retried.
yolosearch_ingest_queue_capacity_bytes gauge — Current global encoded-byte admission capacity of this router.
yolosearch_ingest_queued_bytes gauge — Encoded batch bytes currently reserved by this router across all indexes.
yolosearch_ingest_sessions gauge — Open Ingest streams.
yolosearch_ingest_unavailable_targets_total counter reason Builder health observations excluded from routing by bounded reason.

Builder

Metric Type Labels Meaning
yolosearch_builder_open_build_age_seconds gauge — Age of the oldest open build's first document; zero when nothing is open.
yolosearch_builder_open_build_bytes gauge — Spooled bytes of open builds.
yolosearch_builder_open_build_documents gauge — Spooled documents of open builds.
yolosearch_builder_pending_builds gauge — Sealed builds not yet announced (queued, building, publishing, or retrying).
yolosearch_builder_publication_lag_seconds histogram — First document spooled to the generation holding it being served.
yolosearch_builder_seal_phase_seconds histogram phase, outcome Seal pipeline phase duration by bounded phase and outcome.
yolosearch_builder_seals_total counter seal_trigger, outcome Builds sealed by bounded trigger and outcome.

Maintenance

Metric Type Labels Meaning
yolosearch_maintenance_compaction_runs_total counter outcome Scheduled compaction cycles by bounded outcome: completed, noop, refused, or failed.
yolosearch_maintenance_gc_sweeps_total counter outcome Destructive garbage-collection sweep cycles by bounded outcome: completed, noop, refused, or failed.

Embedding

Metric Type Labels Meaning
yolosearch_embedding_duration_seconds histogram mode, outcome Embedding provider batch duration by mode and bounded outcome.
yolosearch_embedding_query_cache_total counter outcome Completed query-vector cache lookups by bounded outcome.
yolosearch_embedding_requests_total counter mode, outcome Embedding provider batches by schema-selected mode and bounded outcome.
yolosearch_embedding_texts_total counter mode, outcome Texts submitted to embedding providers by mode and bounded outcome.
yolosearch_embedding_truncated_texts_total counter — Texts whose gRPC embedding response reported truncation to the model's token window.

Vector queries

Metric Type Labels Meaning
yolosearch_vector_queries_total counter similarity, candidate_mode, outcome Vector query executions by similarity, candidate mode, and bounded outcome.

Approximate tail

Metric Type Labels Meaning
yolosearch_tail_executions_total counter tail_ordering, outcome Tail executions by actual ordering and bounded outcome.

Configuration

Metric Type Labels Meaning
yolosearch_config_reload_failures_total counter — Configuration file reloads that were refused; the previous configuration stayed.
yolosearch_config_reloads_total counter — Configuration file reloads that were applied.

Admin streams

Metric Type Labels Meaning
yolosearch_admin_fleet_assemblies_total counter outcome Fleet frames assembled, by whether every configured node contributed.
yolosearch_admin_fleet_nodes_total counter outcome Per-node contributions summed over fleet assemblies, by the node's condition.
yolosearch_admin_frames_total counter outcome Admin stats frames delivered to clients, and frames a slow client was skipped past.

Custom collectors

The custom collectors below expose object I/O totals and WavesDB cache resources.

Whole-process object I/O

Metric Type Meaning
yolosearch_storage_process_object_read_bytes histogram Whole-process object-store read request count and transferred-byte sum, including metadata requests
yolosearch_storage_process_object_write_bytes histogram Whole-process object-store write request count and transferred-byte sum

These are constant histograms. Their sample count is the request total and their sample sum is transferred bytes — they have no meaningful bucket distribution. Read them as:

promql
rate(yolosearch_storage_process_object_read_bytes_sum[5m])    # bytes per second
rate(yolosearch_storage_process_object_read_bytes_count[5m])  # requests per second

The counting wrapper encloses the complete object store, so the totals include ingest publication, catalog access, compaction, cache fills, and query reads — not only request-scoped search traffic.

Reads count Stat, FullGet, RangeGet, List, Exists, and Size; writes count Put, Create, CompareAndSwap, and Delete.

WavesDB read resources

Metric Type Labels Meaning
yolosearch_storage_wavesdb_read_cache_events_total counter cache, outcome Read-cache events separated by decoded-block and table-reader cache
yolosearch_storage_wavesdb_read_cache_bytes gauge cache, state Resident and configured-limit bytes
yolosearch_storage_wavesdb_read_cache_entries gauge cache, state Resident and configured-limit entry counts

cache is block, table, or file; outcome is hit, miss, or eviction; state is resident or limit.

The block-entry family has only state="resident": the decoded-block cache is bounded by bytes, not by an independent entry count. Rebinding an already registered process owner is idempotent, so overlapping generation readers do not multiply either limits or reported occupancy.

The admin console's families

The console process exposes six label-free gauges and nothing else. They describe the bridge, not any index.

Metric Type Meaning
yolosearch_admin_build_info gauge Always 1; admin console process information
yolosearch_admin_configured_targets gauge Allowlisted upstream targets
yolosearch_admin_fleet_collectors gauge Shared upstream fleet streams currently active
yolosearch_admin_fleet_subscribers gauge Browser fleet-stream subscribers currently active
yolosearch_admin_search_in_flight gauge Query Lab searches currently occupying a bounded slot
yolosearch_admin_search_capacity gauge Maximum concurrent Query Lab searches

Standard collectors

Every data-plane registry also registers the Prometheus Go client's process and Go collectors, so process_* and go_* families are present with their usual meanings.

Next

  • Observability — scrape configuration and the shipped alerts
  • Sizing — reading these as a capacity signal
  • Glossary — the vocabulary the label values use