devdocs/vectordata/ holds the guide and remote_smoke_test.sh, which
checks base, query, and neighbor-index facets of one profile by
eagerly warming exactly the records each check reads — with the plan
announced and download progress on stderr — aborting on the first
failure. VECTORDATA_CATALOG and DATASET are required, DATASET as
'<dataset>' or '<dataset>:<profile>' with the profile defaulting to
'default', and NB selects how nosqlbench is invoked. The guide keeps
the Facet, windowed-background, and whole-profile warm-up forms as
separate examples.
Eager and background prefetches announce the plan's cost before any
bytes move, emit rate-limited progress on stderr while the download
runs — a meter, not a log event, so it shows regardless of logging
configuration — and confirm completion. Silent when there is nothing
to fetch, so local and already-warm facets add no noise.
Published sized profiles name only what differs: a windowed base
override and their own neighbor facets, with no extends key. Facets a
profile does not declare — query vectors included — resolve from
'default' implicitly, matching the expanded layout the publishing
toolchain emits into catalogs. Java honored only explicit extends, so
a released dataset failed with 'Profile 100k lacks facet
query_vectors' while its own catalog showed the facet plainly.
The window: key also accepts the serializer's structured form (a list
of {min_incl, max_excl} maps) and bare counts; dropping an
unrecognized form silently would read the whole facet where a window
was declared, so it fails instead.
Verified against the real remote dataset: profile 100k resolves all
four facets, reads query vectors, and clips the base to its window.
Every unit test now builds explicit VectorDataSettings with a temporary
cache directory instead of reaching VectorDataSettings.defaults(). The
defaults path deliberately refuses to auto-configure a cache when the
largest writable mount is outside the home filesystem — exactly the
shape of a GitHub runner — so any test depending on it fails there by
design. This broke the 5.25.14-release build, the first CI run to
execute these suites after they were tagged into the unit group.
Environment-variable catalog selection with an isolated VECTORDATA_HOME
and three stdout-driver one-liners: profile-window warm-up, a
caller-named ordinal range in the background, and demand-paged access.
Java API, expr, and inline-op examples for remote reads and for
precache/prefetch in both profile-window and caller-named ordinal-range
forms, plus the verification patterns the integration tests use.
metadataResults, facet, windowedFacet, and variableFacet round out
facet access from expressions; prefetchPlan prices a caller-named
record window without fetching, prefetch blocks until it is resident,
and prefetchBackground returns the running handle. Unresolvable
windows are refused rather than quietly fetching the whole facet.
The standard mappers (BaseVectors, QueryVectors, NeighborIndices,
NeighborDistances, new MetadataResults) address facets by canonical
name, so Rust-published datasets resolve; each takes an optional record
window — clipping the reader and warming exactly those bytes — and a
prefetch mode of eager, background, or none. Facet reads any named
facet including scalars and custom facets; VariableFacet reads vvec
records, warming a window through the offset index while indices stay
absolute. A binding window composes with a profile-declared window by
shifting to absolute records for the fetch.
A profile window names a range someone wants repeatedly; it is not a
fence. TestDataView gains prefetchPlan, prefetch, and
prefetchInBackground, taking a DSWindow in record coordinates for any
facet:
- per-format mapping: xvec by header stride, scalars at the element
stride, vvec through the offset index — loaded whole, cached on the
view's facet handle, and never rebuilt by a remote walk while
planning; locally walked indexes persist atomically beside the data
- plans price chunk-level residency, coalesce ranges at chunk
adjacency, and separate what was asked from what will be issued
- an unresolvable window degrades to a whole-facet fetch that
WholeFacetFallback refuses unless the caller consents; an empty
window is a request for the whole facet, never a degrade
- background prefetch exposes a handle with plan, counters,
range-granular cancel, and a join that surfaces worker failures
- manifests accept the window-suffix sugar on source strings and key
resolved facets canonically, so Rust-published dataset.yaml files
resolve through the standard accessors
Behavior tracks vectordata-rs at 96d787722b39, per RUST_COMPATIBILITY.md.
Two layouts sit next to real data: N record starts (the vectordata-rs
walk rebuild) and N+1 entries ending in a payload-size sentinel (what
nbdatatools publishes). No record can start at the payload size, so the
reader detects the sentinel and both layouts describe N records; a
Rust-walked local dataset is no longer unreadable here. Sidecar probing
is size-directed — the width the payload calls for, first — and reader
windows parse the DSWindow grammar, apply the first interval, and clamp
both bounds to the data instead of throwing.
RangeFill reports what fetching a byte range would cost at chunk
granularity — resident chunks are free, and alignment overfetch is
visible before the fetch. prebufferRange drives only the covering
chunks; CacheStats carries the transfer-chunk size so consumers can map
byte offsets to chunk ordinals; rangeCapable distinguishes sources that
cannot service partial fetches.
Ports the StandardFacet canonical/alias table: base_vectors and kin are
the dataset.yaml keys, with base, train, query, gt, ground_truth, the
metadata shorthands, and the prefiltered/postfiltered families accepted
as aliases. The test pins the whole table so drift from the reference
is a deliberate change, not an accident.
The canonical window spelling from the vectordata-rs dataset-source
parser: half-open intervals, open ends, bracket bound adjustment,
count/size suffixes with compound forms, and comma-separated interval
lists. Intervals that select no records are rejected at parse, naming
the comma-for-'..' mistake.
Surefire runs only tests tagged 'unit' (nb.junit.tags), so the module's
untagged suites were invisible to 'mvn -B test' in CI. Tag every unit
test class; the *IntegrationTest suites already run under failsafe.
- Persisted min/max/mean/stddev in a new sample_statistics SQLite table; added StatsCommand and made InstantCommand/StatsCommand use per-metric latest timestamp.
- Fixed lost activity metrics on short runs: ActivityExecutor now triggers a bypass-bucket snapshot before op dispensers detach; SQLite reporter is silent under post-close ticks.
- New stress-report bundled app + dummy_session.yaml workload with two new diag op-tasks (cycledelay, errormodulo); c-stress-shaped per-op output, A/B equivalence mode, comparability warning at the top.
- removed NBCLI's foundational-label re-injection.
Jonathan Shook