Environment-variable catalog selection with an isolated VECTORDATA_HOME
and three stdout-driver one-liners: profile-window warm-up, a
caller-named ordinal range in the background, and demand-paged access.
Java API, expr, and inline-op examples for remote reads and for
precache/prefetch in both profile-window and caller-named ordinal-range
forms, plus the verification patterns the integration tests use.
metadataResults, facet, windowedFacet, and variableFacet round out
facet access from expressions; prefetchPlan prices a caller-named
record window without fetching, prefetch blocks until it is resident,
and prefetchBackground returns the running handle. Unresolvable
windows are refused rather than quietly fetching the whole facet.
The standard mappers (BaseVectors, QueryVectors, NeighborIndices,
NeighborDistances, new MetadataResults) address facets by canonical
name, so Rust-published datasets resolve; each takes an optional record
window — clipping the reader and warming exactly those bytes — and a
prefetch mode of eager, background, or none. Facet reads any named
facet including scalars and custom facets; VariableFacet reads vvec
records, warming a window through the offset index while indices stay
absolute. A binding window composes with a profile-declared window by
shifting to absolute records for the fetch.
A profile window names a range someone wants repeatedly; it is not a
fence. TestDataView gains prefetchPlan, prefetch, and
prefetchInBackground, taking a DSWindow in record coordinates for any
facet:
- per-format mapping: xvec by header stride, scalars at the element
stride, vvec through the offset index — loaded whole, cached on the
view's facet handle, and never rebuilt by a remote walk while
planning; locally walked indexes persist atomically beside the data
- plans price chunk-level residency, coalesce ranges at chunk
adjacency, and separate what was asked from what will be issued
- an unresolvable window degrades to a whole-facet fetch that
WholeFacetFallback refuses unless the caller consents; an empty
window is a request for the whole facet, never a degrade
- background prefetch exposes a handle with plan, counters,
range-granular cancel, and a join that surfaces worker failures
- manifests accept the window-suffix sugar on source strings and key
resolved facets canonically, so Rust-published dataset.yaml files
resolve through the standard accessors
Behavior tracks vectordata-rs at 96d787722b39, per RUST_COMPATIBILITY.md.
Two layouts sit next to real data: N record starts (the vectordata-rs
walk rebuild) and N+1 entries ending in a payload-size sentinel (what
nbdatatools publishes). No record can start at the payload size, so the
reader detects the sentinel and both layouts describe N records; a
Rust-walked local dataset is no longer unreadable here. Sidecar probing
is size-directed — the width the payload calls for, first — and reader
windows parse the DSWindow grammar, apply the first interval, and clamp
both bounds to the data instead of throwing.
RangeFill reports what fetching a byte range would cost at chunk
granularity — resident chunks are free, and alignment overfetch is
visible before the fetch. prebufferRange drives only the covering
chunks; CacheStats carries the transfer-chunk size so consumers can map
byte offsets to chunk ordinals; rangeCapable distinguishes sources that
cannot service partial fetches.
Ports the StandardFacet canonical/alias table: base_vectors and kin are
the dataset.yaml keys, with base, train, query, gt, ground_truth, the
metadata shorthands, and the prefiltered/postfiltered families accepted
as aliases. The test pins the whole table so drift from the reference
is a deliberate change, not an accident.
The canonical window spelling from the vectordata-rs dataset-source
parser: half-open intervals, open ends, bracket bound adjustment,
count/size suffixes with compound forms, and comma-separated interval
lists. Intervals that select no records are rejected at parse, naming
the comma-for-'..' mistake.
Surefire runs only tests tagged 'unit' (nb.junit.tags), so the module's
untagged suites were invisible to 'mvn -B test' in CI. Tag every unit
test class; the *IntegrationTest suites already run under failsafe.
Jonathan Shook
Youki Shiraishi
dependabot[bot]
allcontributors[bot]