- a facet may be a series: the uniform form names `base__NNNN.fvec`
with shard_stride, shard_count, and record_count; the explicit form
lists entries in the source grammar, each bare, `=`-counted,
windowed, or both, checked against each other
- ordinals map in O(1) whenever the lengths are uniform however the
series was spelled, and by prefix-sum search otherwise; an entry
window is in that file's ordinals, the facet window is in facet
ordinals, and a suffix on a uniform pattern is the facet window
- storage, sidecars, and offset indexes are per file, so two shards of
one file share one of each; the readers present a single surface,
refuse shards that disagree on shape, and name a missing shard
rather than reading a shorter facet
- prefetch plans carry ShardRange in place of ByteRange, decompose a
window across the shards it spans, never merge across shards, and
sum prerequisite bytes per file; a whole-facet request names every
shard, and cache statistics report the weakest access mode
- format_version is a minimum reader requirement: a dataset above 2,
or one stating less than its content needs, is refused naming both
numbers; declarations are checked for self-consistency at load, and
a directory falls back to knn_entries.yaml only when dataset.yaml is
absent, so a refusal keeps its diagnosis
- a series holds open no more files than a quarter of the descriptor
soft limit, floor 8, or VECTORDATA_SHARD_FD_CAP
The ISO forms of LocalTime and LocalDateTime omit the seconds field
when the second is 00, so currentTime(), currentDateTime(),
currentTimeInZone() and millisToDateTime() each returned a shorter
string once a minute — HH:mm instead of HH:mm:ss. That failed
testCurrentTime in CI roughly one run in sixty, and silently handed
workloads a differently-shaped timestamp the rest of the time.
Each now formats with an explicit pattern rather than toString(), which
also makes the truncatedTo(SECONDS) calls redundant.
The new test pins the boundary through millisToDateTime, whose instant
comes from the caller: the wall-clock functions can only be asserted
for shape, which catches this one second in sixty. It builds a
timestamp on a whole minute, requires the seconds to be there, and
repeats at second 30 so a fix in the other direction is caught too.
A binding window re-based the reader to zero, so BaseVectors(spec,
'[50000..100000)') exposed indices 0..49999. A workload running the
cycles that window names then indexed past the end and failed, and it
only ever worked from cycle 0, where the re-basing is the identity. It
also disagreed with the two surfaces beside it: VariableFacet and
prefetchCycles both speak absolute ordinals.
The window now selects which records to warm and nothing else. Indices
are absolute everywhere, so cycles address the records they name, and
the fetch still covers only the window — a [50000..100000) slice of a
100k profile warms 74 MiB rather than the full 146 MiB.
Reads outside the window demand-page rather than failing, which is
load-bearing rather than lenient: VirtData probes a newly resolved
function with a small sample index before any cycle runs, so refusing
out-of-window reads leaves a window above zero unable to resolve at
all. A test pins that.
WindowedReader keeps its restricting behaviour for the windowedFacet
expression function, where a caller asks for a restricted reader
directly and no resolver probe is involved.
Asking for a prefetch with progress meant asking for prefetch=false and
then a binding mode — two knobs that read as contradicting each other,
because the background expr function was the one prefetch form that
never got a meter, so the only way to see a background download was to
route it through a binding.
prefetchBackground now announces its plan and watches the handle like
every other form, and prefetchCyclesBackground takes the ordinal range
directly. With the API reachable at the surface, the demo asks for what
it wants in one word: prefetch=load blocks on prefetchCycles before the
run, prefetch=background hands back a metered handle that fetches while
the cycles run, prefetch=none leaves reads to demand-page. The binding
mode knob is gone.
The documented 'nb5 run vectordata_demo' does not work: run is itself a
command, so the workload name is parsed as a second one and resolves to
no service. The scenario form drops the run — 'nb5 vectordata_demo
dataset=<name>:<profile>' — and the long form names the full bundled
path, 'nb5 run workload=activities/examples/vectordata_demo.yaml'.
Both are now shown in the workload and the guide.
Verified against the 5.25.15-release binary: the workload is bundled
and listed, both corrected forms load it, and a real dataset run meters
the download and reads base, query, and neighbor records.
The demo only worked from a checkout. It now ships in the stdout
adapter's activities, so 'nb5 run vectordata_demo dataset=<name>:<profile>'
works from the distribution alone, with --cat and --copy for reading or
editing it.
Every catalog-touching expression is guarded on the dataset parameter
being given. Bundled workloads are parsed through the expression
pipeline by --list-workloads and --list-scenarios, so a workload that
opened a catalog unconditionally would fail listing for every user
without one configured; with no dataset named it now renders usage text
and touches nothing.
vectordata_demo.yaml needs no database — the stdout driver exercises
dataset access alone. It prints what it derived from the dataset,
fetches a record range with the meter running, and reads records back,
so a download can be watched from one command line. prefetch=false
with mode=background moves the fetch off the load path and overlaps it
with the cycles instead.
Binding meter labels now carry the profile, matching the expr form, so
both name the same thing in a run that uses both.
A prefetch driven from a workload was silent and awkward to aim:
- every expr prefetch now reports through the same meter the bindings
use, so a download started from a workload announces its cost, emits
progress while the bytes move, and confirms completion
- prefetchProfile(spec) warms every facet the profile declares, each to
the window it declares, in one call — the form that cannot
accidentally pull a whole shared base file, since an empty window
means the entire facet by design
- prefetchCycles(spec, facet, start, end) takes the ordinal range
directly, which is how an activity already describes its work
The docs record the three things that bite: expressions evaluate at
load time rather than when a tagged block runs, an empty window is a
request for everything, and assignment sigils render their value where
they are written.
cql_vector_dataset.yaml takes one dataset input —
dataset='<dataset>:<profile>' — and derives everything dataset-shaped
through the expr system at load time: the schema's vector dimensions
and similarity function, rampup cycles from the base count, search
cycles from the query count, and the ANN LIMIT and recall k from the
ground-truth width. Bindings read through the vectordata mappers, so
the profile's base window warms eagerly with download progress before
the first cycle; the plan's cost is echoed into the rendered
description. Operational knobs (keyspace, table, rf, cl) stay
conventional TEMPLATE parameters.
Verified through the real load-path transforms (TemplateRewriter +
ExprPreprocessor) against a published remote dataset: dimensions 384,
train 100k, test 10k, k 100, dot_product, all spliced correctly.
The manifest's attributes block — distance_function and whatever else
the publisher recorded — was parsed and discarded; the Rust view
carries it for metadata accessors. TestDataView.attributes() returns
it on every profile view, which is how a workload derives
configuration from the dataset instead of restating it.
In this catalog format, ivec/ivecs records are length-qualified but
fixed throughout — ground-truth neighbor files — so a window maps at
the header stride like any other xvec, and only the *vvec extensions
carry variable-length records. Classifying ivec as variable sent
neighbor-facet windows through the offset-index path, and a remote
ground-truth file publishes no sidecar, so the window degraded and the
consent gate refused an eager warm-up of three records.
ElementType now recognizes the *vvec extension families, and the
variable-length test fixtures use them instead of overloading .ivecs.
Verified against the real remote dataset: a neighbor window resolves
to exactly its records' bytes and the reader decodes them.
The Rust source's is_vvec_ext still classifies ivec as variable; noted
in RUST_COMPATIBILITY.md for upstream reconciliation.
devdocs/vectordata/ holds the guide and remote_smoke_test.sh, which
checks base, query, and neighbor-index facets of one profile by
eagerly warming exactly the records each check reads — with the plan
announced and download progress on stderr — aborting on the first
failure. VECTORDATA_CATALOG and DATASET are required, DATASET as
'<dataset>' or '<dataset>:<profile>' with the profile defaulting to
'default', and NB selects how nosqlbench is invoked. The guide keeps
the Facet, windowed-background, and whole-profile warm-up forms as
separate examples.
Eager and background prefetches announce the plan's cost before any
bytes move, emit rate-limited progress on stderr while the download
runs — a meter, not a log event, so it shows regardless of logging
configuration — and confirm completion. Silent when there is nothing
to fetch, so local and already-warm facets add no noise.
Published sized profiles name only what differs: a windowed base
override and their own neighbor facets, with no extends key. Facets a
profile does not declare — query vectors included — resolve from
'default' implicitly, matching the expanded layout the publishing
toolchain emits into catalogs. Java honored only explicit extends, so
a released dataset failed with 'Profile 100k lacks facet
query_vectors' while its own catalog showed the facet plainly.
The window: key also accepts the serializer's structured form (a list
of {min_incl, max_excl} maps) and bare counts; dropping an
unrecognized form silently would read the whole facet where a window
was declared, so it fails instead.
Verified against the real remote dataset: profile 100k resolves all
four facets, reads query vectors, and clips the base to its window.
Every unit test now builds explicit VectorDataSettings with a temporary
cache directory instead of reaching VectorDataSettings.defaults(). The
defaults path deliberately refuses to auto-configure a cache when the
largest writable mount is outside the home filesystem — exactly the
shape of a GitHub runner — so any test depending on it fails there by
design. This broke the 5.25.14-release build, the first CI run to
execute these suites after they were tagged into the unit group.
Environment-variable catalog selection with an isolated VECTORDATA_HOME
and three stdout-driver one-liners: profile-window warm-up, a
caller-named ordinal range in the background, and demand-paged access.
Java API, expr, and inline-op examples for remote reads and for
precache/prefetch in both profile-window and caller-named ordinal-range
forms, plus the verification patterns the integration tests use.
metadataResults, facet, windowedFacet, and variableFacet round out
facet access from expressions; prefetchPlan prices a caller-named
record window without fetching, prefetch blocks until it is resident,
and prefetchBackground returns the running handle. Unresolvable
windows are refused rather than quietly fetching the whole facet.
The standard mappers (BaseVectors, QueryVectors, NeighborIndices,
NeighborDistances, new MetadataResults) address facets by canonical
name, so Rust-published datasets resolve; each takes an optional record
window — clipping the reader and warming exactly those bytes — and a
prefetch mode of eager, background, or none. Facet reads any named
facet including scalars and custom facets; VariableFacet reads vvec
records, warming a window through the offset index while indices stay
absolute. A binding window composes with a profile-declared window by
shifting to absolute records for the fetch.
A profile window names a range someone wants repeatedly; it is not a
fence. TestDataView gains prefetchPlan, prefetch, and
prefetchInBackground, taking a DSWindow in record coordinates for any
facet:
- per-format mapping: xvec by header stride, scalars at the element
stride, vvec through the offset index — loaded whole, cached on the
view's facet handle, and never rebuilt by a remote walk while
planning; locally walked indexes persist atomically beside the data
- plans price chunk-level residency, coalesce ranges at chunk
adjacency, and separate what was asked from what will be issued
- an unresolvable window degrades to a whole-facet fetch that
WholeFacetFallback refuses unless the caller consents; an empty
window is a request for the whole facet, never a degrade
- background prefetch exposes a handle with plan, counters,
range-granular cancel, and a join that surfaces worker failures
- manifests accept the window-suffix sugar on source strings and key
resolved facets canonically, so Rust-published dataset.yaml files
resolve through the standard accessors
Behavior tracks vectordata-rs at 96d787722b39, per RUST_COMPATIBILITY.md.
Two layouts sit next to real data: N record starts (the vectordata-rs
walk rebuild) and N+1 entries ending in a payload-size sentinel (what
nbdatatools publishes). No record can start at the payload size, so the
reader detects the sentinel and both layouts describe N records; a
Rust-walked local dataset is no longer unreadable here. Sidecar probing
is size-directed — the width the payload calls for, first — and reader
windows parse the DSWindow grammar, apply the first interval, and clamp
both bounds to the data instead of throwing.
RangeFill reports what fetching a byte range would cost at chunk
granularity — resident chunks are free, and alignment overfetch is
visible before the fetch. prebufferRange drives only the covering
chunks; CacheStats carries the transfer-chunk size so consumers can map
byte offsets to chunk ordinals; rangeCapable distinguishes sources that
cannot service partial fetches.
Ports the StandardFacet canonical/alias table: base_vectors and kin are
the dataset.yaml keys, with base, train, query, gt, ground_truth, the
metadata shorthands, and the prefiltered/postfiltered families accepted
as aliases. The test pins the whole table so drift from the reference
is a deliberate change, not an accident.
The canonical window spelling from the vectordata-rs dataset-source
parser: half-open intervals, open ends, bracket bound adjustment,
count/size suffixes with compound forms, and comma-separated interval
lists. Intervals that select no records are rejected at parse, naming
the comma-for-'..' mistake.
Surefire runs only tests tagged 'unit' (nb.junit.tags), so the module's
untagged suites were invisible to 'mvn -B test' in CI. Tag every unit
test class; the *IntegrationTest suites already run under failsafe.
Jonathan Shook
dependabot[bot]
Youki Shiraishi