Every dataset-shaped parameter is derived from the profile: the table's
metadata columns from the record layout, dimensions and k from the
facets, the similarity from the attributes. Rampup binds each base
vector with its metadata record field by field through a prepared
insert; search runs each query under the predicate published for it,
rendered as a WHERE clause since the predicates differ in shape, with
SAI indexes on every column they reference, and verifies against the
pre-filtered ground truth, padding a short result set the way the
ground truth pads a short match set.
RecordText renders each record of a facet in a named vernacular — the
rendering contract beside the binding one, for a statement whose shape
changes per cycle. CqlColumns derives CQL column definitions from a
record layout, typed from the bind types so the columns accept what the
binders produce, and the recordLayout, recordFields, and recordColumns
expression functions expose the layout to a workload at load time.
A CQL WHERE clause is a bare list of relations and Cassandra rejects a
conjunction wrapped in parentheses, which is what the reference's CQL
vernacular emits. Render the bare form for CQL, keep the grouping for
SQL and for the != and OR rewrites, and record the deviation in the
contract for upstream reconciliation.
Four mappers over a facet of opaque records, each learning the layout
once and resolving no field name per cycle: RecordField projects one
field to the Java value its wire type says, RecordFields binds a record
to named values in bind order, RecordForm binds under a form the facet
declares, and PredicateParams binds a predicate's comparands as
parameters typed from the metadata fields they constrain. The shared
projection maps bind types to driver-ready Java values, reading varint
and decimal payloads as big-endian two's complement.
State the binding contract and the bind-type asymmetry, replace the
note that binding forms are out of scope with the three deviations that
remain — field views as objects, form extras as node values, and the
raw predicate scanner not ported — and show the cycle loop.
The scaffolding projective functions build on: a Layout learned once
from a facet's first record, a Binder compiled against it that binds
by position with parameter renames applied at compile time, forms read
from the facet's forms namespace with one implicit form when it has
none, and a PredicateBinder that types each condition from the field's
tag rather than the comparand's variant and refuses a record whose
leader byte or shape disagrees with the template. BindType is its own
exhaustive mapping from the tag, distinct from the rendering one.
Port the reference's raw scan: a record's fields reached as views over
its own bytes, names skipped rather than copied, every tag advanced at
its true width. A Field reads its payload through accessors typed by
the wire form — a Half as a float, the temporal trio as text, Nanos as
two numbers — or materializes the AST view on request. Schema discovery
and predicate flattening ride on the same walk.
State the record, wire-format, vernacular, and facet-shape parts of the
contract, and replace the note that slab records are not decoded with
the three deviations that remain: a tree codec in place of serde,
document-order JSON parsing, and binding forms out of scope.
A slab facet opens through openFacetRecords as a RecordFacet, one
container per file with each shard of a series based at zero, and a
codec applied to it yields a typed reader. Every extension the spec
names carries a facet shape, facetShape answers it, and each reader
refuses the other shape by naming the one that opens it. Reading one
remote record costs its page, not the file.
Stage 2 of the codec: the thirteen vernaculars the reference names,
each rendering both node kinds in its own syntax. JSON, YAML, SQL, CQL,
CDDL, and readout parse back with the reference's type inference; the
schema, value, and display forms refuse to, naming themselves. The tree
form stands in for a serde target.
Port the reference's stage-1 codec: an MNode as a leader byte, a field
count, and tagged values over the 29 type tags; a PNode as a pre-order
predicate tree in the indexed, named legacy, and named typed wire forms.
Fingerprints and congruence compare shape without values, the Display
grammar reads back through PNodeDisplay, and PredicateEvaluator applies
a tree to a record with the reference's coercions.
- the canary asserted only that the base had records; it now reads
base records at both ends of the profile's window, across a shard
seam when the base is a series, a query, and a neighbor row whose
entries name base records
- every facet is planned against the window it declares, a slab's
included, and none may degrade to a whole-facet fetch — the case a
sized profile over a slab-backed dataset presents
- a small base window is prefetched and read back resident, and the
run prints what it found so a release log carries the numbers
- a binding window over a series warms the shards it spans and still
addresses records by absolute ordinal across the seam
- a background prefetch over a series plans one request per shard
- a sized profile over a series clips at its base_count, serves its
own ground truth, and inherits queries from default
- a series whose shards disagree on shape is refused naming the odd
one; a gap in the middle of a series is named, never skipped
- two shards sliced from one file share one storage and address their
own windows of it; a local series reports local and complete
- the weakest access mode is the one true of every file, and a series
with one unverified shard reports it over HTTP; a range-less server
hands each shard over whole and the series still reads as one
- a mapped series window is prefetched without a fallback in play
- the whole-profile prebuffer visits slab facets whole, a sharded slab
shard by shard, and an explicit entry that slices a slab plans its
pages by file ordinal
- slab containers are planned, prefetched, and prebuffered by the pages
a window spans, through the index in the slab's tail
- a source may name a slab namespace, and the rule that keeps a URL
scheme or drive letter from reading as one
- slab records are not decoded here; the clarification names what a
slab facet does and does not take part in
- a slab window still reported degrades_to_full_download, so a sized
profile over a slab-backed dataset was refused by the whole-profile
prebuffer where the reference fetches the pages the window spans
- the slab's page index is read from the tail: the 16-byte footer, the
terminal pages page or namespaces page, the named namespace's pages
page when indirected, and the last page's own record count — a
bounded read of two short pages, never a walk of the file, reported
as prerequisite bytes
- a window maps to the byte extent of the pages holding it, one
contiguous range since pages lie in ordinal order; a window running
past the end stops at the last page, one starting past the end
degrades rather than fabricating a range
- tests write slabs in the slabtastic layout, single- and
multi-namespace, and check every ordinal against its page, the
seams, the planner's bounds, a slab series across its seam, and over
HTTP that ten records fetch a page and a sized profile fetches only
its window's pages
- a source may name a namespace within a slab — `m.slab:content`,
`m.slab:ns:[0..1K]` — or carry one beside it under namespace:/ns:,
which applies to every source that names none of its own
- the text after the last `:` is a namespace only when the path before
it has an extension and the text after names no directory, so a URL
scheme or a drive letter never reads as one
- the namespace travels with the facet descriptor and with each shard
entry, and the per-format mapping is addressed by the locator — the
file plus its namespace — rather than the file alone
- baseline moves from crate 1.8.0 to 2.1.1; the contract records the
series model, the two window coordinate spaces, format_version,
the `=count` suffix, axis-aware inheritance, shard-qualified plans,
the window-aware whole-profile prebuffer, and the shard error
taxonomy
- clarifications named where Java differs: a windowed reader's
prebuffer fetches its window where the reference's is a no-op, and
evicting a file from a series' open-file budget releases a
reference rather than closing shared storage
- the README shows the uniform and explicit declaration forms and the
whole-profile prebuffer
- the whole-profile prebuffer fetched every facet's file whole, so a
sized profile over a large base downloaded the whole base while the
selective prefetch beside it fetched only the window
- prebuffer(fallback, progress) now plans every facet with the
shard-aware planner against the window it declares for itself, a
series included, and refuses a declared window the format cannot
map under REFUSE exactly as a requested window is; every facet is
planned before any is fetched, and prebuffer(progress) is the
REFUSE form
- the reader a view hands back for a windowed facet warms its window
rather than the file it was cut from
- remote tests: a 500-record window over a three-shard base leaves the
last shard untouched, through the view, through an inherited
base_count window, and through the reader
- neighbor facets no longer cross the size axis: a sized profile that
omits its own ground truth fails with "lacks facet" instead of
silently serving the full base's, since ground truth is derived
from base_count
- base_vectors and metadata_content inherit under the child's
base_count window unless the parent's already carries one
- `inherits:` names a parent, with `extends:` kept as a spelling; an
unknown or self parent falls back to default, and a cycle leaves its
members, and anything inheriting from them, with what they declare
- a `partition: true` profile is an oracle partition with independent
base vectors and inherits nothing
- a facet may be a series: the uniform form names `base__NNNN.fvec`
with shard_stride, shard_count, and record_count; the explicit form
lists entries in the source grammar, each bare, `=`-counted,
windowed, or both, checked against each other
- ordinals map in O(1) whenever the lengths are uniform however the
series was spelled, and by prefix-sum search otherwise; an entry
window is in that file's ordinals, the facet window is in facet
ordinals, and a suffix on a uniform pattern is the facet window
- storage, sidecars, and offset indexes are per file, so two shards of
one file share one of each; the readers present a single surface,
refuse shards that disagree on shape, and name a missing shard
rather than reading a shorter facet
- prefetch plans carry ShardRange in place of ByteRange, decompose a
window across the shards it spans, never merge across shards, and
sum prerequisite bytes per file; a whole-facet request names every
shard, and cache statistics report the weakest access mode
- format_version is a minimum reader requirement: a dataset above 2,
or one stating less than its content needs, is refused naming both
numbers; declarations are checked for self-consistency at load, and
a directory falls back to knn_entries.yaml only when dataset.yaml is
absent, so a refusal keeps its diagnosis
- a series holds open no more files than a quarter of the descriptor
soft limit, floor 8, or VECTORDATA_SHARD_FD_CAP
The ISO forms of LocalTime and LocalDateTime omit the seconds field
when the second is 00, so currentTime(), currentDateTime(),
currentTimeInZone() and millisToDateTime() each returned a shorter
string once a minute — HH:mm instead of HH:mm:ss. That failed
testCurrentTime in CI roughly one run in sixty, and silently handed
workloads a differently-shaped timestamp the rest of the time.
Each now formats with an explicit pattern rather than toString(), which
also makes the truncatedTo(SECONDS) calls redundant.
The new test pins the boundary through millisToDateTime, whose instant
comes from the caller: the wall-clock functions can only be asserted
for shape, which catches this one second in sixty. It builds a
timestamp on a whole minute, requires the seconds to be there, and
repeats at second 30 so a fix in the other direction is caught too.
A binding window re-based the reader to zero, so BaseVectors(spec,
'[50000..100000)') exposed indices 0..49999. A workload running the
cycles that window names then indexed past the end and failed, and it
only ever worked from cycle 0, where the re-basing is the identity. It
also disagreed with the two surfaces beside it: VariableFacet and
prefetchCycles both speak absolute ordinals.
The window now selects which records to warm and nothing else. Indices
are absolute everywhere, so cycles address the records they name, and
the fetch still covers only the window — a [50000..100000) slice of a
100k profile warms 74 MiB rather than the full 146 MiB.
Reads outside the window demand-page rather than failing, which is
load-bearing rather than lenient: VirtData probes a newly resolved
function with a small sample index before any cycle runs, so refusing
out-of-window reads leaves a window above zero unable to resolve at
all. A test pins that.
WindowedReader keeps its restricting behaviour for the windowedFacet
expression function, where a caller asks for a restricted reader
directly and no resolver probe is involved.
Asking for a prefetch with progress meant asking for prefetch=false and
then a binding mode — two knobs that read as contradicting each other,
because the background expr function was the one prefetch form that
never got a meter, so the only way to see a background download was to
route it through a binding.
prefetchBackground now announces its plan and watches the handle like
every other form, and prefetchCyclesBackground takes the ordinal range
directly. With the API reachable at the surface, the demo asks for what
it wants in one word: prefetch=load blocks on prefetchCycles before the
run, prefetch=background hands back a metered handle that fetches while
the cycles run, prefetch=none leaves reads to demand-page. The binding
mode knob is gone.
The documented 'nb5 run vectordata_demo' does not work: run is itself a
command, so the workload name is parsed as a second one and resolves to
no service. The scenario form drops the run — 'nb5 vectordata_demo
dataset=<name>:<profile>' — and the long form names the full bundled
path, 'nb5 run workload=activities/examples/vectordata_demo.yaml'.
Both are now shown in the workload and the guide.
Verified against the 5.25.15-release binary: the workload is bundled
and listed, both corrected forms load it, and a real dataset run meters
the download and reads base, query, and neighbor records.
The demo only worked from a checkout. It now ships in the stdout
adapter's activities, so 'nb5 run vectordata_demo dataset=<name>:<profile>'
works from the distribution alone, with --cat and --copy for reading or
editing it.
Every catalog-touching expression is guarded on the dataset parameter
being given. Bundled workloads are parsed through the expression
pipeline by --list-workloads and --list-scenarios, so a workload that
opened a catalog unconditionally would fail listing for every user
without one configured; with no dataset named it now renders usage text
and touches nothing.
vectordata_demo.yaml needs no database — the stdout driver exercises
dataset access alone. It prints what it derived from the dataset,
fetches a record range with the meter running, and reads records back,
so a download can be watched from one command line. prefetch=false
with mode=background moves the fetch off the load path and overlaps it
with the cycles instead.
Binding meter labels now carry the profile, matching the expr form, so
both name the same thing in a run that uses both.
A prefetch driven from a workload was silent and awkward to aim:
- every expr prefetch now reports through the same meter the bindings
use, so a download started from a workload announces its cost, emits
progress while the bytes move, and confirms completion
- prefetchProfile(spec) warms every facet the profile declares, each to
the window it declares, in one call — the form that cannot
accidentally pull a whole shared base file, since an empty window
means the entire facet by design
- prefetchCycles(spec, facet, start, end) takes the ordinal range
directly, which is how an activity already describes its work
The docs record the three things that bite: expressions evaluate at
load time rather than when a tagged block runs, an empty window is a
request for everything, and assignment sigils render their value where
they are written.
cql_vector_dataset.yaml takes one dataset input —
dataset='<dataset>:<profile>' — and derives everything dataset-shaped
through the expr system at load time: the schema's vector dimensions
and similarity function, rampup cycles from the base count, search
cycles from the query count, and the ANN LIMIT and recall k from the
ground-truth width. Bindings read through the vectordata mappers, so
the profile's base window warms eagerly with download progress before
the first cycle; the plan's cost is echoed into the rendered
description. Operational knobs (keyspace, table, rf, cl) stay
conventional TEMPLATE parameters.
Verified through the real load-path transforms (TemplateRewriter +
ExprPreprocessor) against a published remote dataset: dimensions 384,
train 100k, test 10k, k 100, dot_product, all spliced correctly.
The manifest's attributes block — distance_function and whatever else
the publisher recorded — was parsed and discarded; the Rust view
carries it for metadata accessors. TestDataView.attributes() returns
it on every profile view, which is how a workload derives
configuration from the dataset instead of restating it.
In this catalog format, ivec/ivecs records are length-qualified but
fixed throughout — ground-truth neighbor files — so a window maps at
the header stride like any other xvec, and only the *vvec extensions
carry variable-length records. Classifying ivec as variable sent
neighbor-facet windows through the offset-index path, and a remote
ground-truth file publishes no sidecar, so the window degraded and the
consent gate refused an eager warm-up of three records.
ElementType now recognizes the *vvec extension families, and the
variable-length test fixtures use them instead of overloading .ivecs.
Verified against the real remote dataset: a neighbor window resolves
to exactly its records' bytes and the reader decodes them.
The Rust source's is_vvec_ext still classifies ivec as variable; noted
in RUST_COMPATIBILITY.md for upstream reconciliation.
devdocs/vectordata/ holds the guide and remote_smoke_test.sh, which
checks base, query, and neighbor-index facets of one profile by
eagerly warming exactly the records each check reads — with the plan
announced and download progress on stderr — aborting on the first
failure. VECTORDATA_CATALOG and DATASET are required, DATASET as
'<dataset>' or '<dataset>:<profile>' with the profile defaulting to
'default', and NB selects how nosqlbench is invoked. The guide keeps
the Facet, windowed-background, and whole-profile warm-up forms as
separate examples.
Eager and background prefetches announce the plan's cost before any
bytes move, emit rate-limited progress on stderr while the download
runs — a meter, not a log event, so it shows regardless of logging
configuration — and confirm completion. Silent when there is nothing
to fetch, so local and already-warm facets add no noise.
Published sized profiles name only what differs: a windowed base
override and their own neighbor facets, with no extends key. Facets a
profile does not declare — query vectors included — resolve from
'default' implicitly, matching the expanded layout the publishing
toolchain emits into catalogs. Java honored only explicit extends, so
a released dataset failed with 'Profile 100k lacks facet
query_vectors' while its own catalog showed the facet plainly.
The window: key also accepts the serializer's structured form (a list
of {min_incl, max_excl} maps) and bare counts; dropping an
unrecognized form silently would read the whole facet where a window
was declared, so it fails instead.
Verified against the real remote dataset: profile 100k resolves all
four facets, reads query vectors, and clips the base to its window.
Every unit test now builds explicit VectorDataSettings with a temporary
cache directory instead of reaching VectorDataSettings.defaults(). The
defaults path deliberately refuses to auto-configure a cache when the
largest writable mount is outside the home filesystem — exactly the
shape of a GitHub runner — so any test depending on it fails there by
design. This broke the 5.25.14-release build, the first CI run to
execute these suites after they were tagged into the unit group.
Environment-variable catalog selection with an isolated VECTORDATA_HOME
and three stdout-driver one-liners: profile-window warm-up, a
caller-named ordinal range in the background, and demand-paged access.
Java API, expr, and inline-op examples for remote reads and for
precache/prefetch in both profile-window and caller-named ordinal-range
forms, plus the verification patterns the integration tests use.
Jonathan Shook
dependabot[bot]