An unversioned manifest is held to version 1 and refused when it carries a multi-file facet, the rule the loader took from the reference this morning. The fixture predates that rule and now states the version its content needs, as every sharded fixture in nb-vectordata does.
A filtered ground truth is only as deep as each predicate's match set, so a selectivity ladder cut for a full dataset walks off the bottom of a small slice and pads its rows with sentinels; the strict recall then caps at the profile's ceiling however well the search does. The guide derives that ceiling, tabulates the three measured runs, and states the regime: survey the ground truth before loading, report the attainable measure beside the strict one, and choose the profile by what the survey says it can pay out.
The workload surveys its ground truth in its derived parameters and refuses to run below min_attainable (default 0.95; 0 for a smoke run), and its verifier reports recall_attainable beside recall. The adapter docs state the regime: survey before loading, report the attainable measure beside the strict one, and choose the profile by what the survey says it can pay out, since selectivity times base count is only a first guess.
groundTruthCoverage reads the ground-truth facet alone and reports how many queries have all k neighbors, how many have none, and the recall@k ceiling a perfect search would score. requireAttainableRecall refuses a run below a threshold before a million rows are loaded for a number that could never be read, naming the ceiling and the profiles that would read. On tessera, 100k mixed ceils at 0.1428, 300k-uniform-2-1e-3 at 0.988, and 1m-uniform-2-1e-3 at 1.0.
A filtered ground truth pads a query whose predicate matches fewer than k rows with sentinels no search can return, and the strict recall divides by k regardless, so a small slice reads low by construction. attainable_recall leaves the sentinels out of the denominator: the share of the neighbors that exist which were found. A row with nothing attainable scores 1.0; how many such rows there are is a fact about the ground truth, not the search.
The compatibility contract moves to the reference's 33fedae commit and states the absent-version rule, stated parents, the derived axis, the tag schema, the selector grammar and cardinality, the group prebuffer, and the remaining representation differences. The README documents selecting profiles and the version-3 layout.
A group prebuffers the named profiles in the order a set selector resolved to, each facet against its declared window through the same planner, after tallying the announced total and reporting it as a large download at 250 MiB. One view per profile serves both the tally and the fetch, so the offset index a plan loads is the one the prebuffer uses.
A group answers what a selector reads of every profile, structure as loaded after inheritance and attributes its own, in size order: default first, then by base_count or the name read as a count, then naturally by name. A catalog open takes a selector and refuses a set, openProfiles returns the set, openProfile splits a spec by its head's shape, and a catalog entry resolves the same selection from its listed layout before anything is fetched, exposing the listed format version.
Version 3 is stated parents and a profile tag schema: every profile but default names a real parent or the load is refused naming the profiles involved, and the schema loads in naming order. An unversioned manifest is read as the least it could be and refused, naming the version to declare, when its content needs more. The axis of an inheritance step is derived from base_count rather than the parent's name: base facets are re-cut to the child's count, the per-size outputs, metadata_results now among them, do not cross a size step, and base_count and maxk inherit as the reference's do. A parent YAML reads as a number is the name it spells. Test fixtures that declare a series now state version 2.
The reference's one selector parser: the grammar, the value readings by spelling, the structural keys, list and map matching, case folding, and the set and single-profile resolution rules with their messages. A spec's head is found by shape, so a URL's port and a selector's colon are told apart. A regular expression outside the RE2 subset is refused rather than accepted here and rejected there.
The engine's default JSON schema reads a tilde as the string it spells; the reference writes a naming tag in a profile_tags schema as a tilde, and reads under the core schema, where it is null.
Derive the form survey into the description, note that the rampup
must be throttled against a single small node since an overloaded node
drops mutations that errors=count only counts, and say which of the
nine indexed columns carry the predicates so a lighter variant is an
informed choice.
PredicateClause reads every predicate once when it is built,
initializes each distinct shape then, and reports the forms on the
console with the number of predicates of each, so how many prepared
statements a run will hold is known before it starts. A shape that
cannot be prepared is reported with the others and refused after the
survey rather than on the cycle that would have met it. The
predicateForms expression takes the same survey quietly for a
workload's description.
The predicate's structure is reified through the template: the
predicate bind point takes a fragment, the adapter prepares one
statement per predicate shape keyed by its fingerprint, and the
comparands bind as parameters typed from the metadata fields. The
rendered form stays available through RecordText for a driver that
cannot prepare.
When a bind point of a prepared op yields a fragment, its text takes
the bind point's place in the statement and its values take the place
of that bind value. The space holds one prepared statement per op and
form key, prepared the first time the form is seen; a cycle without a
fragment takes the single-statement path unchanged. A template with one
fragment is keyed by the fragment's own key, so a cycle allocates no
key.
PredicateClause turns each predicate into a prepared fragment: the
shape's WHERE clause with a marker per comparand, the comparands typed
from the metadata fields they constrain, and the fingerprint as the
form key. A form is initialized once, its clause composed and its
binder compiled on the first predicate of that shape, and every later
cycle of the shape costs the record read, the fingerprint lookup, and
the value projection.
A caller that decodes a record itself, to read its fingerprint and pick
one binder per shape, binds the node directly with the same shape check
rather than decoding twice; the binder also names the shape it was
compiled against.
A bind value that is part statement and part parameters: text with
positional markers, the values for them, and the key of the form the
text has. Equal keys mean equal text, so an adapter prepares each form
once and binds every later fragment of that form through it.
Every dataset-shaped parameter is derived from the profile: the table's
metadata columns from the record layout, dimensions and k from the
facets, the similarity from the attributes. Rampup binds each base
vector with its metadata record field by field through a prepared
insert; search runs each query under the predicate published for it,
rendered as a WHERE clause since the predicates differ in shape, with
SAI indexes on every column they reference, and verifies against the
pre-filtered ground truth, padding a short result set the way the
ground truth pads a short match set.
RecordText renders each record of a facet in a named vernacular — the
rendering contract beside the binding one, for a statement whose shape
changes per cycle. CqlColumns derives CQL column definitions from a
record layout, typed from the bind types so the columns accept what the
binders produce, and the recordLayout, recordFields, and recordColumns
expression functions expose the layout to a workload at load time.
A CQL WHERE clause is a bare list of relations and Cassandra rejects a
conjunction wrapped in parentheses, which is what the reference's CQL
vernacular emits. Render the bare form for CQL, keep the grouping for
SQL and for the != and OR rewrites, and record the deviation in the
contract for upstream reconciliation.
Four mappers over a facet of opaque records, each learning the layout
once and resolving no field name per cycle: RecordField projects one
field to the Java value its wire type says, RecordFields binds a record
to named values in bind order, RecordForm binds under a form the facet
declares, and PredicateParams binds a predicate's comparands as
parameters typed from the metadata fields they constrain. The shared
projection maps bind types to driver-ready Java values, reading varint
and decimal payloads as big-endian two's complement.
State the binding contract and the bind-type asymmetry, replace the
note that binding forms are out of scope with the three deviations that
remain — field views as objects, form extras as node values, and the
raw predicate scanner not ported — and show the cycle loop.
Stefano Lottini
dependabot[bot]
Jonathan Shook