A filtered ground truth pads a query whose predicate matches fewer than k rows with sentinels no search can return, and the strict recall divides by k regardless, so a small slice reads low by construction. attainable_recall leaves the sentinels out of the denominator: the share of the neighbors that exist which were found. A row with nothing attainable scores 1.0; how many such rows there are is a fact about the ground truth, not the search.
The compatibility contract moves to the reference's 33fedae commit and states the absent-version rule, stated parents, the derived axis, the tag schema, the selector grammar and cardinality, the group prebuffer, and the remaining representation differences. The README documents selecting profiles and the version-3 layout.
A group prebuffers the named profiles in the order a set selector resolved to, each facet against its declared window through the same planner, after tallying the announced total and reporting it as a large download at 250 MiB. One view per profile serves both the tally and the fetch, so the offset index a plan loads is the one the prebuffer uses.
A group answers what a selector reads of every profile, structure as loaded after inheritance and attributes its own, in size order: default first, then by base_count or the name read as a count, then naturally by name. A catalog open takes a selector and refuses a set, openProfiles returns the set, openProfile splits a spec by its head's shape, and a catalog entry resolves the same selection from its listed layout before anything is fetched, exposing the listed format version.
Version 3 is stated parents and a profile tag schema: every profile but default names a real parent or the load is refused naming the profiles involved, and the schema loads in naming order. An unversioned manifest is read as the least it could be and refused, naming the version to declare, when its content needs more. The axis of an inheritance step is derived from base_count rather than the parent's name: base facets are re-cut to the child's count, the per-size outputs, metadata_results now among them, do not cross a size step, and base_count and maxk inherit as the reference's do. A parent YAML reads as a number is the name it spells. Test fixtures that declare a series now state version 2.
The reference's one selector parser: the grammar, the value readings by spelling, the structural keys, list and map matching, case folding, and the set and single-profile resolution rules with their messages. A spec's head is found by shape, so a URL's port and a selector's colon are told apart. A regular expression outside the RE2 subset is refused rather than accepted here and rejected there.
The engine's default JSON schema reads a tilde as the string it spells; the reference writes a naming tag in a profile_tags schema as a tilde, and reads under the core schema, where it is null.
Derive the form survey into the description, note that the rampup
must be throttled against a single small node since an overloaded node
drops mutations that errors=count only counts, and say which of the
nine indexed columns carry the predicates so a lighter variant is an
informed choice.
PredicateClause reads every predicate once when it is built,
initializes each distinct shape then, and reports the forms on the
console with the number of predicates of each, so how many prepared
statements a run will hold is known before it starts. A shape that
cannot be prepared is reported with the others and refused after the
survey rather than on the cycle that would have met it. The
predicateForms expression takes the same survey quietly for a
workload's description.
The predicate's structure is reified through the template: the
predicate bind point takes a fragment, the adapter prepares one
statement per predicate shape keyed by its fingerprint, and the
comparands bind as parameters typed from the metadata fields. The
rendered form stays available through RecordText for a driver that
cannot prepare.
When a bind point of a prepared op yields a fragment, its text takes
the bind point's place in the statement and its values take the place
of that bind value. The space holds one prepared statement per op and
form key, prepared the first time the form is seen; a cycle without a
fragment takes the single-statement path unchanged. A template with one
fragment is keyed by the fragment's own key, so a cycle allocates no
key.
PredicateClause turns each predicate into a prepared fragment: the
shape's WHERE clause with a marker per comparand, the comparands typed
from the metadata fields they constrain, and the fingerprint as the
form key. A form is initialized once, its clause composed and its
binder compiled on the first predicate of that shape, and every later
cycle of the shape costs the record read, the fingerprint lookup, and
the value projection.
A caller that decodes a record itself, to read its fingerprint and pick
one binder per shape, binds the node directly with the same shape check
rather than decoding twice; the binder also names the shape it was
compiled against.
A bind value that is part statement and part parameters: text with
positional markers, the values for them, and the key of the form the
text has. Equal keys mean equal text, so an adapter prepares each form
once and binds every later fragment of that form through it.
Every dataset-shaped parameter is derived from the profile: the table's
metadata columns from the record layout, dimensions and k from the
facets, the similarity from the attributes. Rampup binds each base
vector with its metadata record field by field through a prepared
insert; search runs each query under the predicate published for it,
rendered as a WHERE clause since the predicates differ in shape, with
SAI indexes on every column they reference, and verifies against the
pre-filtered ground truth, padding a short result set the way the
ground truth pads a short match set.
RecordText renders each record of a facet in a named vernacular — the
rendering contract beside the binding one, for a statement whose shape
changes per cycle. CqlColumns derives CQL column definitions from a
record layout, typed from the bind types so the columns accept what the
binders produce, and the recordLayout, recordFields, and recordColumns
expression functions expose the layout to a workload at load time.
A CQL WHERE clause is a bare list of relations and Cassandra rejects a
conjunction wrapped in parentheses, which is what the reference's CQL
vernacular emits. Render the bare form for CQL, keep the grouping for
SQL and for the != and OR rewrites, and record the deviation in the
contract for upstream reconciliation.
Four mappers over a facet of opaque records, each learning the layout
once and resolving no field name per cycle: RecordField projects one
field to the Java value its wire type says, RecordFields binds a record
to named values in bind order, RecordForm binds under a form the facet
declares, and PredicateParams binds a predicate's comparands as
parameters typed from the metadata fields they constrain. The shared
projection maps bind types to driver-ready Java values, reading varint
and decimal payloads as big-endian two's complement.
State the binding contract and the bind-type asymmetry, replace the
note that binding forms are out of scope with the three deviations that
remain — field views as objects, form extras as node values, and the
raw predicate scanner not ported — and show the cycle loop.
The scaffolding projective functions build on: a Layout learned once
from a facet's first record, a Binder compiled against it that binds
by position with parameter renames applied at compile time, forms read
from the facet's forms namespace with one implicit form when it has
none, and a PredicateBinder that types each condition from the field's
tag rather than the comparand's variant and refuses a record whose
leader byte or shape disagrees with the template. BindType is its own
exhaustive mapping from the tag, distinct from the rendering one.
Port the reference's raw scan: a record's fields reached as views over
its own bytes, names skipped rather than copied, every tag advanced at
its true width. A Field reads its payload through accessors typed by
the wire form — a Half as a float, the temporal trio as text, Nanos as
two numbers — or materializes the AST view on request. Schema discovery
and predicate flattening ride on the same walk.
State the record, wire-format, vernacular, and facet-shape parts of the
contract, and replace the note that slab records are not decoded with
the three deviations that remain: a tree codec in place of serde,
document-order JSON parsing, and binding forms out of scope.
A slab facet opens through openFacetRecords as a RecordFacet, one
container per file with each shard of a series based at zero, and a
codec applied to it yields a typed reader. Every extension the spec
names carries a facet shape, facetShape answers it, and each reader
refuses the other shape by naming the one that opens it. Reading one
remote record costs its page, not the file.
Stage 2 of the codec: the thirteen vernaculars the reference names,
each rendering both node kinds in its own syntax. JSON, YAML, SQL, CQL,
CDDL, and readout parse back with the reference's type inference; the
schema, value, and display forms refuse to, naming themselves. The tree
form stands in for a serde target.
Port the reference's stage-1 codec: an MNode as a leader byte, a field
count, and tagged values over the 29 type tags; a PNode as a pre-order
predicate tree in the indexed, named legacy, and named typed wire forms.
Fingerprints and congruence compare shape without values, the Display
grammar reads back through PNodeDisplay, and PredicateEvaluator applies
a tree to a record with the reference's coercions.
- the canary asserted only that the base had records; it now reads
base records at both ends of the profile's window, across a shard
seam when the base is a series, a query, and a neighbor row whose
entries name base records
- every facet is planned against the window it declares, a slab's
included, and none may degrade to a whole-facet fetch — the case a
sized profile over a slab-backed dataset presents
- a small base window is prefetched and read back resident, and the
run prints what it found so a release log carries the numbers
- a binding window over a series warms the shards it spans and still
addresses records by absolute ordinal across the seam
- a background prefetch over a series plans one request per shard
- a sized profile over a series clips at its base_count, serves its
own ground truth, and inherits queries from default
- a series whose shards disagree on shape is refused naming the odd
one; a gap in the middle of a series is named, never skipped
- two shards sliced from one file share one storage and address their
own windows of it; a local series reports local and complete
- the weakest access mode is the one true of every file, and a series
with one unverified shard reports it over HTTP; a range-less server
hands each shard over whole and the series still reads as one
- a mapped series window is prefetched without a fallback in play
- the whole-profile prebuffer visits slab facets whole, a sharded slab
shard by shard, and an explicit entry that slices a slab plans its
pages by file ordinal
- slab containers are planned, prefetched, and prebuffered by the pages
a window spans, through the index in the slab's tail
- a source may name a slab namespace, and the rule that keeps a URL
scheme or drive letter from reading as one
- slab records are not decoded here; the clarification names what a
slab facet does and does not take part in
- a slab window still reported degrades_to_full_download, so a sized
profile over a slab-backed dataset was refused by the whole-profile
prebuffer where the reference fetches the pages the window spans
- the slab's page index is read from the tail: the 16-byte footer, the
terminal pages page or namespaces page, the named namespace's pages
page when indirected, and the last page's own record count — a
bounded read of two short pages, never a walk of the file, reported
as prerequisite bytes
- a window maps to the byte extent of the pages holding it, one
contiguous range since pages lie in ordinal order; a window running
past the end stops at the last page, one starting past the end
degrades rather than fabricating a range
- tests write slabs in the slabtastic layout, single- and
multi-namespace, and check every ordinal against its page, the
seams, the planner's bounds, a slab series across its seam, and over
HTTP that ten records fetch a page and a sized profile fetches only
its window's pages
- a source may name a namespace within a slab — `m.slab:content`,
`m.slab:ns:[0..1K]` — or carry one beside it under namespace:/ns:,
which applies to every source that names none of its own
- the text after the last `:` is a namespace only when the path before
it has an extension and the text after names no directory, so a URL
scheme or a drive letter never reads as one
- the namespace travels with the facet descriptor and with each shard
entry, and the per-format mapping is addressed by the locator — the
file plus its namespace — rather than the file alone
- baseline moves from crate 1.8.0 to 2.1.1; the contract records the
series model, the two window coordinate spaces, format_version,
the `=count` suffix, axis-aware inheritance, shard-qualified plans,
the window-aware whole-profile prebuffer, and the shard error
taxonomy
- clarifications named where Java differs: a windowed reader's
prebuffer fetches its window where the reference's is a no-op, and
evicting a file from a series' open-file budget releases a
reference rather than closing shared storage
- the README shows the uniform and explicit declaration forms and the
whole-profile prebuffer
- the whole-profile prebuffer fetched every facet's file whole, so a
sized profile over a large base downloaded the whole base while the
selective prefetch beside it fetched only the window
- prebuffer(fallback, progress) now plans every facet with the
shard-aware planner against the window it declares for itself, a
series included, and refuses a declared window the format cannot
map under REFUSE exactly as a requested window is; every facet is
planned before any is fetched, and prebuffer(progress) is the
REFUSE form
- the reader a view hands back for a windowed facet warms its window
rather than the file it was cut from
- remote tests: a 500-record window over a three-shard base leaves the
last shard untouched, through the view, through an inherited
base_count window, and through the reader
- neighbor facets no longer cross the size axis: a sized profile that
omits its own ground truth fails with "lacks facet" instead of
silently serving the full base's, since ground truth is derived
from base_count
- base_vectors and metadata_content inherit under the child's
base_count window unless the parent's already carries one
- `inherits:` names a parent, with `extends:` kept as a spelling; an
unknown or self parent falls back to default, and a cycle leaves its
members, and anything inheriting from them, with what they declare
- a `partition: true` profile is an oracle partition with independent
base vectors and inherits nothing
- a facet may be a series: the uniform form names `base__NNNN.fvec`
with shard_stride, shard_count, and record_count; the explicit form
lists entries in the source grammar, each bare, `=`-counted,
windowed, or both, checked against each other
- ordinals map in O(1) whenever the lengths are uniform however the
series was spelled, and by prefix-sum search otherwise; an entry
window is in that file's ordinals, the facet window is in facet
ordinals, and a suffix on a uniform pattern is the facet window
- storage, sidecars, and offset indexes are per file, so two shards of
one file share one of each; the readers present a single surface,
refuse shards that disagree on shape, and name a missing shard
rather than reading a shorter facet
- prefetch plans carry ShardRange in place of ByteRange, decompose a
window across the shards it spans, never merge across shards, and
sum prerequisite bytes per file; a whole-facet request names every
shard, and cache statistics report the weakest access mode
- format_version is a minimum reader requirement: a dataset above 2,
or one stating less than its content needs, is refused naming both
numbers; declarations are checked for self-consistency at load, and
a directory falls back to knn_entries.yaml only when dataset.yaml is
absent, so a refusal keeps its diagnosis
- a series holds open no more files than a quarter of the descriptor
soft limit, floor 8, or VECTORDATA_SHARD_FD_CAP
The ISO forms of LocalTime and LocalDateTime omit the seconds field
when the second is 00, so currentTime(), currentDateTime(),
currentTimeInZone() and millisToDateTime() each returned a shorter
string once a minute — HH:mm instead of HH:mm:ss. That failed
testCurrentTime in CI roughly one run in sixty, and silently handed
workloads a differently-shaped timestamp the rest of the time.
Each now formats with an explicit pattern rather than toString(), which
also makes the truncatedTo(SECONDS) calls redundant.
The new test pins the boundary through millisToDateTime, whose instant
comes from the caller: the wall-clock functions can only be asserted
for shape, which catches this one second in sixty. It builds a
timestamp on a whole minute, requires the seconds to be there, and
repeats at second 30 so a fix in the other direction is caught too.
A binding window re-based the reader to zero, so BaseVectors(spec,
'[50000..100000)') exposed indices 0..49999. A workload running the
cycles that window names then indexed past the end and failed, and it
only ever worked from cycle 0, where the re-basing is the identity. It
also disagreed with the two surfaces beside it: VariableFacet and
prefetchCycles both speak absolute ordinals.
The window now selects which records to warm and nothing else. Indices
are absolute everywhere, so cycles address the records they name, and
the fetch still covers only the window — a [50000..100000) slice of a
100k profile warms 74 MiB rather than the full 146 MiB.
Reads outside the window demand-page rather than failing, which is
load-bearing rather than lenient: VirtData probes a newly resolved
function with a small sample index before any cycle runs, so refusing
out-of-window reads leaves a window above zero unable to resolve at
all. A test pins that.
WindowedReader keeps its restricting behaviour for the windowedFacet
expression function, where a caller asks for a restricted reader
directly and no resolver probe is involved.
Asking for a prefetch with progress meant asking for prefetch=false and
then a binding mode — two knobs that read as contradicting each other,
because the background expr function was the one prefetch form that
never got a meter, so the only way to see a background download was to
route it through a binding.
prefetchBackground now announces its plan and watches the handle like
every other form, and prefetchCyclesBackground takes the ordinal range
directly. With the API reachable at the surface, the demo asks for what
it wants in one word: prefetch=load blocks on prefetchCycles before the
run, prefetch=background hands back a metered handle that fetches while
the cycles run, prefetch=none leaves reads to demand-page. The binding
mode knob is gone.
The documented 'nb5 run vectordata_demo' does not work: run is itself a
command, so the workload name is parsed as a second one and resolves to
no service. The scenario form drops the run — 'nb5 vectordata_demo
dataset=<name>:<profile>' — and the long form names the full bundled
path, 'nb5 run workload=activities/examples/vectordata_demo.yaml'.
Both are now shown in the workload and the guide.
Verified against the 5.25.15-release binary: the workload is bundled
and listed, both corrected forms load it, and a real dataset run meters
the download and reads base, query, and neighbor records.
The demo only worked from a checkout. It now ships in the stdout
adapter's activities, so 'nb5 run vectordata_demo dataset=<name>:<profile>'
works from the distribution alone, with --cat and --copy for reading or
editing it.
Every catalog-touching expression is guarded on the dataset parameter
being given. Bundled workloads are parsed through the expression
pipeline by --list-workloads and --list-scenarios, so a workload that
opened a catalog unconditionally would fail listing for every user
without one configured; with no dataset named it now renders usage text
and touches nothing.
Jonathan Shook