docs(vectordata): record the 2.1.2 baseline, selectors, and version 3

The compatibility contract moves to the reference's 33fedae commit and states the absent-version rule, stated parents, the derived axis, the tag schema, the selector grammar and cardinality, the group prebuffer, and the remaining representation differences. The README documents selecting profiles and the version-3 layout.

nb-vectordata/README.md

@@ -21,6 +21,35 @@ Local paths and HTTP(S) URLs are both accepted for manifests, catalogs, and
 facet sources. HTTP data is range-cached and uses a `.mref` sidecar whenever a
 server provides one.
 
+## Selecting profiles
+
+The second argument to `open` is a **selector**: a bare profile name as it
+always was, or an expression over the profile's automatic `profile` tag, its
+structural fields (`base_count`, `maxk`, `partition`, `inherits`), and its
+declared `attributes:`. `null` means `default`; `profile=*` means every
+profile. A single-profile surface refuses a set, and the set surface returns
+every match in the dataset's size order:
+
+```java
+TestDataView one = catalog.open("tessera", "size=10m,predicates=uniform-2");   // exactly one, or an error
+List<TestDataView> ladder = catalog.openProfiles("tessera", "family=uniform,selectivity=1e-3..1e-2");
+TestDataView spec = catalog.openProfile("https://host/ds:profile=^10m-.*$");    // the head is found by its shape
+TestDataGroup group = catalog.openGroup("tessera");
+List<String> names = group.select("or(10m,20m)");                              // size-ordered
+group.prebuffer(names, WholeFacetFallback.REFUSE, profile -> meter.forProfile(profile), total -> warn(total));
+```
+
+Values are read by their spelling — `^…`/`…$` a regular expression, `*`/`?`/`[`
+a glob, `lo..hi` a half-open interval, `10m`/`128mi`/`1e-3` numbers under the
+window count rule, `true`/`false`, anything else or anything quoted a literal
+— and every comparison folds case. A `,` is AND; `and(…)`, `or(…)`, and
+`not(…)` nest; an absent attribute matches nothing, `not(key=x)` included. A
+catalog entry resolves the same selector before anything is fetched
+(`entry.select(...)`) and states the dataset's `formatVersion()`, so a
+consumer can refuse a listed dataset it cannot read. A failure is a
+`SelectionException` whose `kind()` is `SYNTAX`, `NO_MATCH`, `AMBIGUOUS`, or
+`NO_DEFAULT`, with the profiles on offer in the message.
+
 ## Prefetching a window
 
 Any facet can be warmed for a record range the caller names — a profile's
@@ -154,7 +183,49 @@ series total and `get(o)` reads from the shard that owns ordinal `o`. A
 window in the facet's `window:` (or as a suffix on the uniform pattern) is
 in facet ordinals and clips the series, not a shard; a window on an explicit
 entry is in that file's ordinals and carves the shard out of it. Prefetch
-plans decompose a window across the shards it touches and fetch only those.
+plans decompose a window across the shards it touches and fetch only those,
+and a `FacetDescriptor` reports its `shardCount()` before the series is
+realized.
+
+## Format versions, layers, and tags
+
+`format_version` is a minimum reader requirement, and an absent field means
+`1` with the dataset held to it: a manifest that never said what it is is
+read as single-file, pre-inheritance, pre-tag, and content that needs more
+is refused naming the version to declare. Version `2` is a multi-file facet;
+version `3` is a dataset whose profiles state their parents — `inherits:`
+on every profile but `default`, every parent real — and may declare a
+`profile_tags` schema:
+
+```yaml
+format_version: 3
+profile_tags:
+  size: ~                # a naming tag: no default, set per profile
+  predicates: ~
+  family: stratified     # carried by every profile, but not a name
+profiles:
+  default:
+    base_vectors: base.fvec
+    metadata_predicates: profiles/base/predicates.slab
+  10m:                   # a size layer: the unfiltered benchmark at ten million
+    inherits: default
+    base_count: 10000000
+    neighbor_indices: profiles/10m/neighbor_indices.ivecs
+    attributes: { size: 10m }
+  10m-uniform-2-1e-2:    # a predicate set at the layer's size
+    inherits: 10m
+    metadata_predicates: profiles/10m-uniform-2-1e-2/predicates.slab
+    metadata_results: profiles/10m-uniform-2-1e-2/results.slab
+    attributes: { size: 10m, predicates: uniform-2, selectivity: 1e-2, family: uniform }
+```
+
+What a profile inherits depends on the axis of the step, which is derived
+from `base_count`: a child whose count differs from its parent's is a size
+step — the base facets arrive re-cut to `[0..base_count)`, and the per-size
+outputs (ground truth, `metadata_results`) do not cross — while a child at
+its parent's count inherits everything it does not override, `base_count`
+included. `group.profileTags()` and `group.formatVersion()` expose the
+schema and the version a caller may need before choosing a profile.
 
 For a release canary against a Rust-hosted dataset, run Maven with
 `-Dvectordata.canary.catalog=<catalog-url>` and

nb-vectordata/RUST_COMPATIBILITY.md

@@ -1,7 +1,7 @@
 # vectordata-rs compatibility contract
 
 This module implements the programmatic dataset-access surface of
-`vectordata-rs` (baseline: crate 2.1.1, source commit `53f95ab2005d0f11266566d2ceec524d47f3a0b5`).
+`vectordata-rs` (baseline: crate 2.1.2, source commit `33fedae` — "Upload on the streams --concurrency promises").
 When this document and an older Java implementation disagree, the Rust
 implementation's current behavior and its format tests are normative.
 
@@ -16,13 +16,23 @@ implementation's current behavior and its format tests are normative.
   order: `dataset.yaml`, then `knn_entries.yaml` — the fallback applies only
   when `dataset.yaml` is absent; one that exists and is refused stays refused,
   with its own diagnosis.
-- Format version: an optional top-level `format_version`. Absent means `1`,
-  which is every dataset written before the field existed. The number is a
-  minimum reader requirement: a dataset above `FormatVersion.SUPPORTED` (`2`)
-  is refused naming both numbers before anything else is read, and a stated
-  version below what the content requires — `1` on a sharded dataset — is
-  refused as a declaration that understates what it holds. An absent field
-  is not a claim, and a generous one is accepted.
+- Format version: an optional top-level `format_version`. The number is a
+  minimum reader requirement: a dataset above `FormatVersion.SUPPORTED`
+  (`3`) is refused naming both numbers before anything else is read, and a
+  stated version below what the content requires — `1` on a sharded
+  dataset, `2` on one that names a parent — is refused as a declaration
+  that understates what it holds. An **absent** field means `1` and the
+  dataset is held to it: an unversioned manifest is read as pre-shard,
+  pre-inheritance, pre-tag, and content that needs more is refused naming
+  the version to declare (`declare format_version: 2`). A generous version
+  is accepted. Version `2` is what a multi-file facet needs; version `3`
+  is what a `profile_tags` schema or a profile naming a parent other than
+  `default` needs. The version a dataset states is exposed on
+  `TestDataGroup.formatVersion()` and, from a catalog listing's `layout`,
+  on `CatalogEntry.formatVersion()`, so a caller can decide before it
+  fetches. YAML is read under the 1.2 core schema the reference writes,
+  so `~` is null — a naming tag in a schema — rather than the string it
+  spells.
 - Data: scalar, fixed-dimension xvec records, and variable-dimension vvec
   records with `IDXFOR__<file>.i32` or `.i64` index sidecars. Both sidecar
   layouts are accepted: `N+1` entries ending in an end-of-data sentinel,
@@ -68,22 +78,81 @@ implementation's current behavior and its format tests are normative.
   namespace only when the path before it has an extension and the text
   after names no directory, so a URL scheme or a drive letter never is.
 - Profiles: a non-default profile inherits unstated facets from the profile
-  it names with `inherits:`, else from `default`; an unknown or self parent
-  falls back to `default`, and a cycle leaves its members with what they
-  declare. What inherits depends on the axis. Across the size axis (parent
-  `default`) `base_vectors` and `metadata_content` inherit under the child's
-  `base_count` window unless already windowed, while the neighbor facets do
-  not — ground truth is derived from `base_count`, so a sized profile that
-  omits its own fails with "lacks facet" rather than serving the full
-  base's. Across any other axis every facet is invariant and inherits as
-  is. A `partition: true` profile is an oracle partition with independent
-  base vectors and inherits nothing.
+  it names with `inherits:`, else from `default`. What inherits depends on
+  the axis, and **the axis is derived from `base_count`**: a step is on
+  the size axis when the child's count differs from its parent's effective
+  one, whatever the parent is called, and at one size otherwise. Across
+  the size axis `base_vectors` and `metadata_content` inherit **re-cut**
+  to `[0..base_count)` — a window the parent carries is replaced, so a
+  `20m` built on `10m` reads the first twenty million of the same file —
+  while the per-size outputs do not cross: the neighbor facets and
+  `metadata_results`, all derived from `base_count`, so a sized profile
+  that omits its own fails with "lacks facet" rather than serving the
+  full base's. Across a step at one size every facet is invariant and
+  inherits as is, and so does `base_count`; `maxk` crosses every step.
+  A `partition: true` profile is an oracle partition with independent
+  base vectors and inherits nothing. A parent YAML reads as a number — a
+  rung named `100` — is the name it spells.
+- Stated parents: from `format_version` 3 every profile other than
+  `default` states `inherits:`, and a stated 3 is a claim that every
+  parent is real — an absent parent ("names no parent; state `inherits:
+  default` or the layer it builds on"), an unknown or self parent, a
+  cycle (named once by its sorted members), `default` naming a parent,
+  and `partition: true` beside `inherits:` are load refusals naming the
+  profiles involved, joined by `; `. Below 3 an unknown or self parent
+  falls back to `default` and a cycle leaves its members with what they
+  declare; `ProfileParents.advisories` phrases the same conditions as
+  what version 3 will refuse. Because a parent other than `default`
+  itself requires version 3, those fallbacks are reachable only through
+  in-memory legacy groups, as in the reference.
+- Profile tags: an optional top-level `profile_tags` schema, naming tags
+  in order, each with a default or `~` for a naming tag, loaded as
+  written on `TestDataGroup.profileTags()` and from a catalog listing on
+  `CatalogEntry.profileTags()`. Its presence requires version 3.
+- Selectors: `ProfileSelector` is the reference's one parser — the
+  grammar (`,` is AND; `and(…)`, `or(…)`, `not(…)`; a bare name is
+  `profile=<name>` with a literal value), the value readings by spelling
+  (regex by `^`/`$`, glob by `*`/`?`/`[`, `lo..hi` interval, number under
+  the window count rule, boolean, literal; quoted is literal; every
+  reading folds case), the structural keys read before attributes
+  (`profile`, `base_count`, `maxk`, `partition`, `inherits`), dotted keys
+  into map attributes, lists matching on any element, absence matching
+  nothing under any operator, a comparison against a non-number being
+  false rather than an error, and a `<`/`>` against a pattern being a
+  parse error. `ProfileFacts` is what a selector reads: structure **as
+  loaded** after inheritance, attributes the profile's own. `resolve`
+  yields the set in the group's size-sorted order — `default` first, then
+  by `base_count` or the name read as a count, then naturally by name —
+  with `null` meaning `default`, `profile=*` meaning all, and no match
+  failing with the offer listed; `resolveOne` refuses a set. Every
+  failure is a `SelectionException` of kind `SYNTAX` (with position),
+  `NO_MATCH`, `AMBIGUOUS`, or `NO_DEFAULT`, with the reference's message.
+  `TestDataGroup.select`/`selectOne`/`profileFacts`/`profileNames`,
+  `CatalogEntry.select`/`selectOne` (structure read through the declared
+  `inherits` chain: `maxk` across any parent, `base_count` only across a
+  named non-default one), `Catalog.open(dataset, selector)` (one
+  profile, `null` for `default`), `Catalog.openProfiles(dataset,
+  selector)` (the set), and `DatasetSpec` — the head found by shape (a
+  URL's `://` and port, a drive letter), the selector starting at the
+  first colon after it, a malformed selector reported at its position in
+  the spec — replace the last-colon split. A bare name still opens the
+  one profile it names on every surface, case folded.
+- Group prebuffer: `TestDataGroup.prebuffer(names, fallback, progress,
+  largeDownload)` prebuffers the named profiles in order — what a set
+  selector resolved to — each facet against its declared window through
+  the same planner, after tallying the announced total and reporting it
+  when it reaches `PREBUFFER_LARGE_WARNING_BYTES` (250 MiB);
+  `prebufferAll` is the same over every profile.
 - Prefetch: caller-supplied record windows on any facet via
   `prefetchPlan`, `prefetch`, and `prefetchInBackground` on
   `TestDataView`, with `WholeFacetFallback` consent gating, chunk-level
   `RangeFill` residency accounting, chunk-adjacency range coalescing, an
   offset-index cache scoped to the view's facet handle, and an empty
-  window meaning the whole facet (a request, never a degrade). Plan ranges
+  window meaning the whole facet (a request, never a degrade). A
+  `FacetDescriptor` answers `shardCount()` — a uniform series'
+  `shard_count`, `null` for a single file and for the explicit form,
+  whose files are its entries — so a caller can report what opening a
+  facet costs without realizing the series. Plan ranges
   are `ShardRange`s, qualified by the shard they lie in — a single file is
   shard `0` — because across a series the same byte offset exists in every
   file. A window decomposes into one sub-window per shard it spans, each
@@ -242,6 +311,13 @@ consent gate instead), sentinel-tolerant sidecar parsing with
 and atomic persistence of locally walked indexes — follow the Rust
 source directly as of the baseline commit above.
 
+The reference's writer-side surfaces — layering a generated rung into a
+size layer and a `-mixed` set, naming generated profiles from their
+tags, the textual `dataset.yaml` editors, `veks check` advisories, push
+and precache commands — are dataset authoring, not the access API, and
+are not ported; the loader-side rules they depend on (stated parents,
+the derived axis, tag schemas, selectors) are.
+
 Remaining representation differences:
 
 - **`ivec`/`ivecs` facets are uniform-stride.** The format requires
@@ -262,6 +338,16 @@ Remaining representation differences:
   relies on its JSON library's map, whose order is that library's choice.
   Numbers follow the reference's inference: no point or exponent and fits
   a long is an integer, anything else numeric is a float.
+- **A regular expression outside RE2 is refused at parse.** The
+  reference evaluates the Rust `regex` crate's RE2 subset; Java's engine
+  accepts more, so a lookaround (`(?=`, `(?!`, `(?<=`, `(?<!`), an atomic
+  group, or a backreference is refused as a syntax error rather than
+  accepted here and rejected there. Everything inside the subset is
+  matched case-insensitively against the whole canonical text with
+  `find`, as `is_match` does.
+- **An explicit series answers no `shardCount()`.** The reference's
+  `shard_count()` is `Some` only for the uniform form; the explicit form's
+  file count is `series().entries().size()`.
 - **A bound field is a small view object.** The reference hands out a
   borrowed `Field` by value; here each field the walk meets is one
   object holding offsets into the record's bytes, and its primitive
@@ -294,6 +380,11 @@ Remaining representation differences:
   would — its declared window, planned per shard — rather than the file it
   was cut from. A sharded reader's own `prebuffer` fetches every file whole,
   as the reference's does; the window-scoped fetch is the view's.
+- **The group prebuffer's progress is per profile.** The reference's
+  callback takes `(profile, facet, progress)`; here `progress` is a
+  function from the profile name to the `PrebufferProgress` that profile's
+  facets report through, since the view-level prebuffer reports byte
+  totals without naming the facet.
 - **A non-range-capable remote file degrades a window** that touches it,
   rather than planning a partial fetch it cannot perform. Rust reaches the
   same end state by downloading such a file whole at open; Java opens