mirror of
https://github.com/nosqlbench/nosqlbench.git
synced 2026-10-12 09:24:52 -05:00
docs(vectordata): record slab windowing and namespaces in the contract
- slab containers are planned, prefetched, and prebuffered by the pages a window spans, through the index in the slab's tail - a source may name a slab namespace, and the rule that keeps a URL scheme or drive letter from reading as one - slab records are not decoded here; the clarification names what a slab facet does and does not take part in
This commit is contained in:
1 parent
989e6e82b5
commit
9be7d0c0b5
2 files changed
+29
-3
No files matched your search
@@ -53,7 +53,10 @@ view.prebuffer(WholeFacetFallback.REFUSE, (cached, total) -> meter.update(cached
|
||||
|
||||
Every facet is planned before any is fetched. A declared window the format
|
||||
cannot map is refused under `REFUSE`, exactly as a requested window would be;
|
||||
`ALLOW` accepts the whole facet instead.
|
||||
`ALLOW` accepts the whole facet instead. Slab facets — paged metadata,
|
||||
addressable as `m.slab` or by namespace as `m.slab:content` — are planned
|
||||
and fetched by the pages a window spans, located through the index in the
|
||||
slab's tail; this module does not decode their records.
|
||||
|
||||
## Facets spread across several files
|
||||
|
||||
|
||||
@@ -27,6 +27,10 @@ implementation's current behavior and its format tests are normative.
|
||||
records with `IDXFOR__<file>.i32` or `.i64` index sidecars. Both sidecar
|
||||
layouts are accepted: `N+1` entries ending in an end-of-data sentinel,
|
||||
and the Rust walk-built form of `N` record starts with no sentinel.
|
||||
Slab containers (`.slab`) — paged records indexed by ordinal through a
|
||||
pages page in the file's tail, with optional named namespaces — are
|
||||
planned, prefetched, and prebuffered by the pages a window spans; their
|
||||
records are not decoded here (see below).
|
||||
- Multi-file facets: a facet may be a *series* of files forming one dense,
|
||||
gapless ordinal space, declared in either of the reference forms —
|
||||
uniform (`source: base__NNNN.fvec` with `shard_stride`, `shard_count`,
|
||||
@@ -58,7 +62,11 @@ implementation's current behavior and its format tests are normative.
|
||||
count. The serialized structured form — a list of `{min_incl, max_excl}`
|
||||
maps — and bare counts (`[0..N)`) are accepted on the `window:` key.
|
||||
Intervals that select no records are rejected at parse. Readers apply
|
||||
the first interval and clamp both bounds to the data.
|
||||
the first interval and clamp both bounds to the data. A source may name
|
||||
a slab namespace (`m.slab:content`, `m.slab:ns:[0..1K]`), or carry one
|
||||
beside it under `namespace:`/`ns:`; the text after the last `:` is a
|
||||
namespace only when the path before it has an extension and the text
|
||||
after names no directory, so a URL scheme or a drive letter never is.
|
||||
- Profiles: a non-default profile inherits unstated facets from the profile
|
||||
it names with `inherits:`, else from `default`; an unknown or self parent
|
||||
falls back to `default`, and a cycle leaves its members with what they
|
||||
@@ -81,7 +89,15 @@ implementation's current behavior and its format tests are normative.
|
||||
file. A window decomposes into one sub-window per shard it spans, each
|
||||
mapped by the format's rule against its own file; ranges in different
|
||||
shards never merge; prerequisite bytes sum over the files touched; a
|
||||
whole-facet request names every shard.
|
||||
whole-facet request names every shard. A slab window maps to the byte
|
||||
extent of the pages holding it — one contiguous range, since pages lie
|
||||
in ordinal order — through the index read from the slab's tail: the
|
||||
16-byte footer, the terminal pages page or namespaces page, the named
|
||||
namespace's pages page when indirected, and the last page's own record
|
||||
count. That bounded read, never a walk of the file, is what the plan
|
||||
reports as prerequisite bytes. A window starting past the end degrades
|
||||
rather than fabricating a range; one running past the end stops at the
|
||||
last page.
|
||||
- Whole-profile prebuffer: `prebuffer(WholeFacetFallback, PrebufferProgress)`
|
||||
plans every facet against the window it declares for itself, with the
|
||||
same shard-aware planner the selective prefetch uses, refusing a declared
|
||||
@@ -157,6 +173,13 @@ Remaining representation differences:
|
||||
`*vvec` extensions carry variable-length records. (The Rust source's
|
||||
`is_vvec_ext` still classifies `ivec` as variable; to be reconciled
|
||||
upstream.)
|
||||
- **Slab records are not decoded.** The reference reads slab records
|
||||
through its ANode codecs and binds them to operation parameters; this
|
||||
module carries none of that, so `openFacet` on a slab facet fails with
|
||||
an unsupported extension. Slab facets take part in everything that
|
||||
does not depend on what a record *is* — windows, shards, residency,
|
||||
planning, prefetch, and the whole-profile prebuffer — which is what a
|
||||
sized profile over a slab-backed dataset needs.
|
||||
- **A windowed reader's `prebuffer` fetches its window.** The reference's
|
||||
windowed reader inherits a no-op `precache`; here the reader a view hands
|
||||
back for a windowed facet warms the same bytes the whole-profile prebuffer
|
||||
|
||||
Reference in new issue
Block a user