feat(virtdata): meter the background prefetch and reach it directly

Asking for a prefetch with progress meant asking for prefetch=false and
then a binding mode — two knobs that read as contradicting each other,
because the background expr function was the one prefetch form that
never got a meter, so the only way to see a background download was to
route it through a binding.

prefetchBackground now announces its plan and watches the handle like
every other form, and prefetchCyclesBackground takes the ordinal range
directly. With the API reachable at the surface, the demo asks for what
it wants in one word: prefetch=load blocks on prefetchCycles before the
run, prefetch=background hands back a metered handle that fetches while
the cycles run, prefetch=none leaves reads to demand-page. The binding
mode knob is gone.

devdocs/vectordata/vectordata_remote_access.md

@@ -58,19 +58,23 @@ The meter goes to stderr while the bytes move:
 [vectordata] mydataset:myprofile:base_vectors: fetch complete (29.0 MiB)
 ```
 
-To watch the fetch overlap the run instead of preceding it, turn off the
-load-time prefetch and let the binding warm in the background:
+To watch the fetch overlap the run instead of preceding it, ask for the
+background form — give it a range big enough and enough cycles to
+outlast the download:
 
 ```bash
 nb5 vectordata_demo dataset=mydataset:myprofile \
-  prefetch=false mode=background cycles=1000
+  prefetch=background records=100000 cycles=200
 ```
 
-Parameters: `cycles`, `records` (how many ordinals the demo spans),
-`prefetch` (`true|false`), and `mode` (`none|eager|background`). The
-meter is silent when there is nothing to fetch, so a second run against
-a warm cache prints nothing — point `VECTORDATA_HOME` at an empty
-directory to see it fetch again.
+`prefetch` selects which prefetch call does the work: `load` (the
+default) blocks on `prefetchCycles` before cycle 0, `background` hands
+back a handle from `prefetchCyclesBackground` and meters it while the
+run proceeds, and `none` leaves reads to demand-page. `records` sets how
+many ordinals the demo spans, `cycles` how long it runs. The meter is
+silent when there is nothing to fetch, so a second run against a warm
+cache prints nothing — point `VECTORDATA_HOME` at an empty directory to
+see it fetch again.
 
 Beyond the script's checks:
 
@@ -200,7 +204,10 @@ passes through untouched. The vectordata expr functions:
 # {{= prefetchPlan("example:default", "base_vectors", "[0..100k)").bytesToFetch()}}
 # {{= prefetch("example:default", "base_vectors", "[0..100k)").rangesFetched()}}
 
-# Background warm-up captured in an expr variable, joined later:
+# The same, in the background: the handle comes back immediately and
+# the fetch overlaps whatever runs next. Both background forms meter
+# from the handle, so the download reports itself either way.
+# {{= prefetchCyclesBackground("example:default", "base_vectors", 0, 1000000).plan().requests()}}
 # {{warmup = prefetchBackground("example:default", "base_vectors", "[0..1M)")}}
 # {{= warmup.join().rangesFetched()}}
 

nb-adapters/adapter-stdout/src/main/resources/activities/examples/vectordata_demo.yaml

@@ -7,11 +7,6 @@ description: |
 
     nb5 vectordata_demo dataset=<name>:<profile>
 
-  That is the scenario form — the workload name stands alone, without a
-  `run` in front of it. The equivalent long form names the full path:
-
-    nb5 run workload=activities/examples/vectordata_demo.yaml dataset=<name>:<profile>
-
   Point it at a catalog first, either of these:
 
     export VECTORDATA_CATALOG='https://your.host/path/catalog.yaml'
@@ -23,6 +18,7 @@ description: |
 
     dataset:       {{spec == paramOr('dataset','MISSING-dataset-parameter')}}
     records:       {{records == paramOr('records','20000') as Long}}
+    prefetch:      {{prefetchMode == paramOr('prefetch','load')}}
   {{= spec == 'MISSING-dataset-parameter' ? '  (pass dataset=<name>:<profile> to derive the rest from a dataset)' :
       '  dimensions:    ' + baseVectors(spec).dimension() + '\n' +
       '    base records:  ' + baseVectors(spec).count() + '\n' +
@@ -30,29 +26,42 @@ description: |
       '    neighbors k:   ' + neighborIndices(spec).dimension() + '\n' +
       '    similarity:    ' + dataset(spec).attributes().getOrDefault('distance_function','unspecified')}}
 
-  The demo works within the first `records` ordinals of the base facet,
-  and something has to fetch them. That fetch is announced, metered
-  while it runs, and confirmed — all on stderr, which is where the
-  progress shows up:
+  The demo reads the first `records` ordinals of the base facet, and
+  `prefetch` chooses which prefetch call fetches them. Both fetching
+  forms report the same way — the plan is announced, progress is metered
+  while the bytes move, and completion is confirmed, all on stderr:
 
-  {{= spec == 'MISSING-dataset-parameter' || paramOr('prefetch','true') != 'true' ? '  warm at load: skipped' :
-      '  warm at load: ' + prefetchCycles(spec,'base_vectors',0,records).rangesFetched() + ' range(s) fetched'}}
+    prefetch=load        prefetchCycles(...) — fetches before the run
+                         starts, so cycle 0 reads warm data. Blocks
+                         until resident. This is the default.
+    prefetch=background  prefetchCyclesBackground(...) — returns a
+                         handle immediately and fetches while the cycles
+                         run, so reads and the download overlap.
+                         Progress is polled from the handle. Give it
+                         enough cycles to outlast the download, or the
+                         run ends first and the fetch stops partway.
+    prefetch=none        no warm-up; reads demand-page whatever they
+                         touch. Nothing to meter, so it stays quiet.
+
+  {{= spec == 'MISSING-dataset-parameter' ? '  warm-up: skipped (no dataset given)' :
+      prefetchMode == 'load' ? '  warm-up: fetched ' + prefetchCycles(spec,'base_vectors',0,records).rangesFetched() + ' range(s) before the run' :
+      prefetchMode == 'background' ? '  warm-up: started in the background, ' + prefetchCyclesBackground(spec,'base_vectors',0,records).plan().requests() + ' range(s) planned' :
+      '  warm-up: none, reads demand-page'}}
 
   Two ways to watch it:
 
-    # 1. warm at load, then read — the meter runs before any cycles
+    # fetch up front, then read — the meter runs before any cycles
     nb5 vectordata_demo dataset=<name>:<profile>
 
-    # 2. warm in the background while cycles run — turn off the
-    #    load-time fetch so the binding does the work instead
+    # fetch during the run — a large range and enough cycles to see it
     nb5 vectordata_demo dataset=<name>:<profile> \
-      prefetch=false mode=background cycles=1000
+      prefetch=background records=100000 cycles=200
 
   Parameters: dataset (required), cycles (default 3), records (default
-  20000), prefetch (true|false, default true), mode (none|eager|
-  background, default none). The meter only speaks when there are bytes
-  to move, so a second run against a warm cache is silent by design —
-  point VECTORDATA_HOME at an empty directory to watch it fetch again.
+  20000), prefetch (load|background|none, default load). The meter only
+  speaks when there are bytes to move, so a second run against a warm
+  cache is silent by design — point VECTORDATA_HOME at an empty
+  directory to watch it fetch again.
 
 scenarios:
   default: >-
@@ -60,10 +69,9 @@ scenarios:
 
 bindings:
   cycle: Identity()
-  # The window keeps reads inside the demo's record range, and mode
-  # selects who warms it: 'none' leaves that to the load-time prefetch
-  # above, 'eager' blocks on it here, 'background' overlaps it with the
-  # run.
-  base_vector: BaseVectors('{{@spec}}','[0..{{@records}})','TEMPLATE(mode,none)'); Stringify()
+  # Reads are demand-paged here on purpose: the prefetch above is what
+  # warms the data, so the meter has one owner and this window only
+  # clips what the bindings expose.
+  base_vector: BaseVectors('{{@spec}}','[0..{{@records}})','none'); Stringify()
   query_vector: QueryVectors('{{@spec}}',false); Stringify()
   neighbor_indices: NeighborIndices('{{@spec}}',false); Stringify()

nb-virtdata/virtdata-lib-vectors/src/main/java/io/nosqlbench/exprs/lib/vectors/VectorDataExprs.java

@@ -218,7 +218,28 @@ public class VectorDataExprs implements ExprFunctionProvider {
         description = "Start fetching a record window of a facet on another thread and return the handle; join() waits, cancel() stops between ranges."
     )
     public PrefetchHandle prefetchBackground(String datasetNameAndProfile, String facetName, String window) {
-        return dataset(datasetNameAndProfile).prefetchInBackground(facetName, DSWindow.parse(window), WholeFacetFallback.REFUSE);
+        TestDataView view = dataset(datasetNameAndProfile);
+        DSWindow parsed = DSWindow.parse(window);
+        // Metered like every other prefetch form: the plan is announced
+        // before the handle is returned, and a watcher reports the
+        // handle's byte counter while the fetch runs alongside whatever
+        // the caller does next.
+        PrefetchMeter meter = new PrefetchMeter(datasetNameAndProfile + ":" + facetName, view.prefetchPlan(facetName, parsed));
+        PrefetchHandle handle = view.prefetchInBackground(facetName, parsed, WholeFacetFallback.REFUSE);
+        meter.watch(handle);
+        return handle;
+    }
+
+    @ExprExample(args = {"\"airports:demo\"", "\"base_vectors\"", "0", "1000"}, expectNotNull = true)
+    @ExprExample(args = {"\"airports:demo\"", "\"query_vectors\"", "0", "10"}, expectNotNull = true)
+    @ExprFunctionSpec(
+        name = "prefetchCyclesBackground",
+        synopsis = "prefetchCyclesBackground(\"dataset:profile\", \"facet_name\", start, end)",
+        description = "Start fetching the records a cycle range will read, on another thread, and return the handle. The background counterpart of prefetchCycles, metered the same way."
+    )
+    public PrefetchHandle prefetchCyclesBackground(String datasetNameAndProfile, String facetName, Number start, Number end) {
+        return prefetchBackground(datasetNameAndProfile, facetName,
+            "[" + start.longValue() + ".." + end.longValue() + ")");
     }
 
 }