start tracking feature news in a file

devdocs/updatenotes.md

@@ -0,0 +1,232 @@
+## Naming Change
+* binary name
+    - `nb`
+    - `nb5`
+
+## Runtime Requirements
+* Java 17 + GC improvements
+    - woo ZGC
+* Deprecated old drivers (conflicts, etc)
+    - version conflicts (removing cql drivers 1.9.* 3.*)
+
+## new Docs
+javadoc site
+developer guide
+first-time-user guide
+Ideally:
+* imported bindings details
+* imported plugins details
+* imported addons details
+* imported app details
+* imported scenario script details
+* imported workload details
+* imported format details
+
+
+## Added Drivers
+- cql driver 4
+- dynamodb
+- mongodb
+- others refreshed
+
+Any driver carried forward has been updated to the DriverAdapter API.
+Any driver which is explicitly incompatible with newer driver versions is deprecated and removed.
+This includes drivers which would require using Maven shading to avoid conflicts.
+This puts NB on a footing to be "Modular Jar" compatible, which is a step towards being fully modular in the jpms sense.
+
+## Bundled Scenario Scripts
+- stepup
+- optimo
+- findmax redux
+
+## Removed Drivers
+
+## New command line options
+--docker-prom-retention-days=183d
+--list-scripts
+--list-apps
+
+## New help topics
+## New scripting plugins
+- globalvars
+- fileaccess
+- csvoutput
+- shutdown
+- s3uploader
+
+## URL support
+- s3 (any usage of an S3 URL should work effectively like an equivalent http* URL)
+
+## New binding functions
+- SaveGlobalVars + Expr updates
+- DecimalFormat
+- DateRangeDuring
+- DateRangeFunc
+- DateRangeOnOrAfter
+- DateRangeOnOrBefore
+- DateRangeParser
+- double -> ToBigDecimal
+- int -> ToBigDecimal
+- long -> ToBigDecimal
+- String -> ToBigDecimal
+- distinct HashRange vs HashInterval functions
+- ToByteBuffer
+- long -> FixedValues() -> double
+- String -> ReplaceAll(...) -> String
+- String -> ReplaceRegex(...) -> String
+- long -> HashRangeScaled(...) -> int
+- long -> HashRangeScaled(...) -> long
+- int -> HashRangeScaled(...) -> int
+- Object -> ToJSON -> String
+- Object -> ToJSONPretty -> String
+- long -> ByteBufferSizedHashed -> ByteBuffer
+- long -> ByteBufferToHex -> String
+- long -> HashedByteBufferExtract -> ByteBuffer
+- long -> HashedCharBufferExtract -> CharBuffer
+- CharBuffer -> ToByteBuffer() -> ByteBuffer
+- String -> ToCharBuffer() -> String
+- long -> CharBufferExtract() -> CharBuffer
+- long -> CharBufImage() -> String
+- long -> CurrentEpochMillis() -> long
+- long -> ElapsedNanoTime() -> long
+- double -> LongStats() -> double
+- long -> ThreadNum() -> long
+- long -> ThreadNum() -> int
+- String -> EscapeJSON() -> String
+- long -> Expr(...) -> Object
+- long -> ToJavaInstant() -> (Java) Instant
+- long -> ToJodaInstant() -> (Joda) Instant
+- long -> ToLocalTime() -> LocalTime
+
+
+## Optimizations
+- avoid preparing ops which are not selected
+- skip over ops with ratio 0
+
+## UX Improvements
+- secure field indirection, username, userfile, password, passfile,
+- op templates can pull from activity params for some fields
+- All yaml can be in mapping form (everything is named)
+- All elements in a yaml can have a description field
+- Uniform Workload Format
+- Uniform Error Handling
+- Uniform Driver API
+  - this means:
+    - standard core activity params across all drivers, like 'retry'
+    - standard metrics across all drivers
+    - standard error handler support
+    - standard op meta-behaviors, like start-timers and stop-timers across all drivers
+    - interoperable data between drivers
+    - standard diagnostic tools across all drivers
+* Named Scenario subsections
+    - `nb5 namedscenario.yaml`
+    - `nb5 namedscenario.yaml default.schema`
+    - `nb5 namedscenario.yaml [<name>]`
+    - `nb5 namedscenario.yaml <scenarioname>.<stepname>`
+* Multiple named scenarios can be chained on the command line, just like any command:
+  - `nb5 workloadfile scenario scenario2 scenario3.step1`
+- Uniform Workload format
+* In-Line Bindings
+    - `{bindingname}`
+    - `{{NumberNameToString()}}`
+- anything can be an op template, even a command line option
+    - `nb5 run driver=cql op="select * from baselines.keyvalue where foo='{{ToString()}} limit 10"`
+* type-safe parameters
+    - extraneous parameters are verboten
+    - parameter types are adapted as needed or an error is thrown
+    - auto-suggested alternatives based on Levenshtein distance
+* improved progress meter
+    - fine-grained
+    - based on actual op metrics in real time instead of internal batching state
+* open-ended binding structures now support multi-layer scaffolding of generated values with a
+  mix of static and dynamic values. This means you can write templates for JSON which look like
+  the payload you want to send.
+  * supports all combinations of lists and maps and strings, with interspersed dynamic values
+    for individual elements or as collection generators, with literals, templated strings, or
+    direct binding references, and in-line bindings recipes.
+* argsfile allows defaults to be set programmatically or on the command line.
+* tag filtering conjugates
+* auto-injected statement block and statement name tags.
+  - this means: You can now construct filters for specific blocks or statements simply by
+    knowing their name:
+  - `tags=block:schema` or `tags='main-.*'`
+* safe usage of activity params and template vars are compatible, but may not be ambiguous. This
+  means that if you have a template variable in myworkload.yaml, it must be distinctly named
+  from any valid activity parameters, or an error is thrown. This eliminates a confusing source
+  of ambiguity.
+* ANSI color support in some places
+* template vars can set defaults the first time they are seen
+
+### Scripting
+- scenario.stop(...) now allows regexes for activity aliases
+
+
+## Core machinery & API Improvements
+- dynamic lib loading via jars
+- startup logging now captures basic system hardware details and NB version, to assist in
+  troubleshooting
+- redesigned developer API around op template, op mapper, op dispenser, and parsed op types.
+    - This means:
+      - ParsedOp - op template -> op mapping -> op dispensing follows an incremental path of
+        construction, allowing functional programming methods to be used *only* when needed.
+      - This means that you only pay for what you use or change. Op builder methods are only
+        added to the construction logic when you need to digress from the defaults.
+    * simpler and more powerful API for driver developers
+        - consolidated logic into core machinery
+        - thin out and focus the driver APIs
+        - pick concepts and API terms which are very responsibility focused
+            - Op Mapper
+            - Op Dispenser
+            - ParsedOp (mapping from op templates to native driver calls)
+    * support for op generators, or ops that inject other ops into the activity. (LWT
+      statement retry, client-side join, etc)
+    * Type-And-Target op template convention which represents special op types and their primary
+      payload template as a key and value.
+
+- drivers are specified per-op. (The activity param just sets the default)
+- native driver contexts, known as a driver `space` allows for instancing native drivers in a data-driven way
+
+## Driver Improvements
+* modular diag driver
+* cqld4 driver
+  * all statement forms are supported
+  * allows file or parameter-based configuration
+  * some backward support for classic nb cql driver options
+  * op templates can support any valid statement builder option
+
+## Bundled Applications
+
+Bundled apps have been part of NoSQLBench for some time, but these were mostly used for
+behind-the-scenes work, like generating docs for binding functions. Now, these apps are
+discoverable and usable by anybody who runs the `--list-apps` command.
+These fall into roughly two categories: 1) NB utilities which can be used to slice and dice
+test data and 2) driver-specific utilites which are useful when testing or analyzing a specific
+type of system.
+
+- `--list-apps`
+- `nb5 <appname> -h`
+- `nb5 <appname> ...`
+
+### cqlgen
+
+cqlgen - takes schema.cql tablestats -> workload.yaml
+- obfuscation
+- weighting by op ratios from tablestats
+- point to docs ->
+
+sstablegen
+
+* yaml+nb version checks
+    - `min_version: "4.17.15"`
+
+
+* Mac M1 support
+    - as of 08/15, all Mac M1 systems should be supported for the.jar and the docker image
+
+review:
+- 7578e91d773a9ea8113250899ef46b7aacf95e70
+- 392bbcc5954019ae58956850c980646cef14a1f7
+- b6627943e2aed2a80ba1aa13d0a929c1ed04b685
+- 394580d66c107127cc68f88cdc64a59e9c481d43
+- 0330c18a7ba0904b3b3420b94416b04ee73dd7fb
+- 11dd8f62daf3d1603b75cdd85fbb872dbfaac111