nosqlbench/devdocs/devguide/drivers/driver_standards.md

# NoSQLBench Driver Standards

This is the document to read if you want to know if your NoSQLBench driver
is complete. Within this document, the phrase `conformant` will be taken
to mean that a driver or feature is implemented according to the design
intent and standards of the NoSQLBench driver API.

While it may be possible to partially implement a driver for basic use,
following the guidelines in this document will ensure that contributed
drivers for NoSQLBench work in a familiar and reliable way for users from
one driver to another.

Over time, the standards in this guide will be programmatically enforced
by the NoSQLBench driver API.

## Terms

- NB Driver - The NoSQLBench level driver, the code that this document
  refers to.
- Native driver - An underlying driver which is provided by a vendor or
  project.

## Op Templates

The core building block of a NoSQLBench driver is the op template. This is
the form of a statement or operation that users add to a yaml or workload
editor to represent a single operation.

It is the driver's responsibility to create a quick-draw version of an
operation. This is done by using the OpTemplate API. Rules for how a
developer maps an op template to an op function are not set in stone, but
here are some guidelines:

1. Pre-compute as much as you can.
2. Store re-usable elements of an operation in thread-safe form and re-use
   it wherever possible.
3. Allow as much to be deferred till cycle time as reasonable, assuming
   you can cache it effectively.

A moderately advanced example of caching objects by name is included in
the pulsar driver.

In contrast to the rules about how you map your op templates to op
functions (and then ops), it is *crucial* tha tyou document the rules for
how the fields of an template are used. The content that users provide in
a YAML file are the substance of an op template. It is very important that
you document what this means for users, specifically in terms of how field
names and values map to a specific operation.

## Op Sequencing

A conformant driver should use the standard method of creating an
operational sequence. This means that a driver simply has to provide a
function to map an OpTemplate to a more ready to use form that is specific
to the low level driver in question.

## Metrics

At a minimum, a conformant driver should provide the following metrics:

- bind (timer) - A timer around the code that prepares an executable form
  of a statement.
- execute (timer) - A timer around the code that submits work to a native
  driver. This is the section of code which enqueues an operation to
  complete, but not the part that waits for a response. If a given driver
  doesn't have the ability to hand off a request to an underlying driver
  asynchronously, then do not include this metric.
- result (timer) - A timer around the code that awaits and processes
  results from a native driver. This timer should be included around all
  operations, successful ones and errors too. The timer should start
  immediately when the operation is submitted to the native ddriver, which
  is immediately after the bind timer above is stopped for non-blocking
  APIs, or immediately before an operation is submitted to the native
  driver API for all others.
- result-success (timer) - A timer around the code that awaits and
  processes results from a native driver. This timer should only be
  updated for successful operations. The same timer values should be used
  as those on the result timer, but they should be applied only in the
  case of no exceptions during the operation's execution.
- errorcounts-... (counters)- Each uniquely named exception or error type
  that is known to the native driver should be counted. This is provided
  for you as a side effect of using the NBErrorHandler API.
- tries (histogram) - The number of tries for a given operation. This
  number is incremented before each execution of a native operation, and
  when the result timer is updated, this value should be updated as well
  (for all operations). This includes errored operations.

## Error Handling

Users often want to control what level of sensitivity their tests have to
errors. Testing requirements vary from the basic "shutdown the test when
any error occurs" to the more advanced "tell me when the error rate
exceeds some threshold", and so on. The essential point here is that
without flexibility in error handling, users may not be able to do
reasonable testing for their requirements, thus configurable error
handling is essential.

A core library, NBErrorHandler is provided as a uniform way to handle
these errors. It is documented separately in this dev guide. If you add
this error handler to your action implementation, users will automatically
get a completely configurable and standard way to decide what happens for
specific errors in their workload.

## Result Validation

TBD

## Diagnostic Mode

TBD

## Naming Conventions

TBD

### Parameter naming

Parameters should be formatted as snake_case by default. Hyphens or camel
case often cause issues when using mixed media such as command lines and
yaml formats. Snake case is a simple common denominator which works across
all these forms with little risk of ambiguity when parsing or documenting
how parameters are set apart from other syntax.

## Documentation

Each activity is required to have a set of markdown documentation in its
resource directory. The name of the driver should also be used as the name
of the documentation for that driver.

Additional documentation can be added beyond this file. However, all
documentation for a given driver must start with the drivers name and a
hyphen.

If a driver wants to include topics, the convention is to mention these
other topics within the driver's main help. Any markdown file which is
included in the resources of a driver module will be viewable by users
with the help command `nb help <name>`. For example, if a driver module
contains `../src/main/resources/mydriver-specials.md`, then a user would
be able to find this help by running `nb help mydriver-specials`.

These sources of documentation can be wired into the main NoSQLBench
documentation system with a set of content descriptors.

## Named Scenarios

Conformant driver implementations should come with one or more examples of
a workload under the activities directory path. Useful driver
implementations should come with one or more examples of a workloads under
the activities directory path. These examples should employ the "named
scenarios" format as described in the main docs. By including named
scenarios in the yaml format, these named scenarios then become available
to users when they look for scenarios to call with the
`--list-scenarios` command.

To include such scenario, simply add a working yaml with a scenarios
section to the root of your module under the
`src/main/resources/activities` directory.

## Included Examples

Useful driver implementations should come with a set of examples under the
examples directory path which demonstrate useful patterns, bindings, or
statement forms.

Users can find these examples in the same way as they can find the named
scenarios above with the only difference being their location. By
convention the directory `src/main/resources/examples` directory is where
these are located.

The format is the same as for named scenarios, because the examples *are*
named scenarios. Users can find these by using the `--include=examples`
option in addition to the `--list-scenarios` command.

## Testing and Docs

Complete driver implementations should also come with a set of examples
under the examples directory path.

Unit testing within the NB code base is necessary in many places, but not
in others. Use your judgement about when to *not* add unit testing, but
default to adding it when it seems subjective. A treatise on when and how
to choose appropriate unit testing won't fit here, but suffice it to say
that you can always ask the project maintainers for help on this if you
need.

Non-trivial code in pull requests without any form of quality checks or
testing will not be merged until or unless the project maintainers are
satisfied that there is little risk of user impact. Experimental features
clearly labeled as such will be given more wiggle room here, but the label
will not be removable unless/until a degree of robustness is proven in
some testing layer.

### Testing Futures

In the future, the integration testing and the docs system are intended to
become part of one whole. Particularly, docs should provide executable
examples which can also be used to explain how NB or drivers work. Until
this is done, use the guidelines above.

## Handling secrets

Reading passwords ...

## Parameter Use

Activity parameters *and* statement parameters must combine in intuitive
ways.

### ActivityType Parameters

The documentation for an activity type should have an explanation of all
the activity parameters that are unique to it. Examples of each of these
should be given. The default values for these parameters should be given.
Further, if there are some common settings that may be useful to users,
these should be included in the examples.

### Statement Parameters

The documentation for an activity type should have an explanation of all
the statement parameters that are unique to it. Examples of each of these
should be given. The default values for these parameters should be given.

### Additive Configuration

If there is a configuration element in the activity type which can be
modified in multiple ways that are not mutually exclusive, each time that
configuration element is modified, it should be done additively. This
means that users should not be surprised when they use multiple parameters
that modify the configuration element with only the last one being
applied. An example of this would be adding a load-balancing policy to a
cql driver and then, separately adding another. The second one should wrap
the first, as this is expected to be additive by nature of the native
driver's API.

### Parameter Conflicts

If it is possible for parameters to conflict with each other in a way that
would provide an invalid configuration when both are applied, or in a way
that the underlying API would not strictly allow, then these conditions
must be detected by the activity type, with an error thrown to the user
explaining the conflict.

### Parameter Diagnostics

Each and every activity parameter that is set on an activity *must* be
logged at DEBUG level with the
pattern `ACTIVITY PARAMETER: <activity alias>` included in the log line,
so that the user may verify applied parameter settings. Further, an
explanation for what this parameter does to the specific activity *should*
be included in a following log line.

Each and every statement parameter that is set on a statement *must* be
logged at DEBUG level with the
pattern `STATEMENT PARAMETER: <statement name>: ` included in the log
line, so that the user may verify applied statement settings. Further, an
explanation for what this parameter does to the specific statement *
should* be included in a following log line.

### Environment Variables

Environment variable may be hoisted into a driver's configuration, but only
using explicit mechanisms. By default, environment variables are not injected into
any NoSQLBench usage context where it is not explicitly enabled by the user.
The mechanism of enabling environment variables is simple indirection, using
a symbolic variable reference where they would normally use a value.

Further, the variable must be explicitly enabled for env interpolation
by the developer, and documented as such. Having variables which often use
`$...` formats for other purposes besides environment variables is a nuisance.
Conversely, not supporting env vars in `$...` values which are historically
enabled for such is also a nuisance.

#### format

such as `myparam=$ENV_VAR_FOO`, where the env var name must follow
this pattern:

1. A `$` literal dollar sign.
2. Any alphabetic or underscore character (`[a-zA-Z_]`)
3. Zero or more trailing characters to include optional dots and digits. (`[a-zA-Z0-9_]*`)

Alternately, the `${...}` form is less strict, and allows any characters which are not `}`.