IntenseFinance/nosqlbench
3
0
Fork 0
mirror of https://github.com/nosqlbench/nosqlbench.git synced 2025-02-25 18:55:28 -06:00
Code Issues Packages Projects Releases Wiki Activity

update wip devguide

Browse Source
This commit is contained in:
Jonathan Shook
2021-02-04 17:47:13 -06:00
parent 516aee2ef5
commit 9526208af8
23 changed files with 802 additions and 206 deletions
Download Patch File Download Diff File Expand all files Collapse all files

58
devdocs/devguide/README.md Normal file
View File

@@ -0,0 +1,58 @@
# Developing with NoSQLBench
This is an overview of how to develop within the NoSQLBench ecosystem.
This guide
## Welcome All
The NoSQLBench project welcomes all contributors, so long as you are
willing to abide by the code
of [CODE OF CONDUCT](../../CODE_OF_CONDUCT.md). Further, you must be
willing to license your contributions under the
[APLv2](../../LICENSE.txt).
## Basics
Herein, and in other docs, you may see NoSQLBench abbreviated as NB
(the project) or nb (the command).
This dev guide covers detailed topics which can help you get bootstrapped
as a contributor. It does not try to cover all the topics which are
included in the main documentation. What you see in this guide wil assume
you have some familiarity with NoSQLBench as a user.
## Topics
- [Design Principles](design_principles.md) - This is a short treatise
that explains the architectural principles behind NoSQLBench.
- [Project Structure](project_structure.md) - An explanation of the module
naming and dependency structure.
- [Dynamic Endpoints](apis/dynamic_endpoints.md) - An introduction to the
appserver mode and how to write new webservices for it.
- [Driver How-To](drivers/README.md) - An introduction for new driver
designers, including API guides and driver standards.
- [NBUI Development](nbui/README.md) - An introduction to the UI
subsystem (NBUI), and tricks for effective development.
- [Java versions](java_versions.md) - A map of where java versions are
configured in the project.
- [Scripting Extensions](scripting_extensions.md) - A guide on what
scripting extensions are, how they are used, and how to build one.
- [Documentation Sources](nb_docs.md) - Where docs are located
- [Adding Scenarios](adding_scenarios.md) - How to add built-in scenarios
### API Naming
In general, internal APIs which are suggested for use throughout
NoSQLBench will start with the *NB* prefix, such as NBIO, or NBErrors.
This makes these APIs easier to find and reference when building new
drivers or core functionality.
New contributors should familiarize themselves with these APIs so that
they can employ them when appropriate. These APIs should also have
sufficient documenation here and in Javadoc to explain their usage.
### API Guides
- [NBIO - File and System IO](nbio_api.md) - A standard way to get files
and other resources from a running NB process.

14
devdocs/devguide/adding_scenarios.md Normal file
View File

@@ -0,0 +1,14 @@
# Adding Scenarios
All of the named scenarios that are bundled with NoSQLBench are simply
named yaml files with scenarios included.
If you are a developer and want to add one of these, you can submit a pull
request to the main project. However, to be accepted, any named scenario
should follow all the conventions described in the main
_Designing Workloads_ documentation, including naming, tagging, and
default values.
If you are not a developer, and you have come up with a workload that
you'd like to make part of the standard NoSQLBench distribution, you can
add it to an issue on the main issue tracker.

33
devdocs/devguide/apis/dynamic_endpoints.md Normal file
View File

@@ -0,0 +1,33 @@
# Dynamic Endpoints
NoSQLBench is more than just a test client. It is a test scenario
execution machine with many pluggable behaviors. It also has a daemon mode
which can be used to enable different types of persistent services:
- Documentation Server
- [NBUI](../nbui/README.md) - The fledgling NoSQLBench UI
- Workload Services
These are all enabled by a set of web services backed by jetty and powered
by jax-rs and related libraries.
## Service Discovery
When NoSQLBench is invoked in appserver mode (classically called
`docserver` mode), it starts up with a set of web services. Like many
other runtime elements of NB, these services are discovered internally as
long as they implement the WebServiceObject tagging interface and are
registered as a service with an annotation like
@Service(WebServiceObject.class)
Any such class is processed as a jax-rs annotated class and automatically
added to the active services accordingly.
This is all you have to do to add endpoints to NoSQLBench!
## Useful Links
* [JAX-RS 2.0 spec](http://download.oracle.com/otn-pub/jcp/jaxrs-2_0-fr-eval-spec/jsr339-jaxrs-2.0-final-spec.pdf)

121
devdocs/devguide/design_principles.md Normal file
View File

@@ -0,0 +1,121 @@
# NoSQLBench Design Principles
This is a basic readout on the design principles that inform the core
NoSQLBench design. They can be taken as a set of non-feature requirements
for a serious testing instrument. These principles do not represent the
actual state of NoSQLBench. In fact, some of them may appear more
aspirational than descriptive. Nonetheless, they describe a set of guiding
principles that any contributor should strive to keep in mind as they
build on NoSQLBench.
Ideally, any feature or component will have the best of all of these
attributes. When there is tension between them, designs should always
defer to what a reasonable user would want. This list may adapt and change
over time with more contributions and discussion.
The rest of this guide is rather detailed and delves into design
philosphy.
## Concept Reuse
NoSQLBench aims to be a power tool that is safe for casual users while
also being a trusted tool by serious users. As such, it needs to provide a
set of useful concepts which apply in an axiomatic way across all
NoSQLBench capabilities. This means that once you know what a `cycle` is,
you know that the purpose and usage of it will be consistent no matter
where, when, or how you find it. As such, it is a material part of the
user experience. Concepts like this are often disregarded in tool design.
This puts users as a disadvantage when they want to do anything
non-trivial. Designing robust concepts that are tangible and composable is
one of the hardest aspects of design, and one that is often kicked down
the road. Not attacking this problem directly is one of the deepest forms
of accidental complexity and one that is difficult for a desing to recover
from.
In NoSQLBench, users should take for granted that an understanding of
concepts like _activity_, _cycle_, and _binding_
is directly empowering, and worthy of their attention. Such an
understanding gives them access to additional layers of capability and
expression, and allows all users to speak the same language to each other
with little concern for ambiguity.
## Modularity
Modularity is used throughout NoSQLBench. The reasons are many, but the
practical effect is that once you understand the mechanisms used to
achieve modularity, you can add your own modules of a given type to the
runtime with little fanfare.
Modularity in a project is also a form of applying the Liskov substitution
principle at a higher level. It encourages healthy design boundaries at
key points of re-use. This means that both service providers and service
consumers can depend on the notions of responsibility based design as well
as design-by-contract.
Many runtime elements of NoSQLBench are provided as bundled plugins. Any
time you see a `@Service` annotation, it describes a service entry for
something that is defined by an interface.
## Efficiency
A serious testing tool must be an effective measurement instrument. In the
end all testing tools are merely measurement tools of some sort. With a
load-generating tool, the business of driving a workload can not interfere
with the need to measure accurately. There are multiple paths to achieving
an effective separation of these concerns, but they are all non-trivial. (
load bank orchestration, local resource marshaling, etc.)
One of the greatest simplifications we can provide in a testing tool is to
allow it to do meaningful work without requiring the same level of
orchestration, coordination, and aggregation that many other tools require
for basic workloads. Optimization for performance is absolutely critical
in a testing tool. It doesn't matter what your testing system can do if it
is always the slowest part of the composed system. Thus, many constructs
found within NoSQLBench are purpose built for efficiency. It is "maturely
optimized" because that is what makes NoSQLBench suited for its purpose.
## Flexibility
A testing system which can only do one thing or be used in on particular
way will not be suitable for many common testing tasks. This lack of fit
for many simple purposes would make a system less reusable overall. In the
world of systems testing, requirements are extremely varied, and
circumstances are always changing. Thus, it is uncommon for users to
invest much time in one trick ponies. Such tools have their purposes, and
serve a need at times. However, they do not generally reward users with
incremental knowledge, nor help them to adapt their investment of testing
effort to new or more nuanced needs. As such, they are often used and then
disregarded for further consideration. Even more costly is that these
tools often over-simplify the testing surface area to such a degree that
users are left without even basic answers to the questions they are
asking, or worse, wrong or inaccurate answers to specific questions.
NoSQLBench should provide the building blocks that it has with clarity and
purposes. It should allow the user to recombine and reconfigure these
building blocks as needed for whatever level of testing they require.
### Scalable Experience
For common tasks, or simple usage patterns, it should be possible to use
NoSQLBench quickly. A user should be able to ask a simple question and get
a simple answer without having to debate or wrestle with the system to get
it running. These capabilities should simply be packaged in a pre-baked
form. Each time a user asks more from the testing tool, it should be able
to
### Discoverability
Another element of flexibility, in practical terms, is discoverability. A
tool which can do both simple and sophisticated things is wasted if users
are unable to find and understand the incremental levels of capability it
offers. While the initial surface area of the tool may be intentionally
simplified, there must be some obvious way for a user to opt-in to more
knowledge, more tooling, and more sophistication if and when they want to.
## Interoperability
Where possible, common standard and modern conventions should be used for
interfacing with external systems. This means that most basic tooling in
the modern development ecosystem should be easy to integrate with
NoSQLBench when needed. This includes aspects such as configuration
formats, logging tools, web endpoints, and so on.

24
devdocs/devguide/drivers/README.md Normal file
View File

@@ -0,0 +1,24 @@
# Driver Development
This section should have everything you need to know to build a successful
driver in NoSQLBench.
If you are new to NoSQLBench concepts, you may want to
read [about drivers](drivers_overview.md).
You'll want to be generally familiar with the current NoSQLBench
[driver standards](driver_standards.md). This document explains what a
well-behaved driver can do. Much of what is requested in driver standards
is directly supported by a set of supporting APIs which are provided to
all driver implementations. It may seem like a high bar to request this of
developers, but without such guidelines, a bar is set nonetheless. The aim
of this is to help define and clarify how to make a
_good_ driver.
## Driver APIs
- [NBOpTemplate](optemplate_api.md) - Op Templating - This is the
recommended way to map user-level semantics to driver-specific
operations.
- [NBErrors](nberrors_api.md) - Uniform error handling - A modular and
configurable error handler which new drivers should use by default.

283
devdocs/devguide/driver_standards.md → devdocs/devguide/drivers/driver_standards.md
View File

@@ -1,34 +1,17 @@
# NoSQLBench Driver Standards
This is a work in progress...
This is the document to read if you want to know if your NoSQLBench driver
is complete. Within this document, the phrase `conformant` will be taken
to mean that a driver or feature is implemented according to the design
intent and standards of the NoSQLBench driver API.
While it may be possible to partially implement a driver for basic use,
following the guidelines in this document will ensure that contributed
drivers for NoSQLBench work in a familiar and reliable way for users from
one driver to another.
This is the document to read if you want to know if your NoSQLBench driver is complete.
Within this document, the phrase `conformant` will be taken to mean that a driver or feature
is implemented according to the design intent and standards of the NoSQLBench driver API.
While it may be possible to partially implement a driver for basic use, following the guidelines
in this document will ensure that contributed drivers for NoSQLBench work in a familiar and
reliable way for users from one driver to another.
Over time, the standards in this guide will be programmatically enforced by the NoSQLBench
driver API.
## Op Templates
The core building block of a NoSQLBench driver is the op template. This is the form of a
statement or operation that users add to a yaml or workload editor to represent a single operation.
For example, in the CQL driver, this is called a "statement template", but going forward, they will
all be called Op Templates and internal API names will reflect that.
It is the driver's responsibility to create a quick-draw version of an operation.
## Op Sequencing
A conformant driver should use the standard method of creating an operational sequence. This means
that a driver simply has to provide a function to map an OpTemplate to a more ready to use form that
is specific to the low level driver in question.
Over time, the standards in this guide will be programmatically enforced
by the NoSQLBench driver API.
## Terms
@@ -37,6 +20,40 @@ is specific to the low level driver in question.
- Native driver - An underlying driver which is provided by a vendor or
project.
## Op Templates
The core building block of a NoSQLBench driver is the op template. This is
the form of a statement or operation that users add to a yaml or workload
editor to represent a single operation.
It is the driver's responsibility to create a quick-draw version of an
operation. This is done by using the OpTemplate API. Rules for how a
developer maps an op template to an op function are not set in stone, but
here are some guidelines:
1. Pre-compute as much as you can.
2. Store re-usable elements of an operation in thread-safe form and re-use
it wherever possible.
3. Allow as much to be deferred till cycle time as reasonable, assuming
you can cache it effectively.
A moderately advanced example of caching objects by name is included in
the pulsar driver.
In contrast to the rules about how you map your op templates to op
functions (and then ops), it is *crucial* tha tyou document the rules for
how the fields of an template are used. The content that users provide in
a YAML file are the substance of an op template. It is very important that
you document what this means for users, specifically in terms of how field
names and values map to a specific operation.
## Op Sequencing
A conformant driver should use the standard method of creating an
operational sequence. This means that a driver simply has to provide a
function to map an OpTemplate to a more ready to use form that is specific
to the low level driver in question.
## Metrics
At a minimum, a conformant driver should provide the following metrics:
@@ -61,7 +78,8 @@ At a minimum, a conformant driver should provide the following metrics:
as those on the result timer, but they should be applied only in the
case of no exceptions during the operation's execution.
- errorcounts-... (counters)- Each uniquely named exception or error type
that is known to the native driver should be counted.
that is known to the native driver should be counted. This is provided
for you as a side effect of using the NBErrorHandler API.
- tries (histogram) - The number of tries for a given operation. This
number is incremented before each execution of a native operation, and
when the result timer is updated, this value should be updated as well
@@ -77,72 +95,19 @@ without flexibility in error handling, users may not be able to do
reasonable testing for their requirements, thus configurable error
handling is essential.
Until the error handling subsystem is put in place, these types of error
handling levels are suggested:
1. stop
2. warn
3. retry
4. histogram
5. count
6. ignore
This serves as an error handling stack, where the user chooses the entry
point. From the user-selected entry point, all of the remaining actions
are taken until the end if possible.
### stop
If an exception occurs, and the user has selected the `stop` error
handler, then the activity should be stopped. This is triggered by
allowing the NB driver to propagate a runtime exception up the stack.
Since this error handler interrupts flow of an activity, no further error
handling is done.
### warn
If an exception occurs and the user has selected the `warn` error handler,
the the exception should be logged at WARN level.
The next error handler `retry` should also be called.
### retry
If an exception occurs and the user has selected the `retry` error
handler, **AND** the exception represents a type of error which could
reasonably be expected to be non-persistent, then the operation should be
re-submitted after incrementing the tries metric.
Whether or not the operation is retried, the next error handler
`histogram` should also be called.
### histogram
If an exception occurs and the user has selected the `histogram` error
handler,the error should be recorded with the help class
`ExceptionHistoMetrics`. This adds metrics under the `errorhistos` label
under the activity's name.
The next error handler `count` should also be called.
### count
If an exception occurs and the user has selected the `count` error
handler, then the error should be counted with the helper class
`ExceptionCountMetrics`. This adds metrics under the `errorcounts` label
under the activity's name.
The next exception handler `ignore` should also be called, but this is
simply a named 'no-op' which is generally the last fall-through case in a
switch statement.
TBD
A core library, NBErrorHandler is provided as a uniform way to handle
these errors. It is documented separately in this dev guide. If you add
this error handler to your action implementation, users will automatically
get a completely configurable and standard way to decide what happens for
specific errors in their workload.
## Result Validation
TBD
## Diagnostic Mode
TBD
## Naming Conventions
@@ -173,18 +138,18 @@ with the help command `nb help <name>`. For example, if a driver module
contains `../src/main/resources/mydriver-specials.md`, then a user would
be able to find this help by running `nb help mydriver-specials`.
These sources of documentation can be wired into the main NoSQLBench documentation system with a set
of content descriptors.
These sources of documentation can be wired into the main NoSQLBench
documentation system with a set of content descriptors.
## Named Scenarios
Conformant driver implementations should come with one or more examples of a workload under the
activities directory path.
Useful driver implementations should come with one or more examples of a
workloads under the activities directory path. These examples should
employ the "named scenarios" format as described in the main docs. By
including named scenarios in the yaml format, these named scenarios then
become available to users when they look for scenarios to call with the
Conformant driver implementations should come with one or more examples of
a workload under the activities directory path. Useful driver
implementations should come with one or more examples of a workloads under
the activities directory path. These examples should employ the "named
scenarios" format as described in the main docs. By including named
scenarios in the yaml format, these named scenarios then become available
to users when they look for scenarios to call with the
`--list-scenarios` command.
To include such scenario, simply add a working yaml with a scenarios
@@ -208,8 +173,8 @@ option in addition to the `--list-scenarios` command.
## Testing and Docs
Complete driver implementations should also come with a set of examples under the examples
directory path.
Complete driver implementations should also come with a set of examples
under the examples directory path.
Unit testing within the NB code base is necessary in many places, but not
in others. Use your judgement about when to *not* add unit testing, but
@@ -232,69 +197,61 @@ become part of one whole. Particularly, docs should provide executable
examples which can also be used to explain how NB or drivers work. Until
this is done, use the guidelines above.
## Usage of the Op Template
The operation which will be executed in a driver should be derivable from
the YAML as provided by a user. Thankfully, NoSQLBench goes to great
lengths to make this easy for both to the user and to the driver
developer. In particular, NB presents the user with a set of formatting
options in the YAML which are highly flexible in terms of syntax and
structure. On the other side, it presents the driver developer with a
service interface which contains all the input from the user as a complete
data structure.
This means that the driver developer needs to make it clear how different
forms of content from the YAML will map into an operation. Fundamentally,
a driver is responsible for mapping the fully realized data structure of
an `op template` into an executable operation by NoSQLBench.
In some protocols or syntaxes, the phrase _the statement_ makes sense, as
it is the normative way to describe what an operation does. This is true,
for example, with CQL. In CQL You have a statement which provides a very
clear indication of what a user is expecting to happen. At a more abstract
level, and more correctly, the content that a user puts into the YAML is a
`statement template`, and more generally within NoSQLBench, and `operation
template`. This is simply called an `op template` going forward, but as a
driver developer, you should know that these are simply different levels
of detail around the same basic idea: an executable operation derived from
a templating format.
Since there are different ways to employ the op template, a few examples
are provided here. As a driver developer, you should make sure that your
primary docs include examples like these for users. Good NB driver docs
will make it clear to users how their op templates map to executable
operations.
### op template: statement form
In this form, the op template is provided as a map, which is also an
element of the statements array. The convention here is that the values
for and _the statement_name_ and _the statement_ are taken as the first
key and value. Otherwise, the special properties `name` and `stmt` are
explicitly recognized.
```text
statements:
- aname: the syntax of the statement with binding {b1}
tags:
tag1: tagvalue1
params:
param1: paramvalue1
freeparamfoo: freevaluebar
bindings:
b1: NumberNameToString()
```
### op template: map form
```text
statements:
- cmd_type:
```
Structural variations and conventions.
## Handling secrets
Reading passwords ...
## Parameter Use
Activity parameters *and* statement parameters must combine in intuitive
ways.
### ActivityType Parameters
The documentation for an activity type should have an explanation of all
the activity parameters that are unique to it. Examples of each of these
should be given. The default values for these parameters should be given.
Further, if there are some common settings that may be useful to users,
these should be included in the examples.
### Statement Parameters
The documentation for an activity type should have an explanation of all
the statement parameters that are unique to it. Examples of each of these
should be given. The default values for these parameters should be given.
### Additive Configuration
If there is a configuration element in the activity type which can be
modified in multiple ways that are not mutually exclusive, each time that
configuration element is modified, it should be done additively. This
means that users should not be surprised when they use multiple parameters
that modify the configuration element with only the last one being
applied. An example of this would be adding a load-balancing policy to a
cql driver and then, separately adding another. The second one should wrap
the first, as this is expected to be additive by nature of the native
driver's API.
### Parameter Conflicts
If it is possible for parameters to conflict with each other in a way that
would provide an invalid configuration when both are applied, or in a way
that the underlying API would not strictly allow, then these conditions
must be detected by the activity type, with an error thrown to the user
explaining the conflict.
### Parameter Diagnostics
Each and every activity parameter that is set on an activity *must* be
logged at DEBUG level with the
pattern `ACTIVITY PARAMETER: <activity alias>` included in the log line,
so that the user may verify applied parameter settings. Further, an
explanation for what this parameter does to the specific activity *should*
be included in a following log line.
Each and every statement parameter that is set on a statement *must* be
logged at DEBUG level with the
pattern `STATEMENT PARAMETER: <statement name>: ` included in the log
line, so that the user may verify applied statement settings. Further, an
explanation for what this parameter does to the specific statement *
should* be included in a following log line.

35
devdocs/devguide/drivers/drivers_overview.md Normal file
View File

@@ -0,0 +1,35 @@
# Contributing a Driver
Drivers in NoSQLBench are how you get the core machinery to speak a
particular protocol or syntax. At a high level, a NB driver is responsible
for mapping a data structure into an executable operation that can be
called by NoSQLBench.
Internally, the name `ActivityType` is used to name different high level
drivers within NoSQLBench. This should avoid confusion with other driver
terms.
Drivers in NoSQLBench are separate maven modules that get added to the
main nb.jar artifact (and thus the AppImage binary). For now, all these
driver live in-tree, but we may start allowing these to be packaged as
separate jar files. Let us know if this would help your integration
efforts.
## Start with Examples
There are a few activity types which can be used as templates. It is
recommended that you study the stdout activity type as your first example
of how to use the runtime API. The HTTP driver is also fairly new and thus
uses the simpler paths in the API to construct operations. Both of these
recommended as starting points for new developers.
## Consult the Dev Guide
The developers guide is not complete, but it does call out some of the
features that any well-built NB driver should have. If you are one of the
early builders of NB drivers, please help us improve the dev guide as you
find things you wish you had known before, or ways of getting started.
The developer's guid is a work in progress. It lives under
devdocs/devguide in the root of the project.

35
devdocs/devguide/drivers/nberrors_api.md Normal file
View File

@@ -0,0 +1,35 @@
# NBErrorHandler API
NoSQLBench provides a standard way of configuring error handling across
driver implementations. This is provided by NBErrorHandler and associated
classes.
This error handler allows a chain of specific error handler behaviors to
be invoked based on which type of error occurs. Additionally, the error
handler is responsible for determining the cycle code to be used in
advanced testing as well as for indicating whether or not an operation is
eligible to be retried or not.
The entire implementation is of this error handler is in package:
io.nosqlbench.engine.api.activityapi.errorhandling.modular
The canonical example of using this API can be found in HttpAction.
## NBErrorHandler Sketch
As a feature that changes control-flow based on user input, it strongly
suggests a specific type of flow within an action. The following
pseudo-code explains this pattern.
1. When an activity is initialized, also initialize an instance of
NBErrorHandler
2. Each time an Action is called, loop while retries are not exhausted.
1. catch all exceptions and pass them to the error handler
2. Read the returned ErrorDetail.
3. If the error detail indicates that the error was retryable and
max_tries is not exhausted, retry the operation Otherwise, stop the
cycle.
4. In any case, when the cycle is complete, always return the cycle
code in the error detail, or 0 if no error was found

168
devdocs/devguide/drivers/optemplate_api.md Normal file
View File

@@ -0,0 +1,168 @@
# OpTemplate API
The operation which will be executed in a driver should be derivable from
the YAML as provided by a user. Thankfully, NoSQLBench goes to great
lengths to make this easy for both to the user and to the driver
developer. In particular, NB presents the user with a set of formatting
options in the YAML which are highly flexible in terms of syntax and
structure. On the other side, it presents the driver developer with a
service interface which contains all the input from the user as a complete
data structure.
This means that the driver developer needs to make it clear how different
forms of content from the YAML will map into an operation. Fundamentally,
a driver is responsible for mapping the fully realized data structure of
an `op template` into an executable operation by NoSQLBench.
In some protocols or syntaxes, the phrase _the statement_ makes sense, as
it is the normative way to describe what an operation does. This is true,
for example, with CQL. In CQL You have a statement which provides a very
clear indication of what a user is expecting to happen. At a more abstract
level, and more correctly, the content that a user puts into the YAML is a
`statement template`, and more generally within NoSQLBench,
and `operation template`. This is simply called an `op template` going
forward, but as a driver developer, you should know that these are simply
different levels of detail around the same basic idea: an executable
operation derived from a templating format.
## Op Function Mapping
This is accomplished by using a technique called _Op Function Mapping_
-- the act of mapping an op template to an op function. There are two
functional steps in this process: First, at activity initialization, and
then later within each cycle.
This process is governed by a single type:
Function<OpTemplate, LongFunction<T>>
In this type, the operation you will use is generic type T, and is up to
you, the driver developer. At activity initialization time, a function of
the type above will be used to construct a sequence of LongFunction<T>.
Within each cycle, one of these LongFunction<T> will be selected from the
sequence and called with the cycle to produce a fully realized operation.
This is the most challenging part of building a NoSQLBench driver, but it
is arguably the most important. This section is worth some study if you
want to understand one of the key mechanisms of NoSQLBench.
## Preparing Operations
During execution, NB needs to be fast and lean in terms of the prep work
needed to execute an operation. This means that, at startup, an activity
is responsible for doing two critical steps:
1) (Driver Logic) Map the YAML structure to a set of driver-specific
operation templates.
2) (Core Logic, with Type specified by driver) Cache the operations in a
ready form which is efficient to use.
3) (Core Logic) Assemble these ready operations into a sequence based on
ratios.
Much of the boilerplate for doing this is handled already by the default
Activity implementation which is used by all drivers: the
_SimpleActivity_.
## Op Mapping
Step 1 above is all about mapping the YAML content for each statement to a
driver-specific type. For the sake of illustration, we'll call that simply
a ReadyOp. In order to provide this, you call
`super.createOpSequence(...)` with a function from _OpTemplate_ to your
ReadyOp type, and store the result in your activity.
As long as your function, say a _Function<OpTemplate,ReadyOp>_ knows what
the user means to do depending on what they put in the YAML structure,
steps #2 and #3 are already handled! You're done.
But wait, there's more. If you unpack the phrase above "what the user
means to do ...", you'll see that it is loaded with ambiguity. The rest of
this guide is meant to explain why and how you actually create that
all-important mapping function.
## Op Template Semantics
**NOTE:**
To fully understand this section, it is important that you understand what
the NB Standard YAML format looks like. A detailed explanation of this
format is found under the user guide, with detailed examples under the "
Designing Workloads", with each feature of the YAML format explained in
detail.
The actual type of mapping function provided by your driver will be up to
you, but it is not an actual operation to be executed. Each cycle will
synthesize the operation to be executed by combining your ready op type
above with a cycle number. Thus, from the yaml format to the operation,
there are two stages of resolution:
1. YAML to ready op, as controlled by your own Ready___Op implementation,
as a Function<OpTemplate,YourTypeHere>.
2. ready op to operation, as controlled by the
### Reserved parameters
Some param names used in the YAML format are reserved for current or
future use by the NoSQLBench core. These are:
- name (used to name all elements within the Standard YAML format)
- ratio (used to set ratios in the Standard YAML format)
- type,driver,context,instance (future use)
These parameter names are sometimes visible as part of the OpTemplate, but
driver implementors are not allowed to reassign what they mean or how they
are used.
## Op Function Mapping
Op Function Mapping, or simply _Op Mapping_ is the process of interpreting
the op template and rendering a partially resolved operation back to
NoSQLBench.
Generally, Op Function Mapping is controlled by a driver-provided function
of type Function<OpTemplate,LongFunction<T>>. The result of calling this
function is another function which is used later during each cycle to
render a fully realized and executable operation.
The OpTemplate is a parsed and assembled view of everything the user has
specified for a statement in the yaml, including layering in values from
blocks and root document levels. This includes bindings, params, tags, and
so on. For bindings, it also knows which fields are specified as static
and which ones are tied to bindings with the
`{binding}` syntax.
This allows you, as the developer, to know which parts of an operation's
definition are intended to be fixed and which ones are to be deferred
until a specific cycle. The methods on the OpTemplate allow you to
interrogate the data.
Ideally, you would create an operation that is as fully resolved as
possible. Sometimes this is quite possible and sometimes it is not. For
example, if a user specifies that the type of operation is controlled by a
binding like `optype: {myoptype}` it is impossible to know what that type
will be until the cycle is being executed. Generally, you want to reject
this type of configuration as invalid, since this defeats the ability to
streamline the execution of ops with pre-configuration.
NoSQLBench configures an activity by calling your Op Function Mapping
function for each active instance of an op template as defined in the
YAML.
## Op Binding
Later, the runtime of NoSQLBench will use the result of your provided
function to render an executable operation, fully qualified with payload
details and any other dynamic features you allowed for in the op template
documentation for your driver. This is known as _binding an op template to
a cycle_, or simply _op binding_. There is nothing else for you do to
enable this. The function produced by your Op Template Mapping phase will
be used to power this within each cycle.
## Mapping and Binding Infographic
The infographic below provides an overview of the flow of information and
types throughout this process. The driver developer must provide an
implementation of an Op mapping function which is assignable to
a `Function<OpTemplate,LongFunction<T>>` as shown. The 4 steps in the _Op
Mapping Logic_ box describe a generalized approach that can be used for
each driver.
![Mapping And Binding](optemplates.png)

18
devdocs/devguide/java_versions.md Normal file
View File

@@ -0,0 +1,18 @@
# Java Version Updates
This is a list of all the places that the Java version is referenced in
the NoSQLBench project. If you change versions or need to do
version-specific troubleshooting, this list will help.
- In the Dockerfile, in the parent image "FROM" tag.
[here](../../Dockerfile)
- In the Maven pom files, under `<source>...</source>`, `<release>...
</release>`, `<target>...</target>`
[here](../../mvn-defaults/pom.xml)
- In some cases, a Maven variable '<java.target.version>...</java.target.
version>` is used.
[here](../../mvn-defaults/pom.xml)
- In the nb appimage build scripts under nb/build-bin.sh.
[here](../../nb/build-bin.sh)
- In the github actions workflows for the Java runtime version

37
devdocs/devguide/nb_docs.md Normal file
View File

@@ -0,0 +1,37 @@
# Editing NoSQLBench Docs
The docs that are hosted at docs.nosqlbench.io are built from markdown
files found in this project. To make changes or to author these docs, you
need to know where to find them.
## NoSQLBench docs
The core docs are found under the [engine-docs](../../engine-docs)
module
under [/src/main/resources/docs-for-nb/](../../engine-docs/src/main/resources/docs-for-nb)
.
By browsing this directory structure and looking at the frontmatter on
each markdown file, you'll get a sense for how they are orgainzed.
## Driver Docs
Some of the other docs are found within each driver module. For example,
the cql docs are found in the resources root directory of the cql driver
module. This is the case for any basic docs provided for a driver. The
docs are bundled with modules to allow for them to be maintained by the
driver maintainers directly.
## Binding Function Docs
All of the binding function docs are generated automatically from source.
Javadoc source as well as annotation details are used to decorate the
binding functions so that the can be cataloged and shared to the doc site.
To improve the binding function docs, you must improve the markdown
rendering code which is responsible for this.
## Suggestions
The doc system in NoSQLBench is a core element, but it is not really great
yet nor is it done. Any help on improving it is appreciated, in any form.

68
devdocs/devguide/nbio_api.md Normal file
View File

@@ -0,0 +1,68 @@
# NBIO API
The NBIO class provides a way to access file and URL resources from within
a running NoSQLBench process. This centralizes the logic for accessing
files in classpath, filesystem, or on the net.
Managing file, classpath, and URL resources when you need access to all of
them can be complicated business. The NBIO API was introduced to
consolidate this into one easy-to-use API with substantial testing. It
also provides a uniform interface for accessing all of these resource
types so that users have a consistent experience regardless of where their
source data lives.
## Design Intent
Unless there is a good reason to avoid it, developers should delegate to
the NBUI API _anywhere_ in NoSQLBench that a user needs to load a file.
This imbues such call sites with the ability to read local and remote
resources with no further effort required by the programmer.
## API Usage
To use this API, simply call one of the following methods and use method
chaining to delve into what you are looking for:
- `NBIO.all()...` - Look in the filesystem, the classpath, and on the
net (for any patterns which are a URL).
- `NBIO.fs()...` - Look only in the filesystem.
- `NBIO.classpath()...` - Look only in the classpath.
- `NBIO.local()...` - Look in filesystem or classpath.
- `NBIO.remote()...` - Look only for remote resources.
## Searching Semantics
The fluent-style API will layout the following search parameters for you.
The details on these are explained in the javadoc which will be provided
in most cases as you expand the API. Each of these search parameters
supports zero or more values, and all values are searched that are
provided.
- `prefix(...)` - Zero or more prefix paths to search.
- `name(...)` - Zero or more resources or filenames or urls to look for.
- `extension(...)` - Zero or more extensions to try to match with.
## Getting Results
At runtime, the local process has visibility to many things. Due to how
classpath resources are assembled in pre-JDPA days, it is possible to find
multiple resources under the same name. Thus, you have to be specific
about how many you expect to find, and thus what is considered an error
condition:
- `list()` - All found content in one list
- `one()` - A single source of content should be found -- more or less is
an error.
- `first()` - At least one source of content should be found, and you only
want to see the first one.
- `resolveEach()` - List of lists, allows full cardinality visibility when
using the bulk interface with multiple name terms.
## Consuming Content
To support efficient resolution and consumption of local and remote
content, the type of element returned by the NBIO API is a _Content_
object. However, it provides convenient accessors for getting content when
needed. Remote content will be read lazily only once per NBIO result.
Local content is also read lazily, but it is not cached until GC.

185
devdocs/devguide/nbui/README.md Normal file
View File

@@ -0,0 +1,185 @@
# NBUI
This is where the developer docs are kept for the NoSQLBench UI stack.
## Status
Presently (June/2020), NoSQLBench contains a web application and services layer, but it has not been
used for more than the documentation system. We are putting out request for comments and ideas and
for contributors who can build UIs in Vue.js around these ideas. This document is the first
consolidated view of how UIs work with NoSQLBench.
## NBUI Stack
The current _NBUI Stack_ consists of the NoSQLBench "nb" process, including the built-in web server,
The JAX-RS services which provide endpoints for accessing NoSQLBench functionality, and a set of
embedded web applications which are bundled within the nb application itself.
The docs for NoSQLBench are in-fact a Vue application. We use Nuxt as an framework/wrapper around
Vue to allow us to handle pre-rendering of the application for bundling. The docs application is
only an example of a Vue app in the NBUI stack, but it can be helpful for understanding how the
stack works.
The web apps are written in Vue.js as a separate developer workflow, but within the umbrella of the
NoSQLBench project. The documentation app lives in `src/main/node/docsys` under the docsys module.
It is a node and npm enabled Nuxt.js application. You can go to that directory and run it separately
under npm or yarn, which is what you would normally do when building any Vue/Nuxt application with
node js.
For bundling applications within the NoSQLBench distribution (jar or binary), Nuxt is used to
generate the "static site" form of the application. The result is packaged in
`src/main/resources/docsys-guidebook` and registered for sharing via the internal NoSQLBench static
content handler by way of the *DocsysDefaultAppPath* class, which is how all content sources are
found in NoSQLBench.
## NBUI Modes
Because the backing content for the docserver can come from a variety of sources, NBUI applications
can be run in different modes. Which mode you use depends on what you are doing, as explained below.
### Embedded Mode
This is the default mode that you get when you run `nosqlbench docserver`. In this mode, nosqlbench
starts up a Jetty container to serve content and JAX-RS services. One of the services is a static
content service which serves up the static copy of the generated UI application. This application is
static content on the server, but it includes a client-side application that further does dynamic
rendering and makes its own requests back to the server to fetch additional content or interact with
NoSQLBench service endpoints. This mode is illustrated by the following diagram:
![DocServer Embedded Mode](embedded_mode.png)
### Development Mode
This is the mode that UI developers will care about the most. In this mode, UI developers can run
Vue apps in the usual dev mode while accessing endpoints in a running nosqlbench instance. To use
this mode, you start nosqlbench with the `nosqlbench docserver` command, and then separately run the
local Nuxt app with `npm dev` which is configured to run nuxt in dev mode.
This causes the services to wire together as illustrated in this diagram:
![DocServer Dev Mode](dev_mode.png)
### Static Site Mode
This mode is used to host an NBUI app on a static site, where endpoint services are not available.
In this mode, content which would be returned from an endpoint path is statically exported into
files in the same path, such that a static content server with no dynamic page rendering can still
provide a snapshot of content to be used by the client-side applications.
The release pipeline creates this content with content export utilities, so that when new NoSQLBench
versions are released, the docs site is automatically updated with refresh of current content from
the source tree.
Client-side logic still runs (The client-side apps are always active), but the server simply hands
static content back from the file system:
![Static Site Mode](staticsite_mode.png)
## Interactive Or Not?
The three modes are useful and needed for different purposes:
- **Dev Mode** is needed for rapid development and prototyping of UIs and endpoint services. Dev
mode is the only mode where you can make code changes and immediately refresh to see the results
of your work. Dev mode supports full UI interactivity and endpoint services, so all application
features can be developed.
- **Embedded Mode** is for users of NoSQLBench who need to actually use the built UIs. Like Dev
mode, it supports full UI interactivity and endpoint services.
- **Static Mode** is for non-interactive use, and does not support endpoint service against an
instance of NoSQLBench, since it is not served from a NoSQLBench process that can also provide
these services.
Given that the static mode can't work with stateful services on a NoSQLBench server, a distinction
has to be made within an application about whether should support static mode. We'll put a
client-side check in where needed once we have other examples to work with. At the minimum, either
applications, or specific (interactive) application features will be disabled automatically if the
client-side application detects that endpoint services are not available. A future enhancement will
probably mean that the endpoint servers are fully configured and enabled by the client user as
needed.
### Developing with Nuxt
Using dev mode, you can build a vue.js application, configure it for being used as a _generated_ web
application, and then have it be hosted by NoSQLBench.
## Proposed Features
We propose these UI features to make using NoSQLBench easier:
### NBUI Wishlist
A user should be able to see all the NBUI apps from one place. They should be able to see the name
of each app, and some hover/title docs about what the app is used for. Users should be able to
switch to an app by clicking its name/menu entry. If a user has some unsaved work in one app that
they would lose by switching to another app, they should be prompted and given the choice to save or
discard their work first, or to cancel the action.
The NBUI Overview is not a highly functional app, but what it does is very important; It ties the
rest of the apps together within one cohesive view for the user. As such, it establishes a pattern
for the rest of the visual design of NBUI.
Sketch: A simple implementation of NBUI would be a frame holder with a menu of apps on the left. The
name of the current app could be the only app name showing by default. When the current app name is
clicked, it could unroll into a list of available app that the user could then pick between. Picking
another app would switch the currently active app within the overview frame and roll-up the app
names again to a single value. This basic design would leave a usable menu area below the app name
(and a subtle divider) so that the screen is still usable by each app, including the left. The left
panel could be toggled to be hidden, with a small corner visible to call it back.
### Scenario Runner
A user should be able to find all the named scenarios and select one to
run. They should be able to easily modify the parameters which the named scenario provides in
template variables. They should be able to run the configured scenario from the UI.
@phact (Sebastian) has already built one of these as a prototype. It would be a good starting point
for learning how web apps work in NoSQLBench. It will also need some updates to mesh well with
recent named scenarios features. This app needs to be tested and integrated into the main NBUI view.
Users should be able to run arbitrary commands in the scenario runner as well, just as they would on
the command line. This would allow them to use and test the rest of the UI features from a single
starting point.
### Scenario Status
An instance of a NoSQLBench process can run multiple scenarios, even concurrently. A user running
scenarios from NBUI will want to be able to see their overall status. This does not include metrics
status at this point, since the best view of this is largely provided already within the docker
metrics view.
Sketch: The scenario status app should list an info panel for each started or completed scenario. It
should contain the following details:
* The name of the scenario (All scenarios have a provided or auto-generated name)
* The commands which were used to start the scenario
* When it was started
* When it completed OR The current progress
* An ETA of completion
### VirtData Function Sandbox
A user should be able to find functions, chain them together, adjust their parameters within valid
ranges/values, and see example outputs over some input cycle range.
The preview of output values should be selectable between these types of visualizations:
- A histogram plot
- A list of values
Each output type is either numeric or non-numeric. In the degenerate case, numeric values should be
allowed to be converted to string form.
A user should be able to choose how they visualize the output: As either list of cycle to value
mappings, or a list of values, a summary of list values, or as a histogram. Additionally, numeric
output types should be plottable if possible.
### Workload Builder
Users should be able to build their workloads from the ground up. They should be able to take an
existing workload yaml as a starting point and tailor it as needed, and then save it.
This can take on the form of a syntax checking text editor, or as a node-based editor that allows
you to only add valid elements in the right place. In any case, the yaml form of the workload should
be accessible.

BIN
devdocs/devguide/nbui/dev_mode.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 128 KiB

50
devdocs/devguide/nbui/dev_mode.puml Normal file
View File

@@ -0,0 +1,50 @@
@startuml
title NoSQLBench docserver in dev mode
participant "UserBrowser" as u
participant "Nuxt dev mode\n:3000 (HTTP)" as dev
participant "NoSQLBench\n:12345 (HTTP)" as nb
participant "Static Content\nService" as content
participant "Markdown Content\nService" as SPI
u -> dev : Load Page
activate dev
dev -> dev : render Vue app
u <- dev : <app resources>
deactivate dev
note over dev
Vue.js develpment occurs
in the Nuxt/Vue instance,
and supports dynamic layout
and reloading.
end note
u -> nb: Read Content Manifest
note over nb
Nuxt/Vue selects
this port for services
when Nuxt is in dev mode
on port 3000
end note
activate nb
nb -> SPI : List Content
activate SPI
SPI -> SPI: Discover and \nEnumerate
nb <- SPI : markdown\nmanifest
deactivate SPI
u <- nb: markdown\nmanifest
deactivate nb
u -> nb: Read Named Content
activate nb
nb -> SPI : Lookup
activate SPI
SPI -> SPI: Lookup
nb <- SPI : Named Content
deactivate SPI
u <- nb: Named Content
deactivate nb
@enduml

BIN
devdocs/devguide/nbui/embedded_mode.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 116 KiB

46
devdocs/devguide/nbui/embedded_mode.puml Normal file
View File

@@ -0,0 +1,46 @@
@startuml
title NoSQLBench docserver in embedded mode
participant "UserBrowser" as u
participant "NoSQLBench\n:12345 (HTTP)" as nb
participant "Static Content\nService" as content
participant "Markdown Content\nService" as SPI
u -> nb : Load Page
activate nb
nb -> content : Read Generated App
activate content
nb <- content : static files
deactivate content
u <- nb : HTTP Content
deactivate nb
note over u
The client app initializes
and makes subsequent
calls to the origin server
for content and services
end note
u -> nb: Read Content Manifest
activate nb
nb -> SPI : List Content
activate SPI
SPI -> SPI: Discover and \nEnumerate
nb <- SPI : markdown\nmanifest
deactivate SPI
u <- nb: markdown\nmanifest
deactivate nb
u -> nb: Read Named Content
activate nb
nb -> SPI : Lookup
activate SPI
SPI -> SPI: Lookup
nb <- SPI : Named Content
deactivate SPI
u <- nb: Named Content
deactivate nb
... Subsequent calls are similar ...
@enduml

29
devdocs/devguide/nbui/nbui_design_guide.md Normal file
View File

@@ -0,0 +1,29 @@
# NBUI Design Guide
General design settings for new NBUI work:
```css
{
font-family: "Mulish, sans-serif"
}
```
## Dashboard Views
- Use layered sections with clear headings.
- Populate each section with cards that layout left to right.
- Show all cards, or allow the view to scroll horizontally with arrows.
- If scrolling horizontally, show the number of elements and allow the
user to show them all in a left-to-right, top-to-bottom layout.
- Each card should show top-level stats and state of the element.
- Each card should have useful hover info for key details.
- Each card should have clickable links to zoom into element details on
different views where available.
- For actions which can affect the elements in a section, use a clearly
labeled button across the top right of the section, horizontally on the
same level as the section heading.
- Add sections which offer help for the user where possible, including
- links to docs, guides, or related services or integrations
- links to videos with preview

87
devdocs/devguide/project_structure.md Normal file
View File

@@ -0,0 +1,87 @@
# Project Structure
nosqlbench is packaged as a
[Maven Reactor](https://maven.apache.org/guides/mini/guide-multiple-modules.html)
project.
## Defaults and Dependencies
Maven reactor projects often confuse developers. In this document, we'll
explain the basic structure of the nosqlbench project and the reasons for
it.
Firstly, there is a parent for each of the modules. In Maven parlance, you
can think of a parent project as a defaults template for projects that
reference it. One of the reasons you would do this is to supply common
build or dependency settings across many maven projects or modules. That
is exactly why we do that here. The 'parent' project for all nosqlbench
modules is aptly named 'mvn-defaults', as that is exactly what we use it
for.
As well, there is a "root" project, which is simply the project at the
project's base directory. It pulls in the modules of the project
explicitly as in:
~~~
<modules>
<module>mvn-defaults</module>
<module>nb</module>
<module>nb-api</module>
<module>nb-annotations</module>
...
~~~
This means that when you build the root project, it will build all the
modules included, but only after linearizing the build order around the
inter-module dependencies. This is an important detail, as it is often
overlooked that this is the purpose of a reactor-style project.
The dependencies between the modules is not implicit. Each module listed
in the root pom.xml has its own explicit dependencies to other modules in
the project. We could cause them to have a common set of dependencies by
adding those dependencies to the 'mvn-defaults' module, but this would
mostly prevent us from making the dependencies for each as lean and
specific as we like. That is why the dependencies in the mvn-defaults **
module** module are very limited. Only those modules which are to be taken
for granted as dependencies everywhere in the project should be added to
the mvn-defaults module.
The mvn-defaults module contains build, locale, and project identity
settings. You can consider these cross-cutting aspects of all of the other
modules in the project. If you want to put something in the mvn-defaults
module, and it is not strictly cross-cutting across the other modules,
then don't. That's how you keep maven reactor projects functioning and
maintainable.
To be clear, cross-cutting build behavior and per-module dependencies are
two separate axes of build management. Try to keep this in mind when
thinking about modular projects.
## Inter-module Dependencies
Modularity at runtime is enabled via the
[Java ServiceLoader](https://docs.oracle.com/javase/8/docs/api/java/util/ServiceLoader.html)
.
The engine-core module uses the engine-api module to know the loadable
activity types. Driver implementations use the engine-api module to
implement the loadable activity types. In this way, they both depend on
the engine-api module to provide the common types needed for this to work.
In this way, multiple modules depending on a single API allows them to
speak together using the language of that API. This also means that the
API must be external to each of the participating modules which use it as
a communication layer.
The nb module allows the separate implementations of the core and the
activity type implementations to exist together in the same classpath.
This occurs specifically because the nb module is an _aggregator_
module which depends on multiple other modules. When the artifact for the
nb module is built, it has all these dependencies together. Since the nb
module is built in a way to include all dependencies, a jar is built that
contains them all in one single file.
## Module Naming
A consistent naming scheme is prescribed for all modules within
nosqlbench, as described in [MODULES](../../MODULES.md)

27
devdocs/devguide/scripting_extensions.md Normal file
View File

@@ -0,0 +1,27 @@
# Scripting Extensions
When a scenario runs, it executes a scenario control script. This script *
is* the scenario in programmatic terms. Even when users don't see the
scenario script as such, it is still there as an intermediary between the
user commands and the core runtime of NoSQLBench. As an executive control
layer, the scenario script doesn't directly run operations with drivers,
although it is able to observe and modify activity metrics and parameters
in real time.
## Scripting Environment
The scripting environment is a _javascript_ environment powered by
GraalJS. Initially, a set of service objects is published into this
environment that allows for the scenario, activity parameters, and
activity metrics to all be accessed directly.
## Scripting Extensions
Additional services can be published into the scripting environment when
it is initialized. These services are simply named objects. They are found
using SPI Java mechanism as long as they are published as services with
a `@Service(ScriptingPluginInfo.class)` annotation and
implement `ScriptingPluginInfo<T>`. This API is pretty basic. You can look
at the `ExamplePluginData` class for a clear example for how to build a
scripting extension.