IntenseFinance/nosqlbench
3
0
Fork 0
mirror of https://github.com/nosqlbench/nosqlbench.git synced 2025-02-25 18:55:28 -06:00
Code Issues Packages Projects Releases Wiki Activity

improve docs for new developers

Browse Source
This commit is contained in:
Jonathan Shook
2020-09-08 17:47:19 -05:00
parent 7ba14409ad
commit 073a0a1995
6 changed files with 326 additions and 72 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
.github/ISSUE_TEMPLATE/bug_report.md vendored
View File

@@ -19,6 +19,9 @@ A clear and concise description of what you expected to happen.
**Additional context**
Add any other context about the problem here.
OS: Windows, Linux (distribution), macOS ...
environment: k8s, docker, ...
version info (`./nb --version` or `java -jar nb.jar --version`)
**Screenshots, if applicable**
If applicable, add screenshots to help explain your problem.

92
CONTRIBUTING.md
View File

@@ -1,18 +1,35 @@
NoSQLBench is an ambitious project. It aims to solve long-standing problems in distributed systems
testing. There are *many* ways you can contribute! Please take a moment to review this document
testing. There are *many* ways you can contribute! Please take a moment to review this document
in order to make the contribution process easy and effective for everyone involved.
## Code of Conduct
This project follows a [CODE_OF_CONDUCT](CODE_OF_CONDUCT.md).
Please read through it at least once if you are going to contribute to our endeavor.
This project follows a [CODE_OF_CONDUCT](CODE_OF_CONDUCT.md). Please read
through it at least once if you are going to contribute to NoSQLBench.
## Licensing
All source code in this repository is licensed exclusively under
[Apache License, Version 2.0](http://www.apache.org/licenses/LICENSE-2.0).
## Ways to Contribute
## Jumping Right In
Some quick how-to docs have been written for some of the subject-matter
areas in NoSQLBench. If you need an onramp that is not listed, here let us
know!
[I am a developer and I want to contribute a driver.](devdocs/contributing_drivers.md)
[I am a user and I want to improve the documentation.](devdocs/improving_documentation.md)
[I am a user and I want to contribute built-in scenarios.](devdocs/contributing_builtins.md)
[I am a UI developer and I want to improve the NoSQLBench UI (NBUI)](devdocs/developint_nbui.md)
## Contribution Ideas
There are lots of ways to contribute to the project. Some ideas on how to
get started are described here.
### Feedback
@@ -22,23 +39,23 @@ All source code in this repository is licensed exclusively under
This can be helpful to help us understand better what conditions cause the issue to occur. If want to help
address a non-trivial issue that has been reported, you can follow the steps that the original user
reported. Updating issues with such findings is immensely helpful to the maintainers.
### Pull Requests
All contributions to this project are made in the form of pull requests, including:
- Easy Picks - Any issue labeled `easy pick` is a good first issue for new contributors. These are tagged
by the maintainers specifically to provide a gentle on-ramp for those joining our endeavor.
- Fixing bugs - Bug fixes are awesome! If you submit a pull request with a bug fix, we will review it
and provide feedback if refinements are needed. Otherwise we'll merge it in for the next release.
- Testing Improvements - Often, a good test is the best form of documentation. Test coverage can always
be improved. Writing tests is also a great way to get familiar with the project up close.
- Writing documentation - Our users need good documentation. Good documentation is, however, a community
effort. If you have used NoSQLBench and want to improve the journey for others, good documentation
with examples is really helpful.
with examples is really helpful.
- Writing web apps - The doc system that is bunded with NoSQLBench is a web application. This is just
the first web app, but we expect there will be more. If you are interested in helping with this part,
@@ -49,41 +66,54 @@ All contributions to this project are made in the form of pull requests, includi
with internal architecture, there is work to be done. This is on the more advanced end of contribution,
and requires some study of the codebase. If you take the time to understand the code a bit before submitting
pull requests, then the maintainers will take the time to help you make your pull requests better.
### Maintainers
We are looking for more community ownership in this project. That means that we will be supportive of
new project contributors who wish to build a sense of trust with the maintainers.
- Reviewing Pull Requests - If you are a maintainer on this project, then you may be called on to review
pull requests. Unless requested directly, pull requests will be allocated to reviewers automatically.
- Enforcing Conduct - You may be called up on as a maintainer of this project to enforce the code of conduct
as it is written.
- Developing Maintainers - It is important that there be a solid core of project maintainers who can depend
on one another and whom the project user base can depend on do govern the project fairly and equitably.
As a project maintainer, you will be expected to help guide contributors as they learn about the project.
You may be responsible for choosing maintainers or voting on maintainer membership.
- Reviewing Pull Requests - If you are a maintainer on this project, then
you may be called on to review pull requests. Unless requested directly,
pull requests will be allocated to reviewers automatically.
- Enforcing Conduct - You may be called up on as a maintainer of this
project to enforce the code of conduct as it is written.
- Developing Maintainers - It is important that there be a solid core of
project maintainers who can depend on one another and whom the project
user base can depend on do govern the project fairly and equitably. As a
project maintainer, you will be expected to help guide contributors as
they learn about the project. You may be responsible for choosing
maintainers or voting on maintainer membership.
- Documenting APIs - The developer docs are pretty lean right now. We need
some examples of building activity types, adapting statement templates
and so on. This level of contribution requires an intimate awareness of
how the core engine works, but it is also some of the most valuable work
that you can do in terms of expanding the nosqlbench ecosystem.
Contributions are welcome here!
- Documenting APIs - The developer docs are pretty lean right now. We need some examples of building
activity types, adapting statement templates and so on. This level of contribution requires an intimate
awareness of how the core engine works, but it is also some of the most valuable work that you can
do in terms of expanding the nosqlbench ecosystem. Contributions are welcome here!
## Outreach
Help us get the word out on NoSQLBench. It is a newly opened project in its current form, and we
are eager to get it into the hands of users who need it.
- Writing blog posts - A blog post that helps others see the good in NoSQLBench is a service to our community.
This is even better when it comes from a new user who has seen the merit of the tooling. We appreciate any
help.
- Helping other users - Helping new users get to a productive state is a great way to build bridges in the
community. The more community advocates we have helping each other the better!
- Writing blog posts - A blog post that helps others see the good in
NoSQLBench is a service to our community. This is even better when it
comes from a new user who has seen the merit of the tooling. We
appreciate any help.
- Helping other users - Helping new users get to a productive state is a
great way to build bridges in the community. The more community
advocates we have helping each other the better!
- Community Development - The NoSQLBench project endeavors to build a
strong, inclusive, and robust support system around users and
contributors alike. This takes on many forms. It is essential that we
keep looking for ways to connect the NoSQLBench community, doing more of
what works and less of what doesn't. If you want to help with community
development, please join our slack channel and raise your hand!

2
README.md
View File

@@ -6,6 +6,8 @@
[Get it Here](DOWNLOADS.md)
[Contribute to NoSQLBench](CONTRIBUTING.md)
[Read the Docs](http://docs.nosqlbench.io/)
## What is NoSQLBench?

35
devdocs/contributing_drivers.md Normal file
View File

@@ -0,0 +1,35 @@
# Contributing a Driver
Drivers in NoSQLBench are how you get the core machinery to speak a
particular protocol or syntax. At a high level, a NB driver is responsible
for mapping a data structure into an executable operation that can be
called by NoSQLBench.
Internally, the name `ActivityType` is used to name different high level
drivers within NoSQLBench. This should avoid confusion with other driver
terms.
Drivers in NoSQLBench are separate maven modules that get added to the
main nb.jar artifact (and thus the AppImage binary). For now, all these
driver live in-tree, but we may start allowing these to be packaged as
separate jar files. Let us know if this would help your integration
efforts.
## Start with Examples
There are a few activity types which can be used as templates. It is
recommended that you study the stdout activity type as your first example
of how to use the runtime API. The HTTP driver is also fairly new and thus
uses the simpler paths in the API to construct operations. Both of these
recommended as starting points for new developers.
## Consult the Dev Guide
The developers guide is not complete, but it does call out some of the
features that any well-built NB driver should have. If you are one of the
early builders of NB drivers, please help us improve the dev guide as you
find things you wish you had known before, or ways of getting started.
The developer's guid is a work in progress. It lives under
devdocs/devguide in the root of the project.

246
devdocs/devguide/driver_standards.md
View File

@@ -13,56 +13,254 @@ reliable way for users from one driver to another.
Over time, the standards in this guide will be programmatically enforced by the NoSQLBench
driver API.
## Terms
- NB Driver - The NoSQLBench level driver, the code that this document
refers to.
- Native driver - An underlying driver which is provided by a vendor or
project.
## Metrics
At a minimum, a conformant driver should provide the following metrics:
- bind (timer) - A timer around the code that prepares an executable form of a statement
- execute (timer) - A timer around the code that submits work to a native driver
- result (timer) - A timer around the code that awaits and processes results from a native driver. This
timer should be included around all operations, successful ones and errors too.
- result-success (timer) - A timer around the code that awaits and processes results from a native driver.
This timer should only be updated for successful operations.
- errorcounts-... (counters)- Each uniquely named exception or error type that is known to the native driver
should be counted.
- tries (histogram) - The number of tries for a given operation. This number is incremented before each
execution of a native operation, and when the result timer is updated, this value should be updated
as well (for all operations). This includes errored operations.
- bind (timer) - A timer around the code that prepares an executable form
of a statement.
- execute (timer) - A timer around the code that submits work to a native
driver. This is the section of code which enqueues an operation to
complete, but not the part that waits for a response. If a given driver
doesn't have the ability to hand off a request to an underlying driver
asynchronously, then do not include this metric.
- result (timer) - A timer around the code that awaits and processes
results from a native driver. This timer should be included around all
operations, successful ones and errors too. The timer should start
immediately when the operation is submitted to the native ddriver, which
is immediately after the bind timer above is stopped for non-blocking
APIs, or immediately before an operation is submitted to the native
driver API for all others.
- result-success (timer) - A timer around the code that awaits and
processes results from a native driver. This timer should only be
updated for successful operations. The same timer values should be used
as those on the result timer, but they should be applied only in the
case of no exceptions during the operation's execution.
- errorcounts-... (counters)- Each uniquely named exception or error type
that is known to the native driver should be counted.
- tries (histogram) - The number of tries for a given operation. This
number is incremented before each execution of a native operation, and
when the result timer is updated, this value should be updated as well
(for all operations). This includes errored operations.
## Error Handling
Users often want to control what level of sensitivity their tests have to errors. Testing requirements
vary from the basic "shutdown the test when any error occurs" to the more advanced "tell me when the
error rate exceeds some threshold", and so on.
Users often want to control what level of sensitivity their tests have to
errors. Testing requirements vary from the basic "shutdown the test when
any error occurs" to the more advanced "tell me when the error rate
exceeds some threshold", and so on. The essential point here is that
without flexibility in error handling, users may not be able to do
reasonable testing for their requirements, thus configurable error
handling is essential.
Configurable error handling is essential.
Until the error handling subsystem is put in place, these types of error
handling levels are suggested:
1. stop
2. warn
3. retry
4. histogram
5. count
6. ignore
This serves as an error handling stack, where the user chooses the entry
point. From the user-selected entry point, all of the remaining actions
are taken until the end if possible.
### stop
If an exception occurs, and the user has selected the `stop` error
handler, then the activity should be stopped. This is triggered by
allowing the NB driver to propagate a runtime exception up the stack.
Since this error handler interrupts flow of an activity, no further error
handling is done.
### warn
If an exception occurs and the user has selected the `warn` error handler,
the the exception should be logged at WARN level.
The next error handler `retry` should also be called.
### retry
If an exception occurs and the user has selected the `retry` error
handler, **AND** the exception represents a type of error which could
reasonably be expected to be non-persistent, then the operation should be
re-submitted after incrementing the tries metric.
Whether or not the operation is retried, the next error handler
`histogram` should also be called.
### histogram
If an exception occurs and the user has selected the `histogram` error
handler,the error should be recorded with the help class
`ExceptionHistoMetrics`. This adds metrics under the `errorhistos` label
under the activity's name.
The next error handler `count` should also be called.
### count
If an exception occurs and the user has selected the `count` error
handler, then the error should be counted with the helper class
`ExceptionCountMetrics`. This adds metrics under the `errorcounts` label
under the activity's name.
The next exception handler `ignore` should also be called, but this is
simply a named 'no-op' which is generally the last fall-through case in a
switch statement.
## Naming Conventions
### Parameter naming
Parameters should be formatted as snake_case by default. Hyphens or camel
case often cause issues when using mixed media such as command lines and
yaml formats. Snake case is a simple common denominator which works across
all these forms with little risk of ambiguity when parsing or documenting
how parameters are set apart from other syntax.
## Documentation
Each activity is required to have a set of markdown documentation in its resource directory.
The name of the driver should also be used as the name of the documentation for that driver.
Each activity is required to have a set of markdown documentation in its
resource directory. The name of the driver should also be used as the name
of the documentation for that driver.
Additional documentation can be added beyond this file. However, all documentation for a given driver
must start with the drivers name and a hyphen.
Additional documentation can be added beyond this file. However, all
documentation for a given driver must start with the drivers name and a
hyphen.
If a driver wants to include topics, the convention is to mention these
other topics within the driver's main help. Any markdown file which is
included in the resources of a driver module will be viewable by users
with the help command `nb help <name>`. For example, if a driver module
contains `../src/main/resources/mydriver-specials.md`, then a user would
be able to find this help by running `nb help mydriver-specials`.
## Named Scenarios
Confirmant driver implementations should come with one or more examples of a workload under the activities directory path.
Useful driver implementations should come with one or more examples of a
workloads under the activities directory path. These examples should
employ the "named scenarios" format as described in the main docs. By
including named scenarios in the yaml format, these named scenarios then
become available to users when they look for scenarios to call with the
`--list-scenarios` command.
Complete driver implementations should also come with a set of eaxmples under the examples directory path.
To include such scenario, simply add a working yaml with a scenarios
section to the root of your module under the
`src/main/resources/activities` directory.
## Examples
## Included Examples
Useful driver implementations should come with a set of examples under the
examples directory path which demonstrate useful patterns, bindings, or
statement forms.
Users can find these examples in the same way as they can find the named
scenarios above with the only difference being their location. By
convention the directory `src/main/resources/examples` directory is where
these are located.
The format is the same as for named scenarios, because the examples *are*
named scenarios. Users can find these by using the `--include=examples`
option in addition to the `--list-scenarios` command.
## Testing and Docs
Testing Examples in Docs vs Testing Doc Examples, practices and intentions
Unit testing within the NB code base is necessary in many places, but not
in others. Use your judgement about when to *not* add unit testing, but
default to adding it when it seems subjective. A treatise on when and how
to choose appropriate unit testing won't fit here, but suffice it to say
that you can always ask the project maintainers for help on this if you
need.
Non-trivial code in pull requests without any form of quality checks or
testing will not be merged until or unless the project maintainers are
satisfied that there is little risk of user impact. Experimental features
clearly labeled as such will be given more wiggle room here, but the label
will not be removable unless/until a degree of robustness is proven in
some testing layer.
### Testing Futures
In the future, the integration testing and the docs system are intended to
become part of one whole. Particularly, docs should provide executable
examples which can also be used to explain how NB or drivers work. Until
this is done, use the guidelines above.
## Usage of the Op Template
The operation which will be executed in a driver should be derivable from
the YAML as provided by a user. Thankfully, NoSQLBench goes to great
lengths to make this easy for both to the user and to the driver
developer. In particular, NB presents the user with a set of formatting
options in the YAML which are highly flexible in terms of syntax and
structure. On the other side, it presents the driver developer with a
service interface which contains all the input from the user as a complete
data structure.
This means that the driver developer needs to make it clear how different
forms of content from the YAML will map into an operation. Fundamentally,
a driver is responsible for mapping the fully realized data structure of
an `op template` into an executable operation by NoSQLBench.
In some protocols or syntaxes, the phrase _the statement_ makes sense, as
it is the normative way to describe what an operation does. This is true,
for example, with CQL. In CQL You have a statement which provides a very
clear indication of what a user is expecting to happen. At a more abstract
level, and more correctly, the content that a user puts into the YAML is a
`statement template`, and more generally within NoSQLBench, and `operation
template`. This is simply called an `op template` going forward, but as a
driver developer, you should know that these are simply different levels
of detail around the same basic idea: an executable operation derived from
a templating format.
Since there are different ways to employ the op template, a few examples
are provided here. As a driver developer, you should make sure that your
primary docs include examples like these for users. Good NB driver docs
will make it clear to users how their op templates map to executable
operations.
### op template: statement form
In this form, the op template is provided as a map, which is also an
element of the statements array. The convention here is that the values
for and _the statement_name_ and _the statement_ are taken as the first
key and value. Otherwise, the special properties `name` and `stmt` are
explicitly recognized.
```text
statements:
- aname: the syntax of the statement with binding {b1}
tags:
tag1: tagvalue1
params:
param1: paramvalue1
freeparamfoo: freevaluebar
bindings:
b1: NumberNameToString()
```
### op template: map form
```text
statements:
- cmd_type:
```
Structural variations and conventions.
## Handling secrets
Reading passwords ...
Reading passwords ...

20
driver-http/src/main/java/io/nosqlbench/activitytype/http/HttpActivity.java
View File

@@ -1,33 +1,19 @@
package io.nosqlbench.activitytype.http;
import io.nosqlbench.activitytype.cmds.ReadyHttpRequest;
import io.nosqlbench.engine.api.activityconfig.ParsedStmt;
import io.nosqlbench.engine.api.activityconfig.StatementsLoader;
import io.nosqlbench.engine.api.activityconfig.yaml.StmtDef;
import io.nosqlbench.engine.api.activityconfig.yaml.StmtsDocList;
import com.codahale.metrics.Histogram;
import com.codahale.metrics.Meter;
import com.codahale.metrics.Timer;
import io.nosqlbench.activitytype.cmds.ReadyHttpRequest;
import io.nosqlbench.engine.api.activityapi.core.Activity;
import io.nosqlbench.engine.api.activityapi.core.ActivityDefObserver;
import io.nosqlbench.engine.api.activityapi.planning.OpSequence;
import io.nosqlbench.engine.api.activityapi.planning.SequencePlanner;
import io.nosqlbench.engine.api.activityapi.planning.SequencerType;
import io.nosqlbench.engine.api.activityconfig.yaml.StmtsDocList;
import io.nosqlbench.engine.api.activityimpl.ActivityDef;
import io.nosqlbench.engine.api.activityimpl.SimpleActivity;
import io.nosqlbench.engine.api.metrics.ActivityMetrics;
import io.nosqlbench.engine.api.templating.CommandTemplate;
import io.nosqlbench.engine.api.templating.StrInterpolator;
import io.nosqlbench.nb.api.errors.BasicError;
import io.nosqlbench.virtdata.core.bindings.BindingsTemplate;
import io.nosqlbench.virtdata.core.templates.StringBindings;
import io.nosqlbench.virtdata.core.templates.StringBindingsTemplate;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.util.*;
import java.util.function.Function;
public class HttpActivity extends SimpleActivity implements Activity, ActivityDefObserver {
private final static Logger logger = LoggerFactory.getLogger(HttpActivity.class);
private final ActivityDef activityDef;
@@ -76,7 +62,7 @@ public class HttpActivity extends SimpleActivity implements Activity, ActivityDe
// port = activityDef.getParams().getOptionalInteger("port").orElse(80);
this.opSequence = createOpSequence(ReadyHttpRequest::new);
setDefaultsFromOpSequence(opSequence);
this.setDefaultsFromOpSequence(opSequence);
bindTimer = ActivityMetrics.timer(activityDef, "bind");
executeTimer = ActivityMetrics.timer(activityDef, "execute");