mirror of
https://github.com/nosqlbench/nosqlbench.git
synced 2025-01-24 14:36:32 -06:00
move common topics to resources base
This commit is contained in:
Jonathan Shook
parent
6a8dfef109
commit
fd74ccba69
@ -1,87 +0,0 @@
|
||||
# Args Files
|
||||
|
||||
An argsfile (Command Line Arguments File) is a simple text file which
|
||||
contains defaults for command-line arguments. You can use an args
|
||||
file to contain a set of global defaults that you want to use by
|
||||
default and automatically.
|
||||
|
||||
A command, `--argsfile <path>` is used to specify an args file. You can
|
||||
use it like an instant import statement in the middle of a command
|
||||
line. Notice that this option uses only a single dash. This
|
||||
distinguishes the argsfile options from the others in general. These are meta
|
||||
options which can modify how options are loaded, so it is important
|
||||
that the look distinctive from everything else.
|
||||
|
||||
## Default argsfile
|
||||
|
||||
The default args file location is `$HOME/.nosqlbench/argsfile`. If this
|
||||
file is present, it is loaded by nosqlbench when it starts even if you
|
||||
don't ask it to. That is, nosqlbench behaves as if your first set of
|
||||
command line arguments is `--argsfile "$HOME/.nosqlbench/argsfile
|
||||
`. However, unlike when you specify `--argsfile ...` explicitly on
|
||||
your command line, this form will not throw an error if the file is
|
||||
missing. This means that when you explicitly ask for an args file
|
||||
to be loaded, and it does not exist, an error is thrown. If you
|
||||
don't ask for it, but the default one does exist, it is loaded
|
||||
automatically before other options are processed.
|
||||
|
||||
## Args file format
|
||||
|
||||
An args file simply contains an argument on each line, like this:
|
||||
|
||||
--docker-metrics
|
||||
--annotate all
|
||||
--grafana-baseurl http://localhost:3000/
|
||||
|
||||
## Pinning options
|
||||
|
||||
It is possible to pin an option to the default args file by use of the
|
||||
`--pin` meta-option. This option will take the following command line
|
||||
argument and add it to the currently active args file. That means, if
|
||||
you use `--pin --docker-metrics`, then `--docker-metrics` is added to
|
||||
the args file. If there is an exact duplicate of the same option
|
||||
and value, then it is skipped, but if the option name is the same
|
||||
with a different value, then it is added at the end. This allows
|
||||
for options which may be called multiple times normally.
|
||||
|
||||
If the `--pin` option occurs after an explicit use of `--argsfile
|
||||
<filename>`, then the filename used in this argument is the one that
|
||||
is modified.
|
||||
|
||||
After the `--pin` option, the following argument is taken as any global
|
||||
option (--with-double-dashes) and any non-option values after it which
|
||||
are not commands (reserved words)
|
||||
|
||||
When the `--pin` option is used, it does not cause the pinned option
|
||||
to be excluded from the current command line call. The effects of the
|
||||
pinned option will take place in the current nosqlbench invocation
|
||||
just as they would without the `--pin`. However, when pinning global
|
||||
options when there are no commands on the command line, nosqlbench
|
||||
will not run a scenario, so this form is suitable for setting
|
||||
arguments.
|
||||
|
||||
As a special case, if the `--pin` is the last option of
|
||||
|
||||
## Unpinning options.
|
||||
|
||||
To reverse the effect of pinning an option, you simply use
|
||||
`--unpin ...`.
|
||||
|
||||
The behavior of --unpin is slightly different than --pin. Specifically,
|
||||
an option which is unpinned will be removed from the arg list, and will
|
||||
not be used in the current invocation of nosqlbench after removal.
|
||||
|
||||
Further, you can specify `--unpin --grafana-baseurl` to unpin an option
|
||||
which
|
||||
normally has an argument, and all instances of that argument will be
|
||||
removed. If you want to unpin a specific instance of a multi-valued
|
||||
option, or one that can be specified more than once with different
|
||||
parameter values, then you must provide the value as well, as in
|
||||
`--unpin --log-histograms 'histodata.log:.*:1m'`
|
||||
|
||||
# Setting defaults, the simple way
|
||||
|
||||
To simply set global defaults, you can run nosqlbench with a command
|
||||
line like this:
|
||||
|
||||
./nb --pin --docker-metrics-at metricsnode --pin --annotate all
|
||||
101
engine-cli/src/main/resources/argsfile.md
Normal file
101
engine-cli/src/main/resources/argsfile.md
Normal file
@ -0,0 +1,101 @@
|
||||
# Args Files
|
||||
|
||||
An argsfile (Command Line Arguments File) is a simple text file which
|
||||
contains defaults for command-line arguments. You can use an args
|
||||
file to contain a set of global defaults that you want to use by
|
||||
default and automatically.
|
||||
|
||||
A command, `--argsfile <path>` is used to specify an args file. You can
|
||||
use it like an instant import statement in the middle of a command
|
||||
line. There are three variants of this option. The `--argsfile <path>`
|
||||
variant will warn you if the file doesn't exist. If you want to load
|
||||
an argsfile if it exist, but avoid warnings if it doesn't, then use
|
||||
the `--argsfile-optional <path>` form. If you want to throw an error if
|
||||
the argsfile doesn't exist, then use the `--argsfile-required <path>`
|
||||
form.
|
||||
|
||||
## Default argsfile
|
||||
|
||||
The default args file location is `$NBSTATEDIR/argsfile`.
|
||||
|
||||
After the NBSTATEDIR environment variable or default is resolved,
|
||||
the default argsfile will be searched for in that directory.
|
||||
|
||||
`$NBSTATEDIR` is a mechanism for setting and finding the local state
|
||||
directory for NoSQLBench. It is a search path, delimited by
|
||||
colons, and allowing Java system properties and shell environment
|
||||
variables. When the NBSTATEDIR location is first needed,
|
||||
the paths are checked in order, and the first one found is used.
|
||||
If one is not found on the filesystem, the first expanded value
|
||||
is used to create the state directory.
|
||||
|
||||
|
||||
If the default argsfile is is present, it is loaded by nosqlbench when
|
||||
it starts even if you don't ask it to. That is, nosqlbench behaves as
|
||||
if your first set of command line arguments is
|
||||
|
||||
--argsfile-optional "$NBSTATEDIR/argsfile
|
||||
|
||||
Just as with the NBSTATEDIR location, the argsfile can also be used
|
||||
like a search path. That is, if the value provided is a colon-delimited
|
||||
set of possible paths, the first one found (after variable expansion)
|
||||
will be used. If needed, the first expanded path will be used to create
|
||||
an argsfile when pinning options are used.
|
||||
|
||||
## Args file format
|
||||
|
||||
An args file simply contains an argument on each line, like this:
|
||||
|
||||
--docker-metrics
|
||||
--annotate all
|
||||
--grafana-baseurl http://localhost:3000/
|
||||
|
||||
## Pinning options
|
||||
|
||||
It is possible to pin an option to the default args file by use of the
|
||||
`--pin` meta-option. This option will take the following command line
|
||||
argument and add it to the currently active args file. That means, if
|
||||
you use `--pin --docker-metrics`, then `--docker-metrics` is added to
|
||||
the args file. If there is an exact duplicate of the same option
|
||||
and value, then it is skipped, but if the option name is the same
|
||||
with a different value, then it is added at the end. This allows
|
||||
for options which may be called multiple times normally.
|
||||
|
||||
If the `--pin` option occurs after an explicit use of `--argsfile
|
||||
<filename>`, then the filename used in this argument is the one that
|
||||
is modified.
|
||||
|
||||
After the `--pin` option, the following argument is taken as any global
|
||||
option (--with-double-dashes) and any non-option values after it which
|
||||
are not commands (reserved words)
|
||||
|
||||
When the `--pin` option is used, it does not cause the pinned option
|
||||
to be excluded from the current command line call. The effects of the
|
||||
pinned option will take place in the current nosqlbench invocation
|
||||
just as they would without the `--pin`. However, when pinning global
|
||||
options when there are no commands on the command line, nosqlbench
|
||||
will not run a scenario, so this form is suitable for setting
|
||||
arguments.
|
||||
|
||||
## Unpinning options.
|
||||
|
||||
To reverse the effect of pinning an option, you simply use
|
||||
`--unpin ...`.
|
||||
|
||||
The behavior of --unpin is slightly different than --pin. Specifically,
|
||||
an option which is unpinned will be removed from the arg list, and will
|
||||
not be used in the current invocation of nosqlbench after removal.
|
||||
|
||||
Further, you can specify `--unpin --grafana-baseurl` to unpin an option
|
||||
which normally has an argument, and all instances of that argument will be
|
||||
removed. If you want to unpin a specific instance of a multi-valued
|
||||
option, or one that can be specified more than once with different
|
||||
parameter values, then you must provide the value as well, as in
|
||||
`--unpin --log-histograms 'histodata.log:.*:1m'`
|
||||
|
||||
# Setting defaults, the simple way
|
||||
|
||||
To simply set global defaults, you can run nosqlbench with a command
|
||||
line like this:
|
||||
|
||||
./nb --pin --docker-metrics-at metricsnode --pin --annotate all
|
||||
93
engine-cli/src/main/resources/docker-metrics.md
Normal file
93
engine-cli/src/main/resources/docker-metrics.md
Normal file
@ -0,0 +1,93 @@
|
||||
# docker-metrics
|
||||
|
||||
Enlist nosqlbench to stand up your metrics infrastructure using a local
|
||||
docker runtime:
|
||||
|
||||
--docker-metrics
|
||||
|
||||
When this option is set, nosqlbench will start graphite, prometheus,
|
||||
and grafana dockers (if-needed) automatically on your local system
|
||||
, configure them to work together, and point nosqlbench to send metrics
|
||||
and annotations to the system automatically.
|
||||
|
||||
The inclued NoSQLBench dashboard uses the default grafana credentials of
|
||||
admin:admin. You can find this dashboard by browsing to the "manage
|
||||
dashboards" section of grafana.
|
||||
|
||||
# remote docker-metrics
|
||||
|
||||
It is possible to use `--docker-metrics` to set up a metrics collector
|
||||
stack on one system and use it from multiple other systems. In order
|
||||
to point client system at the collector, use an option like this:
|
||||
|
||||
--docker-metrics-at 192.168.192.168
|
||||
|
||||
This will configure graphite and grafana annotations to point to
|
||||
the docker stack at the configured address.
|
||||
|
||||
Further, if you want to do one-time configuration on the collector node
|
||||
and other nodes, you can use this pattern:
|
||||
|
||||
# on the collector node
|
||||
... --pin --docker-metrics
|
||||
|
||||
# on other nodes
|
||||
... --pin --docker-metrics-at <collector node ip>
|
||||
|
||||
This causes these options to be configured by default in an argsfile
|
||||
at `$HOME/.nosqlbench/argsfile`. The options above are pinned to
|
||||
be included by default in every command run from that point forward.
|
||||
|
||||
## Docker Details
|
||||
|
||||
If you want to know exactly what nosqlbench is doing, it's the equivalent
|
||||
of running the following by hand:
|
||||
|
||||
# pull and run the graphite-exporter container
|
||||
docker run -d -p 9108:9108 -p 9109:9109 -p 9109:9109/udp prom/graphite-exporter
|
||||
|
||||
Configuration files which are used by the docker containers are stored in:
|
||||
|
||||
$HOME/.nosqlbench
|
||||
|
||||
## Resetting docker state
|
||||
|
||||
If you need to clear the state for a local docker metrics stack, you
|
||||
can remove these directories.
|
||||
|
||||
# DASHBOARDS AND METRICS WILL BE LOST IF YOU DO THIS
|
||||
rm ~/.nosqlbench/{grafana,prometheus,prometheus-conf,graphite-exporter}
|
||||
|
||||
## Manually installing dockers
|
||||
|
||||
# pull and run the prometheus container
|
||||
docker run -d -p 9090:9090 -v '<USER HOME>/.prometheus:/etc/prometheus' prom/prometheus --config.file=/etc/prometheus/prometheus.yml" --storage.tsdb.path=/prometheus" --storage.tsdb.retention=183d --web.enable-lifecycle
|
||||
|
||||
# pull and run the grafana container
|
||||
docker run -d -p 3000:3000 -v grafana/grafana
|
||||
|
||||
## Experimental environment variables
|
||||
|
||||
These may allow you to send snapshot data to a specially configured
|
||||
remote grafana instance.
|
||||
|
||||
GF_SECURITY_ADMIN_PASSWORD=admin
|
||||
GF_AUTH_ANONYMOUS_ENABLED="true"
|
||||
GF_SNAPSHOTS_EXTERNAL_SNAPSHOT_URL=http://54.165.144.56:3001
|
||||
GF_SNAPSHOTS_EXTERNAL_SNAPSHOT_NAME="Send to Wei"
|
||||
|
||||
## Configuration Endpoints (Experimental)
|
||||
|
||||
You can use the grafana api to set up the datasource and dashboard
|
||||
if you have other tools which integrate with grafana:
|
||||
|
||||
# These are not commands, they are only provides API parameters
|
||||
|
||||
POST http://localhost:3000/api/dashboards/db
|
||||
analysis.json
|
||||
# (found in resources/docker/dashboards/analysis.json)
|
||||
|
||||
POST http://localhost:3000/api/datasources
|
||||
prometheus-datasource.yaml
|
||||
# (found in resources/docker/datasources/prometheus-datasource.yaml)
|
||||
|
||||
@ -1,3 +1,6 @@
|
||||
# global help topics
|
||||
- commandline
|
||||
- cli-scripting
|
||||
- argsfile
|
||||
- threads
|
||||
- docker-metrics
|
||||
|
||||
@ -1,60 +0,0 @@
|
||||
## docker metrics
|
||||
|
||||
### summary
|
||||
|
||||
Enlist nosqlbench to stand up your metrics infrastructure using a local docker runtime:
|
||||
|
||||
--docker-metrics
|
||||
|
||||
When this option is set, nosqlbench will start graphite, prometheus, and grafana automatically
|
||||
on your local docker, configure them to work together, and point nosqlbench to send metrics
|
||||
the system automatically. It also imports a base dashboard for nosqlbench and configures grafana
|
||||
snapshot export to share with a central DataStax grafana instance (grafana can be found on localhost:3000
|
||||
with the default credentials admin/admin).
|
||||
|
||||
### details
|
||||
|
||||
If you want to know exactly what nosqlbench is doing, it's the equivalent of running the following by hand:
|
||||
|
||||
#### pull and run the graphite-exporter container
|
||||
|
||||
docker run -d -p 9108:9108 -p 9109:9109 -p 9109:9109/udp prom/graphite-exporter
|
||||
|
||||
#### prometheus config
|
||||
|
||||
place prometheus config in .prometheus:
|
||||
|
||||
prometheus.yml (found in resources/docker/prometheus/prometheus.yml)
|
||||
|
||||
|
||||
#### pull and run the prometheus container
|
||||
|
||||
docker run -d -p 9090:9090 -v '<USER HOME>/.prometheus:/etc/prometheus' prom/prometheus --config.file=/etc/prometheus/prometheus.yml" --storage.tsdb.path=/prometheus" --storage.tsdb.retention=183d --web.enable-lifecycle
|
||||
|
||||
#### pull and run the grafana container
|
||||
|
||||
docker run -d -p 3000:3000 -v grafana/grafana
|
||||
|
||||
with the following environment variables:
|
||||
|
||||
GF_SECURITY_ADMIN_PASSWORD=admin
|
||||
GF_AUTH_ANONYMOUS_ENABLED="true"
|
||||
GF_SNAPSHOTS_EXTERNAL_SNAPSHOT_URL=http://54.165.144.56:3001
|
||||
GF_SNAPSHOTS_EXTERNAL_SNAPSHOT_NAME="Send to Wei"
|
||||
|
||||
#### configure grafana
|
||||
|
||||
use the grafana api to set up the datasource and dashboard
|
||||
|
||||
POST
|
||||
http://localhost:3000/api/dashboards/db
|
||||
|
||||
Payload:
|
||||
analysis.json (found in resources/docker/dashboards/analysis.json)
|
||||
|
||||
POST
|
||||
http://localhost:3000/api/datasources
|
||||
|
||||
Payload:
|
||||
prometheus-datasource.yaml (found in resources/docker/datasources/prometheus-datasource.yaml)
|
||||
|
||||
Loading…
Reference in New Issue
Block a user