diff --git a/devdocs/sketches/argfiles.md b/devdocs/sketches/argfiles.md deleted file mode 100644 index b29b34db5..000000000 --- a/devdocs/sketches/argfiles.md +++ /dev/null @@ -1,87 +0,0 @@ -# Args Files - -An argsfile (Command Line Arguments File) is a simple text file which - contains defaults for command-line arguments. You can use an args - file to contain a set of global defaults that you want to use by - default and automatically. - -A command, `--argsfile ` is used to specify an args file. You can - use it like an instant import statement in the middle of a command - line. Notice that this option uses only a single dash. This - distinguishes the argsfile options from the others in general. These are meta - options which can modify how options are loaded, so it is important - that the look distinctive from everything else. - -## Default argsfile - -The default args file location is `$HOME/.nosqlbench/argsfile`. If this -file is present, it is loaded by nosqlbench when it starts even if you - don't ask it to. That is, nosqlbench behaves as if your first set of - command line arguments is `--argsfile "$HOME/.nosqlbench/argsfile - `. However, unlike when you specify `--argsfile ...` explicitly on - your command line, this form will not throw an error if the file is - missing. This means that when you explicitly ask for an args file - to be loaded, and it does not exist, an error is thrown. If you - don't ask for it, but the default one does exist, it is loaded - automatically before other options are processed. - -## Args file format - -An args file simply contains an argument on each line, like this: - - --docker-metrics - --annotate all - --grafana-baseurl http://localhost:3000/ - -## Pinning options - -It is possible to pin an option to the default args file by use of the - `--pin` meta-option. This option will take the following command line - argument and add it to the currently active args file. That means, if - you use `--pin --docker-metrics`, then `--docker-metrics` is added to - the args file. If there is an exact duplicate of the same option - and value, then it is skipped, but if the option name is the same - with a different value, then it is added at the end. This allows - for options which may be called multiple times normally. - -If the `--pin` option occurs after an explicit use of `--argsfile - `, then the filename used in this argument is the one that - is modified. - -After the `--pin` option, the following argument is taken as any global - option (--with-double-dashes) and any non-option values after it which - are not commands (reserved words) - -When the `--pin` option is used, it does not cause the pinned option - to be excluded from the current command line call. The effects of the - pinned option will take place in the current nosqlbench invocation - just as they would without the `--pin`. However, when pinning global - options when there are no commands on the command line, nosqlbench - will not run a scenario, so this form is suitable for setting - arguments. - -As a special case, if the `--pin` is the last option of - -## Unpinning options. - -To reverse the effect of pinning an option, you simply use - `--unpin ...`. - -The behavior of --unpin is slightly different than --pin. Specifically, -an option which is unpinned will be removed from the arg list, and will - not be used in the current invocation of nosqlbench after removal. - -Further, you can specify `--unpin --grafana-baseurl` to unpin an option - which - normally has an argument, and all instances of that argument will be - removed. If you want to unpin a specific instance of a multi-valued - option, or one that can be specified more than once with different - parameter values, then you must provide the value as well, as in - `--unpin --log-histograms 'histodata.log:.*:1m'` - -# Setting defaults, the simple way - -To simply set global defaults, you can run nosqlbench with a command - line like this: - - ./nb --pin --docker-metrics-at metricsnode --pin --annotate all diff --git a/engine-cli/src/main/resources/argsfile.md b/engine-cli/src/main/resources/argsfile.md new file mode 100644 index 000000000..6dba1011c --- /dev/null +++ b/engine-cli/src/main/resources/argsfile.md @@ -0,0 +1,101 @@ +# Args Files + +An argsfile (Command Line Arguments File) is a simple text file which +contains defaults for command-line arguments. You can use an args +file to contain a set of global defaults that you want to use by +default and automatically. + +A command, `--argsfile ` is used to specify an args file. You can +use it like an instant import statement in the middle of a command +line. There are three variants of this option. The `--argsfile ` +variant will warn you if the file doesn't exist. If you want to load +an argsfile if it exist, but avoid warnings if it doesn't, then use +the `--argsfile-optional ` form. If you want to throw an error if +the argsfile doesn't exist, then use the `--argsfile-required ` +form. + +## Default argsfile + +The default args file location is `$NBSTATEDIR/argsfile`. + +After the NBSTATEDIR environment variable or default is resolved, +the default argsfile will be searched for in that directory. + +`$NBSTATEDIR` is a mechanism for setting and finding the local state +directory for NoSQLBench. It is a search path, delimited by +colons, and allowing Java system properties and shell environment +variables. When the NBSTATEDIR location is first needed, +the paths are checked in order, and the first one found is used. +If one is not found on the filesystem, the first expanded value +is used to create the state directory. + + +If the default argsfile is is present, it is loaded by nosqlbench when +it starts even if you don't ask it to. That is, nosqlbench behaves as + if your first set of command line arguments is + + --argsfile-optional "$NBSTATEDIR/argsfile + +Just as with the NBSTATEDIR location, the argsfile can also be used +like a search path. That is, if the value provided is a colon-delimited +set of possible paths, the first one found (after variable expansion) +will be used. If needed, the first expanded path will be used to create +an argsfile when pinning options are used. + +## Args file format + +An args file simply contains an argument on each line, like this: + + --docker-metrics + --annotate all + --grafana-baseurl http://localhost:3000/ + +## Pinning options + +It is possible to pin an option to the default args file by use of the +`--pin` meta-option. This option will take the following command line +argument and add it to the currently active args file. That means, if +you use `--pin --docker-metrics`, then `--docker-metrics` is added to +the args file. If there is an exact duplicate of the same option +and value, then it is skipped, but if the option name is the same +with a different value, then it is added at the end. This allows +for options which may be called multiple times normally. + +If the `--pin` option occurs after an explicit use of `--argsfile +`, then the filename used in this argument is the one that +is modified. + +After the `--pin` option, the following argument is taken as any global +option (--with-double-dashes) and any non-option values after it which +are not commands (reserved words) + +When the `--pin` option is used, it does not cause the pinned option +to be excluded from the current command line call. The effects of the +pinned option will take place in the current nosqlbench invocation +just as they would without the `--pin`. However, when pinning global +options when there are no commands on the command line, nosqlbench +will not run a scenario, so this form is suitable for setting +arguments. + +## Unpinning options. + +To reverse the effect of pinning an option, you simply use + `--unpin ...`. + +The behavior of --unpin is slightly different than --pin. Specifically, +an option which is unpinned will be removed from the arg list, and will +not be used in the current invocation of nosqlbench after removal. + +Further, you can specify `--unpin --grafana-baseurl` to unpin an option +which normally has an argument, and all instances of that argument will be +removed. If you want to unpin a specific instance of a multi-valued +option, or one that can be specified more than once with different +parameter values, then you must provide the value as well, as in +`--unpin --log-histograms 'histodata.log:.*:1m'` + +# Setting defaults, the simple way + +To simply set global defaults, you can run nosqlbench with a command +line like this: + + ./nb --pin --docker-metrics-at metricsnode --pin --annotate all diff --git a/engine-cli/src/main/resources/docker-metrics.md b/engine-cli/src/main/resources/docker-metrics.md new file mode 100644 index 000000000..b4039b678 --- /dev/null +++ b/engine-cli/src/main/resources/docker-metrics.md @@ -0,0 +1,93 @@ +# docker-metrics + +Enlist nosqlbench to stand up your metrics infrastructure using a local +docker runtime: + + --docker-metrics + +When this option is set, nosqlbench will start graphite, prometheus, +and grafana dockers (if-needed) automatically on your local system +, configure them to work together, and point nosqlbench to send metrics +and annotations to the system automatically. + +The inclued NoSQLBench dashboard uses the default grafana credentials of + admin:admin. You can find this dashboard by browsing to the "manage + dashboards" section of grafana. + +# remote docker-metrics + +It is possible to use `--docker-metrics` to set up a metrics collector +stack on one system and use it from multiple other systems. In order +to point client system at the collector, use an option like this: + + --docker-metrics-at 192.168.192.168 + +This will configure graphite and grafana annotations to point to +the docker stack at the configured address. + +Further, if you want to do one-time configuration on the collector node +and other nodes, you can use this pattern: + + # on the collector node + ... --pin --docker-metrics + + # on other nodes + ... --pin --docker-metrics-at + +This causes these options to be configured by default in an argsfile +at `$HOME/.nosqlbench/argsfile`. The options above are pinned to +be included by default in every command run from that point forward. + +## Docker Details + +If you want to know exactly what nosqlbench is doing, it's the equivalent +of running the following by hand: + + # pull and run the graphite-exporter container + docker run -d -p 9108:9108 -p 9109:9109 -p 9109:9109/udp prom/graphite-exporter + +Configuration files which are used by the docker containers are stored in: + + $HOME/.nosqlbench + +## Resetting docker state + +If you need to clear the state for a local docker metrics stack, you + can remove these directories. + + # DASHBOARDS AND METRICS WILL BE LOST IF YOU DO THIS + rm ~/.nosqlbench/{grafana,prometheus,prometheus-conf,graphite-exporter} + +## Manually installing dockers + + # pull and run the prometheus container + docker run -d -p 9090:9090 -v '/.prometheus:/etc/prometheus' prom/prometheus --config.file=/etc/prometheus/prometheus.yml" --storage.tsdb.path=/prometheus" --storage.tsdb.retention=183d --web.enable-lifecycle + + # pull and run the grafana container + docker run -d -p 3000:3000 -v grafana/grafana + +## Experimental environment variables + +These may allow you to send snapshot data to a specially configured +remote grafana instance. + + GF_SECURITY_ADMIN_PASSWORD=admin + GF_AUTH_ANONYMOUS_ENABLED="true" + GF_SNAPSHOTS_EXTERNAL_SNAPSHOT_URL=http://54.165.144.56:3001 + GF_SNAPSHOTS_EXTERNAL_SNAPSHOT_NAME="Send to Wei" + +## Configuration Endpoints (Experimental) + +You can use the grafana api to set up the datasource and dashboard +if you have other tools which integrate with grafana: + + # These are not commands, they are only provides API parameters + + POST http://localhost:3000/api/dashboards/db + analysis.json + # (found in resources/docker/dashboards/analysis.json) + + POST http://localhost:3000/api/datasources + prometheus-datasource.yaml + # (found in resources/docker/datasources/prometheus-datasource.yaml) + diff --git a/engine-docs/src/main/resources/docs-for-ebhistoric/nb-cli-docs/threads.md b/engine-cli/src/main/resources/threads.md similarity index 100% rename from engine-docs/src/main/resources/docs-for-ebhistoric/nb-cli-docs/threads.md rename to engine-cli/src/main/resources/threads.md diff --git a/engine-cli/src/main/resources/topics.md b/engine-cli/src/main/resources/topics.md index 14cf57c7b..6541bdd77 100644 --- a/engine-cli/src/main/resources/topics.md +++ b/engine-cli/src/main/resources/topics.md @@ -1,3 +1,6 @@ # global help topics - commandline - cli-scripting +- argsfile +- threads +- docker-metrics diff --git a/engine-docs/src/main/resources/docs-for-ebhistoric/nb-cli-docs/docker.md b/engine-docs/src/main/resources/docs-for-ebhistoric/nb-cli-docs/docker.md deleted file mode 100644 index 8f9900bf0..000000000 --- a/engine-docs/src/main/resources/docs-for-ebhistoric/nb-cli-docs/docker.md +++ /dev/null @@ -1,60 +0,0 @@ -## docker metrics - -### summary - -Enlist nosqlbench to stand up your metrics infrastructure using a local docker runtime: - - --docker-metrics - -When this option is set, nosqlbench will start graphite, prometheus, and grafana automatically -on your local docker, configure them to work together, and point nosqlbench to send metrics -the system automatically. It also imports a base dashboard for nosqlbench and configures grafana -snapshot export to share with a central DataStax grafana instance (grafana can be found on localhost:3000 -with the default credentials admin/admin). - -### details - -If you want to know exactly what nosqlbench is doing, it's the equivalent of running the following by hand: - -#### pull and run the graphite-exporter container - - docker run -d -p 9108:9108 -p 9109:9109 -p 9109:9109/udp prom/graphite-exporter - -#### prometheus config - -place prometheus config in .prometheus: - -prometheus.yml (found in resources/docker/prometheus/prometheus.yml) - - -#### pull and run the prometheus container - - docker run -d -p 9090:9090 -v '/.prometheus:/etc/prometheus' prom/prometheus --config.file=/etc/prometheus/prometheus.yml" --storage.tsdb.path=/prometheus" --storage.tsdb.retention=183d --web.enable-lifecycle - -#### pull and run the grafana container - - docker run -d -p 3000:3000 -v grafana/grafana - -with the following environment variables: - - GF_SECURITY_ADMIN_PASSWORD=admin - GF_AUTH_ANONYMOUS_ENABLED="true" - GF_SNAPSHOTS_EXTERNAL_SNAPSHOT_URL=http://54.165.144.56:3001 - GF_SNAPSHOTS_EXTERNAL_SNAPSHOT_NAME="Send to Wei" - -#### configure grafana - -use the grafana api to set up the datasource and dashboard - -POST -http://localhost:3000/api/dashboards/db - -Payload: -analysis.json (found in resources/docker/dashboards/analysis.json) - -POST -http://localhost:3000/api/datasources - -Payload: -prometheus-datasource.yaml (found in resources/docker/datasources/prometheus-datasource.yaml) -