list : entries) {
+ composed.addAll(list);
+ }
+ return composed;
+ }
+
+ /**
+ * Load the args file into an args array. The returned format follows
+ * the standard pattern of args as you would see for a main method, although
+ * the internal format is structured to support easy editing and clarity.
+ *
+ *
+ * The args file is stored in a simple option-per-line format which
+ * follows these rules:
+ *
+ * - Lines ending with a backslash (\) only are concatenated to the next
+ * line with the backslash removed.
+ *
- Line content consists of one option and one optional argument.
+ * - Options must start with at least one dash (-).
+ * - If an argument is provided for an option, it follows the option and a space.
+ * - Empty lines and lines which start with '//' or '#' are ignored.
+ * - Lines which are identical after applying the above rules are elided
+ * down to the last occurence.
+ *
+ *
+ *
+ *
+ * This allows for multi-valued options, or options which can be specified multiple
+ * times with different arguments to be supported, so long as each occurrence has a
+ * unique option value.
+ *
+ *
+ * @param argspath The path of the argsfile to load
+ * @param mode The level of feedback to provide in the case of a missing file
+ * @return The argsfile content, structured like an args array
+ */
+ private LinkedHashSet readArgsFile(Path argspath, Selection mode) {
+ LinkedHashSet args = new LinkedHashSet<>();
+
+ if (!assertArgsFileExists(argspath, mode)) {
+ return args;
+ }
+
+ List lines = null;
+ try {
+ lines = Files.readAllLines(argspath);
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ List content = lines.stream()
+ .filter(s -> !s.startsWith("#"))
+ .filter(s -> !s.startsWith("/"))
+ .filter(s -> !s.isBlank())
+ .filter(s -> !s.isEmpty())
+ .collect(Collectors.toList());
+ StringBuilder splitword = new StringBuilder();
+ LinkedHashSet loaded = new LinkedHashSet<>();
+ for (String s : content) {
+ splitword.append(s);
+ if (!s.endsWith("\\")) {
+ loaded.add(splitword.toString());
+ splitword.setLength(0);
+ } else {
+ splitword.setLength(splitword.length() - 1);
+ }
+ }
+ if (splitword.length() > 0) {
+ throw new RuntimeException("unqualified line continuation for '" + splitword.toString() + "'");
+ }
+
+ return loaded;
+ }
+
+ /**
+ * Write the argsfile in the format specified by {@link #readArgsFile(Path, Selection)}
+ *
+ * This method requires that an argsFile has been set by a previous
+ * --argsfile or --argsfile-required or --argsfile-optional option.
+ *
+ * @param args The args to write in one-arg-per-line form
+ */
+ private void writeArgsFile(LinkedHashSet args) {
+ if (this.argsPath == null) {
+ throw new RuntimeException("No argsfile has been selected before using the pin option.");
+ }
+
+ try {
+ Files.createDirectories(
+ this.argsPath.getParent(),
+ PosixFilePermissions.asFileAttribute(PosixFilePermissions.fromString("rwxrwx---"))
+ );
+ Files.write(this.argsPath, args);
+ } catch (IOException e) {
+ throw new BasicError("unable to write '" + this.argsPath + "': " + e.getMessage());
+ }
+ }
+
+ private boolean assertArgsFileExists(Path argspath, Selection mode) {
+ if (!Files.exists(argsPath)) {
+ switch (mode) {
+ case ErrorIfMissing:
+ throw new RuntimeException("A required argsfile was specified, but it does not exist: '" + argspath + "'");
+ case WarnIfMissing:
+ logger.warn("An argsfile was specified, but it does not exist: '" + argspath + "'");
+ case IgnoreIfMissing:
+ }
+ return false;
+ }
+ return true;
+ }
+
+ private void setArgsFile(String argspath, Selection mode) {
+ Path selected = null;
+ String[] possibles = argspath.split(":");
+ for (String possible : possibles) {
+ Optional expanded = Environment.INSTANCE.interpolate(possible);
+ if (expanded.isPresent()) {
+ Path possiblePath = Path.of(expanded.get());
+ if (Files.exists(possiblePath)) {
+ selected = possiblePath;
+ break;
+ }
+ }
+ }
+
+ if (selected == null) {
+ String defaultFirst = possibles[0];
+ defaultFirst = Environment.INSTANCE.interpolate(defaultFirst)
+ .orElseThrow(() -> new RuntimeException("Invalid default argsfile: '" + possibles[0] + "'"));
+ selected = Path.of(defaultFirst);
+ }
+
+ this.argsPath = selected;
+ logger.debug("argsfile path is now '" + this.argsPath.toString() + "'");
+ }
+
+ /**
+ * Convert argv arguments to consolidated form which is used in the args file.
+ * This means that options and their (optional) arguments are on the
+ * same line, concatenated with a space after the option.
+ *
+ * @return The arg-per-line form
+ */
+ LinkedHashSet argsToLines(List args) {
+ LinkedHashSet lines = new LinkedHashSet<>();
+ Iterator iter = args.iterator();
+ List element = new ArrayList<>();
+ while (iter.hasNext()) {
+ String word = iter.next();
+ if (word.startsWith("-")) {
+ if (element.size() > 0) {
+ lines.add(Strings.join(element, " "));
+ element.clear();
+ }
+ }
+ element.add(word);
+ }
+ lines.add(Strings.join(element, " "));
+ return lines;
+ }
+
+ /**
+ * Convert arg lines as used in an args file to the argv which
+ * is used on the command line.
+ *
+ * @return The argv list as you would see with {@code main(String[] argv)}
+ */
+ LinkedList linesToArgs(Collection lines) {
+ LinkedList args = new LinkedList<>();
+ for (String line : lines) {
+ if (line.startsWith("-")) {
+ String[] words = line.split(" ", 2);
+ args.addAll(Arrays.asList(words));
+ } else {
+ args.add(line);
+ }
+ }
+ return args;
+ }
+
+ private LinkedList unpin(LinkedList arglist) {
+ if (this.argsPath == null) {
+ throw new RuntimeException("No argsfile has been selected before using the pin option.");
+ }
+ return arglist;
+ }
+
+ /**
+ * Read the current command line option from the argument list,
+ * so long as it is a dash or double-dash option, and is not a
+ * reserved word, and any argument that goes with it, if any.
+ *
+ * @param arglist The command line containing the option
+ * @return A list containing the current command line option
+ */
+ private LinkedList readOptionAndArg(LinkedList arglist, boolean consume) {
+ LinkedList option = new LinkedList<>();
+ ListIterator iter = arglist.listIterator();
+
+ if (!iter.hasNext()) {
+ throw new RuntimeException("Arguments must follow the --pin option");
+ }
+ String opt = iter.next();
+
+ if (!opt.startsWith("-") || stopWords.contains(opt)) {
+ throw new RuntimeException("Arguments following the --pin option must not" +
+ " be commands like '" + opt + "'");
+ }
+ option.add(opt);
+ if (consume) {
+ iter.remove();
+ }
+
+ if (iter.hasNext()) {
+ opt = iter.next();
+ if (!stopWords.contains(opt) && !opt.startsWith("-")) {
+ option.add(opt);
+ if (consume) {
+ iter.remove();
+ }
+ }
+ }
+ return option;
+ }
+
+
+ /**
+ * Consume the next word from the beginning of the linked list. If there is
+ * no word to consume, then throw an error with the description.
+ *
+ * @param commandline A list of words
+ * @param description A description of what the next word value is meant to represent
+ * @return The next word from the list
+ */
+ private String readWordOrThrow(LinkedList commandline, String description) {
+ String found = commandline.peekFirst();
+ if (found == null) {
+ throw new RuntimeException("Unable to read argument top option for " + description);
+ }
+ return commandline.removeFirst();
+ }
+}
diff --git a/engine-cli/src/main/java/io/nosqlbench/engine/cli/NBCLIOptions.java b/engine-cli/src/main/java/io/nosqlbench/engine/cli/NBCLIOptions.java
index c92aa52f3..fe0398152 100644
--- a/engine-cli/src/main/java/io/nosqlbench/engine/cli/NBCLIOptions.java
+++ b/engine-cli/src/main/java/io/nosqlbench/engine/cli/NBCLIOptions.java
@@ -4,11 +4,15 @@ import ch.qos.logback.classic.Level;
import io.nosqlbench.engine.api.metrics.IndicatorMode;
import io.nosqlbench.engine.api.util.Unit;
import io.nosqlbench.engine.core.script.Scenario;
+import io.nosqlbench.nb.api.errors.BasicError;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
import java.nio.file.Path;
+import java.nio.file.attribute.PosixFilePermissions;
import java.security.InvalidParameterException;
import java.util.*;
import java.util.stream.Collectors;
@@ -19,19 +23,16 @@ import java.util.stream.Collectors;
*/
public class NBCLIOptions {
- private final static String userHome = System.getProperty("user.home");
- private final static Path defaultOptFile = Path.of(userHome, ".nosqlbench/options");
-
- private final static Logger logger = LoggerFactory.getLogger(NBCLIOptions.class);
-
- // Options which may contextualize other CLI options or commands.
- // These must be parsed first
- private static final String ARGS_FILE = "--argsfile";
- private static final String ARGS_FILE_DEFAULT = "$HOME/.nosqlbench/argsfile";
- private static final String ARGS_PIN = "--pin";
- private static final String ARGS_UNPIN = "--unpin";
+ private final static Logger logger = LoggerFactory.getLogger("OPTIONS");
+ private final static String NB_STATE_DIR = "--statedir";
+ private final static String NB_STATEDIR_PATHS = "$NBSTATEDIR:$PWD/.nosqlbench:$HOME/.nosqlbench";
+ public static final String ARGS_FILE_DEFAULT = "$NBSTATEDIR/argsfile";
private static final String INCLUDE = "--include";
+
+ private final static String userHome = System.getProperty("user.home");
+
+
private static final String METRICS_PREFIX = "--metrics-prefix";
// private static final String ANNOTATE_TO_GRAFANA = "--grafana-baseurl";
@@ -39,7 +40,6 @@ public class NBCLIOptions {
private static final String ANNOTATORS_CONFIG = "--annotators";
private static final String DEFAULT_ANNOTATORS = "all";
-
// Discovery
private static final String HELP = "--help";
private static final String LIST_METRICS = "--list-metrics";
@@ -91,6 +91,7 @@ public class NBCLIOptions {
private static final String DOCKER_GRAFANA_TAG = "--docker-grafana-tag";
private static final String DEFAULT_CONSOLE_LOGGING_PATTERN = "%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n";
+ public static final String NBSTATEDIR = "NBSTATEDIR";
private final LinkedList cmdList = new LinkedList<>();
private int logsMax = 0;
@@ -130,13 +131,16 @@ public class NBCLIOptions {
private Scenario.Engine engine = Scenario.Engine.Graalvm;
private boolean graaljs_compat = false;
private int hdr_digits = 4;
- private String docker_grafana_tag = "7.0.1";
+ private String docker_grafana_tag = "7.2.2";
private boolean showStackTraces = false;
private boolean compileScript = false;
private String scriptFile = null;
private String[] annotateEvents = new String[]{"ALL"};
private String dockerMetricsHost;
private String annotatorsConfig = "";
+ private String statedirs = NB_STATEDIR_PATHS;
+ private Path statepath;
+ private List statePathAccesses = new ArrayList<>();
public String getAnnotatorsConfig() {
return annotatorsConfig;
@@ -163,8 +167,6 @@ public class NBCLIOptions {
}
private LinkedList parseGlobalOptions(String[] args) {
- ArgsFile argsfile = new ArgsFile();
- argsfile.preload("--argsfile-optional", ARGS_FILE_DEFAULT);
LinkedList arglist = new LinkedList<>() {{
addAll(Arrays.asList(args));
@@ -175,7 +177,8 @@ public class NBCLIOptions {
return arglist;
}
- // Preprocess --include regardless of position
+ // Process --include and --statedir, separately first
+ // regardless of position
LinkedList nonincludes = new LinkedList<>();
while (arglist.peekFirst() != null) {
String word = arglist.peekFirst();
@@ -188,9 +191,53 @@ public class NBCLIOptions {
}
switch (word) {
- case ARGS_FILE:
- case ARGS_PIN:
- case ARGS_UNPIN:
+ case NB_STATE_DIR:
+ arglist.removeFirst();
+ this.statedirs = readWordOrThrow(arglist, "nosqlbench global state directory");
+ break;
+ case INCLUDE:
+ arglist.removeFirst();
+ String include = readWordOrThrow(arglist, "path to include");
+ wantsToIncludePaths.add(include);
+ break;
+ default:
+ nonincludes.addLast(arglist.removeFirst());
+ }
+ }
+ this.statedirs = (this.statedirs != null ? this.statedirs : NB_STATEDIR_PATHS);
+ this.setStatePath();
+
+ arglist = nonincludes;
+ nonincludes = new LinkedList<>();
+
+ // Now that statdirs is settled, auto load argsfile if it is present
+ NBCLIArgsFile argsfile = new NBCLIArgsFile();
+ argsfile.reserved(NBCLICommandParser.RESERVED_WORDS);
+ argsfile.preload("--argsfile-optional", ARGS_FILE_DEFAULT);
+ arglist = argsfile.process(arglist);
+
+ // Parse all --argsfile... and other high level options
+
+ while (arglist.peekFirst() != null) {
+ String word = arglist.peekFirst();
+ if (word.startsWith("--") && word.contains("=")) {
+ String wordToSplit = arglist.removeFirst();
+ String[] split = wordToSplit.split("=", 2);
+ arglist.offerFirst(split[1]);
+ arglist.offerFirst(split[0]);
+ continue;
+ }
+
+ switch (word) {
+ // These options modify other options. They should be processed early.
+ case NBCLIArgsFile.ARGS_FILE:
+ case NBCLIArgsFile.ARGS_FILE_OPTIONAL:
+ case NBCLIArgsFile.ARGS_FILE_REQUIRED:
+ case NBCLIArgsFile.ARGS_PIN:
+ case NBCLIArgsFile.ARGS_UNPIN:
+ if (this.statepath == null) {
+ setStatePath();
+ }
arglist = argsfile.process(arglist);
break;
case ANNOTATE_EVENTS:
@@ -198,19 +245,10 @@ public class NBCLIOptions {
String toAnnotate = readWordOrThrow(arglist, "annotated events");
annotateEvents = toAnnotate.split("\\\\s*,\\\\s*");
break;
-// case ANNOTATE_TO_GRAFANA:
-// arglist.removeFirst();
-// grafanaEndpoint = readWordOrThrow(arglist,"grafana API endpoint");
-// break;
case ANNOTATORS_CONFIG:
arglist.removeFirst();
this.annotatorsConfig = readWordOrThrow(arglist, "annotators config");
break;
- case INCLUDE:
- arglist.removeFirst();
- String include = readWordOrThrow(arglist, "path to include");
- wantsToIncludePaths.add(include);
- break;
case REPORT_GRAPHITE_TO:
arglist.removeFirst();
reportGraphiteTo = arglist.removeFirst();
@@ -245,13 +283,57 @@ public class NBCLIOptions {
break;
default:
nonincludes.addLast(arglist.removeFirst());
-
}
-
}
+
return nonincludes;
}
+ private Path setStatePath() {
+ if (statePathAccesses.size() > 0) {
+ throw new BasicError("The statedir must be set before it is used by other\n" +
+ " options. If you want to change the statedir, be sure you do it before\n" +
+ " dependent options. These parameters were called before this --statedir:\n" +
+ statePathAccesses.stream().map(s -> "> " + s).collect(Collectors.joining("\n")));
+ }
+ if (this.statepath != null) {
+ return this.statepath;
+ }
+
+ List paths = Environment.INSTANCE.interpolate(":", statedirs);
+ Path selected = null;
+
+ for (String pathName : paths) {
+ Path path = Path.of(pathName);
+ if (Files.exists(path)) {
+ if (Files.isDirectory(path)) {
+ selected = path;
+ break;
+ } else {
+ logger.warn("possible state dir path is not a directory: '" + path.toString() + "'");
+ }
+ }
+ }
+ if (selected == null) {
+ selected = Path.of(paths.get(0));
+ }
+
+ if (!Files.exists(selected)) {
+ try {
+ Files.createDirectories(
+ selected,
+ PosixFilePermissions.asFileAttribute(PosixFilePermissions.fromString("rwxrwx---"))
+ );
+ } catch (IOException e) {
+ throw new BasicError("Could not create state directory at '" + selected.toString() + "': " + e.getMessage());
+ }
+ }
+
+ Environment.INSTANCE.put(NBSTATEDIR, selected.toString());
+
+ return selected;
+ }
+
private void parseAllOptions(String[] args) {
LinkedList arglist = parseGlobalOptions(args);
@@ -565,7 +647,7 @@ public class NBCLIOptions {
spec.indicatorMode = IndicatorMode.logonly;
} else if (this.getCommands().stream().anyMatch(cmd -> cmd.getCmdType().equals(Cmd.CmdType.script))) {
logger.info("Command line includes script calls, so progress data on console is " +
- "suppressed.");
+ "suppressed.");
spec.indicatorMode = IndicatorMode.logonly;
}
}
@@ -577,7 +659,7 @@ public class NBCLIOptions {
configs.stream().map(LoggerConfig::getFilename).forEach(s -> {
if (files.contains(s)) {
logger.warn(s + " is included in " + configName + " more than once. It will only be included " +
- "in the first matching config. Reorder your options if you need to control this.");
+ "in the first matching config. Reorder your options if you need to control this.");
}
files.add(s);
});
@@ -688,8 +770,8 @@ public class NBCLIOptions {
break;
default:
throw new RuntimeException(
- LOG_HISTOGRAMS +
- " options must be in either 'regex:filename:interval' or 'regex:filename' or 'filename' format"
+ LOG_HISTOGRAMS +
+ " options must be in either 'regex:filename:interval' or 'regex:filename' or 'filename' format"
);
}
}
@@ -714,7 +796,7 @@ public class NBCLIOptions {
switch (parts.length) {
case 2:
Unit.msFor(parts[1]).orElseThrow(
- () -> new RuntimeException("Unable to parse progress indicator indicatorSpec '" + parts[1] + "'")
+ () -> new RuntimeException("Unable to parse progress indicator indicatorSpec '" + parts[1] + "'")
);
progressSpec.intervalSpec = parts[1];
case 1:
diff --git a/engine-cli/src/test/java/io/nosqlbench/engine/cli/ArgsFileTest.java b/engine-cli/src/test/java/io/nosqlbench/engine/cli/ArgsFileTest.java
deleted file mode 100644
index 91bb05ecc..000000000
--- a/engine-cli/src/test/java/io/nosqlbench/engine/cli/ArgsFileTest.java
+++ /dev/null
@@ -1,40 +0,0 @@
-package io.nosqlbench.engine.cli;
-
-import org.junit.Test;
-
-import java.util.LinkedList;
-import java.util.List;
-
-import static org.assertj.core.api.Assertions.assertThat;
-
-public class ArgsFileTest {
-
- @Test
- public void testLoadingArgs() {
- LinkedList result;
- ArgsFile argsFile = new ArgsFile();
- result = argsFile.process("--argsfile", "src/test/resources/argsfiles/nonextant.cli");
- assertThat(result).containsExactly();
- result = argsFile.process("--argsfile", "src/test/resources/argsfiles/alphagamma.cli");
- assertThat(result).containsExactly("alpha", "gamma");
- }
-
- @Test(expected = RuntimeException.class)
- public void testLoadingMissingRequiredFails() {
- LinkedList result;
- ArgsFile argsFile = new ArgsFile();
- result = argsFile.process("--argsfile-required", "src/test/resources/argsfiles/nonextant.cli");
- }
-
- @Test
- public void testLoadingInPlace() {
- LinkedList result;
- LinkedList commands = new LinkedList<>(List.of("--abc", "--def", "--argsfile", "src/test/resources" +
- "/argsfiles/alphagamma.cli"));
- ArgsFile argsFile = new ArgsFile().preload("--argsfile-optional", "src/test/resources/argsfiles/alphagamma" +
- ".cli");
- result = argsFile.process(commands);
- assertThat(result).containsExactly("alpha", "gamma", "--abc", "--def", "alpha", "gamma");
- }
-
-}
\ No newline at end of file
diff --git a/engine-cli/src/test/java/io/nosqlbench/engine/cli/NBCLIArgsFileTest.java b/engine-cli/src/test/java/io/nosqlbench/engine/cli/NBCLIArgsFileTest.java
new file mode 100644
index 000000000..9b2b8f55b
--- /dev/null
+++ b/engine-cli/src/test/java/io/nosqlbench/engine/cli/NBCLIArgsFileTest.java
@@ -0,0 +1,117 @@
+package io.nosqlbench.engine.cli;
+
+import org.junit.Test;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.attribute.PosixFilePermissions;
+import java.util.LinkedList;
+import java.util.List;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+public class NBCLIArgsFileTest {
+
+ @Test
+ public void testLoadingArgs() {
+ LinkedList result;
+ NBCLIArgsFile argsFile = new NBCLIArgsFile();
+ result = argsFile.process("--argsfile", "src/test/resources/argsfiles/nonextant.cli");
+ assertThat(result).containsExactly();
+ result = argsFile.process("--argsfile", "src/test/resources/argsfiles/alphagamma.cli");
+ assertThat(result).containsExactly("alpha", "gamma");
+ }
+
+ @Test
+ public void loadParamsWithEnv() {
+ NBCLIArgsFile argsfile = new NBCLIArgsFile();
+ LinkedList result = argsfile.process("--argsfile-required", "src/test/resources/argsfiles/home_env.cli");
+ System.out.println(result);
+ }
+
+ @Test(expected = RuntimeException.class)
+ public void testLoadingMissingRequiredFails() {
+ LinkedList result;
+ NBCLIArgsFile argsFile = new NBCLIArgsFile();
+ result = argsFile.process("--argsfile-required", "src/test/resources/argsfiles/nonextant.cli");
+ }
+
+ @Test
+ public void testLoadingInPlace() {
+ LinkedList result;
+ LinkedList commands = new LinkedList<>(List.of("--abc", "--def"));
+
+ NBCLIArgsFile argsFile = new NBCLIArgsFile().preload("--argsfile-optional", "src/test/resources/argsfiles/alphagamma" +
+ ".cli");
+ result = argsFile.process(commands);
+ assertThat(result).containsExactly("alpha", "gamma", "--abc", "--def");
+ }
+
+ @Test
+ public void testLinesToArgs() {
+ NBCLIArgsFile argsfile = new NBCLIArgsFile().reserved("reservedword");
+ LinkedList args = argsfile.linesToArgs(
+ List.of("--optionname argument", "--optionname2", "--opt3 reservedword")
+ );
+ assertThat(args).containsExactly("--optionname", "argument", "--optionname2", "--opt3", "reservedword");
+ }
+
+ @Test
+ public void testOptionPinning() {
+ Path tempFile = null;
+ try {
+ tempFile = Files.createTempFile(
+ "tmpfile",
+ "cli",
+ PosixFilePermissions.asFileAttribute(PosixFilePermissions.fromString("rw-rw-r--"))
+ );
+
+ // preloading is a way to have global defaults based on
+ // the presence of a default file
+ NBCLIArgsFile argsfile = new NBCLIArgsFile().preload(
+ "--argsfile-optional", tempFile.toAbsolutePath().toString()
+ );
+
+ LinkedList commandline;
+
+ commandline = argsfile.process(
+ "--pin", "--option1",
+ "--pin", "--option1",
+ "--pin", "--option2", "arg2"
+ );
+
+ String filecontents;
+ filecontents = Files.readString(tempFile);
+
+ // logging should indicate no changes
+ commandline = argsfile.process(
+ "--pin", "--option1",
+ "--pin", "--option1",
+ "--pin", "--option2", "arg2"
+ );
+
+ // unpinned options should be discarded
+ commandline = argsfile.process(
+ "--unpin", "--option1",
+ "--option4", "--option5"
+ );
+
+ assertThat(commandline).containsExactly(
+ "--option4", "--option5"
+ );
+
+ assertThat(filecontents).isEqualTo("--option1\n--option2 arg2\n");
+
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ } finally {
+ try {
+ Files.deleteIfExists(tempFile);
+ } catch (Exception ignored) {
+ }
+ }
+ }
+
+
+}
\ No newline at end of file