diff --git a/devdocs/sketches/linearized/linearized.md b/devdocs/sketches/linearized/linearized.md
index 49ea5e705..47198a363 100644
--- a/devdocs/sketches/linearized/linearized.md
+++ b/devdocs/sketches/linearized/linearized.md
@@ -1,114 +1,97 @@
# Linearized Operations
-NOTE: This is a sketch/work in progress and will not be suitable for
-earnest review until this notice is removed.
+NOTE: This is a sketch/work in progress and will not be suitable for earnest review until this
+notice is removed.
-Thanks to Seb and Wei for helping this design along with their discussions
-along the way.
+Thanks to Seb and Wei for helping this design along with their discussions along the way.
See https://github.com/nosqlbench/nosqlbench/issues/136
-Presently, it is possible to stitch together rudimentary chained
-operations, as long as you already know how statement sequences, bindings
-functions, and thread-local state work. This is a significant amount of
-knowledge to expect from a user who simply wants to configure chained
-operations with internal dependencies.
+Presently, it is possible to stitch together rudimentary chained operations, as long as you already
+know how statement sequences, bindings functions, and thread-local state work. This is a significant
+amount of knowledge to expect from a user who simply wants to configure chained operations with
+internal dependencies.
-The design changes needed to make this easy to express are non-trivial and
-cut across a few of the extant runtime systems within nosqlbench. This
-design sketch will try to capture each of the requirements and approached
-sufficiently for discussion and feedback.
+The design changes needed to make this easier are non-trivial and cut across the runtime systems
+within nosqlbench. This design sketch will try to capture each of the requirements and approaches
+for discussion and feedback.
# Sync and Async
## As it is: Sync vs Async
-The current default mode (without `async=`) emulates a request-per-thread
-model, with operations being planned in a deterministic sequence. In this
-mode, each thread dispatches operations from the sequence only after the
-previous one is fully completed, even if there is no dependence between
+The current default mode (without `async=`) emulates a request-per-thread model, with operations
+being planned in a deterministic sequence. In this mode, each thread dispatches operations from the
+sequence only after the previous one is fully completed, even if there is no dependence between
them. This is typical of many applications, even today, but not all.
-On the other end of the spectrum is the fully asynchronous dispatch mode
-enabled with the `async=` option. This uses a completely different
-internal API to allow threads to juggle a number of operations. In
-contrast to the default mode, the async mode dispatches operations eagerly
-as long as the user's selected concurrency level is not yet met. This
-means that operations may overlap and also occur out of order with respect
-to the sequence.
+On the other end of the spectrum is the fully asynchronous dispatch mode enabled with the `async=`
+option. This uses a completely different internal API to allow threads to juggle a number of
+operations. In contrast to the default mode, the async mode dispatches operations eagerly as long as
+the user's selected concurrency level is not yet met. This means that operations may overlap and
+also occur out of order with respect to the sequence.
-Choosing between these modes is a hard choice that does not offer a
-uniform way of looking at operations. As well, it also forces users to
-pick between two extremes of all request-per-thread or all asynchronous,
-which is becoming less common in application designs, and at the very
-least does not rise to the level of expressivity of the toolchains that
-most users have access to.
+Choosing between these modes is a hard choice that does not offer a uniform way of looking at
+operations. As well, it also forces users to pick between two extremes of all request-per-thread or
+all asynchronous, which is becoming less common in application designs, and at the very least does
+not rise to the level of expressivity of the toolchains that most users have access to.
## As it should be: Async with Explicit Dependencies
-* The user should be able to create explicit dependencies from one
- operation to another.
-* Operations which are not dependent on other operations should be
- dispatched as soon as possible within the concurrency limits of the
- workload.
-* Operations with dependencies on other operations should only be
- dispatched if the upstream operations completed successfully.
-* Users should have clear expectations of how error handling will occur
- for individual operations as well as chains of operations.
+* The user should be able to create explicit dependencies from one operation to another.
+* Operations which are not dependent on other operations should be dispatched as soon as possible
+ within the concurrency limits of the workload.
+* Operations with dependencies on other operations should only be dispatched if the upstream
+ operations completed successfully.
+* Users should have clear expectations of how error handling will occur for individual operations as
+ well as chains of operations.
# Dependent Ops
-We are using the phrase _dependent ops_ to capture the notions of
-data-flow dependency between ops (implying linearization in ordering and
-isolation of input and output boundaries), successful execution, and data
-sharing within an appropriate scope.
+We are using the phrase _dependent ops_ to capture the notions of data-flow dependency between ops (
+implying linearization in ordering and isolation of input and output boundaries), successful
+execution, and data sharing within an appropriate scope.
## As it is: Data Flow
-Presently, you can store state within a thread local object map in order
-to share data between operations. This is using the implied scope of "
-thread local" which works well with the "sequence per thread, request per
-thread" model. This works because both the op sequence as well as the
-variable state used in binding functions are thread local.
+Presently, you can store state within a thread local object map in order to share data between
+operations. This is using the implied scope of "
+thread local" which works well with the "sequence per thread, request per thread" model. This works
+because both the op sequence as well as the variable state used in binding functions are thread
+local.
-However, it does not work well with the async mode, since there is no
-implied scope to tie the variable state to the op sequence. There can be
-many operations within a thread operating on the same state even
-concurrently. This may appear to function, but will create problems for
-users who are not aware of the limitation.
+However, it does not work well with the async mode, since there is no implied scope to tie the
+variable state to the op sequence. There can be many operations within a thread operating on the
+same state even concurrently. This may appear to function, but will create problems for users who
+are not aware of the limitation.
## As it should be: Data Flow
-* Data flow between operations should be easily expressed with a standard
- configuration primitive which can work across all driver types.
-* The scope of data shared should be
-
-The scope of a captured value should be clear to users
+* Data flow between operations should be easily expressed with a standard configuration primitive
+ which can work across all driver types.
+* The scope of data shared should be clear to users when configuring op templates, and in any
+ diagnostic outputs from failed operations.
## As it is: Data Capture
-Presently, the CQL driver has additional internal operators which allow
-for the capture of values. These decorator behaviors allow for configured
-statements to do more than just dispatch an operation. However, they are
-not built upon standard data capture and sharing operations which are
-implemented uniformly across driver types. This makes scope management
-largely a matter of convention, which is ok for the first implementation (
+Presently, the CQL driver has additional internal operators which allow for the capture of values.
+These decorator behaviors allow for configured statements to do more than just dispatch an
+operation. However, they are not built upon standard data capture and sharing operations which are
+implemented uniformly across driver types. This makes scope management largely a matter of
+convention, which is ok for the first implementation (
in the CQL driver) but not as a building block for cross-driver behaviors.
# Injecting Operations
## As it is: Injecting Operations
-Presently operations are derived from statement templates on a
-deterministic op sequence which is of a fixed length known as the stride.
-This follows closely the pattern of assuming each operation comes from one
-distinct cycle and that there is always a one-to-one relationship with
-cycles. This has carried some weight internally in how metrics for cycles
-are derived, etc. There is presently no separate operational queue for
-statements except by modifying statements in the existing sequence with
-side-effect binding assignment. It is difficult to reason about additional
-operations as independent without decoupling these two into separate
-mechanisms.
+Presently operations are derived from statement templates on a deterministic op sequence which is of
+a fixed length known as the stride. This follows closely the pattern of assuming each operation
+comes from one distinct cycle and that there is always a one-to-one relationship with cycles. This
+has carried some weight internally in how metrics for cycles are derived, etc. There is presently no
+separate operational queue for statements except by modifying statements in the existing sequence
+with side-effect binding assignment. It is difficult to reason about additional operations as
+independent without decoupling these two into separate mechanisms.
## As it should be: Injecting Operations
@@ -126,8 +109,7 @@ Open concerns
- before: variable state was per-thread
- now: variable state is per opflow
-- (opflow state is back-filled into thread local as the default
- implementation)
+- (opflow state is back-filled into thread local as the default implementation)
* gives scope for enumerating op flows, meaning you opflow 0... opflow (
cycles/stride)
@@ -157,52 +139,46 @@ statements:
## Capture Syntax
-Capturing of variables in statement templates will be signified
-with `[varname]`. This examples represents the simplest case where the
-user just wants to capture a varaible. Thus the above is taken to mean:
+Capturing of variables in statement templates will be signified with `[varname]`. This examples
+represents the simplest case where the user just wants to capture a variable. Thus the above is
+taken to mean:
- The scope of the captured variable is the OpFlow.
-- The operation is required to succeed. Any other operation which depends
- on a `varname` value will be skipped and counted as such.
-- The captured type of `varname` is a single object, to be determined
- dynamically, with no type checking required.
-- A field named `varname` is required to be present in the result set for
- the statement that included it.
+- The operation is required to succeed. Any other operation which depends on a `varname` value will
+ be skipped and counted as such.
+- The captured type of `varname` is a single object, to be determined dynamically, with no type
+ checking required.
+- A field named `varname` is required to be present in the result set for the statement that
+ included it.
- Exactly one value for `varname` is required to be present.
-- Without other settings to relax sanity constraints, any other appearance
- of `[varname]` in another active statement should yield a warning to the
- user.
+- Without other settings to relax sanity constraints, any other appearance of `[varname]` in another
+ active statement should yield a warning to the user.
-All behavioral variations that diverge from the above will be signified
-within the capture syntax as a variation on the above example.
+All behavioral variations that diverge from the above will be signified within the capture syntax as
+a variation on the above example.
## Inject Syntax
-Similar to binding tokens used in statement templates like '{varname}', it
-is possible to inject captured variables into statement templates with
-the `{[varname]}` syntax. This indicates that the user explicitly wants to
-pull a value directly from the captured variable. It is necessary to
-indicate variable capture and variable injection distinctly from each
-other, and this syntax supports that while remaining familiar to the
-bindings formats already supported.
+Similar to binding tokens used in statement templates like '{varname}', it is possible to inject
+captured variables into statement templates with the `{[varname]}` syntax. This indicates that the
+user explicitly wants to pull a value directly from the captured variable. It is necessary to
+indicate variable capture and variable injection distinctly from each other, and this syntax
+supports that while remaining familiar to the bindings formats already supported.
-The above syntax example represents the case where the user simply wants
-to refer to a variable of a given name. This is the simplest case, and is
-taken to mean:
+The above syntax example represents the case where the user simply wants to refer to a variable of a
+given name. This is the simplest case, and is taken to mean:
-- The scope of the variable is not specified. The value may come from
- OpFlow, thread, global or any scope that is available. By default,
- scopes should be consulted with the shortest-lived inner scopes first
- and widened only if needed to find the variable.
-- The variable must be defined in some available scope. By default, It is
- an error to refer to a variable for injection that is not defined.
-- The type of the variable is not checked on access. The type is presumed
- to be compatible with any assignments which are made within whatever
- driver type is in use.
+- The scope of the variable is not specified. The value may come from OpFlow, thread, global or any
+ scope that is available. By default, scopes should be consulted with the shortest-lived inner
+ scopes first and widened only if needed to find the variable.
+- The variable must be defined in some available scope. By default, It is an error to refer to a
+ variable for injection that is not defined.
+- The type of the variable is not checked on access. The type is presumed to be compatible with any
+ assignments which are made within whatever driver type is in use.
- The variable is assumed to be a single-valued type.
-All behavioral variations that diverge from the above will be signified
-within the variable injection syntax as a variation on the above syntax.
+All behavioral variations that diverge from the above will be signified within the variable
+injection syntax as a variation on the above syntax.
## Scenarios to Consider
@@ -215,22 +191,18 @@ advanced scenarios:
- some ops may be required to produce a value
- some ops may be required to produce multiple values
-* The carrier of op state should enable the following programmatic
- constructions:
+* The carrier of op state should enable the following programmatic constructions:
* Metric measuring the service time of the op on failure
* Metric measuring the service time of the op on success
* Metric measuring the size of the op on success
- * Hooks for transforming or acting upon the op or cycle before the op
- executes
- * Hooks for transforming or acting upon the op or cycle after the op
- executes, regardless of result
+ * Hooks for transforming or acting upon the op or cycle before the op executes
+ * Hooks for transforming or acting upon the op or cycle after the op executes, regardless of
+ result
* Additional modifiers on the op, as in transformers.
-* All op contextual actions should be presented as a function on the op
- type
+* All op contextual actions should be presented as a function on the op type
-* Completion Stages that support the op API should come from built-in
- template implementations that already include metrics options, logging
- support, etc.
+* Completion Stages that support the op API should come from built-in template implementations that
+ already include metrics options, logging support, etc.
diff --git a/driver-cql-shaded/src/main/java/io/nosqlbench/activitytype/cql/core/CQLBindHelper.java b/driver-cql-shaded/src/main/java/io/nosqlbench/activitytype/cql/core/CQLBindHelper.java
index c3898dbfb..7ccd7bbaa 100644
--- a/driver-cql-shaded/src/main/java/io/nosqlbench/activitytype/cql/core/CQLBindHelper.java
+++ b/driver-cql-shaded/src/main/java/io/nosqlbench/activitytype/cql/core/CQLBindHelper.java
@@ -129,7 +129,7 @@ public class CQLBindHelper {
}
}
- public static Map parseAndGetSpecificBindings(OpTemplate> opDef, ParsedStmt parsed) {
+ public static Map parseAndGetSpecificBindings(OpTemplate opDef, ParsedStmt parsed) {
String statement = opDef.getStmt();
Set extraBindings = new HashSet<>(opDef.getBindings().keySet());
diff --git a/driver-cql-shaded/src/main/java/io/nosqlbench/activitytype/cql/core/CqlActivity.java b/driver-cql-shaded/src/main/java/io/nosqlbench/activitytype/cql/core/CqlActivity.java
index 8c571a002..f263eacbc 100644
--- a/driver-cql-shaded/src/main/java/io/nosqlbench/activitytype/cql/core/CqlActivity.java
+++ b/driver-cql-shaded/src/main/java/io/nosqlbench/activitytype/cql/core/CqlActivity.java
@@ -70,7 +70,7 @@ public class CqlActivity extends SimpleActivity implements Activity, ActivityDef
private final ExceptionHistoMetrics exceptionHistoMetrics;
private final ActivityDef activityDef;
private final Map namedWriters = new HashMap<>();
- protected List> stmts;
+ protected List stmts;
Timer retryDelayTimer;
Timer pagesTimer;
Histogram skippedTokensHisto;
@@ -180,7 +180,7 @@ public class CqlActivity extends SimpleActivity implements Activity, ActivityDef
Set timerStarts = new HashSet<>();
Set timerStops = new HashSet<>();
- for (OpTemplate> stmtDef : stmts) {
+ for (OpTemplate stmtDef : stmts) {
ParsedStmt parsed = stmtDef.getParsed(CqlActivity::canonicalizeBindings).orError();
boolean prepared = stmtDef.getParamOrDefault("prepared", true);
diff --git a/driver-cql-shaded/src/main/java/io/nosqlbench/activitytype/cql/statements/rowoperators/verification/VerifierBuilder.java b/driver-cql-shaded/src/main/java/io/nosqlbench/activitytype/cql/statements/rowoperators/verification/VerifierBuilder.java
index 9ee30cfe6..bac12f7db 100644
--- a/driver-cql-shaded/src/main/java/io/nosqlbench/activitytype/cql/statements/rowoperators/verification/VerifierBuilder.java
+++ b/driver-cql-shaded/src/main/java/io/nosqlbench/activitytype/cql/statements/rowoperators/verification/VerifierBuilder.java
@@ -8,7 +8,7 @@ import java.util.List;
import java.util.Map;
public class VerifierBuilder {
- public static BindingsTemplate getExpectedValuesTemplate(OpTemplate> stmtDef) {
+ public static BindingsTemplate getExpectedValuesTemplate(OpTemplate stmtDef) {
BindingsTemplate expected = new BindingsTemplate();
diff --git a/driver-cqld3-shaded/src/main/java/io/nosqlbench/activitytype/cql/core/CQLBindHelper.java b/driver-cqld3-shaded/src/main/java/io/nosqlbench/activitytype/cql/core/CQLBindHelper.java
index bcd27b421..06764b2e5 100644
--- a/driver-cqld3-shaded/src/main/java/io/nosqlbench/activitytype/cql/core/CQLBindHelper.java
+++ b/driver-cqld3-shaded/src/main/java/io/nosqlbench/activitytype/cql/core/CQLBindHelper.java
@@ -122,7 +122,7 @@ public class CQLBindHelper {
}
}
- public static Map parseAndGetSpecificBindings(OpDef> opDef, ParsedStmt parsed) {
+ public static Map parseAndGetSpecificBindings(OpDef opDef, ParsedStmt parsed) {
List spans = new ArrayList<>();
String statement = opDef.getStmt();
diff --git a/driver-cqld3-shaded/src/main/java/io/nosqlbench/activitytype/cql/core/CqlActivity.java b/driver-cqld3-shaded/src/main/java/io/nosqlbench/activitytype/cql/core/CqlActivity.java
index 3a46d8499..3b6df62e7 100644
--- a/driver-cqld3-shaded/src/main/java/io/nosqlbench/activitytype/cql/core/CqlActivity.java
+++ b/driver-cqld3-shaded/src/main/java/io/nosqlbench/activitytype/cql/core/CqlActivity.java
@@ -70,7 +70,7 @@ public class CqlActivity extends SimpleActivity implements Activity, ActivityDef
private final ExceptionHistoMetrics exceptionHistoMetrics;
private final ActivityDef activityDef;
private final Map namedWriters = new HashMap<>();
- protected List> stmts;
+ protected List stmts;
Timer retryDelayTimer;
Timer pagesTimer;
private Histogram triesHisto;
@@ -179,7 +179,7 @@ public class CqlActivity extends SimpleActivity implements Activity, ActivityDef
Set timerStarts = new HashSet<>();
Set timerStops = new HashSet<>();
- for (OpTemplate> stmtDef : stmts) {
+ for (OpTemplate stmtDef : stmts) {
ParsedStmt parsed = stmtDef.getParsed(CqlActivity::canonicalizeBindings).orError();
boolean prepared = stmtDef.getParamOrDefault("prepared", true);
diff --git a/driver-cqld3-shaded/src/main/java/io/nosqlbench/activitytype/cql/statements/rowoperators/verification/VerifierBuilder.java b/driver-cqld3-shaded/src/main/java/io/nosqlbench/activitytype/cql/statements/rowoperators/verification/VerifierBuilder.java
index 9ee30cfe6..bac12f7db 100644
--- a/driver-cqld3-shaded/src/main/java/io/nosqlbench/activitytype/cql/statements/rowoperators/verification/VerifierBuilder.java
+++ b/driver-cqld3-shaded/src/main/java/io/nosqlbench/activitytype/cql/statements/rowoperators/verification/VerifierBuilder.java
@@ -8,7 +8,7 @@ import java.util.List;
import java.util.Map;
public class VerifierBuilder {
- public static BindingsTemplate getExpectedValuesTemplate(OpTemplate> stmtDef) {
+ public static BindingsTemplate getExpectedValuesTemplate(OpTemplate stmtDef) {
BindingsTemplate expected = new BindingsTemplate();
diff --git a/driver-dsegraph-shaded/src/main/java/com/datastax/ebdrivers/dsegraph/GraphActivity.java b/driver-dsegraph-shaded/src/main/java/com/datastax/ebdrivers/dsegraph/GraphActivity.java
index 28c14b2e3..2107bddf0 100644
--- a/driver-dsegraph-shaded/src/main/java/com/datastax/ebdrivers/dsegraph/GraphActivity.java
+++ b/driver-dsegraph-shaded/src/main/java/com/datastax/ebdrivers/dsegraph/GraphActivity.java
@@ -46,7 +46,7 @@ public class GraphActivity extends SimpleActivity implements ActivityDefObserver
public Timer resultTimer;
public Timer logicalGraphOps;
public Histogram triesHisto;
- protected List> stmts;
+ protected List stmts;
private int stride;
private DseSession session;
private DseCluster cluster;
@@ -100,7 +100,7 @@ public class GraphActivity extends SimpleActivity implements ActivityDefObserver
throw new RuntimeException("There were no unfiltered statements found for this activity.");
}
- for (OpTemplate> stmtDef : stmts) {
+ for (OpTemplate stmtDef : stmts) {
ParsedStmt parsed = stmtDef.getParsed().orError();
diff --git a/driver-kafka/src/main/java/com/datastax/ebdrivers/kafkaproducer/KafkaProducerActivity.java b/driver-kafka/src/main/java/com/datastax/ebdrivers/kafkaproducer/KafkaProducerActivity.java
index 734733eed..54ec6513b 100644
--- a/driver-kafka/src/main/java/com/datastax/ebdrivers/kafkaproducer/KafkaProducerActivity.java
+++ b/driver-kafka/src/main/java/com/datastax/ebdrivers/kafkaproducer/KafkaProducerActivity.java
@@ -73,12 +73,12 @@ public class KafkaProducerActivity extends SimpleActivity {
String tagFilter = activityDef.getParams().getOptionalString("tags").orElse("");
StmtsDocList stmtsDocList = StatementsLoader.loadPath(logger, yamlLoc, new StrInterpolator(activityDef),
"activities");
- List> statements = stmtsDocList.getStmts(tagFilter);
+ List statements = stmtsDocList.getStmts(tagFilter);
String format = getParams().getOptionalString("format").orElse(null);
if (statements.size() > 0) {
- for (OpTemplate> statement : statements) {
+ for (OpTemplate statement : statements) {
sequencer.addOp(
new KafkaStatement(statement,
servers,
diff --git a/driver-kafka/src/main/java/com/datastax/ebdrivers/kafkaproducer/KafkaStatement.java b/driver-kafka/src/main/java/com/datastax/ebdrivers/kafkaproducer/KafkaStatement.java
index d89a40c0b..819a207b1 100644
--- a/driver-kafka/src/main/java/com/datastax/ebdrivers/kafkaproducer/KafkaStatement.java
+++ b/driver-kafka/src/main/java/com/datastax/ebdrivers/kafkaproducer/KafkaStatement.java
@@ -31,7 +31,7 @@ public class KafkaStatement {
private AvroSchema valueSerializerSchema = null;
private final String key;
- public KafkaStatement(OpTemplate> stmtDef, String servers, String clientId, String schemaRegistryUrl) {
+ public KafkaStatement(OpTemplate stmtDef, String servers, String clientId, String schemaRegistryUrl) {
ParsedTemplate paramTemplate = new ParsedTemplate(stmtDef.getStmt(), stmtDef.getBindings());
BindingsTemplate paramBindings = new BindingsTemplate(paramTemplate.getCheckedBindPoints());
StringBindingsTemplate template = new StringBindingsTemplate(stmtDef.getStmt(), paramBindings);
diff --git a/driver-mongodb/src/main/java/io/nosqlbench/driver/mongodb/MongoActivity.java b/driver-mongodb/src/main/java/io/nosqlbench/driver/mongodb/MongoActivity.java
index 02bc0a56f..bc08e18ca 100644
--- a/driver-mongodb/src/main/java/io/nosqlbench/driver/mongodb/MongoActivity.java
+++ b/driver-mongodb/src/main/java/io/nosqlbench/driver/mongodb/MongoActivity.java
@@ -112,11 +112,11 @@ public class MongoActivity extends SimpleActivity implements ActivityDefObserver
TagFilter tagFilter = new TagFilter(tagfilter);
stmtsDocList.getStmts().stream().map(tagFilter::matchesTaggedResult).forEach(r -> logger.info(r.getLog()));
- List> stmts = stmtsDocList.getStmts(tagfilter);
+ List stmts = stmtsDocList.getStmts(tagfilter);
if (stmts.isEmpty()) {
logger.error("No statements found for this activity");
} else {
- for (OpTemplate> stmt : stmts) {
+ for (OpTemplate stmt : stmts) {
ParsedStmt parsed = stmt.getParsed().orError();
String statement = parsed.getPositionalStatement(Function.identity());
Objects.requireNonNull(statement);
diff --git a/driver-mongodb/src/main/java/io/nosqlbench/driver/mongodb/ReadyMongoStatement.java b/driver-mongodb/src/main/java/io/nosqlbench/driver/mongodb/ReadyMongoStatement.java
index 199e3b3dc..7edf7893c 100644
--- a/driver-mongodb/src/main/java/io/nosqlbench/driver/mongodb/ReadyMongoStatement.java
+++ b/driver-mongodb/src/main/java/io/nosqlbench/driver/mongodb/ReadyMongoStatement.java
@@ -14,7 +14,7 @@ public class ReadyMongoStatement {
private final StringBindings bindings;
private final ReadPreference readPreference;
- public ReadyMongoStatement(OpTemplate> stmtDef) {
+ public ReadyMongoStatement(OpTemplate stmtDef) {
ParsedTemplate paramTemplate = new ParsedTemplate(stmtDef.getStmt(), stmtDef.getBindings());
BindingsTemplate paramBindings = new BindingsTemplate(paramTemplate.getCheckedBindPoints());
StringBindingsTemplate template = new StringBindingsTemplate(stmtDef.getStmt(), paramBindings);
diff --git a/driver-stdout/src/main/java/io/nosqlbench/activitytype/stdout/StdoutActivity.java b/driver-stdout/src/main/java/io/nosqlbench/activitytype/stdout/StdoutActivity.java
index 3527f0421..3f5ed094c 100644
--- a/driver-stdout/src/main/java/io/nosqlbench/activitytype/stdout/StdoutActivity.java
+++ b/driver-stdout/src/main/java/io/nosqlbench/activitytype/stdout/StdoutActivity.java
@@ -134,7 +134,7 @@ public class StdoutActivity extends SimpleActivity implements ActivityDefObserve
SequencePlanner sequencer = new SequencePlanner<>(sequencerType);
String tagfilter = activityDef.getParams().getOptionalString("tags").orElse("");
- List> stmts = stmtsDocList.getStmts(tagfilter);
+ List stmts = stmtsDocList.getStmts(tagfilter);
String format = getParams().getOptionalString("format").orElse(null);
@@ -179,7 +179,7 @@ public class StdoutActivity extends SimpleActivity implements ActivityDefObserve
sequencer.addOp(sb, 1L);
}
} else if (stmts.size() > 0) {
- for (OpTemplate> stmt : stmts) {
+ for (OpTemplate stmt : stmts) {
ParsedStmt parsed = stmt.getParsed().orError();
BindingsTemplate bt = new BindingsTemplate(parsed.getBindPoints());
String statement = parsed.getPositionalStatement(Function.identity());
diff --git a/driver-web/src/main/java/io/nosqlbench/driver/webdriver/WebDriverActivity.java b/driver-web/src/main/java/io/nosqlbench/driver/webdriver/WebDriverActivity.java
index 71dd2548f..489e96566 100644
--- a/driver-web/src/main/java/io/nosqlbench/driver/webdriver/WebDriverActivity.java
+++ b/driver-web/src/main/java/io/nosqlbench/driver/webdriver/WebDriverActivity.java
@@ -180,13 +180,13 @@ public class WebDriverActivity extends SimpleActivity {
SequencePlanner planner = new SequencePlanner<>(sequencerType);
String tagfilter = activityDef.getParams().getOptionalString("tags").orElse("");
- List> stmts = stmtsDocList.getStmts(tagfilter);
+ List stmts = stmtsDocList.getStmts(tagfilter);
if (stmts.size() == 0) {
throw new BasicError("There were no active statements with tag filter '" + tagfilter + "'");
}
- for (OpTemplate> optemplate : stmts) {
+ for (OpTemplate optemplate : stmts) {
long ratio = optemplate.getParamOrDefault("ratio", 1);
CommandTemplate cmd = new CommandTemplate(optemplate);
planner.addOp(cmd, ratio);
diff --git a/engine-api/src/main/java/io/nosqlbench/engine/api/activityconfig/ParsedStmt.java b/engine-api/src/main/java/io/nosqlbench/engine/api/activityconfig/ParsedStmt.java
index 166a2fe0b..43052c048 100644
--- a/engine-api/src/main/java/io/nosqlbench/engine/api/activityconfig/ParsedStmt.java
+++ b/engine-api/src/main/java/io/nosqlbench/engine/api/activityconfig/ParsedStmt.java
@@ -73,18 +73,6 @@ public class ParsedStmt {
return parsed.hasError();
}
- /**
- * The list of binding names returned by this method does not
- * constitute an error. They may be used for
- * for informational purposes in error handlers, for example.
- *
- * @return a set of bindings names which were provided to
- * this parsed statement, but which were not referenced
- * in either {anchor} or ?anchor
form.
- */
- public Set getExtraBindings() {
- return parsed.getExtraBindings();
- }
/**
* Returns a list of binding names which were referenced
@@ -96,18 +84,7 @@ public class ParsedStmt {
* @return A list of binding names which were referenced but not defined*
*/
public Set getMissingBindings() {
- return parsed.getMissingBindings();
- }
-
- /**
- * Return a map of bindings which were referenced in the statement.
- * This is an easy way to get the list of effective bindings for
- * a statement for diagnostic purposes without including a potentially
- * long list of library bindings.
- * @return a bindings map of referenced bindings in the statement
- */
- public Map getSpecificBindings() {
- return parsed.getSpecificBindings();
+ return parsed.getMissing();
}
/**
@@ -170,7 +147,7 @@ public class ParsedStmt {
}
public List getBindPoints() {
- return parsed.getBindPoints();
+ return parsed.getCheckedBindPoints();
}
}
diff --git a/engine-api/src/main/java/io/nosqlbench/engine/api/activityconfig/yaml/OpDef.java b/engine-api/src/main/java/io/nosqlbench/engine/api/activityconfig/yaml/OpDef.java
index 8fe14ee48..b85a0f562 100644
--- a/engine-api/src/main/java/io/nosqlbench/engine/api/activityconfig/yaml/OpDef.java
+++ b/engine-api/src/main/java/io/nosqlbench/engine/api/activityconfig/yaml/OpDef.java
@@ -23,13 +23,10 @@ import io.nosqlbench.engine.api.activityconfig.ParsedStmt;
import io.nosqlbench.engine.api.activityconfig.rawyaml.RawStmtDef;
import io.nosqlbench.nb.api.errors.BasicError;
-import java.util.LinkedHashMap;
-import java.util.Map;
-import java.util.Objects;
-import java.util.Optional;
+import java.util.*;
import java.util.function.Function;
-public class OpDef implements OpTemplate {
+public class OpDef implements OpTemplate {
private static final String FIELD_DESC = "description";
private static final String FIELD_NAME = "name";
@@ -58,8 +55,20 @@ public class OpDef implements OpTemplate {
}
@Override
- public T getOp() {
- return (T) rawStmtDef.getOp();
+ public Map getOp() {
+ Object op = rawStmtDef.getOp();
+ HashMap newmap = new LinkedHashMap<>();
+ if (op instanceof Map) {
+ ((Map,?>)op).forEach((k,v) -> {
+ newmap.put(k.toString(),v);
+ });
+ } else if (op instanceof CharSequence) {
+ newmap.put("stmt",op.toString());
+ } else {
+ throw new BasicError("Unable to coerce a '" + op.getClass().getCanonicalName() + "' into an op template");
+ }
+
+ return newmap;
}
@Override
diff --git a/engine-api/src/main/java/io/nosqlbench/engine/api/activityconfig/yaml/OpTemplate.java b/engine-api/src/main/java/io/nosqlbench/engine/api/activityconfig/yaml/OpTemplate.java
index 9cfaea540..e471c1c28 100644
--- a/engine-api/src/main/java/io/nosqlbench/engine/api/activityconfig/yaml/OpTemplate.java
+++ b/engine-api/src/main/java/io/nosqlbench/engine/api/activityconfig/yaml/OpTemplate.java
@@ -121,11 +121,11 @@ import java.util.function.Function;
* p2: v2
* }
*/
-public interface OpTemplate extends Tagged {
+public interface OpTemplate extends Tagged {
String getName();
- T getOp();
+ Map getOp();
Map getBindings();
diff --git a/engine-api/src/main/java/io/nosqlbench/engine/api/activityconfig/yaml/StmtsDocList.java b/engine-api/src/main/java/io/nosqlbench/engine/api/activityconfig/yaml/StmtsDocList.java
index b69b876eb..f26d9e87e 100644
--- a/engine-api/src/main/java/io/nosqlbench/engine/api/activityconfig/yaml/StmtsDocList.java
+++ b/engine-api/src/main/java/io/nosqlbench/engine/api/activityconfig/yaml/StmtsDocList.java
@@ -44,7 +44,7 @@ public class StmtsDocList implements Iterable {
.collect(Collectors.toList());
}
- public List> getStmts() {
+ public List getStmts() {
return getStmts("");
}
@@ -53,9 +53,9 @@ public class StmtsDocList implements Iterable {
* @return The list of all included statements for all included blocks of in this document,
* including the inherited and overridden values from the this doc and the parent block.
*/
- public List> getStmts(String tagFilterSpec) {
+ public List getStmts(String tagFilterSpec) {
TagFilter ts = new TagFilter(tagFilterSpec);
- List> opTemplates = new ArrayList<>();
+ List opTemplates = new ArrayList<>();
getStmtDocs().stream()
diff --git a/engine-api/src/main/java/io/nosqlbench/engine/api/activityimpl/SimpleActivity.java b/engine-api/src/main/java/io/nosqlbench/engine/api/activityimpl/SimpleActivity.java
index a9bd332d8..030f7c6f4 100644
--- a/engine-api/src/main/java/io/nosqlbench/engine/api/activityimpl/SimpleActivity.java
+++ b/engine-api/src/main/java/io/nosqlbench/engine/api/activityimpl/SimpleActivity.java
@@ -468,10 +468,10 @@ public class SimpleActivity implements Activity, ProgressCapable {
stmtsDocList = StatementsLoader.loadPath(logger, op_yaml_loc.get(), interp, "activities");
}
- List> stmts = stmtsDocList.getStmts(tagfilter);
+ List stmts = stmtsDocList.getStmts(tagfilter);
List ratios = new ArrayList<>(stmts.size());
for (int i = 0; i < stmts.size(); i++) {
- OpTemplate> opTemplate = stmts.get(i);
+ OpTemplate opTemplate = stmts.get(i);
long ratio = opTemplate.removeParamOrDefault("ratio", 1);
ratios.add(ratio);
}
diff --git a/engine-api/src/main/java/io/nosqlbench/engine/api/templating/CommandTemplate.java b/engine-api/src/main/java/io/nosqlbench/engine/api/templating/CommandTemplate.java
index b2a372754..8ad33f6c1 100644
--- a/engine-api/src/main/java/io/nosqlbench/engine/api/templating/CommandTemplate.java
+++ b/engine-api/src/main/java/io/nosqlbench/engine/api/templating/CommandTemplate.java
@@ -48,7 +48,7 @@ public class CommandTemplate {
*
* @param optpl An OpTemplate
*/
- public CommandTemplate(OpTemplate> optpl) {
+ public CommandTemplate(OpTemplate optpl) {
this(optpl.getName(), optpl.getOp().toString(), optpl.getParamsAsValueType(String.class), optpl.getBindings(), List.of());
}
@@ -66,7 +66,7 @@ public class CommandTemplate {
* @param optpl An OpTemplate
* @param parsers A list of parser functions
*/
- public CommandTemplate(OpTemplate> optpl, List>> parsers) {
+ public CommandTemplate(OpTemplate optpl, List>> parsers) {
this(optpl.getName(), optpl.getOp(), optpl.getParamsAsValueType(String.class), optpl.getBindings(), parsers);
}
diff --git a/engine-api/src/main/java/io/nosqlbench/engine/api/templating/ParsedCommand.java b/engine-api/src/main/java/io/nosqlbench/engine/api/templating/ParsedCommand.java
new file mode 100644
index 000000000..64b8187d1
--- /dev/null
+++ b/engine-api/src/main/java/io/nosqlbench/engine/api/templating/ParsedCommand.java
@@ -0,0 +1,123 @@
+package io.nosqlbench.engine.api.templating;
+
+import io.nosqlbench.engine.api.activityconfig.yaml.OpTemplate;
+import io.nosqlbench.nb.api.config.ParamsParser;
+import io.nosqlbench.nb.api.errors.BasicError;
+import io.nosqlbench.virtdata.core.templates.ParsedTemplate;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+
+/**
+ * Parse an OpTemplate into a ParsedCommand
+ */
+public class ParsedCommand {
+
+ private final static Logger logger = LogManager.getLogger(ParsedCommand.class);
+
+ /** the name of this operation **/
+ private final String name;
+
+ /** The fields which are statically assigned **/
+ private final Map statics = new LinkedHashMap<>();
+
+ /**
+ * The fields which are dynamic, and must be realized via functions.
+ * This map contains keys which identify the field names, and values, which may be null or undefined.
+ */
+ private final Map dynamics = new LinkedHashMap<>();
+
+ /**
+ * The names of payload values in the result of the operation which should be saved.
+ * The keys in this map represent the name of the value as it would be found in the native
+ * representation of a result. If the values are defined, then each one represents the name
+ * that the found value should be saved as instead of the original name.
+ */
+ private final Map captures = new LinkedHashMap<>();
+
+ /**
+ * Create a parsed command from an Op template. The op template is simply the normalized view of
+ * op template structure which is uniform regardless of the original format.
+ * @param ot An OpTemplate representing an operation to be performed in a native driver.
+ */
+ ParsedCommand(OpTemplate ot) {
+ this(ot,List.of());
+ }
+
+ ParsedCommand(OpTemplate ot, List>> optionalParsers) {
+ this.name = ot.getName();
+
+ Map cmd = new LinkedHashMap<>();
+
+ if (ot.getOp() instanceof CharSequence) {
+
+ String oneline = ot.getOp().toString();
+ List>> parserlist = new ArrayList<>(optionalParsers);
+ boolean didParse = false;
+ parserlist.add(s -> ParamsParser.parse(s, false));
+
+ for (Function> parser : parserlist) {
+ Map parsed = parser.apply(oneline);
+ if (parsed != null) {
+ logger.debug("parsed request: " + parsed);
+ cmd.putAll(parsed);
+ didParse = true;
+ break;
+ }
+ }
+ } else if (ot.getOp() instanceof Map) {
+ Map,?> map = ot.getOp();
+ for (Map.Entry, ?> entry : map.entrySet()) {
+ cmd.put(entry.getKey().toString(),entry.getValue());
+ }
+ } else {
+ throw new BasicError("op template has op type of " + ot.getOp().getClass().getCanonicalName() + ", which is not supported.");
+ }
+ resolveCmdMap(cmd,ot.getBindings());
+
+// ArrayList>> _parsers = new ArrayList<>(parsers);
+
+ }
+
+ private void resolveCmdMap(Map cmd, Map bindings) {
+ Map resolved = new LinkedHashMap<>();
+ cmd.forEach((k,v) -> {
+ if (v instanceof CharSequence) {
+ ParsedTemplate parsed = new ParsedTemplate(v.toString(), bindings);
+ switch (parsed.getForm()) {
+ case literal:
+ break;
+ case rawbind:
+ case template:
+ this.dynamics.put(k,v.toString());
+ break;
+ }
+ } else if (v instanceof Map) {
+ Map m = (Map) v;
+// ((Map, ?>) v).forEach((k,v) -> {
+//
+// });
+ resolved.put(k,Map.of("type","Map"));
+ } else {
+
+ }
+ });
+ }
+
+ public String getName() {
+ return name;
+ }
+
+ public Map getStatics() {
+ return statics;
+ }
+
+ public Map getDynamics() {
+ return dynamics;
+ }
+}
diff --git a/engine-api/src/main/java/io/nosqlbench/engine/api/templating/ResolvedCommand.java b/engine-api/src/main/java/io/nosqlbench/engine/api/templating/ResolvedCommand.java
new file mode 100644
index 000000000..d06639af8
--- /dev/null
+++ b/engine-api/src/main/java/io/nosqlbench/engine/api/templating/ResolvedCommand.java
@@ -0,0 +1,42 @@
+package io.nosqlbench.engine.api.templating;
+
+import io.nosqlbench.virtdata.core.bindings.Binder;
+
+import java.util.HashMap;
+import java.util.Map;
+
+public class ResolvedCommand {
+
+ private final ParsedCommand command;
+ private final int mapsize;
+ private final Map statics;
+ private final Map> dynamics;
+
+ public ResolvedCommand(ParsedCommand command) {
+ this.command = command;
+ this.statics = command.getStatics();
+ this.dynamics = resolveDynamics(command);
+ this.mapsize = command.getStatics().size() + command.getDynamics().size();
+ }
+
+ private Map> resolveDynamics(ParsedCommand command) {
+ command.getDynamics().forEach((k,v) -> {
+
+ });
+
+ return null;
+ }
+
+ public Map getCommand(long seed) {
+ HashMap map = new HashMap<>(mapsize);
+ map.putAll(statics);
+
+ dynamics.forEach((k, v) -> {
+ map.put(k, v.bind(seed));
+ });
+ return map;
+
+ }
+
+
+}
diff --git a/engine-api/src/main/resources/workload_definition/command_api.md b/engine-api/src/main/resources/workload_definition/command_api.md
index 1f4fa656a..eeccca0ad 100644
--- a/engine-api/src/main/resources/workload_definition/command_api.md
+++ b/engine-api/src/main/resources/workload_definition/command_api.md
@@ -1,5 +1,227 @@
# Command API
+In the workload template examples, we show statements as being formed from a string value. This is a
+specific type of statement form, although it is possible to provide structured op templates as well.
+
+**The Command API is responsible for converting all valid op template forms into a consistent and
+unambiguous model.** Thus, the rules for mapping the various forms to the command model must be
+precise. Those rules are the substance of this specification.
+
+## Op Synthesis
+
+The method of turning an op template, some data generation functions, and some seed values into an
+executable operation is called *Op Synthesis* in NoSQLBench. This is done in incremental stages:
+
+1. During activity initialization, NoSQLBench parses the workload template and op templates
+ contained within. Each active op template (after filtering) is converted to a parsed command.
+2. The NB driver uses the parsed command to guide the construction of an OpDispenser. This is a
+ dispenser of operations that can be executed by the driver's Action implementation.
+3. When it is time to create an actual operation to be executed, unique with its own procedurally
+ generated payload and settings, the OpDispenser is invoked as a LongFunction. The input
+ provided to this function is the cycle number of the operation. This is essentially a seed that
+ determines the content of all the dynamic fields in the operation.
+
+This process is non-trivial in that it is an incremental creational pattern, where the resultant
+object is contextual to some native API. The command API is intended to guide and enable op
+synthesis without tying developers' hands.
+
+## Command Fields
+
+A command structure is intended to provide all the fields needed to fully realize a native
+operation. Some of these fields will be constant, or *static* in the op template, expressed simply
+as strings, numbers, lists or maps. These are parsed from the op template as such and are cached in
+the command structure as statics.
+
+Other fields are only prescribed as recipes. This comes in two parts: 1) The description for how to
+create the value from a binding function, and 2) the binding point within the op template. Suppose
+you have a string-based op template like this:
+
+```yaml
+ops:
+ - op1: select * from users where userid={userid}
+ bindings:
+ userid: ToString();
+```
+
+In this case, there is only one op in the list of ops, having a name `op1` and a string form op
+template of `select * from users where userid={userid}`.
+
+## Parsed Command Structure
+
+Once an op template is parsed into a *parsed command*, it has the state shown in the data structure
+schematic below:
+
+```json5
+{
+ "name": "some-map-name",
+ "statics": {
+ "s1": "v1",
+ "s2": {
+ "f1": "valfoo"
+ }
+ },
+ "dynamics": {
+ "d1": "NumberNameToString()"
+ },
+ "captures": {
+ "resultprop1": "asname1"
+ }
+}
+```
+
+If either an **op** or **stmt** field is provided, then the same structure as above is used:
+
+```json5
+{
+ "name": "some-string-op",
+ "statics": {
+ },
+ "dynamics": {
+ "op": "select username from table where name userid={userid}"
+ },
+ "captures": {
+ }
+}
+```
+
+The parts of a parsed command structure are:
+
+### command name
+
+Each command knows its name, just like an op template does. This can be useful for diagnostics and
+metric naming.
+
+### static fields
+
+The field names which are statically assigned and their values of any type. Since these values are
+not generated per-op, they are kept separate as reference data. Knowing which fields are static and
+which are not makes it possible for developers to optimize op synthesis.
+
+### dynamic fields
+
+Named bindings points within the op template. These values will only be known for a given cycle.
+
+### variable captures
+
+Names of result values to save, and the variable names they are to be saved as. The names represent
+the name as it would be found in the native driver's API, such as the name `userid`
+in `select userid from ...`. In string form statements, users can specify that the userid should be
+saved as the thread-local variable named *userid* simply by tagging it
+like `select [userid] from ...`. They can also specify that this value should be captured under a
+different name with a variation like `select [userid as user_id] from ...`. This is the standard
+variable capture syntax for any string-based statement form.
+
+# Resolved Command Structure
+
+Once an op template has been parsed into a command structure, the runtime has everything it needs to
+know in order to realize a specific set of field values, *given a cycle number*. Within a cycle, the
+cycle number is effectively a seed value that drives the generation of all dynamic data for that
+cycle.
+
+However, this seed value is only known by the runtime once it is time to execute a specific cycle.
+Thus, it is the developer's job to tell the NoSQLBench runtime how to map from the parsed structure
+to a native type of executable operation suitable for execution with that driver.
+
+# Interpretation
+
+A command structure does not necessarily describe a specific low-level operation to be performed by
+a native driver. It *should* do so, but it is up to the user to provide a valid op template
+according to the documented rules of op construction for that driver type. These rules should be
+clearly documented by the driver developer.
+
+Once the command structure is provided, the driver takes over and maps the fields into an executable
+op -- *almost*. In fact, the driver developer defines the ways that a command structure can be
+turned into an executable operation. This is expressed as a *Function* where T is
+the type used in the native driver's API.
+
+How a developer maps a structure like the above to an operations is up to them. The general rule of
+thumb is to use the most obvious and familiar representation of an operation as it would appear to a
+user. If this is CQL or SQL, then recommend use that as the statement form. If it GraphQL, use that.
+In both of these cases, you have access to
+
+## String Form
+
+Basic operations are made from a statement in some type of query language:
+
+```yaml
+ops:
+ - stringform: select [userid] from db.users where user='{username}';
+ bindings:
+ username: NumberNameToString()
+```
+
+## Structured Form
+
+Some operations can't be easily represented by a single statement. Some operations are built from a
+set of fields which describe more about an operation than the basic statement form. These types of
+operations are expressed to NoSQLBench in map or *object* form, where the fields within the op can
+be specified independently.
+
+```yaml
+ops:
+ - structured1:
+ stmt: select * from db.users where user='{username}}';
+ prepared: true
+ consistency_level: LOCAL_QUORUM
+ bindings:
+ username: NumberNameToString();
+ - structured2:
+ cmdtype: "put"
+ namespace: users
+ key: { userkey }
+ body: "User42 was here"
+ bindings:
+ userkey: FixedValue(42)
+```
+
+In the first case, the op named *structured1* is provided as a string value within a map structure.
+The *stmt* field is a reserved word (synonomous with op and operation). When you are reading an op
+from the command API, these will represented in exactly the same way as the stringform example
+above.
+
+In the second case,
+
+In the second, the op named *structured form* is provided as a map. Both of these examples would
+make sense to a user, as they are fairly self-explanatory.
+
+Op templates may specify an op as either a string or a map. No other types are allowed. However,
+there are no restrictions placed on the elements below a map.
+
+The driver developer should not have to parse all the possible structural forms that users can
+provide. There should be one way to access all of these in a consistent and unambiguous API.
+
+## Command Structure
+
+Here is an example data structure which illustrates all the possible elements of a parsed command:
+
+```json5
+{
+ "statics": {
+ "prepared": "true",
+ "consistency_level'"
+ }
+}
+```
+
+Users provide a template form of an operation in each op template. This contains a sketch of what an
+operation might look like, and includes the following optional parts:
+
+- properties of the operation, whether meta (like a statement) or payload content
+- the binding points where generated field values will be injected
+- The names of values to be extracted from the result of a successful operation.
+
+## Statement Forms
+
+Sometimes operations are derived from a query language, and are thus self-contained in a string
+form.
+
+When mapping the template of an operation provided by users to an executable operation in some
+native driver with specific values, you have to know
+
+* The s
+* The substance of the operation: The name and values of the fields that the user provides as part
+ of the operation
+
Command templates are the third layer of workload templating. As described in other spec documents,
the other layers are:
diff --git a/engine-api/src/main/resources/workload_definition/templated_operations.md b/engine-api/src/main/resources/workload_definition/templated_operations.md
index 1637aab76..8768e695a 100644
--- a/engine-api/src/main/resources/workload_definition/templated_operations.md
+++ b/engine-api/src/main/resources/workload_definition/templated_operations.md
@@ -48,7 +48,9 @@ op: select * from bar.table;
[
{
"name": "block0--stmt1",
- "op": "select * from bar.table;"
+ "op": {
+ "stmt": "select * from bar.table;"
+ }
}
]
```
@@ -78,7 +80,9 @@ ops:
[
{
"name": "block0--stmt1",
- "op": "select * from bar.table;"
+ "op": {
+ "stmt": "select * from bar.table;"
+ }
}
]
```
@@ -113,7 +117,9 @@ ops:
[
{
"name": "block0--op1",
- "op": "select * from bar.table;"
+ "op": {
+ "stmt": "select * from bar.table;"
+ }
}
]
```
@@ -147,7 +153,9 @@ ops:
[
{
"name": "block0--op1",
- "op": "select * from bar.table;"
+ "op": {
+ "stmt": "select * from bar.table;"
+ }
}
]
```
@@ -177,7 +185,9 @@ ops:
[
{
"name": "block0--op1",
- "op": "select * from bar.table;"
+ "op": {
+ "stmt": "select * from bar.table;"
+ }
}
]
```
@@ -210,7 +220,9 @@ ops:
[
{
"name": "block0--op1",
- "op": "select * from bar.table;"
+ "op": {
+ "stmt": "select * from bar.table;"
+ }
}
]
```
@@ -275,7 +287,9 @@ ops:
},
"description": "This is just an example operation",
"name": "block0--special-op-name",
- "op": "select * from ks1.tb1;",
+ "op": {
+ "stmt": "select * from ks1.tb1;"
+ },
"params": {
"prepated": false
},
@@ -368,7 +382,9 @@ blocks:
"binding1": "NumberNameToString();"
},
"name": "block-named-fred--special-op-name",
- "op": "select * from ks1.tb1;",
+ "op": {
+ "stmt": "select * from ks1.tb1;"
+ },
"params": {
"prepared": false
},
diff --git a/engine-api/src/main/resources/workload_definition/templated_workloads.md b/engine-api/src/main/resources/workload_definition/templated_workloads.md
index 72d25688c..2db754bc1 100644
--- a/engine-api/src/main/resources/workload_definition/templated_workloads.md
+++ b/engine-api/src/main/resources/workload_definition/templated_workloads.md
@@ -388,11 +388,15 @@ blocks:
[
{
"name": "namedblock1--op1",
- "op": "select * from bar.table;"
+ "op": {
+ "stmt": "select * from bar.table;"
+ }
},
{
"name": "namedblock1--op2",
- "op": "insert into bar.table (a,b,c) values (1,2,3);",
+ "op": {
+ "stmt": "insert into bar.table (a,b,c) values (1,2,3);"
+ },
"params": {
"type": "batch"
}
@@ -449,18 +453,24 @@ blocks:
[
{
"name": "block1--op1",
- "op": "select * from bar.table;"
+ "op": {
+ "stmt": "select * from bar.table;"
+ }
},
{
"name": "block1--op2",
- "op": "insert into bar.table (a,b,c) values (1,2,3);",
+ "op": {
+ "stmt": "insert into bar.table (a,b,c) values (1,2,3);"
+ },
"params": {
"type": "batch"
}
},
{
"name": "this-is-block-2--op3",
- "op": "select * from foo.table;"
+ "op": {
+ "stmt": "select * from foo.table;"
+ }
}
]
```
@@ -501,7 +511,9 @@ blocks:
[
{
"name": "myblock--stmt1",
- "op": "test op"
+ "op": {
+ "stmt": "test op"
+ }
}
]
```
diff --git a/engine-api/src/test/java/io/nosqlbench/engine/api/activityconfig/rawyaml/UniformWorkloadSpecificationTest.java b/engine-api/src/test/java/io/nosqlbench/engine/api/activityconfig/rawyaml/UniformWorkloadSpecificationTest.java
index 5f8f9ccf1..09de1b950 100644
--- a/engine-api/src/test/java/io/nosqlbench/engine/api/activityconfig/rawyaml/UniformWorkloadSpecificationTest.java
+++ b/engine-api/src/test/java/io/nosqlbench/engine/api/activityconfig/rawyaml/UniformWorkloadSpecificationTest.java
@@ -83,6 +83,15 @@ public class UniformWorkloadSpecificationTest {
);
}
+ @Test
+ public void testCommandAPI() {
+ testSpecPath(
+ NBIO.fs().prefix("target/classes/workload_definition/")
+ .name("command_api.md")
+ .one().asPath()
+ );
+ }
+
public String summarize(Node node) {
StringBuilder sb = new StringBuilder();
@@ -293,7 +302,12 @@ public class UniformWorkloadSpecificationTest {
System.out.println("OK");
} catch (Exception e) {
- System.out.println("ERROR");
+// System.out.println("Error while validating equivalence between the yaml and the rendered op context:");
+// System.out.println("yaml:");
+// System.out.println(yaml);
+// System.out.println("ops:");
+// System.out.println(json);
+ throw new RuntimeException(e);
}
diff --git a/engine-api/src/test/java/io/nosqlbench/engine/api/activityconfig/yaml/ParsedStmtTest.java b/engine-api/src/test/java/io/nosqlbench/engine/api/activityconfig/yaml/ParsedStmtTest.java
index 1a294115f..2b2b49489 100644
--- a/engine-api/src/test/java/io/nosqlbench/engine/api/activityconfig/yaml/ParsedStmtTest.java
+++ b/engine-api/src/test/java/io/nosqlbench/engine/api/activityconfig/yaml/ParsedStmtTest.java
@@ -40,14 +40,12 @@ public class ParsedStmtTest {
StmtsBlock block0 = doclist.getStmtDocs().get(0).getBlocks().get(0);
OpTemplate stmtDef0 = block0.getOps().get(0);
ParsedStmt parsed0 = stmtDef0.getParsed();
- assertThat(parsed0.getExtraBindings()).containsExactly("alpha","gamma");
assertThat(parsed0.getMissingBindings()).containsExactly("delta");
assertThat(parsed0.hasError()).isTrue();
StmtsBlock block1 = doclist.getStmtDocs().get(0).getBlocks().get(1);
OpTemplate stmtDef1 = block1.getOps().get(0);
ParsedStmt parsed1 = stmtDef1.getParsed();
- assertThat(parsed1.getExtraBindings()).containsExactly();
assertThat(parsed1.getMissingBindings()).containsExactly();
assertThat(parsed1.hasError()).isFalse();
}
@@ -63,7 +61,7 @@ public class ParsedStmtTest {
OpTemplate stmtDef1 = block2.getOps().get(1);
ParsedStmt parsed1 = stmtDef1.getParsed();
- assertThat(parsed1.getMissingBindings().isEmpty());
+ assertThat(parsed1.getMissingBindings()).isEmpty();
assertThat(parsed1.hasError()).isFalse();
}
diff --git a/nb-api/src/main/java/io/nosqlbench/nb/api/config/ParamsParser.java b/nb-api/src/main/java/io/nosqlbench/nb/api/config/ParamsParser.java
index 9a71b78d6..0245e3a78 100644
--- a/nb-api/src/main/java/io/nosqlbench/nb/api/config/ParamsParser.java
+++ b/nb-api/src/main/java/io/nosqlbench/nb/api/config/ParamsParser.java
@@ -41,8 +41,8 @@ import java.util.Map;
* These occur as {@code name=value}, and may be delimited by spaces or semicolons, like
* {@code name1=value1 name2=value2} or {@code name1=value1;name2=value2}.
* Values can even contain spaces, even though space is a possible delimiter.
- * Thus, {@code name1= value foo bar
- * name2=baz} works, with the values {@code value foo bar} and {@code baz}.
+ * Thus, {@code name1= value foo bar
+ * name2=baz} works, with the values {@code value foo bar} and {@code baz}.
*
*
* Names
@@ -237,7 +237,7 @@ public class ParamsParser {
s = ParseState.expectingName;
break;
case readingName:
- parms.put(lastVarname,parms.get(lastVarname)+' '+varname.toString());
+ parms.put(lastVarname,parms.get(lastVarname)+' '+ varname);
varname.setLength(0);
break;
default:
diff --git a/virtdata-api/src/main/java/io/nosqlbench/virtdata/core/templates/ParsedTemplate.java b/virtdata-api/src/main/java/io/nosqlbench/virtdata/core/templates/ParsedTemplate.java
index 3114fc815..fd46105cb 100644
--- a/virtdata-api/src/main/java/io/nosqlbench/virtdata/core/templates/ParsedTemplate.java
+++ b/virtdata-api/src/main/java/io/nosqlbench/virtdata/core/templates/ParsedTemplate.java
@@ -65,6 +65,12 @@ import java.util.stream.StreamSupport;
*/
public class ParsedTemplate {
+ public enum Form {
+ literal,
+ rawbind,
+ template,
+ }
+
/**
* The canonical template pattern follows the pattern of an opening curly brace,
* followed by a word character, followed by any contiguous
@@ -78,24 +84,29 @@ public class ParsedTemplate {
*
*/
+
public final static Pattern STANDARD_ANCHOR = Pattern.compile("\\{(?\\w+[-_\\d\\w.]*)}");
public final static Pattern EXTENDED_ANCHOR = Pattern.compile("\\{\\{(?.*?)}}");
-
private final static Logger logger = LogManager.getLogger(ParsedTemplate.class);
- private final Pattern[] patterns;
- // Spans is an even-odd form of (literal, variable, ..., ..., literal)
- private final String rawtemplate;
- private final String[] spans;
- private final Set missingBindings = new HashSet<>();
- private final Set extraBindings = new HashSet<>();
+// private final Pattern[] patterns;
+
+ /**
+ * Spans is an even-odd form of (literal, variable, ..., ..., literal)
+ * Thus a 1-length span is a single literal
+ **/
+ private final String[] spans;
+// private final String rawtemplate;
+
+ /**
+ * A map of binding names and recipes (or null)
+ */
private final Map bindings = new LinkedHashMap<>();
- private final Map specificBindings = new LinkedHashMap<>();
/**
* Construct a new ParsedTemplate from the provided statement template.
*
- * @param rawtemplate The string that contains literal sections and anchor sections interspersed
+ * @param rawtemplate The string that contains literal sections and anchor sections interspersed
* @param providedBindings The bindings that are provided for the template to be parsed
*/
public ParsedTemplate(String rawtemplate, Map providedBindings) {
@@ -114,20 +125,28 @@ public class ParsedTemplate {
* have a named group with the name 'anchor', as in (?<anchor>...)
*
*
- * @param rawtemplate A string template which contains optionally embedded named anchors
- * @param providedBindings The bindings which are provided by the user to fulfill the named anchors in this raw template
- * @param providedPatterns The patterns which match the named anchor format and extract anchor names from the raw template
+ * @param rawtemplate A string template which contains optionally embedded named anchors
+ * @param availableBindings The bindings which are provided by the user to fulfill the named anchors in this raw template
+ * @param parserPatterns The patterns which match the named anchor format and extract anchor names from the raw template
*/
- public ParsedTemplate(String rawtemplate, Map providedBindings, Pattern... providedPatterns) {
- this.rawtemplate = rawtemplate;
- this.bindings.putAll(providedBindings);
- this.patterns = providedPatterns;
- this.spans = parse();
+ public ParsedTemplate(String rawtemplate, Map availableBindings, Pattern... parserPatterns) {
+ this.bindings.putAll(availableBindings);
+ this.spans = parse(rawtemplate, availableBindings, parserPatterns);
+ }
+
+ public Form getForm() {
+ if (this.spans.length == 1) {
+ return Form.literal;
+ } else if (this.spans[0].isEmpty() && this.spans[2].isEmpty()) {
+ return Form.rawbind;
+ } else {
+ return Form.template;
+ }
}
public ParsedTemplate orError() {
if (hasError()) {
- throw new RuntimeException("Unable to parse statement: " + this.toString());
+ throw new RuntimeException("Unable to parse statement: " + this);
}
return this;
}
@@ -140,11 +159,9 @@ public class ParsedTemplate {
* specificBindings will contain an ordered map of the binding definitions
*
*/
- private String[] parse() {
+ private String[] parse(String rawtemplate, Map providedBindings, Pattern[] patterns) {
List spans = new ArrayList<>();
- Set usedAnchors = new HashSet<>();
- extraBindings.addAll(bindings.keySet());
String statement = rawtemplate;
int patternsMatched = 0;
@@ -152,8 +169,8 @@ public class ParsedTemplate {
for (Pattern pattern : patterns) {
if (!pattern.toString().contains("?")) {
- throw new InvalidParameterException("The provided pattern '" + pattern.toString() + "' must contain a named group called anchor," +
- "as in '(?...)'");
+ throw new InvalidParameterException("The provided pattern '" + pattern + "' must contain a named group called anchor," +
+ "as in '(?...)'");
}
Matcher m = pattern.matcher(rawtemplate);
@@ -171,14 +188,7 @@ public class ParsedTemplate {
spans.add(tokenName);
- if (extraBindings.contains(tokenName)) {
- usedAnchors.add(tokenName);
- specificBindings.put(tokenName, bindings.get(tokenName));
- } else {
- missingBindings.add(tokenName);
- }
}
- usedAnchors.forEach(extraBindings::remove);
break; // If the last matcher worked at all, only do one cycle
}
@@ -189,42 +199,25 @@ public class ParsedTemplate {
spans.add(statement);
}
+
return spans.toArray(new String[0]);
}
public String toString() {
- StringBuilder sb = new StringBuilder();
- sb.append("template: '").append(rawtemplate).append("'");
- sb.append("\n parsed: ");
- sb.append(StreamSupport.stream(Arrays.spliterator(spans), false)
- .map(s -> "[" + s + "]").collect(Collectors.joining(",")));
- sb.append("\n missing bindings: ")
- .append(missingBindings.stream().collect(Collectors.joining(",", "[", "]")));
- sb.append(" extra bindings: ");
- sb.append("\n extra bindings: ")
- .append(extraBindings.stream().collect(Collectors.joining(",", "[", "]")));
- return sb.toString();
+ String sb = "\n parsed: " +
+ StreamSupport.stream(Arrays.spliterator(spans), false)
+ .map(s -> "[" + s + "]").collect(Collectors.joining(",")) +
+ "\n missing bindings: " +
+ getMissing().stream().collect(Collectors.joining(",", "[", "]"));
+ return sb;
}
/**
* @return true if the parsed statement is not usable.
*/
public boolean hasError() {
- return missingBindings.size() > 0;
- }
-
- /**
- * The list of binding names returned by this method does not
- * constitute an error. They may be used for
- * for informational purposes in error handlers, for example.
- *
- * @return a set of bindings names which were provided to
- * this parsed statement, but which were not referenced
- * in either {anchor} or ?anchor
form.
- */
- public Set getExtraBindings() {
- return extraBindings;
+ return getMissing().size() > 0;
}
/**
@@ -236,22 +229,19 @@ public class ParsedTemplate {
*
* @return A list of binding names which were referenced but not defined*
*/
- public Set getMissingBindings() {
- return missingBindings;
- }
+ public Set getMissing() {
+ if (spans.length == 1) {
+ return Set.of();
+ }
- /**
- * Return a map of bindings which were referenced in the statement.
- * This is an easy way to get the list of effective bindings for
- * a statement for diagnostic purposes without including a potentially
- * long list of library bindings. This method does not
- * represent all of the binding points, as when anchor names are
- * used more than once.
- *
- * @return a bindings map of referenced bindings in the statement
- */
- public Map getSpecificBindings() {
- return specificBindings;
+ HashSet missing = new HashSet<>();
+ for (int i = 1; i < spans.length; i += 2) {
+ if (!bindings.containsKey(spans[i])) {
+ missing.add(spans[i]);
+ }
+ }
+
+ return missing;
}
/**
@@ -275,11 +265,11 @@ public class ParsedTemplate {
* @throws InvalidParameterException if the template has an error,
* such as an anchor which has no provided binding.
*/
- public List getBindPoints() {
+ public List getCheckedBindPoints() {
List bindpoints = new ArrayList<>();
for (int i = 1; i < spans.length; i += 2) {
if (!bindings.containsKey(spans[i])) {
- throw new InvalidParameterException("Binding named '" + spans[i] + "' is not provided for template '" + rawtemplate + "'");
+ throw new InvalidParameterException("Binding named '" + spans[i] + "' is not contained in the bindings map.");
}
bindpoints.add(new BindPoint(spans[i], bindings.get(spans[i])));
}
@@ -287,6 +277,16 @@ public class ParsedTemplate {
return bindpoints;
}
+ public List getUncheckedBindPoints() {
+ List bindpoints = new ArrayList<>();
+ for (int i = 1; i < spans.length; i += 2) {
+ bindpoints.add(new BindPoint(spans[i], bindings.getOrDefault(spans[i], null)));
+ }
+
+ return bindpoints;
+
+ }
+
/**
* Return the statement that can be used as-is by any driver specific version.
* This uses the anchor token as provided to yield a version of the statement
@@ -299,7 +299,7 @@ public class ParsedTemplate {
StringBuilder sb = new StringBuilder(spans[0]);
for (int i = 1; i < spans.length; i += 2) {
- sb.append(tokenFormatter!=null ? tokenFormatter.apply(spans[i]) : spans[i]);
+ sb.append(tokenFormatter != null ? tokenFormatter.apply(spans[i]) : spans[i]);
sb.append(spans[i + 1]);
}
return sb.toString();
@@ -315,4 +315,18 @@ public class ParsedTemplate {
}
+ /**
+ * Returns the parsed template as a single binding spec if and only if the pattern matches
+ * a single binding anchor with no prefix or suffix.
+ *
+ * @return A single binding spec if that is all that was specified.
+ */
+ public Optional asBinding() {
+ if (spans.length == 3 && spans[0].isEmpty() && spans[2].isEmpty()) {
+ return Optional.of(new BindPoint(spans[1], bindings.getOrDefault(spans[1], null)));
+ } else {
+ return Optional.empty();
+ }
+ }
+
}
diff --git a/virtdata-api/src/test/java/io/nosqlbench/virtdata/core/templates/ParsedTemplateTest.java b/virtdata-api/src/test/java/io/nosqlbench/virtdata/core/templates/ParsedTemplateTest.java
new file mode 100644
index 000000000..61c5e7690
--- /dev/null
+++ b/virtdata-api/src/test/java/io/nosqlbench/virtdata/core/templates/ParsedTemplateTest.java
@@ -0,0 +1,50 @@
+package io.nosqlbench.virtdata.core.templates;
+
+import com.google.gson.Gson;
+import com.google.gson.GsonBuilder;
+import org.junit.Test;
+
+import java.util.Map;
+import java.util.Optional;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+public class ParsedTemplateTest {
+
+ @Test
+ public void testParsedTemplate() {
+ ParsedTemplate pt = new ParsedTemplate("test template", Map.of());
+ assertThat(pt.getAnchors()).isEmpty();
+ assertThat(pt.getCheckedBindPoints()).isEmpty();
+ assertThat(pt.getSpans()).contains("test template");
+ assertThat(pt.getMissing()).isEmpty();
+ }
+
+ @Test
+ public void testBindPoints() {
+ ParsedTemplate pt = new ParsedTemplate("test template {missing1}", Map.of("b1","v1"));
+ assertThat(pt.getSpans()).contains("test template ");
+ assertThat(pt.getAnchors()).containsExactly("missing1");
+ assertThat(pt.getUncheckedBindPoints()).containsExactly(new BindPoint("missing1",null));
+ }
+
+ @Test
+ public void testSingleBinding() {
+ ParsedTemplate pt = new ParsedTemplate("{single}", Map.of());
+ Optional sb = pt.asBinding();
+ assertThat(sb).isPresent();
+ assertThat(sb).contains(new BindPoint("single",null));
+ }
+
+ @Test
+ public void testJsonFormat() {
+ ParsedTemplate pt = new ParsedTemplate("test template {missing1}", Map.of("b1","v1"));
+ Gson gson = new GsonBuilder().setPrettyPrinting().create();
+ String format = gson.toJson(pt);
+ System.out.println(format);
+
+
+ }
+
+
+}