IntenseFinance/nosqlbench
3
0
Fork 0
mirror of https://github.com/nosqlbench/nosqlbench.git synced 2025-02-25 18:55:28 -06:00
Code Issues Packages Projects Releases Wiki Activity

make op templates map based internally

Browse Source
This commit is contained in:
Jonathan Shook
2021-06-17 09:51:45 -05:00
parent 070086e655
commit beea2fdbf6
30 changed files with 726 additions and 277 deletions
Download Patch File Download Diff File Expand all files Collapse all files

216
devdocs/sketches/linearized/linearized.md
View File

@@ -1,114 +1,97 @@
# Linearized Operations
NOTE: This is a sketch/work in progress and will not be suitable for
earnest review until this notice is removed.
NOTE: This is a sketch/work in progress and will not be suitable for earnest review until this
notice is removed.
Thanks to Seb and Wei for helping this design along with their discussions
along the way.
Thanks to Seb and Wei for helping this design along with their discussions along the way.
See https://github.com/nosqlbench/nosqlbench/issues/136
Presently, it is possible to stitch together rudimentary chained
operations, as long as you already know how statement sequences, bindings
functions, and thread-local state work. This is a significant amount of
knowledge to expect from a user who simply wants to configure chained
operations with internal dependencies.
Presently, it is possible to stitch together rudimentary chained operations, as long as you already
know how statement sequences, bindings functions, and thread-local state work. This is a significant
amount of knowledge to expect from a user who simply wants to configure chained operations with
internal dependencies.
The design changes needed to make this easy to express are non-trivial and
cut across a few of the extant runtime systems within nosqlbench. This
design sketch will try to capture each of the requirements and approached
sufficiently for discussion and feedback.
The design changes needed to make this easier are non-trivial and cut across the runtime systems
within nosqlbench. This design sketch will try to capture each of the requirements and approaches
for discussion and feedback.
# Sync and Async
## As it is: Sync vs Async
The current default mode (without `async=`) emulates a request-per-thread
model, with operations being planned in a deterministic sequence. In this
mode, each thread dispatches operations from the sequence only after the
previous one is fully completed, even if there is no dependence between
The current default mode (without `async=`) emulates a request-per-thread model, with operations
being planned in a deterministic sequence. In this mode, each thread dispatches operations from the
sequence only after the previous one is fully completed, even if there is no dependence between
them. This is typical of many applications, even today, but not all.
On the other end of the spectrum is the fully asynchronous dispatch mode
enabled with the `async=` option. This uses a completely different
internal API to allow threads to juggle a number of operations. In
contrast to the default mode, the async mode dispatches operations eagerly
as long as the user's selected concurrency level is not yet met. This
means that operations may overlap and also occur out of order with respect
to the sequence.
On the other end of the spectrum is the fully asynchronous dispatch mode enabled with the `async=`
option. This uses a completely different internal API to allow threads to juggle a number of
operations. In contrast to the default mode, the async mode dispatches operations eagerly as long as
the user's selected concurrency level is not yet met. This means that operations may overlap and
also occur out of order with respect to the sequence.
Choosing between these modes is a hard choice that does not offer a
uniform way of looking at operations. As well, it also forces users to
pick between two extremes of all request-per-thread or all asynchronous,
which is becoming less common in application designs, and at the very
least does not rise to the level of expressivity of the toolchains that
most users have access to.
Choosing between these modes is a hard choice that does not offer a uniform way of looking at
operations. As well, it also forces users to pick between two extremes of all request-per-thread or
all asynchronous, which is becoming less common in application designs, and at the very least does
not rise to the level of expressivity of the toolchains that most users have access to.
## As it should be: Async with Explicit Dependencies
* The user should be able to create explicit dependencies from one
operation to another.
* Operations which are not dependent on other operations should be
dispatched as soon as possible within the concurrency limits of the
workload.
* Operations with dependencies on other operations should only be
dispatched if the upstream operations completed successfully.
* Users should have clear expectations of how error handling will occur
for individual operations as well as chains of operations.
* The user should be able to create explicit dependencies from one operation to another.
* Operations which are not dependent on other operations should be dispatched as soon as possible
within the concurrency limits of the workload.
* Operations with dependencies on other operations should only be dispatched if the upstream
operations completed successfully.
* Users should have clear expectations of how error handling will occur for individual operations as
well as chains of operations.
# Dependent Ops
We are using the phrase _dependent ops_ to capture the notions of
data-flow dependency between ops (implying linearization in ordering and
isolation of input and output boundaries), successful execution, and data
sharing within an appropriate scope.
We are using the phrase _dependent ops_ to capture the notions of data-flow dependency between ops (
implying linearization in ordering and isolation of input and output boundaries), successful
execution, and data sharing within an appropriate scope.
## As it is: Data Flow
Presently, you can store state within a thread local object map in order
to share data between operations. This is using the implied scope of "
thread local" which works well with the "sequence per thread, request per
thread" model. This works because both the op sequence as well as the
variable state used in binding functions are thread local.
Presently, you can store state within a thread local object map in order to share data between
operations. This is using the implied scope of "
thread local" which works well with the "sequence per thread, request per thread" model. This works
because both the op sequence as well as the variable state used in binding functions are thread
local.
However, it does not work well with the async mode, since there is no
implied scope to tie the variable state to the op sequence. There can be
many operations within a thread operating on the same state even
concurrently. This may appear to function, but will create problems for
users who are not aware of the limitation.
However, it does not work well with the async mode, since there is no implied scope to tie the
variable state to the op sequence. There can be many operations within a thread operating on the
same state even concurrently. This may appear to function, but will create problems for users who
are not aware of the limitation.
## As it should be: Data Flow
* Data flow between operations should be easily expressed with a standard
configuration primitive which can work across all driver types.
* The scope of data shared should be
The scope of a captured value should be clear to users
* Data flow between operations should be easily expressed with a standard configuration primitive
which can work across all driver types.
* The scope of data shared should be clear to users when configuring op templates, and in any
diagnostic outputs from failed operations.
## As it is: Data Capture
Presently, the CQL driver has additional internal operators which allow
for the capture of values. These decorator behaviors allow for configured
statements to do more than just dispatch an operation. However, they are
not built upon standard data capture and sharing operations which are
implemented uniformly across driver types. This makes scope management
largely a matter of convention, which is ok for the first implementation (
Presently, the CQL driver has additional internal operators which allow for the capture of values.
These decorator behaviors allow for configured statements to do more than just dispatch an
operation. However, they are not built upon standard data capture and sharing operations which are
implemented uniformly across driver types. This makes scope management largely a matter of
convention, which is ok for the first implementation (
in the CQL driver) but not as a building block for cross-driver behaviors.
# Injecting Operations
## As it is: Injecting Operations
Presently operations are derived from statement templates on a
deterministic op sequence which is of a fixed length known as the stride.
This follows closely the pattern of assuming each operation comes from one
distinct cycle and that there is always a one-to-one relationship with
cycles. This has carried some weight internally in how metrics for cycles
are derived, etc. There is presently no separate operational queue for
statements except by modifying statements in the existing sequence with
side-effect binding assignment. It is difficult to reason about additional
operations as independent without decoupling these two into separate
mechanisms.
Presently operations are derived from statement templates on a deterministic op sequence which is of
a fixed length known as the stride. This follows closely the pattern of assuming each operation
comes from one distinct cycle and that there is always a one-to-one relationship with cycles. This
has carried some weight internally in how metrics for cycles are derived, etc. There is presently no
separate operational queue for statements except by modifying statements in the existing sequence
with side-effect binding assignment. It is difficult to reason about additional operations as
independent without decoupling these two into separate mechanisms.
## As it should be: Injecting Operations
@@ -126,8 +109,7 @@ Open concerns
- before: variable state was per-thread
- now: variable state is per opflow
- (opflow state is back-filled into thread local as the default
implementation)
- (opflow state is back-filled into thread local as the default implementation)
* gives scope for enumerating op flows, meaning you opflow 0... opflow (
cycles/stride)
@@ -157,52 +139,46 @@ statements:
## Capture Syntax
Capturing of variables in statement templates will be signified
with `[varname]`. This examples represents the simplest case where the
user just wants to capture a varaible. Thus the above is taken to mean:
Capturing of variables in statement templates will be signified with `[varname]`. This examples
represents the simplest case where the user just wants to capture a variable. Thus the above is
taken to mean:
- The scope of the captured variable is the OpFlow.
- The operation is required to succeed. Any other operation which depends
on a `varname` value will be skipped and counted as such.
- The captured type of `varname` is a single object, to be determined
dynamically, with no type checking required.
- A field named `varname` is required to be present in the result set for
the statement that included it.
- The operation is required to succeed. Any other operation which depends on a `varname` value will
be skipped and counted as such.
- The captured type of `varname` is a single object, to be determined dynamically, with no type
checking required.
- A field named `varname` is required to be present in the result set for the statement that
included it.
- Exactly one value for `varname` is required to be present.
- Without other settings to relax sanity constraints, any other appearance
of `[varname]` in another active statement should yield a warning to the
user.
- Without other settings to relax sanity constraints, any other appearance of `[varname]` in another
active statement should yield a warning to the user.
All behavioral variations that diverge from the above will be signified
within the capture syntax as a variation on the above example.
All behavioral variations that diverge from the above will be signified within the capture syntax as
a variation on the above example.
## Inject Syntax
Similar to binding tokens used in statement templates like '{varname}', it
is possible to inject captured variables into statement templates with
the `{[varname]}` syntax. This indicates that the user explicitly wants to
pull a value directly from the captured variable. It is necessary to
indicate variable capture and variable injection distinctly from each
other, and this syntax supports that while remaining familiar to the
bindings formats already supported.
Similar to binding tokens used in statement templates like '{varname}', it is possible to inject
captured variables into statement templates with the `{[varname]}` syntax. This indicates that the
user explicitly wants to pull a value directly from the captured variable. It is necessary to
indicate variable capture and variable injection distinctly from each other, and this syntax
supports that while remaining familiar to the bindings formats already supported.
The above syntax example represents the case where the user simply wants
to refer to a variable of a given name. This is the simplest case, and is
taken to mean:
The above syntax example represents the case where the user simply wants to refer to a variable of a
given name. This is the simplest case, and is taken to mean:
- The scope of the variable is not specified. The value may come from
OpFlow, thread, global or any scope that is available. By default,
scopes should be consulted with the shortest-lived inner scopes first
and widened only if needed to find the variable.
- The variable must be defined in some available scope. By default, It is
an error to refer to a variable for injection that is not defined.
- The type of the variable is not checked on access. The type is presumed
to be compatible with any assignments which are made within whatever
driver type is in use.
- The scope of the variable is not specified. The value may come from OpFlow, thread, global or any
scope that is available. By default, scopes should be consulted with the shortest-lived inner
scopes first and widened only if needed to find the variable.
- The variable must be defined in some available scope. By default, It is an error to refer to a
variable for injection that is not defined.
- The type of the variable is not checked on access. The type is presumed to be compatible with any
assignments which are made within whatever driver type is in use.
- The variable is assumed to be a single-valued type.
All behavioral variations that diverge from the above will be signified
within the variable injection syntax as a variation on the above syntax.
All behavioral variations that diverge from the above will be signified within the variable
injection syntax as a variation on the above syntax.
## Scenarios to Consider
@@ -215,22 +191,18 @@ advanced scenarios:
- some ops may be required to produce a value
- some ops may be required to produce multiple values
* The carrier of op state should enable the following programmatic
constructions:
* The carrier of op state should enable the following programmatic constructions:
* Metric measuring the service time of the op on failure
* Metric measuring the service time of the op on success
* Metric measuring the size of the op on success
* Hooks for transforming or acting upon the op or cycle before the op
executes
* Hooks for transforming or acting upon the op or cycle after the op
executes, regardless of result
* Hooks for transforming or acting upon the op or cycle before the op executes
* Hooks for transforming or acting upon the op or cycle after the op executes, regardless of
result
* Additional modifiers on the op, as in transformers.
* All op contextual actions should be presented as a function on the op
type
* All op contextual actions should be presented as a function on the op type
* Completion Stages that support the op API should come from built-in
template implementations that already include metrics options, logging
support, etc.
* Completion Stages that support the op API should come from built-in template implementations that
already include metrics options, logging support, etc.

2
driver-cql-shaded/src/main/java/io/nosqlbench/activitytype/cql/core/CQLBindHelper.java
View File

@@ -129,7 +129,7 @@ public class CQLBindHelper {
}
}
public static Map<String, String> parseAndGetSpecificBindings(OpTemplate<?> opDef, ParsedStmt parsed) {
public static Map<String, String> parseAndGetSpecificBindings(OpTemplate opDef, ParsedStmt parsed) {
String statement = opDef.getStmt();
Set<String> extraBindings = new HashSet<>(opDef.getBindings().keySet());

4
driver-cql-shaded/src/main/java/io/nosqlbench/activitytype/cql/core/CqlActivity.java
View File

@@ -70,7 +70,7 @@ public class CqlActivity extends SimpleActivity implements Activity, ActivityDef
private final ExceptionHistoMetrics exceptionHistoMetrics;
private final ActivityDef activityDef;
private final Map<String, Writer> namedWriters = new HashMap<>();
protected List<OpTemplate<?>> stmts;
protected List<OpTemplate> stmts;
Timer retryDelayTimer;
Timer pagesTimer;
Histogram skippedTokensHisto;
@@ -180,7 +180,7 @@ public class CqlActivity extends SimpleActivity implements Activity, ActivityDef
Set<String> timerStarts = new HashSet<>();
Set<String> timerStops = new HashSet<>();
for (OpTemplate<?> stmtDef : stmts) {
for (OpTemplate stmtDef : stmts) {
ParsedStmt parsed = stmtDef.getParsed(CqlActivity::canonicalizeBindings).orError();
boolean prepared = stmtDef.getParamOrDefault("prepared", true);

2
driver-cql-shaded/src/main/java/io/nosqlbench/activitytype/cql/statements/rowoperators/verification/VerifierBuilder.java
View File

@@ -8,7 +8,7 @@ import java.util.List;
import java.util.Map;
public class VerifierBuilder {
public static BindingsTemplate getExpectedValuesTemplate(OpTemplate<?> stmtDef) {
public static BindingsTemplate getExpectedValuesTemplate(OpTemplate stmtDef) {
BindingsTemplate expected = new BindingsTemplate();

2
driver-cqld3-shaded/src/main/java/io/nosqlbench/activitytype/cql/core/CQLBindHelper.java
View File

@@ -122,7 +122,7 @@ public class CQLBindHelper {
}
}
public static Map<String, String> parseAndGetSpecificBindings(OpDef<?> opDef, ParsedStmt parsed) {
public static Map<String, String> parseAndGetSpecificBindings(OpDef opDef, ParsedStmt parsed) {
List<String> spans = new ArrayList<>();
String statement = opDef.getStmt();

4
driver-cqld3-shaded/src/main/java/io/nosqlbench/activitytype/cql/core/CqlActivity.java
View File

@@ -70,7 +70,7 @@ public class CqlActivity extends SimpleActivity implements Activity, ActivityDef
private final ExceptionHistoMetrics exceptionHistoMetrics;
private final ActivityDef activityDef;
private final Map<String, Writer> namedWriters = new HashMap<>();
protected List<OpTemplate<?>> stmts;
protected List<OpTemplate> stmts;
Timer retryDelayTimer;
Timer pagesTimer;
private Histogram triesHisto;
@@ -179,7 +179,7 @@ public class CqlActivity extends SimpleActivity implements Activity, ActivityDef
Set<String> timerStarts = new HashSet<>();
Set<String> timerStops = new HashSet<>();
for (OpTemplate<?> stmtDef : stmts) {
for (OpTemplate stmtDef : stmts) {
ParsedStmt parsed = stmtDef.getParsed(CqlActivity::canonicalizeBindings).orError();
boolean prepared = stmtDef.getParamOrDefault("prepared", true);

2
driver-cqld3-shaded/src/main/java/io/nosqlbench/activitytype/cql/statements/rowoperators/verification/VerifierBuilder.java
View File

@@ -8,7 +8,7 @@ import java.util.List;
import java.util.Map;
public class VerifierBuilder {
public static BindingsTemplate getExpectedValuesTemplate(OpTemplate<?> stmtDef) {
public static BindingsTemplate getExpectedValuesTemplate(OpTemplate stmtDef) {
BindingsTemplate expected = new BindingsTemplate();

4
driver-dsegraph-shaded/src/main/java/com/datastax/ebdrivers/dsegraph/GraphActivity.java
View File

@@ -46,7 +46,7 @@ public class GraphActivity extends SimpleActivity implements ActivityDefObserver
public Timer resultTimer;
public Timer logicalGraphOps;
public Histogram triesHisto;
protected List<OpTemplate<?>> stmts;
protected List<OpTemplate> stmts;
private int stride;
private DseSession session;
private DseCluster cluster;
@@ -100,7 +100,7 @@ public class GraphActivity extends SimpleActivity implements ActivityDefObserver
throw new RuntimeException("There were no unfiltered statements found for this activity.");
}
for (OpTemplate<?> stmtDef : stmts) {
for (OpTemplate stmtDef : stmts) {
ParsedStmt parsed = stmtDef.getParsed().orError();

4
driver-kafka/src/main/java/com/datastax/ebdrivers/kafkaproducer/KafkaProducerActivity.java
View File

@@ -73,12 +73,12 @@ public class KafkaProducerActivity extends SimpleActivity {
String tagFilter = activityDef.getParams().getOptionalString("tags").orElse("");
StmtsDocList stmtsDocList = StatementsLoader.loadPath(logger, yamlLoc, new StrInterpolator(activityDef),
"activities");
List<OpTemplate<?>> statements = stmtsDocList.getStmts(tagFilter);
List<OpTemplate> statements = stmtsDocList.getStmts(tagFilter);
String format = getParams().getOptionalString("format").orElse(null);
if (statements.size() > 0) {
for (OpTemplate<?> statement : statements) {
for (OpTemplate statement : statements) {
sequencer.addOp(
new KafkaStatement(statement,
servers,

2
driver-kafka/src/main/java/com/datastax/ebdrivers/kafkaproducer/KafkaStatement.java
View File

@@ -31,7 +31,7 @@ public class KafkaStatement {
private AvroSchema valueSerializerSchema = null;
private final String key;
public KafkaStatement(OpTemplate<?> stmtDef, String servers, String clientId, String schemaRegistryUrl) {
public KafkaStatement(OpTemplate stmtDef, String servers, String clientId, String schemaRegistryUrl) {
ParsedTemplate paramTemplate = new ParsedTemplate(stmtDef.getStmt(), stmtDef.getBindings());
BindingsTemplate paramBindings = new BindingsTemplate(paramTemplate.getCheckedBindPoints());
StringBindingsTemplate template = new StringBindingsTemplate(stmtDef.getStmt(), paramBindings);

4
driver-mongodb/src/main/java/io/nosqlbench/driver/mongodb/MongoActivity.java
View File

@@ -112,11 +112,11 @@ public class MongoActivity extends SimpleActivity implements ActivityDefObserver
TagFilter tagFilter = new TagFilter(tagfilter);
stmtsDocList.getStmts().stream().map(tagFilter::matchesTaggedResult).forEach(r -> logger.info(r.getLog()));
List<OpTemplate<?>> stmts = stmtsDocList.getStmts(tagfilter);
List<OpTemplate> stmts = stmtsDocList.getStmts(tagfilter);
if (stmts.isEmpty()) {
logger.error("No statements found for this activity");
} else {
for (OpTemplate<?> stmt : stmts) {
for (OpTemplate stmt : stmts) {
ParsedStmt parsed = stmt.getParsed().orError();
String statement = parsed.getPositionalStatement(Function.identity());
Objects.requireNonNull(statement);

2
driver-mongodb/src/main/java/io/nosqlbench/driver/mongodb/ReadyMongoStatement.java
View File

@@ -14,7 +14,7 @@ public class ReadyMongoStatement {
private final StringBindings bindings;
private final ReadPreference readPreference;
public ReadyMongoStatement(OpTemplate<?> stmtDef) {
public ReadyMongoStatement(OpTemplate stmtDef) {
ParsedTemplate paramTemplate = new ParsedTemplate(stmtDef.getStmt(), stmtDef.getBindings());
BindingsTemplate paramBindings = new BindingsTemplate(paramTemplate.getCheckedBindPoints());
StringBindingsTemplate template = new StringBindingsTemplate(stmtDef.getStmt(), paramBindings);

4
driver-stdout/src/main/java/io/nosqlbench/activitytype/stdout/StdoutActivity.java
View File

@@ -134,7 +134,7 @@ public class StdoutActivity extends SimpleActivity implements ActivityDefObserve
SequencePlanner<StringBindings> sequencer = new SequencePlanner<>(sequencerType);
String tagfilter = activityDef.getParams().getOptionalString("tags").orElse("");
List<OpTemplate<?>> stmts = stmtsDocList.getStmts(tagfilter);
List<OpTemplate> stmts = stmtsDocList.getStmts(tagfilter);
String format = getParams().getOptionalString("format").orElse(null);
@@ -179,7 +179,7 @@ public class StdoutActivity extends SimpleActivity implements ActivityDefObserve
sequencer.addOp(sb, 1L);
}
} else if (stmts.size() > 0) {
for (OpTemplate<?> stmt : stmts) {
for (OpTemplate stmt : stmts) {
ParsedStmt parsed = stmt.getParsed().orError();
BindingsTemplate bt = new BindingsTemplate(parsed.getBindPoints());
String statement = parsed.getPositionalStatement(Function.identity());

4
driver-web/src/main/java/io/nosqlbench/driver/webdriver/WebDriverActivity.java
View File

@@ -180,13 +180,13 @@ public class WebDriverActivity extends SimpleActivity {
SequencePlanner<CommandTemplate> planner = new SequencePlanner<>(sequencerType);
String tagfilter = activityDef.getParams().getOptionalString("tags").orElse("");
List<OpTemplate<?>> stmts = stmtsDocList.getStmts(tagfilter);
List<OpTemplate> stmts = stmtsDocList.getStmts(tagfilter);
if (stmts.size() == 0) {
throw new BasicError("There were no active statements with tag filter '" + tagfilter + "'");
}
for (OpTemplate<?> optemplate : stmts) {
for (OpTemplate optemplate : stmts) {
long ratio = optemplate.getParamOrDefault("ratio", 1);
CommandTemplate cmd = new CommandTemplate(optemplate);
planner.addOp(cmd, ratio);

27
engine-api/src/main/java/io/nosqlbench/engine/api/activityconfig/ParsedStmt.java
View File

@@ -73,18 +73,6 @@ public class ParsedStmt {
return parsed.hasError();
}
/**
* The list of binding names returned by this method does not
* constitute an error. They may be used for
* for informational purposes in error handlers, for example.
*
* @return a set of bindings names which were provided to
* this parsed statement, but which were not referenced
* in either <pre>{anchor}</pre> or <pre>?anchor</pre> form.
*/
public Set<String> getExtraBindings() {
return parsed.getExtraBindings();
}
/**
* Returns a list of binding names which were referenced
@@ -96,18 +84,7 @@ public class ParsedStmt {
* @return A list of binding names which were referenced but not defined*
*/
public Set<String> getMissingBindings() {
return parsed.getMissingBindings();
}
/**
* Return a map of bindings which were referenced in the statement.
* This is an easy way to get the list of effective bindings for
* a statement for diagnostic purposes without including a potentially
* long list of library bindings.
* @return a bindings map of referenced bindings in the statement
*/
public Map<String, String> getSpecificBindings() {
return parsed.getSpecificBindings();
return parsed.getMissing();
}
/**
@@ -170,7 +147,7 @@ public class ParsedStmt {
}
public List<BindPoint> getBindPoints() {
return parsed.getBindPoints();
return parsed.getCheckedBindPoints();
}
}

23
engine-api/src/main/java/io/nosqlbench/engine/api/activityconfig/yaml/OpDef.java
View File

@@ -23,13 +23,10 @@ import io.nosqlbench.engine.api.activityconfig.ParsedStmt;
import io.nosqlbench.engine.api.activityconfig.rawyaml.RawStmtDef;
import io.nosqlbench.nb.api.errors.BasicError;
import java.util.LinkedHashMap;
import java.util.Map;
import java.util.Objects;
import java.util.Optional;
import java.util.*;
import java.util.function.Function;
public class OpDef<T> implements OpTemplate<T> {
public class OpDef implements OpTemplate {
private static final String FIELD_DESC = "description";
private static final String FIELD_NAME = "name";
@@ -58,8 +55,20 @@ public class OpDef<T> implements OpTemplate<T> {
}
@Override
public T getOp() {
return (T) rawStmtDef.getOp();
public Map<String,?> getOp() {
Object op = rawStmtDef.getOp();
HashMap<String, Object> newmap = new LinkedHashMap<>();
if (op instanceof Map) {
((Map<?,?>)op).forEach((k,v) -> {
newmap.put(k.toString(),v);
});
} else if (op instanceof CharSequence) {
newmap.put("stmt",op.toString());
} else {
throw new BasicError("Unable to coerce a '" + op.getClass().getCanonicalName() + "' into an op template");
}
return newmap;
}
@Override

4
engine-api/src/main/java/io/nosqlbench/engine/api/activityconfig/yaml/OpTemplate.java
View File

@@ -121,11 +121,11 @@ import java.util.function.Function;
* p2: v2
* }</pre>
*/
public interface OpTemplate<T> extends Tagged {
public interface OpTemplate extends Tagged {
String getName();
T getOp();
Map<String,?> getOp();
Map<String, String> getBindings();

6
engine-api/src/main/java/io/nosqlbench/engine/api/activityconfig/yaml/StmtsDocList.java
View File

@@ -44,7 +44,7 @@ public class StmtsDocList implements Iterable<StmtsDoc> {
.collect(Collectors.toList());
}
public List<OpTemplate<?>> getStmts() {
public List<OpTemplate> getStmts() {
return getStmts("");
}
@@ -53,9 +53,9 @@ public class StmtsDocList implements Iterable<StmtsDoc> {
* @return The list of all included statements for all included blocks of in this document,
* including the inherited and overridden values from the this doc and the parent block.
*/
public List<OpTemplate<?>> getStmts(String tagFilterSpec) {
public List<OpTemplate> getStmts(String tagFilterSpec) {
TagFilter ts = new TagFilter(tagFilterSpec);
List<OpTemplate<?>> opTemplates = new ArrayList<>();
List<OpTemplate> opTemplates = new ArrayList<>();
getStmtDocs().stream()

4
engine-api/src/main/java/io/nosqlbench/engine/api/activityimpl/SimpleActivity.java
View File

@@ -468,10 +468,10 @@ public class SimpleActivity implements Activity, ProgressCapable {
stmtsDocList = StatementsLoader.loadPath(logger, op_yaml_loc.get(), interp, "activities");
}
List<OpTemplate<?>> stmts = stmtsDocList.getStmts(tagfilter);
List<OpTemplate> stmts = stmtsDocList.getStmts(tagfilter);
List<Long> ratios = new ArrayList<>(stmts.size());
for (int i = 0; i < stmts.size(); i++) {
OpTemplate<?> opTemplate = stmts.get(i);
OpTemplate opTemplate = stmts.get(i);
long ratio = opTemplate.removeParamOrDefault("ratio", 1);
ratios.add(ratio);
}

4
engine-api/src/main/java/io/nosqlbench/engine/api/templating/CommandTemplate.java
View File

@@ -48,7 +48,7 @@ public class CommandTemplate {
*
* @param optpl An OpTemplate
*/
public CommandTemplate(OpTemplate<?> optpl) {
public CommandTemplate(OpTemplate optpl) {
this(optpl.getName(), optpl.getOp().toString(), optpl.getParamsAsValueType(String.class), optpl.getBindings(), List.of());
}
@@ -66,7 +66,7 @@ public class CommandTemplate {
* @param optpl An OpTemplate
* @param parsers A list of parser functions
*/
public CommandTemplate(OpTemplate<?> optpl, List<Function<String, Map<String, String>>> parsers) {
public CommandTemplate(OpTemplate optpl, List<Function<String, Map<String, String>>> parsers) {
this(optpl.getName(), optpl.getOp(), optpl.getParamsAsValueType(String.class), optpl.getBindings(), parsers);
}

123
engine-api/src/main/java/io/nosqlbench/engine/api/templating/ParsedCommand.java Normal file
View File

@@ -0,0 +1,123 @@
package io.nosqlbench.engine.api.templating;
import io.nosqlbench.engine.api.activityconfig.yaml.OpTemplate;
import io.nosqlbench.nb.api.config.ParamsParser;
import io.nosqlbench.nb.api.errors.BasicError;
import io.nosqlbench.virtdata.core.templates.ParsedTemplate;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.function.Function;
/**
* Parse an OpTemplate into a ParsedCommand
*/
public class ParsedCommand {
private final static Logger logger = LogManager.getLogger(ParsedCommand.class);
/** the name of this operation **/
private final String name;
/** The fields which are statically assigned **/
private final Map<String,Object> statics = new LinkedHashMap<>();
/**
* The fields which are dynamic, and must be realized via functions.
* This map contains keys which identify the field names, and values, which may be null or undefined.
*/
private final Map<String,String> dynamics = new LinkedHashMap<>();
/**
* The names of payload values in the result of the operation which should be saved.
* The keys in this map represent the name of the value as it would be found in the native
* representation of a result. If the values are defined, then each one represents the name
* that the found value should be saved as instead of the original name.
*/
private final Map<String,String> captures = new LinkedHashMap<>();
/**
* Create a parsed command from an Op template. The op template is simply the normalized view of
* op template structure which is uniform regardless of the original format.
* @param ot An OpTemplate representing an operation to be performed in a native driver.
*/
ParsedCommand(OpTemplate ot) {
this(ot,List.of());
}
ParsedCommand(OpTemplate ot, List<Function<String, Map<String, String>>> optionalParsers) {
this.name = ot.getName();
Map<String,Object> cmd = new LinkedHashMap<>();
if (ot.getOp() instanceof CharSequence) {
String oneline = ot.getOp().toString();
List<Function<String, Map<String, String>>> parserlist = new ArrayList<>(optionalParsers);
boolean didParse = false;
parserlist.add(s -> ParamsParser.parse(s, false));
for (Function<String, Map<String, String>> parser : parserlist) {
Map<String, String> parsed = parser.apply(oneline);
if (parsed != null) {
logger.debug("parsed request: " + parsed);
cmd.putAll(parsed);
didParse = true;
break;
}
}
} else if (ot.getOp() instanceof Map) {
Map<?,?> map = ot.getOp();
for (Map.Entry<?, ?> entry : map.entrySet()) {
cmd.put(entry.getKey().toString(),entry.getValue());
}
} else {
throw new BasicError("op template has op type of " + ot.getOp().getClass().getCanonicalName() + ", which is not supported.");
}
resolveCmdMap(cmd,ot.getBindings());
// ArrayList<Function<String, Map<String, String>>> _parsers = new ArrayList<>(parsers);
}
private void resolveCmdMap(Map<String, Object> cmd, Map<String, String> bindings) {
Map<String,Object> resolved = new LinkedHashMap<>();
cmd.forEach((k,v) -> {
if (v instanceof CharSequence) {
ParsedTemplate parsed = new ParsedTemplate(v.toString(), bindings);
switch (parsed.getForm()) {
case literal:
break;
case rawbind:
case template:
this.dynamics.put(k,v.toString());
break;
}
} else if (v instanceof Map) {
Map<String,Object> m = (Map<String, Object>) v;
// ((Map<?, ?>) v).forEach((k,v) -> {
//
// });
resolved.put(k,Map.of("type","Map"));
} else {
}
});
}
public String getName() {
return name;
}
public Map<String, Object> getStatics() {
return statics;
}
public Map<String, String> getDynamics() {
return dynamics;
}
}

42
engine-api/src/main/java/io/nosqlbench/engine/api/templating/ResolvedCommand.java Normal file
View File

@@ -0,0 +1,42 @@
package io.nosqlbench.engine.api.templating;
import io.nosqlbench.virtdata.core.bindings.Binder;
import java.util.HashMap;
import java.util.Map;
public class ResolvedCommand<T> {
private final ParsedCommand command;
private final int mapsize;
private final Map<String, Object> statics;
private final Map<String, Binder<?>> dynamics;
public ResolvedCommand(ParsedCommand command) {
this.command = command;
this.statics = command.getStatics();
this.dynamics = resolveDynamics(command);
this.mapsize = command.getStatics().size() + command.getDynamics().size();
}
private Map<String, Binder<?>> resolveDynamics(ParsedCommand command) {
command.getDynamics().forEach((k,v) -> {
});
return null;
}
public Map<String,Object> getCommand(long seed) {
HashMap<String, Object> map = new HashMap<>(mapsize);
map.putAll(statics);
dynamics.forEach((k, v) -> {
map.put(k, v.bind(seed));
});
return map;
}
}

222
engine-api/src/main/resources/workload_definition/command_api.md
View File

@@ -1,5 +1,227 @@
# Command API
In the workload template examples, we show statements as being formed from a string value. This is a
specific type of statement form, although it is possible to provide structured op templates as well.
**The Command API is responsible for converting all valid op template forms into a consistent and
unambiguous model.** Thus, the rules for mapping the various forms to the command model must be
precise. Those rules are the substance of this specification.
## Op Synthesis
The method of turning an op template, some data generation functions, and some seed values into an
executable operation is called *Op Synthesis* in NoSQLBench. This is done in incremental stages:
1. During activity initialization, NoSQLBench parses the workload template and op templates
contained within. Each active op template (after filtering) is converted to a parsed command.
2. The NB driver uses the parsed command to guide the construction of an OpDispenser<T>. This is a
dispenser of operations that can be executed by the driver's Action implementation.
3. When it is time to create an actual operation to be executed, unique with its own procedurally
generated payload and settings, the OpDispenser<T> is invoked as a LongFunction<T>. The input
provided to this function is the cycle number of the operation. This is essentially a seed that
determines the content of all the dynamic fields in the operation.
This process is non-trivial in that it is an incremental creational pattern, where the resultant
object is contextual to some native API. The command API is intended to guide and enable op
synthesis without tying developers' hands.
## Command Fields
A command structure is intended to provide all the fields needed to fully realize a native
operation. Some of these fields will be constant, or *static* in the op template, expressed simply
as strings, numbers, lists or maps. These are parsed from the op template as such and are cached in
the command structure as statics.
Other fields are only prescribed as recipes. This comes in two parts: 1) The description for how to
create the value from a binding function, and 2) the binding point within the op template. Suppose
you have a string-based op template like this:
```yaml
ops:
- op1: select * from users where userid={userid}
bindings:
userid: ToString();
```
In this case, there is only one op in the list of ops, having a name `op1` and a string form op
template of `select * from users where userid={userid}`.
## Parsed Command Structure
Once an op template is parsed into a *parsed command*, it has the state shown in the data structure
schematic below:
```json5
{
"name": "some-map-name",
"statics": {
"s1": "v1",
"s2": {
"f1": "valfoo"
}
},
"dynamics": {
"d1": "NumberNameToString()"
},
"captures": {
"resultprop1": "asname1"
}
}
```
If either an **op** or **stmt** field is provided, then the same structure as above is used:
```json5
{
"name": "some-string-op",
"statics": {
},
"dynamics": {
"op": "select username from table where name userid={userid}"
},
"captures": {
}
}
```
The parts of a parsed command structure are:
### command name
Each command knows its name, just like an op template does. This can be useful for diagnostics and
metric naming.
### static fields
The field names which are statically assigned and their values of any type. Since these values are
not generated per-op, they are kept separate as reference data. Knowing which fields are static and
which are not makes it possible for developers to optimize op synthesis.
### dynamic fields
Named bindings points within the op template. These values will only be known for a given cycle.
### variable captures
Names of result values to save, and the variable names they are to be saved as. The names represent
the name as it would be found in the native driver's API, such as the name `userid`
in `select userid from ...`. In string form statements, users can specify that the userid should be
saved as the thread-local variable named *userid* simply by tagging it
like `select [userid] from ...`. They can also specify that this value should be captured under a
different name with a variation like `select [userid as user_id] from ...`. This is the standard
variable capture syntax for any string-based statement form.
# Resolved Command Structure
Once an op template has been parsed into a command structure, the runtime has everything it needs to
know in order to realize a specific set of field values, *given a cycle number*. Within a cycle, the
cycle number is effectively a seed value that drives the generation of all dynamic data for that
cycle.
However, this seed value is only known by the runtime once it is time to execute a specific cycle.
Thus, it is the developer's job to tell the NoSQLBench runtime how to map from the parsed structure
to a native type of executable operation suitable for execution with that driver.
# Interpretation
A command structure does not necessarily describe a specific low-level operation to be performed by
a native driver. It *should* do so, but it is up to the user to provide a valid op template
according to the documented rules of op construction for that driver type. These rules should be
clearly documented by the driver developer.
Once the command structure is provided, the driver takes over and maps the fields into an executable
op -- *almost*. In fact, the driver developer defines the ways that a command structure can be
turned into an executable operation. This is expressed as a *Function<CommandTemplate,T>* where T is
the type used in the native driver's API.
How a developer maps a structure like the above to an operations is up to them. The general rule of
thumb is to use the most obvious and familiar representation of an operation as it would appear to a
user. If this is CQL or SQL, then recommend use that as the statement form. If it GraphQL, use that.
In both of these cases, you have access to
## String Form
Basic operations are made from a statement in some type of query language:
```yaml
ops:
- stringform: select [userid] from db.users where user='{username}';
bindings:
username: NumberNameToString()
```
## Structured Form
Some operations can't be easily represented by a single statement. Some operations are built from a
set of fields which describe more about an operation than the basic statement form. These types of
operations are expressed to NoSQLBench in map or *object* form, where the fields within the op can
be specified independently.
```yaml
ops:
- structured1:
stmt: select * from db.users where user='{username}}';
prepared: true
consistency_level: LOCAL_QUORUM
bindings:
username: NumberNameToString();
- structured2:
cmdtype: "put"
namespace: users
key: { userkey }
body: "User42 was here"
bindings:
userkey: FixedValue(42)
```
In the first case, the op named *structured1* is provided as a string value within a map structure.
The *stmt* field is a reserved word (synonomous with op and operation). When you are reading an op
from the command API, these will represented in exactly the same way as the stringform example
above.
In the second case,
In the second, the op named *structured form* is provided as a map. Both of these examples would
make sense to a user, as they are fairly self-explanatory.
Op templates may specify an op as either a string or a map. No other types are allowed. However,
there are no restrictions placed on the elements below a map.
The driver developer should not have to parse all the possible structural forms that users can
provide. There should be one way to access all of these in a consistent and unambiguous API.
## Command Structure
Here is an example data structure which illustrates all the possible elements of a parsed command:
```json5
{
"statics": {
"prepared": "true",
"consistency_level'"
}
}
```
Users provide a template form of an operation in each op template. This contains a sketch of what an
operation might look like, and includes the following optional parts:
- properties of the operation, whether meta (like a statement) or payload content
- the binding points where generated field values will be injected
- The names of values to be extracted from the result of a successful operation.
## Statement Forms
Sometimes operations are derived from a query language, and are thus self-contained in a string
form.
When mapping the template of an operation provided by users to an executable operation in some
native driver with specific values, you have to know
* The s
* The substance of the operation: The name and values of the fields that the user provides as part
of the operation
Command templates are the third layer of workload templating. As described in other spec documents,
the other layers are:

32
engine-api/src/main/resources/workload_definition/templated_operations.md
View File

@@ -48,7 +48,9 @@ op: select * from bar.table;
[
{
"name": "block0--stmt1",
"op": "select * from bar.table;"
"op": {
"stmt": "select * from bar.table;"
}
}
]
```
@@ -78,7 +80,9 @@ ops:
[
{
"name": "block0--stmt1",
"op": "select * from bar.table;"
"op": {
"stmt": "select * from bar.table;"
}
}
]
```
@@ -113,7 +117,9 @@ ops:
[
{
"name": "block0--op1",
"op": "select * from bar.table;"
"op": {
"stmt": "select * from bar.table;"
}
}
]
```
@@ -147,7 +153,9 @@ ops:
[
{
"name": "block0--op1",
"op": "select * from bar.table;"
"op": {
"stmt": "select * from bar.table;"
}
}
]
```
@@ -177,7 +185,9 @@ ops:
[
{
"name": "block0--op1",
"op": "select * from bar.table;"
"op": {
"stmt": "select * from bar.table;"
}
}
]
```
@@ -210,7 +220,9 @@ ops:
[
{
"name": "block0--op1",
"op": "select * from bar.table;"
"op": {
"stmt": "select * from bar.table;"
}
}
]
```
@@ -275,7 +287,9 @@ ops:
},
"description": "This is just an example operation",
"name": "block0--special-op-name",
"op": "select * from ks1.tb1;",
"op": {
"stmt": "select * from ks1.tb1;"
},
"params": {
"prepated": false
},
@@ -368,7 +382,9 @@ blocks:
"binding1": "NumberNameToString();"
},
"name": "block-named-fred--special-op-name",
"op": "select * from ks1.tb1;",
"op": {
"stmt": "select * from ks1.tb1;"
},
"params": {
"prepared": false
},

24
engine-api/src/main/resources/workload_definition/templated_workloads.md
View File

@@ -388,11 +388,15 @@ blocks:
[
{
"name": "namedblock1--op1",
"op": "select * from bar.table;"
"op": {
"stmt": "select * from bar.table;"
}
},
{
"name": "namedblock1--op2",
"op": "insert into bar.table (a,b,c) values (1,2,3);",
"op": {
"stmt": "insert into bar.table (a,b,c) values (1,2,3);"
},
"params": {
"type": "batch"
}
@@ -449,18 +453,24 @@ blocks:
[
{
"name": "block1--op1",
"op": "select * from bar.table;"
"op": {
"stmt": "select * from bar.table;"
}
},
{
"name": "block1--op2",
"op": "insert into bar.table (a,b,c) values (1,2,3);",
"op": {
"stmt": "insert into bar.table (a,b,c) values (1,2,3);"
},
"params": {
"type": "batch"
}
},
{
"name": "this-is-block-2--op3",
"op": "select * from foo.table;"
"op": {
"stmt": "select * from foo.table;"
}
}
]
```
@@ -501,7 +511,9 @@ blocks:
[
{
"name": "myblock--stmt1",
"op": "test op"
"op": {
"stmt": "test op"
}
}
]
```

16
engine-api/src/test/java/io/nosqlbench/engine/api/activityconfig/rawyaml/UniformWorkloadSpecificationTest.java
View File

@@ -83,6 +83,15 @@ public class UniformWorkloadSpecificationTest {
);
}
@Test
public void testCommandAPI() {
testSpecPath(
NBIO.fs().prefix("target/classes/workload_definition/")
.name("command_api.md")
.one().asPath()
);
}
public String summarize(Node node) {
StringBuilder sb = new StringBuilder();
@@ -293,7 +302,12 @@ public class UniformWorkloadSpecificationTest {
System.out.println("OK");
} catch (Exception e) {
System.out.println("ERROR");
// System.out.println("Error while validating equivalence between the yaml and the rendered op context:");
// System.out.println("yaml:");
// System.out.println(yaml);
// System.out.println("ops:");
// System.out.println(json);
throw new RuntimeException(e);
}

4
engine-api/src/test/java/io/nosqlbench/engine/api/activityconfig/yaml/ParsedStmtTest.java
View File

@@ -40,14 +40,12 @@ public class ParsedStmtTest {
StmtsBlock block0 = doclist.getStmtDocs().get(0).getBlocks().get(0);
OpTemplate stmtDef0 = block0.getOps().get(0);
ParsedStmt parsed0 = stmtDef0.getParsed();
assertThat(parsed0.getExtraBindings()).containsExactly("alpha","gamma");
assertThat(parsed0.getMissingBindings()).containsExactly("delta");
assertThat(parsed0.hasError()).isTrue();
StmtsBlock block1 = doclist.getStmtDocs().get(0).getBlocks().get(1);
OpTemplate stmtDef1 = block1.getOps().get(0);
ParsedStmt parsed1 = stmtDef1.getParsed();
assertThat(parsed1.getExtraBindings()).containsExactly();
assertThat(parsed1.getMissingBindings()).containsExactly();
assertThat(parsed1.hasError()).isFalse();
}
@@ -63,7 +61,7 @@ public class ParsedStmtTest {
OpTemplate stmtDef1 = block2.getOps().get(1);
ParsedStmt parsed1 = stmtDef1.getParsed();
assertThat(parsed1.getMissingBindings().isEmpty());
assertThat(parsed1.getMissingBindings()).isEmpty();
assertThat(parsed1.hasError()).isFalse();
}

6
nb-api/src/main/java/io/nosqlbench/nb/api/config/ParamsParser.java
View File

@@ -41,8 +41,8 @@ import java.util.Map;
* These occur as {@code name=value}, and may be delimited by spaces or semicolons, like
* {@code name1=value1 name2=value2} or {@code name1=value1;name2=value2}.
* Values can even contain spaces, even though space is a possible delimiter.
* Thus, {@code name1= value foo bar
* name2=baz} works, with the values {@code value foo bar} and {@code baz}.
* Thus, <em>{@code name1= value foo bar
* name2=baz}</em> works, with the values {@code value foo bar} and {@code baz}.
* </p>
*
* <h3>Names</h3>
@@ -237,7 +237,7 @@ public class ParamsParser {
s = ParseState.expectingName;
break;
case readingName:
parms.put(lastVarname,parms.get(lastVarname)+' '+varname.toString());
parms.put(lastVarname,parms.get(lastVarname)+' '+ varname);
varname.setLength(0);
break;
default:

160
virtdata-api/src/main/java/io/nosqlbench/virtdata/core/templates/ParsedTemplate.java
View File

@@ -65,6 +65,12 @@ import java.util.stream.StreamSupport;
*/
public class ParsedTemplate {
public enum Form {
literal,
rawbind,
template,
}
/**
* The canonical template pattern follows the pattern of an opening curly brace,
* followed by a word character, followed by any contiguous
@@ -78,24 +84,29 @@ public class ParsedTemplate {
* </pre>
*/
public final static Pattern STANDARD_ANCHOR = Pattern.compile("\\{(?<anchor>\\w+[-_\\d\\w.]*)}");
public final static Pattern EXTENDED_ANCHOR = Pattern.compile("\\{\\{(?<anchor>.*?)}}");
private final static Logger logger = LogManager.getLogger(ParsedTemplate.class);
private final Pattern[] patterns;
// Spans is an even-odd form of (literal, variable, ..., ..., literal)
private final String rawtemplate;
private final String[] spans;
private final Set<String> missingBindings = new HashSet<>();
private final Set<String> extraBindings = new HashSet<>();
// private final Pattern[] patterns;
/**
* Spans is an even-odd form of (literal, variable, ..., ..., literal)
* Thus a 1-length span is a single literal
**/
private final String[] spans;
// private final String rawtemplate;
/**
* A map of binding names and recipes (or null)
*/
private final Map<String, String> bindings = new LinkedHashMap<>();
private final Map<String, String> specificBindings = new LinkedHashMap<>();
/**
* Construct a new ParsedTemplate from the provided statement template.
*
* @param rawtemplate The string that contains literal sections and anchor sections interspersed
* @param rawtemplate The string that contains literal sections and anchor sections interspersed
* @param providedBindings The bindings that are provided for the template to be parsed
*/
public ParsedTemplate(String rawtemplate, Map<String, String> providedBindings) {
@@ -114,20 +125,28 @@ public class ParsedTemplate {
* have a named group with the name 'anchor', as in (?&lt;anchor&gt;...)
* </P>
*
* @param rawtemplate A string template which contains optionally embedded named anchors
* @param providedBindings The bindings which are provided by the user to fulfill the named anchors in this raw template
* @param providedPatterns The patterns which match the named anchor format and extract anchor names from the raw template
* @param rawtemplate A string template which contains optionally embedded named anchors
* @param availableBindings The bindings which are provided by the user to fulfill the named anchors in this raw template
* @param parserPatterns The patterns which match the named anchor format and extract anchor names from the raw template
*/
public ParsedTemplate(String rawtemplate, Map<String, String> providedBindings, Pattern... providedPatterns) {
this.rawtemplate = rawtemplate;
this.bindings.putAll(providedBindings);
this.patterns = providedPatterns;
this.spans = parse();
public ParsedTemplate(String rawtemplate, Map<String, String> availableBindings, Pattern... parserPatterns) {
this.bindings.putAll(availableBindings);
this.spans = parse(rawtemplate, availableBindings, parserPatterns);
}
public Form getForm() {
if (this.spans.length == 1) {
return Form.literal;
} else if (this.spans[0].isEmpty() && this.spans[2].isEmpty()) {
return Form.rawbind;
} else {
return Form.template;
}
}
public ParsedTemplate orError() {
if (hasError()) {
throw new RuntimeException("Unable to parse statement: " + this.toString());
throw new RuntimeException("Unable to parse statement: " + this);
}
return this;
}
@@ -140,11 +159,9 @@ public class ParsedTemplate {
* <li>specificBindings will contain an ordered map of the binding definitions</li>
* </ul>
*/
private String[] parse() {
private String[] parse(String rawtemplate, Map<String, String> providedBindings, Pattern[] patterns) {
List<String> spans = new ArrayList<>();
Set<String> usedAnchors = new HashSet<>();
extraBindings.addAll(bindings.keySet());
String statement = rawtemplate;
int patternsMatched = 0;
@@ -152,8 +169,8 @@ public class ParsedTemplate {
for (Pattern pattern : patterns) {
if (!pattern.toString().contains("?<anchor>")) {
throw new InvalidParameterException("The provided pattern '" + pattern.toString() + "' must contain a named group called anchor," +
"as in '(?<anchor>...)'");
throw new InvalidParameterException("The provided pattern '" + pattern + "' must contain a named group called anchor," +
"as in '(?<anchor>...)'");
}
Matcher m = pattern.matcher(rawtemplate);
@@ -171,14 +188,7 @@ public class ParsedTemplate {
spans.add(tokenName);
if (extraBindings.contains(tokenName)) {
usedAnchors.add(tokenName);
specificBindings.put(tokenName, bindings.get(tokenName));
} else {
missingBindings.add(tokenName);
}
}
usedAnchors.forEach(extraBindings::remove);
break; // If the last matcher worked at all, only do one cycle
}
@@ -189,42 +199,25 @@ public class ParsedTemplate {
spans.add(statement);
}
return spans.toArray(new String[0]);
}
public String toString() {
StringBuilder sb = new StringBuilder();
sb.append("template: '").append(rawtemplate).append("'");
sb.append("\n parsed: ");
sb.append(StreamSupport.stream(Arrays.spliterator(spans), false)
.map(s -> "[" + s + "]").collect(Collectors.joining(",")));
sb.append("\n missing bindings: ")
.append(missingBindings.stream().collect(Collectors.joining(",", "[", "]")));
sb.append(" extra bindings: ");
sb.append("\n extra bindings: ")
.append(extraBindings.stream().collect(Collectors.joining(",", "[", "]")));
return sb.toString();
String sb = "\n parsed: " +
StreamSupport.stream(Arrays.spliterator(spans), false)
.map(s -> "[" + s + "]").collect(Collectors.joining(",")) +
"\n missing bindings: " +
getMissing().stream().collect(Collectors.joining(",", "[", "]"));
return sb;
}
/**
* @return true if the parsed statement is not usable.
*/
public boolean hasError() {
return missingBindings.size() > 0;
}
/**
* The list of binding names returned by this method does not
* constitute an error. They may be used for
* for informational purposes in error handlers, for example.
*
* @return a set of bindings names which were provided to
* this parsed statement, but which were not referenced
* in either <pre>{anchor}</pre> or <pre>?anchor</pre> form.
*/
public Set<String> getExtraBindings() {
return extraBindings;
return getMissing().size() > 0;
}
/**
@@ -236,22 +229,19 @@ public class ParsedTemplate {
*
* @return A list of binding names which were referenced but not defined*
*/
public Set<String> getMissingBindings() {
return missingBindings;
}
public Set<String> getMissing() {
if (spans.length == 1) {
return Set.of();
}
/**
* Return a map of bindings which were referenced in the statement.
* This is an easy way to get the list of effective bindings for
* a statement for diagnostic purposes without including a potentially
* long list of library bindings. This method does <em>not</em>
* represent all of the binding points, as when anchor names are
* used more than once.
*
* @return a bindings map of referenced bindings in the statement
*/
public Map<String, String> getSpecificBindings() {
return specificBindings;
HashSet<String> missing = new HashSet<>();
for (int i = 1; i < spans.length; i += 2) {
if (!bindings.containsKey(spans[i])) {
missing.add(spans[i]);
}
}
return missing;
}
/**
@@ -275,11 +265,11 @@ public class ParsedTemplate {
* @throws InvalidParameterException if the template has an error,
* such as an anchor which has no provided binding.
*/
public List<BindPoint> getBindPoints() {
public List<BindPoint> getCheckedBindPoints() {
List<BindPoint> bindpoints = new ArrayList<>();
for (int i = 1; i < spans.length; i += 2) {
if (!bindings.containsKey(spans[i])) {
throw new InvalidParameterException("Binding named '" + spans[i] + "' is not provided for template '" + rawtemplate + "'");
throw new InvalidParameterException("Binding named '" + spans[i] + "' is not contained in the bindings map.");
}
bindpoints.add(new BindPoint(spans[i], bindings.get(spans[i])));
}
@@ -287,6 +277,16 @@ public class ParsedTemplate {
return bindpoints;
}
public List<BindPoint> getUncheckedBindPoints() {
List<BindPoint> bindpoints = new ArrayList<>();
for (int i = 1; i < spans.length; i += 2) {
bindpoints.add(new BindPoint(spans[i], bindings.getOrDefault(spans[i], null)));
}
return bindpoints;
}
/**
* Return the statement that can be used as-is by any driver specific version.
* This uses the anchor token as provided to yield a version of the statement
@@ -299,7 +299,7 @@ public class ParsedTemplate {
StringBuilder sb = new StringBuilder(spans[0]);
for (int i = 1; i < spans.length; i += 2) {
sb.append(tokenFormatter!=null ? tokenFormatter.apply(spans[i]) : spans[i]);
sb.append(tokenFormatter != null ? tokenFormatter.apply(spans[i]) : spans[i]);
sb.append(spans[i + 1]);
}
return sb.toString();
@@ -315,4 +315,18 @@ public class ParsedTemplate {
}
/**
* Returns the parsed template as a single binding spec if and only if the pattern matches
* a single binding anchor with no prefix or suffix.
*
* @return A single binding spec if that is all that was specified.
*/
public Optional<BindPoint> asBinding() {
if (spans.length == 3 && spans[0].isEmpty() && spans[2].isEmpty()) {
return Optional.of(new BindPoint(spans[1], bindings.getOrDefault(spans[1], null)));
} else {
return Optional.empty();
}
}
}

50
virtdata-api/src/test/java/io/nosqlbench/virtdata/core/templates/ParsedTemplateTest.java Normal file
View File

@@ -0,0 +1,50 @@
package io.nosqlbench.virtdata.core.templates;
import com.google.gson.Gson;
import com.google.gson.GsonBuilder;
import org.junit.Test;
import java.util.Map;
import java.util.Optional;
import static org.assertj.core.api.Assertions.assertThat;
public class ParsedTemplateTest {
@Test
public void testParsedTemplate() {
ParsedTemplate pt = new ParsedTemplate("test template", Map.of());
assertThat(pt.getAnchors()).isEmpty();
assertThat(pt.getCheckedBindPoints()).isEmpty();
assertThat(pt.getSpans()).contains("test template");
assertThat(pt.getMissing()).isEmpty();
}
@Test
public void testBindPoints() {
ParsedTemplate pt = new ParsedTemplate("test template {missing1}", Map.of("b1","v1"));
assertThat(pt.getSpans()).contains("test template ");
assertThat(pt.getAnchors()).containsExactly("missing1");
assertThat(pt.getUncheckedBindPoints()).containsExactly(new BindPoint("missing1",null));
}
@Test
public void testSingleBinding() {
ParsedTemplate pt = new ParsedTemplate("{single}", Map.of());
Optional<BindPoint> sb = pt.asBinding();
assertThat(sb).isPresent();
assertThat(sb).contains(new BindPoint("single",null));
}
@Test
public void testJsonFormat() {
ParsedTemplate pt = new ParsedTemplate("test template {missing1}", Map.of("b1","v1"));
Gson gson = new GsonBuilder().setPrettyPrinting().create();
String format = gson.toJson(pt);
System.out.println(format);
}
}