mirror of
https://github.com/nosqlbench/nosqlbench.git
synced 2024-12-01 04:49:18 -06:00
doc link fixes - made compatible with the site generator by explicitly qualifying anchor names
This commit is contained in:
Jonathan Shook
parent
186e87ac1b
commit
5d0a890e28
@ -4,16 +4,16 @@ title: AMQP
|
||||
---
|
||||
- [1. Overview](#1-overview)
|
||||
- [2. NB AMQP Usage](#2-nb-amqp-usage)
|
||||
- [2.1. Workload Definition](#21-workload-definition)
|
||||
- [2.1.1. Named Scenarios](#211-named-scenarios)
|
||||
- [2.2. CLI parameters](#22-cli-parameters)
|
||||
- [2.3. Configuration Properties](#23-configuration-properties)
|
||||
- [2.3.1. Global Properties File](#231-global-properties-file)
|
||||
- [2.3.2. Scenario Document Level Properties](#232-scenario-document-level-properties)
|
||||
- [2.1. Workload Definition](#21-workload-definition)
|
||||
- [2.1.1. Named Scenarios](#211-named-scenarios)
|
||||
- [2.2. CLI parameters](#22-cli-parameters)
|
||||
- [2.3. Configuration Properties](#23-configuration-properties)
|
||||
- [2.3.1. Global Properties File](#231-global-properties-file)
|
||||
- [2.3.2. Scenario Document Level Properties](#232-scenario-document-level-properties)
|
||||
|
||||
---
|
||||
|
||||
# 1. Overview
|
||||
# 1. Overview {#1-overview}
|
||||
|
||||
The NB AMQP adapter allows sending messages to or receiving messages from
|
||||
* an AMQP 0-9-1 based server (e.g. RabbitMQ), or
|
||||
@ -22,17 +22,17 @@ The NB AMQP adapter allows sending messages to or receiving messages from
|
||||
At high level, this adapter supports the following AMQP 0-9-1 functionalities
|
||||
* Creating AMQP connections and channels
|
||||
* Declaring AMQP exchanges
|
||||
* The following exchange types are supported: `direct`, `fanout`, `topic`, and `headers`
|
||||
* The following exchange types are supported: `direct`, `fanout`, `topic`, and `headers`
|
||||
* Sending messages to AMQP exchanges with sync. or async. publisher confirms
|
||||
* For sync confirms, it supports both single and batch confirms
|
||||
* Supports message-send based on routing keys
|
||||
* For sync confirms, it supports both single and batch confirms
|
||||
* Supports message-send based on routing keys
|
||||
* Declaring and binding AMQP queues
|
||||
* Supports message-receive based on binding keys
|
||||
* Supports message-receive based on binding keys
|
||||
* Receiving messages from AMQP queues with async. consumer acks
|
||||
|
||||
# 2. NB AMQP Usage
|
||||
# 2. NB AMQP Usage {#2-nb-amqp-usage}
|
||||
|
||||
## 2.1. Workload Definition
|
||||
## 2.1. Workload Definition {#21-workload-definition}
|
||||
|
||||
There are two main types of workloads supported by this adapter:
|
||||
* Message sender workload (see [amqp_msg_sender.yaml](scenarios/amqp_msg_sender.yaml))
|
||||
@ -53,7 +53,7 @@ $ <nb_cmd> run driver=amqp -vv cycles=200 strict_msg_error_handling=0 \
|
||||
config=/path/to/amqp_config.properties
|
||||
```
|
||||
|
||||
### 2.1.1. Named Scenarios
|
||||
### 2.1.1. Named Scenarios {#211-named-scenarios}
|
||||
|
||||
For workload execution convenience, NB engine has the concept of **named scenario** ([doc](https://docs.nosqlbench.io/workloads-101/11-named-scenarios/)).
|
||||
|
||||
@ -68,7 +68,7 @@ $ <nb_cmd> nbamqp_msg_proc_named msg_send
|
||||
$ <nb_cmd> nbamqp_msg_proc_named msg_recv
|
||||
```
|
||||
|
||||
## 2.2. CLI parameters
|
||||
## 2.2. CLI parameters {#22-cli-parameters}
|
||||
|
||||
The following CLI parameters are unique to this adapter:
|
||||
|
||||
@ -77,12 +77,12 @@ The following CLI parameters are unique to this adapter:
|
||||
* `num_exchange`: the number of AMQP exchanges to create for each channel
|
||||
* `num_queue`: the number of AMQP queues to create for each channel (only relevant for message receiver workload)
|
||||
* `num_msg_client`: the number of message clients to create for each channel
|
||||
* for message sender workload, it is the number of message publishers for each exchange
|
||||
* for message receiver workload, it is the number of message consumers for each queue
|
||||
* for message sender workload, it is the number of message publishers for each exchange
|
||||
* for message receiver workload, it is the number of message consumers for each queue
|
||||
|
||||
## 2.3. Configuration Properties
|
||||
## 2.3. Configuration Properties {#23-configuration-properties}
|
||||
|
||||
### 2.3.1. Global Properties File
|
||||
### 2.3.1. Global Properties File {#231-global-properties-file}
|
||||
|
||||
A global AMQP properties file can be specified via the `config` CLI parameter. It includes the following required properties:
|
||||
* `amqpSrvHost`: AMQP server host (e.g. An Astra Streaming cluster with S4R enabled)
|
||||
@ -95,15 +95,15 @@ A global AMQP properties file can be specified via the `config` CLI parameter. I
|
||||
|
||||
An example of this file can be found from: [amqp_config.properties](conf/amqp_config.properties)
|
||||
|
||||
### 2.3.2. Scenario Document Level Properties
|
||||
### 2.3.2. Scenario Document Level Properties {#232-scenario-document-level-properties}
|
||||
|
||||
For message sender workload, the following Document level configuration parameters are supported in the YAML file:
|
||||
* `publisher_confirm`: whether to use publisher confirms
|
||||
* `confirm_mode`: When `publisher_confirm` is true, the following 3 confirm modes are supported:
|
||||
* `individual`: wait for confirm individually
|
||||
* `batch`: wait for confirm in batch
|
||||
* `async`: [default] no wait for confirm
|
||||
* `individual`: wait for confirm individually
|
||||
* `batch`: wait for confirm in batch
|
||||
* `async`: [default] no wait for confirm
|
||||
* `confirm_batch_num`: batch size for waiting for **sync** publisher confirms
|
||||
* Only relevant when `publisher_confirm` is true and `confirm_mode` is "batch"
|
||||
* Only relevant when `publisher_confirm` is true and `confirm_mode` is "batch"
|
||||
* `dft_confirm_timeout_ms`: default timeout in milliseconds for waiting publisher confirms
|
||||
* Only relevant when `publisher_confirm` is true and `confirm_mode` is "individual" or "batch"
|
||||
* Only relevant when `publisher_confirm` is true and `confirm_mode` is "individual" or "batch"
|
||||
|
||||
@ -35,11 +35,11 @@ The NB Pulsar driver allows you to simulate and run different types of workloads
|
||||
* Negative acknowledgement and acknowledgement timeout redelivery backoff policy
|
||||
|
||||
|
||||
## 1.1. Issues Tracker
|
||||
## 1.1. Issues Tracker {#11-issues-tracker}
|
||||
|
||||
If you have issues or new requirements for this driver, please add them at the [pulsar issues tracker](https://github.com/nosqlbench/nosqlbench/issues/new?labels=pulsar).
|
||||
|
||||
# 2. Execute the NB Pulsar Driver Workload
|
||||
# 2. Execute the NB Pulsar Driver Workload {#2-execute-the-nb-pulsar-driver-workload}
|
||||
|
||||
In order to run a NB Pulsar driver workload, it follows similar command as other NB driver types. But it does have its unique execution parameters. The general command has the following format:
|
||||
|
||||
@ -52,7 +52,7 @@ In the above command, make sure the driver type is **pulsar** and provide the fo
|
||||
* ***service_url***: Pulsar native protocol service url and default to "pulsar://localhost:6650"
|
||||
* ***config***: Pulsar schema/client/producer/consumer related configuration (as a property file)
|
||||
|
||||
## 2.1. NB Pulsar Driver Yaml File High Level Structure
|
||||
## 2.1. NB Pulsar Driver Yaml File High Level Structure {#21-nb-pulsar-driver-yaml-file-high-level-structure}
|
||||
|
||||
Just like other NB driver types, the actual NB Pulsar workload is defined in a YAML file with the following high level structure:
|
||||
|
||||
@ -84,7 +84,7 @@ blocks:
|
||||
* ***params***: This section defines **Document level** configuration parameters that apply to all OpTemplate blocks.
|
||||
* ***blocks***: This section defines the OpTemplate blocks that are needed to execute Pulsar specific workloads. Each OpTemplate block may contain multiple OpTemplates.
|
||||
|
||||
## 2.2. NB Pulsar Driver Configuration Parameters
|
||||
## 2.2. NB Pulsar Driver Configuration Parameters {#22-nb-pulsar-driver-configuration-parameters}
|
||||
|
||||
The NB Pulsar driver configuration parameters can be set at 3 different levels:
|
||||
* Global level
|
||||
@ -95,7 +95,7 @@ The NB Pulsar driver configuration parameters can be set at 3 different levels:
|
||||
|
||||
Please **NOTE** that when a parameter is specified at multiple levels, the one at the lowest level takes precedence.
|
||||
|
||||
### 2.2.1. Global Level Parameters
|
||||
### 2.2.1. Global Level Parameters {#221-global-level-parameters}
|
||||
|
||||
The parameters at this level are those listed in the command line config properties file.
|
||||
|
||||
@ -152,7 +152,7 @@ categories of the configuration parameters:
|
||||
* This section defines all configuration parameters that are related with defining a Pulsar Consumer object.
|
||||
* See [Pulsar Doc Reference](http://pulsar.apache.org/docs/en/client-libraries-java/#configure-consumer)
|
||||
|
||||
### 2.2.2. Document Level Parameters
|
||||
### 2.2.2. Document Level Parameters {#222-document-level-parameters}
|
||||
|
||||
For the Pulsar NB driver, Document level parameters can only be statically bound; and currently, the following Document level configuration parameters are supported:
|
||||
|
||||
@ -180,7 +180,7 @@ For the Pulsar NB driver, Document level parameters can only be statically bound
|
||||
* `message_event_time` : uses the message event timestamp as the starting time
|
||||
* `message_property_e2e_starting_time` : uses a message property `e2e_starting_time` as the starting time.
|
||||
|
||||
# 3. NB Pulsar Driver OpTemplates
|
||||
# 3. NB Pulsar Driver OpTemplates {#3-nb-pulsar-driver-optemplates}
|
||||
|
||||
For the NB Pulsar driver, each OpTemplate has the following format:
|
||||
```yaml
|
||||
@ -214,9 +214,9 @@ Its value is mandatory and depending on the actual identifier, its value can be
|
||||
|
||||
Each Pulsar `OpType` may have optional Op specific parameters. Please refer to [here](yaml_examples) for the example NB Pulsar YAML files for each OpType
|
||||
|
||||
# 4. Message Generation and Schema Support
|
||||
# 4. Message Generation and Schema Support {#4-message-generation-and-schema-support}
|
||||
|
||||
## 4.1. Message Generation
|
||||
## 4.1. Message Generation {#41-message-generation}
|
||||
|
||||
A Pulsar message has three main components: message key, message properties, and message payload. Among them, message payload is mandatory when creating a message.
|
||||
|
||||
@ -244,7 +244,7 @@ For `msg_value`, its value can be either
|
||||
* a plain simple text, or
|
||||
* a JSON string that follows the specified "value" Avro schema (when Avro schema or KeyValue schema is used)
|
||||
|
||||
## 4.2. Schema Support
|
||||
## 4.2. Schema Support {#42-schema-support}
|
||||
|
||||
The NB Pulsar driver supports the following Pulsar schema types:
|
||||
* Primitive schema types
|
||||
|
||||
@ -6,21 +6,21 @@ title: S4J
|
||||
- [2. Execute NB S4J Workload](#2-execute-nb-s4j-workload)
|
||||
- [3. NB S4J Driver Configuration Parameter File](#3-nb-s4j-driver-configuration-parameter-file)
|
||||
- [4. NB S4J Scenario Definition File](#4-nb-s4j-scenario-definition-file)
|
||||
- [4.1. Document Level Parameters](#41-document-level-parameters)
|
||||
- [4.2. NB S4J Workload Types](#42-nb-s4j-workload-types)
|
||||
- [4.2.1. Publish Messages to a JMS Destination, Queue or Topic](#421-publish-messages-to-a-jms-destination-queue-or-topic)
|
||||
- [4.2.2. Receiving Messages from a JMS Destination, Queue or Topic](#422-receiving-messages-from-a-jms-destination-queue-or-topic)
|
||||
- [4.3. S4J Named Scenario](#43-s4j-named-scenario)
|
||||
- [4.1. Document Level Parameters](#41-document-level-parameters)
|
||||
- [4.2. NB S4J Workload Types](#42-nb-s4j-workload-types)
|
||||
- [4.2.1. Publish Messages to a JMS Destination, Queue or Topic](#421-publish-messages-to-a-jms-destination-queue-or-topic)
|
||||
- [4.2.2. Receiving Messages from a JMS Destination, Queue or Topic](#422-receiving-messages-from-a-jms-destination-queue-or-topic)
|
||||
- [4.3. S4J Named Scenario](#43-s4j-named-scenario)
|
||||
|
||||
---
|
||||
|
||||
# 1. Overview
|
||||
# 1. Overview {#1-overview}
|
||||
|
||||
This driver is similar to [NB Pulsar driver](../../../../adapter-pulsar/src/main/resources/pulsar.md) that allows NB based workload generation and performance testing against a Pulsar cluster. It also follows a similar pattern to configure and connect to the Pulsar cluster for workload execution.
|
||||
|
||||
However, the major difference is instead of simulating native Pulsar client workloads, the NB S4J driver allows simulating JMS oriented workloads (that follows JMS spec 2.0 and 1.1) to be executed on the Pulsar cluster. Under the hood, this is achieved through DataStax's [Starlight for JMS API] (https://github.com/datastax/pulsar-jms).
|
||||
|
||||
# 2. Execute NB S4J Workload
|
||||
# 2. Execute NB S4J Workload {#2-execute-nb-s4j-workload}
|
||||
|
||||
The following is an example of executing a NB S4J workload (defined as *pulsar_s4j.yaml*)
|
||||
|
||||
@ -31,16 +31,16 @@ $ <nb_cmd> run driver=s4j cycles=10000 threads=4 num_conn=2 num_session=2 sessio
|
||||
In the above NB CLI command, the S4J driver specific parameters are listed as below:
|
||||
* num_conn: the number of JMS connections to be created
|
||||
* num_session: the number of JMS sessions per JMS connection
|
||||
* Note that multiple JMS sessions can be created from one JMS connection, and they share the same connection characteristics.
|
||||
* Note that multiple JMS sessions can be created from one JMS connection, and they share the same connection characteristics.
|
||||
* session_mode: the session mode used when creating a JMS session
|
||||
* web_url: the URL of the Pulsar web service
|
||||
* service_url: the URL of the Pulsar native protocol service
|
||||
* (optional) strict_msg_error_handling: whether to do strict error handling
|
||||
* when true, Pulsar client error will not stop NB S4J execution
|
||||
* otherwise, any Pulsar client error will stop NB S4J execution
|
||||
* when true, Pulsar client error will not stop NB S4J execution
|
||||
* otherwise, any Pulsar client error will stop NB S4J execution
|
||||
* (optional) max_s4jop_time: maximum time (in seconds) to execute the actual S4J operations (e.g. message sending or receiving). If NB execution time is beyond this limit, each NB cycle is just a no-op. Please NOTE:
|
||||
* this is useful when controlled NB execution is needed with NB CLI scripting.
|
||||
* if this parameter is not specified or the value is 0, it means no time limitation. Every single NB cycle will trigger an actual S4J operation.
|
||||
* this is useful when controlled NB execution is needed with NB CLI scripting.
|
||||
* if this parameter is not specified or the value is 0, it means no time limitation. Every single NB cycle will trigger an actual S4J operation.
|
||||
* (optional) track_msg_cnt: When set to true (with default as false), the S4J driver will keep track of the confirmed response count for message sending and receiving.
|
||||
|
||||
Other NB engine parameters are straight forward:
|
||||
@ -49,7 +49,7 @@ Other NB engine parameters are straight forward:
|
||||
* yamL: the NB S4J scenario definition yaml file
|
||||
* config: specify the file that contains the connection parameters used by the S4J API
|
||||
|
||||
# 3. NB S4J Driver Configuration Parameter File
|
||||
# 3. NB S4J Driver Configuration Parameter File {#3-nb-s4j-driver-configuration-parameter-file}
|
||||
|
||||
The S4J API has a list of configuration options that can be found here: https://docs.datastax.com/en/streaming/starlight-for-jms/latest/reference/pulsar-jms-reference.html.
|
||||
|
||||
@ -104,7 +104,7 @@ producer.blockIfQueueFull=true
|
||||
#...
|
||||
```
|
||||
|
||||
# 4. NB S4J Scenario Definition File
|
||||
# 4. NB S4J Scenario Definition File {#4-nb-s4j-scenario-definition-file}
|
||||
|
||||
Like any NB scenario yaml file, the NB S4J yaml file is composed of 3 major components:
|
||||
* bindings: define NB bindings
|
||||
@ -120,7 +120,7 @@ blocks:
|
||||
... ...
|
||||
```
|
||||
|
||||
## 4.1. Document Level Parameters
|
||||
## 4.1. Document Level Parameters {#41-document-level-parameters}
|
||||
|
||||
The parameters defined in this section will be applicable to all statement blocks. An example of some common parameters that can be set at the document level is listed below:
|
||||
* temporary_dest: whether JMS workload is dealing with a temporary destination
|
||||
@ -139,15 +139,15 @@ params:
|
||||
|
||||
Please **NOTE** that the above parameters won't necessarily be specified at the document level. If they're specified at the statement level, they will only impact the statement within which they're specified.
|
||||
|
||||
## 4.2. NB S4J Workload Types
|
||||
## 4.2. NB S4J Workload Types {#42-nb-s4j-workload-types}
|
||||
|
||||
The NB S4J driver supports 2 types of JMS operations:
|
||||
* One for message producing/sending/publishing
|
||||
* this is identified by NB Op identifier ***MessageProduce***
|
||||
* this is identified by NB Op identifier ***MessageProduce***
|
||||
* One for message consuming/receiving/subscribing
|
||||
* this is identified by NB Op identifier ***MessageConsume***
|
||||
* this is identified by NB Op identifier ***MessageConsume***
|
||||
|
||||
### 4.2.1. Publish Messages to a JMS Destination, Queue or Topic
|
||||
### 4.2.1. Publish Messages to a JMS Destination, Queue or Topic {#421-publish-messages-to-a-jms-destination-queue-or-topic}
|
||||
|
||||
***NOTE**: Please see [pulsar_s4j_producer.yaml](scenarios/pulsar_s4j_producer.yaml) as the complete example.*
|
||||
|
||||
@ -184,7 +184,7 @@ blocks:
|
||||
msg_body: "{mytext_val}"
|
||||
```
|
||||
|
||||
### 4.2.2. Receiving Messages from a JMS Destination, Queue or Topic
|
||||
### 4.2.2. Receiving Messages from a JMS Destination, Queue or Topic {#422-receiving-messages-from-a-jms-destination-queue-or-topic}
|
||||
|
||||
***NOTE**: Please see [pulsar_s4j_consumer.yaml](scenarios/pulsar_s4j_consumer.yaml) as the complete example.*
|
||||
|
||||
@ -192,15 +192,15 @@ The generic NB S4J statement block for receiving messages to a JMS destination (
|
||||
* **msg_selector**: Message selector string
|
||||
* **no_local**: Only applicable to a Topic as the destination. This allows a subscriber to inhibit the delivery of messages published by its own connection.
|
||||
* **read_timeout**: The timeout value for receiving a message from a destination
|
||||
* This setting only works if **no_wait** is false
|
||||
* If the **read_timeout** value is 0, it behaves the same as **no_wait** is true
|
||||
* This setting only works if **no_wait** is false
|
||||
* If the **read_timeout** value is 0, it behaves the same as **no_wait** is true
|
||||
* **no_wait**: Whether to receive the next message immediately if one is available
|
||||
* **msg_ack_ratio**: the ratio of the received messages being acknowledged
|
||||
* **slow_ack_in_sec**: whether to simulate a slow consumer (pause before acknowledging after receiving a message)
|
||||
* value 0 means no simulation (consumer acknowledges right away)
|
||||
* value 0 means no simulation (consumer acknowledges right away)
|
||||
* negative ack/ack timeout/deadletter topic related settings
|
||||
* The settings here (as the scenario specific settings) will be merged with the
|
||||
* global settings in *s4j_config.properties* file
|
||||
* The settings here (as the scenario specific settings) will be merged with the
|
||||
* global settings in *s4j_config.properties* file
|
||||
|
||||
```yaml
|
||||
blocks:
|
||||
@ -254,7 +254,7 @@ blocks:
|
||||
}
|
||||
```
|
||||
|
||||
## 4.3. S4J Named Scenario
|
||||
## 4.3. S4J Named Scenario {#43-s4j-named-scenario}
|
||||
|
||||
For workload execution convenience, NB engine has the concept of **named scenario** ([doc](https://docs.nosqlbench.io/workloads-101/11-named-scenarios/)).
|
||||
|
||||
|
||||
Loading…
Reference in New Issue
Block a user