Commit Graph

51 Commits

Author SHA1 Message Date
yabinmeng
2730b0aa24 Update NBS4J version to 4.0.1 (which is based on Pulsar client version 3.0.0) and NB Pulsar version to 3.0.0 2023-06-23 21:47:19 -05:00
yabinmeng
046c0fc1c6 Update NBS4J version to 4.0.1 (which is based on Pulsar client version 3.0.0) and NB Pulsar version to 3.0.0 2023-06-23 21:44:30 -05:00
Jeff Banks
a9fb0d7e63 Revert "fix: upgrade org.apache.pulsar:pulsar-client from 2.10.4 to 2.11.1 (#1282)"
This reverts commit 4de2aa67d6.
2023-05-19 16:14:38 -05:00
Jonathan Shook
4de2aa67d6
fix: upgrade org.apache.pulsar:pulsar-client from 2.10.4 to 2.11.1 (#1282)
Snyk has created this PR to upgrade org.apache.pulsar:pulsar-client from 2.10.4 to 2.11.1.

See this package in Maven Repository:
https://mvnrepository.com/artifact/org.apache.pulsar/pulsar-client/

See this project in Snyk:
https://app.snyk.io/org/nosqlbench/project/829dafa3-4836-462a-b1f4-15be99fe372a?utm_source=github&utm_medium=referral&page=upgrade-pr

Co-authored-by: snyk-bot <snyk-bot@snyk.io>
2023-05-19 08:51:40 -05:00
yabinmeng
7a5831b4ff 1) NB Kafka and NB S4J adaptor code efficiency improvement (following PR 10115 by Lari on Pulsar adaptor)
2) Downgrade NB Pulsar version from 2.11.1 to 2.10.4 to avoid NB S4J execution failure (runtime version conflict)
3) Fix S4J Adaptor metrics labeling conflict
4) Change the space shutdown logic for NB Pulsar, Kafka, and S4J such that to simply log an error message instead of throwing out a run-time exception (NB is shutting down anyway).
5) Minor update of NB Kafka and NB S4J properties file and add settings for connecting to Astra Streaming
2023-05-16 19:59:39 -05:00
Jeff Banks
02ff160b3c
Metrics Publishing (#1234)
Included the following with core changes to allow labeled metrics for Prometheus exposition format publishing.
* javadoc updates
* remove extra types
* use NBLabeledElement instead of NBNamedElement
* contextualize NBLabeledElement for graphite/metrics
* externalize labeled ScriptContext to API
* add labels to NicerTimer
* remove partial packaging
* more progress on formatting for prom exposition format
* added metrics diagram
* resolve build issues with label params
* resolve build issues with label params
* prometheus export services
* added PromExpoFormat Tests for NBMetricMeter(Counting+Sampling) and NBMetricTimer(Counting)
* added test for Gauge Formatting
* added Gauge Formatting as well as Sampling values (count, stdev ...)
* added sketch for metrics labeling contexts
* add NBLabeledElement in all the places, retool calling paths to use it
* synchronize antlr versions after partial snyk change
* unbreak static initializer block after IntelliJ "fixed" it.
* engine-api - adapt to NBLabeledElement
* adapters-api - adapt to NBLabeledElement
* nb-api - adapt to NBLabeledElement
* engine-core - adapt to NBLabeledElement
* misc-adapters - adapt to NBLabeledElement
* streaming-adapters - adapt to NBLabeledElement
* add missing test
* initial implementation of a prom push reporter
* Resolve build issue with parseGlobalOptions
* replaced with PromPushReporter
* cleanup unused deps
* dependency removal for micrometer
* allow empty labels for tests
* change space.getName to space.getSpaceName
* cleanup poms
* graphite linearization now includes space element
* http adapter should only depend on adapters API
* http space does not create its own metric names
* import cleanups
* improved javadocs
* introduce component concepts

---------

Co-authored-by: Jonathan Shook <jshook@gmail.com>
Co-authored-by: Mike Yaacoub <mike.yaacoub@datastax.com>
2023-05-09 09:52:42 -05:00
Jonathan Shook
ed6593af06
fix: upgrade org.apache.pulsar:pulsar-client from 2.11.0 to 2.11.1 (#1249)
Snyk has created this PR to upgrade org.apache.pulsar:pulsar-client from 2.11.0 to 2.11.1.

See this package in Maven Repository:
https://mvnrepository.com/artifact/org.apache.pulsar/pulsar-client/

See this project in Snyk:
https://app.snyk.io/org/nosqlbench/project/829dafa3-4836-462a-b1f4-15be99fe372a?utm_source=github&utm_medium=referral&page=upgrade-pr

Co-authored-by: snyk-bot <snyk-bot@snyk.io>
2023-05-05 09:12:35 -05:00
Jonathan Shook
9295adf03f
Merge pull request #1143 from MMirelli/mm-ls797-continue-add-e2e-error-metrics-to-kafka-adapter
[kafka-adapter] Add e2e error metrics -- to be continued
2023-04-25 09:10:32 -05:00
snyk-bot
265dcce899
fix: upgrade org.apache.commons:commons-configuration2 from 2.8.0 to 2.9.0
Snyk has created this PR to upgrade org.apache.commons:commons-configuration2 from 2.8.0 to 2.9.0.

See this package in Maven Repository:
https://mvnrepository.com/artifact/org.apache.commons/commons-configuration2/

See this project in Snyk:
https://app.snyk.io/org/nosqlbench/project/829dafa3-4836-462a-b1f4-15be99fe372a?utm_source=github&utm_medium=referral&page=upgrade-pr
2023-04-18 02:15:34 +00:00
Massimiliano Mirelli
ef92631595 Add flag to run tests for adapters-api and kafka / pulsar adapters 2023-03-07 15:13:47 +02:00
Massimiliano Mirelli
0a2ab1dcca Add scripts to build pulsar and kafka adapters and manually test against local pulsar cluster 2023-03-07 14:52:21 +02:00
Massimiliano Mirelli
8eb9498fa9 Add examples of simple testing pulsar workload 2023-03-07 14:51:23 +02:00
Massimiliano Mirelli
7d5ecf2f32 Enhance pulsar and kafka adapters documentation 2023-03-07 14:15:21 +02:00
Massimiliano Mirelli
971c5d3c6f Add sequence tracking number when seqTracking is enabled in producer 2023-03-07 14:09:34 +02:00
Massimiliano Mirelli
b333898249 Decouple e2e error metrics from adapter modules
Add classes implementing e2e error metrics to adapters-api module in `import io.nosqlbench.engine.api.metrics.ReceivedMessageSequenceTracker`
2023-03-07 14:06:15 +02:00
Massimiliano Mirelli
70d66719db Edit copyright year 2023-02-24 10:44:14 +02:00
Jonathan Shook
42bc77f0f2
Merge pull request #1017 from lhotari/lh-readd-removed-tests
Re-add tests for MessageSequenceNumberSendingHandler and ReceivedMessageSequenceTracker
2023-02-07 13:52:29 -06:00
Lari Hotari
f9fe79818a Re-add tests for MessageSequenceNumberSendingHandler and ReceivedMessageSequenceTracker
- migrate tests from NB4 to NB5
2023-02-07 08:45:53 +02:00
Lari Hotari
5491790751 Optimize cache keys and fix races with caches in PulsarSpace 2023-02-07 08:10:45 +02:00
Lari Hotari
10da1699ed Optimize enum mappings and cache Avro schemas in PulsarAdapterUtil 2023-02-07 08:10:45 +02:00
Jonathan Shook
1eead1add8 centralize jacoco template for reconfiguration 2023-01-26 23:32:15 -06:00
Jonathan Shook
696576a317 Merge branch 'main' of github.com:nosqlbench/nosqlbench 2023-01-23 13:44:11 -06:00
snyk-bot
87074249c1
fix: upgrade org.apache.pulsar:pulsar-client from 2.10.3 to 2.11.0
Snyk has created this PR to upgrade org.apache.pulsar:pulsar-client from 2.10.3 to 2.11.0.

See this package in Maven Repository:
https://mvnrepository.com/artifact/org.apache.pulsar/pulsar-client/

See this project in Snyk:
https://app.snyk.io/org/nosqlbench/project/829dafa3-4836-462a-b1f4-15be99fe372a?utm_source=github&utm_medium=referral&page=upgrade-pr
2023-01-22 13:42:54 +00:00
Jonathan Shook
f63c7aeadb replace CI+CD friendly revision 2023-01-19 16:54:06 -06:00
snyk-bot
218eb5f44c
fix: upgrade org.apache.pulsar:pulsar-client-admin from 2.10.1 to 2.10.3
Snyk has created this PR to upgrade org.apache.pulsar:pulsar-client-admin from 2.10.1 to 2.10.3.

See this package in Maven Repository:
https://mvnrepository.com/artifact/org.apache.pulsar/pulsar-client-admin/

See this project in Snyk:
https://app.snyk.io/org/nosqlbench/project/829dafa3-4836-462a-b1f4-15be99fe372a?utm_source=github&utm_medium=referral&page=upgrade-pr
2023-01-16 23:37:35 +00:00
nb-droid
a49a9e9380 [maven-release-plugin] prepare for next development iteration 2023-01-13 19:51:57 +00:00
nb-droid
67aeba7a34 [maven-release-plugin] prepare release nosqlbench-5.17.0 2023-01-13 19:51:55 +00:00
Jonathan Shook
e04eef814c temporary work-around for release plugin incompatibility 2023-01-13 13:19:35 -06:00
Jonathan Shook
ff188d4860
actions-for-docs (#919)
* updated docs action with version state

* script updates

* misc doc system updates

* Update build.yml

remove versions logic, since zip contains version info now.

* Update build.yml

corrected git add command

* update version format

* updated docs site url
2023-01-13 11:57:08 -06:00
Jonathan Shook
31478ae591
Nosqlbench 904 version5 (#908)
* Ignore .flattened-pom.xml files from the CI-friendly flattening

* Allow newer versions to squelch version warning

* * switch to CI-friendly revisions in Maven
* add flattener
* refactor version tags
* remove unecesary indirection in poms which break version intellisense

* * switch to CI-friendly revisions in Maven
* add flattener
* refactor version tags
* remove unnecessary indirection in poms which break version intellisense

* deleting commented code
* version dep updates
* streamline LUT trimming options
* use a more explicit regex substituion technique
* specify updated okhttp version
* update resources plugin to bugfixed version
* exclude filtering for problematic files
2023-01-12 12:48:50 -06:00
Jeff Banks
9c25ad9336
Code coverage integration (#897)
* Jacoco introduced for full reporting of test coverage.
* All modules included in scope.
* Ability to scope to module OR overall reporting.
* Ability to run in dev environment and bypass as needed w/out GHA.
2023-01-12 11:25:40 -06:00
nb-droid
6e93468632 [maven-release-plugin] prepare for next development iteration 2023-01-09 19:36:28 +00:00
nb-droid
d1b9c91e44 [maven-release-plugin] prepare release nosqlbench-4.17.32 2023-01-09 19:36:26 +00:00
Jonathan Shook
08fb6ca905 make anchor links zola compatible 2022-12-22 23:54:57 -06:00
Jonathan Shook
972a6af654 remove duplicitous doc file 2022-12-22 23:54:36 -06:00
Mike Yaacoub
74bfbc691d Merge branch 'main' into my-NB5-I446 2022-12-19 16:58:50 -05:00
Mike Yaacoub
b63eba35c5 fixed pulsar.md for export-docs after merge 2022-12-19 16:16:50 -05:00
Mike Yaacoub
a06e1556f0 merged my-NB5-I446 with main 2022-12-19 15:57:58 -05:00
MikeYaacoubStax
e7705d13de NB5-I446 Modified export-docs command to export organized driver and binding docs, also removed gendocs command (VirtDataGenDocsApp now used for generating bindings in StringBuilders to pass to the zipstream)
added the export-docs build workflow job
Updated build.yml to fix path to nb5 in workflow
uploading/downloading docs artifacts in build workflow
Attempt to correct the path to the exported docs to upload them
Corrected exported-docs path
Corrected yml build workflow error
corrected naming of exported docs
2022-12-08 12:20:11 -05:00
yabinmeng
9a00af0806
Merge branch 'nosqlbench:main' into main 2022-12-07 21:58:35 -06:00
nb-droid
02522a468e [maven-release-plugin] prepare for next development iteration 2022-12-07 23:30:05 +00:00
nb-droid
fb131e9b09 [maven-release-plugin] prepare release nosqlbench-4.17.31 2022-12-07 23:30:02 +00:00
yabinmeng
b3d3f59d40 Migrate NBS4J from NB4 to NB5 2022-12-06 17:24:52 -06:00
yabinmeng
c077218e8a Fix example yaml file producer and consumer parameter names 2022-11-21 21:05:26 -06:00
yabinmeng
f7c9e50f35 Add README file and adjust sample yaml files 2022-11-19 18:53:58 -06:00
yabinmeng
bd8aff9081 Added support for 1) subscription initial position and 2) regex subscription mode
Also fixed a consumer configuration loading issue with dlq/negAck/ackTimeout policy using Pulsar client's loadConf() API.

Addressed the review comments in PR #791; reverted the changes in ActivityMetrics and BasedOpDispenser
2022-11-18 18:22:04 -06:00
yabinmeng
0de7102884 Add support for
1) producer message compression
2) consumer DLT/negAck/ackTimeout policy

Add sample NB scenario yaml files
2022-11-17 18:47:37 -06:00
yabinmeng
12f8697e0e Complete the first draft of NB5 Pulsar adapter code, except the per-thread rate limiter. 2022-11-17 10:39:37 -06:00
yabinmeng
0c71696b15 Completed NB4 Pulsar driver code migration to NB5. But there are still a few lingering issues due to the difference of how NB4 vs NB5 driver/adapter interacts with the NB engine. 2022-11-16 15:37:26 -06:00
yabinmeng
ca6f2b052b Add PulsarAdapter specific metrics 2022-11-14 15:27:57 -06:00