Commit Graph

27 Commits

Author SHA1 Message Date
yabinmeng
7a5831b4ff 1) NB Kafka and NB S4J adaptor code efficiency improvement (following PR 10115 by Lari on Pulsar adaptor)
2) Downgrade NB Pulsar version from 2.11.1 to 2.10.4 to avoid NB S4J execution failure (runtime version conflict)
3) Fix S4J Adaptor metrics labeling conflict
4) Change the space shutdown logic for NB Pulsar, Kafka, and S4J such that to simply log an error message instead of throwing out a run-time exception (NB is shutting down anyway).
5) Minor update of NB Kafka and NB S4J properties file and add settings for connecting to Astra Streaming
2023-05-16 19:59:39 -05:00
Jeff Banks
02ff160b3c
Metrics Publishing (#1234)
Included the following with core changes to allow labeled metrics for Prometheus exposition format publishing.
* javadoc updates
* remove extra types
* use NBLabeledElement instead of NBNamedElement
* contextualize NBLabeledElement for graphite/metrics
* externalize labeled ScriptContext to API
* add labels to NicerTimer
* remove partial packaging
* more progress on formatting for prom exposition format
* added metrics diagram
* resolve build issues with label params
* resolve build issues with label params
* prometheus export services
* added PromExpoFormat Tests for NBMetricMeter(Counting+Sampling) and NBMetricTimer(Counting)
* added test for Gauge Formatting
* added Gauge Formatting as well as Sampling values (count, stdev ...)
* added sketch for metrics labeling contexts
* add NBLabeledElement in all the places, retool calling paths to use it
* synchronize antlr versions after partial snyk change
* unbreak static initializer block after IntelliJ "fixed" it.
* engine-api - adapt to NBLabeledElement
* adapters-api - adapt to NBLabeledElement
* nb-api - adapt to NBLabeledElement
* engine-core - adapt to NBLabeledElement
* misc-adapters - adapt to NBLabeledElement
* streaming-adapters - adapt to NBLabeledElement
* add missing test
* initial implementation of a prom push reporter
* Resolve build issue with parseGlobalOptions
* replaced with PromPushReporter
* cleanup unused deps
* dependency removal for micrometer
* allow empty labels for tests
* change space.getName to space.getSpaceName
* cleanup poms
* graphite linearization now includes space element
* http adapter should only depend on adapters API
* http space does not create its own metric names
* import cleanups
* improved javadocs
* introduce component concepts

---------

Co-authored-by: Jonathan Shook <jshook@gmail.com>
Co-authored-by: Mike Yaacoub <mike.yaacoub@datastax.com>
2023-05-09 09:52:42 -05:00
Massimiliano Mirelli
ef92631595 Add flag to run tests for adapters-api and kafka / pulsar adapters 2023-03-07 15:13:47 +02:00
Massimiliano Mirelli
0a2ab1dcca Add scripts to build pulsar and kafka adapters and manually test against local pulsar cluster 2023-03-07 14:52:21 +02:00
Massimiliano Mirelli
8eb9498fa9 Add examples of simple testing pulsar workload 2023-03-07 14:51:23 +02:00
Massimiliano Mirelli
7d5ecf2f32 Enhance pulsar and kafka adapters documentation 2023-03-07 14:15:21 +02:00
Massimiliano Mirelli
971c5d3c6f Add sequence tracking number when seqTracking is enabled in producer 2023-03-07 14:09:34 +02:00
Massimiliano Mirelli
b333898249 Decouple e2e error metrics from adapter modules
Add classes implementing e2e error metrics to adapters-api module in `import io.nosqlbench.engine.api.metrics.ReceivedMessageSequenceTracker`
2023-03-07 14:06:15 +02:00
Massimiliano Mirelli
70d66719db Edit copyright year 2023-02-24 10:44:14 +02:00
Lari Hotari
5491790751 Optimize cache keys and fix races with caches in PulsarSpace 2023-02-07 08:10:45 +02:00
Lari Hotari
10da1699ed Optimize enum mappings and cache Avro schemas in PulsarAdapterUtil 2023-02-07 08:10:45 +02:00
Jonathan Shook
ff188d4860
actions-for-docs (#919)
* updated docs action with version state

* script updates

* misc doc system updates

* Update build.yml

remove versions logic, since zip contains version info now.

* Update build.yml

corrected git add command

* update version format

* updated docs site url
2023-01-13 11:57:08 -06:00
Jonathan Shook
08fb6ca905 make anchor links zola compatible 2022-12-22 23:54:57 -06:00
Jonathan Shook
972a6af654 remove duplicitous doc file 2022-12-22 23:54:36 -06:00
Mike Yaacoub
74bfbc691d Merge branch 'main' into my-NB5-I446 2022-12-19 16:58:50 -05:00
Mike Yaacoub
b63eba35c5 fixed pulsar.md for export-docs after merge 2022-12-19 16:16:50 -05:00
Mike Yaacoub
a06e1556f0 merged my-NB5-I446 with main 2022-12-19 15:57:58 -05:00
MikeYaacoubStax
e7705d13de NB5-I446 Modified export-docs command to export organized driver and binding docs, also removed gendocs command (VirtDataGenDocsApp now used for generating bindings in StringBuilders to pass to the zipstream)
added the export-docs build workflow job
Updated build.yml to fix path to nb5 in workflow
uploading/downloading docs artifacts in build workflow
Attempt to correct the path to the exported docs to upload them
Corrected exported-docs path
Corrected yml build workflow error
corrected naming of exported docs
2022-12-08 12:20:11 -05:00
yabinmeng
b3d3f59d40 Migrate NBS4J from NB4 to NB5 2022-12-06 17:24:52 -06:00
yabinmeng
c077218e8a Fix example yaml file producer and consumer parameter names 2022-11-21 21:05:26 -06:00
yabinmeng
f7c9e50f35 Add README file and adjust sample yaml files 2022-11-19 18:53:58 -06:00
yabinmeng
bd8aff9081 Added support for 1) subscription initial position and 2) regex subscription mode
Also fixed a consumer configuration loading issue with dlq/negAck/ackTimeout policy using Pulsar client's loadConf() API.

Addressed the review comments in PR #791; reverted the changes in ActivityMetrics and BasedOpDispenser
2022-11-18 18:22:04 -06:00
yabinmeng
0de7102884 Add support for
1) producer message compression
2) consumer DLT/negAck/ackTimeout policy

Add sample NB scenario yaml files
2022-11-17 18:47:37 -06:00
yabinmeng
12f8697e0e Complete the first draft of NB5 Pulsar adapter code, except the per-thread rate limiter. 2022-11-17 10:39:37 -06:00
yabinmeng
0c71696b15 Completed NB4 Pulsar driver code migration to NB5. But there are still a few lingering issues due to the difference of how NB4 vs NB5 driver/adapter interacts with the NB engine. 2022-11-16 15:37:26 -06:00
yabinmeng
ca6f2b052b Add PulsarAdapter specific metrics 2022-11-14 15:27:57 -06:00
yabinmeng
c309d200ac - Delete unsed code "driver-jms"
- NB5 Pulsar driver Admin functionality in place - create/delete tenant, namespace, and topic
2022-11-11 18:36:06 -06:00