Yabin Meng

There are two traditional ways of doing performance testing with&nbsp;<a href="https://pulsar.apache.org/">Apache Pulsar</a>. One is with the&nbsp;<a href="https://pulsar.apache.org/docs/en/performance-pulsar-perf/">pulsar-perf</a>&nbsp;command-line tool that&rsquo;s part of the Pulsar distribution. Another is through the&nbsp;<a href="http://openmessaging.cloud/docs/benchmarks/">OpenMessaging Benchmark (OMB)</a>&nbsp;tool &ndash; a generic testing framework for a variety of messaging platforms including Apache Pulsar and Kafka.
From a workload generation and performance testing perspective, both tools can be used to simulate high-throughput message publishing or consuming against one or multiple Pulsar topics. Pulsar-perf can also simulate other types of Pulsar workloads like Pulsar reader, WebSocket producer, and managed ledger.
Both tools, however, have limitations when it comes to simulating realistic production workloads.
<ul>
<li>Support for message keys is limited.&nbsp;In OMB, there&rsquo;s no easy way to specify message keys as part of the workload generation. On the other hand, pulsar-perf only supports key generation either as auto-increment or purely random </li>
<li>Both tools can only test Pulsar topics&nbsp;within a specific tenant and namespace </li>
<li>Neither tool supports message schema.&nbsp;They can only use the default binary format as the message payload type </li>
<li>Message payload&nbsp;is either randomly generated or statically assigned for each message </li>
<li>The OMB tool is only available on AWS public cloud platform. This may not represent the actual Pulsar production infrastructure environment </li>
<li>Not all tuning parameters are available&nbsp;to fine control the Pulsar client execution behavior</li>
</ul>
So there&rsquo;s a need for a new performance testing tool suitable for simulating real production workloads. This is why we created&nbsp;<a href="https://github.com/nosqlbench/nosqlbench">NoSQLBench Pulsar Driver</a>. Here&rsquo;s an introduction to the&nbsp;<a href="https://docs.nosqlbench.io/">NoSQLBench</a>&nbsp;tool and how the NoSQLBench Pulsar Driver simulates a realistic production Pulsar workload.
<h2>Introducing NoSQLBench</h2>
<a href="https://www.datastax.com/blog/nosqlbench">NoSQLBench</a>&nbsp;is a generic workload generation and performance testing tool for a distributed system. It started as an internal performance testing tool for&nbsp;<a href="https://www.datastax.com/cassandra">Apache Cassandra&reg;</a>&nbsp;at DataStax.&nbsp;
It gradually expanded to other distributed systems like Apache Pulsar. NoSQLBench was open-sourced in 2020 under Apache 2.0 license (APLv2).
Compared with other performance testing tools for a distributed system, NoSQLBench has some unique features and benefits.
<ul>
<li>Recipe-oriented procedural data generation through built-in function flows. Data can be both statically and dynamically bound</li>
<li>Deterministic and repeatable workload behavior</li>
<li>Modular protocol support via driver extensions for different types of distributed systems</li>
<li>Configuration (YAML) based testing scenario design and access pattern modeling using the target system&rsquo;s native approach</li>
</ul>
NoSQLBench workloads are defined in YAML configuration files. At its core, a NoSQLBench workload is composed of statements. Each statement can have its own parameters. A set of statement blocks can be grouped together under a common name and selected by tags.&nbsp;
Data used in each statement can either be statically or dynamically bound. Each binding represents how one particular piece of data is generated via a series of built-in functions.
The example YAML file below &ndash; hello_world.yaml &ndash; shows the basic structure of a NoSQLBench workload for Cassandra. Two statements that can be grouped together under a common phase, main. One for Cassandra write and another for Cassandra read.&nbsp;
Cassandra write/read ratio (4 vs. 1) is specified by the statement parameter ratio. The data for Cassandra write and read is dynamically bound according to the binding function flows.
<code>bindings:</code>
<code>&nbsp;cycle: Identity()</code>
<code>&nbsp;cyclename: NumberNameToString()</code>
<code>&nbsp;sample: Normal(100.0D, 10.0D)</code>
<code>&nbsp;randomish_cycle: HashRangeScaled()</code>
<code>blocks:</code>
<code>&nbsp;- tags:</code>
<code>phase: main</code>
<code>statements:</code>
<code>- insert-sample: |</code>
<code>insert into hello.world (cycle,name,sample)&nbsp;</code>
<code>values ({cycle},{cyclename},{sample});</code>
<code>ratio: 4</code>
<code>- read-sample: |</code>
<code>select * from hello.world where cycle={randomish_cycle}</code>
<code>ratio: 1</code>
To execute this workload, simply run the following NoSQLBench command to trigger one million Cassandra operations with 800K writes and 200K reads.
<code>nb run driver=cql yaml=hello-world.yaml tags=phase:main host=&lt;Cassandra_host_name&gt; cycles=1M</code>
The NoSQLBench tool has more features and capabilities than we can cover here. You can learn more in the&nbsp;<a href="http://docs.nosqlbench.io/">NoSQLBench documentation</a>&nbsp;and this&nbsp;<a href="https://jaxenter.com/nosqlbench-tutorial-170790.html">NoSQLBench tutorial</a>&nbsp;by Jonathan Shook.
<h2>NoSQLBench Pulsar Driver in detail</h2>
NoSQLBench provides a common framework for workload generation and performance testing. The actual workload execution against a particular distributed system is achieved via custom-built drivers. This is what&nbsp;<a href="https://docs.nosqlbench.io/docs/drivers/pulsar/">NoSQLBench Pulsar Driver</a>&nbsp;is designed to do for performance testing against Apache Pulsar.
At a high level, we can achieve the following testing objectives with the NoSQLBench Pulsar Driver that are hard or impossible to implement with other existing Apache Pulsar performance testing tools.
<ul>
<li>Concurrent message publishing&nbsp;and consuming for topics under multiple tenants and namespaces</li>
<li>Realistic production workload simulation&nbsp;with fine-controls of message keys, message properties, and message payloads</li>
<li>Complex Pulsar message schema types&nbsp;like&nbsp;<a href="https://avro.apache.org/docs/current/">Apache Avro</a></li>
<li>Track end-to-end message processing latency&nbsp;from message publishing to receipt</li>
<li>Detecting abnormal message processing errors&nbsp;like loss, duplication, and out-of-order</li>
<li>Easy integration of rich client-side metrics&nbsp;with Pulsar cluster metrics and a holistic view of the system performance behaviors</li>
</ul>
<h2>Deterministic workload definition</h2>
The Pulsar Driver defines Pulsar workloads in YAML files like in the generic NoSQLBench workload definition pattern. At a high level, a Pulsar workload YAML file has three main sections: bindings, params, and blocks. 
<ul>
<li>Bindings&nbsp;define how message data is actually generated. For this part, the Pulsar driver follows the generic NoSQLBench data binding principle</li>
<li>Params&nbsp;define the document-level configuration settings that may impact multiple or all Pulsar workload statements. More detail on this in the next section</li>
<li>Blocks&nbsp;define Pulsar workload details as NoSQLBench statement blocks. Depending on Pulsar workload type, the statement block content may be different</li>
</ul>
A template YAML file is like this:
<code>bindings:</code>
<code>&hellip; &hellip;</code>
<code>params:</code>
<code>&hellip; &hellip;</code>
<code>blocks:</code>
<code>- name: &lt;statement_block_1&gt;</code>
<code>tags:</code>
<code>phase: &hellip; &hellip;&nbsp;</code>
<code>statements:</code>
<code>- name: &lt;statement_name_1&gt;</code>
<code>&hellip; &hellip;&nbsp;</code>
<code>- name: &lt;statement_name_2&gt;</code>
<code>&hellip; &hellip;</code>
NoSQLBench Pulsar Driver currently supports the following types of Pulsar workload.
<ul>
<li>Create or delete Pulsar tenants, namespaces and topics</li>
<li>Message reading</li>
<li>Message publishing</li>
<li>Message batch publishing (only relevant when using Pulsar sync API)</li>
<li>Message consuming (also using topic list or pattern)</li>
</ul>
The definitions of these Pulsar workload types as NoSQLBench statement blocks are described in detail in the&nbsp;<a href="https://docs.nosqlbench.io/docs/drivers/pulsar/">NoSQLBench Pulsar Driver documentation</a>.
<h2>Complete workload behavior configuration</h2>
The behavior of the Pulsar workload execution is impacted by configuration settings that can be set at different levels described in detail below.
<ul>
<li>Global</li>
<li>Document</li>
<li>Statement</li>
</ul>
<h3>Global Level configuration</h3>
The NoSQLBench Pulsar Driver is built upon Pulsar&rsquo;s Java client API. With this API, various configuration settings can be used to fine-tune the behaviors for the following.
<ul>
<li><a href="http://pulsar.apache.org/docs/en/client-libraries-java/#client">Pulsar client connection</a></li>
<li><a href="http://pulsar.apache.org/docs/en/client-libraries-java/#producer">Pulsar producer</a></li>
<li><a href="http://pulsar.apache.org/docs/en/client-libraries-java/#configure-messages">Pulsar consumer</a></li>
<li><a href="http://pulsar.apache.org/docs/en/client-libraries-java/#configure-reader">Pulsar reader</a></li>
<li><a href="https://pulsar.apache.org/docs/en/schema-get-started/">Pulsar schema</a></li>
</ul>
The NoSQLBench Pulsar Driver supports all these configuration settings. You can define the desired configuration settings in a Java property file like config.properties with the right prefixes.&nbsp;
For example, for all client connection-related configuration settings, the prefix is &ldquo;client.&rdquo; For Pulsar producer-related configuration settings, the prefix is &ldquo;producer&rdquo;, and so on. To give you a better idea, here&rsquo;s an&nbsp;<a href="https://github.com/nosqlbench/nosqlbench/blob/main/driver-pulsar/src/main/resources/activities/config.properties">example file of global level configuration parameters</a>.
The settings under global level configuration will impact all NoSQLBench Pulsar workload definition YAML files. They&rsquo;re provided as a NoSQLBench runtime execution parameter, config, as shown below.
<code>nb run driver=pulsar tags=phase:producer cycles=1M web_url=http://localhost:8080 service_url=pulsar://localhost:6650 config=&lt;dir&gt;/config.properties yaml=&lt;dir&gt;/pulsar_nb.yaml</code>
<h3>Document and Statement Level configuration</h3>
Another set of configuration settings can only impact one particular Pulsar workload definition YAML file. Depending on the scope of their impacts in the file, these settings are further categorized as Document Level settings and Statement Level settings.
Document level configuration settings are placed under the &ldquo;params:&rdquo; section of the NoSQLBench workload definition YAML file. They impact multiple or all Pulsar workload types defined in the YAML file. There are currently four Document Level configuration settings for the NoSQLBench Pulsar Driver.
One Pulsar workload type may include one or more statements grouped under different statement blocks. Each statement may have its unique configuration settings. The settings under one statement have no impact on other statements.
You can find a detailed description of the Document Level and the Statement Level configuration settings in the&nbsp;<a href="http://docs.nosqlbench.io/">NoSQLBench Pulsar Driver documentation</a>.
<h2>Realistic message simulation</h2>
When we use the NoSQLBench Pulsar Driver for Pulsar message processing testing, we get a realistic message workload simulation as close to a production environment as possible. We&rsquo;ll use a simple oil and gas sensor IoT example to see how it works.
We&rsquo;re assuming that each message corresponds to a piece of sensor data containing the following information.
<ul>
<li>Message Key: The drilling bit identifier</li>
<li>Message Property: Various static properties of a drilling bit</li>
<li>Message Payload: Actual data collected by one sensor on a drilling bit</li>
</ul>
Simulating a message format in such detail isn&rsquo;t possible with any existing Pulsar performance testing tools. But NoSQLBench Pulsar Driver makes it simple. Below is an example of a NoSQLBench workload definition YAML file.
<code>bindings:</code>
<code>drill_id: ToUUID(); ToString();&nbsp;</code>
<code>prop1_val: AlphaNumericString(10);</code>
<code>prop2_val: AlphaNumericString(20);</code>
<code>sensor_id: ToUUID(); ToString();</code>
<code>sensor_type: HashedLineToString('sens_type_values.txt')</code>
<code>reading_time: ToDateTime();</code>
<code>reading_value: ToFloat(100);</code>
<code>params:</code>
<code>&hellip; &hellip;&nbsp;</code>
<code>blocks:&nbsp;</code>
<code>- name: producer-block</code>
<code>tags:</code>
<code>phase: producer</code>
<code>statements:</code>
<code>- name: s1</code>
<code>optype: msg-send</code>
<code>msg_key: "{drill_id}"</code>
<code>msg_property: |</code>
<code>{</code>
<code>"drill_prop1": "{prop1_val}",</code>
<code>"drill_prop2": "{prop2_val}"</code>
<code>}</code>
<code>msg_value: |</code>
<code>{</code>
<code>"SensorID": "{sensor_id}",</code>
<code>"SensorType": "{sensor_type}",</code>
<code>"ReadingTime": "{reading_time}",</code>
<code>"ReadingValue": {reading_value}</code>
<code>}</code>
The message key is a UUID string representing a specific drilling bit ID. The message property is defined by a JSON string that contains a list of key/value pairs representing various drilling bit properties. The message payload is also defined by a JSON string that follows an Avro schema definition.
<code>{</code>
<code>"type": "record",</code>
<code>"name": "IotSensor",</code>
<code>"namespace": "TestNS",</code>
<code>"fields" : [</code>
<code>{"name": "SensorID", "type": "string"},</code>
<code>{"name": "SensorType", "type": "string"},</code>
<code>{"name": "ReadingTime", "type": "string"},</code>
<code>{"name": "ReadingValue", "type": "float"}</code>
<code>]</code>
The Avro schema definition is made available to the Pulsar workload by the following global level configuration settings.
<code>schema.type=avro</code>
<code>schema.definition=file:///path/to/iot-example.avsc</code>
<h2>Authentic multi-tenancy message processing testing</h2>
Real multi-tenancy message processing testing is another big advantage of using NoSQLBench Pulsar Driver over Pulsar performance testing tools. We can get this by dynamical data binding to Pulsar topic names. For a bit more context.
<ul>
<li>Statically binding means&nbsp;the Pulsar topic name will be the same for all NoSQLBench execution cycles. Each cycle represents one particular operation like publishing a message. This is the single-topic testing we know from other Pulsar testing tools</li>
<li>Dynamically binding means&nbsp;the Pulsar topic name is different for each NoSQLBench execution cycle. The actual Pulsar topic name per cycle will follow the NoSQLBench data binding principle</li>
</ul>
In the next example, we create 100 Pulsar topics under 10 tenants with two namespaces per tenant and five topics per namespace. The Pulsar topic names that are generated follow the pattern of persistent://tnt[0~9]/ns[0~1]/t[0~4].
<code>bindings:</code>
<code>tenant: Mod(100); Div(10L); ToString(); Prefix("tnt")</code>
<code>namespace: Mod(10); Div(5L); ToString(); Prefix("ns")</code>
<code>core_topic_name: Mod(5); ToString(); Prefix("t")</code>
<code>params:</code>
<code>topic_uri: "persistent://{tenant}/{namespace}/{core_topic_name}"</code>
When running a NoSQLBench Pulsar message publishing workload with one million execution cycles (publishing one million messages), NoSQLBench will make sure that the one million messages are evenly distributed among 100 topics under 20 namespaces of 10 tenants.
<h2>Conclusion</h2>
We&rsquo;ve explored a powerful, open-source performance testing utility for Apache Pulsar:&nbsp;<a href="https://github.com/nosqlbench/nosqlbench/tree/main/driver-pulsar">NoSQLBench Pulsar Driver</a>. It has distinct advantages compared with other Pulsar performance testing utilities. It enables deterministic, realistic, and production-like Pulsar workload simulation. Execution is easy and flexible.
Another distinguishing feature is to detect abnormal Pulsar message processing errors like message loss, message duplication, and message out-of-order. The deterministic nature of this utility&rsquo;s workload generation and execution makes it easy to capture, analyze and debug these errors. This feature has helped our Pulsar engineers uncover a few Apache Pulsar bugs in some rare, racing-related situations. It has other useful features that we haven&rsquo;t covered here.&nbsp;
We made it open-source &ndash; and wrote this post &ndash; in the hopes that the Apache Pulsar community will benefit from the powers of NoSQLBench Pulsar Driver. We look forward to seeing how it helps your performance testing, messaging model design, sizing analysis, and cluster deployment evaluation for applications and systems based on Apache Pulsar.
Follow&nbsp;<a href="https://dtsx.io/3s2oooG">DataStax on Medium</a>&nbsp;for more posts on all things Pulsar, Cassandra, streaming, Kubernetes, and more. To join a buzzing community of developers from around the world and stay in the data loop, follow DataStaxDevs on&nbsp;<a href="https://dtsx.io/3LyCN3y">Twitter</a>&nbsp;and&nbsp;<a href="https://dtsx.io/3OEG5Et">LinkedIn</a>.
<h2>Resources</h2>
<ol>
<li><a href="https://www.datastax.com/blog/nosqlbench">NoSQLBench | DataStax</a></li>
<li><a href="https://github.com/nosqlbench/nosqlbench/tree/main/driver-pulsar">GitHub: NoSQLBench Pulsar Driver</a></li>
<li><a href="http://docs.nosqlbench.io/">NoSQLBench Documentation</a></li>
<li><a href="https://jaxenter.com/nosqlbench-tutorial-170790.html">Tutorial &mdash; Getting Started with NoSQLBench</a></li>
<li><a href="https://dtsx.io/3EYSfmZ">NoSQLBench Discord Community</a></li>
</ol>

Apache Pulsar Performance Testing with NoSQLBench

Yabin MengDataStax

Discover more

Share

Share

Introducing NoSQLBench

NoSQLBench Pulsar Driver in detail

Deterministic workload definition

Complete workload behavior configuration

Global Level configuration

Document and Statement Level configuration

Realistic message simulation

Authentic multi-tenancy message processing testing

Conclusion

Resources

More Technology

How Winweb Built its AI Assistant with DataStax Astra DB and LangChain

Vercel + Astra DB: Get Data into Your GenAI Apps Fast

Simplifying Agent Development with Astra DB Connector for Vertex AI Search

Making Astra DB easier for MongoDB developers

One-stop Data API for Production GenAI