MQTT load test settings

When creating new MQTT load test from MQTTBox app, there are wide range of load test settings ranging from connection protocol, number of message to publish or subscribe, number of parallel instances to run, rate at which messages needs to published or subscribed and many more. This document explains each load test attributes in details to better understand load test tool and use it in right way to test your MQTT load.

Load Test Name: Name to identify MQTT load testcase and display on dashboard. It can be any string value.
e.g: load_test_1
Protocol: Network protocol used to connect with MQTT broker.
MQTTBox supports TCP, SSL/TLS, MQTT, MQTTS, WebSockets(WS) and Secure WebSockets(WSS). Depending on the platform you are on, all Protocol may not be supported because of platform limitations.
Please check here for list of MQTTBox features supported by each platform.
Host: MQTT host to connect. Make sure you specify right host and port number depending on MQTT connection protocol you selected. MQTT client may not get connected if you mention wrong port number or interchange port numbers.
e.g: test.mosquitto.org:8080
Load Test Type: Specifies whether you want to load test MQTT publishing or MQTT subscribing.
# of messages to publish: This attribute is for publisher load test only. Total number of messages to publish in given runtime to topic by each instance.
e.g: If # of messages to publish = 20 and runtime = 5, total of 20 messages are published in 5 seconds per instance. Its approximately (20/5) = 4 messages/second are published for given topic per instance.
# of messages to subscribe: This attribute is for subscriber load test only. Total number of messages to subscribe to topic by each instance. Load test will wait until specified number of messages are received or time out value is reached.
e.g: If # of messages to subscribe = 20 and Timeout = 60, testcase will wait until 20 messages are received or 60 seconds is passed. If 60 seconds are passed, testcase will fail with timeout error
Runtime (in seconds): This attribute is for publisher load test only. Total Runtime of testcase to publish messages per instance. Depending on runtime specified, MQTTBox calculates number of messages to publish/second. (# of mesages to publish / Runtime). MQTTBox will always tries to publish number of calculated messages per second however it may take more time than calculated if your OS has limited resources.
e.g: If runtime=100 and # of messages to publish = 20000, MQTTBox tries to publish 200 messages per second for given topic (20000/100) per instance
Timeout (in seconds): Testcase timeout. For some reason, if testcase is blocked or MQTTBox is waiting for broker connection to establish, we don't want testcase to wait forever. If Timeout is reached, regardless of testcase state (running/hung/ etc), testcase is terminated. Make sure timeout is always way greater than Runtime.
In case of subscriber load test, if "# of messages to subscribe" are not received within timeout interval, testcase is failed with timeout error.
In case of publisher load test, if "# of messages to publish" are still being sent after timeout interval is reached, testcase is failed with timeout error and no further messages are published.
e.g: If Timeout=60 and if load is still running after 60 seconds, load test will be terminated and marked as Timeout error for that instance.
# of instances: How many separate instances to spin up to run in parallel with same settings.
1. If # of instances = 6 and # of messages to publish =10, total of 6 instances are running with each instance publishing 10 messages (total 6*10 = 60 messages are published together in given runtime).
2. If # of instances = 6 and # of messages to subscribe =10, then each instance will wait until it receives 10 messages or timed out.
Topic: Topic name to which all messages needs to be published or subscribed.
For subscriber load testing, you can specify Single Level(+) and Multilevel(#) subscription to topics.
QoS: Set the mqtt qos for messages to published or subscribed. Default to 1.
Payload: This attribute is for publisher load test only. Payload message to publish for given topic. You can specify multiple payloads and each payload is published serially but order is not guaranteed. Each message is counted towards # of messages to published.
If # of messages to publish = 10 and 1 payload is mentioned, same payload is published 10 times.
If # of messages to publish = 10 and 4 payloads are mentioned, MQTTBox tries to send in following order - 1,2,3,4,1,2,3,4,1,2 however order is not guaranteed.
You can add/delete upto 100 payloads.

MQTT load test metrics

Once load test is complete, MQTTBox displays test metrics and shows how your test was performed. This document explains each metric attribute represents. You can view load test logs for each instance on dashboard. Log has in detail messages with timestamp and shows when the connection was established, number of times connection was dropped and many more. Each instance has following metric attributes.

Name: Name of the instance. Name is auto generated depending on number of instances you mentioned while creating load test.
e.g: Instance 1
Status: Execution status of testcase. Can be in any one of these states - 'In Progress', 'Done', 'Failed', 'Stopped', 'Error' and 'Timeout'.
Published Time: This attribute is for publisher load test only. Total time taken to publish messages. This doesn't include time for QoS responses. Its just time taken to fire all messages to given topic
Published Messages: This attribute is for publisher load test only. Total number of messages published to topic by respective instance
QoS Response: This attribute is for publisher load test only. Number of QoS responses received after publishing messages. If you have mentioned QoS=0, this will always be equal to message sent as load test will not wait for QoS responses.
QoS Time: This attribute is for publisher load test only. Total time taken to publish messages and receive QoS responses for all message published.
Subscribed Time: This attribute is for subscriber load test only. Total time taken to receive messages for given topic specified by "Messages Received" attribute.
Messages Received: This attribute is for subscriber load test only. Total number of messages received by instance from given topic.

Graph metrics

Once load test is completed, you can view graphs by clicking on "View Graph" button at top of dashboard page. Graph represents number of messages published or subscribed over time for all instances together or you can select specific instance to view.

You can also change sampling rate to plot graph. Sampling rate represents number of data points you want to take in given time. More the sampling rate you select, more data points are created in given time which represents more accurate data at given time. However selecting more sampling rate is not always useful like when number of messages are less and runtime is high. General rule of thumb is when more data is published or subscribed in given second, more sampling rate is useful.

e.g: Say testcase was run for 10 seconds publishing 1000 messages. If you select sampling rate of 100, 10 seconds time interval is divided into 100 points and each point represents average number of messages published between current and previous data point.

Data metrics

Once load test is complete, you can view all data published or received in table format. Click on "View Data" button at top of dashboard page. Each data point has following information.

Time: Time when data was published or received.
Message Id: Message Id generated by MQTT broker for each message. If QoS is 0, Message Id is not generated by MQTT broker.
Topic: Topic from which message was published or subscribed.
QoS: QoS level for each message
Instance: Instance from which payload was published or subscribed.
Payload: Payload published or received from given topic.