Category: Model Based Testing

Using the Eclipse Paho “Test” Broker to Help Test MQTT

I may not have finished all my goals for the test material in the Eclipse Paho project, as outlined in this blog post of mine, but some components are still useful in their current state.

Recently my IBM and Paho colleague, James Sutton, needed to check the behaviour of the Java and Android clients when receiving an error code in response to an MQTT subscribe request. This error code, returned in the MQTT suback packet, was introduced in the last, and current, version of MQTT, 3.1.1. It takes the form of an 0x80 value in the granted QoS (Quality of Service) field, for which only 0, 1 or 2 are valid values – the set of integers which QoS can take in MQTT.

Now you could fire up a broker like Eclipse Mosquitto and configure it to disallow a subscription to a certain topic. If you know the broker well enough, this may be quick and easy. It’s possible you might have to do some fishing around, and figure out whether that broker is actually returning 0x80 as you wanted.

So James turned to the Paho test broker (startbroker.py in this repo.) (You must use Python 3 to run it, not Python 2). As it stands, this broker will return 0x80 if you try to subscribe to the topic “test/nosubscribe” (see line 322 in MQTTBrokers.py). That’s pretty easy. If you checked out that file, you will notice another two topics: “test/QoS 1 only” and “test/QoS 0 only”, which will return granted QoSs of a maximum of 1 and 0 respectively. These behaviours can be hard to elicit out of a standard MQTT broker.

Ultimately I guess I should make the specific topics which these responses are attached to configurable, but they aren’t right now.

There are some other characteristics of this broker, which single it out as a “test” broker rather than a product:

  • the goal of the coding is clarity rather than performance. You can tell me whether I succeeded
  • there is no persistence: if you want to simulate stopping and restarting a broker, this broker can just remain running, and disconnect all clients
  • MQTT specification conformance statements are embedded in the code, so that when a test suite is run against the broker, it can tell you which statements were encountered, and which weren’t.
  • the broker has parameters to choose behaviours which can vary but still conform to the MQTT specification:
    1. whether to publish QoS 2 messages on PUBREL or not
    2. whether multiple matching subscriptions result in one publication, or more than one
    3. whether queued QoS 0 messages are dropped if the client is disconnected, or not
    4. whether zero_length_clientids are allowed

    it’s possible more might be added in the future.

The main missing feature of this broker is TLS – but it does have WebSocket support. TLS is on my to do list, as will be updates for the next version of MQTT, 5, which we are working on, and is tentatively scheduled for completion next year.

As outlined in the blog post, this broker is meant to help with broker testing as well, as an oracle.

For some more information, see the Eclipse Paho website for the test tools.

You can use issues on this project to ask for new features or identify bugs, or pull requests to offer your own contributions.

More Rigorous Testing for MQTT Servers

I’ve had a sustained interest in a rigorous approach to test design ever since I studied formal methods in the 1990s. This is reflected in the domain name of this blog. [1]All the test material referred to in this post is in the Eclipse Paho project.

Testing both client libraries and servers for adherence to the MQTT specification seems a natural candidate. Any form of rigorous testing will need more investment than a few manual experiments, but the payoff is potentially large (unless you really like a boring, repetitive life). The MQTT standardization process is nearly complete. A major goal of any standardization is to encourage implementations, so there will be plenty of opportunity for a rigorous test suite to recoup the investment made in it.

If a test suite is a haphazard collection of acquired test programs, it tends to suffer from gaps in coverage and a lack of consistent approach and motivation. When the object of that test suite changes, the suite will be hard to maintain because you have to work out the intent of each of the component test programs. Even explaining the exact reasoning behind an individual test program can be hard. With the approach outlined here I hope to make the goals and attainments of the test suite much more transparent, and maintenance easier and more logical.

I start with the MQTT 3.1.1 Specification. The standardized MQTT specification is a great improvement over the previous version because it is has been argued over intensively, with the (planned for) result that many of the ambiguities have been resolved. Each identified conformance statement has been numbered and listed in Appendix B. I use that list as a starting point for measuring test coverage of the complete MQTT specification. [2]It has been pointed out to me, by Peter Niblett of IBM, that this list has been constructed by fallible human beings, and thus itself is not rigorous.  I knew that already, but you have to start … Continue reading

An example statement from the table:

Normative Statement Number Normative Statement
[MQTT-1.5.3-1] The character data in a UTF-8 encoded string MUST be well-formed UTF-8 as defined by the Unicode specification [Unicode] and restated in RFC 3629 [RFC3629]. In particular this data MUST NOT include encodings of code points between U+D800 and U+DFFF. If a Server or Client receives a Control Packet containing ill-formed UTF-8 it MUST close the Network Connection.

What we need first is a model of the expected behaviour of an MQTT implementation, a test oracle. I had previously written an MQTT server in Python, for test purposes, so I updated it for MQTT 3.1.1. I also seeded it with log entries for conformance statements from the specification at appropriate points [3]Look for logger.info("[MQTT … statements in this example. During the execution of this server, any conformance statements encountered are logged. At the end of the test, we can measure the number of conformance statements that that test invoked in the server. We can assess a test suite both by the conformance statements invoked, and by those which are not.

Next we need a model of the test inputs, the MQTT packets that we want to send to the model MQTT server.  We wouldn’t need this model if we could exercise the whole input space without worrying, but for any non-trivial software, this space is so vast that we need to focus.  Even a computer needs to focus.  The first model I’ve created is intended to cover the basic MQTT test scenarios. It responds to QoS 2 publish flows correctly for instance. In the basic scenarios, we want to make sure that the “golden paths” are executed, and some of the error cases. Other input models will be needed for complete coverage — when a large message takes longer to send than the keep alive interval, for instance.

With the Python script suite_generate.py the input model is used to generate a sequence of inputs which are sent to the model MQTT server.  The outputs from the model MQTT server are recorded along with the inputs, to result in a self-checking test case.  The exact same sequence of inputs are not used more than once in the same test. The resulting test case can be used directly against a real MQTT server with the run_test.py. There are some ‘little’ complications — MQTT packet IDs do not have to be the same on each run, so they have to be remapped. But in general we replay the same recorded scenario against the MQTT server we want to test.

Currently, the generated test suites reach about 55% coverage of the conformance statements. I have to work on this input model, and maybe some others to increase that coverage to as near 100% as I can manage.   Another complication is that some behaviours of MQTT are optional or can vary, so we have to be able to generate test suites for each of these variations. I aim to be working on these and other items in the upcoming weeks and months before EclipseCon NA next year.

Some more information is available on the Paho wiki. I hope this post is helpful in explaining my intentions. Please do ask questions. I expect at least two other follow on posts:

  • More Rigorous Testing for MQTT Clients
  • A “model” implementation of an MQTT server

so this won’t be the last word on the subject.

References

References
↑1 All the test material referred to in this post is in the Eclipse Paho project.
↑2 It has been pointed out to me, by Peter Niblett of IBM, that this list has been constructed by fallible human beings, and thus itself is not rigorous.  I knew that already, but you have to start from somewhere, and this is better than any other specification I have previously tested against.
↑3 Look for logger.info("[MQTT … statements in this example.