A New Fate for the Eclipse Paho “Test” Broker?

If you’re familiar with the Eclipse Paho test material, you’ll know that I wrote an MQTT 3.1.1 broker in Python with the purpose of providing a benchmark or oracle against which other implementations could be compared. Last year, I added support for the new MQTT V5.0 definition, so that client implementations of V5, including my own, could be easily tested. It has been used by Paho colleagues of mine, James Sutton and Allan Stockdill-Mander to develop MQTT V5 Java and Golang client libraries.

Then, I added the missing TLS support, so that it could satisfy testing requirements for secured connections. Adding TLS support in Python turned out to be really easy, much easier than I expected. I know how hard it is to add OpenSSL support in C, and I was prepared for at least some proportion of that effort, but there was none. I enabled multiple MQTT listeners to be configured, a la RSMB/Mosquitto. That meant the broker was comprehensive enough to be used for automated testing of existing client implementations. I used it to replace a remote Mosquitto broker in the automated Paho C client testing for Windows using AppVeyor, as installing Mosquitto on Windows is a little tricky, and it works fine in that capacity. All I had to do in the AppVeyor configuration file was to add these lines:

- cmd: git clone https://github.com/eclipse/paho.mqtt.testing.git
- cmd: cd paho.mqtt.testing\interoperability
- ps: Start-Process C:\Python36\python -ArgumentList 'startbroker.py -c client_testing.conf'

to get the broker running locally. WebSockets support was already included, so it means that each MQTT listener supports both 3.1.1 and V5, WebSockets and TLS. The 3.1.1 and V5 clients can communicate with each other (although this work is not complete and tested yet), both through normal publish commands and retained messages.

In a recent project at work I had added persistence to a Python project by using the ZODB object database, and that had turned out to be really easy too. These experiences got me to thinking that it should be easy to add MQTT-SN support as well, and create a broker with the capabilities that I had envisaged for RSMB before it was superseded by Mosquitto. Roger Light, the author of Mosquitto, has intended to add MQTT-SN support for a long time, but wanted to restructure the internals first. Due to other commitments, and some licensing obstacles, this has not yet come about, and with the advent of MQTT V5 which will have a higher priority, looks even further out.

A broker written in Python will not be as efficient as Mosquitto, nor is it particularly scalable as currently written because it uses Python’s threading model which does not take advantage of multiple cores. But as it is so easy to add new capabilities, it could well be one of the most complete MQTT implementations, soonest. In this way it would be complementary to Mosquitto.

I have now experimented with adding MQTT-SN support, a framework for bridges, and an HTTP listener which can provide APIs to query and update the broker state and configuration. We can add APIs to allow queries to return information about the currently connected clients, their subscriptions and messages for instance. The set of capabilities I envisage looks like this:

  • TCP listener supporting WebSockets, TLS, MQTT 3.1.1 and 5.0
  • UDP listener supporting MQTT-SN with DTLS, multicast
  • other potential MQTT-SN listeners: BLE, ZigBee
  • HTTP listener for state/configuration APIs. The broker behaviour for test purposes can potentially be reconfigured dynamically
  • TCP bridge capable supporting MQTT 3.1.1 and 5.0, WebSockets and TLS
  • UDP bridge supporting MQTT-SN with DTLS, multicast
  • HTTP bridge supporting webhooks?
  • dynamic bridge connections – adding/deleting bridges
  • variety of persistence options including ZODB

I think it’s obvious to state that with these capabilities it could serve as an MQTT-SN gateway for services that support MQTT, as a conversion mechanism between MQTT V3.1.1 and V5, and a dynamic flexible component of an MQTT network. I would still keep a focus on enabling MQTT testing, including for test suite generation. A lot of this work could be completed in the first half of 2018, other commitments depending.

This brings me on to my main motivation for writing this blog post. Perhaps this broker should have a life of its own, separate to Paho, as an Eclipse IoT project. It could quite well do the job as part of Paho, conversely it could have a higher profile and encourage more community participation if stood on its own. So, if you have an interest in seeing this broker as an Eclipse IoT standalone project, from the point of view of using it, or also contributing to it, please let me know – thanks.