Author: Ian Craggs

I’m almost at the end of implementing the changes to the Eclipse Paho C client that I described in an earlier post to support MQTT 5.0. I haven’t quite finished off the release yet, so I thought I’d allow a little more time for anyone to comment. I had to compromise in my naming of the APIs for MQTT 5.0 support; none of the proposals I created were perfect, so I’m particularly interested in any feedback in that area.

MQTT 5.0 is not the only change for this release of the C client, but it is the biggest. You can find the complete list in the milestone. Another major addition is WebSocket support, thanks to the contribution of Keith Holman.

Another improvement I’ve been able to make is the update of the command line utilities. Back in 2010, I called them stdinpub and stdoutsub, the idea being that they could be used in Unix pipe combinations. Stdinpub reads data from stdin and publishes the results to an MQTT server; stdoutsub subscribes to an MQTT server and sends the data to stdout. I hadn’t really touched these programs in the meantime except to change the names to paho_c_pub and paho_c_sub. I always had intended to add TLS and other functionality to make them comparable to the Eclipse Mosquitto utilities mosquitto_pub and mosquitto_sub.

Now I’ve actually done that (small cheer!). Along with TLS I’ve added support for WebSockets and of course, MQTT 5.0. In each case, I have followed the Mosquitto utilities options where possible, so that it should be easy to switch from one to the other. There are actually two versions of each, one for each style of Paho C library, MQTTClient and MQTTAsync:

• paho_c_pub – MQTTAsync C publisher
• paho_c_sub – MQTTAsync C subscriber
• paho_cs_pub – MQTTClient C publisher
• paho_cs_sub – MQTTClient C subscriber

As well as being a useful way to experiment with V5 support, they also serve as examples on how to use the various APIs, for instance, how to write an application which combines support for all MQTT versions. You can try the following examples of using the command line utilities against the Paho test broker which supports all the combinations needed: WebSockets, MQTT 5.0 and TLS. To to start it up:

git clone git@github.com:eclipse/paho.mqtt.testing.git
cd git/paho.mqtt.testing/interoperability
python3 startbroker.py -c localhost_testing.conf

Typing paho_c_pub on the command line will display the option help:

-t (--topic)        : MQTT topic to publish to
-c (--connection)   : connection string, overrides host/port e.g wss://hostname:port/ws.  Use this option rather than host/port to connect with TLS and/or web sockets. No default.
-h (--host)         : host to connect to.  Default is localhost.
-p (--port)         : network port to connect to. Default is 1883.
-q (--qos)          : MQTT QoS to publish with (0, 1 or 2). Default is 0.
-V (--MQTTversion)  : MQTT version (31, 311, or 5).  Default is 311.
--quiet             : do not print error messages.
--trace             : print internal trace ("error", "min", "max" or "protocol").
-r (--retained)     : use MQTT retain option.  Default is off.
-n (--null-message) : send 0-length message.
-m (--message)      : the payload to send.
-f (--filename)     : use the contents of the named file as the payload.
-i (--clientid)     : MQTT client id. Default is paho-c-pub.
-u (--username)     : MQTT username. No default.
-P (--password)     : MQTT password. No default.
-k (--keepalive)    : MQTT keepalive timeout value. Default is 10 seconds.
--delimiter         : delimiter string.  Default is \n.
--maxdatalen        : maximum length of data to read when publishing strings (default is 100)
--message-expiry    : MQTT 5 only.  Sets the message expiry property in seconds.
--user-property     : MQTT 5 only.  Sets a user property.
--will-topic        : will topic on connect.  No default.
--will-payload      : will message.  If the will topic is set, but not payload, a null message will be set.
--will-retain       : set the retained flag on the will message.  The default is off.
--will-qos          : the will message QoS.  The default is 0.
--cafile            : a filename for the TLS truststore.
--capath            : a directory name containing TLS trusted server certificates.
--cert              : a filename for the TLS keystore containing client certificates.
--key               : client private key file.
--keypass           : password for the client private key file.
--ciphers           : the list of cipher suites that the client will present to the server during the TLS handshake.
--insecure          : don't check that the server certificate common name matches the hostname.

Here are some examples of use. To publish to the test broker on localhost over WebSockets using MQTT 5.0:

paho_c_pub -t topic --connection ws://localhost:1883 -V 5

To subscribe to the test broker on localhost over WebSockets with TLS server authentication using MQTT 3.1.1 by default:

paho_c_sub -t topic --connection wss://localhost:18885 --cafile test/ssl/test-root-ca.crt

And with mutual authentication without WebSockets:

paho_c_pub x --connection ssl://localhost:18884 --cafile test/ssl/test-root-ca.crt --cert test/ssl/client.pem --trace protocol

The –trace option displays various levels of trace, for debugging. A particularly useful level is ‘protocol’, which displays the MQTT and OpenSSL exchanges:

paho_c_pub x --connection ws://localhost:1883 -V 5 --trace protocol
Trace : 3, =========================================================
Trace : 3,                    Trace Output
Trace : 3, Product name: Eclipse Paho Asynchronous MQTT C Client Library
Trace : 3, Version: 1.2.1
Trace : 3, Build level: 2018-08-06T19:00:57Z
Trace : 3, OpenSSL version: OpenSSL 1.0.2o  27 Mar 2018
Trace : 3, OpenSSL flags: compiler: clang -I. -I.. -I../include  -fPIC -fno-common -DOPENSSL_PIC -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -arch x86_64 -O3 -DL_ENDIAN -Wall -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM -DECP_NISTZ256_ASM
Trace : 3, OpenSSL build timestamp: built on: reproducible build, date unspecified
Trace : 3, OpenSSL platform: platform: darwin64-x86_64-cc
Trace : 3, OpenSSL directory: OPENSSLDIR: "/usr/local/etc/openssl"
Trace : 3, =========================================================
Trace : 4, 20180809 144216.424 Connecting to serverURI localhost:1883 with MQTT version 5
Trace : 4, 20180809 144216.525 WebSocket connection upgraded
Trace : 4, 20180809 144216.525 5 paho-c-pub -> CONNECT cleansession: 0 (0)
Trace : 4, 20180809 144216.528 5 paho-c-pub <- CONNACK rc: 0
dfsdf
Trace : 4, 20180809 144259.493 5 paho-c-pub -> PUBLISH qos: 0 retained: 0 (0)

If using MQTT 5.0, you can attach user and message expiry properties to the sent message:

paho_c_pub -t topic --connection ws://localhost:1883 -V 5 --user-property name value --message-expiry 55

when received can be displayed with the –verbose option:

paho_c_sub x -V 5 -v
URL is localhost:1883
Subscribing to topic x with client paho-c-sub at QoS 0
13 x	dadasd
Property name MESSAGE_EXPIRY_INTERVAL value 55
Property name USER_PROPERTY key name value value

These utilities now have man pages too, as does the API itself, so I’ll have to make sure those are installed appropriately.

I gave a talk about MQTT V5 today at EclipseCon France, and at the end I gave the usual spiel about ways that people could help. After the talk, Cyrille Francois tweeted a great pictorial summary of the talk, as he did for other talks he attended. (Thanks Cyrille!) As part of the title of the tweet was the phrase “we need help”.

I’d like to add some more context to this. It’s of course true that we can always use more help. It’s also true that we are getting a lot of help already: issues, PRs, comments and feedback of all sorts. For example, one recent contribution to the Paho C client adds WebSocket support – thanks Keith! Paho is one of the most active Eclipse IoT projects as shown by this table extracted from an Eclipse IoT Metrics report:

Paho is also one of the most popular, in website visits:

although that metric is not definitive by any means, nor is this one:

as people obtain software from all the projects by many different means, not just downloads, but I don’t mind showing it given where Paho lies on it! At the time of writing, the Paho components have been starred 4108 times in total (up from 1766 last year) and users have forked the code 1969 times (up from 967). The number of committers now stands at 13, and the number of code authors is at least 88 (up from 50).

Paho is composed of distinct projects; client libraries and utilities, so those 13 committers are spread thinly across them. It’s an Eclipse convention that one committer doesn’t interfere in another’s repository without permission. Even without that, each committer is generally responsible for one or two specific components.

So I think we could do with some more committers, on just about all the Paho components. There are some that are in more need than others. The JavaScript client doesn’t really have a current owner, James Sutton and I step in when we can and as needed. We plan to update it for MQTT 5.0 later this year. The .Net client hasn’t been updated for a while and we’ve asked for help. All the other components could be helped by having more than one committer responsible.  Benjamin Cabé recently asked what would happen to the Paho C client if I was unable to continue (for whatever reason!) for instance.

When I became project lead I read the Eclipse documents quite thoroughly to learn what was expected.  About new committers, this is written:

Contributors who have the trust of the project’s committers can, through election, be promoted committer for that project. The breadth of a committer’s influence corresponds to the breadth of their contribution. A development team’s contributors and committers may (and should) come from a diverse set of organizations. A committer gains voting rights allowing them to affect the future of the project.

Becoming a committer is a privilege that is earned by contributing and showing discipline and good judgment. It is a responsibility that should be neither given nor taken lightly, nor is it a right based on employment by an Eclipse member company or any company employing existing committers.

I have taken these words quite seriously, maybe too seriously.  Should I lighten up a bit?

If anyone would like to become a committer then you definitely let me know and we can start talking.  Whether that’s for a particular component, or a wider remit such as helping with the website, or release process, it isn’t just coding.  There’s process development, design, documentation, tests for continuous integration, release packaging and distribution for example.

Of course you don’t need to be a committer to help by opening issues, responding to issues you didn’t open, raising PRs, or answering questions on mailing lists or forums.  If you haven’t done any of that already, you should start there.  There are many reasons to contribute to open source projects too.  Here’s what looks like a pretty helpful guide.

At EclipseCon today, Simon Phipps gave a great talk about Open Source, where it came from, where it’s going and why you should think about contributing if you use open source projects.  I expect that the slides will be up soon and you should take a look.  Seeing such talks in person is of course one of the benefits of attending conferences such as EclipseCon – you should do that too!

I’ve started to write the new MQTT version 5.0 support for the Eclipse Paho C clients. As I’ve reached the stage where some very simple tests are running, I thought this was a good time to try to elicit some feedback. I’ve had to make some compromises in the API, balancing the minimization of disruption to application programs with the natural incorporation the new MQTT 5.0 capabilities.

If you’ve used the main Paho C client, you’ll know it comes in two flavours:

MQTTClient

The synchronous version – intended to be easy to get started. Most calls block, waiting for a response from the server. This was the first API, and the only one when I created it. I modelled it after the existing Java API for consistency.

MQTTAsync

The asynchronous version – (almost) entirely non-blocking. Responses are reported in callback functions. This was a response to the need to run in asynchronous environments such as iOS and Windows.

The mqttv5 branch in the Github repo contains all the MQTT V5 updates to date. The test15 and test45 test programs have the first tests for MQTTClient and MQTTAsync libraries respectively. (At the moment, only test1 in each file has been migrated to MQTT V5, so only look at those).

MQTTClient for V5

First we have to create the client object with MQTTClient_create – this stays the same.

rc = MQTTClient_create(&client, options.connection,
 "single_threaded_test", MQTTCLIENT_PERSISTENCE_DEFAULT, NULL);

Next we must connect to the server.  As outlined in my previous post the main difference in MQTT V5 packets is the addition of the properties section, and for acknowledgements and disconnect, a reason code byte. Properties exist on both requests and their acknowledgements.

The connect call thus adds a properties object, for the connect request itself, plus the will message, and as part of the response. The MQTT V3 code:

MQTTClient_connectOptions conn_opts = MQTTClient_connectOptions_initializer;

int rc = MQTTClient_connect(client, &conn_opts))

becomes:

MQTTClient_connectOptions opts = MQTTClient_connectOptions_initializer;
MQTTProperties props = MQTTProperties_initializer;
MQTTProperties willProps = MQTTProperties_initializer;
MQTTResponse response = {SUCCESS, NULL};

response = MQTTClient_connect5(c, &opts, &props, &willProps);

The property and will property fields can be NULL if not required. The properties structure is manipulated with the add, free and copy functions. To add properties:

property.identifier = SESSION_EXPIRY_INTERVAL;
property.value.integer4 = 30;
MQTTProperties_add(&props, &property);

and

property.identifier = USER_PROPERTY;
property.value.data.data = "test user property";
property.value.data.len = strlen(property.value.data.data);
property.value.value.data = "test user property value";
property.value.value.len = strlen(property.value.value.data);
MQTTProperties_add(&props, &property);

Memory is allocated in the process of adding a property to the properties structure, so when you’ve finished with it this must be freed:

MQTTProperties_free(response.properties);

The response structure looks like this:

typedef struct MQTTResponse
{
  enum MQTTReasonCodes reasonCode;
  MQTTProperties* properties; /* optional */
} MQTTResponse;

I’m not particularly happy with the name of the function, MQTTClient_connect5, but it does have the benefit of being short with a minimal change of name. Other options I considered were:

• MQTT5Client_connect
• MQTTClient_connectWithProperties
• MQTTClient_connectExt

I quite like the “connectWithProperties” option, but it didn’t get much approval from the audience I’ve questioned so far — it also doesn’t reflect all the parameters required.

The other MQTT functions have similar changes. An MQTTResponse return structure instead of a single int, and an extra input parameter for properties. For the publishMessage function, the properties are added to the MQTTClient_message structure:

MQTTClient_message pubmsg = MQTTClient_message_initializer;

property.identifier = USER_PROPERTY;
property.value.data.data = "test user property";
property.value.data.len = strlen(property.value.data.data);
property.value.value.data = "test user property value";
property.value.value.len = strlen(property.value.value.data);
MQTTProperties_add(&pubmsg.properties, &property);

response = MQTTClient_publishMessage5(c, test_topic, &pubmsg, &dt);

And for the subscribe function, there are added subscribe-specific options:

MQTTSubscribe_options subopts = MQTTSubscribe_options_initializer;

subopts.retainAsPublished = 1;
subopts.noLocal = 0;
subopts.retainHandling = 0; /* 0, 1 or 2*/

response = MQTTClient_subscribe5(client, test_topic, subsqos, &subopts, &props);

Finally the disconnect function has both input properties and reason code:

enum MQTTReasonCodes reasonCode = SUCCESS;

rc = MQTTClient_disconnect5(client, 0, reasonCode, &props);

At the moment, the return from disconnect is still just an integer, as there is no MQTT response to a disconnect packet other than closing the connection.

I haven’t implemented the new MQTT V5 auth exchange packets yet, but I envisage this callback and call:

MQTTResponse MQTTClient_authArrived(void* context, enum MQTTReasonCodes rc, MQTTProperties props);

response = MQTTClient_auth(client, reasonCode, &props);

The callback will be used to implement an extended auth exchange initiated by the server, the function when the exchange is started by the client.

MQTTAsync for V5

Most of the changes to the MQTTAsync API are in parameter structures and in callback signatures rather than the API signatures themselves. The “create” and “setCallbacks” calls remain the same. V5 properties for both the connect and will messages are added to the existing connect options along with new callback signatures for success and failure:

MQTTAsync_connectOptions opts = MQTTAsync_connectOptions_initializer;
MQTTAsync_willOptions wopts = MQTTAsync_willOptions_initializer;
opts.onSuccess5 = test1_onConnect;
opts.onFailure = NULL;
opts.context = c;

property.identifier = SESSION_EXPIRY_INTERVAL;
property.value.integer4 = 30;
MQTTProperties_add(&props, &property);

property.identifier = USER_PROPERTY;
property.value.data.data = "test user property";
property.value.data.len = strlen(property.value.data.data);
property.value.value.data = "test user property value";
property.value.value.len = strlen(property.value.value.data);
MQTTProperties_add(&props, &property);

opts.connectProperties = &props;
opts.willProperties = &willProps;

rc = MQTTAsync_connect(c, &opts);

The return from the connect call remains an integer, as the expanded response information is made available in the callbacks in the MQTTAsync_successData5 structure:

typedef struct
{
	...
	enum MQTTReasonCodes reasonCode; /* MQTT V5 reason code returned */
	MQTTProperties props; /* MQTT V5 properties returned, if any */
	/** A union of the different values that can be returned for subscribe, unsubscribe and publish. */
	union
	{
		...
		/* For connect, the server connected to, MQTT version used, and sessionPresent flag */
		struct
		{
			char* serverURI;
			int MQTTVersion;
			int sessionPresent;
		} connect;
		...
	} alt;
} MQTTAsync_successData5;

In the test, the contents of the properties returned by the connack are logged in the success callback:

void test1_onConnect(void* context, MQTTAsync_successData5* response)
{
	...
	MyLog(LOGA_INFO, "Connack properties:");
	logProperties(&response->props);
	...
}

In subscribe, unsubscribe and send (publish) calls, the response options structure contains the properties as well as the new success and callback function pointers. As the options are now not just for responses, there is a synonym for the structure, called callOptions:

MQTTAsync_message pubmsg = MQTTAsync_message_initializer;
MQTTAsync_callOptions opts = MQTTAsync_callOptions_initializer;
MQTTProperty property;
MQTTProperties props = MQTTProperties_initializer;

property.identifier = USER_PROPERTY;
property.value.data.data = "test user property";
property.value.data.len = strlen(property.value.data.data);
property.value.value.data = "test user property value";
property.value.value.len = strlen(property.value.value.data);
MQTTProperties_add(&props, &property);
opts.properties = props;

pubmsg.payload = "a much longer message that we can shorten to the extent that we need to payload up to 11";
pubmsg.payloadlen = 11;
pubmsg.qos = 2;
pubmsg.retained = 0;
rc = MQTTAsync_sendMessage(c, test_topic, &pubmsg, &opts);
MQTTProperties_free(&props);

The call options structure also includes the new V5 subscribe options:

MQTTAsync_callOptions opts = MQTTAsync_callOptions_initializer;
opts.properties = props;

opts.subscribe_options.retainAsPublished = 1;
rc = MQTTAsync_subscribe(c, test_topic, 2, &opts);
MQTTProperties_free(&props);

and for disconnect, similarly as for connect, the disconnectOptions structure is extended:

MQTTAsync_disconnectOptions opts = MQTTAsync_disconnectOptions_initializer;
MQTTProperty property;
int rc;

opts.onSuccess = test1_onDisconnect;
opts.context = c;
opts.reasonCode = UNSPECIFIED_ERROR;

property.identifier = SESSION_EXPIRY_INTERVAL;
property.value.integer4 = 0;
MQTTProperties_add(&opts.properties, &property);

rc = MQTTAsync_disconnect(c, &opts);
MQTTProperties_free(&opts.properties);

As for the MQTTClient API I’ve not added the AUTH packet capabilities yet. They will echo the other APIs:

int rc = MQTTAsync_setAuthReceived(c, context, callback_pointer);

rc = MQTTAsync_auth(c, &opts);

The MQTTAsync_auth call, to send an AUTH packet, will be able to be called from the AuthReceived callback in response to an AUTH packet from the server, or separately to enable the client to initiate an authentication exchange.

That rounds up the summary for now, I’m very interested to hear any thoughts.

If you’re familiar with the Eclipse Paho test material, you’ll know that I wrote an MQTT 3.1.1 broker in Python with the purpose of providing a benchmark or oracle against which other implementations could be compared. Last year, I added support for the new MQTT V5.0 definition, so that client implementations of V5, including my own, could be easily tested. It has been used by Paho colleagues of mine, James Sutton and Allan Stockdill-Mander to develop MQTT V5 Java and Golang client libraries.

Then, I added the missing TLS support, so that it could satisfy testing requirements for secured connections. Adding TLS support in Python turned out to be really easy, much easier than I expected. I know how hard it is to add OpenSSL support in C, and I was prepared for at least some proportion of that effort, but there was none. I enabled multiple MQTT listeners to be configured, a la RSMB/Mosquitto. That meant the broker was comprehensive enough to be used for automated testing of existing client implementations. I used it to replace a remote Mosquitto broker in the automated Paho C client testing for Windows using AppVeyor, as installing Mosquitto on Windows is a little tricky, and it works fine in that capacity. All I had to do in the AppVeyor configuration file was to add these lines:


- cmd: git clone https://github.com/eclipse/paho.mqtt.testing.git
- cmd: cd paho.mqtt.testing\interoperability
- ps: Start-Process C:\Python36\python -ArgumentList 'startbroker.py -c client_testing.conf'


to get the broker running locally. WebSockets support was already included, so it means that each MQTT listener supports both 3.1.1 and V5, WebSockets and TLS. The 3.1.1 and V5 clients can communicate with each other (although this work is not complete and tested yet), both through normal publish commands and retained messages.

In a recent project at work I had added persistence to a Python project by using the ZODB object database, and that had turned out to be really easy too. These experiences got me to thinking that it should be easy to add MQTT-SN support as well, and create a broker with the capabilities that I had envisaged for RSMB before it was superseded by Mosquitto. Roger Light, the author of Mosquitto, has intended to add MQTT-SN support for a long time, but wanted to restructure the internals first. Due to other commitments, and some licensing obstacles, this has not yet come about, and with the advent of MQTT V5 which will have a higher priority, looks even further out.

A broker written in Python will not be as efficient as Mosquitto, nor is it particularly scalable as currently written because it uses Python’s threading model which does not take advantage of multiple cores. But as it is so easy to add new capabilities, it could well be one of the most complete MQTT implementations, soonest. In this way it would be complementary to Mosquitto.

I have now experimented with adding MQTT-SN support, a framework for bridges, and an HTTP listener which can provide APIs to query and update the broker state and configuration. We can add APIs to allow queries to return information about the currently connected clients, their subscriptions and messages for instance. The set of capabilities I envisage looks like this:

• TCP listener supporting WebSockets, TLS, MQTT 3.1.1 and 5.0
• UDP listener supporting MQTT-SN with DTLS, multicast
• other potential MQTT-SN listeners: BLE, ZigBee
• HTTP listener for state/configuration APIs. The broker behaviour for test purposes can potentially be reconfigured dynamically
• TCP bridge capable supporting MQTT 3.1.1 and 5.0, WebSockets and TLS
• UDP bridge supporting MQTT-SN with DTLS, multicast
• HTTP bridge supporting webhooks?
• dynamic bridge connections – adding/deleting bridges
• variety of persistence options including ZODB

I think it’s obvious to state that with these capabilities it could serve as an MQTT-SN gateway for services that support MQTT, as a conversion mechanism between MQTT V3.1.1 and V5, and a dynamic flexible component of an MQTT network. I would still keep a focus on enabling MQTT testing, including for test suite generation. A lot of this work could be completed in the first half of 2018, other commitments depending.

This brings me on to my main motivation for writing this blog post. Perhaps this broker should have a life of its own, separate to Paho, as an Eclipse IoT project. It could quite well do the job as part of Paho, conversely it could have a higher profile and encourage more community participation if stood on its own. So, if you have an interest in seeing this broker as an Eclipse IoT standalone project, from the point of view of using it, or also contributing to it, please let me know – thanks.