Using MQTT V5 with the IBM Watson IoT Platform and the Eclipse Paho C client

I’ve just finished off release 1.3 of the Eclipse Paho C client which includes MQTT 5.0 and WebSockets support. Another thing I’ve done is to update the command line utilities to be much more capable, so in this post I’ll describe how they can be used to connect to the IBM Watson IoT Platform using MQTT V5.

MQTT V5 is the latest version of the successful protocol which is the core of Paho’s capability. I’ve delivered a number of talks about MQTT V5, if you’d like to learn more about it that way. I’ve also previously written about how the C client command line utilities can use MQTT V5 to connect an MQTT server which supports V5 – we have one within the Paho project.

First you need to install the Eclipse Paho C client, if you haven’t already. As described on the linked page, the easiest way on Windows is probably to install the pre-built binaries, while on Linux and the Mac, for the time being, building from source. Running the command

paho_c_pub

will check that it runs, and display the full list of options.

IBM’s IoT Platform has a quick start playground, where you don’t have to sign up to try sending events. We’re going to connect to that first of all. Go to the web page, accept IBM’s terms of use, then enter a device id in the input box. This can be any sequence of characters you like (there are some restrictions and there is a length limit) – its purpose is to distinguish your device from other peoples’. So make it something unique enough, then push ‘Go’.

To connect the Paho publisher program to the platform with MQTT version 5, run the command:

paho_c_pub -c ssl://quickstart.messaging.internetofthings.ibmcloud.com:8883 -t iot-2/evt/myevent/fmt/json -i d:quickstart:my_device_type:my_device_id -V 5

replacing my_device_id with the device id you typed into the IoT Platform input box. Now the program is waiting for you to type an input message. The message should be in JSON format, because that’s what the quick start application is expecting, so type something like this:

{"t":2}

and press enter. You should see a point appear on the graph. Try sending a few more values, varying the number but keeping the rest the same, and you should see a line graph drawn something like this:

And that’s it! The -V 5 option at the end of the command line indicates that MQTT 5.0 was used rather than MQTT 3.1.1. There are two MQTT V5 specific options taken currently by the utility – user-property and message-expiry, although the platform doesn’t make use of them yet. The “–trace protocol” option will display details of the TLS exchange and MQTT packets sent and received.

If you already use the platform, to connect to an existing organization, use:

paho_c_pub -c ssl://orgid.messaging.internetofthings.ibmcloud.com:8883 -t iot-2/evt/event-id/fmt/json -i d:orgid:device-type:device-id --password auth-token --username use-token-auth -V5 

substituting appropriate values:

orgid – your organization id
event-id – the event id you want send events on
device-type – your device type
device-id – your device id
auth-token – the authentication token of the device previously defined

You can then send event data in the same way.

Paho C Client – MQTT 5.0 and Command-Line Utilities

I’m almost at the end of implementing the changes to the Eclipse Paho C client that I described in an earlier post to support MQTT 5.0. I haven’t quite finished off the release yet, so I thought I’d allow a little more time for anyone to comment. I had to compromise in my naming of the APIs for MQTT 5.0 support; none of the proposals I created were perfect, so I’m particularly interested in any feedback in that area.

MQTT 5.0 is not the only change for this release of the C client, but it is the biggest. You can find the complete list in the milestone. Another major addition is WebSocket support, thanks to the contribution of Keith Holman.

Another improvement I’ve been able to make is the update of the command line utilities. Back in 2010, I called them stdinpub and stdoutsub, the idea being that they could be used in Unix pipe combinations. Stdinpub reads data from stdin and publishes the results to an MQTT server; stdoutsub subscribes to an MQTT server and sends the data to stdout. I hadn’t really touched these programs in the meantime except to change the names to paho_c_pub and paho_c_sub. I always had intended to add TLS and other functionality to make them comparable to the Eclipse Mosquitto utilities mosquitto_pub and mosquitto_sub.

Now I’ve actually done that (small cheer!). Along with TLS I’ve added support for WebSockets and of course, MQTT 5.0. In each case, I have followed the Mosquitto utilities options where possible, so that it should be easy to switch from one to the other. There are actually two versions of each, one for each style of Paho C library, MQTTClient and MQTTAsync:

As well as being a useful way to experiment with V5 support, they also serve as examples on how to use the various APIs, for instance, how to write an application which combines support for all MQTT versions. You can try the following examples of using the command line utilities against the Paho test broker which supports all the combinations needed: WebSockets, MQTT 5.0 and TLS. To to start it up:

git clone git@github.com:eclipse/paho.mqtt.testing.git
cd git/paho.mqtt.testing/interoperability
python3 startbroker.py -c localhost_testing.conf

Typing paho_c_pub on the command line will display the option help:

-t (--topic)        : MQTT topic to publish to
-c (--connection)   : connection string, overrides host/port e.g wss://hostname:port/ws.  Use this option rather than host/port to connect with TLS and/or web sockets. No default.
-h (--host)         : host to connect to.  Default is localhost.
-p (--port)         : network port to connect to. Default is 1883.
-q (--qos)          : MQTT QoS to publish with (0, 1 or 2). Default is 0.
-V (--MQTTversion)  : MQTT version (31, 311, or 5).  Default is 311.
--quiet             : do not print error messages.
--trace             : print internal trace ("error", "min", "max" or "protocol").
-r (--retained)     : use MQTT retain option.  Default is off.
-n (--null-message) : send 0-length message.
-m (--message)      : the payload to send.
-f (--filename)     : use the contents of the named file as the payload.
-i (--clientid)     : MQTT client id. Default is paho-c-pub.
-u (--username)     : MQTT username. No default.
-P (--password)     : MQTT password. No default.
-k (--keepalive)    : MQTT keepalive timeout value. Default is 10 seconds.
--delimiter         : delimiter string.  Default is \n.
--maxdatalen        : maximum length of data to read when publishing strings (default is 100)
--message-expiry    : MQTT 5 only.  Sets the message expiry property in seconds.
--user-property     : MQTT 5 only.  Sets a user property.
--will-topic        : will topic on connect.  No default.
--will-payload      : will message.  If the will topic is set, but not payload, a null message will be set.
--will-retain       : set the retained flag on the will message.  The default is off.
--will-qos          : the will message QoS.  The default is 0.
--cafile            : a filename for the TLS truststore.
--capath            : a directory name containing TLS trusted server certificates.
--cert              : a filename for the TLS keystore containing client certificates.
--key               : client private key file.
--keypass           : password for the client private key file.
--ciphers           : the list of cipher suites that the client will present to the server during the TLS handshake.
--insecure          : don't check that the server certificate common name matches the hostname.

Here are some examples of use. To publish to the test broker on localhost over WebSockets using MQTT 5.0:

paho_c_pub -t topic --connection ws://localhost:1883 -V 5

To subscribe to the test broker on localhost over WebSockets with TLS server authentication using MQTT 3.1.1 by default:

paho_c_sub -t topic --connection wss://localhost:18885 --cafile test/ssl/test-root-ca.crt

And with mutual authentication without WebSockets:

paho_c_pub x --connection ssl://localhost:18884 --cafile test/ssl/test-root-ca.crt --cert test/ssl/client.pem --trace protocol

The –trace option displays various levels of trace, for debugging. A particularly useful level is ‘protocol’, which displays the MQTT and OpenSSL exchanges:

paho_c_pub x --connection ws://localhost:1883 -V 5 --trace protocol
Trace : 3, =========================================================
Trace : 3,                    Trace Output
Trace : 3, Product name: Eclipse Paho Asynchronous MQTT C Client Library
Trace : 3, Version: 1.2.1
Trace : 3, Build level: 2018-08-06T19:00:57Z
Trace : 3, OpenSSL version: OpenSSL 1.0.2o  27 Mar 2018
Trace : 3, OpenSSL flags: compiler: clang -I. -I.. -I../include  -fPIC -fno-common -DOPENSSL_PIC -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -arch x86_64 -O3 -DL_ENDIAN -Wall -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM -DECP_NISTZ256_ASM
Trace : 3, OpenSSL build timestamp: built on: reproducible build, date unspecified
Trace : 3, OpenSSL platform: platform: darwin64-x86_64-cc
Trace : 3, OpenSSL directory: OPENSSLDIR: "/usr/local/etc/openssl"
Trace : 3, =========================================================
Trace : 4, 20180809 144216.424 Connecting to serverURI localhost:1883 with MQTT version 5
Trace : 4, 20180809 144216.525 WebSocket connection upgraded
Trace : 4, 20180809 144216.525 5 paho-c-pub -> CONNECT cleansession: 0 (0)
Trace : 4, 20180809 144216.528 5 paho-c-pub <- CONNACK rc: 0
dfsdf
Trace : 4, 20180809 144259.493 5 paho-c-pub -> PUBLISH qos: 0 retained: 0 (0)

If using MQTT 5.0, you can attach user and message expiry properties to the sent message:

paho_c_pub -t topic --connection ws://localhost:1883 -V 5 --user-property name value --message-expiry 55

when received can be displayed with the –verbose option:

paho_c_sub x -V 5 -v
URL is localhost:1883
Subscribing to topic x with client paho-c-sub at QoS 0
13 x	dadasd
Property name MESSAGE_EXPIRY_INTERVAL value 55
Property name USER_PROPERTY key name value value

These utilities now have man pages too, as does the API itself, so I’ll have to make sure those are installed appropriately.

Thoughts on Help for Eclipse Paho

I gave a talk about MQTT V5 today at EclipseCon France, and at the end I gave the usual spiel about ways that people could help. After the talk, Cyrille Francois tweeted a great pictorial summary of the talk, as he did for other talks he attended. (Thanks Cyrille!) As part of the title of the tweet was the phrase “we need help”.

I’d like to add some more context to this. It’s of course true that we can always use more help. It’s also true that we are getting a lot of help already: issues, PRs, comments and feedback of all sorts. For example, one recent contribution to the Paho C client adds WebSocket support – thanks Keith! Paho is one of the most active Eclipse IoT projects as shown by this table extracted from an Eclipse IoT Metrics report:

Paho is also one of the most popular, in website visits:

although that metric is not definitive by any means, nor is this one:

as people obtain software from all the projects by many different means, not just downloads, but I don’t mind showing it given where Paho lies on it! At the time of writing, the Paho components have been starred 4108 times in total (up from 1766 last year) and users have forked the code 1969 times (up from 967). The number of committers now stands at 13, and the number of code authors is at least 88 (up from 50).

Paho is composed of distinct projects; client libraries and utilities, so those 13 committers are spread thinly across them. It’s an Eclipse convention that one committer doesn’t interfere in another’s repository without permission. Even without that, each committer is generally responsible for one or two specific components.

So I think we could do with some more committers, on just about all the Paho components. There are some that are in more need than others. The JavaScript client doesn’t really have a current owner, James Sutton and I step in when we can and as needed. We plan to update it for MQTT 5.0 later this year. The .Net client hasn’t been updated for a while and we’ve asked for help. All the other components could be helped by having more than one committer responsible.  Benjamin Cabé recently asked what would happen to the Paho C client if I was unable to continue (for whatever reason!) for instance.

When I became project lead I read the Eclipse documents quite thoroughly to learn what was expected.  About new committers, this is written:

Contributors who have the trust of the project’s committers can, through election, be promoted committer for that project. The breadth of a committer’s influence corresponds to the breadth of their contribution. A development team’s contributors and committers may (and should) come from a diverse set of organizations. A committer gains voting rights allowing them to affect the future of the project.

Becoming a committer is a privilege that is earned by contributing and showing discipline and good judgment. It is a responsibility that should be neither given nor taken lightly, nor is it a right based on employment by an Eclipse member company or any company employing existing committers.

I have taken these words quite seriously, maybe too seriously.  Should I lighten up a bit?

If anyone would like to become a committer then you definitely let me know and we can start talking.  Whether that’s for a particular component, or a wider remit such as helping with the website, or release process, it isn’t just coding.  There’s process development, design, documentation, tests for continuous integration, release packaging and distribution for example.

Of course you don’t need to be a committer to help by opening issues, responding to issues you didn’t open, raising PRs, or answering questions on mailing lists or forums.  If you haven’t done any of that already, you should start there.  There are many reasons to contribute to open source projects too.  Here’s what looks like a pretty helpful guide.

At EclipseCon today, Simon Phipps gave a great talk about Open Source, where it came from, where it’s going and why you should think about contributing if you use open source projects.  I expect that the slides will be up soon and you should take a look.  Seeing such talks in person is of course one of the benefits of attending conferences such as EclipseCon – you should do that too!

The new MQTT V5 API for the Eclipse Paho C client

I’ve started to write the new MQTT version 5.0 support for the Eclipse Paho C clients. As I’ve reached the stage where some very simple tests are running, I thought this was a good time to try to elicit some feedback. I’ve had to make some compromises in the API, balancing the minimization of disruption to application programs with the natural incorporation the new MQTT 5.0 capabilities.

If you’ve used the main Paho C client, you’ll know it comes in two flavours:

MQTTClient
The synchronous version – intended to be easy to get started. Most calls block, waiting for a response from the server. This was the first API, and the only one when I created it. I modelled it after the existing Java API for consistency.
MQTTAsync
The asynchronous version – (almost) entirely non-blocking. Responses are reported in callback functions. This was a response to the need to run in asynchronous environments such as iOS and Windows.

The mqttv5 branch in the Github repo contains all the MQTT V5 updates to date. The test15 and test45 test programs have the first tests for MQTTClient and MQTTAsync libraries respectively. (At the moment, only test1 in each file has been migrated to MQTT V5, so only look at those).

MQTTClient for V5

First we have to create the client object with MQTTClient_create – this stays the same.


rc = MQTTClient_create(&client, options.connection,
 "single_threaded_test", MQTTCLIENT_PERSISTENCE_DEFAULT, NULL);

Next we must connect to the server.  As outlined in my previous post the main difference in MQTT V5 packets is the addition of the properties section, and for acknowledgements and disconnect, a reason code byte. Properties exist on both requests and their acknowledgements.

The connect call thus adds a properties object, for the connect request itself, plus the will message, and as part of the response. The MQTT V3 code:


MQTTClient_connectOptions conn_opts = MQTTClient_connectOptions_initializer;

int rc = MQTTClient_connect(client, &conn_opts))

becomes:


MQTTClient_connectOptions opts = MQTTClient_connectOptions_initializer;
MQTTProperties props = MQTTProperties_initializer;
MQTTProperties willProps = MQTTProperties_initializer;
MQTTResponse response = {SUCCESS, NULL};

response = MQTTClient_connect5(c, &opts, &props, &willProps);

The property and will property fields can be NULL if not required. The properties structure is manipulated with the add, free and copy functions. To add properties:


property.identifier = SESSION_EXPIRY_INTERVAL;
property.value.integer4 = 30;
MQTTProperties_add(&props, &property);

and


property.identifier = USER_PROPERTY;
property.value.data.data = "test user property";
property.value.data.len = strlen(property.value.data.data);
property.value.value.data = "test user property value";
property.value.value.len = strlen(property.value.value.data);
MQTTProperties_add(&props, &property);

Memory is allocated in the process of adding a property to the properties structure, so when you’ve finished with it this must be freed:


MQTTProperties_free(response.properties);

The response structure looks like this:


typedef struct MQTTResponse
{
  enum MQTTReasonCodes reasonCode;
  MQTTProperties* properties; /* optional */
} MQTTResponse;

I’m not particularly happy with the name of the function, MQTTClient_connect5, but it does have the benefit of being short with a minimal change of name. Other options I considered were:

  • MQTT5Client_connect
  • MQTTClient_connectWithProperties
  • MQTTClient_connectExt

I quite like the “connectWithProperties” option, but it didn’t get much approval from the audience I’ve questioned so far — it also doesn’t reflect all the parameters required.

The other MQTT functions have similar changes. An MQTTResponse return structure instead of a single int, and an extra input parameter for properties. For the publishMessage function, the properties are added to the MQTTClient_message structure:


MQTTClient_message pubmsg = MQTTClient_message_initializer;

property.identifier = USER_PROPERTY;
property.value.data.data = "test user property";
property.value.data.len = strlen(property.value.data.data);
property.value.value.data = "test user property value";
property.value.value.len = strlen(property.value.value.data);
MQTTProperties_add(&pubmsg.properties, &property);

response = MQTTClient_publishMessage5(c, test_topic, &pubmsg, &dt);

And for the subscribe function, there are added subscribe-specific options:


MQTTSubscribe_options subopts = MQTTSubscribe_options_initializer;

subopts.retainAsPublished = 1;
subopts.noLocal = 0;
subopts.retainHandling = 0; /* 0, 1 or 2*/

response = MQTTClient_subscribe5(client, test_topic, subsqos, &subopts, &props);

Finally the disconnect function has both input properties and reason code:


enum MQTTReasonCodes reasonCode = SUCCESS;

rc = MQTTClient_disconnect5(client, 0, reasonCode, &props);

At the moment, the return from disconnect is still just an integer, as there is no MQTT response to a disconnect packet other than closing the connection.

I haven’t implemented the new MQTT V5 auth exchange packets yet, but I envisage this callback and call:



MQTTResponse MQTTClient_authArrived(void* context, enum MQTTReasonCodes rc, MQTTProperties props);

response = MQTTClient_auth(client, reasonCode, &props);

The callback will be used to implement an extended auth exchange initiated by the server, the function when the exchange is started by the client.

MQTTAsync for V5

Most of the changes to the MQTTAsync API are in parameter structures and in callback signatures rather than the API signatures themselves. The “create” and “setCallbacks” calls remain the same. V5 properties for both the connect and will messages are added to the existing connect options along with new callback signatures for success and failure:


MQTTAsync_connectOptions opts = MQTTAsync_connectOptions_initializer;
MQTTAsync_willOptions wopts = MQTTAsync_willOptions_initializer;
opts.onSuccess5 = test1_onConnect;
opts.onFailure = NULL;
opts.context = c;

property.identifier = SESSION_EXPIRY_INTERVAL;
property.value.integer4 = 30;
MQTTProperties_add(&props, &property);

property.identifier = USER_PROPERTY;
property.value.data.data = "test user property";
property.value.data.len = strlen(property.value.data.data);
property.value.value.data = "test user property value";
property.value.value.len = strlen(property.value.value.data);
MQTTProperties_add(&props, &property);

opts.connectProperties = &props;
opts.willProperties = &willProps;

rc = MQTTAsync_connect(c, &opts);

The return from the connect call remains an integer, as the expanded response information is made available in the callbacks in the MQTTAsync_successData5 structure:


typedef struct
{
	...
	enum MQTTReasonCodes reasonCode; /* MQTT V5 reason code returned */
	MQTTProperties props; /* MQTT V5 properties returned, if any */
	/** A union of the different values that can be returned for subscribe, unsubscribe and publish. */
	union
	{
		...
		/* For connect, the server connected to, MQTT version used, and sessionPresent flag */
		struct
		{
			char* serverURI;
			int MQTTVersion;
			int sessionPresent;
		} connect;
		...
	} alt;
} MQTTAsync_successData5;

In the test, the contents of the properties returned by the connack are logged in the success callback:


void test1_onConnect(void* context, MQTTAsync_successData5* response)
{
	...
	MyLog(LOGA_INFO, "Connack properties:");
	logProperties(&response->props);
	...
}

In subscribe, unsubscribe and send (publish) calls, the response options structure contains the properties as well as the new success and callback function pointers. As the options are now not just for responses, there is a synonym for the structure, called callOptions:


MQTTAsync_message pubmsg = MQTTAsync_message_initializer;
MQTTAsync_callOptions opts = MQTTAsync_callOptions_initializer;
MQTTProperty property;
MQTTProperties props = MQTTProperties_initializer;

property.identifier = USER_PROPERTY;
property.value.data.data = "test user property";
property.value.data.len = strlen(property.value.data.data);
property.value.value.data = "test user property value";
property.value.value.len = strlen(property.value.value.data);
MQTTProperties_add(&props, &property);
opts.properties = props;

pubmsg.payload = "a much longer message that we can shorten to the extent that we need to payload up to 11";
pubmsg.payloadlen = 11;
pubmsg.qos = 2;
pubmsg.retained = 0;
rc = MQTTAsync_sendMessage(c, test_topic, &pubmsg, &opts);
MQTTProperties_free(&props);

The call options structure also includes the new V5 subscribe options:


MQTTAsync_callOptions opts = MQTTAsync_callOptions_initializer;
opts.properties = props;

opts.subscribe_options.retainAsPublished = 1;
rc = MQTTAsync_subscribe(c, test_topic, 2, &opts);
MQTTProperties_free(&props);

and for disconnect, similarly as for connect, the disconnectOptions structure is extended:


MQTTAsync_disconnectOptions opts = MQTTAsync_disconnectOptions_initializer;
MQTTProperty property;
int rc;

opts.onSuccess = test1_onDisconnect;
opts.context = c;
opts.reasonCode = UNSPECIFIED_ERROR;

property.identifier = SESSION_EXPIRY_INTERVAL;
property.value.integer4 = 0;
MQTTProperties_add(&opts.properties, &property);

rc = MQTTAsync_disconnect(c, &opts);
MQTTProperties_free(&opts.properties);

As for the MQTTClient API I’ve not added the AUTH packet capabilities yet. They will echo the other APIs:


int rc = MQTTAsync_setAuthReceived(c, context, callback_pointer);

rc = MQTTAsync_auth(c, &opts);

The MQTTAsync_auth call, to send an AUTH packet, will be able to be called from the AuthReceived callback in response to an AUTH packet from the server, or separately to enable the client to initiate an authentication exchange.

That rounds up the summary for now, I’m very interested to hear any thoughts.

A Story of MQTT 5.0

The MQTT protocol has been around since the late 90s when it was created to enable the monitoring of a long distance oil pipeline. It went through several iterations before landing on version 3.1, published by IBM.

The next step was standardisation, at the OASIS standards body. As anyone who has taken part in a standardisation committee will know, this process is necessarily bureaucratic and slow. To speed up adoption, the main imperative was minimising disruption to existing implementations, as set out in the Technical Committee (TC) charter.

As a result, wholesale changes to the MQTT 3.1 specification were not allowed in the 3.1.1 standard. This meant that many irritating flaws could not be fixed nor widely sought enhancements included. This is where MQTT 5.0 comes in. While we still wanted to minimise disruption (no-one wanted to repeat the experiences of say AMQP 0.9 to 1.0), we also wanted to address the MQTT wish-list as far as possible so that major changes would not be needed for a long time to come. Whether we succeeded in that aim, time will tell.

To help us make sense of the multitude of items on that wish list, as I wrote in September 2016, they were grouped into four Big Ideas:

  • Improved error reporting
  • Extensible metadata
  • Scalability and large scale systems
  • Resource Constrained Clients and Performance Improvements

At that time, many of the solutions were not decided upon, but now with the availability of Committee Specification 01, I can write about the details. We are in the final stages of the standardisation process for 5.0. We hope to complete the process of rubber stamping in the next few months, and expect no substantive changes during that time.

Improved Error Reporting

Negative responses, or nacks, were the biggest omission from earlier versions of MQTT. If the client or server had a problem with the request or packet from the other end, the only recourse in many circumstances was to close the TCP connection. Connect packets were the exception to this: the connack always had a return code. MQTT 3.1.1 added a negative response code to subscribe requests because there was space available in the “granted QoS” field of the suback packet. Publish requests, however, were still not catered for. This is now remedied.

Reason Codes

As all ack packets now have reason codes, they have been consolidated into one set, which starts like this:

Reason codes are one byte. Values from 0 to 127 inclusive indicate successful outcomes, those from 128 to 0xFF unsuccessful. So as the subscribe response can have an error code:

msc {
    arcgradient="4";

    c [label="client"], b [label="broker"];

    c => b [label="connect"];
    b => c [label="connack(rc=0)"];

    c => b [label="subscribe"];
    b => c [label="suback(rc=0x80)"];

    ...;
}

so can the publish, when the QoS is greater than 0:

msc {
    arcgradient="4";

    c [label="client"], b [label="broker"];

    c => b [label="connect"];
    b => c [label="connack(rc=0)"];
    ...;
    c => b [label="publish, qos=1"];
    b => c [label="puback(rc=0x80)"];
    ...;
    c => b [label="publish, qos=2"];
    b => c [label="pubrec(rc=0x80)"];
    ...;
    c => b [label="publish, qos=2"];
    b => c [label="pubrec(rc=0x0)"];
    c => b [label="pubrel(rc=0x80)"];
    ...;
    c => b [label="publish, qos=2"];
    b => c [label="pubrec(rc=0x0)"];
    c => b [label="pubrel(rc=0x0)"];
    b => c [label="pubcomp(rc=0x80)"];
    ...;
}

For the QoS 2 exchange, it stops if any of the reason codes are 0x80 or above. This is a major improvement on previous versions of MQTT, where continuing with the exchange or terminating the connection were the only options.

Server initiated disconnect

In MQTT versions prior to 5.0, only the client could send a disconnect packet. This meant that in any case where the server wanted to end the conversation with a client, there was no option but to just terminate the TCP connection. A common case is when the server shuts down – there is no error in the interaction between broker and client, but the client has no idea what’s happening. In MQTT 5.0, the server can send a disconnect packet with a “Server shutting down” reason code:

msc {
    arcgradient="4";

    c [label="client"], b [label="broker"];

    c => b [label="connect"];
    b => c [label="connack(rc=0)"];
    ...;
    c => b [label="publish, qos=1"];
    b => c [label="disconnect(rc=139)"];
}

In this case, the client might wait for a while before attempting a reconnect, knowing that the server might not be available for a while.

Extensible metadata

The big change here is in addition to reason codes, each packet (apart from pings) can have properties. This is an extract of the full list:

Properties can be used to add extra information to responses, such as a reason string, or extra parameters to requests. A lot of the rest of the changes rely on properties because now we had a mechanism for adding that extra information to packets, we had to use it!

Request/response

The new request/response capability makes good use of properties. The requester subscribes to the topic it expects to receive responses on, then sets the value of the “Response Topic” property to that topic name. The responder simply uses that property to set the topic name for its response.

msc {
    arcgradient="4";

    c1 [label="client1"], b [label="broker"], c2 [label="client2"];

    c1 => b [label="subscribe, topic=resp"];
    b => c1 [label="suback"];

    c1 => b [label="publish, response_topic=resp"];
    b => c2 [label="publish, response_topic=resp"];

    ...;

    c2 => b [label="publish, topic=resp"];
    b => c1 [label="publish, topic=resp"];
}

The “Correlation Data” property can be used to set an id for each request, so that replies can be matched to requests by the requester.

Payload format indicator

There were fairly contentious discussions about how much flexibility there should be in payload format settings. Some were in favour of user definable payload formats. Others felt that if people could define their own formats it was no better than the current position, unless some body kept an approved list of format indicators and their meanings. That seemed a step too far for MQTT. MIME types were discussed, but the final approach is minimalistic – just two values: binary, as 3.1.1, or UTF-8 data.

Enhancements for Scalability

Improved error reporting helps scalability because exchanges between servers and clients become more efficient. Properties are again crucial to the following functions.

Simplified session state

One of the other big irritations with 3.1.1, along with the lack of nacks for publish commands, is the behaviour of the “clean session” flag. In earlier versions of MQTT, this started out as the “clean start” flag, where the session state was only cleaned up at the start of a session, not at the end. This was good for clients, because it meant you could ensure a clean starting point, and leave the session around in case you needed to reconnect. Not so good for servers, because clients would tend to leave the state lying around for ever.

Later on, this flag was changed to “clean session”, cleaning the session state both at the start and end of the session. Good for servers. For clients, if they want to ensure a clean slate to start with, but then want to have session state saved, they have to connect twice:

msc {
    arcgradient="4";

    c [label="client"], b [label="broker"];

    c => b [label="connect cleansession=true"];
    b => c [label="connack"];
    c => b [label="disconnect"];
    ...;
    c => b [label="connect cleansession=false"];
}

We knew we should fix this situation once and for all. The “clean session” flag becomes “clean start” once more – session state is only cleaned up at the start of the session. Then there is the “session expiry interval” property, a four-byte integer value in seconds which defaults to zero if omitted. If it is set to 0xFFFFFFFF (UINT_MAX), the session does not expire. To accomplish the above scenario:

msc {
    arcgradient="4", wordwraparcs=on;

    c [label="client"], b [label="broker"];

    c => b [label="connect cleanstart=true, expiry_interval=0xFFFFFFFF"];
    ...;
}

The MQTT-SN “offline keep alive” scenario is also catered for. By setting the expiry interval to a suitable non-zero value, the client can ensure that the session state is saved as long as it reconnects regularly. If the client disappears entirely, the session state will be cleaned up at some point. Both clients and servers are happy.

Shared subscriptions

To allow load balancing of high throughput topics, the concept of shared subscriptions is introduced to MQTT. Messages on these topics are sent to one of a group of subscribers rather than to them all. The subscriber indicates that the subscription is shared simply by subscribing to a special topic pattern:


$share/{ShareName}/{filter}

where ShareName is the name of the shared subscription group, and filter is the usual topic filter used in the subscribe request.

Optional server capabilities

Some server functionality is expensive to implement at large scale. In MQTT 5.0, the server can advertise the limitations on the functionality it provides in the connack properties. Some examples:

Retain Available
are retained messages supported?
Maximum QoS
the maximum publish QoS the server will accept
Maximum Packet Size
the maximum packet size the server will accept
Receive maximum
the maximum number of concurrent QoS 1 and 2 message the server will handle

Resource Constrained Clients and Performance Improvements

Various features fall into this category, including some already described. Some further examples follow.

Nolocal subscriptions

Up until MQTT 5.0, the publisher of a message will receive that message back if it is subscribed to the same topic. People often find this out in their first experience of writing an MQTT application, when they implement a shared chat room. There is now a subscribe option noLocal which when set, indicates that the publishing application should not receive its own messages.

msc {
    arcgradient="4", wordwraparcs=on;

    c [label="client"], b [label="broker"];

    c => b [label="subscribe topic=a"];
    b => c [label="suback"];
    c => b [label="publish topic=a"];
    b => c [label="publish topic=a"];
    ...;
    c => b [label="subscribe topic=b, noLocal"];
    b => c [label="suback"];
    c => b [label="publish topic=b"];
    ...;
}

Retained message control

Options on the subscribe request have been added to:

  • 0 = Send retained messages at the time of the subscribe
  • 1 = Send retained messages at subscribe only if the subscription does not currently exist
  • 2 = Do not send retained messages at the time of the subscribe

This could help particularly with the implementation of MQTT bridges from one broker to another.

Topic aliases

This capability exists in MQTT-SN, to reduce the size of the publish packet when long topic names are used. The publish request allows a numeric topic alias to be specified, which can be used in subsequent publish packets. Topic aliases on the client and the server are independent of each other, in much the same way as packet ids are.

msc {
    arcgradient="4", wordwraparcs=on;

    c [label="client"], b [label="broker"];

    c => b [label="publish topic=long_name,alias=1"];
    c => b [label="publish alias=1"];
    ...;
    b => c [label="publish topic=server_long_name, alias=1"];
    b => c [label="publish alias=1"];
    ...;
}

Topic aliases only exist for the lifetime of a TCP connection.

Specifying client limitations

To help protect implementations on small devices, the client can specify its limitations using properties on the connect packet. Some examples:

Maximum Packet Size
the maximum packet size the client can accept
Receive maximum
the maximum number of concurrent QoS 1 and 2 message the client can handle

It is an administrative action or decision on the part of the server to decide what to do with messages that it receives bound for a client for which that message exceeds the constraints. This is not particularly different from 3.1.1 where the message would be sent anyway, and then the client might be forced to disconnect as its only recourse. At a minimum, the server should probably emit a warning log message.

Eclipse Paho Progress

A release of the Eclipse Paho project is planned for June 2018 with its first implementations of MQTT 5.0. I first implemented a broker to test against in the Paho test project. It combines 3.1.1 and 5.0 implementations, and has been used by James Sutton to implement the Java MQTT 5.0 support. It is used in Travis and AppVeyor continuous integration tests for the MQTT 5.0 branches. Example output when you start it up is shown below.

My own C clients, embedded and main are planned to have a June release. The MQTT 5.0 implementations continue in the embedded mqttv5 and mqttv5 branches. Please do give your feedback or thoughts on these implementations as they progress via the GitHub issues:

A New Fate for the Eclipse Paho “Test” Broker?

If you’re familiar with the Eclipse Paho test material, you’ll know that I wrote an MQTT 3.1.1 broker in Python with the purpose of providing a benchmark or oracle against which other implementations could be compared. Last year, I added support for the new MQTT V5.0 definition, so that client implementations of V5, including my own, could be easily tested. It has been used by Paho colleagues of mine, James Sutton and Allan Stockdill-Mander to develop MQTT V5 Java and Golang client libraries.

Then, I added the missing TLS support, so that it could satisfy testing requirements for secured connections. Adding TLS support in Python turned out to be really easy, much easier than I expected. I know how hard it is to add OpenSSL support in C, and I was prepared for at least some proportion of that effort, but there was none. I enabled multiple MQTT listeners to be configured, a la RSMB/Mosquitto. That meant the broker was comprehensive enough to be used for automated testing of existing client implementations. I used it to replace a remote Mosquitto broker in the automated Paho C client testing for Windows using AppVeyor, as installing Mosquitto on Windows is a little tricky, and it works fine in that capacity. All I had to do in the AppVeyor configuration file was to add these lines:


- cmd: git clone https://github.com/eclipse/paho.mqtt.testing.git
- cmd: cd paho.mqtt.testing\interoperability
- ps: Start-Process C:\Python36\python -ArgumentList 'startbroker.py -c client_testing.conf'

to get the broker running locally. WebSockets support was already included, so it means that each MQTT listener supports both 3.1.1 and V5, WebSockets and TLS. The 3.1.1 and V5 clients can communicate with each other (although this work is not complete and tested yet), both through normal publish commands and retained messages.

In a recent project at work I had added persistence to a Python project by using the ZODB object database, and that had turned out to be really easy too. These experiences got me to thinking that it should be easy to add MQTT-SN support as well, and create a broker with the capabilities that I had envisaged for RSMB before it was superseded by Mosquitto. Roger Light, the author of Mosquitto, has intended to add MQTT-SN support for a long time, but wanted to restructure the internals first. Due to other commitments, and some licensing obstacles, this has not yet come about, and with the advent of MQTT V5 which will have a higher priority, looks even further out.

A broker written in Python will not be as efficient as Mosquitto, nor is it particularly scalable as currently written because it uses Python’s threading model which does not take advantage of multiple cores. But as it is so easy to add new capabilities, it could well be one of the most complete MQTT implementations, soonest. In this way it would be complementary to Mosquitto.

I have now experimented with adding MQTT-SN support, a framework for bridges, and an HTTP listener which can provide APIs to query and update the broker state and configuration. We can add APIs to allow queries to return information about the currently connected clients, their subscriptions and messages for instance. The set of capabilities I envisage looks like this:

  • TCP listener supporting WebSockets, TLS, MQTT 3.1.1 and 5.0
  • UDP listener supporting MQTT-SN with DTLS, multicast
  • other potential MQTT-SN listeners: BLE, ZigBee
  • HTTP listener for state/configuration APIs. The broker behaviour for test purposes can potentially be reconfigured dynamically
  • TCP bridge capable supporting MQTT 3.1.1 and 5.0, WebSockets and TLS
  • UDP bridge supporting MQTT-SN with DTLS, multicast
  • HTTP bridge supporting webhooks?
  • dynamic bridge connections – adding/deleting bridges
  • variety of persistence options including ZODB

I think it’s obvious to state that with these capabilities it could serve as an MQTT-SN gateway for services that support MQTT, as a conversion mechanism between MQTT V3.1.1 and V5, and a dynamic flexible component of an MQTT network. I would still keep a focus on enabling MQTT testing, including for test suite generation. A lot of this work could be completed in the first half of 2018, other commitments depending.

This brings me on to my main motivation for writing this blog post. Perhaps this broker should have a life of its own, separate to Paho, as an Eclipse IoT project. It could quite well do the job as part of Paho, conversely it could have a higher profile and encourage more community participation if stood on its own. So, if you have an interest in seeing this broker as an Eclipse IoT standalone project, from the point of view of using it, or also contributing to it, please let me know – thanks.

What’s happening with RSMB?

Some people will be aware that I wrote a small MQTT broker in about 2008, made available on IBM’s alphaWorks website. It was called RSMB (Really Small Message Broker) when released, because the name we used first for it, Nanobroker, was already taken. Somewhat amusingly, an IBM website for RSMB still exists.

A year or two later, Roger Light asked Andy Stanford-Clark why RSMB wasn’t open source (not my decision), so Andy suggested Roger write his own. And that’s how Mosquitto started, as a drop in replacement for RSMB. When MQTT software was being contributed to the Eclipse foundation, IBM contributed a Java client and my C client to the Paho project, and Mosquitto was contributed as the broker. IBM did contribute the RSMB source to the Mosquitto project on my encouragement, to serve as a repository of potentially useful code, and because it had support for MQTT-SN.

So in my mind, Mosquitto became the official replacement for RSMB, and I expected RSMB to outlive its usefulness pretty quickly. As it happens, MQTT-SN support in Mosquitto has been on the back burner ever since, because Roger wanted to rebase the internals of Mosquitto on an event library before tackling it. Unfortunately, this ran into a number of issues, social and technical. I’m still hoping that it will happen.

But one of the alternative approaches to MQTT-SN support is now available in Paho (written by Tomoaki Yamaguchi) – a transparent gateway which converts MQTT-SN into MQTT. Transparent because it creates a new MQTT connection for each MQTT-SN client, so that the MQTT broker has visibility of those clients. RSMB acts as an aggregating gateway, where one MQTT bridge connection carries the traffic for all MQTT-SN clients, and the MQTT broker sees only one.

I do wonder if an MQTT-SN to MQTT gateway will in fact be a better solution because it may allow easier support of additional underlying transports. The gateway has UDP and XBee right now, others such as BLE and even serial could be useful.

Mosquitto in the meantime has added further capabilities such as TLS and WebSocket support, and many more. If there were a niche that Mosquitto has left open then I would be happy to support RSMB in that, but I don’t think there is. The combination of Mosquitto and the Paho transparent gateway will do a better job all round.

Where are we with MQTT-SN?

This question was posed to me recently, with the added observation that it seemed no progress had been made in the last two years.  From my perspective this isn’t quite true.  Although the specification is still maintained by IBM, no movement has been made to standardize it at OASIS like MQTT has been. That is not to say that the subject of MQTT-SN has not arisen in the MQTT OASIS Technical Committee (TC) discussions, it has. But amongst the numerous improvements that we wanted to make to MQTT 3.1.1, and an ambitious timescale — publishing the new version MQTT 5.0 this year (2017) — addressing non TCP networks in MQTT was put to one side.

There is a continual wavering in my mind as to the importance of MQTT-SN. On one hand, edge computing platforms are getting ever more powerful, on the other, low power consumption is very important for battery power. The powerful edge processors are likely to have TCP stacks for which MQTT is appropriate: but there are still use cases for which low power use is the crucial factor. UDP, Bluetooth Low Energy (BLE) are typical transports used in this case.

If there is enough interest in a standardized MQTT-SN, its features could be addressed by the MQTT OASIS TC. There are a number of options. Incorporating MQTT-SN or its features into the main MQTT specification is one. However MQTT currently requires a reliable underlying network transport (TCP). To change that assumption we would have to consider very carefully the implications, which could take a significant amount of time. It has been suggested that MQTT-SN could be dealt with as a committee note: I’m not sure what form that would take.

In the meantime, I’ve been trying to get more MQTT-SN capabilities added to the Eclipse® Paho project. After I contributed an initial MQTT-SN packet library a couple of years ago, and the RSMB broker with MQTT-SN support to the Eclipse Mosquitto project at its inception, things stalled for a while (but not for a lack of interest on my part). There are forks of RSMB with fixes to the MQTT-SN support, and various gateways and clients around. One of those, written by Tomoaki Yamaguchi, had gained some support and I forget exactly how it happened, but Tomoaki has now contributed an MQTT-SN transparent gateway to Paho.

Over the past months this gateway has matured nicely. I recently used it to replace the use of RSMB in this experiment of Benjamin Cabé’s and it worked well. If we add BLE support to the gateway, then the Node.js BLE to UDP MQTT-SN forwarder in that experiment would not be required either. The gateway is written with the intention of allowing other transports to be added, so this should be eminently feasible.

I do remember seeing an email or post or something recently describing some other MQTT-SN components written by the poster. But now I can’t find or remember where I saw it. If this wasn’t a dream, and you know of such a thing, or it was you, please do let me know.

Where do we go from here?  Two avenues: practical implementations and the specification.  After the MQTT 5.0 standard is published, we can see if there is any interest in the OASIS TC in pursuing the application of MQTT to non-TCP networks.  In the meantime, I will continue working on and encouraging MQTT-SN contributions of the current specification to Paho and elsewhere.

If you are interested in MQTT-SN and would like to see it considered by the OASIS MQTT TC, or make any other comments to the TC, then you can use the mailing list.  If you know of any other implementations or have any questions, or suggestions for Paho, then I will be happy to hear of them

The UK Referendum on Leaving the EU

I want to record my thoughts on the UK referendum, so that in the future I can refer to them, and remember my feelings in the immediate aftermath.

I was going to the Glastonbury festival on Friday 24th June. Normally I go on the Wednesday, but this time I’d just come back from Montreal the week before, so I left it until later. On the Thursday evening, I started watching the results, but ended up watching all night, in some disbelief. At Glastonbury, the mood was sombre amongst my friends and the people we met, and amongst the performers who spoke out.

I think my overwhelming feeling was one of loss: I thought we were all in this together, we being Europe, and then the world. This felt like a step backwards, splitting apart instead of coming together to work out our problems. I wanted our borders, a purely human divisive contrivance, to become less important, not more. Of course, in this day and age of increasingly easy mass communication, this is not really going to happen unless civilization takes a severely different turn, which I hope is unlikely.

My perception is that people voted to leave the EU for a variety of reasons, not all of which are compatible. In the process of leaving, some of those who voted to leave will be dissatisfied. For those on far enough left of the political spectrum, the EU is too right-wing; for those far enough on the right, the EU is too left-wing. I think those on the left disapprove of TTIP, and the effects of global capitalism. I’m not sure what those on the right exactly disapprove of. I did entertain the idea that some were still hoping for the return of the British Empire, then dismissed that thought as insulting. To my astonishment through the referendum campaign, I realized that there really are some in that camp.

Undoubtedly one of the key reasons people voted to leave is immigration. David Cameron promised in 2010 to reduce immigration to the tens of thousands per year, a policy with which I broadly don’t agree. The promise was reiterated in 2015. The failure to do this was blamed on the EU. Yet we have more immigration from outside of the EU – only slightly more, but still more. Even the immigration over which we already have control has not been reduced to the “tens of thousands” per year. Why not? That is just abject failure and dishonesty on the part of the government – making a promise on which they know they can’t deliver. Then it was used as a stick with which to beat the EU.

Another reason was to vote leave was to save the money contributed to the EU. This rough figure is easy enough to estimate, event if the Leave campaign got it wrong by “forgetting” to subtract the money we get back. Even then the figure doesn’t take into account the benefit to us of being within the single market, the growth in the economy that otherwise wouldn’t have happened. Of course, we don’t really know how big that the effect was, we can only speculate. Then the question is, is it worth contributing £160 million per week? If we leave, will we actually save that amount, or will it be eaten away by other costs and expenses?

Perhaps the biggest message was “taking back control”, “loss of sovereignty”, which included immigration. As noted, we couldn’t even reduce the immigration over which we already did have control. In my life in Britain, I’ve never lived in a place where my vote in a general election had any effect. Growing up in South Wales, Labour would always win. Since I started work, I’ve only lived in constituencies where the Conservatives always win. Under the first past the post system, I vote, but it has no effect. Conversely, my MEP is elected by proportional representation. We are largely governed by the privately educated; the head of state inherits his or her position, and the second chamber of parliament is full of appointees. I don’t want to give more power to this leadership – the EU is more democratic.

I subscribe to the Economist. One article referred to a report written by the Open Europe, which I downloaded and read significant parts of. It seemed to be the most balanced analysis of the impact of the UK leaving the EU. It also contains what I feel to be a key statement:

“If the UK puts as much effort into reforming the EU as it would have to in order to make a success of Brexit, the UK and the EU would both be far better off.”

To make the most “success” out of leaving, we would have to abolish or reduce regulations on subjects like environmental and employment protections and climate change mitigation. Those are not topics on which I wish to see less regulation. Over many important aspects of decision making, we already had control – it was our own government not the EU that dictated our position. The EU was just a scapegoat.

Instead of making a concerted effort to make a better EU, years of effort will now be put into unnecessary legal process and trade negotiations. This could take the best part of a decade, the only winners out of which will be the lawyers, politicians, trade negotiators and accountants. Maybe some company board members will accrue a few more millions as a result of reduced employment and environmental legislation. It looks like we’ll get a watered down version of the Universal Declaration of Human Rights, which British representatives helped to draft. According to this entry at the British Library, “British representatives were frustrated that it had moral but no legal obligation” at the time of the original drafting. It sounds suspiciously like we think it’s good enough for everyone but us.

One of the biggest frustrations for me is that no leading campaigner had any specific plans for leaving. There are innumerable options, with various pros and cons, some of which outlined in the Open Europe report. People voted to leave on the expectation that some or other promises would be delivered. In reality, none of them may be: if we end up with a Norway type deal for instance.

Continually, throughout the campaign, I saw people repeating the myths and legends about the EU. The cabbage regulations, the lack of democracy, the £350 million a week we would save. I did think that at least leaving would mean people would stop blaming the EU for everything, but in the aftermath of the vote I realize that even this hope will probably be in vain.

You might have noticed that I haven’t used the term “Brexit”. I can’t bring myself to regularly use that word, because to me it trivializes the whole issue, reduces the impact of the decision to an easily digested soundbite. It’s symptomatic of the whole simplistic leave rhetoric.

Right now, it is quite common to see pro-leave opinion pointing out that the worst predictions of the effects of leaving the EU have not come to pass. This amazes me – not only have we not left; we have not even started the process of leaving; not even decided when to start the process of leaving (by invoking article 50). We could be sitting around in five years time and still wondering if we have actually, really left. I hope we avoid the worst possible outcomes of leaving the EU, but to do so I think we will have to disappoint many who voted leave on the basis of promises impossible to keep.

Using the Eclipse Paho “Test” Broker to Help Test MQTT

I may not have finished all my goals for the test material in the Eclipse Paho project, as outlined in this blog post of mine, but some components are still useful in their current state.

Recently my IBM and Paho colleague, James Sutton, needed to check the behaviour of the Java and Android clients when receiving an error code in response to an MQTT subscribe request. This error code, returned in the MQTT suback packet, was introduced in the last, and current, version of MQTT, 3.1.1. It takes the form of an 0x80 value in the granted QoS (Quality of Service) field, for which only 0, 1 or 2 are valid values – the set of integers which QoS can take in MQTT.

Now you could fire up a broker like Eclipse Mosquitto and configure it to disallow a subscription to a certain topic. If you know the broker well enough, this may be quick and easy. It’s possible you might have to do some fishing around, and figure out whether that broker is actually returning 0x80 as you wanted.

So James turned to the Paho test broker (startbroker.py in this repo.) (You must use Python 3 to run it, not Python 2). As it stands, this broker will return 0x80 if you try to subscribe to the topic “test/nosubscribe” (see line 322 in MQTTBrokers.py). That’s pretty easy. If you checked out that file, you will notice another two topics: “test/QoS 1 only” and “test/QoS 0 only”, which will return granted QoSs of a maximum of 1 and 0 respectively. These behaviours can be hard to elicit out of a standard MQTT broker.

Ultimately I guess I should make the specific topics which these responses are attached to configurable, but they aren’t right now.

There are some other characteristics of this broker, which single it out as a “test” broker rather than a product:

  • the goal of the coding is clarity rather than performance. You can tell me whether I succeeded
  • there is no persistence: if you want to simulate stopping and restarting a broker, this broker can just remain running, and disconnect all clients
  • MQTT specification conformance statements are embedded in the code, so that when a test suite is run against the broker, it can tell you which statements were encountered, and which weren’t.
  • the broker has parameters to choose behaviours which can vary but still conform to the MQTT specification:
    1. whether to publish QoS 2 messages on PUBREL or not
    2. whether multiple matching subscriptions result in one publication, or more than one
    3. whether queued QoS 0 messages are dropped if the client is disconnected, or not
    4. whether zero_length_clientids are allowed

    it’s possible more might be added in the future.

The main missing feature of this broker is TLS – but it does have WebSocket support. TLS is on my to do list, as will be updates for the next version of MQTT, 5, which we are working on, and is tentatively scheduled for completion next year.

As outlined in the blog post, this broker is meant to help with broker testing as well, as an oracle.

For some more information, see the Eclipse Paho website for the test tools.

You can use issues on this project to ask for new features or identify bugs, or pull requests to offer your own contributions.