The new MQTT V5 API for the Eclipse Paho C client

I’ve started to write the new MQTT version 5.0 support for the Eclipse Paho C clients. As I’ve reached the stage where some very simple tests are running, I thought this was a good time to try to elicit some feedback. I’ve had to make some compromises in the API, balancing the minimization of disruption to application programs with the natural incorporation the new MQTT 5.0 capabilities.

If you’ve used the main Paho C client, you’ll know it comes in two flavours:

MQTTClient
The synchronous version – intended to be easy to get started. Most calls block, waiting for a response from the server. This was the first API, and the only one when I created it. I modelled it after the existing Java API for consistency.
MQTTAsync
The asynchronous version – (almost) entirely non-blocking. Responses are reported in callback functions. This was a response to the need to run in asynchronous environments such as iOS and Windows.

The mqttv5 branch in the Github repo contains all the MQTT V5 updates to date. The test15 and test45 test programs have the first tests for MQTTClient and MQTTAsync libraries respectively. (At the moment, only test1 in each file has been migrated to MQTT V5, so only look at those).

MQTTClient for V5

First we have to create the client object with MQTTClient_create – this stays the same.


rc = MQTTClient_create(&client, options.connection,
 "single_threaded_test", MQTTCLIENT_PERSISTENCE_DEFAULT, NULL);

Next we must connect to the server.  As outlined in my previous post the main difference in MQTT V5 packets is the addition of the properties section, and for acknowledgements and disconnect, a reason code byte. Properties exist on both requests and their acknowledgements.

The connect call thus adds a properties object, for the connect request itself, plus the will message, and as part of the response. The MQTT V3 code:


MQTTClient_connectOptions conn_opts = MQTTClient_connectOptions_initializer;

int rc = MQTTClient_connect(client, &conn_opts))

becomes:


MQTTClient_connectOptions opts = MQTTClient_connectOptions_initializer;
MQTTProperties props = MQTTProperties_initializer;
MQTTProperties willProps = MQTTProperties_initializer;
MQTTResponse response = {SUCCESS, NULL};

response = MQTTClient_connect5(c, &opts, &props, &willProps);

The property and will property fields can be NULL if not required. The properties structure is manipulated with the add, free and copy functions. To add properties:


property.identifier = SESSION_EXPIRY_INTERVAL;
property.value.integer4 = 30;
MQTTProperties_add(&props, &property);

and


property.identifier = USER_PROPERTY;
property.value.data.data = "test user property";
property.value.data.len = strlen(property.value.data.data);
property.value.value.data = "test user property value";
property.value.value.len = strlen(property.value.value.data);
MQTTProperties_add(&props, &property);

Memory is allocated in the process of adding a property to the properties structure, so when you’ve finished with it this must be freed:


MQTTProperties_free(response.properties);

The response structure looks like this:


typedef struct MQTTResponse
{
  enum MQTTReasonCodes reasonCode;
  MQTTProperties* properties; /* optional */
} MQTTResponse;

I’m not particularly happy with the name of the function, MQTTClient_connect5, but it does have the benefit of being short with a minimal change of name. Other options I considered were:

  • MQTT5Client_connect
  • MQTTClient_connectWithProperties
  • MQTTClient_connectExt

I quite like the “connectWithProperties” option, but it didn’t get much approval from the audience I’ve questioned so far — it also doesn’t reflect all the parameters required.

The other MQTT functions have similar changes. An MQTTResponse return structure instead of a single int, and an extra input parameter for properties. For the publishMessage function, the properties are added to the MQTTClient_message structure:


MQTTClient_message pubmsg = MQTTClient_message_initializer;

property.identifier = USER_PROPERTY;
property.value.data.data = "test user property";
property.value.data.len = strlen(property.value.data.data);
property.value.value.data = "test user property value";
property.value.value.len = strlen(property.value.value.data);
MQTTProperties_add(&pubmsg.properties, &property);

response = MQTTClient_publishMessage5(c, test_topic, &pubmsg, &dt);

And for the subscribe function, there are added subscribe-specific options:


MQTTSubscribe_options subopts = MQTTSubscribe_options_initializer;

subopts.retainAsPublished = 1;
subopts.noLocal = 0;
subopts.retainHandling = 0; /* 0, 1 or 2*/

response = MQTTClient_subscribe5(client, test_topic, subsqos, &subopts, &props);

Finally the disconnect function has both input properties and reason code:


enum MQTTReasonCodes reasonCode = SUCCESS;

rc = MQTTClient_disconnect5(client, 0, reasonCode, &props);

At the moment, the return from disconnect is still just an integer, as there is no MQTT response to a disconnect packet other than closing the connection.

I haven’t implemented the new MQTT V5 auth exchange packets yet, but I envisage this callback and call:



MQTTResponse MQTTClient_authArrived(void* context, enum MQTTReasonCodes rc, MQTTProperties props);

response = MQTTClient_auth(client, reasonCode, &props);

The callback will be used to implement an extended auth exchange initiated by the server, the function when the exchange is started by the client.

MQTTAsync for V5

Most of the changes to the MQTTAsync API are in parameter structures and in callback signatures rather than the API signatures themselves. The “create” and “setCallbacks” calls remain the same. V5 properties for both the connect and will messages are added to the existing connect options along with new callback signatures for success and failure:


MQTTAsync_connectOptions opts = MQTTAsync_connectOptions_initializer;
MQTTAsync_willOptions wopts = MQTTAsync_willOptions_initializer;
opts.onSuccess5 = test1_onConnect;
opts.onFailure = NULL;
opts.context = c;

property.identifier = SESSION_EXPIRY_INTERVAL;
property.value.integer4 = 30;
MQTTProperties_add(&props, &property);

property.identifier = USER_PROPERTY;
property.value.data.data = "test user property";
property.value.data.len = strlen(property.value.data.data);
property.value.value.data = "test user property value";
property.value.value.len = strlen(property.value.value.data);
MQTTProperties_add(&props, &property);

opts.connectProperties = &props;
opts.willProperties = &willProps;

rc = MQTTAsync_connect(c, &opts);

The return from the connect call remains an integer, as the expanded response information is made available in the callbacks in the MQTTAsync_successData5 structure:


typedef struct
{
	...
	enum MQTTReasonCodes reasonCode; /* MQTT V5 reason code returned */
	MQTTProperties props; /* MQTT V5 properties returned, if any */
	/** A union of the different values that can be returned for subscribe, unsubscribe and publish. */
	union
	{
		...
		/* For connect, the server connected to, MQTT version used, and sessionPresent flag */
		struct
		{
			char* serverURI;
			int MQTTVersion;
			int sessionPresent;
		} connect;
		...
	} alt;
} MQTTAsync_successData5;

In the test, the contents of the properties returned by the connack are logged in the success callback:


void test1_onConnect(void* context, MQTTAsync_successData5* response)
{
	...
	MyLog(LOGA_INFO, "Connack properties:");
	logProperties(&response->props);
	...
}

In subscribe, unsubscribe and send (publish) calls, the response options structure contains the properties as well as the new success and callback function pointers. As the options are now not just for responses, there is a synonym for the structure, called callOptions:


MQTTAsync_message pubmsg = MQTTAsync_message_initializer;
MQTTAsync_callOptions opts = MQTTAsync_callOptions_initializer;
MQTTProperty property;
MQTTProperties props = MQTTProperties_initializer;

property.identifier = USER_PROPERTY;
property.value.data.data = "test user property";
property.value.data.len = strlen(property.value.data.data);
property.value.value.data = "test user property value";
property.value.value.len = strlen(property.value.value.data);
MQTTProperties_add(&props, &property);
opts.properties = props;

pubmsg.payload = "a much longer message that we can shorten to the extent that we need to payload up to 11";
pubmsg.payloadlen = 11;
pubmsg.qos = 2;
pubmsg.retained = 0;
rc = MQTTAsync_sendMessage(c, test_topic, &pubmsg, &opts);
MQTTProperties_free(&props);

The call options structure also includes the new V5 subscribe options:


MQTTAsync_callOptions opts = MQTTAsync_callOptions_initializer;
opts.properties = props;

opts.subscribe_options.retainAsPublished = 1;
rc = MQTTAsync_subscribe(c, test_topic, 2, &opts);
MQTTProperties_free(&props);

and for disconnect, similarly as for connect, the disconnectOptions structure is extended:


MQTTAsync_disconnectOptions opts = MQTTAsync_disconnectOptions_initializer;
MQTTProperty property;
int rc;

opts.onSuccess = test1_onDisconnect;
opts.context = c;
opts.reasonCode = UNSPECIFIED_ERROR;

property.identifier = SESSION_EXPIRY_INTERVAL;
property.value.integer4 = 0;
MQTTProperties_add(&opts.properties, &property);

rc = MQTTAsync_disconnect(c, &opts);
MQTTProperties_free(&opts.properties);

As for the MQTTClient API I’ve not added the AUTH packet capabilities yet. They will echo the other APIs:


int rc = MQTTAsync_setAuthReceived(c, context, callback_pointer);

rc = MQTTAsync_auth(c, &opts);

The MQTTAsync_auth call, to send an AUTH packet, will be able to be called from the AuthReceived callback in response to an AUTH packet from the server, or separately to enable the client to initiate an authentication exchange.

That rounds up the summary for now, I’m very interested to hear any thoughts.

A Story of MQTT 5.0

The MQTT protocol has been around since the late 90s when it was created to enable the monitoring of a long distance oil pipeline. It went through several iterations before landing on version 3.1, published by IBM.

The next step was standardisation, at the OASIS standards body. As anyone who has taken part in a standardisation committee will know, this process is necessarily bureaucratic and slow. To speed up adoption, the main imperative was minimising disruption to existing implementations, as set out in the Technical Committee (TC) charter.

As a result, wholesale changes to the MQTT 3.1 specification were not allowed in the 3.1.1 standard. This meant that many irritating flaws could not be fixed nor widely sought enhancements included. This is where MQTT 5.0 comes in. While we still wanted to minimise disruption (no-one wanted to repeat the experiences of say AMQP 0.9 to 1.0), we also wanted to address the MQTT wish-list as far as possible so that major changes would not be needed for a long time to come. Whether we succeeded in that aim, time will tell.

To help us make sense of the multitude of items on that wish list, as I wrote in September 2016, they were grouped into four Big Ideas:

  • Improved error reporting
  • Extensible metadata
  • Scalability and large scale systems
  • Resource Constrained Clients and Performance Improvements

At that time, many of the solutions were not decided upon, but now with the availability of Committee Specification 01, I can write about the details. We are in the final stages of the standardisation process for 5.0. We hope to complete the process of rubber stamping in the next few months, and expect no substantive changes during that time.

Improved Error Reporting

Negative responses, or nacks, were the biggest omission from earlier versions of MQTT. If the client or server had a problem with the request or packet from the other end, the only recourse in many circumstances was to close the TCP connection. Connect packets were the exception to this: the connack always had a return code. MQTT 3.1.1 added a negative response code to subscribe requests because there was space available in the “granted QoS” field of the suback packet. Publish requests, however, were still not catered for. This is now remedied.

Reason Codes

As all ack packets now have reason codes, they have been consolidated into one set, which starts like this:

Reason codes are one byte. Values from 0 to 0x127 inclusive indicate successful outcomes, those from 0x128 to 0xFF unsuccessful. So as the subscribe response can have an error code:

msc {
    arcgradient="4";

    c [label="client"], b [label="broker"];

    c => b [label="connect"];
    b => c [label="connack(rc=0)"];

    c => b [label="subscribe"];
    b => c [label="suback(rc=0x80)"];

    ...;
}

so can the publish, when the QoS is greater than 0:

msc {
    arcgradient="4";

    c [label="client"], b [label="broker"];

    c => b [label="connect"];
    b => c [label="connack(rc=0)"];
    ...;
    c => b [label="publish, qos=1"];
    b => c [label="puback(rc=0x80)"];
    ...;
    c => b [label="publish, qos=2"];
    b => c [label="pubrec(rc=0x80)"];
    ...;
    c => b [label="publish, qos=2"];
    b => c [label="pubrec(rc=0x0)"];
    c => b [label="pubrel(rc=0x80)"];
    ...;
    c => b [label="publish, qos=2"];
    b => c [label="pubrec(rc=0x0)"];
    c => b [label="pubrel(rc=0x0)"];
    b => c [label="pubcomp(rc=0x80)"];
    ...;
}

For the QoS 2 exchange, it stops if any of the reason codes are 0x80 or above. This is a major improvement on previous versions of MQTT, where continuing with the exchange or terminating the connection were the only options.

Server initiated disconnect

In MQTT versions prior to 5.0, only the client could send a disconnect packet. This meant that in any case where the server wanted to end the conversation with a client, there was no option but to just terminate the TCP connection. A common case is when the server shuts down – there is no error in the interaction between broker and client, but the client has no idea what’s happening. In MQTT 5.0, the server can send a disconnect packet with a “Server shutting down” reason code:

msc {
    arcgradient="4";

    c [label="client"], b [label="broker"];

    c => b [label="connect"];
    b => c [label="connack(rc=0)"];
    ...;
    c => b [label="publish, qos=1"];
    b => c [label="disconnect(rc=139)"];
}

In this case, the client might wait for a while before attempting a reconnect, knowing that the server might not be available for a while.

Extensible metadata

The big change here is in addition to reason codes, each packet (apart from pings) can have properties. This is an extract of the full list:

Properties can be used to add extra information to responses, such as a reason string, or extra parameters to requests. A lot of the rest of the changes rely on properties because now we had a mechanism for adding that extra information to packets, we had to use it!

Request/response

The new request/response capability makes good use of properties. The requester subscribes to the topic it expects to receive responses on, then sets the value of the “Response Topic” property to that topic name. The responder simply uses that property to set the topic name for its response.

msc {
    arcgradient="4";

    c1 [label="client1"], b [label="broker"], c2 [label="client2"];

    c1 => b [label="subscribe, topic=resp"];
    b => c1 [label="suback"];

    c1 => b [label="publish, response_topic=resp"];
    b => c2 [label="publish, response_topic=resp"];

    ...;

    c2 => b [label="publish, topic=resp"];
    b => c1 [label="publish, topic=resp"];
}

The “Correlation Data” property can be used to set an id for each request, so that replies can be matched to requests by the requester.

Payload format indicator

There were fairly contentious discussions about how much flexibility there should be in payload format settings. Some were in favour of user definable payload formats. Others felt that if people could define their own formats it was no better than the current position, unless some body kept an approved list of format indicators and their meanings. That seemed a step too far for MQTT. MIME types were discussed, but the final approach is minimalistic – just two values: binary, as 3.1.1, or UTF-8 data.

Enhancements for Scalability

Improved error reporting helps scalability because exchanges between servers and clients become more efficient. Properties are again crucial to the following functions.

Simplified session state

One of the other big irritations with 3.1.1, along with the lack of nacks for publish commands, is the behaviour of the “clean session” flag. In earlier versions of MQTT, this started out as the “clean start” flag, where the session state was only cleaned up at the start of a session, not at the end. This was good for clients, because it meant you could ensure a clean starting point, and leave the session around in case you needed to reconnect. Not so good for servers, because clients would tend to leave the state lying around for ever.

Later on, this flag was changed to “clean session”, cleaning the session state both at the start and end of the session. Good for servers. For clients, if they want to ensure a clean slate to start with, but then want to have session state saved, they have to connect twice:

msc {
    arcgradient="4";

    c [label="client"], b [label="broker"];

    c => b [label="connect cleansession=true"];
    b => c [label="connack"];
    c => b [label="disconnect"];
    ...;
    c => b [label="connect cleansession=false"];
}

We knew we should fix this situation once and for all. The “clean session” flag becomes “clean start” once more – session state is only cleaned up at the start of the session. Then there is the “session expiry interval” property, a four-byte integer value in seconds which defaults to zero if omitted. If it is set to 0xFFFFFFFF (UINT_MAX), the session does not expire. To accomplish the above scenario:

msc {
    arcgradient="4", wordwraparcs=on;

    c [label="client"], b [label="broker"];

    c => b [label="connect cleanstart=true, expiry_interval=0xFFFFFFFF"];
    ...;
}

The MQTT-SN “offline keep alive” scenario is also catered for. By setting the expiry interval to a suitable non-zero value, the client can ensure that the session state is saved as long as it reconnects regularly. If the client disappears entirely, the session state will be cleaned up at some point. Both clients and servers are happy.

Shared subscriptions

To allow load balancing of high throughput topics, the concept of shared subscriptions is introduced to MQTT. Messages on these topics are sent to one of a group of subscribers rather than to them all. The subscriber indicates that the subscription is shared simply by subscribing to a special topic pattern:


$share/{ShareName}/{filter}

where ShareName is the name of the shared subscription group, and filter is the usual topic filter used in the subscribe request.

Optional server capabilities

Some server functionality is expensive to implement at large scale. In MQTT 5.0, the server can advertise the limitations on the functionality it provides in the connack properties. Some examples:

Retain Available
are retained messages supported?
Maximum QoS
the maximum publish QoS the server will accept
Maximum Packet Size
the maximum packet size the server will accept
Receive maximum
the maximum number of concurrent QoS 1 and 2 message the server will handle

Resource Constrained Clients and Performance Improvements

Various features fall into this category, including some already described. Some further examples follow.

Nolocal subscriptions

Up until MQTT 5.0, the publisher of a message will receive that message back if it is subscribed to the same topic. People often find this out in their first experience of writing an MQTT application, when they implement a shared chat room. There is now a subscribe option noLocal which when set, indicates that the publishing application should not receive its own messages.

msc {
    arcgradient="4", wordwraparcs=on;

    c [label="client"], b [label="broker"];

    c => b [label="subscribe topic=a"];
    b => c [label="suback"];
    c => b [label="publish topic=a"];
    b => c [label="publish topic=a"];
    ...;
    c => b [label="subscribe topic=b, noLocal"];
    b => c [label="suback"];
    c => b [label="publish topic=b"];
    ...;
}

Retained message control

Options on the subscribe request have been added to:

  • 0 = Send retained messages at the time of the subscribe
  • 1 = Send retained messages at subscribe only if the subscription does not currently exist
  • 2 = Do not send retained messages at the time of the subscribe

This could help particularly with the implementation of MQTT bridges from one broker to another.

Topic aliases

This capability exists in MQTT-SN, to reduce the size of the publish packet when long topic names are used. The publish request allows a numeric topic alias to be specified, which can be used in subsequent publish packets. Topic aliases on the client and the server are independent of each other, in much the same way as packet ids are.

msc {
    arcgradient="4", wordwraparcs=on;

    c [label="client"], b [label="broker"];

    c => b [label="publish topic=long_name,alias=1"];
    c => b [label="publish alias=1"];
    ...;
    b => c [label="publish topic=server_long_name, alias=1"];
    b => c [label="publish alias=1"];
    ...;
}

Topic aliases only exist for the lifetime of a TCP connection.

Specifying client limitations

To help protect implementations on small devices, the client can specify its limitations using properties on the connect packet. Some examples:

Maximum Packet Size
the maximum packet size the client can accept
Receive maximum
the maximum number of concurrent QoS 1 and 2 message the client can handle

It is an administrative action or decision on the part of the server to decide what to do with messages that it receives bound for a client for which that message exceeds the constraints. This is not particularly different from 3.1.1 where the message would be sent anyway, and then the client might be forced to disconnect as its only recourse. At a minimum, the server should probably emit a warning log message.

Eclipse Paho Progress

A release of the Eclipse Paho project is planned for June 2018 with its first implementations of MQTT 5.0. I first implemented a broker to test against in the Paho test project. It combines 3.1.1 and 5.0 implementations, and has been used by James Sutton to implement the Java MQTT 5.0 support. It is used in Travis and AppVeyor continuous integration tests for the MQTT 5.0 branches. Example output when you start it up is shown below.

My own C clients, embedded and main are planned to have a June release. The MQTT 5.0 implementations continue in the embedded mqttv5 and mqttv5 branches. Please do give your feedback or thoughts on these implementations as they progress via the GitHub issues:

A New Fate for the Eclipse Paho “Test” Broker?

If you’re familiar with the Eclipse Paho test material, you’ll know that I wrote an MQTT 3.1.1 broker in Python with the purpose of providing a benchmark or oracle against which other implementations could be compared. Last year, I added support for the new MQTT V5.0 definition, so that client implementations of V5, including my own, could be easily tested. It has been used by Paho colleagues of mine, James Sutton and Allan Stockdill-Mander to develop MQTT V5 Java and Golang client libraries.

Then, I added the missing TLS support, so that it could satisfy testing requirements for secured connections. Adding TLS support in Python turned out to be really easy, much easier than I expected. I know how hard it is to add OpenSSL support in C, and I was prepared for at least some proportion of that effort, but there was none. I enabled multiple MQTT listeners to be configured, a la RSMB/Mosquitto. That meant the broker was comprehensive enough to be used for automated testing of existing client implementations. I used it to replace a remote Mosquitto broker in the automated Paho C client testing for Windows using AppVeyor, as installing Mosquitto on Windows is a little tricky, and it works fine in that capacity. All I had to do in the AppVeyor configuration file was to add these lines:


- cmd: git clone https://github.com/eclipse/paho.mqtt.testing.git
- cmd: cd paho.mqtt.testing\interoperability
- ps: Start-Process C:\Python36\python -ArgumentList 'startbroker.py -c client_testing.conf'

to get the broker running locally. WebSockets support was already included, so it means that each MQTT listener supports both 3.1.1 and V5, WebSockets and TLS. The 3.1.1 and V5 clients can communicate with each other (although this work is not complete and tested yet), both through normal publish commands and retained messages.

In a recent project at work I had added persistence to a Python project by using the ZODB object database, and that had turned out to be really easy too. These experiences got me to thinking that it should be easy to add MQTT-SN support as well, and create a broker with the capabilities that I had envisaged for RSMB before it was superseded by Mosquitto. Roger Light, the author of Mosquitto, has intended to add MQTT-SN support for a long time, but wanted to restructure the internals first. Due to other commitments, and some licensing obstacles, this has not yet come about, and with the advent of MQTT V5 which will have a higher priority, looks even further out.

A broker written in Python will not be as efficient as Mosquitto, nor is it particularly scalable as currently written because it uses Python’s threading model which does not take advantage of multiple cores. But as it is so easy to add new capabilities, it could well be one of the most complete MQTT implementations, soonest. In this way it would be complementary to Mosquitto.

I have now experimented with adding MQTT-SN support, a framework for bridges, and an HTTP listener which can provide APIs to query and update the broker state and configuration. We can add APIs to allow queries to return information about the currently connected clients, their subscriptions and messages for instance. The set of capabilities I envisage looks like this:

  • TCP listener supporting WebSockets, TLS, MQTT 3.1.1 and 5.0
  • UDP listener supporting MQTT-SN with DTLS, multicast
  • other potential MQTT-SN listeners: BLE, ZigBee
  • HTTP listener for state/configuration APIs. The broker behaviour for test purposes can potentially be reconfigured dynamically
  • TCP bridge capable supporting MQTT 3.1.1 and 5.0, WebSockets and TLS
  • UDP bridge supporting MQTT-SN with DTLS, multicast
  • HTTP bridge supporting webhooks?
  • dynamic bridge connections – adding/deleting bridges
  • variety of persistence options including ZODB

I think it’s obvious to state that with these capabilities it could serve as an MQTT-SN gateway for services that support MQTT, as a conversion mechanism between MQTT V3.1.1 and V5, and a dynamic flexible component of an MQTT network. I would still keep a focus on enabling MQTT testing, including for test suite generation. A lot of this work could be completed in the first half of 2018, other commitments depending.

This brings me on to my main motivation for writing this blog post. Perhaps this broker should have a life of its own, separate to Paho, as an Eclipse IoT project. It could quite well do the job as part of Paho, conversely it could have a higher profile and encourage more community participation if stood on its own. So, if you have an interest in seeing this broker as an Eclipse IoT standalone project, from the point of view of using it, or also contributing to it, please let me know – thanks.

What’s happening with RSMB?

Some people will be aware that I wrote a small MQTT broker in about 2008, made available on IBM’s alphaWorks website. It was called RSMB (Really Small Message Broker) when released, because the name we used first for it, Nanobroker, was already taken. Somewhat amusingly, an IBM website for RSMB still exists.

A year or two later, Roger Light asked Andy Stanford-Clark why RSMB wasn’t open source (not my decision), so Andy suggested Roger write his own. And that’s how Mosquitto started, as a drop in replacement for RSMB. When MQTT software was being contributed to the Eclipse foundation, IBM contributed a Java client and my C client to the Paho project, and Mosquitto was contributed as the broker. IBM did contribute the RSMB source to the Mosquitto project on my encouragement, to serve as a repository of potentially useful code, and because it had support for MQTT-SN.

So in my mind, Mosquitto became the official replacement for RSMB, and I expected RSMB to outlive its usefulness pretty quickly. As it happens, MQTT-SN support in Mosquitto has been on the back burner ever since, because Roger wanted to rebase the internals of Mosquitto on an event library before tackling it. Unfortunately, this ran into a number of issues, social and technical. I’m still hoping that it will happen.

But one of the alternative approaches to MQTT-SN support is now available in Paho (written by Tomoaki Yamaguchi) – a transparent gateway which converts MQTT-SN into MQTT. Transparent because it creates a new MQTT connection for each MQTT-SN client, so that the MQTT broker has visibility of those clients. RSMB acts as an aggregating gateway, where one MQTT bridge connection carries the traffic for all MQTT-SN clients, and the MQTT broker sees only one.

I do wonder if an MQTT-SN to MQTT gateway will in fact be a better solution because it may allow easier support of additional underlying transports. The gateway has UDP and XBee right now, others such as BLE and even serial could be useful.

Mosquitto in the meantime has added further capabilities such as TLS and WebSocket support, and many more. If there were a niche that Mosquitto has left open then I would be happy to support RSMB in that, but I don’t think there is. The combination of Mosquitto and the Paho transparent gateway will do a better job all round.

Where are we with MQTT-SN?

This question was posed to me recently, with the added observation that it seemed no progress had been made in the last two years.  From my perspective this isn’t quite true.  Although the specification is still maintained by IBM, no movement has been made to standardize it at OASIS like MQTT has been. That is not to say that the subject of MQTT-SN has not arisen in the MQTT OASIS Technical Committee (TC) discussions, it has. But amongst the numerous improvements that we wanted to make to MQTT 3.1.1, and an ambitious timescale — publishing the new version MQTT 5.0 this year (2017) — addressing non TCP networks in MQTT was put to one side.

There is a continual wavering in my mind as to the importance of MQTT-SN. On one hand, edge computing platforms are getting ever more powerful, on the other, low power consumption is very important for battery power. The powerful edge processors are likely to have TCP stacks for which MQTT is appropriate: but there are still use cases for which low power use is the crucial factor. UDP, Bluetooth Low Energy (BLE) are typical transports used in this case.

If there is enough interest in a standardized MQTT-SN, its features could be addressed by the MQTT OASIS TC. There are a number of options. Incorporating MQTT-SN or its features into the main MQTT specification is one. However MQTT currently requires a reliable underlying network transport (TCP). To change that assumption we would have to consider very carefully the implications, which could take a significant amount of time. It has been suggested that MQTT-SN could be dealt with as a committee note: I’m not sure what form that would take.

In the meantime, I’ve been trying to get more MQTT-SN capabilities added to the Eclipse® Paho project. After I contributed an initial MQTT-SN packet library a couple of years ago, and the RSMB broker with MQTT-SN support to the Eclipse Mosquitto project at its inception, things stalled for a while (but not for a lack of interest on my part). There are forks of RSMB with fixes to the MQTT-SN support, and various gateways and clients around. One of those, written by Tomoaki Yamaguchi, had gained some support and I forget exactly how it happened, but Tomoaki has now contributed an MQTT-SN transparent gateway to Paho.

Over the past months this gateway has matured nicely. I recently used it to replace the use of RSMB in this experiment of Benjamin Cabé’s and it worked well. If we add BLE support to the gateway, then the Node.js BLE to UDP MQTT-SN forwarder in that experiment would not be required either. The gateway is written with the intention of allowing other transports to be added, so this should be eminently feasible.

I do remember seeing an email or post or something recently describing some other MQTT-SN components written by the poster. But now I can’t find or remember where I saw it. If this wasn’t a dream, and you know of such a thing, or it was you, please do let me know.

Where do we go from here?  Two avenues: practical implementations and the specification.  After the MQTT 5.0 standard is published, we can see if there is any interest in the OASIS TC in pursuing the application of MQTT to non-TCP networks.  In the meantime, I will continue working on and encouraging MQTT-SN contributions of the current specification to Paho and elsewhere.

If you are interested in MQTT-SN and would like to see it considered by the OASIS MQTT TC, or make any other comments to the TC, then you can use the mailing list.  If you know of any other implementations or have any questions, or suggestions for Paho, then I will be happy to hear of them

The UK Referendum on Leaving the EU

I want to record my thoughts on the UK referendum, so that in the future I can refer to them, and remember my feelings in the immediate aftermath.

I was going to the Glastonbury festival on Friday 24th June. Normally I go on the Wednesday, but this time I’d just come back from Montreal the week before, so I left it until later. On the Thursday evening, I started watching the results, but ended up watching all night, in some disbelief. At Glastonbury, the mood was sombre amongst my friends and the people we met, and amongst the performers who spoke out.

I think my overwhelming feeling was one of loss: I thought we were all in this together, we being Europe, and then the world. This felt like a step backwards, splitting apart instead of coming together to work out our problems. I wanted our borders, a purely human divisive contrivance, to become less important, not more. Of course, in this day and age of increasingly easy mass communication, this is not really going to happen unless civilization takes a severely different turn, which I hope is unlikely.

My perception is that people voted to leave the EU for a variety of reasons, not all of which are compatible. In the process of leaving, some of those who voted to leave will be dissatisfied. For those on far enough left of the political spectrum, the EU is too right-wing; for those far enough on the right, the EU is too left-wing. I think those on the left disapprove of TTIP, and the effects of global capitalism. I’m not sure what those on the right exactly disapprove of. I did entertain the idea that some were still hoping for the return of the British Empire, then dismissed that thought as insulting. To my astonishment through the referendum campaign, I realized that there really are some in that camp.

Undoubtedly one of the key reasons people voted to leave is immigration. David Cameron promised in 2010 to reduce immigration to the tens of thousands per year, a policy with which I broadly don’t agree. The promise was reiterated in 2015. The failure to do this was blamed on the EU. Yet we have more immigration from outside of the EU – only slightly more, but still more. Even the immigration over which we already have control has not been reduced to the “tens of thousands” per year. Why not? That is just abject failure and dishonesty on the part of the government – making a promise on which they know they can’t deliver. Then it was used as a stick with which to beat the EU.

Another reason was to vote leave was to save the money contributed to the EU. This rough figure is easy enough to estimate, event if the Leave campaign got it wrong by “forgetting” to subtract the money we get back. Even then the figure doesn’t take into account the benefit to us of being within the single market, the growth in the economy that otherwise wouldn’t have happened. Of course, we don’t really know how big that the effect was, we can only speculate. Then the question is, is it worth contributing £160 million per week? If we leave, will we actually save that amount, or will it be eaten away by other costs and expenses?

Perhaps the biggest message was “taking back control”, “loss of sovereignty”, which included immigration. As noted, we couldn’t even reduce the immigration over which we already did have control. In my life in Britain, I’ve never lived in a place where my vote in a general election had any effect. Growing up in South Wales, Labour would always win. Since I started work, I’ve only lived in constituencies where the Conservatives always win. Under the first past the post system, I vote, but it has no effect. Conversely, my MEP is elected by proportional representation. We are largely governed by the privately educated; the head of state inherits his or her position, and the second chamber of parliament is full of appointees. I don’t want to give more power to this leadership – the EU is more democratic.

I subscribe to the Economist. One article referred to a report written by the Open Europe, which I downloaded and read significant parts of. It seemed to be the most balanced analysis of the impact of the UK leaving the EU. It also contains what I feel to be a key statement:

“If the UK puts as much effort into reforming the EU as it would have to in order to make a success of Brexit, the UK and the EU would both be far better off.”

To make the most “success” out of leaving, we would have to abolish or reduce regulations on subjects like environmental and employment protections and climate change mitigation. Those are not topics on which I wish to see less regulation. Over many important aspects of decision making, we already had control – it was our own government not the EU that dictated our position. The EU was just a scapegoat.

Instead of making a concerted effort to make a better EU, years of effort will now be put into unnecessary legal process and trade negotiations. This could take the best part of a decade, the only winners out of which will be the lawyers, politicians, trade negotiators and accountants. Maybe some company board members will accrue a few more millions as a result of reduced employment and environmental legislation. It looks like we’ll get a watered down version of the Universal Declaration of Human Rights, which British representatives helped to draft. According to this entry at the British Library, “British representatives were frustrated that it had moral but no legal obligation” at the time of the original drafting. It sounds suspiciously like we think it’s good enough for everyone but us.

One of the biggest frustrations for me is that no leading campaigner had any specific plans for leaving. There are innumerable options, with various pros and cons, some of which outlined in the Open Europe report. People voted to leave on the expectation that some or other promises would be delivered. In reality, none of them may be: if we end up with a Norway type deal for instance.

Continually, throughout the campaign, I saw people repeating the myths and legends about the EU. The cabbage regulations, the lack of democracy, the £350 million a week we would save. I did think that at least leaving would mean people would stop blaming the EU for everything, but in the aftermath of the vote I realize that even this hope will probably be in vain.

You might have noticed that I haven’t used the term “Brexit”. I can’t bring myself to regularly use that word, because to me it trivializes the whole issue, reduces the impact of the decision to an easily digested soundbite. It’s symptomatic of the whole simplistic leave rhetoric.

Right now, it is quite common to see pro-leave opinion pointing out that the worst predictions of the effects of leaving the EU have not come to pass. This amazes me – not only have we not left; we have not even started the process of leaving; not even decided when to start the process of leaving (by invoking article 50). We could be sitting around in five years time and still wondering if we have actually, really left. I hope we avoid the worst possible outcomes of leaving the EU, but to do so I think we will have to disappoint many who voted leave on the basis of promises impossible to keep.

Using the Eclipse Paho “Test” Broker to Help Test MQTT

I may not have finished all my goals for the test material in the Eclipse Paho project, as outlined in this blog post of mine, but some components are still useful in their current state.

Recently my IBM and Paho colleague, James Sutton, needed to check the behaviour of the Java and Android clients when receiving an error code in response to an MQTT subscribe request. This error code, returned in the MQTT suback packet, was introduced in the last, and current, version of MQTT, 3.1.1. It takes the form of an 0x80 value in the granted QoS (Quality of Service) field, for which only 0, 1 or 2 are valid values – the set of integers which QoS can take in MQTT.

Now you could fire up a broker like Eclipse Mosquitto and configure it to disallow a subscription to a certain topic. If you know the broker well enough, this may be quick and easy. It’s possible you might have to do some fishing around, and figure out whether that broker is actually returning 0x80 as you wanted.

So James turned to the Paho test broker (startbroker.py in this repo.) (You must use Python 3 to run it, not Python 2). As it stands, this broker will return 0x80 if you try to subscribe to the topic “test/nosubscribe” (see line 322 in MQTTBrokers.py). That’s pretty easy. If you checked out that file, you will notice another two topics: “test/QoS 1 only” and “test/QoS 0 only”, which will return granted QoSs of a maximum of 1 and 0 respectively. These behaviours can be hard to elicit out of a standard MQTT broker.

Ultimately I guess I should make the specific topics which these responses are attached to configurable, but they aren’t right now.

There are some other characteristics of this broker, which single it out as a “test” broker rather than a product:

  • the goal of the coding is clarity rather than performance. You can tell me whether I succeeded
  • there is no persistence: if you want to simulate stopping and restarting a broker, this broker can just remain running, and disconnect all clients
  • MQTT specification conformance statements are embedded in the code, so that when a test suite is run against the broker, it can tell you which statements were encountered, and which weren’t.
  • the broker has parameters to choose behaviours which can vary but still conform to the MQTT specification:
    1. whether to publish QoS 2 messages on PUBREL or not
    2. whether multiple matching subscriptions result in one publication, or more than one
    3. whether queued QoS 0 messages are dropped if the client is disconnected, or not
    4. whether zero_length_clientids are allowed

    it’s possible more might be added in the future.

The main missing feature of this broker is TLS – but it does have WebSocket support. TLS is on my to do list, as will be updates for the next version of MQTT, 5, which we are working on, and is tentatively scheduled for completion next year.

As outlined in the blog post, this broker is meant to help with broker testing as well, as an oracle.

For some more information, see the Eclipse Paho website for the test tools.

You can use issues on this project to ask for new features or identify bugs, or pull requests to offer your own contributions.

Why doesn’t MQTT have a payload format?

I was talking to Andy Stanford-Clark on Friday, and asked him if he and Arlen ever considered using a payload format in MQTT.  I was interested to know this because we are discussing approaches to MQTT metadata in the standardization process.

Many of the industrial protocols that MQTT was intended to replace, enhance or work alongside go to great lengths to specify the details of the payload.  Data types, metadata, objects, all of these things may be defined in the data.  MQTT does none of these things – the payload is completely undefined.

Andy’s answer follows.

MQTT is intended as a transport, not a protocol.   It’s job is to get the data from one machine to another, not to define what that data is.   Industrial protocols have often conflated the two jobs, so that they are inseparable.  This makes them more complex and difficult to understand than they need to be.

MQTT can be used as a basis for a protocol which also defines the data to be exchanged, but the job of getting the data from one machine to another is already done.  Andy gave the example of a company who used MQTT to defined a new protocol for the first time.  Because MQTT took care of the connection between client and server and transporting the data between them, the process of defining the new protocol took half a day.  Andy asked them how long defining a protocol would have previously taken: the answer was “two weeks”.

Since MQTT was first defined in 1998, data serialization formats have come into and gone out of fashion.  XML was the “obvious” choice for a while.  JSON and Google Protocol Buffers may be the favourite options today.  MQTT is agnostic to them all, it can carry any of them, so it has not outlived its usefulness.

When people first learn about MQTT, they often ask about the lack of payload format, the implicit question being, what use is MQTT then?   The answer being, whatever you want to use it for.  Combine it with some data formatting and exchange sequences, then you have a protocol.  Maybe we will see some standardized protocols in the future which will be based on MQTT, using it for the data transport.

They Buried her Last Tuesday

You may recall my mentioning another acquaintance, “Shakespeare” Smith, an unsuccessful playwright who lived at Horton Bank Top. He told me he had given lots of thought to his name, and wanting to try something different – song writing, he would adopt another. Many passed through his mind like Irving Australia, Amadeus Wagner, but nothing seemed right until he finally decided on Sullivan O’Gilbert. Anyway, this is his latest attempt. He is just trying to find someone to write the music.

‘Twas by the old Rustic Bridge
on the day I met you
You’d broken your crutches
And your good eye was blue.

You’d a wart on your nose
And a boil on your chin
Your front teeth were missing
And you were incredibly thin.

Your hair somewhat sparse
And that was dyed black
But not quite as noticeable
As the hump on your back

Tears on your cheeks
I asked why you cried
“I can’t help it” you said
“My pet dog just died”.

But I married her anyway
Though she was sick and unhealthy
And I’m very happy
She’s exceedingly wealthy

She was buried last Tuesday
And now is no more
She was carried out feet first
Through the front door.

Maybe I’ll miss her
Now she’s gone off
And in my dreams hear
That terrible cough

I’ll remember her fondly
For all that I got –
Two houses in London
A Rolls Royce and a yacht

Some oil wells in Texas
A villa in Crete
A stud farm in Ireland
And a fishing boat fleet

I can’t stay here talking
All’s done and said
I’ve my packing to do
I’m crusing the Med.

–OoOoO–

© Colin Flanagan, 1998

What Will The Internet of Things Ever Do For Us?

There have been any number of warnings in recent years about the potential negative impacts of what has come to be known as the Internet of Things. Warnings about security:

Their own devices
Hacking the planet

about privacy and the control of global corporations:

Against the Smart City – Adam Greenfield
The Epic Struggle of the Internet of Things – Bruce Sterling

and user interfaces amongst others.

These ideas have a long history in fiction, Philip K. Dick’s Ubik’s argument with his front door, the discussions with toasters in Hitchhiker’s Guide to the Galaxy and Red Dwarf, for instance. Now these warnings and stories have particular relevance because we are on the cusp of seeing them turn into reality.

I am conscious of the pitfalls we face, but I also have a positive outlook on the Internet of Things. When I started working with embedded devices and MQTT, it wasn’t called that – we were just getting data and command from and to devices. As time went on, the “Internet of” prefix became a shorthand phrase that we would use to describe connected objects of any sort — see “The Internet of Cows”. The “Internet of Things” phrase has caught on, though, for better or worse.

These are some of the obvious positive potential benefits:

  • medical monitoring – of pacemakers for instance
  • environmental monitoring and prediction – flooding, snowfall,
  • infrastructure monitoring – power, water, fuel

but I also have an image of a world where other things are possible, important or trivial:

  • opening the curtains with wave of my hand
  • nano machines that can cure disease from the inside
  • street furniture that changes colour to match my outfit (Only Forward)
  • painted portraits that respond to me and my questions
  • an active view of the sky on my bedroom ceiling
  • a car which can change colour each day to match my mood or fancy

which have all appeared in fiction before, whether as science or magic. I will probably be explaining the unnecessary if I recall Arthur C. Clarke’s aphorism “any sufficiently advanced technology is indistinguishable from magic”.

So I have an image of a world where magical acts are possible, where books I have read come to life. Where the intrusive technology we have today can fade into the background if we want it to. Where art, science and society and can find new ways to influence and improve our daily lives.

Of course we do need to be aware of the dangers that can potentially accrue. But whatever activities we are involved in have dangerous implications, from accidents or intentional criminal behaviour. So those dangers in themselves are no reason to dismiss the entire field Technology may also have awkward interfaces that cause more problems than they solve, which stems from a lack of forethought, of design.

Adam Greenfield, in his book Everyware: The dawning age of ubiquitous computing has considered just about all of the possibilities. He set out some principles for development and use of “Everyware”, which I quote directly.

Ubiquitous systems must:

  1. default to harmlessness
  2. be self-disclosing
  3. be conservative of face (not take actions to unduly embarrass users)
  4. be conservative of time (must not introduce undue complications into ordinary operations)
  5. be deniable (users must be able to opt out at any time)

I think it is integral to the above aims that the ethos driving these systems is openness:

so that control of our technological environment is not entirely left to our governments and corporations. The more transparent the whole process is, the better. (I do like to remember from time to time, however, that governments and corporations are people too.) Essential infrastructure needs to be secure from attack, from malevolent intent, but each of us should be able to choose what happens to data we own. I see reason to be optimistic that the advance of technology is empowering the individual as well as the organization.

Today we allow organizations to obtain data about us in exchange for free services we value. Maybe we should consider whether a service which charges but gives us control over our data is a better bargain.