MQTT-SN Standardization Progress

Good News! The MQTT-SN standardization is progressing nicely. That isn’t to say at warp speed, but at a steady pace in the right direction with no significant obstacles so far. Of course, someone might decide to discover one in response to this article, but it’s better to know about any now if they exist.

Looking back, I think we had two major objectives, apart from the benefit of standardization itself:

  • To take into account experiences of implementation and improve them. MQTT-SN was devised in a different era of MQTT and networks: things have moved on.
  • To align more closely with MQTT 5.0 which had been developed in the meantime. Ironically, in the odd case such as cleanstart/cleansession, this means moving back in time to pre-MQTT 3.1.1.

Simon Johnson brought the experience of ThingStream’s use of MQTT-SN in a production environment into the fray. We started with that, along with some alignment with MQTT 5.0. I thought at the beginning that I would prefer the fewest changes we could get away with, but as our discussions proceeded I realized there were more changes I would like to see adopted.

So we’ve ended up with a list of about 40 issues. Through the end of 2020 and the first few months of 2021 we proposed solutions to those and had them agreed. In the meantime, Andy Banks had updated the original source specification to the format used at OASIS and started making basic corrections.

Since then, Simon and I have updated the draft specification with the issue proposals to result in the latest working draft. At this stage, or with a few minor extra changes, we will have completed the technical updates needed to implement the new version of MQTT-SN.

Next Steps

There is still a lot of work before the specification is complete: the non-normative sections, introduction, conformance statements and formalization (RFC-izing) of the language for example. I estimate all this will take six months at least.

The immediate steps are:

  • Take the current working draft to the MQTT Technical Committee to hear any objections to our direction of travel. I don’t expect any, but I want to find out.
  • Assuming no objections, we can then embark on the process of completing the non-normative sections and rest of the updates.

In the meantime, Simon and I are planning implementations, Simon in Java, mine in Python. If anyone else wants to start implementing then that would be welcome. Having some implementation experiences is necessary for the specification to become an OASIS standard, as well as double checking our work.

I hope that later in the year, we will have a specification ready for release as a first Committee Specification Draft (CSD), along with a couple of implementations. Reaching a final standard should then be possible in 4Q this year or 1Q the next.

One Thing! – Security

I heard, through Simon, that CoAP has added functionality to deter security issues. I didn’t know what that was, but a quick search resulted in hits about DDoS attacks. From a quick read, I think that MQTT-SN isn’t vulnerable to the same type of attacks, but perhaps QoS -1 publishes (without a connection) could be used and should be thought about carefully before use. If anyone has any thoughts or more information on this topic, do speak up.

MQTT-SN Authentication

This discussion is part of the standardization of MQTT-SN at OASIS. Here is a summary of the current issues we are working on. The issue for authentication is 568.

In the current MQTT-SN specification (1.2), there is no authentication capability included apart from the client id. This applied to the early versions of MQTT as well, so it’s not that surprising as MQTT-SN was derived from MQTT.

Later, userid and password fields were added to the MQTT connect packet, with optional external TLS encryption applied to the TCP/IP connection. In MQTT 5.0 an enhanced authentication capability was added by means of the new AUTH packet.

It’s generally agreed that we should add authentication to MQTT-SN. There are a few considerations which are different to MQTT however. MQTT is aimed at low powered devices, but MQTT-SN is notionally aimed even lower at connectionless network communications such as UDP where packet sizes are limited and TCP capabilities such as fragmentation do not exist.

So we need to be even more aware of data overhead, bloat and efficiency than we do with MQTT. In the base version of MQTT-SN, 1.2, there is a maximum of only one variable length field in each packet. This means that there is no length field apart from that for the whole packet, unlike in MQTT where there are many variable length fields each with its own length. So MQTT-SN is more efficient in terms of the size of each packet, at the expense sometimes of multiplying the number of packets needed. The MQTT-SN will message operations are an example of that.

The MQTT-SN 1.2 connect packet is structured like this:

Length n
(byte 0)
MsgType
(1)
Flags
(2)
ProtocolId
(3)
Duration
(4,5)
ClientId
(6:n-1)
MQTT-SN 1.2 CONNECT packet

The flags byte is an identical format for all the packets that use it, and is fully used – I’ll come back to that in a moment.

To add authentication to MQTT-SN I propose that an additional AUTH flag is added to the connect packet, which tells the server that an AUTH packet is going to be sent following the CONNECT packet. The AUTH packet consists of the following fields:

Length n
(byte 0)
MsgType
(1)
Length m
(2)
Auth Method
(3:m+2)
Auth Data
(m+3:n-1)
Proposed MQTT-SN AUTH packet with two variable length fields

This is designed to allow the operation of SASL framework authentication mechanisms and follows MQTT 5.0 closely. The current list of registered mechanisms includes PLAIN, a single message including the following fields:

  • authorization identity
  • authentication identity
  • password

each separated by a NUL (U+0000) character. Using this mechanism allows the CONNECT packet to be followed by an AUTH packet which includes a userid (authentication identity) and password, and so caters for the simple MQTT like case.

More sophisticated methods can also be used – this should allow any of the registered SASL mechanisms to be used. The server can also send an AUTH packet, either before a CONNACK, or during a session for re-authentication of the client.

It would be neater from a packet structure point of view if the authentication method were a fixed rather than variable length field. However, SASL mechanism names have a maximum length of 20 characters, so including a one-byte length field is going to be more efficient just about all the time.

Another alternative could be to have a one byte code for authentication method. But someone would have to maintain the table of translations from codes to meaning, or leave it to be implementation independent, hindering application portability.

Finally, to fit in the AUTH flag to the connect packet, some space will have to be made. As the CONNECT flags are a completely different set to those used in the other packets I propose that they are separated. The CONNECT flags:

x
(7)
x
(6)
x
(5)
x
(4)
Will
(3)
CleanStart
(2)
x
(1)
Auth
(0)
Proposed CONNECT flags; x means reserved

DUP
(bit 7)
QoS
(6,5)
Retain
(4)
x
(3)
x
(2)
TopicIdType
(1,0)
Proposed flags for other packets

I have left the positions of the existing fields unchanged for continuity, but they could be changed for neatness.

So this is my current first draft proposal. I’d like to get any thoughts that any of you might have. I haven’t used the SASL mechanisms myself so hearing from anyone who has could be particularly useful.

MQTT-SN Alignment with MQTT

We are starting to work on the standardization of MQTT-SN – MQTT for Sensor Networks. The current specification for MQTT-SN is in a similar position to that for MQTT before it became a standard at OASIS; it is published by IBM, freely available with a liberal license, and has been in use for several years. It is not as widely used as MQTT was at the same point, but there are several existing users and implementations. To this end, I propose that in the process of standardization we take the approach that was adopted for MQTT 3.1.1 – minimal changes with the existing specification to allow standardization to proceed as quickly as these things ever do. In the case of MQTT 3.1.1, that was about two years.

While there is a general agreement to get things moving quickly, a concern has been raised from a couple of quarters. That is, the current MQTT-SN specification was written before MQTT 5.0 existed. One of the primary goals of MQTT-SN is to extend MQTT to non-TCP networks – to do so, it must allow the easy interoperation of the two protocols. Messages from MQTT must be able to flow to MQTT-SN and vice versa. The concern is that the current MQTT-SN specification might more closely align with MQTT 3.1.1 rather than 5.0, and we should really be aiming at 5.0 as that is likely to be more frequently used in the future.

Several features in MQTT 5.0 were influenced by MQTT-SN in fact, so the flow of concepts might be towards MQTT rather than MQTT-SN. In this article, I’ll go through the aspects of MQTT-SN and see how they match up to the two MQTT standards, 3.1.1 and 5.0.

MQTT-SN Server

MQTT-SN is both a client/server and peer to peer protocol. An MQTT-SN server can be a broker in the MQTT sense, or a gateway which does little more than act as a mediator between MQTT-SN and MQTT and the underlying transports. Here I will use the term server to refer to both brokers and gateways for MQTT-SN.

Packet Format

Every MQTT packet has a header byte followed by a variable-length remaining length field. Some of the packets have multiple variable-length fields (string or binary) as part of their construction. Although MQTT packet sizes are still kept as small as is feasible, MQTT-SN is intended to be suitable for even lower power devices and used over networks with fewer reliability characteristics than TCP. Each MQTT-SN packet, apart from one possible exception, has a single variable-length field within it, so the only one length field is needed for each packet, helping to reduce their size. As a result, for instance, the MQTT connect packet has been split into several MQTT-SN packets:

  • WILLTOPICREQ – sent by the server to request that a client sends the will topic name
  • WILLTOPIC – sent by the client to tell the server its will topic name
  • WILLMSGREQ – sent by the server to request that a client sends the will message
  • WILLMSG – sent by the client to tell the server its will message
  • WILLTOPICUPD – sent by the client to update its will topic name stored in the server
  • WILLLTOPICRESP -sent by the server to confirm the will topic name has been updated
  • WILLMSGUPD – sent by the client to update its will message stored in the server
  • WILLMSGRESP – sent by the server to confirm the will message has been updated

Connect and Disconnect

The fields in the connect packet are:

  • Will flag – request will topic and message prompting
  • CleanSession – as in MQTT 3.1.1
  • ProtocolId – corresponds to protocol name and version as in both MQTT 3.1.1 and 5.0
  • Duration – keep alive timeout as in both MQTT 3.1.1 and 5.0
  • ClientId – as in both MQTT 3.1.1 and 5.0

The clean session flag operates in a similar manner to MQTT 3.1.1 in that the cleanup operates at both the start and end of the session. In MQTT 5.0, the clean session flag becomes the clean start flag, and a separate property session expiry dictates when the session state is removed on the server. The MQTT 5.0 facilities are much more flexible and I would advocate changing MQTT-SN to match. One way to achieve this would be to add a session expiry 2 byte integer field (matching the duration field) to the CONNECT packet.

The clientid variable length field is allowed to be 0-length in both MQTT 3.1.1 and 5.0, indicating that the server should assign a clientid itself. If we do allow this behaviour in MQTT-SN I think it’s important that that clientid is returned to the client, as it is in MQTT 5.0. This could be done by including the server assigned clientid in the MQTT-SN CONNACK packet, which currently has no variable length field.

The MQTT-SN DISCONNECT packet has a duration field, which operates in a similar way to the MQTT 5.0 session expiry property. In this case MQTT-SN is already closer MQTT 5.0. MQTT-SN also allows the DISCONNECT packet to be sent by the server to the client, so that the client has more information about the reason for the disconnection. This is forced on MQTT-SN anyway, as unlike for MQTT there may be no underlying session (TCP for MQTT) to break, as in UDP for instance. Again, this is closer to MQTT 5.0 than 3.1.1. The latter does not allow the server to send a DISCONNECT packet.

In fact the MQTT-SN behaviour on disconnect is more sophisticated than MQTT 5.0 (see Sleeping Clients in the MQTT-SN specification), but that doesn’t alter the fact that it is closer to 5.0 than 3.1.1.

Will Processing

The will message processing in MQTT-SN uses 8 packets as listed above, so is equally removed from both MQTT 3.1.1 and 5.0. Section 6.3 of the MQTT-SN specification lists the combination of interactions between the clean session and will flags on the connect packet – I think these would remain intact if we changed the clean session flag to clean start to match MQTT 5.0. So I feel there is no need to change the will processing in MQTT-SN to align with MQTT 5.0 more. There may be other reasons but this isn’t one.

Topic Names

In both MQTT 5.0 and MQTT-SN topic ids can be used instead of a full topic string. However, in MQTT-SN this is almost compulsory, because the PUBLISH packet’s one variable length field is the payload. The topic data is limited to a two byte field to hold a topic id (2 byte integer) or a short topic string (2 bytes). In MQTT 5.0 the topic id is registered by including it in the publish packet along with the long topic string. In MQTT-SN this registration is delegated to a separate packet, REGISTER, which must be sent before sending a PUBLISH packet. This applies to both clients and servers.

This does lead to a problem when using the PUBLISH packet in a QoS -1 mode, which is exclusive to MQTT-SN. QoS -1 in MQTT-SN means that a client can send a message to a server outside of the familiar CONNECT/DISCONNECT session start and end range. Typically this could be used in a multi-cast environment where the client is not sure of the location of the server. There are a number of proposals to allow a variable length topic name to be included in a PUBLISH packet. At least one has already been implemented, which is to use the spare topic id type indicator to specify that the topic name is variable length field, and the topic id integer holds the length. This would mean the PUBLISH packet is the only one with two variable length fields (topic name and payload). I would advocate allowing this format for all PUBLISH QoS.

MQTT-SN also has the concept of pre-registered topic ids – there is no parallel in either version of MQTT.

I feel there is no need to change to particularly align with MQTT 5.0 – any problems with the existing MQTT-SN implementation of topic ids should be fixed for their own sake.

Subscribe/Unsubscribe

The MQTT-SN SUBACK packet includes a return code. As in MQTT 3.1.1, the UNSUBACK packet does not. It would seem sensible to add a return code to every ack. I think the UNSUBACK is the only one without. This would mirror MQTT 5.0 too.

Other MQTT-SN Features

Other capabilities of MQTT-SN are straightforward mirrors of simple MQTT function, favouring neither one nor the other, or have no analog in MQTT at all. These include:

  • PING and PINGRESP
  • Keep alive
  • Gateway advertisement and discovery
  • Forwarder encapsulation

Conclusion

It turns out there are more changes that I would like to see than I was expecting before writing this article. However, I feel they are more to do with fixing up some of the more irritating MQTT-SN aspects for themselves, rather than aligning with MQTT 5.0 per se.