More Rigorous Testing for MQTT Servers

I’ve had a sustained interest in a rigorous approach to test design ever since I studied formal methods in the 1990s. This is reflected in the domain name of this blog. 1)All the test material referred to in this post is in the Eclipse Paho project.

Testing both client libraries and servers for adherence to the MQTT specification seems a natural candidate. Any form of rigorous testing will need more investment than a few manual experiments, but the payoff is potentially large (unless you really like a boring, repetitive life). The MQTT standardization process is nearly complete. A major goal of any standardization is to encourage implementations, so there will be plenty of opportunity for a rigorous test suite to recoup the investment made in it.

If a test suite is a haphazard collection of acquired test programs, it tends to suffer from gaps in coverage and a lack of consistent approach and motivation. When the object of that test suite changes, the suite will be hard to maintain because you have to work out the intent of each of the component test programs. Even explaining the exact reasoning behind an individual test program can be hard. With the approach outlined here I hope to make the goals and attainments of the test suite much more transparent, and maintenance easier and more logical.

I start with the MQTT 3.1.1 Specification. The standardized MQTT specification is a great improvement over the previous version because it is has been argued over intensively, with the (planned for) result that many of the ambiguities have been resolved. Each identified conformance statement has been numbered and listed in Appendix B. I use that list as a starting point for measuring test coverage of the complete MQTT specification. 2)It has been pointed out to me, by Peter Niblett of IBM, that this list has been constructed by fallible human beings, and thus itself is not rigorous.  I knew that already, but you have to start from somewhere, and this is better than any other specification I have previously tested against.

An example statement from the table:

Normative Statement Number Normative Statement
[MQTT-1.5.3-1] The character data in a UTF-8 encoded string MUST be well-formed UTF-8 as defined by the Unicode specification [Unicode] and restated in RFC 3629 [RFC3629]. In particular this data MUST NOT include encodings of code points between U+D800 and U+DFFF. If a Server or Client receives a Control Packet containing ill-formed UTF-8 it MUST close the Network Connection.

What we need first is a model of the expected behaviour of an MQTT implementation, a test oracle. I had previously written an MQTT server in Python, for test purposes, so I updated it for MQTT 3.1.1. I also seeded it with log entries for conformance statements from the specification at appropriate points 3)Look for logger.info("[MQTT … statements in this example. During the execution of this server, any conformance statements encountered are logged. At the end of the test, we can measure the number of conformance statements that that test invoked in the server. We can assess a test suite both by the conformance statements invoked, and by those which are not.

Next we need a model of the test inputs, the MQTT packets that we want to send to the model MQTT server.  We wouldn’t need this model if we could exercise the whole input space without worrying, but for any non-trivial software, this space is so vast that we need to focus.  Even a computer needs to focus.  The first model I’ve created is intended to cover the basic MQTT test scenarios. It responds to QoS 2 publish flows correctly for instance. In the basic scenarios, we want to make sure that the “golden paths” are executed, and some of the error cases. Other input models will be needed for complete coverage — when a large message takes longer to send than the keep alive interval, for instance.

With the Python script suite_generate.py the input model is used to generate a sequence of inputs which are sent to the model MQTT server.  The outputs from the model MQTT server are recorded along with the inputs, to result in a self-checking test case.  The exact same sequence of inputs are not used more than once in the same test. The resulting test case can be used directly against a real MQTT server with the run_test.py. There are some ‘little’ complications — MQTT packet IDs do not have to be the same on each run, so they have to be remapped. But in general we replay the same recorded scenario against the MQTT server we want to test.

Currently, the generated test suites reach about 55% coverage of the conformance statements. I have to work on this input model, and maybe some others to increase that coverage to as near 100% as I can manage.   Another complication is that some behaviours of MQTT are optional or can vary, so we have to be able to generate test suites for each of these variations. I aim to be working on these and other items in the upcoming weeks and months before EclipseCon NA next year.

Some more information is available on the Paho wiki. I hope this post is helpful in explaining my intentions. Please do ask questions. I expect at least two other follow on posts:

  • More Rigorous Testing for MQTT Clients
  • A “model” implementation of an MQTT server

so this won’t be the last word on the subject.

References   [ + ]

1. All the test material referred to in this post is in the Eclipse Paho project.
2. It has been pointed out to me, by Peter Niblett of IBM, that this list has been constructed by fallible human beings, and thus itself is not rigorous.  I knew that already, but you have to start from somewhere, and this is better than any other specification I have previously tested against.
3. Look for logger.info("[MQTT … statements in this example.

Porting the Paho synchronous embedded C++ client

The MQTT library on mbed.org also now exists in Eclipse Paho. I intend Paho to be the master copy.

There are two APIs in this library, the largely complete synchronous API in MQTTClient.h, and the unfinished asynchronous API in MQTTAsync.h. In this post I’ll be talking about the synchronous API. I have discussed the major design points previous posts, now I’ll describe how to use the API. The documentation page on mbed can be found here.

The API implementation is intended to avoid system calls to allow it to be easily ported. Its function is limited purely to MQTT rather than networking setup for instance. This does mean that there is some work to use the API on a new OS, but we plan to collect examples in Paho for as many OSes as we can — contributions welcome!

The API is defined as a class template with these parameters:


template<class Network, class Timer, int MAX_MQTT_PACKET_SIZE = 100, int MAX_MESSAGE_HANDLERS = 5>

There are two class parameters, Network and Timer which are needed to provide platform specific implementations of sending and receiving data from the network, and timing, respectively. The third and fourth parameters are limits on the maximum MQTT packet size to be handled and the number of message handling callbacks. For each subscription, a message handling callback is needed, so you can think of this second limit as the maximum number of subscriptions you can have active at one time. Increasing these limits will increase the storage used when allocating the MQTT client object.

The file mbed.h contains an mbed specific implmentation of the Timer class. The file MQTTSocket.h contains an mbed specific network implementation for the Ethernet interface. The only requirements for the network class that you need to supply are the two methods:


int read(unsigned char* buffer, int len, int timeout);
int write(unsigned char* buffer, int len, int timeout);

Where

buffer
is the location to write data to, or read data from
len

is the number of bytes of that datatimeoutis the maximum number of milliseconds for the operation to complete.

The timer class counts down from a value to zero. I chose this way because it is always used for timeouts, so it checks to see whether the time has expired:


bool expired();                 // has the time expired?
void countdown_ms(int ms);      // start the timer for ms milliseconds
void countdown(int seconds);    // start the timer in seconds
int left_ms();                  // return the time left in milliseconds

Looking at the mbed example HelloMQTT, you see we have to connect the network interface before calling the MQTT connect:


MQTTEthernet ipstack = MQTTEthernet();
              
MQTT::Client<MQTTEthernet, Countdown> client = MQTT::Client<MQTTEthernet, Countdown>(ipstack);

int rc = ipstack.connect("m2m.eclipse.org", 1883);
if (rc != 0)
   lcd.printf("rc from TCP connect is %d\n", rc);
 
MQTTPacket_connectData data = MQTTPacket_connectData_initializer;       
data.MQTTVersion = 3;
data.clientID.cstring = "mbed-sample";
data.username.cstring = "testuser";
data.password.cstring = "testpassword";
if ( (rc = client.connect(&data)) != 0)
   lcd.printf("rc from MQTT connect is %d\n", rc);

and then we can call subscribe, publish, unsubscribe and disconnect methods on the client in the usual sort of way for MQTT APIs.

Examples of how to use this API on Linux is in this
directory on Paho, in the hello.cpp and stdoutsub.cpp programs. The API is already running on Arduino, for which we’ll add an example, and for other platforms and operating systems later.

Portable MQTT C/C++ clients for resource-limited platforms

When I started writing the Paho MQTT C client as long ago as 2008, it looked to me like the world of embedded operating systems was converging on Linux and Windows variants. This was based on my (limited) experience with embedded systems such as the Arcom Viper and various WinCE hand held devices. Consequently I wrote the Paho MQTT C client using Posix and Windows system APIs.

Now that decision doesn’t look so appropriate. There is a proliferation of cheap and small embedded platforms – some can run Linux, some can’t. In my last two posts I starting describing a new, more portable MQTT C client at its simplest. That first layer simply takes C data and serializes it into a buffer ready for sending across the network using any network APIs you choose:


len = MQTTSerialize_connect(buf, buflen, &data); /* 1 */
len += MQTTSerialize_publish(buf + len, buflen - len, 0, 0, 0, 0, topicString, payload, payloadlen); /* 2 */
rc = write(mysock, buf, len);

- there are corresponding API calls for deserializing buffers received from the network into C data structures.

API for mbed

This works fine, but I was also asked to write an API that would work well in the mbed world (The mbed development platform is the fastest way to create products based on ARM microcontrollers). C++ is the standard language here, but non-Posix networking APIs. What I’ve come up with so far (after some discussion) is an API that has an outline like this:


template<class Network, class Timer, class Thread, class Mutex>
class Client
{
public:    
    /* connectionLost and default messageArrived callbacks to be added */
    struct Result
    {
        /* success or failure result data */
        Client<Network, Timer, Thread, Mutex>* client;
        int connack_rc;
    };
    typedef void (*resultHandler)(Result*); 

    Client(Network* network, const Limits limits = Limits()); 

    int connect(MQTTPacket_connectData* options = 0, resultHandler fn = 0);     
    int publish(const char* topic, Message* message, resultHandler rh = 0);
    int subscribe(const char* topicFilter, enum QoS qos, messageHandler mh, resultHandler rh = 0);
    int unsubscribe(const char* topicFilter, resultHandler rh = 0);
    int disconnect(int timeout, resultHandler rh = 0);

    void yield(int timeout);
}

This API will look familiar to anyone who is acquainted with the Paho MQTT APIs for C and Java. The resultHandler parameters are optional callback functions which give the outcome of the standard calls. If these are not set, then the MQTT operation calls will block until complete, using no background thread. If resultHandler callback functions are set, then a background thread will be started and will handle MQTT networking while the application can continue with other business.

I am considering splitting the API into two completely separate units, one blocking and single-threaded, the second non-blocking and multi-threaded. This would make the intended use clearer, and reduce the size of both APIs (resultHandlers would be unnecessary in the single-threaded API for instance). Other points to note:

  • C++ Standard Template Library (STL) not used – too heavyweight
  • messageHandler callback per subscription (see discussion) – there will also be a default messageHandler
  • system APIs for networking, timing and threading passed in as template parameters (for the synchronous API threading and mutex classes are not needed)
  • heap memory allocation minimized (I hope to avoid it altogether)
  • memory use limited at object construction time by the limits parameter (see below)
  • network connection handled outside of this API
  • persistence, if/when added, as another template parameter

The limits parameter looks like this:

typedef struct limits
{
    int MAX_MQTT_PACKET_SIZE;       // two buffers, for sending and receiving, will be allocated of this size
    int MAX_MESSAGE_HANDLERS;       // each subscription requires a message handler
    int MAX_CONCURRENT_OPERATIONS;  // each command which runs concurrently can have a result handler, when we are in multi-threaded mode
    int command_timeout;            // how long to wait for any command to complete, in seconds
        
    limits()
    {
        MAX_MQTT_PACKET_SIZE = 100;
        MAX_MESSAGE_HANDLERS = 5;
        MAX_CONCURRENT_OPERATIONS = 1; // 1 indicates single-threaded mode - set to >1 for multithreaded mode
        command_timeout = 30;
    }
} Limits;

Here are example programs to use this API on mbed and Linux. These demonstrate the portability of the API – it’s not just for mbed, or for one networking API.

I’ve not finished this API yet, and it will be pulled back into Paho.

Three Layers

I’ve been thinking that a third layer supplying offline message buffering would complete the picture. (I’m hoping that I will never have to write another MQTT C client API after this effort :-) ) This simply means the ability to send and receive messages from the API when there is no connection to the MQTT server. This feature is frequently asked for, but does increase the internal complexity of an API and thus its memory requirements. Some of the questions that arise when considering a buffered client API are:

  • what if messages are “sent” by the application, but the client library never connects successfully?
  • how many messages should we buffer? what size?
  • should the messages be sent/received when the application is not running?
  • when and how often should the API try to (re-)connect to the server?
  • how should buffered messages be stored? for how long?

There are reasonable answers to these questions which will make such an API suitable for embedded platforms, and certainly for a portable C/C++ MQTT API. The family of MQTT C/C++ APIs for portability/embedded systems would then look like this:

  1. serialization/deserialization API – currently C only
  2. blocking and non-blocking unbuffered MQTT client in a style similar to current C and Java APIs, but portable across system libraries – currently C++ only (on top of library 1)
  3. buffered MQTT API, portable across system libraries (on top of library 2).

Whether each of these layers should be in both C and C++, I’m not sure. I think it depends on the demand. I am relishing using C++ again (without STL) and am reluctant to go back to C for layers 2 and 3 unless there’s a real need.

Comments or suggestions on any of this are welcome.

Receiving messages with the Paho embedded C client

In my previous post I described the motivation behind writing a new MQTT C client, and the first sample – how to publish a message with a small application program.

Now I want to go onto the second sample pub0sub1.c. This is meant to mimic the sort of program that subscribes to a command topic, and publishes a value from sensor at a regular interval – in this case 1 second.

The first part of the program is similar to the first, sending a connect packet. But now because we want to subscription, we should wait for the connack packet to be sent back from the server. First we set the timeout on the socket, so when we read from it, if no data arrives, it will return rather than block. Because we want to send a sensor reading every 1 second, we set the timeout to 1 second.

struct timeval tv;
tv.tv_sec = 1;  
tv.tv_usec = 0;  
setsockopt(mysock, SOL_SOCKET, SO_RCVTIMEO, (char *)&tv, sizeof(struct timeval));

A helper function, MQTTPacket_read is used to get the data for one packet into a supplied buffer buf. The buffer length is specified in buflen – to avoid overruns. MQTT packet lengths are defined in a specific variable-length field, which this function reads, and then attempts to read the rest of that packet.

MQTTPacket_read(buf, buflen, getdata);

The third parameter is a pointer to helper function which is called to read the data from whatever source is required. In this case, a socket:

int mysock = 0;

int getdata(char* buf, size_t count)
{
	return recv(mysock, buf, (size_t)count, 0);
}

The received data, of length count, is placed into the buffer, and the length of received data returned. MQTTPacket_read will return the code of the packet type received, or -1 if there was no data. So we check that the packet was a CONNACK, and then parse it with a call to MQTTDeserialize_connack. This allows us to check the connack return code.

if (MQTTPacket_read(buf, buflen, getdata) == CONNACK)
{
	int connack_rc;

	if (MQTTDeserialize_connack(&connack_rc, buf, buflen) != 1 || connack_rc != 0)
	{
		printf("Unable to connect, return code %d\n", connack_rc);
		goto exit;
	}
}
else
	goto exit;

Next we follow a similar process with sending a subscribe packet and waiting for the suback.

topicString.cstring = "substopic";
len = MQTTSerialize_subscribe(buf, buflen, 0, msgid, 1, &topicString, &req_qos);

rc = write(mysock, buf, len);
if (MQTTPacket_read(buf, buflen, getdata) == SUBACK) 	/* wait for suback */
{
	int submsgid;
	int subcount;
	int granted_qos;

	rc = MQTTDeserialize_suback(&submsgid, 1, &subcount, &granted_qos, buf, buflen);
	if (granted_qos != 0)
	{
		printf("granted qos != 0, %d\n", granted_qos);
		goto exit;
	}
}
else
	goto exit;

Finally we create the main loop. It calls MQTTPacket_read to wait for incoming publish packets. The read times out after 1 second if no packet arrives. If a publish does arrive, we parse it into variables we can use, like payload_in.

while (!toStop)
{
	if (MQTTPacket_read(buf, buflen, getdata) == PUBLISH)
	{
		int dup, qos, retained, msgid, payloadlen_in, rc;
		char* payload_in;
		MQTTString receivedTopic;

		rc = MQTTDeserialize_publish(&dup, &qos, &retained, &msgid, &receivedTopic,
					&payload_in, &payloadlen_in, buf, buflen);
		printf("message arrived %.*s\n", payloadlen_in, payload_in);
	}

	printf("publishing reading\n");
	len = MQTTSerialize_publish(buf, buflen, 0, 0, 0, 0, topicString, payload, payloadlen);
	rc = write(mysock, buf, len);
}

Whether we receive an incoming message or not, we do send a reading from the sensor every time.

The structure MQTTString has two parts – a length delimited string and a C style null-delimited string.

typedef struct
{
	int len;
	char* data;
} MQTTLenString;

typedef struct
{
	char* cstring;
	MQTTLenString lenstring;
} MQTTString;

#define MQTTString_initializer {NULL, {0, NULL}}

MQTT strings are length delimited. To avoid unnecessary copying of received data, by default when a packet is parsed, or deserialized, the string data is not copied into an MQTTString. Instead, the length delimited type has its length set, along with a pointer to the appropriate place in the buffer into which MQTTPacket_read placed the data read from the socket.

Later, if we want to, we can convert the length delimited string into a null-delimited C string (and I will add a helper function for this). This will probably involve allocating some memory, and copying the data so we can add the null character. For efficiency, we were trying to avoid both actions, which is why C-style strings are not created by default.

All the possible error conditions are by no means catered for in this program. Adding that error handling would improve reliability at the cost of size. As it is, whereas the QoS 0 publisher came in at under 5k compiled (64-bit Linux), this program is less than 10k (more than a completely empty program). I’m reasonably happy with that.

These examples are using the standard socket calls. Next step is to try real embedded environments to see how well this MQTT client fits in.

New “Embedded” Paho MQTT C Client

Yesterday I added the first phase of a new C client for “embedded” platforms to the Paho project in a new repository. It might seem odd to start another C client library, but I think the logic is sound.

When I started writing the first version of the Paho C client it was to work with IBM MQTT servers like Microbroker, Message Broker (now IBM Integration Bus Advanced) and my own RSMB. In spite of trying to keep it as lightweight as I could, I had to respond to many requirements without which the library would not be ‘complete’. So it grew. The main priority was to make MQTT programming simple – size was not that important.

The end result is a library which builds to about 80k (if you strip the debugging symbols), and is not easily stripped down if you want something smaller. A later addition was the asynchronous client, in the same repository, which changed the programming model to be entirely non-blocking, but didn’t make the library any smaller.

Even though most devices are getting ever more capable, there are still the smallest, most frugal motes that have limited capabilities. MQTT was intended to work on these devices, which is why the protocol is public – you can write a bespoke client library if you want. The existing client libraries are too big and inflexible, so this is the first stage in attempting to fix that.

I didn’t want to destabilize the existing libraries, so decided to start afresh. These small devices might not use the usual libraries for networking, threads, timers or even memory allocation. So I thought I would start from the bottom up, with a basic lower layer that would omit all network or memory allocation calls. This lower layer merely serializes and deserializes MQTT packet data so that it can be written to or read from the data destination or source. The end goal is a totally asynchronous API – perhaps with an external definition akin to the existing asynchronous API.

The best way to see it in action is to look at some samples. The simplest sample qos0pub.c is to send a QoS 0 MQTT message. It connects, publishes and disconnects all in one go, making it one of the shortest MQTT applications you can write. The core of the program looks like:

MQTTPacket_connectData data = MQTTPacket_connectData_initializer;
int rc = 0;
char buf[200];
MQTTString topicString = MQTTString_initializer;
char* payload = "mypayload";
int payloadlen = strlen(payload);int buflen = sizeof(buf);

data.clientID.cstring = "me";
data.keepAliveInterval = 20;
data.cleansession = 1;
len = MQTTSerialize_connect(buf, buflen, &data); /* 1 */

topicString.cstring = "mytopic";
len += MQTTSerialize_publish(buf + len, buflen - len, 0, 0, 0, 0, topicString, payload, payloadlen); /* 2 */

len += MQTTSerialize_disconnect(buf + len, buflen - len); /* 3 */

rc = Socket_new("127.0.0.1", 1883, &mysock);
rc = write(mysock, buf, len);
rc = close(mysock);

We set up the data we want to connect with, and add that to a buffer (1). Then we add the MQTT publish command to the end of the same buffer (2). Lastly we add the disconnect command to the same buffer (3). The disconnect is not strictly necessary – it’s an act of kindness to the server, for the cost of a few bytes.

We can write this buffer directly to a socket which is connected to an MQTT server, and then close the socket immediately, and we’ve published a message. The entire program comes in at about 5k over and above a completely empty one using RedHat and gcc – which is more like it, although I’d prefer smaller.

I’ll talk about receiving messages in a second post.

Use of Gotos in C code (in RSMB)

I saw that John Donovan criticised my use of gotos in the Really Small Message Broker (RSMB) C code a little while back in his article MQTT and the language of the Internet of Things.

To quote:

And, despite some missing features, a crash bug (for which I have posted a fix), and a scary number of goto statements (i.e. more than 0), it does the job admirably.

The crash bug was due to an unimplemented feature of MQTT-SN, but let’s get onto the use of gotos.  The RSMB code is written in C, not C++.  I made that decision in 2008 when I started to implement RSMB in C++.  I thought I would use all the features of C++ to build a nicely object oriented architecture, while making the code as efficient and small as it could be.  That approach lasted all of a few hours, as I  discovered that template support in the version of gcc I had to hand was decidedly dodgy.

Nothing for it, I had to resort to standard, portable C.  On the bright side, that meant I didn’t have to contend with huge standard libraries being included in the executable.  On the down side (and this is a big downside) I had to obtain or write my own collection libraries.

As RSMB became part of WebSphere MQ for a while, under the name of Telemetry Daemon for Devices, I was required to add some features to aid bug fixing.  One of these is a stack trace (the pros and cons of that belong in a different post).  What that meant is that each function had to end with a macro:


  FUNC_EXIT;
}

In those functions where some code inside a nested loop or two found an error and needed to leave the function, I had two potential approaches (apart from restructuring the loops):

  1. have multiple returns from the function, with multiple FUNC_EXIT macros, or
  2. use a goto to the single exit point of the function.

I chose the latter, as it meant only one FUNC_EXIT call was needed, meaning less scope for forgetting to include it before any return statement. This means that each function only has one exit point, which some people prefer in any case.

The use of goto is therefore constrained to the following pattern:


int function(int parm)
{
...
   goto exit;  /* an error condition */
...
   goto exit;  /* another error condition */
...

exit:
  FUNC_EXIT;
}

It struck me recently, rather belatedly, that this is what exceptions are. So if you dislike my use of gotos, please consider them to be exceptions instead.

With the right sort of macros, maybe I could even make them look like exceptions



#define raise(label) goto label;

#define handle(label) label:

but I think that would be wilfully obscure :-)

Getting started with MQTT-SN in RSMB

Now that the source for RSMB with MQTT-SN is available, here is the quickest of quick starts to get going with MQTT-SN. More to come in due course.

The source is at http://git.eclipse.org/c/mosquitto/org.eclipse.mosquitto.rsmb.git/
in the rsmb directory. To build, go into rsmb/src and call “make”. That works well for Linux. You need a make from cygwin or similar to build for Windows.

This is an example configuration file for RSMB with MQTT-SN:


# will show you packets being sent and received
trace_output protocol						

# normal MQTT listener
listener 1883 INADDR_ANY	
		
# MQTT-S listener
listener 1884 INADDR_ANY mqtts					
	# optional multicast groups to listen on
	multicast_groups 224.0.18.83	
	# optional advertise packets parameters: address, interval, gateway_id			
	advertise 225.0.18.83:1883 30 33			

# MQTT-S outgoing multicast bridge (QoS -1)
connection mqtts_multicast						
	protocol mqtts_multicast  
	# only one address is used at the moment, plan is that a list can be used
	address 225.0.18.83:1883					
	topic a out
	topic abcdef out

# QoS 2 MQTT-S bridge
connection mqtts							
	protocol mqtts
	address 127.0.0.1:1885
	topic a out

At the moment predefined topics and sleeping clients aren’t implemented.

There is a Python MQTT-SN client in rsmb/src/MQTTSClient/Python – see mqtts.py, in the main section, for an example of using it.

MQTT, QoS and Persistence

Over the years, every now and again I get asked “what is the point of supporting QoS 1 or 2 in MQTT without full persistence?” I drew up this table a while back to help answer that question.

First a reminder of MQTT definitions of reliability of QoS:

  • 0 – at most once (fire and forget)
  • 1 – at least once
  • 2 – once only
Error Situation No Persistence Persistence
Communications errors: broken connection from whatever cause (intermittent signals e.g. over wireless, hardware problem, etc) All QoSs apply with any persistence (will hold to their reliability criteria). All QoSs apply with any persistence (will hold to their reliability criteria).
Power failure to remote host machine (bridge connection) All QoSs apply with any persistence (will hold to their reliability criteria). All QoSs apply with any persistence (will hold to their reliability criteria).
Power failure to local host machine All in-progress messages lost. Retained messages and subscriptions lost Shutdown persistence: all state since last restart lost.
Full persistence: full protection for all QoSs, as long as all persistence store write caching is turned off, both in the operating system and in the hardware. Otherwise, data can still be lost
Filesystem corruption All state lost All state lost.
Normal persistence will not help. Raid or other disk mirroring needed
Hard drive failure All state lost All state lost.
Normal persistence will not help. Raid or other disk mirroring needed
Complete system destruction All state lost All state lost.
Normal persistence will not help. Mirroring to another system needed
Destruction of location All state lost All state lost.
Normal persistence will not help. Mirroring to system in remote location needed
End of the world or universe All state lost All state lost.
Normal persistence will not help. Remote location needs to be on another planet or in another universe

Which Paho MQTT C API to use? (And some history)

It may not be obvious if you take a look at the Paho MQTT (http://mqtt.org) C client libraries (http://www.eclipse.org/paho/) , but my intention all along was to avoid proliferation of C APIs.

When I first started writing a C client for MQTT, around 2008, there was one other C MQTT API that I knew about – the now withdrawn IA93 (http://www-01.ibm.com/support/docview.wss?uid=swg24006525).  With all respect to the author, this had an interface which I did not want to copy.  Firstly, there were the variable length option structures, which meant that you had to copy topic strings, for instance, into their correct location in the structure before making the API call.  Secondly, there was the overly complicated method of setting up the API to operate with background threads.

IA93 did have one feature that I felt it was desirable to copy: the ability to work using no background threads.  As MQTT is intended for embedded devices, we would like the library to work in very simple environments.

My other starting point was the MQTT Java API at the time, I saw no reason why the C API should be much more difficult to use.  So I set out to create a C API that was close to the Java API, with the addition that it could be optionally run with no background threads.  This became the MQTTClient API (http://www.eclipse.org/paho/files/mqttdoc/Cclient/index.html).  The parameters are similar to the Java API – no unnecessary variable length structures.  The method I chose to indicate whether you want background threads or not is to simply make a call to MQTTClient_setCallbacks().  Asking for callbacks means that you have to have a background thread, right?

I was developing Really Small Message Broker (RSMB) around the same time, which is single threaded, using multiplexed I/O on the TCP port.  At the time, every MQTT server I knew of was multi-threaded – at least one thread for every connecting MQTT client.  I thought it would be interesting to see how far you could get with a single thread: reminding me of the techniques we had to use in the 1980s to write games on ZX81s and the like.  So, I thought reusing the same approach would be good for the C client – hence only one background thread no matter how many client objects you create (limited to whatever the ‘select’ system call will handle).

Another conscious decision was to use C rather than C++.  I started RSMB in C++, but quickly found out that compiler template support was pretty buggy at the time, so dropped back to ANSI standard C.  That was somewhat disappointing, but several years later does have the advantage of not being incompatible with Objective-C.

The final part of the story so far is the widespread use of graphical environments, or more accurately, environments where the application itself is a series of callbacks.   The ubiquitous example is the JavaScript program running in the web browser.  When the Paho MQTT JavaScript API was devised, it needed to be completely asynchronous.  The previous C API, in the mould of the Java API, had some calls which would block, to make programming easier.  We needed a totally asynchronous C API too, so I followed the model of the JavaScript API – which unfortunately meant a new interface, MQTTAsync (http://www.eclipse.org/paho/files/mqttdoc/Casync/index.html).

So, which to use when?

  1. MQTTClient, single-threaded: when you want to use only one thread for your entire application.
  2. MQTTClient, multi-threaded: when you want ease of use, aren’t bothered about some calls blocking, but still want messages to be sent and received in the background.
  3. MQTTAsync: when you want no calls to block at all.  (Messages will be sent and received on two background threads).

The MQTTAsync API happens to perform better than the MQTTClient API, but that wasn’t the rationale behind it.

Thanks to a Paho contributor, Frank Pagliughi, there is also now a C++ layer over the asynchronous C API.  His intent is to have an API which mirrors the corresponding Java one as much as possible.

Hmm, I think we’ve been here before!