What is MQTT? The leading messaging protocol in IoT

• What is MQTT protocol?
• Who created the MQTT?
• Who is the “owner” of MQTT?
• What is the latest version of MQTT?
• What are the advantages of MQTT?
• Where is MQTT used?
• How does MQTT work?
• What are the differences between MQTT and MQTT-SN?
• How does MQTT guarantee quality of service (QoS)?
• How does MQTT implement security?
• Conclusion

The explosive growth of the Internet of Things (IoT) in recent years has led to a unique situation in our history. A growing number of connected devices surround us, many invisible, that are constantly working to make our environment safer, more comfortable and our routine more efficient.

But on their own, they can only do so much. Their potential is only realised when they can communicate, creating a demand for a lightweight, scalable communication protocol that could be implemented in low-cost devices operating over connections that are unreliable.

This protocol already existed and was waiting for its chance to shine. Created in the mid-1990s to meet the demands of the oil industry, MQTT is fast becoming a “de facto” standard for messaging in the internet of things, used in applications as diverse as controlling a fleet of vehicles or chatting with your friends.

This article will explain MQTT, how it works, where it is used and why it is so attractive for the IoT segment.

What is the MQTT protocol?

MQTT is a protocol for machine-to-machine (M2M, Machine-to-Machine) messaging, whose main features are lightness and simplicity. It was designed for use in devices with limited resources and bandwidth, communicating over unreliable connections or with high latency.

These characteristics make it ideal for Internet of Things (IoT) applications, such as remote monitoring of devices, or message exchange between mobile devices, where bandwidth or power supply is limited, or costly.

Who created MQTT?

MQTT was developed in 1999 by Andy Stanford-Clark (of IBM) and Arlen Nipper (of Eurotech, Inc.). Nipper recalled some circumstances that led to the pattern’s emergence during an appearance on an IBM podcast in 2011.

“In the early, or mid, the 1990s, as you can imagine, we didn’t have the network connectivity that we have today. One of the projects I was working on was with Philips Conoco on a SCADA (supervisory control and data acquisition) system for a new pipeline, where we needed real-time data delivery. At the time, what we were doing in the communications industry was talking over 1,200 Baud or 300 Baud dial-up lines and satellite terminals (VSAT) with minimal bandwidth.” 

“Our five goals in implementing MQTT were: it needs to be simple to implement. It needs to deliver data with Quality of Service (QoS). It needs to be lightweight and bandwidth efficient because at the time bandwidth was expensive. It needs to be data agnostic, and it needs to have session persistence,” he said. 

Later, the emergence of the Internet of Things (IOT) created the need for a protocol with exactly the characteristics of MQTT. According to Angel Diaz, Vice President of Software Standards at IBM: 

“As a planet, we have become more connected. The Internet of things has started to emerge. And the network of devices that are able to communicate with each other and work with each other has really exploded in size.”

“So what happened […] is that a real need emerged for a kind of protocol with a Publisher-Assigner architecture to have bidirectional delivery of messages in a more predictable way.”

“With that, we could start to move from simple point-to-point interaction to more sophisticated interaction over the network. So MQTT is a protocol that allows exactly that: publishing and signing messages and a predictable, two-way delivery of these messages.” 

Originally MQTT was an acronym for “MQ Telemetry Transport”, “MQ” being a reference to an IBM middleware product line called MQSeries, where it stood for “Message Queue”. However, since 2013 the term has become the proper name of the protocol and is no longer an acronym.

Who owns MQTT?

Although it was initially developed by IBM, since 2013 MQTT has been an open protocol maintained by the OASIS (Organization for the Advancement of Structured Information Standards) consortium, based in Massachusetts, USA.

The first version of the protocol released under OASIS responsibility was 3.1.1, in October 2014. This version is also a recommendation of the ISO (International Standards Organization) and IEC (International Electrotechnical Commission), called ISO/IEC 20922:2016.

What is the latest version of MQTT?

The latest version of MQTT is version 5, published by OASIS in March 2019. In defining the objectives of the new version, the committee responsible at OASIS (OASIS Message Queuing Telemetry Transport Technical Committee) had the challenge of adding features long desired by users without increasing load or decreasing ease of use and improving performance and scalability without adding unnecessary complexity.

To meet these demands, the following objectives have been defined for version 5:

• Improvements in scalability and large systems.
• Better error reporting.
• Formalisation of common standards, including capability discovery and response to requests.
• Extensibility mechanisms including user properties.
• Performance improvements and support for small customers.

What are the advantages of MQTT?

In addition to lightness and simplicity, one of the main advantages of MQTT is its flexibility: it focuses on transporting messages between two points, without worrying about who the data in these messages were produced by or how it will be consumed.

Also according to Niper, “When we invent an embedded device, the idea is, well, if I invented a device, then I have to invent a protocol. And then I have to run to the IT side and invent an application that talks to my device. The result is a solution […] that becomes frozen in time. 

“What I learned from working with IBM on this project is that by adopting a Publisher-Assigner architecture […] we can separate the implementation from the data producers, who may be on the business or device side, and the data consumers.”

“And now consumers can write exciting applications on an end-to-end device, publish the data and be completely oblivious to who will consume the data. And on the other side, I can have people consuming them.” 

In addition, the Publisher-Subscriber model also facilitates scalability. No matter if there are 10, 100 or 1,000 subscribers on a given topic, the effort a publisher needs to make to send a message to everyone is the same: a single connection to the intermediary, and a single message. 

Since MQTT is an open standard, many implementations are available, either commercial or open source. This means that no matter what hardware platform, operating system or programming language you use to develop your application, there is sure to be one or more implementations that meet your needs.

Where is MQTT used?

The MQTT protocol is used in countless situations, from industrial environments to control devices in a smart home. Wherever you need to exchange messages between devices, you will likely find MQTT.

German company CASO Design produces smart devices for the kitchen, such as climate-controlled wine cellars and Sous-Vide circulators. When developing its products, the company had among its goals to allow users to be controlled by users from virtually anywhere, for multiple users to command a single device, and for data to be used by multiple applications.

Solutions such as a Bluetooth connection over the UDP protocol were discarded as they require physical proximity between the user and the device. Finally, the company opted for MQTT: using Wi-Fi connections or mobile phone networks a device can “talk” to a user anywhere in the world.

Multiple devices can be controlled by a single app, which sends commands and notifies users when there are changes in temperature or the end of cooking. 

BMW, meanwhile, uses MQTT in BMW Mobility Services, a vehicle-sharing system for fleet operators. A traditional way to remotely control a vehicle would be to send an SMS message to “wake up” an onboard computer, which initiates an HTTP session with a back-end service running in the cloud.

However, as anyone who has ever suffered receiving an authentication code should know, SMS is not always predictable. A message can arrive instantly, take minutes to be delivered, or disappear on the way. In addition, HTTP is slow, and the size of messages and the amount of them needed to establish a connection increase network operating costs.

With its lightness, ease of implementation and guaranteed quality of service, MQTT was the perfect solution. On a typical day, the BMW service processes 90,000 transactions per minute, with more than 80,000 clients connected. And this operates in an environment where network reliability, to the application’s very mobile and dispersed nature, is low.

But according to Stanford-Clark, co-creator of MQTT, the protocol is used in much more than automation systems. In 2011, Facebook adopted it as the transport layer for its popular instant messaging app, Messenger. “Overnight, 800 million people were using MQTT,” he said.

Each conversation generates a thread in Messenger, and chat members subscribe and post to this thread. There are multiple intermediaries instead of one, and a “Topic Director” directs data packets to the intermediary responsible for a specific chat.

In addition, MQTT is used in cloud platforms such as Amazon’s AWS, Microsoft’s Azure, IBM’s Watson and Google’s IoT platforms. With this, MQTT became 2018 the preferred protocol for message transport in the internet of things. 

How does MQTT work?

MQTT works with a Publisher-Subscriber architecture. Clients may send messages (acting as publishers) or receive them (acting as subscribers). There is no direct connection between them: all communication is controlled by an intermediary (broker) that organises the published messages into a hierarchy of topics and distributes them to interested parties.

An analogy is a traditional newspaper newsroom. The publishers would be the reporters, who collect and send the news to the newsroom. This, in turn, acts as an intermediary, organising the information into sections or topics. On the other hand, subscribers consult the newspaper to consume information on the topics that interest them.

In a more practical example, imagine that a client is a sensor in charge of monitoring the temperature of an engine. When the sensor has new information (such as an increase in temperature), it sends a message to the intermediary specifying the topic (e.g. “Temperature”) and the information (e.g. “+10 ºC”). The middleman, in turn, transmits this information to all subscribers of the “Temperature” topic.

It is worth remembering that the roles of publisher and subscriber are not exclusive. The same customer who sends information about engine temperature can subscribe to a thread about the level of coolant used to keep it under control and use this information to act as required.

The intermediary, in turn, can be a dedicated appliance, a PC running Open Source software on the local network, a server in a corporate datacenter, or even a virtualized instance in the cloud. Its location and nature do not matter, as long as it can “talk” to publishers and subscribers via TCP/IP.

Just as a neighbourhood newspaper does not publish information about a discovery in quantum physics, an intermediary discards a message about topics for which there are no subscribers. But optionally, the publisher can specify that this message should be withheld until some subscriber expresses interest in the topic.

In this case, the subscriber will receive the message as soon as he subscribes to the corresponding topic. This allows him to immediately receive the latest information, without waiting for a publisher to send an update. 

Publishers can also define a message known as “Last Will”, which will be sent by the intermediary to all subscribers of a topic if the connection between the intermediary and the publisher is suddenly broken. 

Speaking of connections, MQTT has a feature called “persistent sessions”, which reduces the overhead in reconnecting a client to an intermediary, especially on unstable networks where this can occur frequently.

Usually, when reconnecting with an intermediary, a customer needs to re-sign all topics of interest to him. But when requesting a persistent session, the intermediary store’s information about the customer will be immediately available at the reconnection time.

So the client can resume communication “where it left off”, without wasting time (and resources) re-signing threads, redefining QoS levels, etc. 

MQTT data transmission is usually over TCP/IP protocol (using standard port 1883), but any protocol that offers orderly, bidirectional and lossless packet connections can be used instead. The size of messages varies, and they can be as little as 2 bytes or as much as 256 Megabytes of data, as required.

What are the differences between MQTT and MQTT-SN?

MQTT for Sensor Networks (MQTT-SN) is an extension of the MQTT protocol for use in Wireless Sensor Networks (WSN).

Such networks usually consist of many sensors and actuators (SAs) with limited processing and storage capacity, communicating over wireless connections with reduced bandwidth, short range and low reliability as they are susceptible to interruptions by obstacles or interference.

The MQTT protocol seems ideal for this situation, but it requires a TCP/IP implementation, which may be too costly in terms of processing power for devices and connections as simple as those used in SAS.

MQTT-SN eliminates this need and can operate over any type of network capable of performing bi-directional data transfer between a device and a gateway.

Created in 2007, also by IBM, it was initially designed to run on networks based on the Zigbee protocol, based on the IEEE standard 802.15.4 for Wireless Personal Area Networks (WPANs).

Like MQTT, MQTT-SN is now a standard maintained by OASIS. The most recent version of the specification is 1.2. 

How does MQTT guarantee quality of service in message delivery (QoS)?

When establishing a connection with an intermediary, a customer (either publisher or subscriber) can define three levels of quality of service, according to the needs of the application.

The lowest level, or 0 (zero), is also known as “at most once”. A message is sent only once, and both the customer and the intermediary take no action to ensure that it has been delivered or received. 

An analogy for this level would be a supermarket that hires a company to distribute flyers with offers for its customers to homes in a city. He does not ask for confirmation that each leaflet has been delivered, and the customers do not bother to call the supermarket to say they have received a copy.

Level 1 is known as “at least once”. At this level, if there is no acknowledgement of receipt of the message after a specific time interval, it will be resent until that time elapses. A message can be received or delivered more than once: for example if the acknowledgement is lost en route due to a connection failure. 

In this case, we could replace the leaflets with a collection letter by reusing the previous analogy. If there is no confirmation of receipt (with payment of the bill), it will be resent until this happens.

Finally, level 2 is known as “exactly once”. The client and the intermediary perform a two-level handshake to ensure that only one copy of the message is delivered.

Keeping our postal analogy, it would be as if a letter had been sent with an acknowledgement of receipt (AR), which returns to the sender guaranteeing that it has been delivered.

It is important to remember that in MQTT there is always an intermediary between the publisher and the subscriber, which means that there are two connections in every “conversation”: one between the publisher and the intermediary, and another between the intermediary and the subscriber.

These connections may have different QoS levels. In this case, an intermediary will consistently deliver messages to the subscriber using the QoS level defined by the subscriber, regardless of the one specified by the publisher.

Imagine that a publisher sends messages on a topic like “Fuel Level” with QoS 1. They will be delivered to the intermediary respecting this level, with repeated sending until there is delivery confirmation. But if the subscriber of this topic has specified QoS 0, he will receive the message only once, and the intermediary will have no confirmation. 

How does MQTT implement security?

In version 3.1 of the protocol, it is possible to enter a username and password to control a connection, but this data is transmitted “in the open” (plain text), and there is no encryption. 

It can be implemented using SSL, independent of the MQTT protocol itself, but there is an inherent penalty in network and processing overhead since SSL is not a lightweight protocol. Encrypted connections use port 8883 by default.

Additionally, applications can encrypt the information they send and receive, but this is not something integrated into the protocol to keep it simple and lightweight.

Conclusion

Although it was not created for the Internet of Things, the success of MQTT is directly linked to its expansion, something that is not likely to slow down any time soon. Lightweight, flexible and with numerous implementations available, the protocol is worth considering the next time you develop an IoT device or application.