← All 50 books
MQTT: Messaging for IoT cover

MQTT: Messaging for IoT

Book 31 of 50 — AstolixGen Learning Series For researcher and publication students

Book cover

About This Book

MQTT (Message Queuing Telemetry Transport) is the dominant messaging protocol for the Internet of Things. This book takes you from first principles to research-grade understanding: how MQTT works, how to design topic structures, how quality of service affects reliability and energy, how to secure deployments, and how to use MQTT as the backbone of an IoT research study you can publish.

Learning objectives: - Explain the publish/subscribe model and why it fits IoT - Design topic hierarchies and use wildcards correctly - Select the right QoS level for reliability vs. cost trade-offs - Configure retained messages, last will, and session persistence - Secure MQTT with authentication, authorization, and TLS - Compare MQTT with CoAP, AMQP, and HTTP for research justification - Design and evaluate an MQTT-based experiment for publication

Chapter 1: What Is MQTT and Why IoT Needs It

MQTT is a lightweight messaging protocol designed for constrained devices and unreliable networks. It was created in 1999 by Andy Stanford-Clark (IBM) and Arlen Nipper (Arcom) for monitoring oil pipelines over satellite links — a setting with tiny bandwidth, high latency, and frequent disconnections. Those same constraints describe most IoT deployments today: battery-powered sensors, cellular or LoRa links, and devices that sleep most of the time.

The core problem MQTT solves is this: how do thousands of small devices exchange small messages reliably without each device maintaining expensive direct connections to every other device? The answer is a brokered publish/subscribe architecture with a minimal wire protocol. An MQTT packet header can be as small as 2 bytes. Compare that with HTTP, where headers alone often exceed a kilobyte. On a battery-powered sensor sending a temperature reading every minute, that difference decides whether the battery lasts months or weeks.

MQTT became an OASIS standard (v3.1.1) in 2014 and an ISO standard (ISO/IEC 20922:2016) in 2016. Version 5.0, published by OASIS in 2019, added richer features while keeping backward-compatible design principles. For researchers, this standardization matters: you can cite the standard itself, compare implementations against a fixed specification, and be confident the protocol will still be relevant when your paper is read years later.

Example: A smart agriculture deployment has 200 soil-moisture sensors. Each sensor publishes a 20-byte reading every 10 minutes to a central broker. With MQTT, each transmission costs roughly 25–30 bytes on the wire. With HTTP, the same reading would cost 500+ bytes in headers. Over a year, the MQTT deployment transmits roughly 15 MB total; the HTTP equivalent would exceed 250 MB — a decisive difference on metered cellular links.

For your research: When you justify MQTT in a paper, cite the bandwidth and energy argument quantitatively. Measure or estimate bytes-per-message for your setup and compare against HTTP. Reviewers in IoT venues expect this justification.

Key takeaway: MQTT is a minimal, brokered, publish/subscribe protocol standardized by OASIS and ISO — the default choice for constrained IoT messaging.

Chapter 2: Publish/Subscribe vs Request/Response

In the request/response model (HTTP), a client asks a server for data and waits for an answer. The client must know the server's address, both must be online at the same time, and the interaction is one-to-one. In publish/subscribe, publishers send messages to topics on a broker, and subscribers receive messages for topics they subscribed to. Publishers and subscribers never contact each other directly; they may not even exist at the same time.

This decoupling has three dimensions. Space decoupling: publishers don't know subscribers. Time decoupling: a subscriber can receive a message published while it was offline (with persistent sessions). Synchronization decoupling: neither side blocks waiting for the other. For IoT, where devices sleep, roam, and fail, this decoupling is not a luxury — it is the only sane architecture.

Consider a building with 50 air-quality sensors and 5 dashboard displays. With request/response, each display would poll 50 sensors — 250 connections, most returning "no new data." With publish/subscribe, each sensor publishes once to its topic; the broker fans the message out to whoever subscribed. Adding a sixth display requires zero changes to the sensors. This many-to-many fan-out with zero reconfiguration is why pub/sub dominates IoT.

Example: A student project first used HTTP polling: 10 sensors × 1 request/5 seconds = 172,800 requests/day, mostly wasted. Switching to MQTT publish/subscribe cut traffic by 95% and let the sensors sleep between readings, doubling battery life.

For your research: The pub/sub vs request/response comparison is a standard related-work discussion. Frame your contribution as exploiting decoupling (e.g., "our design leverages time decoupling to support duty-cycled sensors").

Key takeaway: Publish/subscribe decouples publishers from subscribers in space, time, and synchronization — the architectural reason MQTT fits IoT.

Chapter 3: MQTT Architecture — Brokers, Clients, Topics

An MQTT system has three roles. The broker is the central server that receives all messages and routes them to subscribed clients. The publisher (client) sends messages to topics. The subscriber (client) registers interest in topics and receives matching messages. A single device is usually both publisher and subscriber.

The topic is a UTF-8 string with levels separated by slashes, e.g., campus/building-a/room-101/temperature. Topics are not pre-configured; they come into existence when first used. The broker matches published topics against subscriptions using exact match plus two wildcards: + (single level) and # (multi-level, must be last). Subscribing to campus/+/room-101/temperature gets that room's temperature in every building; campus/# gets everything under campus.

The broker is the scalability chokepoint and the research playground. Popular open-source brokers include Mosquitto (lightweight, single-node), EMQX (clustered, millions of connections), and HiveMQ. Broker choice affects your experiments: Mosquitto is ideal for controlled lab studies, while EMQX lets you study clustering and load balancing. For a paper, always name the broker and version — results are not broker-independent.

Example topic hierarchy for a campus deployment:

campus/building-a/floor-2/room-205/temperature
campus/building-a/floor-2/room-205/humidity
campus/building-a/floor-2/room-205/co2
campus/building-b/floor-1/room-102/temperature

A facilities dashboard subscribes to campus/#. A room controller subscribes to campus/building-a/floor-2/room-205/#.

For your research: Topic design is a legitimate design contribution. A paper can propose and evaluate a topic-naming scheme (e.g., optimizing for wildcard efficiency or access control granularity). Document your hierarchy; reviewers will ask why you chose it.

Key takeaway: Brokers route messages between publishing and subscribing clients using hierarchical topics with + and # wildcards.

Chapter 4: Topic Design and Wildcards

Good topic design determines security, efficiency, and maintainability. Follow these principles. Put the most general level first and the most specific last (site/device/metric, not metric/device/site) so wildcards aggregate naturally. Keep topics short — every byte is transmitted with every message. Use consistent naming (all lowercase, hyphens not spaces). Never embed secrets in topics; topics appear in broker logs.

Design for your access-control model. If different users may access different buildings, make building a topic level so the broker's ACL can grant campus/building-a/# to one user and campus/building-b/# to another. If you instead put the building at the end, per-building authorization becomes impossible with simple ACLs.

Wildcard subscriptions are powerful but costly: campus/# forces the broker to evaluate every message against that subscription. In benchmarks, overly broad wildcards are a common cause of broker CPU saturation. Subscribe as narrowly as your application allows.

Example: A poor design: temperature/room-205/building-a — to get all of building A you cannot use a clean wildcard. A good design: campus/building-a/room-205/temperature — campus/building-a/# gives the whole building, campus/+/+/temperature gives every temperature reading.

For your research: If your paper includes a deployment, include a topic-design subsection with rationale. If you benchmark brokers, test wildcard-heavy workloads explicitly — that is where implementations differ most.

Key takeaway: Design topics general-to-specific, keep them short, align levels with access-control boundaries, and avoid overly broad wildcards.

Chapter 5: Quality of Service Levels

MQTT defines three QoS levels, negotiated per message. QoS 0 (at most once): fire and forget. The message is sent once; it may be lost. Lowest overhead, lowest latency. QoS 1 (at least once): the sender retransmits until it receives an acknowledgment (PUBACK). The message arrives at least once but may arrive duplicated. QoS 2 (exactly once): a four-step handshake (PUBLISH → PUBREC → PUBREL → PUBCOMP) guarantees the message arrives exactly once. Highest overhead.

The critical subtlety: QoS is per-hop, not end-to-end. A publisher may send at QoS 2 to the broker, but a subscriber receiving at QoS 0 can still miss messages. The effective QoS for a subscriber is the minimum of the publish QoS and the subscribe QoS. Many production bugs come from assuming QoS 2 publish guarantees delivery to all subscribers.

Choosing QoS is an engineering trade-off you can measure and publish. QoS 0 suits high-frequency sensor data where losing one reading in a thousand is harmless (temperature every 10 seconds). QoS 1 suits commands and alerts where loss is unacceptable but duplicates are tolerable (turn on a pump — make the command idempotent). QoS 2 suits billing or safety-critical messages where duplicates are dangerous, and you pay roughly 4× the packets of QoS 0.

Example: In an irrigation study, soil-moisture readings used QoS 0 (a missed reading is replaced 10 minutes later), while valve-actuation commands used QoS 1 with idempotent command design (repeating "valve=open" is harmless). This cut total traffic by 40% versus using QoS 1 everywhere, with zero functional loss.

For your research: QoS selection experiments are highly publishable: measure latency, throughput, energy, and delivery rate across QoS levels on your hardware. Report the trade-off curve — that is a result, not just a configuration.

Key takeaway: QoS 0/1/2 trade reliability against overhead; QoS is per-hop; choose per message type and measure the trade-off.

Chapter 6: Retained Messages and Last Will

Two MQTT features handle state and failure. A retained message is stored by the broker per topic; any new subscriber immediately receives the last retained message for topics it subscribes to. This solves the "what is the current state?" problem without polling: a sensor publishes its latest reading as retained, and a dashboard that connects later instantly shows current values instead of waiting for the next reading.

Last Will and Testament (LWT) handles ungraceful disconnection. When connecting, a client registers a will message (topic + payload). If the client disconnects without sending DISCONNECT (power loss, network drop), the broker publishes the will — typically device/sensor-42/status = "offline". This gives the system reliable presence detection without heartbeats.

Clean session / session persistence (v3.1.1) and session expiry (v5) control what the broker remembers for a disconnected client: subscriptions and undelivered QoS 1/2 messages. With a persistent session, a subscriber that sleeps for an hour wakes up to its queued messages. This is time decoupling made concrete — and it costs broker memory, which bounds how many sleeping devices one broker can serve.

Example: A cold-chain monitor publishes temperature as retained QoS 1 every 5 minutes and registers LWT truck-7/status = "disconnected". The logistics dashboard always shows the latest temperature on connect, and instantly flags a truck whose device died — no polling, no heartbeat traffic.

For your research: Retained messages and LWT are underused in published IoT work. A paper that quantifies how retained messages reduce join latency, or how LWT compares with heartbeat-based presence in energy cost, fills a real gap.

Key takeaway: Retained messages give new subscribers instant state; LWT gives reliable failure detection; persistent sessions enable time decoupling at a memory cost.

Chapter 7: MQTT Security — Authentication, Authorization, TLS

MQTT's base specification has minimal security: a username/password field in the CONNECT packet, sent in cleartext unless the transport is encrypted. Real deployments layer three mechanisms. Authentication verifies who the client is (username/password, client certificates, or tokens via MQTT v5 enhanced authentication). Authorization controls what an authenticated client may do (topic ACLs: which topics it may publish/subscribe). Transport encryption (TLS) protects credentials and payloads from eavesdropping and tampering.

The common failure mode is deploying a broker with anonymous access on the public internet — automated scanners find and abuse such brokers within hours (they are used for spam relay and as DDoS reflectors). For any deployment beyond the lab, require TLS with server certificate validation, unique credentials per device, and least-privilege topic ACLs. Client certificates (mutual TLS) are the gold standard for fleets: each device gets its own certificate, compromised devices are revoked individually, and there are no shared passwords to leak.

Example ACL for a sensor: allow publish to farm/plot-3/node-12/#, allow subscribe to farm/plot-3/node-12/commands, deny everything else. Even if the device is compromised, the attacker can only spoof one node's data.

For your research: Security evaluations are very publishable. Measure the cost of TLS on constrained hardware (handshake time, energy, memory) — this is a classic result that remains relevant as hardware evolves. Compare pre-shared-key vs certificate approaches on your target MCU.

Key takeaway: MQTT needs TLS + per-device credentials + topic ACLs; never expose an anonymous broker; measure security's cost on your hardware.

Chapter 8: MQTT Version 5 — What's New

MQTT v5 (OASIS, 2019) keeps the v3.1.1 core and adds features that remove long-standing workarounds. Reason codes on every acknowledgment tell you why something failed (not authorized, quota exceeded) instead of silent drops. Session expiry replaces the binary clean-session flag with a time interval. Message expiry lets a publisher set a lifetime; the broker discards stale messages. Topic aliases replace long topic strings with small integers after first use — a major bandwidth saver for constrained links. User properties are key-value metadata on any packet, enabling tracing and custom headers. Shared subscriptions ($share/group/topic) distribute messages across a group of subscribers for load balancing — previously you needed an external queue. Enhanced authentication supports challenge/response schemes (e.g., SCRAM, Kerberos). Flow control (receive maximum) lets constrained clients limit in-flight messages.

For researchers, v5 matters in two ways: new measurable features (topic-alias bandwidth savings, shared-subscription load distribution) and better failure diagnostics (reason codes make experiments debuggable). When writing a paper, state the protocol version — v3.1.1 and v5 results are not interchangeable.

Example: With topic alias 7 mapped to campus/building-a/floor-2/room-205/temperature (52 bytes), subsequent publishes carry 2 bytes instead of 52 — a 96% header reduction that is directly measurable and reportable.

For your research: A clean, small paper: measure topic-alias savings across realistic topic lengths and publish rates. Or evaluate shared-subscription fairness across subscriber counts. Both are v5-only and under-published.

Key takeaway: MQTT v5 adds reason codes, topic aliases, shared subscriptions, message/session expiry, and enhanced auth — new features, new measurable research questions.

Chapter 9: Choosing and Setting Up a Broker

Broker choice shapes what you can study. Mosquitto (Eclipse, C) is the reference lightweight broker: single-node, tiny footprint, runs on a Raspberry Pi. Ideal for teaching, prototyping, and controlled experiments. EMQX (Erlang) is built for scale: clustering, millions of concurrent connections, built-in observability. Ideal for scalability studies. HiveMQ (Java) targets enterprise with strong tooling. Cloud brokers (AWS IoT Core, Azure IoT Hub) offer managed MQTT with deep cloud integration but per-message pricing that punishes chatty research workloads.

For a research deployment: start with Mosquitto in Docker for reproducibility (pin the version, publish your config). Enable persistence, set max queued messages, configure ACLs from day one, and log at a level that lets you reconstruct experiments. Benchmark with a load tool (e.g., mqtt-benchmark, emqtt-bench) before claiming any performance number — "the broker handled it" is not a result; "the broker sustained 10k msg/s at p99 latency 40 ms on this hardware" is.

Example Docker setup for reproducible research:

mosquitto:2.0.18 with config: persistence true, max_queued_messages 10000,
acl_file /mosquitto/acl, password_file /mosquitto/passwd, listener 8883 with TLS

Publish the config with your paper's artifact. Reproducibility is increasingly required by venues.

For your research: A broker comparison paper (Mosquitto vs EMQX vs HiveMQ on identical workloads) is straightforward, useful, and citable — but only if methodology is rigorous: same hardware, same workload generator, multiple runs, reported variance.

Key takeaway: Match the broker to the study (Mosquitto for control, EMQX for scale); pin versions, publish configs, benchmark before claiming.

Chapter 10: MQTT in Research — Datasets and Experiments

MQTT research falls into recognizable types. Measurement studies characterize real deployments (message sizes, publish rates, QoS usage, disconnection patterns). Benchmarking studies compare brokers, client libraries, or protocol versions under controlled load. Optimization studies propose improvements (better topic-alias strategies, smarter QoS selection, energy-aware retransmission). Application studies use MQTT as infrastructure for a domain problem (agriculture, health) and evaluate the end-to-end system.

Design your experiment around a falsifiable claim: "QoS 1 increases delivery from 97% to 99.9% at 2.1× energy cost on ESP32 over Wi-Fi" beats "we used MQTT and it worked." Control variables: broker, network conditions (use tc/netem to emulate loss/latency), payload sizes, publish rates, client count. Report hardware, library versions, and statistical treatment (runs, confidence intervals).

Example experiment matrix: 3 QoS levels × 3 payload sizes (32 B, 256 B, 1 KB) × 2 network profiles (clean, 5% loss) = 18 conditions × 10 runs. Metrics: delivery rate, end-to-end latency (p50/p99), client energy (measured via INA219), broker CPU. That matrix is a paper's evaluation section.

For your research: Publish your workload generator and configs. The IoT community suffers from non-reproducible benchmarks; a reproducible artifact is itself a contribution reviewers reward.

Key takeaway: Frame MQTT experiments as falsifiable claims with controlled variables, real metrics, and published artifacts.

Chapter 11: Comparing MQTT with CoAP, AMQP, and HTTP

CoAP (Constrained Application Protocol, RFC 7252) is HTTP-like (request/response, URIs, methods) over UDP with tiny headers. It suits device-to-device request/response on extremely constrained networks but lacks native pub/sub (though Observe extends it). Choose CoAP when you need REST semantics on 6LoWPAN-class networks.

AMQP (Advanced Message Queuing Protocol, ISO/IEC 19464) is a full-featured enterprise message queue: exchanges, queues, bindings, transactions. Far richer routing than MQTT but heavier — overkill for small sensors, right for backend integration between brokers and enterprise systems.

HTTP is ubiquitous and firewall-friendly but verbose; fine for gateways and dashboards, wasteful for constrained devices.

The research framing: there is no universally best protocol, only best-fit per constraint set. A strong paper states its constraints (device class, network, message pattern, reliability needs), evaluates 2–3 protocols against them, and reports where each wins. "MQTT vs CoAP for LoRa-based soil sensing" is a better paper than "MQTT is the best IoT protocol."

Example decision table (constrained sensor → gateway): MQTT wins on ecosystem and pub/sub; CoAP wins on per-message overhead over UDP; HTTP loses on overhead but wins on tooling. Your paper should show numbers, not adjectives.

For your research: Protocol-comparison papers are evergreen because hardware and networks evolve. Revisit comparisons every few years on new hardware — that is a legitimate contribution.

Key takeaway: MQTT (pub/sub, TCP), CoAP (REST, UDP), AMQP (enterprise queuing), HTTP (ubiquitous but heavy) — compare against stated constraints with numbers.

Chapter 12: Designing Your MQTT-Based IoT Study

Pull it together into a publishable study plan.

  1. Problem: Name the narrow problem (e.g., "reliable delivery of irrigation commands over lossy rural links").
  2. Hypothesis: State the falsifiable claim (e.g., "QoS 1 with idempotent commands achieves ≥99.5% effective delivery at <2× the energy of QoS 0").
  3. Design: Topics, QoS per message type, retained/LWT usage, broker + version, client hardware + library.
  4. Method: Controlled variables, network emulation, metrics (delivery, latency, energy), runs and statistics.
  5. Baseline: Compare against the obvious alternative (QoS 0 everywhere, or HTTP polling).
  6. Threats: What could invalidate the result (broker tuning, Wi-Fi interference, small sample)? State them.
  7. Artifact: Configs, workload generator, raw data — published.

Write the paper in IEEE format: abstract with numbers, introduction ending in contributions, related work organized thematically, design, evaluation with tables/figures, discussion of limitations, conclusion. Your literature review must include the MQTT standards and at least one measurement or benchmarking study for context.

For your research: This chapter is your template. Replace the example problem with yours and you have an assignment-to-paper pipeline: Assignment 1 (problem + lit review) → Assignment 2 (this design) → Assignment 3 (results) → Assignment 4 (presentation).

Key takeaway: A publishable MQTT study = narrow problem + falsifiable claim + controlled method + baseline + honest limitations + artifact.

Learning Dashboard

# Chapter Core idea Research use
1 What is MQTT Lightweight pub/sub, OASIS/ISO standard Cite standard; quantify bandwidth case
2 Pub/sub vs req/resp Space/time/sync decoupling Frame contribution via decoupling
3 Architecture Broker, clients, topics, wildcards Name broker+version; design topics
4 Topic design General→specific, ACL-aligned Topic scheme as design contribution
5 QoS 0/1/2 Reliability vs overhead, per-hop QoS trade-off experiments
6 Retained + LWT Instant state, failure detection Measure join latency / presence cost
7 Security TLS, per-device creds, ACLs Measure TLS cost on MCU
8 MQTT v5 Aliases, shared subs, reason codes v5-only measurement studies
9 Brokers Mosquitto vs EMQX vs HiveMQ Rigorous broker benchmarks
10 Experiments Falsifiable claims, controlled vars Reproducible artifacts
11 Protocol comparison MQTT/CoAP/AMQP/HTTP trade-offs Constraint-based comparisons
12 Study design Problem→claim→method→artifact Assignment-to-paper pipeline

References

[1] A. Banks and R. Gupta, "MQTT Version 3.1.1," OASIS Standard, Oct. 2014. [2] ISO/IEC 20922:2016, "Information technology — Message Queuing Telemetry Transport (MQTT) v3.1.1," 2016. [3] OASIS, "MQTT Version 5.0," OASIS Standard, Mar. 2019. [4] U. Hunkeler, H. L. Truong, and A. Stanford-Clark, "MQTT-S — A publish/subscribe protocol for Wireless Sensor Networks," in Proc. IEEE COMSWARE, 2008. [5] D. Thangavel et al., "Performance evaluation of MQTT and CoAP via a common middleware," in Proc. IEEE ISSNIP, 2014. [6] N. De Caro et al., "MQTT — A protocol for the Internet of Things," Journal of Communications Software and Systems, 2013. (introductory; prefer [1]–[3] as primary) [7] B. Mishra and A. Kertesz, "The use of MQTT in M2M and IoT systems: A survey," IEEE Access, vol. 8, 2020. [8] E. Park et al., "MQTT vulnerabilities and security: A review," IEEE Access, 2021. (survey; verify current version before citing) [9] W. Shi et al., "Edge Computing: Vision and Challenges," IEEE Internet of Things Journal, vol. 3, no. 5, 2016. [10] L. Atzori, A. Iera, and G. Morabito, "The Internet of Things: A survey," Computer Networks, vol. 54, no. 15, 2010.

Glossary

  • Broker — server that routes MQTT messages between clients
  • Topic — hierarchical string naming a message channel
  • Publish/Subscribe — pattern where senders and receivers are decoupled via topics
  • QoS — Quality of Service: 0 (at most once), 1 (at least once), 2 (exactly once)
  • Retained message — last message on a topic stored by the broker for new subscribers
  • Last Will (LWT) — message published by the broker if a client disconnects ungracefully
  • Wildcard — + (one level) or # (many levels) in subscriptions
  • ACL — access control list limiting what a client may publish/subscribe
  • TLS — transport encryption for MQTT connections
  • Session persistence — broker remembering subscriptions/queued messages for offline clients
  • Topic alias — MQTT v5 small-integer substitute for long topic strings
  • Shared subscription — MQTT v5 load-balanced delivery across a subscriber group

Practice Exercises

  1. Explain in your own words why publish/subscribe fits IoT better than request/response.
  2. Design a topic hierarchy for a 3-floor office with temperature and occupancy sensors. Show two useful wildcard subscriptions.
  3. A sensor publishes at QoS 1 but the subscriber uses QoS 0. What delivery guarantee does the subscriber actually get? Why?
  4. When would you choose QoS 2 over QoS 1? Give a concrete example and justify the overhead.
  5. Write the LWT configuration (topic + payload) for a soil sensor that should report "offline" on failure.
  6. List three security misconfigurations commonly found in MQTT deployments and how to fix each.
  7. Calculate the bandwidth saving of a v5 topic alias for a 60-byte topic published 1,000 times.
  8. Design an experiment comparing Mosquitto and EMQX: state your hypothesis, variables, and metrics.
  9. Compare MQTT and CoAP for a battery-powered LoRa sensor network. Which wins and on what measured basis?
  10. Draft a one-paragraph "contributions" statement for a paper built on Chapter 12's study plan, applied to your own problem.

End of Book 31. Next: Book 32 — TinyML: AI on Microcontrollers.