Securing MQTT Communication using Mosquitto MQTT Broker

In this article, we will configure the mosquitto MQTT broker to use Username-Password and TLS/SSL security mechanisms. We will be using openssl to create our own Self-Signed Certificate authority (CA), Server keys and certificates. You can refer SSL TLS Explained article to learn more about SSL, Handshake mechanism, Various certificates, etc.

Unsecured-Unauthenticated MQTT Communication

Mosquitto Broker Configurations:
- Settings:
  - listener 8883
  - listener 1883
  - allow_anonymous true
- listener:
  - Mosquitto Broker listens on a port/ip address combination with above keyword.
  - By using this variable multiple times, mosquitto can listen on more than one port.
  - The port number to listen on must be given. Optionally, an ip address or host name may be supplied as a second, optional argument.
    - Port number is mandatory, but not IP.
  - listener by default listen on all network interfaces, but can also have IP address, or Hostname, or Unix Socket path, which can be used to restrict the broker access to specific interface and network.
  - Only one listener should be sufficient.
  - But, I faced issues to operate mosquitto_sub client on port 8883, it resulted in error: Client <unknown> disconnected due to malformed packet, but it worked fine on port 1883, for some unknown reason, so added both listeners.
- allow_anonymous:
  - Boolean value that determines whether clients that connect without providing a username are allowed to connect.
  - Defaults to false.
  - Setting this parameter to true enables the remote MQTT clients, running on other machines like CloudRail.Box to connect to Mosquitto Broker running on laptop.
  - Without this option enabled, Mosquitto Broker can only connect to local MQTT clients, i.e. clients running on same laptop.
Run Mosquitto MQTT Broker:
```
$ mosquitto -c mosquitto.conf -v
```

Run Mosquitto Clients Receiver/Publisher:

$ mosquitto_sub -t # -v
$ mosquitto_pub -m <message> -t /<topic>

Username and Password Authentication for Mosquitto MQTT

Mosquitto MQTT broker can be configured to require client authentication using a valid username and password for successful connection.
The username and password combination is transmitted from the mosquitto clients to the broker in clear text format, and it is not secure in any form of encryption.(~SSL)
Password File:
- Create Username and Password file:
```
$ mosquitto_passwd -c <passwordfilename> <username>
```
- This command will ask you to set the password.
- It will not echo the characters, so be careful and remember the password.
- This creates a passwordfile which has username and encrypted password in <username>:<encrypted-password>.
- This file needs to be provided to Mosquitto Broker in its config file with parameter password_file.
Mosquitto Broker Configurations:
- Parameter Settings:
  - allow_anonymous false
    - Set this to false, as we are implementing the basic security, access control mechanism.
  - password_file <password_file_path>
    - Provide the passwordfile created with mosquitto_passwd utility with this parameter, along with its absolute/relative path.
Run Mosquitto MQTT Broker:
```
$ mosquitto -c mosquitto.conf -v
```

Run Mosquitto Clients Receiver/Publisher:

$ mosquitto_sub -t # -u <username> -P <password>
$ mosquitto_pub -m <message> -t /<topic>

Secure MQTT Communication with TLS/SSL Certificates

Authentication with TLS/SSL certificates is another way of authenticating & securely connecting the clients to the broker.
A client certificate identifies the client just like the server certificate identifies the server.
Normally certificates are created and distributed to each client that connects to the server/broker that requires them.
We can use certificates in combination with username and password authentication.
Mosquitto Broker Configurations:
- Parameter Settings:
  - listener 8883
    - Recommended port for MQTT over TLS is 8883, but this must be set manually.
  - allow_anonymous false
  - cafile <ca-certificate-file-path>
    - Certificate Authority certificates that will be considered trusted when checking incoming client certificates.
  - certfile <broker-certificate-file-path>
    - Path to the PEM encoded server/broker certificate.
  - keyfile <broker-private-key-file-path>
    - Path to the PEM encoded keyfile.
  - require_certificate true
  - use_identity_as_username true
OpenSSL Configuration
- Required to create for SAN Certificates required to use Private/Public IP Addresses instead of a Domain name/URLs.
- Backup openssl configs:
```
$ cp /etc/openssl<version>/openssl.cnf /tmp/openssl.cnf
```
- Add Private/Public IP Addresses entries to SAN extensions:
```
$ SAN='\n[SAN]\nsubjectAltName=IP:<IP-Address>,IP:<IP-Address>'
$ 
$ echo -e "$SAN" >> /tmp/openssl.cnf
```

Self-Signed CA Certificates:

Create Certificate Authority:

$ openssl req -new -x509 -days 365 -extensions v3_ca -keyout ca.key -out ca.crt

Sample output of above command, Fill all details:

---
Enter PEM pass phrase: <password-for-ca-key>
Verifying - Enter PEM pass phrase: <password-for-ca-key>
---
Country Name (2 letter code) [AU]:IN
State or Province Name (full name) [Some-State]:<State>
Locality Name (eg, city) []:<City>
Organization Name (eg, company) [Internet Widgits Pty Ltd]:<Company-Name>
Organizational Unit Name (eg, section) []:<Org-Unit>
Common Name (e.g. server FQDN or YOUR name) []:<Website-URL>
Email Address []:<support-email-address>
---

Important Note: The Common Name for the CA cannot be the same as the Common Name for the client and server certificates.
- Probable Error: OpenSSL Error[0]: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
Note down the PEM pass phrase, you will need this at a next stage while creating the broker & client certificates.

Check generated CA Certificate, specifically Subject field, which reflect all information provided in above step:
```
$ openssl x509 -in ca.crt -text -noout
```

MQTT Broker/Server Certificate:

Create Private Key for Broker:
```
$ openssl genrsa -out broker.key 2048
```

Create a Certificate Signing Request(CSR) file from Broker key:

$ openssl req -out broker.csr -key broker.key -new -config /tmp/openssl.cnf

Sample output of above command, Fill all details:

---
You are about to be asked to enter information that will be incorporated into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
---
Country Name (2 letter code) [AU]:IN
State or Province Name (full name) [Some-State]:<State>
Locality Name (eg, city) []:<City>
Organization Name (eg, company) [Internet Widgits Pty Ltd]:<Company-Name>
Organizational Unit Name (eg, section) []:<Org-Unit>
Common Name (e.g. server FQDN or YOUR name) []:<IP Address of the MQTT Broker> or localhost
Email Address []:<support-email-address>

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:              
An optional company name []:
---

Important Note: The Common Name for the CA cannot be the same as the Common Name for the client and server certificates.
Optional fields: A challenge password []: & An optional company name []:
Check generated MQTT Broker Certificate Signing Request:
```
$ openssl req -in broker.csr -text -noout
```

Create a MQTT Broker Certificate(.crt) from above CSR Request:

$ openssl x509 -req -in broker.csr -CA ../ca/ca.crt -CAkey ../ca/ca.key -CAcreateserial -out broker.crt -days 365 -extensions SAN -extfile ../openssl.cnf

Important Note: You need to provide PEM pass phrase used in CA certificate creation hereby.

Sample output:

Certificate request self-signature ok
subject=C = IN, ST = Maharashtra, L = Pune, O = CloudRail, OU = IoT, CN = 10.149.0.30, emailAddress = contact@cloudrail.com
Enter pass phrase for ../ca/ca.key:

Check generated MQTT Broker Certificate, specifically Subject & SAN fields, which reflect all information provided in above step:
```
$ openssl x509 -in broker.crt -text -noout
```

MQTT Client Certificate:

Create Private Key for Client:
```
$ openssl genrsa -out client.key 2048
```

Create a Certificate Signing Request(CSR) file from Client key:

$ openssl req -out client.csr -key client.key -new -config /tmp/openssl.cnf

Sample output of above command, Fill all details:

---
You are about to be asked to enter information that will be incorporated into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
---
Country Name (2 letter code) [AU]:IN
State or Province Name (full name) [Some-State]:<State>
Locality Name (eg, city) []:<City>
Organization Name (eg, company) [Internet Widgits Pty Ltd]:<Company-Name>
Organizational Unit Name (eg, section) []:<Org-Unit>
Common Name (e.g. server FQDN or YOUR name) []:<IP Address of the MQTT Broker> or localhost
Email Address []:<support-email-address>

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:              
An optional company name []:
---

Important Note: The Common Name for the CA cannot be the same as the Common Name for the client and server certificates.
Optional fields: A challenge password []: & An optional company name []:
Check generated Client Certificate Signing Request:
```
$ openssl req -in client.csr -text -noout
```

Create a Client Certificate(.crt) from above CSR Request:

$ openssl x509 -req -in broker.csr -CA ../ca/ca.crt -CAkey ../ca/ca.key -CAcreateserial -out client.crt -days 365 -extensions SAN -extfile ../openssl.cnf

Important Note: You need to provide PEM pass phrase used in CA certificate creation hereby.

Sample output:

Certificate request self-signature ok
subject=C = IN, ST = Maharashtra, L = Pune, O = CloudRail, OU = IoT, CN = 10.149.0.30, emailAddress = contact@cloudrail.com
Enter pass phrase for ../ca/ca.key:

Check generated Client Certificate, specifically Subject & SAN fields, which reflect all information provided in above step:
```
$ openssl x509 -in client.crt -text -noout
```

MQTTS Communication Testing:

Run Mosquitto MQTT Broker:
```
$ mosquitto -c mosquitto.conf -v
```

Run Mosquitto Client Receiver:

$ mosquitto_sub -p <1883/8883> --cafile /<path-to-ca-certificate>/ca.crt --cert /<path-to-client-certificate>/client.crt --key /<path-to-client-key>/client.key -h <IPAddress-or-localhost-or-BrokerDomainURL> -t <topic>

Run Mosquitto Client Publisher:

$ mosquitto_pub -p <1883/8883> --cafile /<path-to-ca-certificate>/ca.crt --cert /<path-to-client-certificate>/client.crt --key /<path-to-client-key>/client.key -h <IPAddress-or-localhost-or-BrokerDomainURL> -m <message> -t <topic>

What is SAN Certificate?

A Subject Alternate Name (or SAN) certificate is a digital security certificate which allows multiple hostnames to be protected by a single certificate.
A SAN certificate may also be called a Unified Communication Certificate (or UCC), a multi-domain certificate, or an Exchange certificate.
SAN certificates can be used on unlimited multiple servers concurrently and can be used on unlimited IP addresses with multiple, concurrent private keys (which is great for hosting and virtual hosts).
The entries in any SSL.com SAN certificate: can be a Fully Qualified Domain Name (FQDN) and can be a wildcard domain name (i.e. .domain.com or .store.domain.com) but NOT a multiple-level wildcard (like ..domain.com).
CA Certificate can not be a “Subject Alternative Name (SAN)” certificate, it results in an Error:
- OpenSSL Error[0]: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca.
Error: {"message":"Device muruMsT0IhwpFZeNNPlaYxFjx11Uxikv failed to connect with err Error: Hostname/IP doesn't match certificate's altnames: \"IP: 10.149.0.30 is not in the cert's list: \"\n","timestamp":"2022-06-29T09:45:18.080Z","type":"out","process_id":9,"app_name":"main"}
- Reason:
  - This Error appeared for me while using Self-Signed CA, Server, Client certificates with NodeJS Application, but Mosquitto MQTT clients/broker had no issues using the same self-signed certificates. Also these certificates were created for a Private IP address, and not for any domain URL, as is the usual case. I was testing the local MQTTS communication over local LAN so private IP addresses makes sense.
  - The problem is as indicated that, a Node ‘mqtt’ module seems to need a SAN certificate for Private IP Addresses.
    - According to the CA Browser forum, there may be compatibility issues with certificates for IP addresses unless the IP address is in both the commonName and subjectAltName fields. This is due to legacy SSL implementations which are not aligned with RFC 5280, notably, Windows OS prior to Windows 10.

Encountered Errors & Reasons for them:

Error: mosquitto mqtt tls certificate error Client auto-5F29B532-D3F5-7C93-EF6A-0289F5C3DAEA disconnected, not authorised.
- Reason: Client certificates were not created and broker certificates were tried to use, so this error.
Error: OpenSSL Error: error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
- Reason:
  - TLS expects the connection to be directed to the same CN (Common Name) as the one used in the generation step of the client.csr request file.
  - The CN (Common Name) should be the same as the server/broker hostname if no SAN extension is used (See What is SAN Certificate).
- References:
  - Client using libmosquitto gets “tlsv1 alert internal error” using TLS, works fine without TLS
  - MQTTS : How to use MQTT with TLS?
Error: {“message":"Device XdtNrVk5zzZ5Fvmf8adL9R0dJeG6inkG failed to connect with err Error: write EPROTO 1996375408:error:1409441B:SSL routines:ssl3_read_bytes:tlsv1 alert decrypt error:../deps/openssl/openssl/ssl/s3_pkt.c:1498:SSL alert number 51\n1996375408:error:1409E0E5:SSL routines:ssl3_write_bytes:ssl handshake failure:../deps/openssl/openssl/ssl/s3_pkt.c:659:\n\n","timestamp":"2022-06-29T09:34:12.858Z","type":"out","process_id":9,"app_name":"main"}
- Reason: Certificate mismatch, older certificate used.
New connection from 10.149.0.1:59884 on port 8883.
OpenSSL Error[0]: error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding
OpenSSL Error[1]: error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed
OpenSSL Error[2]: error:0D0C5006:asn1 encoding routines:ASN1_item_verify:EVP lib
OpenSSL Error[3]: error:1417C086:SSL routines:tls_process_client_certificate:certificate verify failed Client <unknown> disconnected: Protocol error.
- Reason: Certificate mismatch, older certificate used.

References:

PREVIOUSAWS IoT Security: Device x.509, CA Certificates

NEXTPravega Sensor Collector: A Pravega Agent for IoT devices