Fixing Mosquitto TLS-PSK Malformed Packet on Port 8883

Problem Details

An air-gapped MQTT broker is configured for TLS-PSK on port 8883 with username/password authentication layered on top. Every mosquitto_pub attempt terminates during the TLS/CONNECT phase before any PUBLISH is accepted.

Observed broker log on the stock Debian 11 (bullseye) Mosquitto package:

2022-09-09T13:19:20: New connection from ::1:45822 on port 8883.
2022-09-09T13:19:20: Client <unknown> disconnected due to protocol error.

After purging the distribution package and installing Mosquitto 2.0.14 from the upstream Mosquitto repositories, the same command and the same broker configuration produce a different, more precise message:

2022-09-09T14:22:33: mosquitto version 2.0.14 running
2022-09-09T14:24:00: New connection from ::1:50112 on port 8883.
2022-09-09T14:24:00: Client <unknown> disconnected due to malformed packet.

Key diagnostic details in the log lines:

• Client <unknown> means the broker never parsed a valid CONNECT packet, so the client ID was never registered. Username/password ACLs are therefore not the cause — authentication is never reached.
• The connection is accepted at TCP level (New connection from ::1:50112), so the listener is bound and reachable. The failure is above TCP.
• The source address ::1 shows the loopback resolved to IPv6 first; the listener is bound on both IPv4 and IPv6. Forcing -h 127.0.0.1 instead of -h localhost produces the same failure, which rules out an address-family problem.

Failing Command and Two Independent Faults

The publish command under test:

echo "testing" | mosquitto_pub \
  -u "testclient" -P "passwordhere" \
  -h "localhost" -t "testlog" \
  --psk "db272179ae67e2df490c8a26405d77f1963c5c0107c33fb6352eeaca52a83f4830f141b92b14fd9d0f5e360ed96b1f8b1070c07b42638c22afe8ccab2a8a074b" \
  --psk-identity "1b67ecac-94b4-4238-9678-2ab3e3668402" \
  -i "local" -d -l

Two separate defects were present in the original test setup. Fix both.

Root Cause

1. -L mqtts:// forces certificate-based TLS

Passing a URL with the mqtts:// scheme to mosquitto_pub -L puts the client into certificate-validating TLS mode. The client then expects the server to present an x509 certificate chain it can verify. A broker configured only for TLS-PSK presents no certificate, the TLS handshake fails, and the broker reports the aborted session as a protocol error or malformed packet depending on version. Removing -L alone does not fix the connection, which is what makes this bug pair confusing — the second fault masks the first.

2. psk_hint is listener-scoped, not global

This is the actual blocker. In mosquitto.conf, options placed after a listener directive apply to that listener; options placed before any listener line belong to the default/global context. psk_hint must be attached to the specific TLS-PSK listener. When it is declared in the global section while the 8883 listener is defined later, the PSK listener has no hint bound to it, PSK ciphersuites are not properly offered on that socket, and every client is dropped before CONNECT is parsed — exactly matching the Client <unknown> log signature.

Solution: Per-Listener TLS-PSK Configuration

1. Define the PSK listener with all PSK options inside the listener block:

   # /etc/mosquitto/mosquitto.conf
   per_listener_settings true
   
   listener 8883
   psk_hint mypskhint
   psk_file /etc/mosquitto/psk.txt
   require_certificate false
   allow_anonymous false
   password_file /etc/mosquitto/passwd

   With per_listener_settings true, authentication and ACL options are evaluated per listener, so password_file and allow_anonymous must also sit under the listener they govern. Setting it and then leaving auth options in the global block is a second common way to break this configuration.

2. Populate the PSK file as identity:hex-key, one entry per line, no 0x prefix, no spaces:

   1b67ecac-94b4-4238-9678-2ab3e3668402:db272179ae67e2df490c8a26405d77f1963c5c0107c33fb6352eeaca52a83f4830f141b92b14fd9d0f5e360ed96b1f8b1070c07b42638c22afe8ccab2a8a074b

   Verify the key is a contiguous hex string. Line-wrapping introduced by copy/paste (a space inside the hex, as appeared in the original test) yields an invalid key and a handshake failure that looks identical to a misconfigured listener.

3. Create the password entry for the MQTT-layer credential:

   mosquitto_passwd -c /etc/mosquitto/passwd testclient

4. Restrict permissions and restart:

   chown mosquitto:mosquitto /etc/mosquitto/psk.txt /etc/mosquitto/passwd
   chmod 600 /etc/mosquitto/psk.txt /etc/mosquitto/passwd
   systemctl restart mosquitto

5. Publish with PSK flags and no -L:

   echo "testing" | mosquitto_pub \
     -u testclient -P passwordhere \
     -h localhost -t privcdnlog \
     --psk "db272179ae67e2df490c8a26405d77f1963c5c0107c33fb6352eeaca52a83f4830f141b92b14fd9d0f5e360ed96b1f8b1070c07b42638c22afe8ccab2a8a074b" \
     --psk-identity "1b67ecac-94b4-4238-9678-2ab3e3668402" \
     -d -l

   When --psk is supplied without an explicit -p, the client targets the TLS port; keep the listener on 8883 to match. Add -p 8883 explicitly if you run multiple listeners.

Verification

1. Run the publisher with -d. A working PSK session shows client-side Client local sending CONNECT followed by Client local received CONNACK (0). Return code 0 confirms both the TLS-PSK handshake and the username/password check passed.
2. Check the broker log. Success replaces Client <unknown> disconnected with a named client line, e.g. New client connected from 127.0.0.1 as local (p2, c1, k60, u'testclient'). The presence of the client ID and username proves CONNECT was parsed.
3. Confirm end-to-end delivery with a second terminal before publishing:

   mosquitto_sub -h localhost -t privcdnlog -u testclient -P passwordhere \
     --psk "db2721...a074b" --psk-identity "1b67ecac-94b4-4238-9678-2ab3e3668402" -d

4. Negative test the auth layer: repeat the publish with a wrong -P value. The TLS handshake should still succeed and the broker should reject at CONNACK with a bad-username-or-password result, not with a malformed packet. If you still get malformed packet, the fault is still in the TLS/listener layer, not in credentials.
5. Negative test the hint scope: temporarily move psk_hint above the listener line, restart, and confirm the Client <unknown> failure returns. This confirms the root cause rather than an incidental fix.

Diagnostic Reference

Upgrading from the Debian 11 distribution package to Mosquitto 2.0.14 from the upstream repositories does not fix the configuration fault, but it is worth doing first: the newer build reports malformed packet instead of the generic protocol error, which narrows the search considerably. Purge the distribution package completely before installing to avoid two config trees and two unit files.