Resolving Siemens IOT2040 Node-RED Crashes Under High-Rate Azure IoT Messaging

The SIMATIC IOT2040 and its successor IOT2050 are hardened industrial edge gateways designed to live on the plant network and bridge SIMATIC PLC data to cloud back-ends. Node-RED is the de-facto flow runtime on these devices because the Siemens example image and the SIMATIC Industrial OS ship it pre-installed. In dozens of field deployments a recurring failure pattern has emerged: when a flow pushes a JSON payload to Microsoft Azure IoT Hub on a 20-30 ms cycle, the gateway stops responding after four to six hours. The crash presents in three escalating signatures, all rooted in the same group of saturation mechanisms. This article documents the failure modes, identifies the root causes, and provides a verified remediation procedure that works without changing the PLC program.

Problem Description

A typical installation looks like this. A SIMATIC S7-1200 or S7-1500 is polled by Node-RED using either the node-red-contrib-s7 node or the Advanced SIM library, which exposes the S7 data blocks as Node-RED msg.payload objects. The flow then assembles a JSON object, signs it with an SAS token, and forwards it to Azure IoT Hub over MQTT or AMQP using the azure-iot-device Node.js SDK.

At a 20-30 ms cycle the flow generates 33-50 messages per second. After roughly four to six hours of continuous operation one of the following symptoms appears:

1. Node-RED stops. The systemd unit reports inactive (dead) or failed. The SSH daemon is still listening and the Ethernet interfaces are still up. Restarting the unit with systemctl restart node-red clears the issue until the next saturation cycle.
2. Node-RED stops and the SSH daemon briefly stops accepting new connections for one to three minutes. ping still returns replies, indicating the kernel and the Ethernet drivers are alive. After a few minutes SSH comes back, but Node-RED is no longer running and journalctl -u node-red shows no panic — only a normal termination line.
3. The full gateway disappears. Neither ping nor SSH answers, and the unit must be power-cycled. On reboot everything works again. Six IOT2040 units in a single field deployment exhibited this behaviour in parallel.

Reducing the cycle to 1 s or 60 s prevents the crash entirely, which confirms the failure is throughput-driven and not caused by a logic bug in the flow itself.

Affected Hardware and Firmware

The IOT2040 ships with 1 GB of soldered DDR3 and 4 GB of eMMC. The Yocto-based images do not create a swap partition by default. The Intel Quark x1000 SoC is a single Pentium-class 400 MHz core with no GPU and no hardware crypto acceleration. The IOT2050 is a much more capable platform but is still constrained by the IOT2000 class form factor (no fan, no display). Refer to the SIMATIC IOT2040 Equipment Manual (entry ID 109741655) and the SIMATIC IOT2050 Equipment Manual (entry ID 109780672) for the full hardware specification.

Symptom Classification Matrix

If you observe signature C, the next section will help you distinguish a kernel OOM kill from a hardware watchdog reset, because the dmesg evidence is very different for each.

Root Cause Analysis

Five root causes, alone or in combination, account for essentially every reported IOT2040 Node-RED crash in this scenario. Each is documented with the diagnostic evidence that confirms it.

1. Node.js heap exhaustion

Node.js defaults --max-old-space-size to 1.5 GB on 64-bit hosts and to roughly 0.7 GB on 32-bit hosts. The IOT2040 is 32-bit x86, but because only 1 GB of physical RAM is present, the kernel needs to be conservative. In practice, allowing the V8 old generation to grow past 256 MB on an IOT2040 invites a hard OOM. The azure-iot-device SDK retains PendingMessage objects in the heap until the service acknowledges them, and on a slow Azure link (e.g. 1 MBit/s uplink) those pending objects can easily outnumber the available heap cells within an hour.

Confirmation: journalctl -u node-red shows <--- Last few GCs ---> followed by FATAL ERROR: out of memory, or simply a clean exit at the same time the heap ceiling is reached.

2. System RAM pressure and kernel OOM

Even when the Node.js heap is bounded, the rest of the system (kernel page cache, journald, connman) competes for the same 1 GB. With 50 msg/s pushing 1-2 KB JSON over an MQTT TLS connection, the kernel encrypts and writes at line rate, leaving little for the page cache. When free RAM falls below 8 MB, the kernel OOM killer reclaims the largest process, which is usually Node-RED.

Confirmation: dmesg | grep -i oom shows Out of memory: Killed process (node-red).

3. DHCP lease churn on the customer router

The customer reports the device's IP address changes during a crash. The IOT2000 example images default to DHCP on interface X1. When Node-RED disappears, the systemd restart logic on the example image occasionally triggers a connman reconfiguration, which can send a DHCPRELEASE and then re-request a new lease, which the router may satisfy with a different address. The user observes the "IP shifted" symptom. After a power cycle the lease is renewed, but the new address may not be the same as before, breaking any port forwarding or any Azure IoT Hub device provisioning that is keyed to a stable device ID / IP pair.

Confirmation: journalctl -u connman shows DHCPRELEASE and DISCOVER immediately before the crash.

4. File descriptor and socket exhaustion

The default ulimit -n on the SIMATIC Industrial OS is 1024. Each AMQP connection to Azure IoT Hub opens multiple sub-sockets (control, telemetry, c2d). Each unfinished TLS handshake holds a temporary socket. When a burst of retries occurs after a transient network blip, the socket table fills and the Node.js event loop blocks on EAI_AGAIN or EMFILE.

Confirmation: cat /proc/$(pidof node-red)/limits shows Max open files 1024 and lsof -p $(pidof node-red) | wc -l reaches the limit at crash time.

5. Hardware watchdog or eMMC hang

The IOT2040 has a TCO watchdog that, if the BIOS is configured to do so, will hard-reset the platform if the Linux kernel fails to pet the watchdog for more than ~120 s. A hung Node.js event loop that does not call schedule() in time can look like a hang to the kernel. Similarly, an eMMC that has reached its end-of-life threshold can hang the MMC driver for tens of seconds, which the watchdog misreads as a kernel hang. The result is a hard reset that requires a power cycle to recover from.

Confirmation: dmesg | grep -i reset shows Watchdog detected hard LOCKUP or reboot: machine restart with no preceding panic.

Diagnostic Procedure

Execute the following steps on a crashed unit to collect evidence. The order matters — collect volatile information first because some evidence (e.g. /proc/*/status) is lost on reboot.

1. Confirm connectivity: ping -c 4 <iot-ip>, then ssh root@<iot-ip> 'uptime; uname -a'. If SSH is dead, jump to the recovery flow at the end of this section.
2. Capture the systemd journal from the last 8 hours: journalctl --since "8 hours ago" > /tmp/journal-snapshot.txt (volatile — copy off-box before reboot).
3. Inspect Node-RED logs: journalctl -u node-red --since "8 hours ago" | tail -200. Look for the strings FATAL ERROR, out of memory, EMFILE, and the Last few GCs marker.
4. Inspect the kernel ring buffer: dmesg -T | tail -300. Look for oom-kill, watchdog, mmc0 errors, and connman DHCP lines.
5. Check the Node-RED service state: systemctl status node-red. The Active: line tells you whether the unit failed, was killed by the OOM killer, or exited cleanly.
6. Check connman state: connmanctl services and connmanctl state. Confirm whether X1 still has the original IP.
7. Check file descriptor usage: ls /proc/$(pidof node-red)/fd | wc -l — if it is close to 1024, you have socket exhaustion.
8. Check the eMMC health: cat /sys/block/mmcblk0/device/life_time on Industrial OS. A value of 0x06 or higher means the device is past 80% of its rated endurance.

Recovery flow for signature C (no ping, no SSH):

1. Power-cycle the unit.
2. Immediately after boot, before the flow starts, run journalctl --since "-2 minutes" > /tmp/post-reboot.txt to capture the kernel boot and any prior-shutdown messages.
3. Copy /tmp/post-reboot.txt, the dmesg buffer, and the connman state file off-box before they rotate.

Step-by-Step Resolution

The fix is a combination of static addressing, Node.js heap tuning, file-descriptor tuning, batching, and firmware updates. Apply the steps in order; each one is necessary to clear one of the five root causes listed above.

Step 1: Convert DHCP to a static IP

On the SIMATIC IOT2000 example image, the network is managed by connman. Use connmanctl to apply a static configuration so the address cannot change after a connman restart:

connmanctl
> services
> config <service-name> ipv4 manual 192.168.0.50 255.255.255.0 192.168.0.1
> config <service-name> nameservers 192.168.0.1
> enable ipv4

Verify with connmanctl state and ip addr show dev eth0. Document the IP in the device's HMI or asset record so future engineers can find it.

Reference: Siemens Knowledge Base entry 109756318 - Setting up a static IP on the SIMATIC IOT2000.

Step 2: Cap the Node.js heap

Edit the systemd unit for Node-RED. On Industrial OS the unit file is /etc/systemd/system/node-red.service:

[Service]
Environment="NODE_OPTIONS=--max-old-space-size=256"
ExecStart=/usr/bin/node --max-old-space-size=256 \
  /usr/lib/node_modules/node-red/red.js -u /root/.node-red
LimitNOFILE=8192

Reload with systemctl daemon-reload and restart. The 256 MB ceiling is a safe starting point on the IOT2040 with 1 GB total RAM. On the IOT2050 with 2 GB you can raise it to 512 MB. Use node --v8-options | grep max-old to confirm the value was applied.

Step 3: Raise the file descriptor limit

Add the LimitNOFILE=8192 line shown above and add the following to /etc/security/limits.conf for the root user:

root  soft  nofile  8192
root  hard  nofile  16384

Re-login for the limits to take effect on interactive shells. The Node-RED process inherits the systemd value directly.

Step 4: Batch the Azure IoT messages

Sending 50 msg/s to Azure IoT Hub is well above the IoT Hub throttling limit of 20 msg/s/unit for the S1 tier. Above that limit, IoT Hub returns throttling errors and the SDK retries, which piles up PendingMessage objects. Two remediations are available:

• Reduce the source cycle to 1 s (50× lower) and accept the lower telemetry fidelity.
• Batch the data with the azure-iot-device sendBatch API. The recommended pattern is to accumulate messages in a node-red-contrib-queue-gate or a 1 s timer node, then call sendBatch on a 1 s interval. This keeps the 20 ms input rate but emits one Azure call per second carrying up to 50 messages.

Reference: Azure IoT Hub device-to-cloud messaging and IoT Hub scaling guidance.

Step 5: Update Node-RED and the Azure IoT SDK

Node-RED 0.18.x is built on Node.js 4.x, which is past end-of-life. Upgrade the firmware to SIMATIC Industrial OS V2.5 or later, which ships Node.js 12 LTS and Node-RED 1.2+. The newer Azure IoT SDK (1.17+) has improved back-pressure handling that does not retain as many in-flight objects. Refer to the SIMATIC Industrial OS V2.5 release notes for the supported upgrade path from the IOT2000 example image.

Step 6: Disable or reconfigure the TCO watchdog

The IOT2040 BIOS exposes a TCO watchdog that, in the default shipping configuration, hard-resets the platform if the kernel does not pet the watchdog for 120 s. If your flow legitimately blocks for longer than 120 s (e.g. during a TLS handshake on a slow uplink), disable the watchdog in the BIOS setup or pass nowatchdog on the kernel command line in /boot/grub/grub.cfg:

linux /boot/bzImage root=/dev/mmcblk0p2 console=ttyS0,115200 nowatchdog

Reference: Siemens Knowledge Base 109762255 - Disabling the TCO watchdog on the IOT2040.

Step 7: Move logs off eMMC

Configure journald to keep logs in RAM and ship them off-box. Edit /etc/systemd/journald.conf:

[Journal]
Storage=volatile
RuntimeMaxFileSize=64M

Alternatively, install a persistent SD card and mount it at /var/log with a journaling file system such as ext4 with data=writeback.

Hardening and Long-Term Reliability

Once the unit is no longer crashing, apply the following hardening measures to make the next deployment reproducible and to detect future regressions automatically.

1. Add a watchdog user-space daemon that pings the TCO watchdog only if the Node-RED process is healthy. systemd already does this; verify with journalctl -u systemd-shutdownd.
2. Enable connman's PreferredTechnologies for Ethernet only, so a flaky Wi-Fi connection cannot displace the wired link.
3. Configure Azure IoT Hub device twin desired properties to report the heartbeat, free RAM, and uptime. Use these to detect creeping heap pressure before it becomes a crash.
4. Set up an logrotate job that copies the journal buffer to a remote syslog server every 15 minutes.
5. Schedule a monthly smartctl -a /dev/mmcblk0 run (if installed) and raise an alert when the eMMC life_time field exceeds 0x05.

For fleet rollouts, treat the changes above as a single upgrade bundle. The IOT2050 already includes most of them, so the bundle is only mandatory for the older IOT2040 hardware.

Verification and Acceptance Test

Run the following acceptance test after the fix to prove the gateway is no longer susceptible to the failure. The test reproduces the original 20-30 ms input cycle and runs for 12 hours continuously.

1. Power-cycle the unit to start from a known state.
2. Run a stress flow that emits 50 msg/s of 1 KB JSON to Azure IoT Hub for 12 hours.
3. At t=0, 1h, 4h, 8h, 12h capture: free -m, ps -o rss,vsz,etime -p $(pidof node-red), ls /proc/$(pidof node-red)/fd | wc -l, and journalctl -u node-red --since "-1 minute" | wc -l.
4. At the end of the run, verify that the RSS of Node-RED is stable to within ±20 % and that no FATAL ERROR or oom-kill line appears in the journal.
5. Confirm that the unit's IP address is unchanged across the entire run (ip addr show dev eth0 before and after).

Acceptance criteria: zero FATAL ERROR, zero OOM events, RSS drift < 20 %, IP address unchanged. Any failure requires re-running the test with one stress parameter relaxed (e.g. cycle to 50 ms instead of 20 ms) to localize the remaining headroom.

Frequently Asked Questions