Resolving Node-RED Crashes on SIMATIC IOT2040 Gateways

Overview: Node-RED on the SIMATIC IOT2040

The SIMATIC IOT2040 is an industrial IoT gateway from Siemens based on an Intel Quark x86 SoC (x1020) with 1 GB RAM and 8 GB internal eMMC storage. It runs a Yocto-based Linux image (the SIMATIC IOT2000 SDK / Example Image) and is intended as an edge device to bridge field-level data (typically S7 / PROFINET / Modbus TCP) with higher-level IT systems, MQTT brokers, cloud services, and HMIs.

Node-RED is a flow-based development tool built on Node.js that many engineers deploy on the IOT2040 because it allows PLC programmers to implement MQTT publish/subscribe, REST endpoints, and dashboard logic without learning a full high-level language. The trade-off is that Node-RED inherits the runtime characteristics of Node.js: a single event loop, garbage-collected JavaScript heap, and a large dependency tree of third-party nodes — including several that load native C++ add-ons compiled against the IOT2000 SDK's libc/glibc and Node ABI version.

When the IOT2040 is moved from the lab bench to a real production cell, intermittent Node-RED crashes are one of the most commonly reported stability problems. The crash is rarely caused by Node-RED itself; it is almost always a downstream node, a misconfigured flow, or an operating-system resource limit on the constrained IOT2040 hardware.

Hardware and Software Boundary Conditions

Parameter	Value / Limit	Source
CPU	Intel Quark x1020, single core @ 400 MHz	Siemens IOT2040 manual
RAM	1 GB DDR3 (approx. 600 MB usable after kernel + base OS)	Siemens IOT2040 manual
Storage	8 GB eMMC, no swap by default	Siemens IOT2000 SDK
OS	Yocto-based Linux, kernel 4.x (IOT2000 Example Image)	Siemens support entry 109741654
Node.js (bundled)	v6.x or v8.x (depends on image revision)	Siemens IOT2000 SDK release notes
Node-RED (bundled)	v0.18.x to v1.x depending on image	Siemens IOT2000 SDK release notes
Default open file descriptors (ulimit -n)	1024 (often insufficient for many sockets)	Process default on Yocto Poky

The 400 MHz single core and the absence of swap mean that any leak in the Node.js heap, any blocking call, or any flood of inbound socket connections will surface as a hard crash far sooner on an IOT2040 than on a desktop PC. Always size your flows against these limits, not against a dev laptop.

Symptom Catalog: What an Unstable Node-RED Looks Like

Before diving into root causes, classify the symptom. Each one points to a different subsystem:

1. Node-RED process (node-red) disappears from ps, no journald entry, port 1880 closed — the Node.js interpreter segfaulted. Almost always a native C++ addon in a node has crashed.
2. Node-RED restarts every few minutes — the process is being killed by the kernel OOM killer, or by a watchdog timer (e.g. systemd Restart=on-failure tripping).
3. Node-RED process is alive but does not respond; curl localhost:1880 hangs — the event loop is blocked; usually an exec / spawn call to a shell, or an infinite synchronous loop in a function node.
4. Node-RED loses connection to S7 PLC and never reconnects until restart — the S7 / node-s7comm node is in a bad state; this is the most field-reported failure mode.
5. Node-RED fails to auto-start at boot — race between network bring-up, MQTT broker, and the Node-RED service; or nodesDir contains a node that fails to load and throws an unhandled exception during startup.
6. Editor hangs, deploying a flow takes 30+ seconds — the editor is on, the runtime is shipping flow metadata to disk inefficiently, or the filesystem is full on the eMMC.

Root Cause: Why Node-RED Crashes on the IOT2040

Node-RED itself is written in pure JavaScript. JavaScript exceptions inside a function node or a flow are caught and surfaced as catch nodes; they do not terminate the runtime. The Node-RED runtime is, in practice, very robust against logic errors in user flows.

Crashes almost always trace back to one of four classes:

RC1. Native C++ Add-on Crashes in Contributed Nodes

Several commonly used Node-RED nodes bind to native C++ libraries through node-gyp. Examples include the node-serialport core, the node-rpio-style GPIO bindings, and the nodes7 / s7comm S7 node. These C++ add-ons are compiled against a specific Node.js ABI version and a specific glibc/musl version.

When the bundled Node.js version in the IOT2000 Example Image is updated (e.g. from Node 6 to Node 8) without recompiling the native add-ons, you get a load-time crash or, more insidiously, a crash after the first N messages when a code path enters the native bridge. Symptoms are a Segmentation fault line in journalctl -u node-red followed by the process exiting with signal 11.

Mitigation: keep the Node.js version pinned to the one shipped with the Siemens image and do not mix npm install -g packages compiled on a different host. If you must upgrade Node, rebuild all native dependencies on the IOT2040 itself with the SDK's toolchain.

RC2. Bugs or Misuse in Custom / Community Nodes

The Node-RED ecosystem contains thousands of contributed nodes. Quality varies. Common defects that manifest on the IOT2040:

• Unbounded array growth in a function node (heap leak, eventually OOM-killed).
• Nodes that call setInterval without ever calling clearInterval, leaking timers.
• Nodes that use synchronous file I/O on the eMMC, which is slow and wears out the flash.
• Nodes that do not handle TCP ECONNRESET cleanly and throw an uncaught exception that propagates out of the node and into the runtime.

RC3. Async / Event-Loop Misuse in User Flows

Node-RED's runtime is single-threaded. Heavy use of setInterval, setTimeout, and async / await in function nodes can starve the event loop. The most common offender is a function node that does a node.warn or node.log in a tight loop driven by an inject at 1 ms or by an MQTT subscription receiving a high message rate.

The visible symptom is not a crash but a Node-RED process that is alive and unresponsive, and eventually the watchdog (systemd or an external ping) declares it dead. From a process-supervision perspective, the result is the same: the runtime is down.

RC4. S7 Communication Node Issues (S7-1200 / S7-1500)

The single most field-reported source of instability on the IOT2040 is the S7 communication node family used to read/write data to a SIMATIC S7-1200 or S7-1500. Common failure modes:

• The S7 connection drops when the PLC goes to STOP / RUN, when the PROFINET cable is unplugged, or when the PLC performs an online firmware update.
• Reading DB areas that are larger than the PLC allows in a single PDU triggers a 0x03 (Object does not exist) or 0x05 (Address out of range) error that is not handled and kills the node.
• The nodes7 package in particular has historically used a fixed internal connection state machine that does not always survive a TCP RST from the S7 CPU's comm service.
• Slot / rack configuration that matches the S7-1500 CPU in TIA Portal but is wrong for the actual connection (e.g. slot 0 vs slot 1) produces a silent disconnect that re-occurs every reconnect cycle, filling the journal and eventually OOMing.

RC5. Operating-System Resource Exhaustion

The IOT2040 ships with conservative defaults inherited from Poky / Yocto:

• ulimit -n = 1024 open file descriptors. A Node-RED flow with 50 S7 connections, 10 MQTT subscriptions, and 5 HTTP endpoints can easily exhaust this.
• No swap. When the Node.js heap approaches the RSS limit, the OOM killer fires and Node-RED is the largest RSS process — so it is the first to die.
• Read-only rootfs with overlay. A flow that writes log files to /tmp will fill the tmpfs and the write will fail with ENOSPC, which can crash nodes that do not check the return value of fs.write.

Solution 1: Hardening the Node-RED Configuration for Production

The single highest-leverage change is to disable the editor in production. The editor is a WebSocket-based IDE; it consumes RAM and CPU even when no one is connected, and every Deploy action triggers a full flow restart internally. In a production cell, the editor has no business running.

Editing settings.js

On the IOT2000 Example Image, Node-RED's user data lives under /home/iot/.node-red/. Edit settings.js:

module.exports = {
    // Disable the editor entirely for production
    disableEditor: true,

    // Lock the admin API behind HTTP basic auth
    adminAuth: {
        type: "credentials",
        users: [ /* generate with: node -e "console.log(require('bcryptjs').hashSync('YourPassword',8))" */ ]
    },

    // Disable the default in-memory context store; use file store
    contextStorage: {
        default: { module: "localfilesystem" },
        memory:    { module: "memory" }
    },

    // Flow file location on the persistent partition
    flowFile: "/home/iot/.node-red/flows_iot2040.json",

    // Diagnostic settings
    logging: {
        console: {
            level: "info",
            metrics: false,
            audit: false
        }
    },

    // Disable palette editor (no npm install

from inside the editor) editorTheme: { projects: { enabled: false } }, // Tighten HTTP timeouts so dead sockets do not accumulate httpNodeAuth: { /* optional */ }, httpAdminRoot: false, // editor off implies admin off; explicit for clarity };

Deploy flows to the file by editing flows_iot2040.json directly with a deploy script from your engineering workstation, or temporarily set disableEditor: false, edit, deploy, then set it back to true and restart Node-RED. The trade-off: every time you change a flow, Node-RED must restart. On an IOT2040 that is acceptable; on a much larger gateway it would not be.

Solution 2: Raising OS Limits

Edit /etc/systemd/system/node-red.service (or the override /etc/systemd/system/node-red.service.d/override.conf):

[Service]
ExecStart=/usr/bin/env node /usr/lib/node_modules/node-red/red.js /home/iot/.node-red/flows_iot2040.json $NODE_RED_OPTIONS
Restart=always
RestartSec=10
User=iot
Environment=NODE_OPTIONS=--max-old-space-size=384
LimitNOFILE=4096
LimitNPROC=2048

Notes:

• --max-old-space-size=384 caps the V8 heap at 384 MB, leaving headroom in the 600 MB usable RAM for the OS, S7 node buffers, and the journal.
• LimitNOFILE=4096 removes the 1024-fd ceiling.
• Restart=always plus RestartSec=10 gives the OS time to free the port 1880 and flush the journal before relaunch.

Reload systemd and restart:

sudo systemctl daemon-reload
sudo systemctl restart node-red
sudo systemctl status node-red

Solution 3: Stabilizing the S7 Connection

The S7 communication node is the most common cause of perceived "Node-RED instability". The following pattern wraps the S7 read/write in a catch + reconnect loop and decouples connection state from message processing:

Recommended S7 Node Selection

Node package	Maintenance state	Stability on IOT2040	Notes
node-red-contrib-s7	Active	Good for PUT/GET on S7-1200/1500 with optimized block access	Requires the PUT/GET checkbox enabled in TIA Portal CPU properties
node-red-contrib-s7comm (s7-comm)	Active community fork	Best in field reports for multi-month uptime	No native C++ addon; pure JavaScript — eliminates RC1 risk
nodes7 (standalone)	Original, slow updates	Workable but has known reconnect bugs on TCP RST	Wrap with external reconnect timer
node-red-contrib-plc	Multi-vendor (Siemens, Beckhoff, Modbus)	Acceptable	Use when you also need Modbus TCP on the same gateway

Field experience: the s7-comm package is widely reported as the most stable S7 node on the IOT2040 in long-running installations, typically operating for 6+ months without restart when configured against an S7-1500 PUT/GET-enabled CPU. The node-red-contrib-s7 node is the second-best option for the same hardware and is generally preferable when the target PLC is an S7-1200 with firmware ≥ 4.x that has the optimized-block access limitations.

S7 Connection Best-Practice Configuration

1. Enable Permit access with PUT/GET in the S7-1200 / S7-1500 CPU properties under Protection & Security → Connection mechanisms. Without this, the S7 node can connect initially and then drop on the first request.
2. Set the connection's rack/slot to the TIA Portal configuration of the CPU. For an S7-1500 this is rack 0, slot 1. For an S7-1200 it is rack 0, slot 1. For a soft PLC (e.g. PLCSIM) it may be rack 0, slot 0 or rack 0, slot 2 — confirm in the device configuration.
3. Set the cycle / poll rate to at least 100 ms. The S7 CPU's communication load budget is finite; a 10 ms poll loop on a heavily loaded PLC will cause the connection to drop with a CPU-side resource error.
4. Use DB reads with explicit length. Do not request DB1 with length 9999 bytes if the actual DB is 200 bytes; the PLC will return a length error that some S7 nodes do not catch.
5. Wrap every S7 node in a flow with a catch node that triggers a node.status({fill:"red",shape:"dot",text:"reconnecting"}) and a 5-second reconnect timer.

Reconnect Pattern (Pseudo-Flow)

[S7 Read]  --success--> [Process payload]   (debug, MQTT publish, etc.)
     |
     +--error--> [Catch all] --> [Delay 5s] --> [Re-trigger S7 Read]
                                |
                                +-- after 3 consecutive errors
                                    --> [node.warn] + [system reboot via exec node]

Solution 4: Auto-Start and Process Supervision

On the IOT2000 Example Image, Node-RED is typically started by a systemd unit named node-red.service. Verify and harden it:

systemctl list-unit-files | grep -i node
systemctl cat node-red.service
systemctl enable node-red.service

Common auto-start failure modes on the IOT2040:

1. Network not yet up when Node-RED starts. If a flow opens an MQTT broker connection on startup and the broker is on a remote host whose IP is brought up via DHCP after the IOT2040's network-online.target, the connection fails and the flow's catch path triggers. Add Wants=network-online.target and After=network-online.target to the unit, and a 30-second ExecStartPre=/bin/sleep 30 if your environment is slow.
2. A node in nodesDir throws on load. If a custom node in ~/.node-red/nodes/ has a syntax error or missing dependency, Node-RED exits with code 1 at startup and the service enters a restart loop. Boot with disableEditor: true still in place but temporarily start Node-RED in the foreground (node-red -v) over SSH to see the actual error.
3. Port 1880 already in use. A previous Node-RED instance that did not shut down cleanly holds the port. The new process exits with EADDRINUSE. sudo lsof -i :1880 and kill the stale PID.

Solution 5: Logging and Diagnostics

All crash root causes must be confirmed from logs before any of the above fixes are deployed. Collect:

1. journalctl -u node-red.service -n 2000 --no-pager — Node-RED's own output.
2. dmesg | tail -200 — kernel OOM kills and segfault backtraces. Look for Out of memory: Killed process or node-red segfault.
3. ls -la ~/.node-red/ — verify flows_*.json timestamp and size; a 0-byte flows file indicates a failed write.
4. df -h /home /tmp — eMMC full or tmpfs full.
5. cat /proc/$(pidof node-red)/limits — confirm the systemd LimitNOFILE actually took effect (you will see Max open files 4096).
6. cat /proc/$(pidof node-red)/status | grep -E 'VmRSS|VmSize' — V8 RSS and virtual size; compare against --max-old-space-size.

Solution 6: When to Move Beyond Node-RED

Node-RED is an excellent glue layer for moderate-complexity flows: a few hundred messages per second, a handful of PLCs, one or two MQTT brokers, a dashboard. It is not a replacement for a SCADA system on a plant line. Use a higher-level language (Python, structured text compiled via a S7-1500, or a dedicated edge runtime such as the Siemens Industrial Edge runtime) when:

• You have hard real-time requirements (sub-100 ms guaranteed latency).
• You need to handle > 1,000 tags at > 10 Hz without packet loss.
• You need deterministic restart / watchdog semantics (Node-RED's restart is best-effort).
• You need to ship a certified product where the runtime must have an audit trail and version pinning.

For the vast majority of IOT2040 deployments — MQTT fan-out from one S7-1500 to a cloud broker, a local Node-RED Dashboard, periodic Modbus polls — Node-RED is fit for purpose, provided the hardening steps in this article are applied.

Verification Checklist

1. Confirm disableEditor: true in settings.js; curl http://iot2040:1880 should return 404 or connection refused.
2. Confirm LimitNOFILE=4096 applied: cat /proc/$(pidof node-red)/limits | grep "open files".
3. Confirm heap cap: V8 reports --max-old-space-size=384 in /proc/$(pidof node-red)/cmdline.
4. Force a PLC stop/start and confirm the S7 node reconnects within 10 seconds (watch the node.status indicator).
5. Unplug the PROFINET cable, plug it back in, confirm reconnect without manual intervention.
6. Run node-red under node --inspect for one day and capture heap snapshots; verify heap returns to baseline after a message burst (no monotonic growth).
7. Reboot the IOT2040; confirm Node-RED comes up within 60 s and all flows resume their PLC polling.
8. Watch journalctl -u node-red -f for 24 h; expect zero uncaught exceptions and zero segfaults.

Troubleshooting Matrix

Symptom	Likely root cause	First diagnostic	Fix
Process gone, no journal entry	Native addon segfault (RC1)	dmesg | grep -i segfault	Pin Node version, rebuild native addons or switch to pure-JS S7 node
Restart every few minutes	OOM killer (RC5)	dmesg | grep -i oom	Cap heap, add watchdog flow, prune custom nodes
Editor slow, deploy takes 30 s+	eMMC full or disableEditor: false in production	df -h, curl :1880	Free disk, set disableEditor: true
Stuck, no response	Event loop blocked (RC3)	node --prof on next start	Remove setInterval tight loops, use inject nodes with rate limit
S7 drop, no reconnect	S7 node state machine (RC4)	PLC diag buffer for comm errors	Enable PUT/GET in TIA, wrap in catch+reconnect, switch to s7-comm
Auto-start fails	Network not up / port in use / broken node	journalctl -u node-red first 30 lines	Add After=network-online.target, kill stale PID, remove broken node
Heap grows over days	Timer / context leak (RC2)	V8 heap snapshot	Use file context store, audit function nodes for unbounded arrays

Edge Cases and Field-Proven Caveats

• The IOT2040's eMMC has limited write endurance. Logging heavy MQTT payloads to a file-based context store at 10 Hz will wear the flash in months. Use an external USB SSD or stream to syslog over the network.
• PROFINET IRT is not supported on the IOT2040's second Ethernet port. Use the port for standard PROFINET RT or for IP traffic only.
• If the IOT2040 is in a NAT environment, the MQTT broker's keepalive must be longer than the NAT idle timeout (typically 30–120 s). A keepalive of 15 s with NAT will cause silent broker-side disconnects that look like Node-RED bugs.
• The bundled node-red-dashboard has no built-in auth. Behind disableEditor: true the dashboard still runs — do not confuse "editor off" with "dashboard off". Add an httpNodeAuth reverse proxy if exposed.
• On a fresh IOT2000 Example Image, the default Node.js is older than current LTS. If you npm install -g n and try to upgrade, you will likely break the bundled native addons. Stay on the bundled version unless you also rebuild all addons against the new ABI.