Resolving Modbus Read Failures Over Time on Siemens IoT2000

Field deployments of the Siemens IoT2000 (SIMATIC IOT2050, IOT2040) reading Delta, Mitsubishi, or generic slave PLCs over RS232 and Modbus RTU frequently exhibit a class of fault where reads work cleanly for 18 to 36 hours and then silently drop one register at a time. A four-register poll reads four, then three, then two, then one, until the link appears to be hung. Polling rate, cable distance, and node timing look correct at install, so the regression is almost always attributed to a software bug when it is actually a stack of physical, task-cycle, and timeout issues that compound under continuous duty.

This reference walks through the protocol fundamentals, the five root causes most likely to surface after a 24-hour burn-in, the diagnostic workflow, and the corrected Node-RED / CODESYS configuration that keeps an IoT2000 master polling four D-registers indefinitely.

1. Problem Statement and Observable Symptoms

The original failure is reproducible:

• Hardware: SIMATIC IoT2000 (IOT2050 or IOT2040) running Node-RED with node-red-contrib-modbus v5.x, or CODESYS 3.5 SP16 with the CAA Modbus Master library.
• Link: Cross DB9 cable, 4 ft (1.2 m), direct to PLC COM port. Pins 7 (RTS) and 8 (CTS) jumpered for self-shake. No 120 Ω termination (RS232 does not require it, but the wiring note matters for diagnostic clarity).
• Slave: PLC exposing holding registers at D1030, D1040, D1044, and D1054. Modbus address mapping depends on the vendor but is functionally 4x1029, 4x1039, 4x1043, and 4x1053 in the standard 1-based holding-register addressing.
• Schedule: Four Modbus Read nodes, scheduled 1 s apart, each requesting one register (FC 03, Quantity = 1). Outer poll loop runs every 60 s.
• Symptom: Reads are stable for 18 to 36 hours, then the first Modbus Read starts returning ILLEGAL_FUNCTION or simply times out. After another few hours a second node drops. The other nodes continue to respond normally, suggesting a per-instance state problem rather than a link-level fault.

The stepped degradation (4 → 3 → 2 → 1) is a signature of a per-node resource problem, not a shared physical-layer failure. If the entire link died at once, the cause would be a cable, termination, ESD, or ground problem. Per-instance degradation points at the Modbus client stack on the IoT2000.

2. Modbus RTU Protocol Constraints That Drive the Failure

Modbus RTU is sensitive to two inter-frame timing rules defined in the Modbus over Serial Line specification V1.02:

The slave treats any gap larger than t1.5 between received bytes as a frame error and discards the PDU. The master treats any gap larger than t3.5 between two ADUs as the end of a frame. The Modbus Application Protocol V1.1b3 explicitly states the inter-frame silence must be observed even when requests are queued back-to-back. On Linux, the UART driver and the Modbus library must cooperate to insert these gaps; otherwise the slave can collapse two ADUs into one frame and discard both, producing silent timeouts that look identical to the original symptom.

For FC 03 with a single register, the request ADU is 8 bytes (1 + 1 + 2 + 2 + 2) and the response ADU is 7 bytes (1 + 1 + 2 + 1 + 2). At 9600 8N1 the round-trip on the wire is approximately:

• Request: 8 × 10 / 9600 = 8.33 ms
• Response: 7 × 10 / 9600 = 7.29 ms
• t3.5 silence after response: 4.48 ms
• Slave turnaround: 1-10 ms (PLC dependent)

Total cycle for a single read is roughly 25-35 ms at 9600 bps. Four back-to-back reads spaced 1 s apart therefore occupy 100-140 ms of radio time inside a 4 s window — well within budget — but the budget collapses if the user-space Modbus client retries internally or holds the file descriptor open across multiple requests.

3. Root Cause Analysis: Five Compounding Mechanisms

3.1 Serial Port Handle Exhaustion (Highest Probability)

Node-RED's node-red-contrib-modbus keeps a per-node ModbusRTU client. Each client opens the underlying TTY, sets termios, and creates a TCP socket to the internal modbus-serial server on 127.0.0.1. If the closeOnTimeout parameter is left at default, a client that hits an inter-frame error closes the socket but leaves the termios-locked /dev/ttyS2 handle in the FLUSHING state. After ~1500-2000 cycles the IoT2000's process table fills with half-closed modbus-serial worker processes that still hold the UART. Subsequent open() calls return EBUSY, which the node surfaces as a silent {} payload — exactly the symptom observed.

Reference: node-red-contrib-modbus official README documents the closeOnTimeout, delayOnAck, and retryOnTimeout knobs. The default closeOnTimeout: true combined with retryOnTimeout: true produces the leak.

3.2 Task Cycle Time Smaller Than t3.5

If the four Modbus Read nodes run inside a Node-RED flow whose exec scheduler calls them within the same 10-50 ms tick, the second request hits the slave before the first t3.5 gap has elapsed. The slave discards both. The client marks the request as RECONNECTING and queues a backoff. Over thousands of cycles the queue grows, latency compounds, and a node that looked healthy on day one drops off at hour 22.

Verify by inspecting modbus-serial logs at DEBUG level. Look for repeating Timed out followed by State: INIT → RECONNECTING.

3.3 Watchdog Reset on the IoT2000

The IoT2000 firmware exposes a hardware watchdog at /dev/watchdog that pings every 60 s. If the user-space Modbus client holds the UART for more than that interval (which can happen on cold-start when the PLC is in STOP mode and the client retries 5 times × 5 s timeout), the watchdog fires and reboots the SIMATIC. After reboot the four Modbus nodes re-init serially, and the race for the TTY handle begins again — masking as a fault that "recovers" when you restart the flow.

Reference: SIMATIC IOT2050 operating instructions, section on watchdog configuration, available on the Siemens Industry Online Support product page for 6ES7647-0BA00-0YA2 / 6ES7647-0BA00-1YA2.

3.4 RS232 Signal Levels and Cable-Induced Errors

RS232 is single-ended, full-duplex, and tolerates cable lengths up to 15 m at 9600 bps, but the cable capacitance budget is consumed by every meter of conductor. A 4 ft cross cable is fine. What is not fine is:

• Pins 7 (RTS) and 8 (CTS) jumpered: This forces permanent request-to-send, defeating hardware flow control. Some Modbus slaves use RTS to gate transmission; if RTS is held asserted, the slave may transmit continuously and corrupt the master's receiver.
• No ground reference between chassis: If the IoT2000 and the PLC are powered from different branch circuits, the GND pin (DB9 pin 5) may float ±5 V relative to the slave's GND. The receiver interprets any differential less than ±3 V as indeterminate. Field experience shows this becomes intermittent at temperature cycles inside enclosures.
• DB9 shield not bonded: Without a bonded shield, the cable picks up VFD-induced common-mode noise that biases the receiver threshold.

3.5 Slave PLC Dropping the Link

The slave may itself be a victim of cumulative state. Delta DVP, Mitsubishi FX5U, and Keyence KV-8000 all implement a Modbus RTU slave with a finite transaction counter and a watchdog that resets the UART after N idle seconds. If the IoT2000's poll rate is 60 s and the slave's internal idle reset is also 60 s, the timer fires before the next request, the slave closes its UART, and the next request is lost. The slave will recover on the next poll, but the master's expectation of four successful reads per cycle is permanently broken until both timers are realigned.

4. Diagnostic Procedure

1. Capture the UART traffic. Connect a USB-to-serial tap (FTDI FT2232H or compatible) in parallel with the existing DB9. Run tio -b 9600 -d 8 -s 1 -p none -f /dev/ttyUSB0 | tee /tmp/capture.log. Look for dropped bytes, FCS errors, and inter-frame gaps shorter than 4 ms at 9600 bps.
2. Log the Modbus client state. Add a debug node to each Modbus Read output and capture msg.payload, msg.error, and the inner ModbusResponse object. Run for 48 hours. The first error's timestamp is the regression point.
3. Inspect the process table. Periodically run ps aux | grep -E 'modbus|node' into a cron-log file. Compare the number of node processes at hour 1 and hour 30. If the count grew, you have a handle leak.
4. Disable the IoT2000 hardware watchdog during testing: systemctl stop watchdog or remove the WATCHDOG=1 environment variable if running a custom systemd unit. If the regression disappears, the watchdog is implicated.
5. Swap the IoT2000 with a Windows machine running Modbus Poll or QModMaster over the same cable. If the symptom disappears on Windows but persists on Linux, the bug is in the user-space Modbus stack, not the physical layer.
6. Use an oscilloscope on TXD, RXD, and GND. Look for signal rise times > 1 ms (indicating capacitive loading), ringing > ±5 V (indicating missing termination on the line driver), or asymmetric mark/space voltages (indicating ground offset).

5. Hardware Verification

Verify the DB9 wiring against the standard EIA-574 pinout (the IoT2000 COM ports implement DTE behavior):

Verify RS232 levels with a multimeter in DC mode between DB9 pin 5 (GND) and pins 2 and 3. The idle state of TXD should be between −3 V and −15 V. The idle state of RXD should also be in that range. Voltages between −3 V and +3 V on either line indicate a ground-offset problem; replace the cable or bond the chassis grounds with a separate 12 AWG conductor.

For runs longer than 2 m or noisy environments (VFDs, soft starters, contactors), migrate to RS485. The IoT2000 COM2 supports RS485 half-duplex on the same DB9 connector when configured via the device tree overlay; reference the SIMATIC IOT2050 manual, section "Serial interface configuration". RS485 requires a 120 Ω termination resistor between the A and B conductors at both ends of the bus; do not place termination on stubs.

6. Software Fix: Node-RED Configuration

The following settings resolve the regression in node-red-contrib-modbus 5.x:

Modbus Read node configuration (per node):
  Name:           D1030
  Topic:          plc-d1030
  Unit-Id:        1
  FC:             3 (FC 3 - Read Holding Registers)
  Address:        1029   // 0-based; equivalent to D1030 in 1-based PLC convention
  Quantity:       1
  Poll Rate:      1000   // ms, used only when no trigger input is wired
  Timeout (ms):   1500
  Delay on Ack:   50     // ms, ensures t3.5 gap is honored between ADUs
  Retry on Timeout: false
  Reconnect on Timeout: true
  Reconnect Timeout:   5000  // ms

Key changes:

• Retry on Timeout: false prevents the inner 3-retry loop from holding the socket open while it cycles.
• Reconnect on Timeout: true with a 5 s backoff gives the slave time to reset its UART without piling up handles.
• Delay on Ack: 50 ms injects the inter-frame silence that the Linux UART driver does not naturally produce when back-to-back write() calls land in the same termios buffer.
• The outer trigger is an inject node firing every 60 s on topic "poll-cycle", connected to all four Modbus Read nodes through a delay chain: 0 s, 1 s, 2 s, 3 s. This guarantees sequential polling and eliminates the same-tick collision risk.

Add a watchdog "kicker" flow that emits a reset message on msg.topic = "modbus-master-reset" if any of the four Modbus Read nodes has not produced a successful payload in 5 minutes. Wire that reset into the Modbus Connector node's close input. This purges accumulated state every 5 minutes of abnormal behavior, which is much less aggressive than waiting 20 hours for a full regression.

7. Software Fix: CODESYS 3.5 SP16+ Modbus Master

If the deployment runs CODESYS Control for IOT2000 instead of Node-RED, the equivalent fix lives in the PLC task configuration and the CAA Modbus library parameters:

// CODESYS task configuration (Task Configuration -> Task1)
Task1:
  Priority:        15
  Cycle time:      20 ms   // t3.5 at 9600 = 4.48 ms; 20 ms guarantees inter-frame
  Watchdog:        100 ms
  Event-driven:    FALSE

// CAA_ModbusMaster parameter block
iSlaveAddr       := 1;
iFuncCode        := MODBUS_FC_READ_HOLDING_REGISTERS; // 16#03
iDataAddr        := 1029;
iDataCount       := 1;
iTimeout         := 1500;     // ms
xBlockUntilDone  := FALSE;
xExecute         := TRUE;

// Re-arm after each successful read:
IF xBusy = FALSE AND xError = FALSE THEN
  xExecute := FALSE;       // mandatory rising-edge reset
  tDelay(IN := TRUE, PT := #50ms);  // 50 ms inter-frame gap
  IF tDelay.Q THEN
    tDelay(IN := FALSE);
    iDataAddr := iDataAddr; // cycle to next register in external sequencer
    xExecute := TRUE;
  END_IF;
END_IF;

The CODESYS task cycle of 20 ms is deliberately larger than the 4.48 ms t3.5 at 9600 bps. If a faster task is needed (sub-10 ms), switch to RS485 with hardware-managed direction control, or insert a TON delay between back-to-back requests.

Reference: CODESYS Online Help, "CAA Modbus Master" library documentation, available from CODESYS Store under the CAA bundle. The library's ModbusChannel FB exposes iTimeout and iPollCycle; both must be aligned with the slave's idle-reset timer.

8. RS485 Migration Path (Recommended for Permanent Installations)

RS232 over a 4 ft cross cable is acceptable for bench commissioning but is not appropriate for permanent industrial deployment. The IoT2000 COM2 can be reconfigured for RS485 half-duplex via the device tree overlay documented in the SIMATIC IOT2050 manual. RS485 differential signaling provides:

• ±7 V common-mode rejection (vs ±2 V for RS232 single-ended).
• Cable runs up to 1200 m at 9600 bps.
• Multi-drop support for up to 32 slaves on a single twisted pair.
• Survivability in VFD, motor-start, and welding environments.

Reference the TIA-485-A standard from TIA for the electrical characteristics and the Modbus over Serial Line V1.02 for the cable, termination, and stub-length requirements. Place a 120 Ω ±1 % resistor at each physical end of the bus. Do not place termination on stubs or Y branches. Use a shielded twisted pair with the shield bonded to chassis ground at one end only (typically the master).

9. Verification Procedure

After applying the fixes, run a 72-hour burn-in with the following observability stack:

1. Capture every Modbus Read payload into a CSV with columns: timestamp, register address, raw value, response time, error code, node state. Use the Node-RED csv node or a function node appending to a rotating file.
2. Sample /proc/$(pidof node-red)/fd | wc -l every 60 minutes. The count must be flat over 72 hours (typically 30-45 file descriptors steady-state).
3. Run strace -p $(pidof node-red) -e read,write,openat -o /tmp/trace.log for 5 minutes every 6 hours. Verify /dev/ttyS2 open count does not grow.
4. Plot response_time over time. A healthy link shows P99 ≤ 50 ms with a flat tail. A regressing link shows a long right tail after hour 18-30 — that's the precursor to the four-to-three dropout.
5. Verify the IoT2000 uptime with uptime. The watchdog should not have fired; the load average should remain under 0.5 on the dual-core ARM.

Acceptance criteria: zero dropped reads over 72 hours, P99 response time under 50 ms, zero RECONNECTING events in the modbus-serial debug log, and process file-descriptor count constant.

10. Fault Code Mapping

Codes 0x06 and 0x0B are the most common co-conspirators in the 20-30 hour regression because they trigger the master's reconnect logic, which compounds handle exhaustion under load.

11. Troubleshooting Matrix

12. Field-Proven Checklist

• Confirm ttyS2 mapping: stty -F /dev/ttyS2 9600 cs8 -cstopb -parenb -crtscts raw before starting Node-RED.
• Set delayOnAck or equivalent inter-frame guard to at least 50 ms.
• Disable inner retry; rely on outer reconnect timer.
• Schedule the four reads sequentially with a 1 s gap, not concurrently.
• Bind reconnectTimeout to 5 s to give the slave UART time to reset.
• Bond chassis ground if devices are on different branch circuits.
• Remove pins 7-8 jumper; allow CTS to float on the master.
• For new installations, specify RS485 with proper termination.
• Plan a 72-hour burn-in with CSV logging before commissioning.
• Monitor file-descriptor count and IoT2000 uptime during the burn-in.