Resolving Siemens IOT2050 Reboot Failure After sudo reboot

Resolving Siemens IOT2050 Reboot Failure After sudo reboot Command

The Siemens SIMATIC IOT2050 is an ARM-based Industrial Edge Device (IED) deployed as a gateway, protocol converter, OPC UA broker, MQTT publisher, or local data collector in brownfield automation cells. Across multiple industrial sites, engineers have documented a reproducible failure pattern: after issuing a software-initiated reboot command — sudo reboot from SSH, an Industrial Edge pipeline restart, a REST-triggered shutdown, or any equivalent systemd path — the device goes offline and remains unreachable on the network. Recovery requires a full power cycle of the 24 V DC supply. Siemens has confirmed this class of failure in the open-source layer meta-iot2050 and shipped a fix in firmware V1.4.3. This reference consolidates the symptom profile, root-cause hypothesis, the official remediation, UART-based diagnostic capture, the hardware watchdog fallback, and the documented hard-reset procedure into a single engineer field-notes document.

1. Problem Overview

Field engineers first noticed the failure because the device stops responding to ping, SSH, HTTPS, and the Industrial Edge management plane within seconds of the reboot command. The on-board LEDs enter a state inconsistent with a successful Linux shutdown — typically a static pattern with no activity, no heartbeat on the user LED, and no DHCP request on the uplink port. After the failure, the device does not auto-recover: the only field-proven recovery path is to remove and re-apply the 24 V DC supply (or PoE on supported SKUs), after which the IOT2050 boots cleanly.

One logged case shows the failure occurring on the 16th of a month, with the device remaining inaccessible through the 17th, and only recovering after a maintenance-window power cycle on the 18th. Siemens Support has acknowledged this as a known issue affecting multiple sites running long-uptime deployments and noted that it is “very difficult to reproduce on our side” without a debug firmware image (V1.4) supplied to the customer for in-field log capture.

2. Affected Hardware, Firmware, and Software Stack

3. Symptoms and Failure Signature

The failure mode has a recognizable signature. Capture the following observations before applying the workaround — they confirm the issue class and help Siemens Support correlate with internal regression data:

1. SSH session disconnects within ~1 second of sudo reboot execution.
2. The last journalctl -k -b -1 lines typically show systemd entering shutdown, services stopping, then no further log lines after the kernel print of systemd-shutdown[1]: Rebooting.
3. The management LED (USER / STATUS) freezes — no heartbeat pattern, no shutdown-blink, no kernel-panic blink code.
4. Network interfaces drop; no gratuitous ARP is sent on the uplink.
5. Console output via the UART debug interface (115200 8N1) stops mid-shutdown, with no Restarting system from U-Boot.
6. On a single-site sample, uptime at the time of failure was measured at ~60 days; shorter uptimes have also triggered, but the correlation dominates.

4. Root Cause Analysis

The failure does not reproduce reliably on Siemens engineering benches, which initially made triage difficult. From the available evidence (uptime-correlated trigger, kernel-shutdown log truncation, clean recovery via power-on reset, and the V1.4.3 patch contents), the failure class is consistent with a Linux power-management regression interacting with long-uptime clock/timer state and the IOT2050 PMIC sequencing path. Two contributing factors have surfaced in the open-source tracker:

1. Stale timer-wheel / clock-event state after extended uptime. Issue #440 in the meta-iot2050 repo documents that the failure appears after extended continuous operation. The hypothesis is that cumulative drift in the timer subsystem prevents the kernel from completing the final CPU off-lining plus system-controller reset handshake, leaving the SoC in a half-powered state from which only a full power-on reset can recover.
2. PMIC / system-controller handshake on the reboot path. The reboot path requires the system controller to acknowledge a warm-reset request. Under the conditions above, the handshake does not complete, and the SoC hangs in a low-power state rather than asserting reset to the PMIC.

Siemens has not published a single root-cause statement naming the exact commit that fixes the issue, but the V1.4.3 release notes on the Industry Online Support downloads page confirm the patch addresses the reboot hang and is the supported remediation.

5. Immediate Workarounds

Until V1.4.3 can be deployed, the following workarounds reduce the probability of a hang in production. Apply all three in combination for highest protection.

5.1 Scheduled hard reboot instead of soft reboot

Replace sudo reboot with a power cycle orchestrated through the upstream PLC, IPC, or managed PDU:

#!/bin/bash
# Trigger PDU outlet cycle rather than OS-level reboot
curl -X POST https://pdu.example.local/api/outlet/3/cycle \
     -H "X-Auth-Token: $PDU_TOKEN"

5.2 Force a sync and short grace before reboot

Issue the kernel a sync and allow the filesystem journals to flush before issuing the reboot:

sync && sleep 5 && systemctl reboot --no-wall

5.3 Enable the hardware watchdog

See Section 8 — the on-board hardware watchdog will reset the SoC within ~60 seconds if the kernel hangs during shutdown, turning a permanent hang into a one-minute self-recovery.

6. Permanent Fix: Firmware V1.4.3

Siemens released firmware V1.4.3 as the official remediation. The release is published on the Siemens Industry Online Support downloads page under the IOT2050 product tree. The same base image is delivered as both a standalone “Example Image” (vanilla Linux for direct use) and an Industrial Edge firmware variant (consumed through IEManager / Industrial Edge Hub).

6.1 Identify current firmware

Before flashing, capture the running version to confirm the upgrade is actually applied:

cat /etc/os-release
cat /etc/iot2050-release 2>/dev/null
dpkg -l | grep -i iot2050

6.2 Acquire the firmware

1. Navigate to the Siemens Industry Online Support downloads page.
2. Search for IOT2050 in the product tree.
3. Locate the entry titled “SIMATIC IOT2050 Firmware Update V1.4.3” (or the matching Industrial Edge firmware V1.4.3 entry).
4. Download the signed .wic or .mender bundle and verify the SHA-256 against the manifest published on the same page.

6.3 Apply the update — two paths

Path A: In-place Example Image upgrade via mender

sudo mender install /path/to/iot2050-image-V1.4.3.mender
sudo mender commit
sudo reboot

Path B: External SD-card / USB flash for full re-image

1. Write the .wic image to an SD card or USB stick using bmaptool or balenaEtcher.
2. Insert the medium into the IOT2050.
3. Hold the recovery / boot-select button on power-up, or use the documented boot-menu key sequence to boot from external media.
4. Re-flash the internal eMMC following the on-screen installer.
5. Remove the external medium and reboot.

6.4 Verify the new image is active

cat /etc/iot2050-release
uname -a
mender show-artifact

Confirm the version string contains V1.4.3 (or later) and that mender show-artifact reports the new artifact as the booted (not pending) partition.

7. UART Debug Interface Diagnostics

The full reboot transition is only observable on the UART debug header — journald cannot persist logs that never completed the post-boot flush cycle. The header location depends on the IOT2050 variant:

7.1 Capture procedure

1. Connect a 3.3 V USB-to-UART cable (FTDI FT232, CP2102, or CH340G). Do not use a 5 V cable — the IOT2050 debug header is 3.3 V only.
2. Connect GND → GND, RX → TX (host RX to IOT2050 TX), TX → RX (host TX to IOT2050 RX).
3. Open a serial terminal: minicom -D /dev/ttyUSB0 -b 115200 or screen /dev/ttyUSB0 115200.
4. Begin a log capture to file: minicom -C iot2050_reboot.log or use tee with picocom.
5. Trigger the reboot from SSH and let the log run for at least 90 seconds — past the watchdog window — so a watchdog-induced recovery is also captured if it occurs.

7.2 What to look for

In a healthy reboot the UART shows a clean transition:

[  OK  ] Reached target Shutdown.
systemd-shutdown[1]: Syncing filesystems and block devices
systemd-shutdown[1]: Sending SIGTERM to remaining processes...
systemd-shutdown[1]: Sending SIGKILL to remaining processes...
systemd-shutdown[1]: Unmounting /sys/fs/cgroup...
systemd-shutdown[1]: All filesystems unmounted.
systemd-shutdown[1]: Deactivating swaps.
systemd-shutdown[1]: All swaps deactivated.
systemd-shutdown[1]: Rebooting.
[    0.000000] Booting Linux on physical CPU 0x0
U-Boot SPL 2021.10 ...

In a hung reboot the trace stops somewhere in the kernel power-off / system-controller handshake, typically:

systemd-shutdown[1]: Rebooting.
[   42.117xxx] reboot: Restarting system
<hang here — no further lines>

Send the captured log to Siemens Support via the Industry Online Support ticket you opened for the issue.

8. Hardware Watchdog Configuration

The IOT2050 exposes an on-board hardware watchdog tied to the system controller. If the kernel fails to service the watchdog within the configured timeout, the SoC is hard-reset by the watchdog logic — independent of the kernel reboot path. Configuring the watchdog turns a long-uptime-correlated hang into a ≤ 60-second self-recovery, which is acceptable for most IED use cases (the device is typically polled or buffered upstream).

8.1 Confirm the watchdog device

ls /dev/watchdog*
dmesg | grep -i watchdog

You should see at least one /dev/watchdogN node and a watchdog driver binding message in dmesg.

8.2 Configure watchdogd via systemd

Edit /etc/systemd/system.conf and set:

[Manager]
RuntimeWatchdogSec=30
RebootWatchdogSec=10

• RuntimeWatchdogSec=30 — keepalive ping every 30 seconds; if the kernel hangs, the SoC resets in ≤ 30 s.
• RebootWatchdogSec=10 — applies during the shutdown phase, the exact window where the documented hang occurs.

Reload systemd:

sudo systemctl daemon-reexec

8.3 Disable the watchdog on demand

For debugging or when running kernel bisects, stop the watchdog without rebooting:

# Temporary local disable
echo 0 > /proc/sys/kernel/watchdog

# Permanent local disable
sudo sed -i 's/^RuntimeWatchdogSec=.*/RuntimeWatchdogSec=0/' /etc/systemd/system.conf
sudo systemctl daemon-reexec

9. Hard Reset Procedure

Use the hard-reset procedure when the device is unresponsive after a soft-reboot hang. Reference: Hard reset — IOT2050 IED One DB documentation.

1. Verify the device is actually hung (no ping, no SSH, no heartbeat LED).
2. Disconnect the 24 V DC supply (or the PoE supply on supported SKUs). Wait at least 10 seconds to allow the PMIC bulk capacitors to discharge below the brown-out threshold.
3. Re-apply the 24 V DC supply.
4. Watch the USER LED pattern:
   • Steady green within ~30 seconds — healthy boot.
   • Rapid red blink — boot failure; capture UART logs before power-cycling again.
   • No LED activity at all — check the supply voltage and polarity before assuming a deeper failure.
5. Once SSH is reachable, capture journalctl -b -1 and the last 5 minutes of /var/log/iot2050/* for the support ticket.

10. Verification Steps

After applying firmware V1.4.3 and the watchdog configuration, run the following verification matrix. Each row is a discrete acceptance test; mark Pass/Fail before declaring the remediation complete.

11. Field Engineering Best Practices

These are field-proven practices for any IOT2050 fleet, regardless of whether the V1.4.3 patch is in place yet.

1. Treat the IOT2050 like an appliance, not a server. Prefer managed power-cycling (PDU, PLC-controlled contactor) over sudo reboot for routine maintenance windows. The kernel soft-reboot path should be reserved for emergency recovery, not scheduled operations.
2. Enable the hardware watchdog on every install. The cost is one config line; the benefit is automatic recovery from any future kernel power-management regression.
3. Keep firmware on a regular cadence. Subscribe to Siemens Industry Online Support notifications for the IOT2050 product tree. The V1.4.3 release was a maintenance update; future updates may address adjacent issues in the same class.
4. Capture UART logs during the first 90 days. For new deployments, leave a serial logger (USB-to-UART → Raspberry Pi → syslog) attached during the burn-in window. The IOT2050 reboots that fail in this window provide the data Siemens Support needs to triage future regressions.
5. Avoid mixing IOT2050 firmware variants. Standardize the fleet on a single firmware baseline (Example Image V1.4.3 or Industrial Edge firmware V1.4.3) — mixing causes inconsistent reboot behavior because the systemd units differ between the two SKUs.
6. Time-bound deployments. If you cannot apply V1.4.3 immediately, schedule a preventive reboot at least every 30 days to stay below the long-uptime correlation threshold documented in issue #440. This is a workaround, not a fix.
7. Document the recovery path. Every site that owns an IOT2050 should have a printed one-page runbook covering: identify a hung device, perform a hard-reset, capture UART logs, open an Industry Online Support ticket.
8. Preserve UART evidence. Each reboot event should leave a timestamped log on an out-of-band serial collector. When a regression reappears in a future firmware, that log is the single most useful artifact for Siemens Support.

12. Decision Flow — Field Triage

Use the following decision flow to triage any IOT2050 reboot failure in the field.

FAQ