Siemens IOT2050 Example Image Boot Failure: STAT LED Red

Problem: IOT2050 Example Image Refuses to Boot, STAT LED Red

A freshly compiled Siemens SIMATIC IOT2050 example image writes successfully to the SD card but the device will not bring up a login shell. The user-observable symptoms are:

• U-Boot loads the kernel and device tree, control passes from bootcmd to the Linux kernel.
• Boot progress halts very early — the console shows no systemd banner, no login: prompt, no getty.
• The front-panel STAT LED transitions from solid green to solid red within seconds of the first userspace activity.
• The same SD card flashed with the binary sample distributed under Siemens Support Entry 109780231 boots normally and exposes all network interfaces.

This pattern indicates the bootloader and storage are healthy — the failure is inside the userspace image itself (kernel/userspace ABI mismatch, missing firmware blobs, or incompatible extlinux.conf entry).

Affected Hardware and Firmware

Root Cause: Kernel/Firmware ABI Break

The Siemens meta-iot2050 Yocto layer tracks upstream linux-yocto and the AM65xx TI BSP. The vendor replaced the legacy 4.19 kernel branch in the layer with a newer mainline kernel branch (5.10+). The on-board firmware (CPSW Ethernet firmware, PRU firmware, System Controller binary) that is pre-flashed at the factory is only ABI-compatible with the 4.19 kernel drivers compiled against the matching firmware interface header. When a newer kernel is loaded:

1. The CPSW netcp driver probes and fails to attach because the firmware interface version reported by the System Controller does not match the in-tree driver's expectation.
2. If a custom systemd unit or udev rule requires the network stack to be online (e.g. Wants=network-online.target) before the watchdog daemon starts, the watchdog pet never happens and the BMC-equivalent watchdog on the IOT2050 asserts a hardware reset, which the firmware re-enters as a STAT = red state on the next U-Boot pass.
3. Even when the watchdog path is removed, newer kernels drop support for the on-board EEPROM layout that the vendor's MAC-address/persistent-storage reader expects, so the kernel may panic at boot on of_get_mac_address with an -EINVAL if the device tree is incompatible.

Siemens' official position (per the GitHub release notes on the V01.01.01 tag) is that the deployed firmware image on existing field devices requires the 4.19 kernel until a paired firmware + kernel update is released. The 4.19 branch was therefore the last supported configuration, and newer branches are pinned to require a firmware update that was scheduled for release "around October" (post-2021/09/10 cut-off for the source incident).

Solution: Pin to meta-iot2050 V01.01.01 (4.19 + Buster)

The supported, documented workaround is to clone or update the meta-iot2050 layer at the V01.01.01 release tag and rebuild the example image against it. This combination was the last release built on kernel 4.19 with Debian 10 (Buster) userland and is the safest option for existing devices that cannot take the new firmware yet.

Step 1 — Pin the layer

In your Yocto build directory's conf/sanity.conf or your top-level init script, change the SRCREV of the layer to the V01.01.01 commit hash:

# In your .conf or bblayers.conf-equivalent
SRCREV_meta-iot2050 = "<commit-hash-of-V01.01.01-tag>"
SRC_URI = "git://github.com/siemens/meta-iot2050.git;branch=master;protocol=https"
# Override to a specific tag by using a srcrev file or BB_GIT_LOCAL_MIRROR

The simpler approach is to use a git checkout in a local mirror before bitbake resolves the layer:

cd /path/to/sources/meta-iot2050
git fetch --tags
git checkout V01.01.01
git log -1 --oneline   # confirm you are on the tagged commit

Step 2 — Verify kernel and distro pins

Confirm the build is producing the expected kernel and userland. In the V01.01.01 release the relevant variables are:

Inspect with bitbake -e virtual/kernel | grep ^LINUX_VERSION= after sourcing the build environment.

Step 3 — Build the example image

source oe-init-build-env
bitbake iot2050-example-image   # or whatever the example IMAGE_INSTALL target is named in V01.01.01

The output .wic or rootfs tarball will land in tmp/deploy/images/iot2050/.

Step 4 — Flash the SD card

Use dd with a verified block size and confirm the write completed:

sudo umount /dev/sdX*
sudo dd if=tmp/deploy/images/iot2050/iot2050-example-image-iot2050.wic \
        of=/dev/sdX bs=4M conv=fsync status=progress
sync

Remove the card, insert it into the IOT2050, and power-cycle.

Step 5 — Connect to the serial console

Use a 3.3 V TTL USB serial adapter on the IOT2050 debug header (115200 8N1):

minicom -D /dev/ttyUSB0 -b 115200
# or
screen /dev/ttyUSB0 115200

You should see the U-Boot banner, the kernel command line (typically console=ttyS2,115200n8 root=/dev/mmcblk0p2 rw rootfstype=ext4 rootwait), and the login: prompt within 25-40 seconds of power-on.

Verification Checklist

1. STAT LED behavior. After power-on, LED goes amber briefly during U-Boot, then green when systemd reaches the multi-user target. It must not turn red.
2. Console banner. Kernel prints Linux version 4.19.x and finishes the Freeing unused kernel memory line within 15 s.
3. systemd default target reached. systemctl is-system-running returns running; multi-user.target is active.
4. Network interfaces up. ip link shows eth0 and any bridge interfaces with LOWER_UP. On V01.01.01 the interface naming has been corrected (it matches the physical labels); older user builds sometimes named them in a way that clashed with udev rules.
5. Watchdog daemon. systemctl status systemd watchdog (or the equivalent wd_keepalive unit in the example image) is active. If it is missing or failing, the watchdog will still fire on idle and you will see a red STAT LED after the configured timeout (default ~60 s).
6. Persistent state. Reboot twice. Some kernel/firmware mismatch symptoms only appear on the second boot when the RTC wakeup path is exercised.

STAT LED Diagnostic Matrix

Bootloader and Image Layout Reference

The IOT2050 SD card uses a two-stage layout managed by U-Boot. Understanding this layout is essential when a STAT-red boot leaves no useful log on the serial console.

If the SD card is unreadable at all, U-Boot's boot script (commonly finduuid then load mmc 0:1 ...) will fail and the device will drop to the U-Boot prompt. If the boot script succeeds but the kernel cannot mount /dev/mmcblk0p2 (e.g. ext4 module missing or root= argument wrong), the kernel panics with VFS: Cannot open root device "mmcblk0p2" — this also produces a red STAT LED via the watchdog, but the console will show the panic.

Workarounds Until the New Firmware Is Released

1. Use V01.01.01 image only. Pin your build to this tag. Do not pull HEAD. This is the supported path until the matching firmware update ships.
2. Use the Siemens-bundled binary sample. For training and demonstration purposes, the prebuilt sample linked from Siemens Support 109780231 is built against the same V01.01.01 base and will boot on factory firmware without modification.
3. Disable the hardware watchdog temporarily for debugging. In U-Boot, append systemd.watchdog_device= to the kernel cmdline, or mask the watchdog service. This will keep the device alive long enough to capture a real panic log over the serial console, but it must not be left in production.
4. Capture a full boot log from serial. Use a hardware serial logger (e.g. an OpenSDC, an FTDI friend with logging software, or a Linux host with screen -L -Logfile boot.log) to record the entire boot. The kernel panic message will identify the exact failing subsystem (CPSW, PRU, EEPROM reader, etc.).

Post-Recovery: Planning the Firmware Update

When Siemens publishes the new IOT2050 firmware capsule (originally indicated for "around October" from the 2021/09 incident), the migration path is:

1. Reflash the System Controller and CPSW firmware using the Siemens-provided capsule and the IOT2050 firmware update procedure (typically a signed U-Boot update from the FAT partition plus a power-cycle).
2. Verify the new firmware version is reported on the serial console during U-Boot (look for SCFW Build Version and CPSW version strings).
3. Rebuild the Yocto image against the newer meta-iot2050 release that matches the new firmware (kernel 5.10 or later).
4. Validate the same boot checks (LED, systemd, network) on the new image.

Do not attempt to load a newer kernel image onto factory firmware. The kernel/firmware ABI mismatch is the exact failure mode that produces the red STAT LED in the original report.

Edge Cases and Field-Proven Caveats

• SD card quality. Industrial SD cards (e.g. Swissbit, ATP, Apacer) are strongly recommended. Consumer cards can produce intermittent read errors that masquerade as image corruption, with the same STAT-red symptom.
• Power supply ramp. The IOT2050 is sensitive to slow-rising 24 V. If the STAT LED never lights at all, measure the 5 V and 3.3 V rails on the base board test points before assuming an image problem.
• U-Boot environment. If uEnv.txt is corrupted (e.g. unterminated quote, missing newline at EOF), U-Boot silently falls back to default environment and may load the wrong device tree branch. Always edit uEnv.txt on a Linux host and verify with hexdump -C uEnv.txt | tail.
• Second boot only. If the first boot succeeds after a reflash and the second boot fails, the watchdog reset path is masking an in-service crash. Check journalctl -b -1 (previous boot) for the real panic message.
• MAC address stability. On IOT2050 the MAC addresses live in the on-board EEPROM. If you see eth0: random ethernet address on every boot, the EEPROM reader failed and the kernel assigned a random MAC — that is a strong indicator the wrong kernel branch is loaded.

Related Reading and Standards

• meta-iot2050 V01.01.01 release notes — Siemens GitHub
• Siemens Support Entry 109780231 — bundled IOT2050 example image binary
• U-Boot 2021.10 source tree — reference bootloader for embedded boards

FAQ