Resolving Siemens IoT2040 Boot Failure with Swissbit SD Cards

1. Problem Overview

The Siemens SIMATIC IoT2040 is an x86-based industrial IoT gateway designed for edge data acquisition, protocol conversion, and cloud connectivity. The device uses an Intel Quark x1000 SoC (a 32-bit, single-core, 400 MHz x86 CPU) with a Yocto-based Linux image that boots from a removable microSD card inserted in the on-board reader.

On certain industrial-grade SD cards, particularly the Swissbit SFSD / S-400u / S-450u series, the IoT2040 fails to complete the boot process. The UEFI firmware loads the bootloader, the systemd-boot menu appears, and the kernel starts to uncompress and initialize hardware. The kernel then attempts to mount the root filesystem (rootfs) from the SD card's second partition. At this point the boot halts: the kernel cannot find the root device, falls through to an initramfs shell or hangs on "Waiting for root device /dev/mmcblk0p2..." while the underlying card is still completing its internal initialization.

When the same image is written to a consumer card such as a SanDisk Ultra 8 GB Class 10, the unit boots reliably. The issue is not image corruption, not a bad card, and not a defective reader; it is a card-initialization latency that exceeds the kernel's default root-device timeout.

2. Affected Hardware, Firmware, and Cards

Cards from other vendors with similar controller/firmware combinations can also trigger the failure. Industrial cards from Toshiba M203 16 GB U1, Transcend 32 GB U1, and Kingston Industrial families have produced identical kernel-level boot hangs on x86 and i.MX8M Mini platforms (see NXP community thread iMX8M Mini: Problem boot from some micro SD cards), and similar behaviour is documented in the Raspberry Pi firmware repository (Rpi2 stops booting from swissbit #838).

3. Root Cause Analysis

The Linux kernel's MMC subsystem and the block-device layer have a tight race window between two events:

1. The kernel finishes device_initcall-driven SD card probe and exposes partitions to userspace.
2. The kernel's prepare_namespace() function tries to mount the root filesystem specified by the root= parameter (defaulting to /dev/mmcblk0p2 on the IoT2040).

The kernel waits for the root device to appear using the rootwait mechanism. The default wait time is controlled by CONFIG_CMDLINE and the root= argument's rootdelay value. On the Siemens BSP image the default behaviour is rootdelay=0 (no extra delay beyond the built-in rootwait polling loop), which is normally sufficient for consumer cards.

Industrial SD cards with pSLC (pseudo-SLC) or ruggedized controllers frequently perform a long post-power-on internal housekeeping pass (wear-levelling table reload, FTL rebuild, bad-block re-scan) that can take 300-800 ms before the card responds to a block-read command. The kernel sees no response on the root device, times out, and either:

• Drops to an initramfs shell with "ALERT! /dev/mmcblk0p2 does not exist. Dropping to a shell!" (when an initramfs is configured), or
• Panics with "VFS: Unable to mount root fs on unknown-block(0,0)" (when no initramfs is present), or
• Silently reboots (when the watchdog/auto-restart path is armed).

Once the card finishes initialization, the partitions do appear in the kernel log, confirming that the card is functionally healthy - it simply answered too late.

4. Reading the Boot Log

Connect a USB-to-TTL serial adapter (3.3 V, FTDI FT232 or CP2102) to the IoT2040's X14 debug header (TX, RX, GND; 115200 baud, 8N1, no flow control). Capture the console output to a log file. The failing sequence typically looks like:

systemd-boot: Booting EFI Default
Loading kernel... ok
Loading initrd... ok
[    0.842100] mmc0: new high speed SDHC card at address 0007
[    0.851000] mmcblk0: mmc0:0007 SD08G 7.42 GiB
[    1.012000] mmcblk0: p1 p2
[    1.118000] List of all partitions:
[    1.121000] No filesystem could mount root, tried:
[    1.121500]  ext4
[    1.122000] Kernel panic - not syncing: VFS: Unable to mount root fs

Note the timestamps: the partitions are enumerated after the kernel's mount attempt has already failed. The same card, when given one additional second, will mount successfully.

5. Solution: Add rootdelay=1

The robust and officially supported fix is to instruct the kernel to wait a configurable time before attempting to mount the root filesystem. This is done by appending rootdelay=1 (or higher, e.g. rootdelay=2 for very slow cards) to the kernel command line in the systemd-boot configuration file.

The configuration file lives on the EFI System Partition (ESP) at /loader/entries/boot.conf. The IoT2040 BSP mounts this partition as /dev/mmcblk0p1 in FAT format.

6. Step-by-Step Procedure

6.1 Prerequisites

• Working card reader (USB or built-in) on a Linux host.
• Swissbit SD card with the IoT2040 image already written.
• Access to a Linux shell with mount, blkid, and write permissions to FAT.

6.2 Mount the EFI System Partition

1. Insert the SD card into the host and identify the device: lsblk or dmesg | tail to see the new /dev/sdX or /dev/mmcblk0.
2. Mount the first (small, ~64 MB) FAT partition:

   sudo mount /dev/sdX1 /mnt/iot2040_esp

3. Verify the mount and inspect the existing boot entry:

   ls /mnt/iot2040_esp/loader/entries/
   cat /mnt/iot2040_esp/loader/entries/boot.conf

6.3 Edit boot.conf

The file is plain text and typically contains three lines: title, linux, and options. Locate the options line, which carries the kernel command-line parameters. The Siemens default looks like:

title        Siemens IoT2040
linux        /bzImage
initrd       /initrd.img
options      root=/dev/mmcblk0p2 rootfstype=ext4 console=ttyS0,115200n8

Append rootdelay=1 to the options line so it reads:

options      root=/dev/mmcblk0p2 rootfstype=ext4 rootdelay=1 console=ttyS0,115200n8

6.4 Unmount and Boot

1. Save the file with any editor (vi, nano, gedit).
2. Unmount the partition cleanly:

   sync
   sudo umount /mnt/iot2040_esp

3. Remove the card, reinsert it in the IoT2040, and power-cycle the gateway.

7. Verification

Watch the serial console (X14) during the next boot. With rootdelay=1 the log will show a one-second pause between the mmcblk0: p1 p2 line and the EXT4-fs (mmcblk0p2): mounted filesystem line, after which systemd will continue normally to the multi-user target. The unit should reach the login prompt or the configured auto-start application within 30-45 seconds of power-on.

If the gateway still fails to mount the root filesystem, increase rootdelay to 2 or 3 and retry. Some industrial cards require up to 5 s in cold-start conditions. The kernel accepts any integer value from 0 to 600 (the upper bound is set by root_delay_max in the kernel's mount code).

8. SD Card Selection Criteria

Not every industrial-grade card is suitable for low-power x86 IoT gateways. The Siemens IoT2040 manual states only that an "industrial SD card" shall be used, but the practical criteria are:

Cards that are known to work on the IoT2040 without any rootdelay workaround include consumer-grade SanDisk Ultra and Samsung EVO Select 8-16 GB cards. Cards that typically require rootdelay=1 or higher include Swissbit SFSD 8-16 GB, Toshiba M203 16 GB, Transcend 32 GB, and Kingston Industrial 8 GB. The NXP community thread confirms that the i.MX8M Mini and other boot-loaders exhibit the same symptom across the same card families, so the rootdelay workaround is portable.

9. Alternative Workarounds

If editing boot.conf is not possible (e.g. locked-down image provisioning), three alternative mitigations exist:

1. Use a smaller-capacity card: 4 GB cards initialize measurably faster than 8 GB+ cards in the same family.
2. Pre-warm the card: Issue several block reads from UEFI or from an early initramfs step to force the controller to load its FTL before the rootfs mount attempt.
3. Replace the card: Switch to a different vendor whose controller firmware has a shorter power-on-to-ready time. The Siemens support knowledge base is the authoritative source for current vendor recommendations.

10. Long-Term Reliability and Field Considerations

Industrial SD cards selected for longevity (Swissbit, ATP, Apacer, Transcend High-Endurance) often trade a slightly longer initialization time for higher endurance and power-loss resilience. The rootdelay=1 fix is a one-time configuration change that should remain in the image for the entire product lifecycle. It costs one second of boot time and zero runtime overhead.

For sites where the IoT2040 is rebooted frequently (e.g. on every shift), monitor the kernel log for the EXT4-fs line on each boot to confirm the card is still responding within the delay window. If the card's initialization time degrades below the configured delay (rare, but possible on cards near end-of-life with failing flash), increase the delay. A practical upper limit is 5 s; beyond that, swap the card.

Reference: the SIMATIC IoT2040 operating instructions, Siemens support entry ID support.industry.siemens.com (search for "IoT2040 manual" or "6ES7647-0AA00-0YA2"). The systemd-boot loader manual is at freedesktop.org - systemd-boot. The SD card full specification (Part 1, Physical Specification) is published by the SD Association.

11. Troubleshooting Matrix

12. FAQ