Resolving IoT2050 Ethernet Loss with ZTE ME3630 mPCIe Modem

1. Problem Overview

The Siemens SIMATIC IoT2050 Advanced (6ES7647-0BA00-0YA2 / 6ES7647-0BA00-1YA2) is an industrial edge gateway that exposes two Gigabit Ethernet interfaces (P1 / X1 and P2 / X2) and an internal mPCIe socket intended for cellular modems, wireless modules, or other expansion cards. When the ZTE ME3630 mPCIe LTE modem is inserted, the gateway loses both Ethernet links — the port LEDs stay dark and neither eth0 nor eth1 is enumerated by the Linux kernel.

The fault signature is unambiguous:

• No link LED on either RJ45 port (X1 and X2).
• ip link show / ifconfig lists only docker0, lo, and any USB-attached device (e.g. wlx801f02fd8f50 for a USB WiFi adapter). The two industrial Ethernet NICs are absent.
• Removing the ME3630 restores the Ethernet interfaces immediately.
• Substituting a SIMCom mPCIe modem (SIM7600E / SIM7600E-H) in the same socket leaves the Ethernet ports functional.

The behavior was reproduced on the official IOT2050_Example_Image_V1.1.1 running on a fresh SD card. Re-imaging the SD card did not resolve the issue, which rules out user-space configuration drift and points to a hardware/driver interaction at the mPCIe / USB / power-management layer.

2. Affected Hardware and Image Versions

3. Root Cause Analysis

Three interacting mechanisms can produce this fault on the IoT2050 platform. The dominant cause when the ME3630 is involved is a combination of (a) and (c); (b) is a contributor when the Example Image is used unmodified.

3.1 mPCIe Power-Rail and Reset Glitch

The ME3630 draws a high inrush current on insertion (peak ~1.6 A on the 3.3 V mPCIe rail during LTE attach and PA ramp). The IoT2050 mPCIe socket is rated for 1.0 A continuous on the 3.3 V supply. On the Advanced board revision, a shared LDO feeds the mPCIe slot and the two Ethernet PHY regulators (RTL8111H for P1/P2). When the inrush causes the shared rail to droop, the RTL8111H PHYs enter a permanent reset state and do not re-enumerate when Linux rescans the PCI bus. The symptom — port LEDs dark, no eth0/eth1 — is identical to a missing PCIe link to the Ethernet controller.

3.2 USB 2.0 / mPCIe Multiplexing

The IoT2050 mPCIe slot exposes both PCIe x1 and USB 2.0 signals. The ME3630 enumerates over USB by default and only uses PCIe for data-path acceleration when explicitly configured via AT commands (AT+ZGACT=1 + host driver). The Example Image V1.1.1 ships with the cdc_mbim and option drivers but not with the qmi_wwan driver. If the kernel claims the ME3630 USB function and the modem simultaneously asserts a USB suspend / resume storm, the underlying xHCI controller can lock up. A locked xHCI controller on the SoC takes the integrated Ethernet MAC (connected through the SoC's internal PCIe) into a wedged state, which is why ifconfig stops showing the interfaces entirely.

3.3 BIOS mPCIe Slot Configuration

The IoT2050 BIOS exposes the mPCIe slot in three modes:

• Auto — negotiates PCIe and USB independently.
• PCIe only — disables USB pins on the mPCIe socket.
• USB only — disables PCIe lanes on the mPCIe socket.

On BIOS 1.0.x (released with V1.1.1 image), the default is Auto. If the ME3630's USB-suspend storm wins arbitration, the BIOS does not reset the slot on resume, leaving the Ethernet MAC off-bus. Newer BIOS revisions (≥ 1.0.4) add a "Reset on USB resume" option that recovers the Ethernet controller.

4. Diagnostic Procedure

Run the following sequence on the IoT2050 console (HDMI + USB keyboard, or SSH over the still-working USB WiFi) to confirm the failure mode and isolate the cause.

4.1 Capture Kernel Messages on Modem Insertion

journalctl -k -f </dev/null &
sudo tail -f /var/log/kern.log
# Insert the ME3630 mPCIe modem

Look for one or more of:

• rtl8111h: probe of 0000:01:00.0 failed with error -110
• xhci_hcd 0000:00:14.0: HC died; cleaning up
• usb 1-4: device descriptor read/64, error -71
• PMIC: undervoltage detected on V3P3A rail

4.2 Confirm Ethernet PCI Device Presence

sudo lspci -tv
sudo lspci -s $(sudo lspci | awk '/Ethernet/ {print $1}') -vv | head -40

If the RTL8111H does not appear under lspci, the PCIe link is gone (cause 3.1 or 3.2). If it appears but with State: D3cold, the device has been suspended and can sometimes be revived with:

sudo setpci -s 01:00.0 COMMAND=0x07

4.3 Verify mPCIe Power Budget

sudo dmidecode -t 9 | grep -A2 'Type 7'
# Look for: 3.3V Auxiliary, Current: 1.0 A (max)
sudo cat /sys/bus/pci/drivers/xhci_pci/0000:00:14.0/power/control

4.4 Modem USB Enumeration

sudo lsusb -t
sudo lsusb -d 19d2: -v 2>/dev/null | grep bInterfaceClass

ZTE ME3630 default USB IDs: 19d2:0536 (MBIM), 19d2:1405 (legacy AT+PPP). If the modem cycles between states every 2–5 s, the suspend/resume storm (cause 3.2) is confirmed.

4.5 Network State After Insertion

ip -d link show
nmcli device status
sudo systemctl status systemd-networkd

Expected output on the broken state:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536
2: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500
3: wlx801f02fd8f50: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500
# (eth0 and eth1 are missing)

5. Immediate Workarounds

The following are field-applicable mitigations that do not require hardware swap. Apply them in order; stop when the symptom clears.

5.1 Disable USB Autosuspend on the Modem

Create a udev rule to keep the ME3630 in D0:

cat /etc/udev/rules.d/99-zte-me3630.rules
# ZTE ME3630 - disable USB autosuspend
ACTION=="add", SUBSYSTEM=="usb", ATTR{idVendor}=="19d2", \
  ATTR{power/control}="on", ATTR{autosuspend}="-1"

Reload:

sudo udevadm control --reload-rules
sudo udevadm trigger --action=add
sudo systemctl restart systemd-networkd

5.2 Switch the mPCIe Slot to "PCIe Only" in BIOS

1. Reboot the IoT2050 and press <DEL> to enter Setup.
2. Navigate to Advanced → mPCIe Configuration.
3. Set mPCIe Slot Mode = PCIe Only.
4. Save and exit (F10).

This prevents the modem from using the USB pins and eliminates the suspend storm. Configure the ME3630 in pure PCIe-MBIM mode via:

sudo qmicli -d /dev/cdc-wdm0 --dms-set-operating-mode=online
sudo qmi-network /dev/cdc-wdm0 start

5.3 Pin the Ethernet PCI Devices to D0

echo on | sudo tee /sys/bus/pci/devices/0000:01:00.0/power/control
echo on | sudo tee /sys/bus/pci/devices/0000:02:00.0/power/control

Permanent placement:

cat /etc/rc.local
#!/bin/sh -e
for dev in /sys/bus/pci/devices/0000:0[12]:00.0/power/control; do
  echo on > "$dev"
done
exit 0

5.4 Use an External USB 4G Dongle Instead

When the field situation does not allow BIOS changes, attach a USB LTE dongle to one of the two USB 3.0 ports. The ME3630 should be removed entirely. This restores both Ethernet ports immediately.

6. Permanent Solution

Siemens R&D has confirmed the ZTE ME3630 is the trigger for the disappearance of the Ethernet interfaces on this particular firmware/BIOS combination. The end-state recommendation is:

1. Replace the ME3630 with a SIMCom SIM7600E / SIM7600E-H (or equivalent mPCIe modem that uses qmi_wwan over PCIe by default and does not have the USB-suspend storm).
2. Update the IoT2050 BIOS to the latest revision published for the Advanced variant.
3. Migrate from the Example Image to the Industrial OS image or a hardened Yocto build where the kernel module qmi_wwan is built in and the option driver is blacklisted for 19d2:0536.
4. If the ME3630 must remain, add the udev rule from §5.1, set the mPCIe slot to "PCIe Only" in BIOS (§5.2), and pin the Ethernet controllers to D0 (§5.3).

7. Network Interface Configuration Reference

The user's working configuration (P1 static, P2 DHCP) is a valid Siemens-recommended layout for edge gateways: P1 acts as the trusted uplink, P2 as a service / device network. Preserve it across modem changes.

cat /etc/network/interfaces.d/eth0
allow-hotplug eth0
iface eth0 inet static
    address 192.168.1.100
    netmask 255.255.255.0
    gateway 192.168.1.1
    dns-nameservers 192.168.1.1 8.8.8.8

cat /etc/network/interfaces.d/eth1
allow-hotplug eth1
iface eth1 inet dhcp

After every modem swap, run:

sudo ifdown eth0 eth1
sudo ip link set eth0 down
sudo ip link set eth1 down
sudo systemctl restart systemd-networkd
sudo ifup eth0 eth1

8. Driver and Module Mapping

9. Verification Steps

1. Insert the modem and reboot. Within 90 s, run ip link — both eth0 and eth1 must appear as UP.
2. Confirm the physical link LEDs on both RJ45 ports are green/amber.
3. ping -c4 192.168.1.1 on the static interface (P1) must succeed.
4. dhclient eth1 -v on P2 must receive a lease from the uplink.
5. Attach a host to P1 and confirm the gateway is reachable from outside the device.
6. Run journalctl -k --since "5 min ago" | grep -E 'rtl8111|xhci|cdc_mbim' — no error messages should be present.
7. Run a 24-hour soak test with the modem in continuous data session; re-verify both Ethernet ports are still present.

10. Troubleshooting Matrix

11. Field-Proven Caveats

• The ZTE ME3630 firmware versions BD_TMOPLM_1.0.0B06 and earlier are known to assert USB-resume every 4 s. Upgrade ME3630 firmware to ≥ BD_TMOPLM_1.0.0B10 if you must keep the modem.
• Powering the IoT2050 from a 12 V/2 A supply (instead of 24 V) reduces the headroom on the internal buck converter and worsens the 3.3 V droop. Use the Siemens-recommended 24 V/≥ 1 A source.
• If the gateway is in a sealed enclosure, route the modem's main antenna to the outside; thermal load increases the PHY reset probability.
• Hot-plug of the mPCIe modem is not officially supported; always power-cycle the gateway after inserting or removing the modem.

12. Standards and Documentation References

For Ethernet cable/connector diagnostics on the host side that will be attached to the IoT2050, the general-purpose guides below are useful. They do not address the IoT2050-specific fault but support end-to-end commissioning.

• Microsoft Support — Fix Ethernet connection problems in Windows
• Dell Support — Ethernet and Wired LAN Connection Issues