Resolving OpenVPN Routing Failures on Siemens IOT2040

1. Problem Overview

When the SIMATIC IOT2040 (6ES7647-0AA00-1YA2) is configured as an OpenVPN endpoint in a custom Yocto image (typically derived from meta-iot2000-example), the tunnel itself establishes cleanly: the TUN device comes up, the client receives an IP from the configured --ifconfig-pool range, and the client-side log reports Initialization Sequence Completed. Despite this, traffic sourced from the OpenVPN client never reaches the destination. Commands such as ping 8.8.8.8, dig google.com, and curl https://example.com produce no output and eventually time out.

The failure pattern is characteristic of a missing source NAT (MASQUERADE) rule on the IOT2040. The kernel is correctly forwarding packets between tun0 and the physical Ethernet interface (usually X2P1LAN on the upper RJ45 jack), but reply packets have no return path because the destination address visible on the public side is the OpenVPN client's tunnel address, not the IOT2040's outbound IP.

This article documents the root cause, the corrective iptables rule, the supporting sysctl change, and the methodology for making the rule survive reboot without breaking console or network connectivity.

2. Affected Hardware, Firmware, and Software Stack

The same root cause applies to the successor IOT2050 (6ES7647-0BA00-0YA2) running ArduPilot-style or Yocto images, but the network interface naming differs (eth0, eth1 instead of X1P1LAN, X2P1LAN).

3. Root Cause Analysis: Why MASQUERADE Is Required

OpenVPN's tun device creates a virtual point-to-point link with its own subnet (for example 10.8.0.0/24). When the server pushes redirect-gateway def1 bypass-dhcp to the client, the client installs a default route pointing at the tunnel endpoint.

For return traffic to flow correctly:

1. The IOT2040 kernel must accept and forward packets from tun0 to the outbound interface (X2P1LAN or whichever port faces the Internet). This is governed by net.ipv4.ip_forward.
2. The IOT2040 must rewrite the source address of forwarded packets so that the upstream router can route the reply back. Without source rewriting, the upstream router sees a packet sourced from the client's tunnel address (e.g., 10.8.0.6) and has no route to deliver the reply to 10.8.0.6.
3. The standard Linux tool for this is a POSTROUTING rule in the nat table that performs MASQUERADE on packets leaving the outbound interface.

The same effect is occasionally described as "site-to-site NAT" or "outbound NAT". On Linux, MASQUERADE differs from SNAT only in that the source address is read from the interface at packet egress, which is the desired behaviour for dynamic-address uplinks (DHCP, cellular, etc.). See the netfilter NAT documentation for the formal model.

3.1 Why a simple forward is not enough

Some users assume that with ip_forward = 1 the tunnel will work end-to-end. That assumption is correct for the forward direction but ignores the return path. Asymmetric routing is the most common reason for "tunnel comes up, no traffic passes" on edge devices, and is documented in the OpenVPN Routing on Linux guide.

4. Prerequisites and Pre-flight Checks

Before applying the fix, confirm that the baseline configuration is intact. Execute each command on the IOT2040 (via serial console, SSH on the management port, or the Yocto sshd service).

1. Verify the tunnel interface exists: ip addr show dev tun0 — should report an address inside the --ifconfig-pool range (e.g., 10.8.0.1/24 for the server side).
2. Verify the default route through the upstream interface: ip route show default — should resolve via X2P1LAN or X1P1LAN and the ISP gateway.
3. Verify the kernel is willing to forward: sysctl net.ipv4.ip_forward — must return 1.
4. Verify the NAT table is loaded: iptables -t nat -L POSTROUTING -n -v — should return without iptables: Table does not exist. On the IOT2040 reference image the iptable_nat module is built-in or pulled in by openvpn automatically.
5. Verify outbound reachability from the IOT2040 itself: ping -c 3 8.8.8.8 — must succeed before you expect tunneled traffic to succeed.

5. Step-by-Step Resolution

5.1 Enable IPv4 forwarding

Edit /etc/sysctl.conf and ensure the following line is present and uncommented:

Apply the change without reboot:

Persist across boots via the same file. The sysctl.conf(5) manual page describes the precedence rules; sysctl -p reads the file in order and applies values to /proc/sys/.

5.2 Confirm the OpenVPN server pushes the redirect

The server configuration (typically /etc/openvpn/server.conf) must contain the directive that informs the client to send all traffic through the tunnel:

The corresponding client configuration must request the same behaviour explicitly so that the gateway change applies even if the server side forgets to push it:

Refer to the OpenVPN 2.6 Reference Manual for the exact semantics of def1 (modify the default gateway rather than prepend a more specific route) and bypass-dhcp (do not overwrite the DHCP-obtained DNS servers on Windows clients).

5.3 Apply the MASQUERADE rule

The minimal, correct fix is a single iptables rule:

Notes on the rule:

• -t nat selects the NAT table. The default filter table is not appropriate for source rewriting.
• -A POSTROUTING appends to the POSTROUTING chain, the only chain where source address modification is legal for routed traffic.
• -o tun0 restricts the rule to traffic leaving via the OpenVPN tunnel. This is the correct direction: the IOT2040 is the server, and tun0 is the interface facing the remote client.
• -j MASQUERADE performs dynamic SNAT, picking the source address from the outgoing interface. The Linux netfilter team recommends MASQUERADE over static SNAT for interfaces whose address can change (DHCP, PPP, cellular). Reference: netfilter NAT HOWTO, section 6.

If the IOT2040 is instead running in client mode and tun0 is the interface towards the remote OpenVPN server, the rule must be applied to the WAN-side interface (for example -o X2P1LAN) so that traffic sourced from the IOT2040 itself appears to originate from the WAN IP. Always match the rule to your traffic direction.

5.4 Confirm with tcpdump

From a remote host with console access on the same network as the IOT2040, capture traffic on the WAN interface to confirm rewriting:

Issue a ping from the OpenVPN client through the tunnel. The captured packets must show the IOT2040's WAN IP as the source, not the client's tunnel IP. If the tunnel IP is still visible, MASQUERADE is not active.

6. Making the Configuration Persistent

The default IOT2040 image uses sysvinit with init scripts under /etc/init.d/. There is no NetworkManager and no firewall service by default, so iptables rules inserted manually are lost on the next reboot. The supported way to make them stick is to embed iptables-restore into the /etc/network/interfaces stanza for the relevant interface.

6.1 Save the current ruleset

Dump the live ruleset (filter, nat, mangle) to the directory Debian-derived images expect:

Verify the file exists and contains the MASQUERADE line. The directory /etc/iptables/ is the conventional location on Debian-derived systems; it is not called /etc/network/ on the IOT2040 image, although the same ruleset can be placed under /etc/network/ if your build uses the networking service. Note that the live image ships with the directory already present at /etc/iptables/, which is why a typographical mistake (writing /etc/network/iptables.rules) results in a silent failure and the next reboot drops the rule.

6.2 Hook the restore into the interface

Append the following pre-up and post-down directives to the X2P1LAN stanza in /etc/network/interfaces:

auto X2P1LAN
iface X2P1LAN inet dhcp
    pre-up iptables-restore < /etc/iptables/rules.v4
    post-down iptables-save -c > /etc/iptables/rules.v4

Restart the interface to validate the persistence path without rebooting the IOT2040:

After the interface comes back up, verify that the rule is in place:

The output should contain -A POSTROUTING -o tun0 -j MASQUERADE.

6.3 Alternative: udev-triggered restore

For builds that use BusyBox mdev or that lack the pre-up hook, an alternative is to create a small init script that runs after openvpn:

#!/bin/sh
iptables-restore < /etc/iptables/rules.v4

Place the file at /etc/init.d/S99nat and mark it executable. The numeric prefix must be greater than the Sxxopenvpn script in the same directory so that OpenVPN is up before the rule is loaded.

7. Verification and Testing

After the rule is applied and persisted, validate the end-to-end path from the OpenVPN client:

1. Establish the tunnel from the remote client. Confirm Initialization Sequence Completed in the client log.
2. Check the routing table on the client. The default route must point to the tunnel endpoint (e.g., 10.8.0.5 dev tun0).
3. From the client, run ping 8.8.8.8. If replies arrive, the forward and return paths are working.
4. From the client, run dig +short example.com. DNS resolution through the tunnel requires that the IOT2040 is a recursive resolver or forwards to one; if the IOT2040 itself cannot resolve, push a public DNS server (e.g., 1.1.1.1) via the OpenVPN dhcp-option DNS directive.
5. From the client, run curl -v https://example.com and confirm HTTP/1.1 200 OK.
6. On the IOT2040, confirm the counter on the MASQUERADE rule is incrementing: iptables -t nat -L POSTROUTING -n -v. The pkts and bytes columns for the MASQUERADE entry should grow during the test.

If the counters increment but the client still cannot reach the destination, the bottleneck is on the WAN side; check the IOT2040 default route, the upstream router, and any provider-side firewalls.

8. Troubleshooting Matrix

9. Performance and Security Notes

The IOT2040's Quark x1020 CPU has no AES-NI and no hardware crypto accelerator. A 128-bit AES-GCM tunnel running at line rate of the 1 Gbps Ethernet ports is not feasible; the platform is typically deployed at site-to-site rates of 10–50 Mbps when VPN is enabled. Measure with iperf3 -c <server> -P 4 -t 30 and choose the cipher accordingly.

Security hardening checklist for an OpenVPN deployment on the IOT2040:

• Use tls-crypt-v2 or at minimum tls-auth to harden the TLS control channel.
• Restrict the management interface: management 127.0.0.1 7505 with a Unix socket or password file.
• Bind OpenVPN to a single interface with local X2P1LAN to prevent the daemon from listening on the LAN side.
• Run the OpenVPN process as a dedicated user (user nobody, group nogroup) so that any kernel vulnerability does not yield root directly.
• Rotate the Easy-RSA PKI annually; the IOT2040 image ships with sample CA files in /etc/openvpn/easy-rsa/ that must not be used in production.
• Enable persist-key and persist-tun so that SIGUSR1 restarts do not leak privileges or interfaces.

These recommendations are derived from the OpenVPN Hardening Guide and the Siemens IOT2040 device manual security appendix.

10. Field-Proven Caveats and Edge Cases

• Multiple LANs on the same subnet. If two networks reachable from the IOT2040 share the same RFC1918 range (for example, two plants on 192.168.1.0/24), MASQUERADE alone will not disambiguate. The fix is a 1:1 static NAT mapping or to renumber one of the sites.
• IPv6. The default IOT2040 image disables IPv6 forwarding. If you need v6 over the tunnel, enable net.ipv6.conf.all.forwarding=1 and add a matching ip6tables rule, but be aware that some Yocto BSPs do not ship the ip6table_nat module.
• MTU and fragmentation. A typical tunnel MTU is 1421 bytes after the OpenVPN overhead. If the underlying WAN path fragments aggressively, lower the link MTU on the IOT2040's WAN interface to 1400 or apply the MSS-clamp rule shown in the matrix.
• Watchdog on production units. Some Siemens recovery images wipe /etc/iptables/ on factory reset. Keep a copy of rules.v4 in your build artifact directory so that a recovery is a single-file copy.

11. Quick Reference: Minimal Working Configuration

For a copy-paste baseline, the complete server-side change set is:

# /etc/sysctl.conf
net.ipv4.ip_forward=1

# /etc/openvpn/server.conf (excerpt)
push "redirect-gateway def1 bypass-dhcp"
push "dhcp-option DNS 1.1.1.1"

# Apply and persist
iptables -t nat -A POSTROUTING -o tun0 -j MASQUERADE
iptables-save > /etc/iptables/rules.v4

# /etc/network/interfaces (excerpt)
auto X2P1LAN
iface X2P1LAN inet dhcp
    pre-up iptables-restore < /etc/iptables/rules.v4
    post-down iptables-save -c > /etc/iptables/rules.v4

Reboot once after editing /etc/network/interfaces to confirm the persistence path end-to-end.