Troubleshooting S7-1500 CPU 1512C Ethernet Ping Dropouts with nodave on Linux

Engineer field-notes: A Siemens SIMATIC S7-1500 CPU 1512C-1 PN (catalog number 6ES7512-1CK00-0AB0 / 6ES7512-1CK01-0AB0) is wired directly to a Linux PC running the open-source nodave ISO-on-TCP library. After commissioning on TIA Portal V14, the link shows baseline 1–2 ms ICMP echo round-trips, but a 40-second "lock-up" of all pings occurs every few minutes. After the engineering laptop running TIA Portal is disconnected, the complete lock-up disappears, but residual 200 ms latency spikes remain. This reference decomposes the symptom, separates the PROFINET root causes, and provides a deterministic resolution path with port isolation, TIA broadcast containment, and nodave keep-alive tuning.

1. Problem Summary and Measured Symptoms

Use the values below as a baseline for what "normal" should look like on an S7-1500 PROFINET port in a 2-node topology (no managed switch, no PROFINET IO devices attached).

The 40-second blackout is the critical symptom: ICMP is a best-effort, kernel-handled service. A 40-second loss on a point-to-point link with auto-negotiated 100 Mbit/s full duplex cannot be caused by congestion. It can only be caused by one of the following:

1. Loss of physical link / port reset / re-initialisation of the integrated switch.
2. CPU firmware watchdog event that pauses the PROFINET stack or routing path.
3. TCP / ISO-on-TCP connection storm that temporarily blocks the Ethernet controller's transmit queue (less likely with 1 s ping).
4. Link-down / link-up event triggered by energy-saving negotiation, PoE-induced powering, or a marginal cable.

2. S7-1500 CPU 1512C PROFINET Interface Architecture

The CPU 1512C-1 PN provides an integrated 2-port managed PROFINET switch. Both RJ45 sockets belong to the same PROFINET IO controller interface (X1P1 / X1P2 in the device labelling) and are isolated by default only at the VLAN / MAC-flooding level, not at the IP routing level. The PROFINET stack itself continues to advertise DCP (Discovery and Configuration Protocol) and LLDP (Link Layer Discovery Protocol) on both ports even if only one cable is connected. Per the SIMATIC S7-1500 Automation System System Manual (entry ID 59191792), the integrated PROFINET interface is a single IP endpoint, not a routed pair.

Key characteristics relevant to this fault:

• Port 1 (X1P1) and Port 2 (X1P2) share the same IP address by default (single subnet, internal switch). The PLC's IP is bound to the PROFINET interface, not to an individual port.
• Both ports are 100 Mbit/s, full-duplex, auto-negotiation, auto-crossover (Auto-MDI/MDIX). 1 Gbit/s is not available on the standard integrated PROFINET interface; the CPU 1512C-1 PN does not ship with a 1 Gbit/s port.
• PROFINET port configuration is performed in TIA Portal under Device view → PROFINET interface → Port [X1] P1 / P2. Options include Transmission medium / duplex, Monitor, Enable port, Enable auto-negotiation, and Enable topology discovery.
• CPU firmware at the time of writing supports separate port enable / disable, but does not provide per-port IP routing. The CPU 1512C-1 PN is a layer-2 + ISO-on-TCP end node, not a layer-3 router.

3. Root Cause Analysis – S7-300 vs S7-1500

The reporter states that the same Linux PC and nodave application "previously worked well on a S7-300 CPU". This is the single most important clue and is the reason for the symptom shift. The S7-300 family (CPU 31x, e.g. 6ES7315-2EH14) uses a fundamentally different PROFINET implementation than the S7-1500 family:

The fact that the same application worked on S7-300 does not exonerate the S7-1500 setup; the higher PROFINET stack activity is the reason 40-second blackouts can occur in a configuration that used to be quiet. The two concrete mechanisms that cause a 40-second blackout on a 2-node link with no switch are detailed below.

3.1 Mechanism A – TIA Portal "Go online" broadcast storm

When the engineering laptop with TIA Portal V14 is connected to the second port of the CPU 1512C, every TIA online action (project tree refresh, accessible nodes scan, online & diagnostics open) triggers the following frames on the link:

• DCP Identify All Multicast (destination 01:0E:CF:00:00:01, every 1 s while a scan is in progress).
• LLDP TLV broadcasts on the second port, which the internal switch forwards to port 1 (the same broadcast domain) unless VLAN tagging is configured.
• SNMPv3 polls against the CPU's SNMP agent (UDP 161) and traps (UDP 162) if SNMP is enabled.

The Linux PC running nodave on port 1 sees these frames in the receive buffer, but its kernel queues ICMP echo replies at a higher priority. However, the S7-1500 PROFINET controller's internal switch uses a small number of buffer descriptors; a sustained broadcast flood of more than ~2000 frames/s can stall the egress queue, dropping not only the PC's TCP acknowledgements but the PLC's own ISO-on-TCP responses. nodave's connect request times out, the connection state machine enters retry, and the resulting chatter can extend the blackout window beyond 30 s.

3.2 Mechanism B – PLC port link-down / link-up

40-second blackouts with no packet loss after disconnecting TIA point to a second, more physical cause: the link is flapping. Auto-negotiation on a marginal Cat 5e / Cat 6 cable can take 4–8 s to re-establish. The S7-1500 PROFINET stack additionally executes a port state machine (init → link-up → data exchange) that, on a port reset, blocks application-layer traffic for the duration of topology rediscovery — typically 5–20 s depending on the number of expected PROFINET IO devices. If a port is misconfigured to "End of topology detection" with a non-zero expected PDEV count, the PLC will wait for all expected devices before resuming data exchange, which can extend the outage to 30–40 s.

To rule out (A) the reporter disconnected the TIA laptop, and the 40-second blackouts disappeared — confirming that the engineering PC is the primary source of the broadcast storm. The remaining 200 ms spikes are a separate issue covered in Section 7.

4. Diagnostic Procedure (Field Checklist)

Execute the following on the Linux PC and the TIA Portal laptop in order. Do not skip steps; each is required to isolate the next layer.

1. Confirm physical link on both ports. From the S7-1500 display or from TIA Portal under Online & Diagnostics → PROFINET interface → Port statistics, verify that Link status = "Up" and that the Speed / duplex shows 100 Mbps / Full duplex on both ports. Mismatched duplex is the most common cause of background packet loss in field installations. Per the S7-1500 PROFINET Function Manual (entry ID 49948856), force 100 Mbps / Full duplex on the S7 side if the link partner is known-good.
2. Capture a 60 s pcap with TIA disconnected. Use sudo tcpdump -i eth0 -w ping_only.pcap icmp while running ping -i 0.2 192.168.0.1 (5 Hz ping) for 60 s. Open in Wireshark. Inspect whether the spike is preceded by a DCP / LLDP / SNMP frame from the PLC. Useful filters: dcp, lldp, snmp, eth.addr == xx:xx:xx:xx:xx:xx (replace with PLC MAC). If yes, the 200 ms spike is an internal switch egress queue stall — fixable with port settings (Section 6).
3. Reconnect the TIA laptop. Re-capture for 60 s with TIA "Go online" idle. Compare the pcap to the previous one. With TIA online but idle, additional traffic should be limited to LLDP every 5 s. If broadcast or multicast traffic exceeds 50 frames/s while TIA is idle, a TIA add-in is generating chatter; update the add-in or close the project.
4. Open TIA "Online → Accessible nodes". This action forces a DCP Identify All. Time the blackout: with a default project, the reachable-nodes scan takes 3–5 s. If the blackout matches the scan window, you have confirmed Mechanism A.
5. Check the CPU diagnostic buffer. In TIA, Online & Diagnostics → Diagnostics buffer. Look for events with Event ID 0x013C (port link-down) or 0x013D (port link-up) and Event ID 0x14xx (PROFINET IO station failure). If a recurring link-down/link-up pair is present at the timestamp of each blackout, the cause is electrical (cable, connector, EMI) and must be solved in the physical layer (Section 8).
6. Verify CPU firmware version. On the CPU display: Settings → Overview → Firmware. For CPU 1512C-1 PN, the article's behavior was first improved in firmware V2.6.0 (broadcast load reduction) and again in V2.9.x (PROFINET stack hardening). Reference: SIMATIC S7-1500 / ET 200 CPU 1500 firmware update V2.9.x – release notes (entry ID 109974728).
7. Inspect nodave connection state. nodave uses ISO-on-TCP (RFC 1006) on TCP port 102. The connection is held open by TCP keep-alive, which defaults to 7,200 s on Linux. On any blackout exceeding 9 min 12 s the Linux kernel will send RST — but during a 40 s outage the connection is not closed; nodave's application layer will see the blackout as a "no response" timeout, not a TCP reset. Confirm by enabling nodave's diagnostic output (setDebug(1) in the C API) and watching for repeated "ISO PDU timeout" messages at the moment of each blackout.

5. Layered Diagnostic – tcpdump Wireshark Filter Set

For a 2-node S7-1500 + Linux PC link, the following Wireshark display filters give a fast view of the contributors. Replace the MAC address xx:xx:xx:xx:xx:xx with the value printed on the PLC's front label (X1).

# All PROFINET traffic on the link
pnio

# All DCP (PROFINET discovery)
dcp

# All LLDP (neighbour discovery)
lldp

# All SNMP (only if the CPU SNMP agent is enabled)
snmp

# Only the S7-1500's own frames
eth.addr == xx:xx:xx:xx:xx:xx

# ISO-on-TCP / TPKT (port 102, used by nodave)
tcp.port == 102 && tpkt

# Watch the S7-1500 broadcast rate per second
statistics -> Conversations -> Ethernet -> sort by Packets

A healthy direct link with no PROFINET IO devices should show fewer than 30 broadcast / multicast frames per second in steady state. Anything above 200 frames/s while TIA is "online but idle" is the broadcast load that triggers the 40-second blackout on a misconfigured port.

6. Port Isolation Configuration (TIA Portal)

The single most effective change is to separate the engineering traffic from the application traffic. The CPU 1512C-1 PN has a single PROFINET interface with two physical ports, so separation must be implemented as a VLAN or as a strict port disable, not as a routing change.

6.1 Recommended Topology

For a 2-node link with TIA engineering on the second port, the S7-1500 cannot physically isolate the two ports. The three practical options are:

Recommended path for this fault: disable port 2 in TIA while developing (Device view → PROFINET interface → X1P2 → Port options → Enable port: unchecked), use a separate unmanaged switch + Wi-Fi bridge for the engineering laptop, or connect the TIA laptop only for short sessions and use TIA's Online → Disconnect explicitly when finished.

6.2 Port Option Settings (Recommended for nodave Link)

7. Residual 200 ms Latency Spikes

With TIA disconnected, the 40-second blackouts vanish, but a few pings in any 5-minute window stretch to ~200 ms. On a 2-node link this is not a 100 Mbit/s congestion artifact; it is a single-frame round-trip anomaly. Common causes, ranked by field frequency:

1. DCP Identify All broadcast from the CPU. The S7-1500 PROFINET stack issues a periodic DCP Identify (typically every 1 s when topology is dirty, every 5 s when stable). If the internal switch forwards the broadcast and the Linux kernel processes the broadcast interrupt before draining the ICMP echo reply queue, the ICMP reply is delayed by 1 kernel tick (default 1 ms) plus DCP processing (variable, up to 200 ms in older firmware). Fix: see Section 6.1 (disable port 2 / topology) and update CPU firmware to ≥ V2.9.7.
2. ARP cache expiry. If the Linux host has ARP suppression enabled but the PLC is re-using the same IP across a CPU reset, the first ARP request after the reset triggers a 200 ms delay while the stack waits for the ARP reply. Fix: pin a static ARP entry on the Linux host: sudo arp -s 192.168.0.1 <MAC-of-PLC-X1>. Use ip neigh show to verify; the entry should show the permanent flag.
3. nodave ISO-on-TCP connection retry. If nodave has a connect-retry timer that fires during a TCP RTO, the resulting SYN/ACK exchange can preempt the ICMP reply. Fix: tune nodave's read/write timeout to ≥ 5 s and disable the application-level retry storm (Section 9).
4. Web server / OPC UA server background activity. The default-enabled web server on S7-1500 firmware ≥ V2.6 occupies CPU time on every port, and the OPC UA server publishes a KeepAlive every 10 s. If a KeepAlive is processed between the ICMP echo request and reply, the response is delayed. Fix: disable the web server in TIA (Properties → Web server access) unless actively used; reduce OPC UA publishing rate to the application minimum.
5. EMC / cable crosstalk. A 200 ms spike without packet loss is the typical signature of a single physical-layer retransmit on the PLC's PROFINET MAC, not of a real collision. Replace the Cat 5e cable with a shielded Cat 6A patch, route at least 30 cm from any VFD output cable, and verify that the shield is bonded to the cabinet backplane at both ends.

8. Physical Layer Inspection

The S7-1500 PROFINET ports are industrial-grade but they are not fault-tolerant. A marginal cable that "mostly" works on a 2-node link will produce ping spikes of the exact magnitude described. Inspect the following on the cable run between the Linux PC and the CPU 1512C X1 P1:

• Cable type: minimum Cat 5e / 24 AWG solid conductor, shielded (SF/UTP) preferred. PROFINET conformance class C requires Cat 6A S/FTP for industrial cabinet runs > 60 m.
• Length: ≤ 80 m for Cat 5e at 100 Mbit/s, ≤ 100 m for Cat 6A. Do not exceed.
• Connector: industrial M12 D-coded if the cabinet uses IP67, or RJ45 with a strain relief boot if IP20. The CPU 1512C-1 PN ships with RJ45 sockets; an M12-to-RJ45 adapter cord is required for M12 trunks.
• Shield bonding: 360° bonded at the cabinet backplane, not to a flying lead. The PROFINET grounding bar (Siemens 6GK1905-0AA00) is the recommended bonding point.
• Routing: minimum 30 cm clearance from VFD motor cables, servo power cables, and any cable carrying > 24 V / > 1 A in non-shielded form. Cross at 90° if intersection is unavoidable.
• PoE / Phantom power: the S7-1500 PROFINET ports are not PoE-powered; verify that the Linux PC's NIC is not injecting power, and that the patch panel mid-span is not equipped with a PoE injector.

Run a single 60-second continuous ping with the TIA laptop powered off and connected to a different electrical outlet. If the 200 ms spikes disappear entirely, the cause was electromagnetic coupling on the engineering PC's power line, not on the data cable.

9. nodave Connection Tuning

nodave uses ISO-on-TCP (RFC 1006). The following parameters can be tuned to make the application layer tolerant of the residual PROFINET jitter without flooding the link with retries. All values are valid for the C API distributed in nodave 0.8.5+; for the Java / .NET ports the parameter names differ.

Minimum C snippet to harden the Linux side of the nodave link. Insert before openSocket and after the daveNewConnection call:

/* nodave example: harden ISO-on-TCP socket before connect */
#include <sys/socket.h>
#include <netinet/in.h>
#include <netinet/tcp.h>

void hardenSocket(int fd) {
    int on = 1;
    /* Disable Nagle so small S7 PDUs do not get coalesced */
    setsockopt(fd, IPPROTO_TCP, TCP_NODELAY, &on, sizeof(on));
    /* Enable kernel TCP keep-alive */
    setsockopt(fd, SOL_SOCKET, SO_KEEPALIVE, &on, sizeof(on));
    /* Optional: tune keep-alive per-socket (Linux 2.6.37+) */
    int keepidle  = 600;  /* seconds before first probe  */
    int keepintvl = 30;   /* seconds between probes      */
    int keepcnt   = 5;    /* probes before declaring dead */
    setsockopt(fd, IPPROTO_TCP, TCP_KEEPIDLE,  &keepidle,  sizeof(keepidle));
    setsockopt(fd, IPPROTO_TCP, TCP_KEEPINTVL, &keepintvl, sizeof(keepintvl));
    setsockopt(fd, IPPROTO_TCP, TCP_KEEPCNT,   &keepcnt,   sizeof(keepcnt));
}

int main(void) {
    daveInterface *di = daveNewInterface(_192_168_0_1__0, 0, "IF1", 0, 0);
    daveConnection *dc = daveNewConnection(di, 0, 0, 0);

    /* Set tolerant timeouts (microseconds) */
    di->timeout     = 5000000;
    di->readTimeout = 3000000;
    di->writeTimeout= 3000000;

    daveSetDebug(dc, 1);  /* commissioning only */
    if (daveConnect(dc) == 0) {
        hardenSocket(dc->fd);  /* harden the live socket */
        /* ... application loop ... */
    }
}

Reference for the protocol itself: IETF RFC 1006 – ISO Transport Service on top of the TCP. nodave speaks the TPKT / ISO 8073 (class 4) subset used by S7 communication; the keep-alive tuning above is the Linux-side guard, the application-level retry is the nodave-side guard.

10. PROFINET Stack Hardening on the S7-1500

Beyond port disable, several TIA configuration changes reduce the broadcast load from the CPU itself. None of these is "for nodave"; they are best practice for any S7-1500 running a peer-to-peer ISO-on-TCP link.

Firmware download: SIMATIC S7-1500 CPU 1500 firmware update V2.9.x – release notes & download (entry ID 109974728). Always read the upgrade notes for OB / FB / FC library changes; the S7-1500 firmware upgrade can change the behaviour of TSEND_C / TRCV_C and the open user communication blocks used by nodave's PDU exchange.

11. Verification Procedure

After applying the changes in Sections 6, 7, 9, 10:

1. Power-cycle the CPU 1512C-1 PN (selector switch to MRES, then RUN) to ensure the port-disable and OPC UA / web server changes take effect on cold start.
2. From the Linux PC, run a 10-minute ping with TIA disconnected: ping -i 0.2 192.168.0.1 | tee /tmp/ping_after_fix.log. Expected: every round-trip is between 1 ms and 5 ms, no loss, no 200 ms spikes.
3. Connect the TIA engineering laptop on port 2 (which should now be disabled) and verify that no TIA traffic is visible on port 1. Use sudo tcpdump -i eth0 -e -p -c 1000 on the Linux PC and confirm that no frame has the TIA laptop's MAC as source address.
4. Enable TIA "Online" and run a 5-minute test with engineering connected. Expected: ping latency returns to 1–2 ms baseline; the only additional frames on the Linux PC are LLDP / DCP from the PLC, not from TIA.
5. Open the CPU diagnostic buffer in TIA. Confirm that the entries from the 10-minute test window show no event ID 0x013C / 0x013D (link-down / link-up) and no event ID 0x14xx (PROFINET IO failure).
6. Run a 30-minute nodave application soak test. Confirm that no ISO PDU timeout is logged, no TCP re-connect is required, and the Linux kernel reports zero TCP retransmissions. Use ss -ti dst 192.168.0.1 — every retransmission entry should read 0 in the retrans column.
7. Stress-test: from the TIA laptop, run Online → Accessible nodes 5 times in 60 s. Confirm that ping latency on the Linux PC remains below 10 ms during the scan (was previously the trigger of the 40 s blackout).

12. Troubleshooting Matrix

13. Edge Cases – Firmware, Hardware and Linux Variants

The fix above is not universal. The following edge cases are recorded in field service tickets on the S7-1500 family and deserve a check before declaring the fault resolved.

13.1 CPU 1512C-1 PN vs CPU 1512C-1 PN/DP

The 1512C-1 PN has an integrated PROFINET interface only. The 1512C-1 PN/DP (catalog 6ES7512-1CK00-0AB0 vs 6ES7512-1DK01-0AB0) adds a PROFIBUS DP master. The DP master is irrelevant to the ping fault but its diagnostics buffer writes an extra event at every PLC STOP → RUN, which can be confused with a PROFINET event when reading the buffer with TIA. Filter on Event ID 0x14xx for PROFINET and on 0x06xx for PROFIBUS.

13.2 S7-1200 (CPU 1215C, 1217C) – Same Symptom, Different Cause

The S7-1200 family uses the same PROFINET stack as the S7-1500 (S7-1500 stack is a superset). The 200 ms ping spike is identical. The fix is the same (disable port 2 in TIA Portal), but the S7-1200 has a different port-disable UI: Device properties → Ethernet addresses → Ethernet port 2. The S7-1200 also has a default "System clock broadcast" enabled, which adds an extra Ethernet broadcast every 10 s — disable it under Properties → Time of day → Synchronization if not used.

13.3 ET 200SP CPU 1510SP-1 PN / 1512SP-1 PN

The ET 200SP CPUs are functionally identical to the S7-1500 in PROFINET behaviour. The port labels differ: X1 P1 / X1 P2 on the BusAdapter interface. The fix in Section 6 is identical; the firmware release notes are shared with the S7-1500 family.

13.4 Linux Kernel Version Sensitivity

The Linux net/ipv4/tcp_input.c handling of broadcast vs unicast is kernel-version-sensitive. Kernels older than 4.14 will queue broadcast frames ahead of unicast echo replies under high broadcast load, which inflates the 200 ms spike. Upgrading the Linux host kernel to 5.4 LTS or later measurably reduces the spike duration. RHEL 8.4+, Ubuntu 20.04 LTS+, and Debian 11 are validated.

13.5 Power-Saving EEE (Energy-Efficient Ethernet)

Some Linux NIC drivers (particularly igb, e1000e) negotiate 100 Mbit/s EEE on direct connect to a 1512C, even when the PLC is set to forced 100/full. EEE is the S7-1500 PROFINET's most common cause of micro-blackouts on idle links. Disable with sudo ethtool --set-eee eth0 eee off and verify with ethtool --show-eee eth0 (the Active flag should be no). Per-port disable via TIA does not disable the PC-side EEE.

13.6 Virtual Machine (VMware / KVM) Linux Host

If the Linux PC running nodave is a VM, the vSwitch / bridge introduces its own broadcast handling and can produce a 200 ms spike even on a perfect physical link. Run cat /sys/class/net/eth0/speed to confirm that the VM NIC is 1000 Mbit/s negotiated to the physical switch; if the VM is bridged to a 10/100 physical NIC, the S7-1500's 100 Mbit/s and the VM's 1 Gbit/s can auto-negotiate to 100 Mbit/s with a duplex mismatch. Pin the VM to a 100 Mbit/s virtual NIC and force full duplex.

14. Commissioning and Safety Considerations

None of the changes in Sections 5–10 affect the S7-1500 safety program. The CPU 1512C-1 PN does not have a safety-integrated PROFINET interface (F-CPU variant is 6ES7512-1SK00-0AB0 or 6ES7512-1SK01-0AB0, e.g. CPU 1512SP F-1 PN). For non-F CPUs, the port-disable and OPC UA / web server changes are commissioning tasks with no impact on the safety program.

Field commissioning sequence recommended for the nodave link:

1. Pre-power checklist: verify that the cabinet has a common protective earth, that the cable shield is bonded to the backplane at both ends, and that the Linux PC is on the same protective earth as the PLC cabinet. A floating ground can produce 200 ms spikes from common-mode voltage on the Ethernet pair.
2. Power-up sequence: CPU 1512C first, wait for "RUN" on the display, then the Linux PC. Reversing the order can cause the PC's NIC to auto-negotiate before the PLC port is up, producing an initial EEE event.
3. Configuration: download the TIA Portal project, perform a "Go online" to verify the CPU, then explicitly "Go offline" before running the nodave application. Do not leave TIA "Go online" on the link during normal operation.
4. Soak test: 30 minutes of nodave polling with the Linux PC alone on the link. Confirm the verification checklist in Section 11 is clean.