Troubleshooting S7-400 STOP Event 16#457F and 16#75D1 in OB35

Troubleshooting S7-400 STOP Event 16#457F and 16#75D1 in OB35

1. Problem Summary

A SIMATIC S7-400 CPU in a sawmill sawing-line station begins to randomly transition from RUN to STOP. The CPU is an F-CPU variant (for example CPU 416F-3 PN/DP or CPU 417-4 H) running the safety program in OB35. The diagnostic buffer captures two paired events at the exact same millisecond timestamp:

• Event ID 16# 457F — STOP caused by STOP command, breakpoint in user program: cyclic interrupt OB (OB 35), priority class 12, FB 4001, module address 2028.
• Event ID 16# 75D1 — Safety program: internal CPU error, internal error information 7:0000, instance DB 12814 of the causing F system block.

Both events timestamp 08:52:30.217 PM, 08/03/2001 and both request the operating-mode transition from RUN to STOP (internal). The 16# 457F entry is the triggering STOP command and the 16# 75D1 entry is the F-runtime recording of the same incident.

The operator observes that the faults correlate with IT-side work on the plant network: switches, firewall and VLAN changes occur within minutes of the faults. After PLC firmware updates and F-block rewrites by the line manufacturer (assisted by Siemens technical support) fail to remediate, the CPU is replaced. Replacement resolves the issue, but the root cause is ambiguous: was it CPU hardware aging (the unit was approximately 12 years in service) or did the network perturbation fatally disturb the F-runtime?

This article reconstructs the diagnostic chain, separates the two possible root causes, and lists the field-proven checks, OEM service options, and prevention steps.

2. Diagnostic Buffer Event Decoding

The S7-400 CPU diagnostic buffer is battery-backed RAM. It stores the last 120 events, each 24 bytes, including fault and stop events. The buffer is read with STEP 7 / TIA Portal: Online > Diagnostics > Diagnostic Buffer, or with the SFC 51 RDSYSST in the user program.

16# 457F is the generic STOP-request event. The "breakpoint in user program" wording indicates the CPU's instruction pointer reached a programming or runtime breakpoint inside OB35 at FB 4001. In a F-CPU runtime, breakpoints inside F-user blocks are not legal because they break determinism in the safety execution model. When the F-runtime detects the breakpoint it raises a fault and forces the CPU to STOP. The 16# 75D1 entry that follows is the F-runtime's own record of the same incident, attributing the cause to iDB 12814.

The "module address 2028" value is the diagnostic ID slot — in this case it does not refer to an I/O slot but to the F-block numbering base. The Coding bytes for 75D1 encode F-runtime internal error codes. Byte 4 ("0007") decodes to F-system internal code class 7 (F-runtime group administration) and bytes 5–6 ("0000") indicate no further sub-classification. Full decoding requires the CPU's firmware-specific diagnostic entry table in the S7-400F and S7-400FH Automation System Manual.

3. OB35 Cyclic Interrupt Context

OB35 is one of nine cyclic interrupt OBs (OB30 through OB38) supported by the S7-400 CPU. Default properties:

In an F-CPU, OB35 is the conventional host OB for the safety program because its deterministic phase matches the PROFIsafe watchdog (typical 100 ms – 200 ms per V2 mode, 1 ms – 25 ms per V2.6 mode). OB35 preempts OB1 regardless of OB1 priority.

The call chain observed in this incident is:

OB35  (priority 12, phase 100 ms)
 &-- FC500  (F-CALL / F-SHIELD wrapper from the F-library)
      &-- FB4001  (F-user block, instance DB 12814)

iDB 12814 is the per-instance data of FB4001. iDB numbers above 10000 are typically auto-assigned by the F-library (Distributed Safety / F-Configuration Pack) and correspond to per-call F-block instances. The OEM line builder owns the F-library configuration; the end user cannot inspect iDB contents without the F-source.

4. F-System Block Fault Analysis

When an F-block raises a runtime fault the F-CPU executes a STOP transition in three steps:

1. F-runtime group raises an internal fault.
2. F-shutdown logic records the faulted iDB number (here 12814) and the fault class (7:0000) into the diagnostic buffer.
3. CPU executes a controlled STOP and the corresponding plain-text event appears in the buffer.

For event 16# 75D1 the "internal error information 7:0000" field indicates the fault class is 7, which in the F-CPU diagnostic table corresponds to F-runtime group administration errors. This class is raised when the F-runtime cannot complete a scheduled pass — typically due to:

• Watchdog timeout in OB35 (configured cycle exceeded)
• F-block code corrupted in load memory
• F-CPU internal CRC mismatch between code and iDB
• Stack overflow or invalid iDB pointer inside the F-block

The first diagnostic step is to compare the configured OB35 cycle against the worst-case execution time of FC500 + FB4001. On a 416F-3 with a typical 1 ms PROFIsafe net, OB35 should finish in less than 20 ms. If the OB35 measured runtime is approaching the configured watchdog, the CPU raises the same fault class. To confirm, monitor OB35 runtime using SFC 6 RD_SINFO or with the STEP 7 cycle-time tool. Suspect an F-block cycle-time problem if the runtime grows linearly with field-device count after a configuration change.

If cycle time is healthy, the next suspect is F-block consistency. Re-compile the F-program from the F-source owned by the OEM line builder, transfer to the CPU offline, and re-issue the safety acceptance test. If the OEM is unable or unwilling to provide the F-source, the only remediation paths are CPU replacement or Siemens repair diagnostics.

5. PROFIsafe and F-I/O on PROFIBUS

The plant uses PROFIBUS DP to reach distributed F-I/O modules. PROFIsafe is the safety protocol layered on top of PROFIBUS DP-V0 / DP-V1 / PROFINET IO. Each PROFIsafe slave has a unique F-destination address (1 – 1023) which the F-host validates on every cycle. Common PROFIsafe module families on PROFIBUS:

If the F-CPU detects a PROFIsafe watchdog violation (for example due to PROFIBUS cable damage, EMC bursts, or a slave going offline) it raises a passivation event (not a STOP) for that specific F-slave. A passivation does NOT generate event 16# 75D1 — it generates 16# 75D0 / 16# 75D2 area codes. The presence of 16# 75D1 implies the F-runtime could not complete the cycle at all, which is consistent with a CPU-internal fault rather than a slave communication fault.

To check PROFIBUS health, read the diagnostic buffer of the CP 443-5 Extended (if used) or use the built-in PROFIBUS diagnostics in STEP 7: PLC > Module Information > Diagnostic Buffer for each F-slave. Confirm that:

• No slave is reporting BUSF (PROFIBUS fault) LED.
• PROFIBUS cycle is less than 50 ms with deterministic token rotation.
• Repeaters are not powered down by IT-side VLAN changes.

Refer to the PROFIsafe Profile for PROFIBUS and PROFINET and the S7-400F and S7-400FH Automation System Manual for the full F-event catalog.

6. Plant Network Interference Considerations

The plant uses two Ethernet cards inside the S7-400 CPU rack. The CPU 41x-3 PN/DP and CPU 41xF-3 PN/DP variants ship with two integrated interfaces:

Both Ethernet interfaces were remotely accessible to the IT network. Common IT-side changes that disturb S7-400 RUN behavior include:

• VLAN / subnet change without re-routing PG/PC connection (CPU becomes unreachable; PNIO stack may go to fault state).
• Spanning-tree re-convergence after switch firmware upgrade — typical 30–60 s; PNIO AR (Application Relationship) times out at 3–60 s.
• Broadcast storm from a misconfigured switch port — CPU PROFINET stack can drop ARs and require AR re-establishment.
• Firewall / NTP / DNS changes that affect S7 communication routing.
• Port-security 802.1X activation — the S7-400 CP/PN stack does not support 802.1X natively except on firmware V6.0.4 and later.

In this incident, the operator tested running the PLC with the Ethernet cables disconnected. The fault still recurred, which rules out an in-flight network packet as the trigger. The fault was a deterministic CPU-internal fault (event 16# 75D1, iDB 12814), independent of the network. Network correlation is therefore co-incidence: IT changes happened on the same day but did not cause the STOP. This is consistent with the firmware update not helping and with the fault not appearing when the line was running without network connectivity.

7. CPU Hardware Aging and Stress Indicators

The CPU had been in service approximately 12 years. S7-400F CPUs are specified for long-term industrial service; however, certain failure modes accelerate at the upper end of the life curve:

If all four LEDs (SF / BF / DC5V / FRCE) start blinking simultaneously the CPU has reached "tired" mode — internal self-test failed. That mode was not observed here, but the F-runtime 7:0000 class of fault is consistent with capacitor-aging-induced timing drift in the mainboard clock tree. Capacitor aging is invisible to STEP 7 diagnostics; it manifests as random runtime faults that correlate with ambient temperature or with the time-since-power-on.

Action: when an S7-400F CPU reaches 12+ years of continuous service and starts to emit F-runtime faults that are not reproducible by recompiling the F-program, the recommended remediation is preventive CPU replacement. Capture the on-line program and F-source, replace the CPU (typical lead-time 1 week, typical cost around EUR 18,000 for a 416F-3 PN/DP), transfer the project offline, run the safety acceptance test, and dispose of the old CPU through the Siemens repair center if a forensic analysis is required.

8. Firmware Update Strategy

Firmware updates on F-CPUs are restricted by the safety certification. The following firmware families apply to common S7-400 CPUs:

Firmware updates require:

1. Download the firmware file (.upd) from Siemens Industry Online Support using the order number.
2. Place the CPU in STOP.
3. Connect via STEP 7 / TIA Portal and select PLC > Update Firmware.
4. For F-CPUs the safety signature is invalidated; the safety acceptance test must be re-run by a certified F-engineer.

In this incident the OEM line builder attempted a firmware update before replacement. The firmware update did not resolve the fault, which is consistent with the fault being a hardware issue (mainboard aging) rather than a firmware bug. A firmware update is appropriate as a first-line test when the fault class is "soft" (communication stack bug) but is rarely effective for F-runtime class 7 faults.

9. Siemens Repair Diagnostic Option

When the root cause is ambiguous and CPU replacement is costly, the failed CPU can be returned to the Siemens repair service for failure-mode analysis. Standard repair process:

1. Open a support request via the Siemens Industry Online Support portal.
2. Request an RMA (Return Material Authorization) for the failed CPU. The RMA is keyed to the CPU's serial number (SVP-number).
3. Ship the CPU with a fault description including the diagnostic buffer dump (.txt or screenshot) and the STEP 7 project archive.
4. Siemens performs the Standard Repair and returns the CPU with a repair report including root-cause classification (hardware fault, firmware fault, user error, environmental).

Standard Repair typically returns the unit within 10–14 working days. Express Repair (3-day turnaround) is available at higher cost. If the fault is non-reproducible the report may classify it as "NFF — No Fault Found", which still provides certification for the safety authority. The repair report can be attached to the plant's safety-case file.

10. Step-by-Step Diagnostic Procedure

Follow this order when the F-CPU repeatedly logs 16# 457F / 16# 75D1 with breakpoint in OB35:

1. Capture the diagnostic buffer to a file: STEP 7 > PLC > Diagnostics/Setting > Diagnostic Buffer > Save As. Note the exact timestamp pattern (recurring same time-of-day versus random).
2. Read OB35 measured runtime via PLC > Module Information > Cycle Time. If OB35 runtime approaches the configured watchdog, increase the OB35 timebase or split the F-program into multiple cyclic OBs.
3. Verify the F-program on the CPU matches the F-source by re-transferring from STEP 7 with the F-source obtained from the OEM line builder. If the OEM cannot provide F-source, escalate via Siemens support.
4. Inspect PROFIBUS health: read the CP 443-5 diagnostic buffer, scan for BUSF flicker, verify repeater power and PROFIBUS termination at both ends.
5. Check the F-CPU battery (BATF LED). If red, replace the backup battery before further diagnostics — RAM corruption can mimic F-runtime faults.
6. Compare the CPU's firmware version against the Siemens product notice list. If a known firmware bug exists for the event class, plan a firmware update (see Section 8).
7. Decouple the CPU from the plant network by disconnecting both Ethernet ports. Run the line for at least 4 hours of normal duty. If the fault persists the root cause is CPU-local; if it disappears the root cause is network-originated.
8. If all above steps fail and the CPU is older than 10 years, replace the CPU with a new or factory-refurbished unit. Capture the old CPU's diagnostic buffer to file and route it to the Siemens repair center for forensic analysis (see Section 9).

11. Verification and Acceptance

After CPU replacement or firmware update, the following acceptance steps are mandatory:

1. Restore the STEP 7 project (including F-source) from the project archive.
2. Perform an online/offline comparison: STEP 7 > PLC > Compare. The on-line and off-line blocks must be byte-identical.
3. Run the safety acceptance test per the F-Configuration Pack checklist. Test every safety function from the safety requirement specification: E-stop, guard-door interlock, two-hand control, safe speed monitor. Record the test certificate.
4. Run the line for 48 hours of normal duty and confirm that the diagnostic buffer contains no 16# 457F, 16# 75D1, 16# 49xx, or 16# 50xx events.
5. Update the change log and safety-case file with the CPU replacement, the firmware version, and the repair report reference (if obtained).

If the new CPU is a different hardware version (for example V6 versus V5), confirm that the F-program compiles correctly with the new CPU's F-library version. Mismatched F-library versions between old CPU and replacement are a common cause of re-occurrence after replacement.

12. Prevention and Best Practices

To prevent recurrence of similar faults and to reduce mean-time-to-repair:

• Network segmentation: place a managed firewall or industrial router between the production PROFINET / PROFIBUS network and the IT backbone. Use a VLAN dedicated to PLC traffic and a separate VLAN for SCADA / HMI.
• 802.1X / port-security: if IT requires 802.1X on production ports, deploy a managed switch with MAC bypass for known PLC MACs, or upgrade to S7-400 CPUs that support 802.1X (firmware V6.0.4+).
• NTP: synchronise the CPU time to a plant NTP server so diagnostic buffer timestamps are consistent across the plant. Use SFC 1 READ_CLK and SFC 0 SET_CLK.
• Spare-part strategy: hold one spare S7-400F CPU of each variant on site for 24-hour swap-out. Lead time for new units from Siemens is typically 1 week, which is unacceptable for a continuous-process sawmill.
• Preventive replacement: for S7-400F CPUs older than 12 years, plan a preventive replacement during a planned shutdown. The cost (EUR 15,000–EUR 25,000) is lower than the cost of an unplanned outage.
• Change management: require IT to submit network changes to a change-control board that includes automation engineering. Reject any change that touches the production VLAN without an automation sign-off.
• Diagnostic buffer offload: periodically offload the CPU diagnostic buffer to a historian (WinCC / PCS 7 OS log). Use SFC 51 RDSYSST with SSL_ID W#16#0132 INDEX W#16#0008 to read the event log and store it externally.

FAQ