Troubleshooting S7-400 CPU416-3 STOP Due to IO Error Fault

Overview of the S7-400 CPU416-3 STOP Due to I/O Error

The Siemens SIMATIC S7-400 CPU 416-3 family (catalog numbers 6ES7416-3XL00, 6ES7416-3ER05, 6ES7416-3ES06, 6ES7416-3FR05, 6ES7416-3FS06, 6ES7416-3JR05, and the current 6ES7416-3XK04) is engineered for high-end, redundant automation. The CPU can leave RUN and enter STOP with diagnostic event W#16#4548 when its integrated I/O management detects that one or more distributed or central I/O modules have failed to acknowledge access, or when hardware faults disrupt the PROFIBUS DP, the central backplane (P bus / K bus), or the communications bus. The mode transition is governed by the OB configuration: if the assigned OB 82 (diagnostic interrupt), OB 85 (priority class error), OB 86 (rack/DP slave failure), or OB 122 (I/O access error) is missing, the CPU defaults to STOP. The mode transition is reported in event class 4, the "Stop events and other mode changes" group, per the official Siemens documentation for S7-300/S7-400 stop events.

On plants that have been in continuous service for a decade, a recurring STOP from a single I/O error is rarely the symptom of one bad module. It is almost always the symptom of a degraded mechanical/electrical layer (backplane pin, connector, shield, terminator) and a missing OB layer that allows a single fault to escalate into a full CPU halt. This article covers the event codes, the diagnostic buffer read procedure, the rack and PROFIBUS inspection workflow, and the OB-level hardening required to keep a CPU 416-3 in RUN after a single node drops.

Event Class 4 — Stop Events and Mode Change Codes

Event class 4 in the S7-400 diagnostic buffer captures all transitions out of RUN. The IDs the field engineer must memorise are listed below; the entries are taken from the Siemens stop events reference.

For the specific case of an I/O loss, the events that should appear in the buffer in succession are: a module removal/failure event (W#16#4571 or W#16#4572), the OB 85/86/122 entry, and finally the W#16#4548 STOP entry. If W#16#4548 appears alone, an OB is missing and is escalating the failure.

W#16#4548 — STOP Caused by I/O Management

Event W#16#4548 is generated when the operating system cannot complete an I/O access, the access would normally be routed to an OB error handler, and that OB is either missing, deselected, or has reached its retry limit. The associated OB is normally OB 85 (priority class error) and/or OB 122 (I/O access error). The event message includes the slot and the logical address that failed, which can be cross-referenced against the hardware configuration.

Sample buffer entry (from a real 6ES7416-3ES06-0AB0):

Event 1 of 100:  STOP caused by I/O management
  Event ID       : 16#4548
  OB             : OB 85 / OB 122 (deselected)
  Module address : 256 / 0  (slot 4 in ET 200M)
  Diagnostic ID  : 16#00FF
  Mode transition: RUN -> STOP
  Date / Time    : 14.03.2024  09:42:17.412

The accompanying event in the buffer is normally W#16#4572 "STOP caused by rack/DP station failure" with the same logical address. Together they identify the failing PROFIBUS node or the central slot that is intermittently dropping. For a CPU 416-3, the diagnostic buffer has space for 100 (small) to 320 (large) entries depending on the MMC; the most recent 100 must be exported before the next STOP so they are not overwritten by the new event storm.

Hardware Topology — Where to Look First

A CPU 416-3 system typically consists of:

• One central rack (CR) with the CPU 416-3 and a power supply (PS 405/PS 407) plus signal and function modules.
• Up to 21 expansion racks (ER) connected by IM 460/IM 461 send/receive interface modules.
• One or more PROFIBUS DP masters (CP 443-5 Extended) and PROFINET IO Controller (integrated PN interface of the CPU 416-3 PN/DP).
• Distributed I/O stations (ET 200M, ET 200S, ET 200pro) connected via PROFIBUS or PROFINET.

When a STOP appears with no single station pointed to, the root cause is generally at one of three layers:

1. Mechanical layer — the central backplane, the IM cable, the PROFIBUS connector, or the shield termination.
2. Electrical layer — power supply decoupling, ground loops, missing or duplicate terminators on PROFIBUS segments.
3. Configuration layer — missing OB (OB 85, OB 86, OB 122), wrong slot, firmware-incompatible module.

On 10-year-old installations, the mechanical layer is the most common. The P bus connectors on the backplane are gold-plated but the spring tension degrades; the IM 461 connecting cable wears where it flexes; PROFIBUS D-sub connectors oxidise and the shield contact opens up.

Diagnostic Buffer Read Procedure (STEP 7 V5.x)

The diagnostic buffer is the primary diagnostic artifact. Always capture it before a power cycle, otherwise the volatile entries are lost.

1. Connect the PG to the CPU 416-3 MPI/DP port (MPI cable 6ES7901-0BF00-0AA0 or PROFIBUS cable 6ES7901-4BD00-0AA0).
2. Open STEP 7 V5.x (V5.5 SP2 or later for full CPU 416-3 support; for TIA Portal use V13 SP1 or V14 and the CPU 416-3 HSP). The "Online > Accessible Nodes" path works when the CPU is still in STOP.
3. Select the CPU, then PLC > Diagnostic/Setting > Diagnostic Buffer.
4. Press the Save button to export the buffer to a text file. The default file is <Project>\<CPU>\DIAGBUF.TXT. Keep the file with a timestamp; DIAGBUF_2024-03-14_094217.TXT is the conventional format.
5. Search the buffer for the first occurrence of 16#4548 in the running 12 months; count how many times it appears.
6. For each 16#4548, open the local diagnostic buffer of the named module via HW Config > Online > double-click on the fault icon. If the module is greyed-out (e.g., CP 443-5 EXT in the source case), it is unreachable — the field engineer must escalate to physical inspection.

To make the procedure reproducible after a STOP, enable the Report system errors option in HW Config and configure OB 82, OB 85, OB 86, OB 100, OB 102, OB 121, OB 122 in the project. The CPU's diagnostic buffer clock is buffered; on power-up the time is recalculated from the runtime meter, so always reconcile with a local NTP source on the S7-300/S7-400 PN side if exact timestamps matter.

Rack and Backplane Inspection Procedure

If the diagnostic buffer points to no single module (entries are uniformly distributed across slots, or the same module reappears only intermittently), inspect the mechanical layer:

1. Lockdown power and ESD-safe the rack. Remove the 24 V supply to the PS 405/PS 407 and wait for the bus connectors to fully discharge. Use a wrist strap bonded to the cabinet ground.
2. Photograph the rack with module order, slot numbers, coding keys, and rotary switches before any removal.
3. Pull each module straight out using the front locks. Do not rock or rotate the module — the circular motion is the single largest source of backplane pin damage. A twisted pin looks bent, offset, or pushed down by 0.5 mm relative to its neighbour.
4. Inspect each P bus socket with a magnifier and a torch. The P bus is the upper row of contacts, the K bus the lower row. Check for any pin that is recessed, twisted, missing gold plating, or shorted with a flake of solder.
5. Measure the backplane continuity slot-to-slot. The P bus carries 5 V; an open or shorted line shows up as a voltage drop across the affected slot.
6. Re-seat each module with a firm, straight push until both locks click. Do not use excessive force. Torque the screw locks where the module has them.
7. Restore power, bring the CPU to STOP, and reload the project from the PG to flush any transient SDB mismatch.
8. Run a 24-hour burn-in with OB 1 only and observe the diagnostic buffer for new entries.

PROFIBUS Node Diagnosis and Cleanup

When the diagnostic buffer shows the same DP slave flooding entries (typical when the node is permanently switched off, the segment is open, or a station is missing entirely), proceed as follows:

1. Open HW Config > Online > PROFIBUS to see the live list of reachable slaves. A greyed-out station means the CP 443-5 EXT cannot reach it.
2. For each PROFIBUS segment, check the bus terminator at both physical ends. The terminator must be ON at exactly two points per segment. Withdraw all other terminators.
3. Measure the PROFIBUS A and B line voltages at the suspected station: idle A is 0 V, idle B is +5 V (with the receiving station pulling B). Anything else indicates a shorted bus or a faulty transceiver.
4. Use a PROFIBUS tester (e.g., Softing PROFIusb or the Siemens BT 200) to capture the telegram and check for repeats, gaps, and illegal telegrams.
5. Replace the DP connector at the failing slave (the 9-pin D-sub with switchable terminator, 6ES7972-0BA12-0XA0 for 35°/90° cable exit). Oxidation on the connector pin is a frequent cause of intermittent faults that look "random" in the buffer.

A node that is intentionally switched off must be removed from the PROFIBUS configuration (HW Config > right-click > Delete) or assigned to OB 86. Leaving the node in the configuration with no power causes the diagnostic buffer to flood within hours and consumes one entry per access scan cycle, eventually pushing out the older entries that may have been the real cause.

CP 443-5 Extended Local Diagnostics

When the CP 443-5 EXT (e.g., 6GK7443-5DX03-0XE0 or 6GK7443-5DX04-0XE0) appears greyed-out in HW Config online, the local diagnostic buffer is unreachable over the backplane and must be retrieved differently:

• Open the front-panel service port of the CP (the X2/PG port) with a serial cable and read the buffer via a terminal program (XMODEM-compatible). Service mode is entered by setting the rotary switch to "3" and powering the rack. This is described in the CP 443-5 Extended manual (article number A5E00369635).
• Alternatively, swap the CP with a known-good spare and re-read the buffer from the suspect CP off-line.

Local buffer content is typically:

Entry 1: PROFIBUS DP bus error, station 3, identifier 0xFF
Entry 2: Repeat telegram, station 3, slot 5
Entry 3: DP station 3 timed out after 3 retries
Entry 4: OK (last successful telegram)

Compare the entries across all CPs that show the same fault. Identical entries mean the network is the cause; different entries mean the CP itself is degrading. The CP 443-5 EXT firmware should match the package version used in HW Config (typical versions: V5.x, V6.x, V7.x for the 6GK7443-5DX04). A firmware downgrade is generally not supported; if the project was created with V5.x, the CP must run V5.x or compatible V6.x per the S7-400 compatibility list.

Configuration Hardening — Required OBs to Avoid Escalation to STOP

The most common preventable cause of an I/O error escalating to STOP is a missing OB. The minimum set the engineer must insert into the project for a CPU 416-3 is:

Insert each as an empty FC call (or a block that increments a counter) so the CPU has somewhere to vector the event. The minimum "survivor" OB 85 is shown below in STL:

FUNCTION_BLOCK FB1
// OB 85 / OB 86 / OB 122 stub
// Call: CALL FB1, DB100
L    L#0
T    MD   0   // clear 32-bit scratch

Or, simpler, place BE at the end of each empty OB and download. The CPU will not STOP on the first event when the OB is present; it will still write the entry to the diagnostic buffer for the engineer to review.

Verification Procedure After the Fix

Once the mechanical, electrical, and OB layers are addressed, the engineer must verify the CPU stays in RUN under fault injection:

1. With the plant in maintenance state, power-cycle the CPU 416-3 and confirm a clean startup (event 16#454F for warm restart or 16#454E for cold restart, no further events).
2. Inject a single DP node failure: power-down one ET 200M station. Verify the CPU does NOT STOP, an OB 86 is called, and the diagnostic buffer shows a W#16#4572 entry followed by a W#16#3921 (DP station failure in/out entry).
3. Re-power the station, verify W#16#3971 (DP station return) appears.
4. Pull one central slot module. Verify the CPU does NOT STOP and W#16#4571 is recorded.
5. Simulate a continuous hardware interrupt by shorting an OB 40 source. Verify the CPU does NOT enter STOP via W#16#4949.
6. Let the plant run for 24 hours; export the diagnostic buffer and confirm no W#16#4548 event during the test.

Preventive Maintenance Schedule

For an S7-400 installation approaching 10 years of service, the recommended maintenance cadence is:

Troubleshooting Matrix

FAQ