Troubleshooting S7-400H: STOP LED After Plant Trip and Field-Side Spark Event

When an S7-400H redundant PLC pair enters a dual-STOP condition after a plant trip and a simultaneous field-side spark event, the failure path is rarely a single root cause. In most documented cases, three concurrent mechanisms combine: a transient over-voltage event on the plant distribution side, a wiring-induced surge on a 24 V I/O segment, and a residual FORCE condition that prevents one CPU from re-entering RUN after a memory reset. This article builds a field-tested diagnostic matrix, walks through the precise recovery sequence, and explains why a warm restart alone is not a sustainable fix for the standby CPU.

1. S7-400H Redundancy Architecture Recap

The S7-400H (S7-400H stands for "Hochverfügbar" - Highly Available) is a hot-standby redundant system built from two AS 400H stations synchronized over fiber-optic redundancy links. Understanding the master/standby behavior is essential to interpreting the LED pattern described in the original field report.

The combination observed in the field report - both Master and Standby in STOP, then after warm restart, Master RUN and Standby STOP - is a textbook dual-CPU transient followed by a partial recovery. According to the S7-400H Fault-Tolerant Systems Manual, a simultaneous STOP on both CPUs almost always points to one of: a common-mode electrical event, an OB121/OB122 storm, a failed firmware update, or a programmed STOP issued from the engineering station.

2. Symptom Timeline Reconstruction

Breaking the field report into discrete events produces the following fault timeline, which is the framework for the rest of the diagnostic work:

1. High-voltage transient on the plant distribution side trips the upstream breaker and de-energizes the MCC.
2. Plant is restarted; two specific control valves no longer respond to DCS commands.
3. Loops and cable continuity are checked on those two valves - all appear normal.
4. When re-terminating a limit switch on one of the affected valves, a visible spark is drawn from the terminal and the entire DCS layer drops.
5. STOP LED on both Master and Standby CPU is lit solid; RUN LED off on both.
6. Operator performs MRES (Memory Reset) on both CPUs; both restart and DCS comes back, but the two control valves remain unoperable.
7. Operator performs a warm restart; both control valves become operable. Master RUN LED green, Standby STOP LED yellow.
8. Standby CPU does not auto-rejoin the redundant pair; plant runs in degraded (non-redundant) mode.

This sequence is consistent with two independent issues: (a) a corrupted output image on the SM 432 / DO module that controls the two valves, recovered by warm restart, and (b) a standby CPU that has accumulated a fault, a pending FORCE, or a redundancy-link synchronization error that MRES did not clear.

3. Root Cause Hypothesis Matrix

4. Memory Reset (MRES) vs Warm Restart - Operational Differences

Operators frequently confuse these two operations. In an S7-400H system, the difference determines whether a CPU can rejoin a running redundant pair or whether it is permanently excluded.

The Siemens S7-400 CPU manual (entry ID 1117747) documents the precise key sequence for MRES: turn the mode switch to STOP, hold the MRES button for 3 s until the STOP LED flashes slowly, release for 1 s, hold again for 3 s - the STOP LED will flash rapidly. If the operator skips the second 3 s press, the CPU performs a warm restart instead, which does not clear a stale FORCE table.

5. The FORCE Function Hazard in H Systems

FORCE is a STEP 7 / TIA Portal function that overrides process values regardless of the program logic. On a single CPU, FORCE is a commissioning convenience. On an S7-400H, FORCE is a documented hazard.

5.1 Why FORCE on an H System Causes Standby STOP

The two CPUs in an H system maintain a deterministic event-synchronous lock-step on the user program. The standby CPU mirrors the master's program execution cycle by cycle. If the master has FORCE values on a memory bit (for example, M 0.0 = TRUE) and the standby does not, the cyclic data comparison will detect a divergence and force the standby into STOP with diagnostic event "Standby CPU has different data state from master". Conversely, if FORCE was last written by an engineering station that is now connected only to the master, the standby's view of the same memory area can desynchronize.

Siemens has documented this behavior in the Siemens FAQ entry on FORCE with H-CPUs and the PCS 7 H System FAQ. The recovery procedure is: disable FORCE on both CPUs, MRES the affected CPU, and let it re-synchronize from the master.

5.2 How to Identify Active FORCE on an S7-400H

Proceed with the following steps before any other recovery action:

1. Open STEP 7 V5.5 (or TIA Portal V16+ for newer projects) and connect online to the AS 400H station.
2. Right-click the CPU in the project tree and select Monitor/Modify.
3. Switch to the Force view (not the Modify view - they are different).
4. If the table is non-empty, every row shows an 'F' status symbol. Each entry must be deleted explicitly with the trash-can icon - closing STEP 7 does not clear FORCE.
5. Repeat for the partner CPU. Both views must be empty before MRES will result in a clean RUN.

For PCS 7 systems the equivalent path is WinCC Explorer → Tag Management → Force Values or via the CFC online test mode. The PCS 7 operator does not see a FORCE indicator on the OS by default, which is why FORCE-related faults are a frequent cause of "phantom" standby failures.

6. The Spark Event - Electrical Forensics

A spark on a limit-switch terminal during re-termination means one of three things:

• Back-EMF from an inductive load (solenoid, contactor coil) on the same 24 V rail that is still energized elsewhere in the cabinet.
• Stray voltage from a parallel power source - e.g. a separately powered field loop or a 4-wire device with a shared common that is not properly bonded.
• Capacitive charge stored on a long cable that was not discharged before re-termination. A 1 km cable can store enough energy to draw a 5-10 mm spark at 24 V, and substantially more if the loop was, in fact, carrying a higher AC voltage through a fault path.

If the plant tripped on high voltage, the upstream event is almost certainly a switching over-voltage or a lightning-induced surge. The IEC 61000-4-5 surge impulse test exposes exactly this kind of failure mode: a 1 kV line-to-line and 2 kV line-to-ground impulse with 1.2/50 µs waveform. A non-protected I/O input will pass that surge directly into the SM module, which will latch the diagnostic interrupt (OB82) and, if the surge is large enough, damage the input driver stage.

7. SM 432 / Output Module Diagnostics

The two control valves that became unoperable are driven from a digital output module - typically SM 432 (in a 400H system it is more likely SM 322 in a 300-side ET 200M, but the diagnostic principles are identical). The diagnostic procedure is:

1. Open HW Config and identify the slot and order number of the affected module (for example 6ES7 322-1BH01-0AA0).
2. Read the diagnostic buffer in STEP 7: CPU → Module Information → Diagnostic Buffer. Filter for entries with the timestamp of the spark event.
3. Open the module's Module Information → I/O Status and check the per-channel error byte. A '1' in bit 0 (channel fault) or bit 3 (external fault) for a particular channel confirms field-side damage.
4. Physically inspect the module: a damaged channel will often show a darkened PCB near the output driver, a lifted SMD pad, or a ruptured TVS diode. Use a thermal camera with the module under load to find the hot channel.
5. Measure the 24 V supply at the module's L+ terminal. If it sags below 19.2 V (the lower limit of the 24 V DC spec per IEC 61131-2), the module may have latched into a protective shutdown.

8. Step-by-Step Recovery Procedure

The following sequence is the field-tested order of operations for a S7-400H system that has been through a plant trip and a field-side electrical event.

8.1 Pre-Conditions

• Plant is in a safe state, all valve actuators de-energized.
• Upstream power quality is verified: voltage within 110 V AC ± 10 % (or 230 V AC ± 10 %), frequency 50/60 Hz ± 5 %.
• Engineering station (PG) is connected to the AS 400H rack via PROFIBUS or Industrial Ethernet and STEP 7 V5.5 (or TIA Portal) is online with the project.
• You have read-only access to the diagnostic buffer and write access to the S7 connections in NetPro.

8.2 Procedure

1. Open the diagnostic buffer on both CPUs. Save a screenshot of the last 50 events. The first event after the trip usually tells the story.
2. Check for active FORCE on both CPUs (Section 5.2). If FORCE is present, document each forced address, then delete all FORCE entries on both CPUs. Confirm both Force tables are empty.
3. Read the I/O diagnostic data on the affected SM module. If a channel is permanently faulted, the module must be replaced - no amount of MRES will restore a damaged driver.
4. Switch the Master CPU mode selector to STOP → wait 3 s → to MRES → hold 3 s → release 1 s → hold 3 s. The STOP LED will flash rapidly. The CPU will perform a full memory reset and reload from the Flash card / MMC if present.
5. Switch the Master back to RUN. The CPU will go through OB100 (warm restart) and rejoin the H system. RUN LED should be solid green within 30 s for a typical PCS 7 project.
6. Wait for the Standby CPU to detect the master and attempt to re-synchronize. The I/F (interface) LED on both will flash during sync, then go off when the pair is fully redundant. This is the moment of truth: if the standby's I/F LED never stops flashing, the fiber or sync error is still present.
7. If the Standby does not rejoin: repeat step 4 on the Standby CPU. The Standby should pick up the program and data from the master via the redundancy link and rejoin in 1-3 minutes.
8. Validate the two control valves: command them from the DCS, observe the output LED on the SM module, observe the field device. If the output LED toggles but the valve does not move, the fault is mechanical (stuck actuator, lost air supply, broken linkage), not electrical.
9. If the output LED does not toggle, the program is not writing to that output. Inspect the CFC / SCL source for the valve block, check the interconnect, and confirm that the output address in HW Config matches the symbol used in the program.
10. Once the system is fully redundant (Master + Standby both RUN, REDF off), perform a controlled role switch: open SIMATIC Manager → H System → Switch master/standby. The previously standby CPU should take over as master and the original master should drop to standby. This validates the redundancy in both directions.

9. Verification Matrix

10. Permanent Mitigation - Engineering Recommendations

After recovery, the following changes should be applied to prevent recurrence:

1. Install surge protection on every I/O cable that leaves the cabinet. For 24 V DC digital signals, a Phoenix Contact PT-IQ-24 or Weidmüller VPU I 24 is appropriate. For analog signals (4-20 mA), use a Pt-IQ-4-20MA or equivalent. Both terminate the line and provide a status contact for remote monitoring.
2. Verify shield bonding on every I/O cable. The standard rule for S7-400 cabinets is single-ended grounding - ground the shield at the cabinet entry only, leave the field end floating. Use EMC cable glands (e.g. Lapp Skintop MS-SC) to bond the shield 360° at the entry plate.
3. Add a separate 24 V DC distribution rail for the I/O field side, with its own circuit breaker and a Phoenix Contact QUINT UPS or SITOP selectivity module to decouple field transients from the PS 407.
4. Review the PCS 7 / STEP 7 program for missing OBs. At a minimum, OB82 (diagnostic interrupt), OB83 (insert/remove), OB121 (programming error), and OB122 (I/O access error) should be present in the offline program and have a defined behavior (typically a counter increment and a latched alarm). Without OB82, a single module diagnostic event can drop the CPU into STOP.
5. Document a plant-wide FORCE policy. FORCE should be allowed only during commissioning, only with a signed permit, and the FORCE table must be cleared as a mandatory step in the handover procedure. PCS 7 V9 SP3+ supports a FORCE permission bit that can be tied to a user role in WinCC.
6. Perform a controlled role switch of the H system once per quarter, ideally during a planned downtime. A H system that has not role-switched in two years is a H system with an unverified redundancy.

11. FAQ - Frequently Asked Questions