Resolving S7-400H Failover Failure from Missing OB88 Error OB

Resolving S7-400H Failover to STOP from a Missing OB88

An S7-400H with CPU 414-5H was operating normally after pre-commissioning redundancy tests. After a CPU0 failure (all LEDs off, PS and CP modules in the same rack still healthy), CPU1 did not take over as the new master. It entered STOP and refused to return to RUN, even after toggling the mode selector. Only a memory reset (MRES) brought CPU1 back online. The root cause turned out to be a missing OB 88 in the program. Loading the OB corrected the behavior. This article documents the architecture, the required fault-tolerant OB set, the diagnostic procedure, and the recommended commissioning checks so that the same failure mode is avoided in other S7-400H systems.

1. S7-400H Redundancy Architecture Overview

The S7-400H is a fault-tolerant PLC platform based on two redundant CPUs running in parallel. The standard configuration pairs two CPUs of the same type (for example CPU 414-5H and CPU 414-5H, or two CPU 417-5H) mounted in separate racks (RACK0 and RACK1). The two subsystems communicate over fiber-optic redundancy links using the IF-966 / IM-153 or the integrated sync modules, depending on the rack layout.

Operating modes supported by S7-400H:

• Standalone — only one CPU is active.
• Redundant — both CPUs are in RUN and execute the user program in lock-step. Both are masters of their respective sub-process image.
• Hot-standby — both CPUs are in RUN but the partner is the backup. Only one CPU is actively controlling the I/O.

When one CPU fails, the partner must detect the failure, take over the process, and continue executing the user program without interruption. The detection and switching logic is implemented in the firmware, but the error response is controlled by the user program via organization blocks (OBs). If a required OB is missing, the CPU's default response is to enter STOP — which is exactly the behavior observed in this case.

2. Failover Sequence and Mode Transitions

When CPU0 is lost, the expected sequence on CPU1 is:

1. CPU1 detects the partner's failure via the redundancy link loss / heartbeat.
2. CPU1 promotes itself to Master.
3. The user program continues running; OB 72 (CPU redundancy error) is called on CPU1 to notify the application of the partner loss.
4. If other events coincide with the failover (rack failure of the partner, communication loss, etc.), additional OBs may be triggered.

If the firmware needs to call an OB that is not loaded in the project, the CPU enters STOP. This is a safety feature: the system prefers a controlled stop with a diagnostic entry rather than executing an undefined error path. The reported incident — RUN → STOP after toggle, but RUN possible only after MRES — is the textbook symptom of a missing error OB.

3. The Role of Error OBs in Fault-Tolerant Operation

OBs are the interface between the CPU firmware (operating system) and the user program. For each error class the firmware attempts to call a specific OB. If the OB exists in the project, control is transferred there. If it does not exist, the CPU enters STOP (or RUN depending on the configuration) and writes an entry to the diagnostic buffer.

In a redundant S7-400H configuration, several OBs are mandatory because the firmware will call them during normal failover and during background self-tests. The complete list of OBs that must be present in the project for a fully functional S7-400H is documented in the SIMATIC S7-400H Fault-Tolerant Systems manual and is summarized in the table below.

4. Required OB Set for S7-400H CPUs

The exact set depends on the configured hardware (PROFIBUS DP, PROFINET IO, redundant CPs, F-modules, etc.), but for a fault-tolerant configuration that uses the standard rack layout with PROFINET/DP and at least one H-sync link, the recommended baseline is:

5. Case Study: Why OB 88 Was the Trigger

The reported incident is consistent with the following chain of events:

1. CPU0 failed hard (all LEDs off, PS and CP in the same rack still alive).
2. CPU1 detected the partner loss and started the switchover logic.
3. During the switchover the firmware attempted to update the process image and ran into an OB that required OB 88 as its escalation path. Common scenarios are:
   • An OB 85 or OB 86 was triggered because the partner's I/O update could not complete (rack failure / obstruction).
   • Another OB was already active when a second error occurred; the firmware needs OB 88 to handle the nesting.
4. Because OB 88 was not in the program, CPU1 entered STOP.
5. The mode selector toggle (RUN → STOP → RUN) re-issued the same internal state, so the CPU went straight back to STOP.
6. A memory reset (MRES) cleared the pending error state and the diagnostic buffer, allowing RUN without the original trigger being present.

Once OB 88 was added to the project and downloaded, the next failover event was handled cleanly. This matches the documented behavior of OB 88: "OB is not loaded or not available, and OB call is not possible (obstruction of the rack/segment)". In an H system the partner rack's obstruction is a regular event during failover, which is why OB 88 cannot be omitted.

6. Diagnostic Buffer Analysis Procedure

If the diagnostic buffer had been retrievable in the incident, it would have shown an entry similar to the following (text from the Siemens manual, paraphrased):

Event 1 of 10: 16:42:13.456  2019-09-12
  OB not loaded (OB 88) - rack failure during OB execution
  Requested OB:   OB 72 (CPU redundancy error)
  Operating mode: RUN (switching)
  Priority class: 25
  Event ID: 0x4548

Procedure to retrieve the diagnostic buffer:

1. Connect the engineering station to CPU1 via MPI/TCP-PROFIBUS/Ethernet.
2. Open SIMATIC Manager (Step 7 V5.x) or TIA Portal, go to Online > Accessible Nodes.
3. Right-click the CPU > CPU Messages > Diagnostic Buffer (or Online & Diagnostics in TIA).
4. Sort by date/time descending. The first event after the failover attempt is the root cause.
5. Look for entries containing "OB not loaded", "rack failure", "obstruction", "redundancy loss", or the fault pattern 0x45xx / 0x88xx.

If both CPUs are accessible after the incident, the buffers should be compared. In an H system, the partner CPU often contains a different failure cascade — for example a sync-link error from CPU0 that can clarify whether the loss was a hard CPU fault, a power event, or a watchdog reset.

7. Step-by-Step: Adding Missing OBs in STEP 7 (Classic)

For legacy S7-400H systems still programmed with SIMATIC Manager / STEP 7 V5.5 / V5.6:

Prerequisites:

• Online connection to CPU1 (and CPU0 once recovered).
• Project source archive (offline program) with the same order number as the running AS.
• The current online program can be downloaded via CPU > Upload Station to PG if no offline project exists.

Procedure:

1. In SIMATIC Manager, open the S7 program of CPU1.
2. Right-click the Blocks folder > Insert New Object > Organization Block.
3. Select OB 88 (Processing aborted / rack failure). Choose STL, LAD, or FBD as the source language.
4. At minimum, populate it with the local error variables so they can be inspected:

   ORGANIZATION_BLOCK OB 88
   TITLE = "Rack Failure - OB not loaded"
   VERSION : 0.1
   VAR_TEMP
     OB88_EV_CLASS   : BYTE;   //16#13 = "incoming", 16#11 = "outgoing"
     OB88_FLT_ID     : BYTE;   // Fault identifier
     OB88_PRIORITY   : WORD;   // Priority class
     OB88_OB_NUMBR   : BYTE;   // OB 88
     OB88_IO_FLAG    : BYTE;
     OB88_FLT_REG    : DWORD;  // Fault code
     OB88_Z1         : WORD;   // Rack number or block number
     OB88_Z2         : WORD;   // Multiplier / further info
     OB88_RESERVED_1 : DWORD;
     OB88_RESERVED_2 : DWORD;
   END_VAR
   BEGIN
     // Log to instance DB / flag area for later upload
     SET;
     SAVE;
   END_ORGANIZATION_BLOCK

5. Repeat steps 2-4 for OB 73, OB 84, OB 90 if they are also missing.
6. Compile the S7 program and check for errors.
7. Download only the new blocks to CPU1 (and CPU0 once recovered) via PLC > Download. In an H system, use Download to target system on the master, then accept the prompt to download to the standby CPU.
8. Watch the diagnostic buffer for a successful download entry.

Verification:

• Open CPU > Accessible Nodes > Blocks on the online view and confirm OB 70, 72, 73, 80, 81, 82, 83, 84, 85, 86, 87, 88, 90 are present on both CPUs.
• Compare the integrity check (block checksum) between RACK0 and RACK1. Use PLC > Compare Blocks Online/Offline or the H diagnostic panel.

8. Step-by-Step: Adding Missing OBs in TIA Portal (H-system)

For newer S7-400H projects programmed with TIA Portal (V15.1 and later, with the S7-400H add-on):

1. Open the project. Expand PLC_1 > Program blocks.
2. Double-click Add new block. In the dialog, select Organization block and choose OB 88.
3. Repeat for OB 73, OB 84, OB 90 if missing.
4. For each OB, set the language and add a minimal body (empty STL block is acceptable; TIA Portal will accept it as long as the OB exists).
5. Compile the project (Compile > Hardware and Software).
6. Go online with the H-station. Use Download to device > Software (only changes). In the dialog, ensure Download to both CPUs is selected.
7. After the download, verify with Online & diagnostics > Diagnostic buffer that no error remains.

9. Should OB 73, OB 84, OB 90 Be Loaded?

The follow-up question from the field is whether OB 73, OB 84, and OB 90 are also required. The answer depends on the configured I/O and on the safety philosophy:

10. Pre-Commissioning Redundancy Verification

After every program change, perform the following checks. These match the Siemens commissioning checklist for S7-400H.

1. Block consistency: Verify that RACK0 and RACK1 contain identical S7 programs. Use Compare Blocks in SIMATIC Manager or the Compare function in TIA Portal. Pay special attention to OBs and to F-library blocks (if present).
2. Memory card checksum: If a Flash card is used, the overall checksum must match between RACK0 and RACK1. Different checksums indicate that one CPU has a stale or partially updated image.
3. Hardware consistency: All main components (CPU, PS, CP, IM, Ethernet / PROFINET modules, redundancy sync modules) must be of identical order number and firmware version.
4. Redundancy role assignment: Use HW Config to confirm that RACK0 is the master and RACK1 is the standby (or vice versa as designed). A wrong assignment can lead to ambiguous switchover behavior.
5. Network settings: For PROFINET interfaces, set the connection to Full duplex with a fixed 100 Mbps or 1 Gbps. Auto-negotiation can cause intermittent link drops that mask as redundancy failures.
6. Diagnostic buffer review: Both CPUs must show a clean startup sequence with no "OB not loaded" entries.
7. Forced failover test: Power down CPU0; CPU1 must take over within the configured switchover time (typically 100 ms to 1 s). Repeat several times.
8. Restore test: Power up CPU0; verify that CPU0 joins as the standby and that no STOP/START cycle occurs.

11. Troubleshooting Matrix

12. Field Notes and Lessons Learned

• Always check OBs first. When an H-system shows STOP on failover, the diagnostic buffer almost always points to a missing OB. Loading it is faster than chasing hardware.
• MRES is not a fix. A memory reset clears the immediate state but leaves the program defect in place. The next event will repeat the failure.
• Empty OBs are acceptable. An empty STL block with the correct OB number is sufficient to keep the firmware happy. The error variables in OB 88 are useful for logging to a flag or DB so that the next diagnostic-buffer dump contains the root cause.
• Document the OB baseline in the program header. Adding a comment block listing all OBs that must be present in every H-system project prevents the issue from recurring when a new programmer removes "unused" blocks.
• Verify both CPUs online. A block present on CPU0 but not on CPU1 (or vice versa) will create a non-deterministic failover. Both must be in sync before signing off commissioning.
• Capture the diagnostic buffer at the first opportunity. In an H-system the window between failover and memory reset is short, but the buffer is the only authoritative source of what the firmware tried to do.

13. Related Standards and References

For deeper background, consult the Siemens SIMATIC S7-400H Fault-Tolerant Systems manual, the Function Manual System and Standard Functions for S7-400H, and the entry "System redundancy on S7-400H" in the TIA Portal manual collection. The ResearchGate document Diagnostic of Fault-tolerant System S7-400H gives an overview of how the firmware maps error conditions to OBs.

For redundancy configurations that involve PROFINET, also review the application guidelines for PROFINET system redundancy (S2 redundancy mode) and for ET 200SP HA / ET 200MP HA. The S7-400H chapter on system redundancy also specifies that PROFINET interfaces must be configured for Full duplex, with the same fixed speed on both sides. Auto-negotiation is not allowed on the redundancy link.

14. Summary of the Fix

The S7-400H 414-5H system stopped during failover because OB 88 was not loaded into the program. The firmware attempted to escalate the partner-loss event into OB 88, found it missing, and stopped CPU1. After loading OB 88 — and as a precaution, OB 73, OB 84, and OB 90 — the failover works as designed. The fix is reproducible on any H-system: load the full S7-400H OB baseline, verify block consistency between the two CPUs, and always capture the diagnostic buffer before clearing the fault.

FAQ