S7-414H Redundancy Solo Mode: Program Mismatch Troubleshooting

Overview: Siemens S7-400H Fault-Tolerant Architecture

The Siemens SIMATIC S7-400H is a high-availability fault-tolerant PLC platform built around two synchronized CPUs (CPU 0 master, CPU 1 standby) interconnected by fiber-optic synchronization links. Both CPUs execute the user program in lockstep and exchange event/state data through redundant sync modules (e.g., 6ES7 960-1AA04-0XA0 sync sub-modules on the 417-4H/414-4H). When one CPU fails, detects an error, or loses its sync partner, the other takes over the process without interruption (bumpless switchover).

When the redundancy manager detects that the two CPUs can no longer remain in synchronized operation, the system degrades to Solo mode: only one CPU actively controls the I/O, while the other enters STOP or remains in a passive state. The system no longer tolerates a CPU failure until the cause is cleared and resynchronization is achieved. One of the most common — and most easily misdiagnosed — root causes of an unexpected transition to Solo mode is a program mismatch between the two CPUs' non-volatile memory (Flash EPROM / MMC / RAM load memory).

Problem Description: Unexpected Solo Mode After a Plant Trip

The reported fault pattern is as follows:

• The S7-414H system was running in redundant (duplicated) mode before the trip event.
• During the trip, an automatic CPU switchover occurred (the standby CPU took over the master role).
• After the trip, only one CPU remains in RUN; the second CPU will not come back up in sync, and the H system stays in Solo mode.
• STEP 7 / PCS 7 hardware diagnostics reports: "There is a mismatch in the program" between the two CPUs.
• Standard remediation steps (re-download of the project, full restart) do not resolve the issue.
• Removing the Flash EPROM from both CPUs and downloading the project backup (to RAM) restores normal redundant operation.
• Reinserting the Flash EPROM causes the Solo mode fault to return.

This pattern is diagnostic of a load-memory content divergence between CPU 0 and CPU 1. The two CPUs are not running identical binaries, and the redundancy link cannot bring them into agreement.

Root Cause Analysis: Program/Load-Memory Mismatch

An S7-400H CPU holds the executable project in three logical locations:

1. RAM load memory (volatile): active running image used for program execution.
2. Work memory (code/data): compiled blocks consumed by the CPU at runtime.
3. Non-volatile load memory (Flash EPROM / MMC): persistent backup that is reloaded into RAM after power-cycle or MRES.

For redundant operation, both CPUs must contain identical blocks in work memory, identical configuration, and identical non-volatile load memory contents. The CPU-to-CPU link-up protocol compares the project signatures; any deviation triggers a redundancy fault and forces the system into Solo mode.

Typical reasons the load memories diverge after a trip:

• Write-once / partial flash update: the Flash EPROM write was interrupted (e.g., the trip event coincided with a download-in-progress), so only one CPU's flash was updated, or both were updated but with different project revisions.
• Inconsistent project state in PG/ES: the project downloaded to CPU 0 was a newer revision than the one downloaded to CPU 1 (e.g., CPU 1's flash was never refreshed after a small change).
• Different runtime options / system data blocks (SDB): one CPU's SDB set diverged (e.g., due to online edits applied to only one side, which is unsupported in H systems).
• CRC / checksum mismatch in flash: rare but possible if the Flash EPROM has a bad block; the corrupted CPU falls back to STOP and refuses to sync.

The user's observation that "once the flash is removed the system runs fine, but reinserting it triggers Solo mode again" is the classic smoking gun: the runtime image (RAM) is correct, but the persistent image (flash) is wrong — and the link-up procedure prefers the flash content as the source of truth for project identity.

Diagnostic Steps: Confirm the Mismatch

1. Open the H-station online in STEP 7 (HW Config). Both CPUs should be visible. If only one is online, mark both rack slots and use PLC > Download to Target…> Configuration to confirm reachability.
2. Read the diagnostic buffer of both CPUs.
   • Select the rack > CPU 0 > PLC > Diagnostic/Setting > Diagnostic Buffer.
   • Repeat for CPU 1.
   • Look for entries such as: "Redundancy partner not in sync," "Program comparison error," "Different project on partner," or OB 70 / OB 72 / OB 80 / OB 101 / OB 102 references.
3. Use the H-specific diagnostic OBs. Insert temporary diagnostic logic in the user program to evaluate the system state SFCs:
   • SFC 90 "H_CTRL" – H-system control
   • SFC 51 "RDSYSST" with SSL ID W#16#0132 – H-system status
   • SFC 51 with SSL ID W#16#0222 – redundant partner status
4. Compare project signatures. In HW Config > Station > Properties > Diagnostic, the Plant Identification / Project Signature field on each CPU must match exactly. A mismatch confirms the divergence.
5. Online block comparison. Use PLC > Compare Blocks (Online/Offline) against both CPUs; any block with status different or only on partner is the source.

Diagnostic indicator	Meaning	Where to find
"Program mismatch" in HW Config	Different project signature on CPU 0 vs CPU 1	HW Config > CPU properties > Diagnostic
Diagnostic buffer entry "Redundancy lost"	Sync link or project identity failure	CPU > Diagnostic Buffer
System state LED on standby CPU	STOP with blinking or solid	CPU front panel
SF / BF3 / IFM LED on sync modules	Sync link failure (separate cause)	Sync sub-module front panel
SFC 51 SSL 0x0132 returns mode = 2	Solo mode confirmed	User program / online watch

Step-by-Step Resolution Procedure

Follow this sequence to clear the mismatch and restore redundant operation. Adjust slot numbers to match the actual H-station layout.

Prerequisites

• STEP 7 V5.5 / V5.6 (or PCS 7) installed with the S7-400H option package.
• The current offline project backup that matches the running process (verified against the last successful commissioning).
• Identical Flash EPROM / MMC cards for both CPUs (same type, same firmware compatibility, e.g., 6ES7 952-1AH00-0AA0 or equivalent for the 414-4H).
• Access to the rack to physically remove/insert cards if needed.

Procedure

1. Stop both CPUs in a controlled way. From the master CPU: PLC > Stop. Confirm CPU 1 also transitions to STOP via the redundancy link.
2. Open the SIMATIC Manager and the offline project. Verify the project signature inside the project matches the one currently in the plant record.
3. Perform a full download to both CPUs (to RAM load memory).
   • Mark CPU 0 in HW Config > PLC > Download to Target… > select CPU 0 > OK > confirm Download to RAM only (uncheck Download to Flash).
   • Switch the online target to CPU 1 and repeat.
4. Start both CPUs. Use PLC > Run/Restart on CPU 0 (master). The H-system should now perform a link-up; CPU 1 will pass through the OB 100 (restart) and synchronize.
5. Confirm the system is in redundant (not solo) mode.
   • HW Config online > CPU 0 properties > Operating mode should read RUN (redundant).
   • CPU 1 should read RUN (redundant, standby).
   • Read SFC 51 SSL 0x0132: a return value of 1 = redundant, 2 = solo.
6. Re-flash the non-volatile memory only after redundant operation is confirmed.
   • From the master CPU: PLC > Download to Target… > select CPU 0 > OK > check Download to Flash.
   • Wait for the flash write to complete (CPU STOP/RUN LED behavior per firmware — typically RUN is briefly interrupted or LED flashes rapidly; do not power-cycle).
   • Repeat for CPU 1.
7. Verify the flash write. In HW Config, check both CPUs' Diagnostic tab — the project signature, timestamp, and checksum must be identical.
Order of operations matters. Always download to RAM first, confirm redundant RUN, and only then re-flash. Flashing first, or flashing a mismatched image, re-introduces the same fault and is the most common cause of the reported symptom returning.

Flash EPROM / MMC Handling Best Practices

The user-side behavior of physically removing the Flash EPROM to recover from Solo mode is a known workaround, but it is not a permanent solution. The recommended long-term approach is:

• Always update the flash of both CPUs in the same online session. If using a serial/MPI/Ethernet download from a single PG, the upload to CPU 0 and CPU 1 must be back-to-back and the process must be allowed to finish before the system is restarted.
• Avoid online editing of the S7-400H from a PG that is only connected to one CPU. Online edits are not automatically replicated to the partner; this is the most common hidden source of divergence after a long service life.
• After every change, re-run the Compare Blocks Online/Offline on both CPUs before signing off the change.
• Keep a dated MMC / Flash backup per CPU in the project documentation. Label them with the project signature and timestamp.
• Never power-cycle a S7-400H during a flash write. A partial flash will leave the card in an inconsistent state and may even brick it; in that case the CPU falls back to RAM and refuses to synchronize.

S7-1500H Reference: OB 70 I/O Redundancy Error

On the newer S7-1500H / S7-1500HF platform, the I/O redundancy error OB is supported from firmware V3.0. Per the official TIA Portal documentation, I/O redundancy error OB (OB 70) is invoked when a redundancy error occurs on a switched or shared I/O device. While this OB is platform-specific to S7-1500H, the same diagnostic principle applies to S7-400H: an OB-driven handler (typically OB 70 / OB 72 / OB 80 in PCS 7) is the right place to capture the event, log it to the operator station, and trigger an alarm to the maintenance team.

On S7-400H, the equivalent OB is OB 72 (CPU redundancy error). Insert it into the program so that you can record the start information:

ORGANIZATION_BLOCK OB72
TITLE = "CPU Redundancy Error"
// OB 72 is called by the S7-400H system when the redundancy
// status changes (loss of partner, switchover, link failure).
VAR_TEMP
    info : ARRAY[0..19] OF BYTE;   // 20 bytes of start info
END_VAR
BEGIN
    // Read diagnostic info, archive, raise operator alarm
    // via SFC 17 / SFC 18 to the OS
    ...
END_ORGANIZATION_BLOCK

Forcing OB 72 to raise a maintenance message via the OS (PCS 7 WinCC) is the cleanest way to shorten the time-to-diagnose on the next occurrence.

Verification: Confirm Redundant Operation

After completing the resolution procedure, perform these checks:

1. Operator station: the redundancy status indicator in WinCC / PCS 7 OS shows Redundant (not Solo). No maintenance alarm from OB 72.
2. Diagnostic buffer of both CPUs: entries indicate a clean link-up and a clean redundancy transition, with no "program mismatch" or "CRC error" entries.
3. Forced switchover test: from STEP 7, on the master CPU, run PLC > H-Systems > Stop Standby CPU and then start it again. The standby should re-sync within seconds and both CPUs should remain in RUN. Repeat in the opposite direction (stop master) to confirm bump-less switchover.
4. Field test: pull the fiber-optic sync cable from CPU 0; CPU 1 should take over without process interruption. Reconnect and confirm automatic resynchronization.
5. Read-back check: compare the offline project, the contents of CPU 0 flash, and the contents of CPU 1 flash. All three must show the same project signature.

Troubleshooting Matrix

Symptom	Likely cause	First action
Solo mode after trip; "program mismatch"	Load memory divergence between CPU 0 and CPU 1	Re-download to RAM on both, then re-flash
Standby CPU in STOP, no link-up	Sync sub-module fault or wrong sync mode	Check BF3 / IFM LEDs; verify both sync modules are configured identically
Both CPUs STOP after power-on	Bad / partial flash content from interrupted write	Remove flash, re-download from PG, run from RAM
OB 72 fires repeatedly	Persistent hardware fault, not a memory issue	Read OB 72 start info; inspect rack / IM / sync link
CRC error in diagnostic buffer	Flash EPROM wear / bad block	Replace flash card; re-download project
Solo mode but both CPUs RUN	Redundancy manager de-qualified the system	Check 1756-RM equivalent (S7-400H: redundancy status in SFC 51); review qual criteria

Preventive Maintenance

• Schedule a quarterly read-back of the project signature from both CPUs and from the offline backup to detect silent drift.
• Restrict online editing of an H-station to a single, controlled change window, with the project being downloaded to both CPUs in a single PG session.
• Keep the STEP 7 project version under formal change control. Whenever the project signature changes, archive the previous flash cards and label them with the timestamp.
• Replace Flash EPROM cards at the manufacturer's recommended write-cycle limit (typically 100k–1M writes depending on series). A worn card will start producing intermittent CRC errors that look like program mismatches.
• Train the maintenance crew to read the OB 72 / OB 100 / OB 102 diagnostic buffer entries — most Solo-mode investigations can be closed in minutes if the buffer is read first.

FAQ

Why does removing the Flash EPROM fix the Solo mode fault on the S7-414H?

When the flash is removed, the CPU boots from RAM load memory only. If the in-RAM image on both CPUs is identical, the redundancy link-up succeeds. The flash card was the source of the project-signature divergence — either the card itself held a corrupted or outdated image, or it had been partially written during a previous interrupted download.

What diagnostic buffer entry confirms a "program mismatch" on S7-400H?

Look for entries of the form "Different project on partner CPU" or "Redundancy: project comparison failed" with a CRC mismatch on the load memory. They appear in the diagnostic buffer of the master CPU at the moment link-up is attempted after the trip.

Can I fix the mismatch by re-downloading only one CPU?

No. Always download the same project (same offline database, same build) to both CPU 0 and CPU 1 in a single session. A single-CPU download is the most common reason the fault returns immediately after a service intervention.

Which OB should I monitor for redundancy events on S7-400H?

OB 72 (CPU redundancy error) is the dedicated OB for H-system events such as loss of the standby partner, master switchover, and redundancy link-up failure. On S7-1500H, the equivalent is OB 70 (I/O redundancy error), supported from firmware V3.0 per Siemens TIA Portal documentation.

Is the issue related to the sync modules or the program memory?

Sync-module faults (BF3 / IFM LEDs, fiber-optic link loss) produce a different diagnostic pattern — typically a sync-link-down entry and immediate switchover, not a "program mismatch" message. The program-mismatch symptom reported here is a load-memory identity issue, not a sync-link issue.

How do I verify the system has returned to fully redundant operation after the fix?

In HW Config online, both CPUs must show RUN, the system must read Redundant (not Solo), and a forced switchover test (stop the standby, then stop the master) must complete without a process trip. The offline project, the in-RAM image, and the flash card content on both CPUs must all show the same project signature.