Troubleshooting PCS7 WinCC Redundant Server Trend Value Mismatch

Problem Overview

On Siemens PCS 7 V6.1 SP3 (and the follow-on V6.1 SP4 / V8.2 branches), a redundant WinCC server pair frequently exhibits a subtle but production-critical fault: tag values shown in the trend display diverge between the Master and Standby servers at exactly the same time stamp, particularly during process surges or high-rate transients. Operators see one value on Server A's trend and a different value on Server B's trend for the same process variable, same time, same tag. The discrepancy is not a slow drift — it is a sample-by-sample disagreement that can mislead control room decisions during upset conditions.

The symptom is independent of the HMI client used to view the trend. Both server-local trends and WinCC Client/Web Navigator clients reproduce the mismatch because the values originate at the server archive, not at the client.

System Architecture Reference

PCS 7 V6.1 SP3 redundancy is built on WinCC V6.2 (AS/OS-Engineering). The typical layout is:

• One AS pair — in this case two S7-400H CPUs (6ES7414-2Hxxx) with redundant fiber-optic sync modules.
• One OS server pair — Server 1 (Master) and Server 2 (Standby) running WinCC with the Redundancy option (WinCC/Redundancy for PCS 7).
• One or more OS clients (WinCC Client) for operator visualization.
• A plant bus (Industrial Ethernet / PROFINET) carrying the S7 protocol between AS and OS, and a terminal bus for OS-to-OS and OS-to-client communication.
• A dedicated synchronization link between the two OS servers — historically a serial null-modem cable on COM1, later a TCP/IP redundancy link over the terminal bus.

Root Cause Analysis

Trend-value mismatch is a data-consistency defect, not a display defect. In PCS 7 V6.1 SP3, the fault is most often traceable to one or more of the following:

1. WinCC archive synchronization latency on transient events. When the AS delivers a burst of value changes, the Tag Logging runtime on the Master writes samples to its local archive and forwards the update frame to the Standby. If the inter-server sync link (COM cable at 115.2 kbit/s or low-priority TCP on a shared terminal bus) cannot drain the frame buffer before the next sample arrives, the Standby's archive slot for that timestamp is left empty and is filled with the next received value — producing an offset of one sample that visually looks like a mismatch.
2. Time-base drift between Master and Standby. If the Standby's WinCC Time Sync sub-tick drifts by more than one scan period relative to the Master, the timestamp written into the Standby archive is not bit-identical to the Master's timestamp, and the trends cannot overlay.
3. Asymmetric S7 connection priorities. If one server is connected to AS 414-2H #1 and the other to AS #2 with different configured fetch priorities, the read cycles can interleave differently on each server, producing per-server sample sequences that look inconsistent even when the underlying process is identical.
4. Hotfix defect — the principal root cause in V6.1 SP3. Siemens identified a Tag Logging redundancy replication defect in the original V6.1 SP3 build that caused archive frame coalescing on transient events. The fix is delivered in Hotfix 31 of PCS 7 V6.1 SP4.

Pre-Diagnostic Checklist

Before applying any patch, isolate the failure mode. The four checks below eliminate 80% of field-reported cases.

Time Synchronization Configuration

WinCC Time Synchronization is enabled per computer through WinCC Explorer → Computer → Properties → Time Synchronization. In a PCS 7 redundant server pair, the typical pattern is:

• Designate Server 1 as the Master for the OS layer; it receives the time from the AS 414-2H via the plant bus (PC station time = PLC time, option "Access point 1: S7ONLINE").
• Configure Server 2 as Slave; it receives the time from Server 1 through the synchronization link or through the terminal bus.
• Clients synchronize from the Master OS server (Server 1) only.

Verify on the Standby:

WinCC Explorer > Computer > Properties > Time Synchronization
  [x] Synchronize via terminal bus
  Slave / Master role:    Slave
  Access point:           S7ONLINE  (for receiving from Server 1)
  Cyclic sync interval:   10 s  (default; reduce to 1 s for tightest trends)

Archive Synchronization

WinCC Redundancy (option WinCC/Redundancy) maintains two parallel archive data streams on Master and Standby. Replication parameters are tuned under:

WinCC Explorer > Computer > Properties > Redundancy
  Master/Standby:               Auto
  Sync data of Tag Logging:     Yes
  Sync data of Alarm Logging:   Yes
  Switchover time on failure:   30 s   (default 25 s; raise to 30 s for slow disks)
  Sync tag value after n polls: 1

The critical parameter on transient events is the Tag Logging sync buffer. In V6.1 SP3 the buffer was undersized for surge rates above ~500 value changes/s. Hotfix 31 enlarges the buffer and re-sequences frames so the Standby receives the same N:1 mapping between time and value that the Master wrote.

COM / Network Sync Cable Verification

The historical sync link is a DB-9 null-modem cable on COM1 of each server, fixed baud 115 200, no hardware flow control. Verify:

1. Cable is a true null-modem (pins 2-3 crossed, 4-5 crossed, 6-20 crossed, shield grounded on one end only). Do not use a straight-through serial cable.
2. BIOS COM port is enabled and assigned to a non-shared IRQ (typically IRQ 4 / IO 3F8).
3. In Device Manager, the COM port is reported without a yellow bang; UART is 16550-compatible or better.
4. No USB-to-serial adapter is in the path — these drop characters under high interrupt load and are a known cause of trend mismatch.

On newer deployments (V6.1 SP4 with HF31 or later, and V8.x) the serial link is replaced by a TCP/IP redundancy link over the terminal bus using RedundancyControl.exe. Recommended settings:

Sync port:            5000 (default)
Heartbeat interval:   1 s
Timeout:              5 s
Bind to terminal bus adapter only (never to all interfaces)

WinCC Redundancy Configuration

Open the redundancy properties on both servers and confirm symmetric configuration. Asymmetric configuration is the most common human error in redundant deployments.

Hotfix 31 — PCS 7 V6.1 SP4

Siemens Hotfix 31 for PCS 7 V6.1 SP4 addresses the Tag Logging redundancy frame coalescing defect that produces sample-by-sample trend mismatch on transient events. Hotfix 31 supersedes earlier hotfixes 1-30 and is cumulative.

1. Identify the exact build string: WinCC Explorer → Help → About. The vulnerable build is V6.1 SP3 with HF 1-25; the corrected build is V6.1 SP4 with Hotfix 31 (or any later cumulative HF).
2. Request Hotfix 31 from Siemens Customer Support under the customer's valid SSP (Siemens Solution Partner) or directly from Siemens Industry Online Support. Cite entry ID 6ES7658-2BX06-1YA5 and the project's OS image number.
3. Schedule a maintenance window. Both servers must be patched. The standby is patched first, then the role is manually switched and the new standby is patched. Runtime is interrupted for ~20 min per server.
4. Run the install with the Computer name service stopped. Hotfix 31 re-registers the CCDBCL.dll and LibraryManager.dll components.
5. After install, verify the build string shows HF31 in the About dialog and the WinCC service starts cleanly.

PCS 7 V8.2 Considerations

A follow-on thread from the same user community reports the same symptom appearing again under PCS 7 V8.2. The V8.x code path is different — it uses WinCC V7.x runtime and benefits from the architectural changes in WinCC Redundancy. The recommended corrective steps in V8.2 are:

• Install the latest cumulative update for V8.2 (SP1 or later as of writing). V8.x ships with the redundancy frame-coalescing fix already integrated; the failure on V8.2 in the linked case was traceable to non-cumulative hotfixes being applied out of order on Server 1 and Server 2, producing an asymmetric runtime DLL set.
• Verify both servers carry an identical WinCC build string via Help → About → Installed Software. Hash the CCRedundancyCtrl.exe and CCDBCL.dll on both servers; they must match.
• If the project migrates from V8.0 to V8.2, re-run the OS Project Editor on both servers and re-compile the OS. The migration can leave the Tag Logging redundancy database in a pre-V8.2 state on one server.

For TIA Portal-based V17+ projects using WinCC Runtime Professional, redundancy is re-architected around the TIA Portal model. The TIA Portal documentation explicitly enumerates the failure scenarios (partner failure, network failure, runtime stop, time-of-day shift) and the recovery behavior — see System behavior in the event of a fault (RT Professional).

S7-400H Integration

CPU 414-2H redundancy is independent of OS redundancy, but the two interact through the S7 connection. Confirm:

• Both OS servers have an S7 connection to both H-CPUs (rack 0, slot 3 in the typical layout). Symmetric connections are required so either OS can survive an H-CPU switch.
• The configured fetch priority in WinCC Explorer → Tag Management → SIMATIC S7 PROTOCOL SUITE → Connections matches on both servers. Asymmetric priority causes the fetch thread to alternate between CPUs differently on each OS, producing the same observable mismatch even when the archive stream is healthy.

Verification Procedure

1. Static check. On both OS servers, run WinCC > Tools > Project Duplicator > Compare or diff the ProjectName.mcp hash. They must be identical.
2. Time drift check. Add an internal tag @TimeSyncOffsetMs driven by a C-script that returns the absolute difference between the two servers' tick counts each second. Trend it. Acceptable envelope: ±200 ms steady, ±500 ms during switchover.
3. Surge replay. Force a controlled surge using the AS simulation: ramp a fast tag from 0 to 100 over 2 s, hold, return. The Master and Standby trends for this tag must overlay within 1 sample interval.
4. Archive consistency. In WinCC Explorer, Tag Logging → Archive Configuration → Properties → Redundancy, click Compare Archives. The tool reports any time-stamp / value pairs that differ. Empty result = pass.
5. Switchover test. Pull the plant-bus cable on the Master. The Standby must become Master within the configured switchover time, the redundant connection must re-establish, and the new Standby's archive must catch up to the new Master without sample loss.

Prevention and Best Practices

• Patch both OS servers on the same day, from the same downloaded hotfix, after re-hashing the downloaded file. Out-of-sync patch levels are the single largest cause of V8.x trend mismatch in the field.
• Keep a 1 Gbit/s copper or fiber terminal bus dedicated to the OS layer. Do not multiplex SCADA traffic with office VLANs.
• Replace legacy COM-cable sync with the TCP/IP sync link on any new deployment or on any system past V6.1 SP3.
• Run the OS Project Editor after any AS or OS firmware change. The editor propagates the new connection priorities that prevent the asymmetric-fetch issue documented above.
• Schedule an annual "redundancy health audit" that performs the surge replay and archive-compare checks above. Document the trend overlay in the audit record.

Field-Proven Caveats

• If a USB-to-serial adapter was retrofitted to a V6.1 SP3 server, the symptom reappears intermittently. Replace with a native COM port or move to TCP sync.
• If the Standby's archive disk is significantly slower than the Master's (e.g. mixed SSD/HDD), the write-side queue fills first on the Standby. The symptom is then a "the Standby is consistently one sample behind the Master". The fix is disk symmetry, not a hotfix.
• If the trend display is via Web Navigator and the mismatch only appears in the browser, the cause is the Web Navigator cache, not the server archive. Clear Cache and Runtime data in the Web Navigator client config and re-test.

FAQ