Troubleshooting PCS 7 V8.1 WinCC Client Freeze and CP1623 Master-Mode Switch

1. Problem Summary

Operator stations in a SIMATIC PCS 7 V8.1 plant running WinCC V7.3 + Update 1 with a redundant OS server pair and four connected clients experienced a progressive freeze. Faceplates stopped opening, and several block icons began rendering with the standard "hatched" status while other icons on the same graphic remained live. The fault cleared only after Server1 was closed and the clients were forced to switch to Server2. During the same window the "CP1623 switched to master mode" diagnostic alarm was logged. The visible impact lasted roughly 7 minutes, but the underlying loss of availability in a process plant is the actual risk.

2. System Environment & Configuration

The reported stack is:

• Engineering / runtime: SIMATIC PCS 7 V8.1 (no SP at the time of the incident).
• SCADA: SIMATIC WinCC Explorer V7.3 + Update 1 (recommended later path: V7.3 + Update 19).
• Topology: 1 × redundant OS server pair (Server1 + Server2) with WinCC Redundancy.
• Operator clients: 4 × WinCC clients, each holding the standard OS_Client project role.
• Connection to automation systems (AS): SIMATIC NET CP1623 (catalog number 6GK1162-3AA00) installed in the OS servers.
• Network: Industrial Ethernet / PROFINET with the S7 protocol used between OS server and AS.

The CP1623 is a SIMATIC NET IE PCI Express card that provides S7 communication, PROFINET IO controller / device, and time synchronization to PCS 7 OS servers. It is the normal termination of the S7 connection for a WinCC server and therefore any CP1623 redundancy or fault event is visible directly on the WinCC server and propagated to every dependent client.

3. Symptom Decomposition

Three sub-symptoms are reported and each points to a different layer of the stack.

3.1 Client-side freeze, sequential, 7-minute window

Clients did not crash — they froze. This is typical of a stalled S7 connection attempt: WinCC continues to render, but the data manager is waiting for a server response. Sequential freezing of four clients is the expected cascade when a single WinCC server process or its network connection is degraded: each client times out slightly after the previous one. A 7-minute duration aligns with the WinCC redundancy monitoring interval plus the application timeout values (see Section 7).

3.2 Hatching of PID and Valve block icons while PT icons stayed normal

In PCS 7 WinCC, "hatching" of a block icon means the value / status update is lost. The display in the WinCC picture is driven by internal tags that mirror the AS-side block state. PT blocks in PCS 7 are typically configured with the standard @PCT faceplate and rely on the cyclic Process tag driver connection. PID and motor-operated valve blocks additionally use the control-relevant Status / Command tag pair, which is sent on a different S7 connection than the raw process value. If only the latter go to "Connection disrupted" while raw measurements keep updating, the most likely root cause is loss of a specific S7 connection (or a specific application connection on the CP1623), not loss of the physical link.

3.3 "CP1623 switched to master mode" alarm

This alarm is generated by the SIMATIC NET redundancy layer on the OS server. In a redundant CP1623 configuration the two cards run in hot-standby; the active one is "master". A switchover implies the previous master either failed, was disconnected, or lost the keep-alive. Even a single CP1623 (no paired card) raises this status text when the soft-PLC redundancy agent re-evaluates the connection state to the AS. Combined with the 7-minute client freeze, it is the most important diagnostic message in the incident.

4. Root Cause Hypotheses

Ranked by probability in a PCS 7 V8.1 / WinCC V7.3 + Upd1 plant:

5. Diagnostic Data Collection (SIMATIC Assessment Suite – Data Collector)

Before any code change, collect a full diagnostic snapshot with the official Siemens tool. The current name is SIMATIC Assessment Suite - Data Collector (SAS-DC); the legacy name SIMATIC Diagnostics Tool (SDT) refers to the same toolset and is still referenced in older entries.

1. Download the latest SAS-DC from the Siemens support entry ID 65976201. The entry also documents the executable and the supported Windows / PCS 7 versions.
2. Run SAS-DC on every OS server and on every client that participated in the incident. The tool produces a ZIP per computer.
3. In the project directory, also export the WinCC redundancy log (<WinCC Project>\Redundancy\*.log), the WinCC_Server_<ComputerName>_<timestamp>.log from Documents\Siemens\Automation\Logs\Install and the SIMATIC NET diagnostic files from %ProgramData%\Siemens\Automation\SIMATIC_NET.
4. From the AS, export the diagnostic buffer (Step 7 / TIA Portal: "PLC → Save Service Data...") and include the S7 connection list (netpro export or SFC RD_DIAG outputs).
5. Collect the OS server Windows event logs (eventvwr), in particular Application, System and the SIMATIC Diagnostic Agent log.
6. Open a Siemens support request via Technical Support and attach every ZIP. The combination of server, client and AS data is what the support engineer needs to discriminate between hypotheses 1, 3 and 4.

6. Step-by-Step Resolution Procedure

Apply the steps below in order. Each step is reversible.

6.1 Stabilize the plant

1. Confirm Server2 is fully synchronized (Redundancy Control Center shows "Standby" for Server2 with all partners reachable).
2. If Server1 has not already been closed, stop the WinCC Explorer service: net stop "WinCC Explorer". The redundant server will pick up the active role within the configured MonitorTime window.
3. Allow clients to re-establish. Verify in the client runtime that block icons return to normal coloring within 60 s.

6.2 Inspect CP1623 / SIMATIC NET redundancy on the failed server

1. Open SIMATIC NET Configuration Console on Server1. Check that the CP1623 entries for the OS-AS connections (typically named S7-OS-AS1...) show "OK" and are not in "Standby (down)" or "Fault".
2. Open PC Station Configuration via Station Configuration Editor. Verify the CP1623 module is assigned to index 1 and that its assigned PC interface (e.g. Intel(R) PRO/1000 EB Ethernet) shows link state "Up".
3. Look at PROFINET diagnostics: any Station failure, Connection abort, or Media redundancy role change event within the 7-minute window is a direct cause candidate.
4. If the CP1623 is paired (redundant), check that the standby CP is in Hot Standby, not Cold Standby. A cold-standby partner has to renegotiate the S7 connection on switchover and explains the 7-minute hang.

6.3 WinCC redundancy parameters

Open the WinCC project on the master server, navigate to Server Data → Redundancy and verify the following parameters against the default and your plant design:

If the active values deviate from the recommendations, align them on both servers and re-activate WinCC.

6.4 AS-side S7 connection count

PCS 7 V8.1 limits the number of OS connections per AS. With four clients + two servers you can reach the limit if the project was duplicated during commissioning. On the AS open HW Config, count the S7 connection entries under the CPU and verify the number against the CPU's published maximum (e.g. AS 410 with a 416 CPU: 64 OS connections). Remove or consolidate unused connections.

6.5 Apply the official update path

The recommended long-term fix is to upgrade to PCS 7 V8.1 SP1 with WinCC V7.3 + Update 19. The WinCC update alone is also valid as a hot-fix. The Update 19 readme lists several corrections relevant to the symptom set described here, including S7 connection stability and redundancy switchover behavior. The upgrade is non-disruptive for the AS side if the OS is switched to the standby server first.

7. WinCC Redundancy & Time Parameters in Detail

The 7-minute window is the sum of three timers:

• WinCC client failover: the client tries the preferred server, then the secondary server, then re-tries. The MaxConnectionRetries × ConnectionTimeout product sets the upper bound.
• WinCC server redundancy switchover: the standby server waits for MonitorTime before declaring the partner dead. The default is 60 s.
• CP1623 re-initialization: the new master CP has to bring the S7 connections up, which typically takes 30–60 s per AS connection.

7 minutes ≈ 6 × (MonitorTime + CP bring-up) when multiple AS connections are involved. Reducing MonitorTime to 30 s and pre-establishing connections at startup cuts the window to ~3 minutes. The trade-off is the risk of false-positive switchover on a single network blip.

8. CP1623 Redundancy Configuration Reference

To read the manual, browse the Siemens Industry Online Support catalog for 6GK1162-3AA00 and download the device manual. The same family includes the CP1628 which is often deployed in newer OS server builds.

9. WinCC Block Icon Hatching Reference

When a PCS 7 block icon shows the hatched pattern, identify the source by reading the associated status message in the WinCC Alarm Control:

Combine this with the alarm log entry "CP1623 switched to master mode" and the diagnostic conclusion is: at least one S7 connection between Server1 and at least one AS was lost on the primary CP, and the secondary CP took over. While the secondary CP was re-establishing the connection, the affected block icons hatched.

10. Verification Checklist

After the fix, run the following acceptance test:

1. Stop Server2 (preferred) and confirm Server1 takes over within 60 s without any hatched icon on the four clients.
2. Re-start Server2, wait for synchronization, then stop Server1 and repeat.
3. Pull the network cable of the primary CP1623 on the active server and confirm the secondary CP takes over within 30 s and no icon hatches for more than 5 s.
4. Generate a synthetic AS-side load (start 10 SFCs in parallel) and verify the client cycle time stays below 1 s in APLOG.
5. Run the SAS-DC collector again and compare the new log set with the old one — there should be no new "Connection abort" entries.

11. Long-Term Hardening & Update Strategy

• Keep the OS server Windows install lean: disable Windows indexing on the WinCC project drives, exclude the project folder from real-time virus scanning.
• Pin the WinCC and PCS 7 versions: do not mix V8.1 OS servers with V8.2 clients in the same plant. The mixed-version combinations are supported only within the compatibility matrix published with each PCS 7 version.
• Schedule a yearly SAS-DC run and archive the ZIPs in a versioned folder. The drift in the log set is the best indicator of an upcoming failure.
• Update to the highest service pack + update supported by your AS firmware before commissioning a new process cell. For PCS 7 V8.1 this is V8.1 SP1 + WinCC V7.3 + Update 19.
• Document the redundancy behavior in your plant's O&M manual: switchover time, expected client reaction, alarms to expect (CP1623 master, redundancy status). The operator on shift must know that a 30-second icon hatch during a switchover is normal.

12. Frequently Asked Questions