Troubleshooting WinCC v6 SP4 Redundant Server Communication

1. Problem Overview

Intermittent communication loss in WinCC v6 SP4 redundant server installations manifests as one or more clients silently losing visibility of substation process data after several days of continuous operation. A client reboot (and frequently a redundant server pair restart) restores the connection, but the failure recurs on a different client after a further interval. This behavior is characteristic of resource exhaustion, connection-table saturation, or licensing gaps inside the WinCC server runtime rather than physical layer faults, because the network itself remains operational for the surviving clients.

The configuration in question runs the following hardware and software stack:

2. Symptom Patterns Extracted from WinCC Diagnostic Logs

WinCC v6 emits per-component log files inside the project's \<ProjectName>\<ServerName>\Diagnostics folder. The relevant files for the issue described are WinCC_Server_0_<date>.log, APL_<date>.log, ArchiveManager.log, and the channel-specific S5TCP* / PES_API* traces. The recurring event classes that map to this symptom set are listed below.

The two most informative lines are the Confirmation timeout and the AWAIT_CLOSE iNAResp=16 messages. They indicate that the S5 ISO-transport connection from the server to a CP-1430 TF did not respond to a keep-alive TPDU inside the configured timeout window. WinCC closes the handle (OpenRef) and re-opens a new one, but the recycled OpenRef numbers grow monotonically until the per-process table fills and new requests start failing with "No Request Object available for confirmation".

3. Root-Cause Tree for Intermittent Loss of Substation Data

The behavior "client C1 stops seeing substation 3" while C2–C6 continue operating isolates the problem to the path WinCC server → CP-1613 → Ethernet switch → CP-1430 TF on substation 3. Once that path is established, the candidates collapse to five root-cause families. Each is presented with the diagnostic evidence that proves or disproves it.

3.1 CP-1613 / CP-1430 TF connection-table exhaustion

Industrial Ethernet on Windows uses the Siemens PES_API (Process Event-driven Socket API) layer. Each S5 connection allocates a request object and an open reference (OpenRef). WinCC v6 SP4 fixes a known issue where the request pool was not released after StartDisconnect if a Confirmation timeout had already torn the link down asynchronously. The patch is delivered via hotfix WinCC_V6_SP4_HF7 (or later) on the Siemens support portal.

3.2 C-Script saturation

WinCC v6 evaluates C-Actions on every tag change. With 6 substations, alarm logging enabled, and "a lot of C-Script intensive customised functions", the scripting engine can be starved of execution time. The Variable <Tag> timeout events (ID 1007006) confirm that the script thread blocks long enough that the channel DLL can no longer meet its 2 s default confirmation timer. Increase the S5 ISO-Transport confirmation timer from 2 s to 6 s, and refactor C-Scripts to remove polling patterns.

3.3 Archive Manager licence gap

The diagnostic trace from the S7/Archive manager reads:

License   installed  needed  to install  Product
SI--A9WTG1----  No   Yes   Yes   WinCC Option Archive 1500 ArchiveTags

When the 1500 tag archive license is not installed, the archive subsystem falls back to an evaluation mode that periodically drops the active archive segment. The Failed to switch archive (Type:2):(null) ! line is the direct symptom. Without a working archive, runtime memory pressure grows because alarm and trend buffers cannot be flushed to disk. The fix is to install the missing license, then reset the archive path.

3.4 RAM ceiling on 32-bit Windows Server 2003

The servers report only 2 GB physical RAM. WinCC v6 with redundancy and option archive under Windows Server 2003 R2 (32-bit) has an effective user-mode virtual address space of 2 GB, of which the runtime reserves up to 1.2 GB for tag buffer, picture cache, and connection tables. A redundant pair therefore needs at minimum 4 GB physical per server, with /3GB or /PAE boot switches to remap the kernel pool.

3.5 Switched (non-cross-over) redundancy link

The redundancy link between S1 and S2 must be a dedicated crossover connection or a private VLAN pair with no Spanning Tree enabled. When run through the same managed switch that carries PLC traffic, BPDU/TCN storms, IGMP snooping mis-configuration, or simply broadcast storm control can drop the redundancy heartbeat. WinCC v6 will demote the standby if the heartbeat is lost for more than HBTIMEOUT seconds, and the demoted server tears down all client connections in a single batch.

4. Diagnostic Procedure

Execute the following steps in order, capturing the result of each before proceeding. The full procedure should be repeated on both S1 and S2.

1. Open WinCC Explorer → Server Data → Diagnostics Folders and export the last 14 days of logs for both servers.
2. Run grep -E "Confirmation timeout|AWAIT_CLOSE|No Request Object" *.log and count the number of distinct OpenRef values. If the maximum OpenRef value exceeds 0x4000, the request table has wrapped and the connection handler is leaking handles.
3. Open Start → Programs &right SIMATIC → WinCC → Tools → Channel Diagnosis. For each S5 connection, capture the "Status", "Last Error", and "Outstanding Requests" columns. Any connection with > 50 outstanding requests is a candidate for timeout.
4. From the command prompt on the server, run perfmon /res and add the counters Memory → Available Bytes, Process → Private Bytes (WinCC_Server.exe), and Process → Handle Count (WinCC_Server.exe). Log for 24 h. A private-bytes growth greater than 50 MB per day indicates a C-Script or archive leak.
5. On the switch, enable port mirroring on the ports that carry CP-1430 TF traffic, then capture 30 minutes of traffic with Wireshark using the s5_iso dissector. Filter for s5_iso.dt == 0xf0 (TPDU CR/CC) and s5_iso.dt == 0x80 (TPDU DR). Excessive DR PDUs without matching CC confirms that the server is initiating disconnects.
6. Open the project in WinCC Configuration Studio, expand Graphics and search for any reference to notvalid.PDL. Replace the missing picture reference with a valid .pdl or delete the variable trigger.
7. Verify the archive license with Start → Programs → SIMATIC → WinCC → License Analysis (Automation License Manager V5.x). Install SI--A9WTG1 or an equivalent archive option license on both servers.
8. Add the boot switch /3GB /USERVA=2800 to boot.ini on both servers, then reboot, provided physical RAM is upgraded to at least 4 GB.
9. Re-seat the CP-1613 in a different PCI slot, preferably one that does not share an IRQ with the RAID controller. A shared IRQ raises the latency of the PES_API completion routine above the 1 ms threshold, which surfaces as a confirmation timeout under load.
10. Apply Siemens hotfix WinCC_V6_SP4_HF7 or later, then re-test.

5. Step-by-Step Resolution

5.1 Reconfigure the S5 ISO-Transport connection parameters

In WinCC Explorer → Tag Management → SIMATIC S5 ISO Transport → CP-1613, open the connection to substation 3 and adjust the following parameters:

5.2 Install the missing archive license

Use the Automation License Manager (ALM) to install the SI--A9WTG1 license file. After installation, run the following in the project context to rebuild the archive catalog:

ccarcsvr.exe -rebuild <ProjectName> -server <ServerName> -type 2

Restart the WinCC runtime on both servers in the order standby first, then master to avoid a forced failover.

5.3 Increase redundancy heartbeat resilience

In the WinCC project, open Computer → Properties → Redundancy:

This allows the redundancy layer to ride through a 1-minute broadcast storm on the plant LAN without triggering an unnecessary failover that tears down client connections.

5.4 Replace the C-Script polling patterns

C-Actions such as the one that generated ExecuteError in Action @ee9 should be replaced by direct tag triggers using the Standard Dynamics wizard. Where C-Script must remain, the following structure avoids blocking the script thread:

// Correct pattern: non-blocking C-Action on tag change
#include "apdefap.h"
void OnTagChange(const char* lpszTagName)
{
    double dVal = GetTagDouble(lpszTagName);
    if (dVal > 95.0) {
        SetTagWord("ALARM_LIMIT_EXCEEDED", 1);
    }
    return;
}

Do not call printf, MessageBox, or any blocking COM call from inside a C-Action. Move long-running logic into a triggered Global Script action with a minimum cycle of 5 s.

5.5 Apply the WinCC v6 SP4 hotfix

Hotfix WinCC_V6_SP4_HF7 corrects a memory leak in the PES_API layer that resulted in increasing OpenRef numbers and a request-table exhaustion after 4–7 days of operation. The symptom exactly matches the failure pattern of "one client drops a substation every few days". Apply the hotfix on both servers, then monitor the Process → Handle Count (WinCC_Server.exe) counter to confirm that the leak is closed (target: < 12,000 handles after 30 days).

6. Verification Procedure

After applying the fixes, the installation is considered stable only when the following checks pass for a continuous 14-day observation period.

1. No new Confirmation timeout events in any S5 channel log. Verify with findstr /C:"Confirmation timeout" "%ProgramFiles%\Siemens\WinCC\Diagnostics\"*.log.
2. No new AWAIT_CLOSE events. Same command, search for AWAIT_CLOSE.
3. OpenRef values stay below 0x2000. Capture the maximum OpenRef with the channel diagnosis tool daily.
4. Archive switch events are normal. A single archive switched event per day at the configured segment boundary is expected; anything more indicates a license or path problem.
5. WinCC_Server.exe private bytes stable within ± 5% of the 24-h mean.
6. Redundancy role unchanged for 14 days. Open WinCC Explorer → Redundancy on the master and verify the partner state is OK.
7. All six clients can navigate to all six substation pictures with no Picture not found (notvalid.PDL) events in the PDLRuntimeSystem log.
8. Simulated failover test: disable the network adapter on the master. The standby should take over within the configured Master / Standby switch time and all six clients should reconnect automatically within 60 s without requiring a local restart.

7. Long-Term Stability Measures

• Schedule a monthly service window to export the diagnostic folder to a network share. WinCC v6 does not rotate the logs automatically beyond the default 7-day window; a longer retention period is essential for post-incident analysis.
• Configure Windows Server 2003 R2 to deliver automatic updates for the operating system, but exclude the WinCC services from being recycled by the OS update agent. Use a maintenance window that runs outside of the production shift.
• Replace the CP-1613 with a CP-1623 or CP-1628 if a PCI-Express slot is available. The CP-1613 is a 32-bit PCI card and is sensitive to IRQ sharing on the Primergy RX300 S3 mainboard.
• Plan a phased migration to WinCC 7.5 or TIA Portal WinCC Professional. The TIA Portal V20 release notes for WinCC describe the communication options that can replace the legacy S5 ISO-Transport channel via a routing gateway. The Siemens TIA Portal V20 readme at docs.tia.siemens.cloud lists the supported HMI connections.
• Capture a baseline Wireshark trace for each substation connection after stabilisation. Store the trace alongside the project backup so that future regression incidents can be diffed against a known-good capture.
• Enable the WinCC Performance Monitor for a 30-day baseline and archive the resulting .rdb file. This is the only built-in tool that resolves tag-side and channel-side latency in the same timeline.

8. Troubleshooting Matrix

9. Field-Proven Cautions

• Never run the WinCC service under a user account that is also a domain administrator. The LSASS token size on Windows Server 2003 R2 grows to more than 8 KB under heavy GPO load, which causes the WinCC_Server.exe heap allocator to fragment and surfaces as a Confirmation timeout after several days of operation.
• Do not run third-party anti-virus real-time scanning on the WinCC project directory. The *.pdl, *.log, and *.ldf files are written in append mode; scanner interception on the file handle can stall the runtime for several hundred milliseconds.
• Avoid placing the redundancy link on the same VLAN as the plant traffic. A single Siemens HMI client that broadcasts 256 ARPs per second on a saturated segment can defeat the default 120 s partner timeout.
• The Siemens Hotline recommendation to change from Windows Server 2003 R2 to plain Windows Server 2003 is incorrect for this symptom pattern. The actual root cause is the request-table leak fixed in WinCC_V6_SP4_HF7. Reverting the OS will not address the leak and removes a critical patch baseline.
• The CP-1430 TF firmware cannot be upgraded from a PC. The firmware lives in a soldered EPROM on the module and is replaced by Siemens as a board-level repair only. The "IC chip" concern raised during the discussion is correct: any firmware change is an RMA action, not a software install.