WinCC V6.0 SP3 Variable Update Freezes: Action Overflow Fix

1. Problem Summary

A WinCC V6.0 SP3 runtime station connected to multiple local and remote OPC servers gradually stops updating analog and digital process values after roughly 3 to 30 days of continuous operation. Updates slow progressively across all OPC channels and tags, the HMI dynamics freeze, and a reboot of the WinCC station temporarily restores service until the next cumulative-runtime threshold is reached. The fault reproduces on both members of a WinCC redundant pair within hours of each other, which rules out a single-point hardware failure on one PC.

The hallmark symptoms reported in the field are:

• Progressive freeze of all tag updates from every OPC channel, not just one.
• Log file entries for AddItem timeouts, generic COM error 80004005, and WinCC action error c0040007.
• Action queue overflow message: "ActionOverflow: more than 5000 Actions to work".
• Tag format errors such as Format error of tag 'TRA_Tasa_tren_A.U'. Format ID: 136314880 = 8200000 (hex).
• Script execution errors of the form ExecuteError in Action @NN.

Each symptom is a single observable of an underlying resource-leak or configuration problem in the WinCC runtime data manager, the action scheduler, or the OPC channel wrapper. The job of the engineer is to read the log, classify the symptom, and apply the correct Siemens KB fix.

2. Affected Environment

WinCC V6.0 SP3 is the last service pack before V6.0 SP4. The action queue and several OPC channel issues were corrected in SP4, so the first action on any new incident is to verify the installed service pack and plan the upgrade path if the system is still on SP3.

3. Diagnostic Data: Log File Forensics

Open the WinCC diagnosis files in the project directory, typically:

The relevant files are WinCC_Sys_01.log, WinCC_Sys_02.log, the *.log files of each OPC channel, and the action runtime log. Each line is comma-delimited in the form:

Below is the mapping of the errors observed in the field to the diagnostic category and the first Siemens KB article to consult.

4. Root Cause Analysis

The progressive freeze across all OPC channels is the visible symptom of three independent defect classes that all manifest over time:

1. Action scheduler saturation (ActionOverflow). The WinCC action worker has a hard queue depth of 5000 pending actions. When a C-action blocks (I/O wait, deadlocked COM call, recursive trigger, or simply a long loop) it consumes a worker slot. Other triggered actions are enqueued. When the queue reaches 5000 entries, WinCC drops the trigger and the action runtime enters degraded mode. New trigger events either pile up or are dropped, and the data manager that fans them out to the picture windows also stalls. After several hours the data manager no longer refreshes the OPC mirror, so the HMI dynamics freeze even though the underlying OPC servers are still healthy.
2. Tag format / format-string corruption. A tag whose configured Format property does not match the value being supplied by the OPC server produces a Format error each scan. The format string is cached, the conversion throws, and the data manager increments an internal error counter. Over weeks the counter and the associated state objects accumulate, and the data manager scan loop slows until it can no longer keep up with the configured acquisition cycle.
3. OPC channel COM failure (80004005 / AddItem timeouts). When a single OPC channel enters COM-failed state (e.g. due to a remote OPC server restart, a DCOM security change, or a slow AddItem call), WinCC keeps the channel open in degraded mode. As more channels accumulate, the shared thread pool used by the data manager is exhausted, and the freeze spreads to all channels.

These three classes are independent, but in practice a long-running SP3 station hits all three. The remediation must therefore fix all three, not only the one that produced the first error in the log.

5. Error Code Reference

6. Diagnostic Tools: APDIAG and GSC Runtime

Two internal WinCC tools give direct visibility into the action scheduler and global script execution. Both must be opened in the running WinCC project, not in WinCC Explorer.

6.1 APDIAG (Action Performance Diagnostics)

1. Open the WinCC project on the runtime PC.
2. From the Start menu, run Start > Simatic > WinCC > Tools > APDIAG (or invoke the executable directly in the WinCC installation directory).
3. Select Info > FirstAction. APDIAG writes a snapshot of the action queue to a text file named stack.txt in the project directory.
4. Open stack.txt. Each entry shows the action ID, the picture name, the trigger (timer / tag / click) and the line number currently executing. The first entry of the snapshot is the action that is blocking the worker and producing the overflow.

If stack.txt is never created, the action runtime is not in a state that can be snapshotted; this is itself diagnostic information and means the scheduler is no longer servicing APDIAG requests. Recover by stopping runtime, fixing the suspected action, and restarting.

6.2 GSC Runtime (Global Script Runtime)

1. On the runtime PC open Start > Simatic > WinCC > Tools > GSC Runtime, or select Smart Object > Application Windows > GSC Runtime inside a WinCC picture.
2. The GSC Runtime window lists every global script action that is currently loaded, with its internal ID.
3. Cross-reference each action ID against the ExecuteError in Action @NN line in the diagnosis log. A logged action ID that does not appear in GSC Runtime means the action failed to load and is silently disabled by WinCC, leaving a triggering tag pointing at a non-existent handler.

7. Remediation: Action Overflow (Siemens KB 2357302)

The action queue is sized in the project configuration, not at runtime. The remediation sequence is:

1. Identify the offending action with APDIAG (Section 6.1).
2. Open the global script C-editor and inspect the action. Common offenders:
   • Recursive triggers: an action that writes the same tag it is triggered on.
   • Synchronous I/O: a C-action that calls printf, MessageBox, file I/O on a slow network share, or a database call to a remote SQL server.
   • Loops without a yield: for (...) loops in a timer-triggered action that can run for more than the trigger interval.
3. Refactor the action so that the trigger returns within one scan. Move any blocking I/O into a background task or wrap it with a watchdog timeout.
4. Reduce the trigger rate of any action that does not need a 250 ms cycle. Most HMI dynamics can be triggered on a 1 s or 2 s cycle without visible degradation.
5. Verify by reloading the project and watching GSC Runtime for the action ID. The ActionOverflow line should disappear from WinCC_Sys_*.log within minutes.

8. Remediation: Tag Format Errors

The format string 8200000 (hex) is the WinCC internal VT_-style code combined with the format-failure bit. The most common user-visible source is a tag whose configured format does not match the OPC-supplied value type, for example a tag configured as Signed 32-bit that the OPC server delivers as Float.

1. Open WinCC Explorer > Tag Management.
2. Locate the tag named in the log, e.g. TRA_Tasa_tren_A.U.
3. Open Properties > Type / Format and verify the data type and the format string match what the OPC server delivers.
4. Re-import the tag from the OPC server, or correct the format string manually, e.g. %d for integer, %f for float, %s for string.
5. Reload the project. The format-error line must not reappear in the channel log.

Run Project > Cross Reference > Tags in Use to find every picture, archive, and script that references the tag, and validate that none of them forces an incompatible cast.

9. Remediation: COM Error 80004005 and AddItem Timeouts

Errors 80004005 and AddItem timeouts on a WinCC OPC channel are almost always DCOM, RPC, or remote-server restart issues. The diagnostic and remediation sequence is:

1. Open the WinCC online help and search for Channel diagnose > Error flag. The full description of 80004005 in the WinCC channel context is documented at Siemens KB 16622456.
2. Open Control Panel > Administrative Tools > Component Services on the WinCC station and on each remote OPC server PC. Validate the DCOM Config > My Computer > Default Properties:

Enable Distributed COM on this computer: ON
Default Authentication Level: Connect
Default Impersonation Level: Identify

3. Add the WinCC runtime user to the local group Distributed COM Users on every remote OPC server PC. Add the OPC server service account to Distributed COM Users on the WinCC station.
4. On the remote OPC server PC, open DCOM Config > OPCServer > Properties > Identity and set The interactive user or This user with the account that owns the OPC server process. Launching user is the most common source of cross-machine 80004005.
5. Open Windows Firewall on the remote OPC server PC and allow the OPC Enum, the OPC Server executable, and the RPC dynamic ports. WinCC OPC requires TCP 135 plus the dynamic RPC range 49152-65535 inbound.
6. In the WinCC OPC channel configuration (OPC > System Parameters), set the AddItem timeout to a value larger than the worst expected remote OPC server startup time; 30 s is a reasonable default for industrial OPC servers.
7. Verify with a third-party OPC client, e.g. OPC DA Client from a separate machine, that the same tags return values. If the third-party client reads the tags cleanly, the channel issue is in WinCC, not the OPC server.

Once a single channel has produced 80004005 it remains in a degraded state until WinCC is restarted. After the DCOM hardening above, restart both members of the redundant pair and verify that the channel log no longer records AddItem timeouts or 80004005.

10. Redundancy Considerations

The user reports that the fault reproduces on both members of the redundant pair within hours of each other. This is consistent with a software fault, not a hardware fault, because both PCs are running the same WinCC project. Three checks apply to the redundant pair:

1. Both partners have the same WinCC version. Mismatched SP levels (e.g. SP3 on one partner, SP4 on the other) are an unsupported configuration. Upgrade both PCs to the same service pack.
2. Action IDs are not duplicated. Global script actions are shared by the project, but the runtime IDs are assigned at project compile. If one partner was edited and reloaded while the other ran the old project, the action ID space diverges and the redundancy handshake logs an error. Always run WinCC Explorer > Redundancy > Synchronize after any project change.
3. Tag Logging archives are independent. When the data manager freezes on the preferred partner, redundancy does not switch partners within seconds; the switch is event-driven. A frozen data manager does not emit switch events, so the standby partner continues running its own copy. Expect both PCs to freeze in the same time window even though only one is preferred.

11. Step-by-Step Recovery Procedure

1. Capture the current Diagnosis folder from both partners and archive it; this is the baseline.
2. Plan and execute the upgrade to WinCC V6.0 SP4 on both partners. Schedule a maintenance window; the upgrade requires a project recompile.
3. On the running project (still SP3 if upgrade is not immediate), open the global script C-editor and audit every triggered action. For each, confirm the trigger tag is not written by the same action, and that no blocking I/O is performed inside the trigger callback.
4. Refactor any action that fails the audit, then reload the project.
5. Open the Tag Management and re-import every OPC tag. Resolve every format error reported in the channel logs.
6. Harden DCOM on every remote OPC server PC per Section 9.
7. Restart the WinCC runtime on both partners and monitor WinCC_Sys_*.log for 24 hours. Any remaining c0040007, 80004005, AddItem timeout, or ActionOverflow line must be investigated before declaring the incident closed.
8. Run the project for at least 14 days and confirm no degradation. The original fault was on the order of 3-30 days, so 14 days of clean operation is a credible but not exhaustive verification.

12. Verification Checklist

13. Frequently Asked Questions