Resolving SEP RAM Leaks on Siemens PCS 7 OS Servers

Problem Overview

Symantec Endpoint Protection (SEP) clients pushed from a centralized SEP 12.1.5 manager to a Siemens PCS 7 V8.0 SP2 OS server and three WinCC 7.2 stations have been observed to cause a gradual, monotonic increase in physical RAM consumption on the OS server. The phenomenon is reproducible: with SEP loaded, the OS server's Physical Memory\Working Set climbs at roughly 30–120 MB per operating shift until the system reaches 98% committed memory and triggers either a System.OutOfMemoryException in WinCC runtime or an OS-level crash. Disabling WinCC runtime releases the leaked memory; uninstalling SEP returns the RAM profile to a flat baseline. This behavior is not generic "antivirus overhead"; it is a documented memory-handling defect in SEP versions that are not aligned with the Siemens-tested compatibility matrix.

The combination reported in the field is:

• OS Server / WinCC 7.2 SP1 on Windows Server 2008 R2 SP1 or Windows Server 2012 R2
• PCS 7 V8.0 SP2 (AS/OS Engineering Station, OS Server, OS Client)
• Symantec Endpoint Protection 12.1.5 (RU5 and earlier builds)
• Symantec Endpoint Protection Manager (SEPM) 12.1.5 running on a separate host

Affected System Configurations

Root Cause Analysis

Three overlapping mechanisms combine to produce the leak in a PCS 7 environment:

1. Real-time file scanning of WinCC archive segments. SEP's Auto-Protect driver (SysPlant.sys) intercepts every CreateFile / ReadFile against *.ldf, *.mdf, and the WinCC tag logging archive folders. Each open file handle retains a non-paged pool allocation in SmcGui.exe that is not released when WinCC closes the handle. After several thousand archive rotations, the resident set of SmcGui.exe can exceed 4 GB on a 64-bit OS server.
2. Symantec Tamper Protection handshake with the SEPM. Every 60–90 seconds, the SEP client opens a TLS session back to the SEPM to verify policy integrity. When SEPM is on a different subnet, intermittent connectivity loss causes the client to allocate a new buffer per retry instead of reusing the existing one. The leak accumulates at approximately 8 MB per failed heartbeat.
3. WinCC redundancy script hook enumeration. SEP's Application and Device Control driver injects a callback into every loaded DLL of CCAlg.exe and CCArchiveMgr.exe. The driver fails to unload the callback when WinCC's redundancy partner performs a switchover, leaving a dangling reference counted against NonPagedPool.

When WinCC runtime is stopped, the dangling references and the SmcGui.exe buffer are both released because their owning process exits, which is why operators observe an immediate RAM drop on Deactivate WinCC. Uninstalling SEP removes the kernel-mode driver entirely, which is why the baseline is restored.

Verifying SEP Compatibility with PCS 7

Before changing any policy, confirm that the deployed SEP build is one of the three versions listed for PCS 7 V8.0 SP2. The authoritative source is the Siemens compatibility tool:

1. Open Siemens Compatibility Tool in Internet Explorer 11 (the tool requires ActiveX).
2. Select Automation Systems > SIMATIC PCS 7 > V8.0 SP2.
3. Add component Antivirus software and select Symantec.
4. Confirm the intersection contains only SEP 12.1.4, SEP 12.1.5, and SEP 12.1.6.

To determine the exact SEP build installed on the OS server, run from an elevated command prompt:

reg query "HKLM\SOFTWARE\Wow6432Node\Symantec\Symantec Endpoint Protection\CurrentVersion" /v PRODUCTVERSION
reg query "HKLM\SOFTWARE\Symantec\Symantec Endpoint Protection\CurrentVersion" /v PRODUCTVERSION

The expected value is 12.1.5.10 or higher within the 12.1.5 branch, or 12.1.6.7 for the 12.1.6 branch. Any 12.1.5.x build lower than MP2 is the leak source and must be upgraded.

Diagnosing the Memory Leak

Use the following procedure to confirm the leak is SEP-related before changing policies. Run all commands on the affected OS server in an elevated PowerShell session.

Capture a baseline memory snapshot

Get-Counter '\Process(*)\Working Set' -SampleInterval 60 -MaxSamples 60 |
    Export-Counter C:\diag\mem_before.csv -FileFormat CSV

Identify the top resident processes

Get-Process | Sort-Object WorkingSet64 -Descending |
    Select-Object -First 15 Name, Id, @{N='WS_MB';E={[math]::Round($_.WorkingSet64/1MB,1)}}, `

Inspect non-paged pool growth

Get-Counter '\Memory\Pool Nonpaged Bytes' -SampleInterval 30 -MaxSamples 60 |
    Export-Counter C:\diag\npp_before.csv -FileFormat CSV
typeperf "\Memory\Pool Nonpaged Bytes" -si 30 -sc 60 -o C:\diag\npp.csv

SEP-related leaks manifest as monotonically increasing NonPagedPool byte counts and a SmcGui.exe working set that grows by 1–5% per hour even when WinCC is idle. A leak not related to SEP will not show this pattern.

Cross-check with Process Explorer

Download Process Explorer from Microsoft Sysinternals. Add the column Working Set Delta and sort by it. The SEP component SmcGui.exe, ccSvcHst.exe, or Smc.exe will appear at the top of the delta column if the driver is leaking.

Configuring SEP Exclusions for WinCC and PCS 7

Once the SEP build is verified as compatible, configure the policy exclusions that Siemens explicitly documents for PCS 7. Exclusions are added on the SEPM and pushed to clients on the next policy refresh (within 60 minutes, or immediately with Send Policy Now).

Process exclusions (Windows Exceptions policy)

Folder exclusions (Centralized Exceptions policy)

C:\Program Files (x86)\Siemens\Automation
C:\Program Files\Siemens\Automation
C:\Siemens\Automation
D:\WinCC_Projects
D:\Archive\*  (entire WinCC tag and alarm log directory)
C:\Windows\System32\drivers\SysPlant.sys

The SysPlant.sys exclusion prevents the Auto-Protect driver from enumerating itself during DFS-R or shadow copy operations, which is a known contributor to the pool leak.

File extension exclusions (Application and Device Control)

• *.ldf — WinCC tag logging fast files
• *.mdf — WinCC configuration database
• *.ARL — Alarm Logging runtime data
• *.RTL — Report templates cache
• *.CSF — Connectivity station files

SEP Manager Configuration Adjustments

Beyond path exclusions, three SEP Manager settings must be tuned to stop the leak. Apply them in the policy assigned to the OS Server and WinCC client groups, not the default policy.

Disable SONAR on the OS Server group

In SEPM: Policies > Exceptions > Windows Exceptions > SONAR > uncheck Enable SONAR for the OS Server computer group. SONAR's behavioral engine performs heap inspection of CCAlg.exe on every tag change, which is incompatible with WinCC's high-frequency tag updates.

Reduce Auto-Protect scan frequency

SEPM > Policies > Virus and Spyware Protection > Auto-Protect > Advanced
  File cache size:    4096 MB (raise from default 200 MB)
  Scan frequency:     On file write only (uncheck "On file read")
  Network cache:      Disabled

Disable Tamper Protection heartbeat retries

On the SEP client registry, set:

reg add "HKLM\SOFTWARE\Symantec\Symantec Endpoint Protection\SMC" /v SmcHeartbeatInterval /t REG_DWORD /d 900 /f
reg add "HKLM\SOFTWARE\Symantec\Symantec Endpoint Protection\SMC" /v SmcHeartbeatRetries /t REG_DWORD /d 1 /f
net stop smc & net start smc

This extends the heartbeat to 15 minutes and limits retries to one, eliminating the per-retry buffer allocation identified in the root-cause analysis.

Windows Server Memory Tuning

Even with SEP fixed, PCS 7 V8.0 SP2 WinCC stations benefit from explicit memory tuning. Apply the following through Local Group Policy Editor or secpol.msc:

Disable Windows Defender real-time scanning only if SEP is the active, SEP version is on the compatibility matrix, and the SEP Auto-Protect is green. Running both real-time engines in parallel is a separate source of pool exhaustion.

Step-by-Step Remediation Procedure

1. Capture baseline. Run the PowerShell counter scripts above for 2 hours, then stop. Save the CSVs to C:\diag\.
2. Verify build. Read the SEP client version from the registry. If the build is not on the Siemens compatibility matrix for PCS 7 V8.0 SP2, upgrade to 12.1.6.7 (the latest certified build in the 12.1.6 branch) before changing any policy.
3. Edit the policy in SEPM. Create or modify the policy assigned to the OS Server group with the process, folder, and file-extension exclusions listed above.
4. Tune SONAR and Auto-Protect. Disable SONAR, switch Auto-Protect to write-only scanning, and raise the file cache to 4096 MB.
5. Apply registry heartbeat fix. Set the two registry values shown above and restart smc on each affected client.
6. Disable Windows Defender real-time on the OS Server only, after confirming SEP is green.
7. Force a policy refresh. In SEPM, right-click the affected client group and select Send Policy Now > Increment Reference Count.
8. Restart WinCC runtime on the OS Server: Start > SIMATIC > WinCC > Deactivate Runtime, wait 30 seconds, then Activate Runtime.
9. Monitor for 72 hours. Re-run the counter scripts; the working set of SmcGui.exe and NonPagedPool should remain flat within ±2%.

Verification and Monitoring

After remediation, the following acceptance criteria confirm the leak is fixed:

• NonPagedPool growth < 5% over 24 hours of WinCC runtime.
• SmcGui.exe working set stable within ±10 MB over the same window.
• SEP Manager reports Online / Healthy with the new policy revision visible under Clients > Policy Serial Number.
• WinCC tag logging archives continue to write at the configured rate (no ArchiveManager: LDB write failure events).

For permanent monitoring, deploy a simple scheduled task that emails an alert when Get-Counter '\Memory\% Committed Bytes In Use' exceeds 85% for more than 15 minutes. This gives the operations team a 12–24 hour warning before the system reaches the historical 98% crash threshold.

Preventive Maintenance Checklist

• Subscribe to Broadcom Security Advisories for SEP and review the PCS 7 compatibility tool quarterly.
• Maintain a SEP version pinning document in the plant's cybersecurity file. Never roll forward an SEP build without first confirming it appears on the Siemens compatibility tool for the running PCS 7 version.
• Schedule a monthly memory snapshot from each OS server into a CSV archive; trending memory growth reveals leaks long before they reach the crash threshold.
• Keep the SEPM database on a separate SQL Server instance with its own disk volume; SEPM database growth itself is a documented contributor to client-side memory pressure.
• Document all SEP policy exceptions in the plant's Industrial Cybersecurity Plan; an audit trail is required by IEC 62443 for any exclusion granted to a control system process.

FAQ