Resolving PCS 7 OS Server Automatic Reboot on Redundant V8/V10

Problem Overview: PCS 7 OS Server Automatic Reboot on Redundant Systems

The SIMATIC PCS 7 OS Server is a Windows-based station that hosts the WinCC Runtime, the alarm and archive services, and the redundancy logic that maintains a hot-standby partner server. On a redundant PCS 7 plant, two OS Servers operate as a Master/Standby pair and synchronize project data, message archives, and process value archives over a dedicated redundant link. When the Master fails, the Standby assumes control within a configurable failover window (typically a few seconds) and the affected Master is brought back to a synchronized state so it can resume the partner role.

"Automatic reboot" can mean two very different things, and the distinction must be made before any diagnostic work begins:

1. Operator-initiated restart through a WinCC popup - the OS generates a "Server will be restarted" dialog that the operator confirms, or that times out unattended.
2. Hard, unannounced restart - the operating system powers off and reboots without any visible application prompts, equivalent to a power-button press.

The Siemens PCS 7 High Availability manual, Chapter 7.13 ("Automatic restart after failure"), explicitly documents the second scenario: when a redundant OS Server fails, the network adapters for terminal bus and plant bus are deactivated, and an automatic restart can be triggered on the affected server if the corresponding option is enabled. This is built-in, expected behavior - not a fault - and must be distinguished from a hardware- or software-induced fault reboot. Reference: SIMATIC PCS 7 High Availability Process Control Systems (V10.0) and PCS 7 High Availability (V9.0).

This article catalogs every documented cause of unplanned PCS 7 OS Server reboots on V8.x, V9.0/V9.1, and V10.0 redundant systems and gives a reproducible investigation procedure ending in a verifiable root cause.

Root Cause Categories and Decision Matrix

The table below groups the seven most common reboot triggers. Each row maps the symptom, the most useful log source, and the typical Event Viewer ID you will see on the affected server.

Investigating Windows Event Viewer Logs

Open Event Viewer (eventvwr.msc) on the affected OS Server immediately. The four logs that contain reboot evidence are:

• Windows Logs > System: Kernel-Power (41), Event log service (6005 boot, 6008 unexpected shutdown, 6009 shutdown startup)
• Windows Logs > Application: Application Error (1000), Application Hang (1005), Windows Error Reporting (1001)
• Applications and Services Logs > Microsoft > Windows > UpdateOrchestrator: Reboot scheduling
• Setup log: WindowsUpdateClient events when a patch initiated the shutdown

Run this PowerShell snippet to retrieve the ten most recent shutdown sources:

Get-WinEvent -FilterHashtable @{LogName='System';Id=1074,6008,6009,41,1001} -MaxEvents 10 |
  Select-Object TimeCreated,Id,ProviderName,Message |
  Format-List

For Event ID 41 (Kernel-Power) you must inspect the BugcheckCode and BugcheckParameter1..4 fields to identify the failing driver. Common BSOD codes seen on PCS 7 stations include:

If a minidump was produced, copy C:\Windows\Minidump\*.dmp to a workstation and analyze with !analyze -v in WinDbg, or upload to the Microsoft WinDbg toolchain for symbol-resolved stack traces. To enable complete memory dumps on the next crash, configure:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\CrashControl
  CrashDumpEnabled = 1      (REG_DWORD; 1=complete, 2=kernel, 3=small)
  DumpFile        = %SystemRoot%\MEMORY.DMP
  AutoReboot      = 0       (REG_DWORD; set to 0 during investigation)

Setting AutoReboot = 0 causes the server to display a BSOD and stop, preserving the dump for later analysis rather than rebooting and overwriting the dump file with boot-time memory contents.

PCS 7 High Availability Fail-Over Behavior (Manual Chapter 7.13)

The PCS 7 High Availability documentation describes an explicit automatic restart behavior that is enabled by configuration, not a fault:

Network adapter for terminal bus and redundant link are deactivated. If automatic restart is activated, it is executed on the server affected by the failure.

Source: SIMATIC PCS 7 High Availability Process Control Systems (V10.0) and the equivalent V9.0 entry at PCS 7 High Availability (V9.0). The architecture is summarized below.

To find this setting:

1. Open WinCC Explorer on the OS Server.
2. Right-click the OS project name > Properties > tab Redundancy.
3. Locate the "Automatic restart after failure" checkbox. By default on PCS 7 V9.0 and V10.0 this is enabled.
4. If you want the operator popup only (no auto-reboot), uncheck it; the reboot then requires manual confirmation.

If the WinCC popup appears and times out without operator intervention, the server reboots unattended. The default timeout is 300 s (5 min). To extend or disable the timeout, modify the registry key:

HKEY_LOCAL_MACHINE\SOFTWARE\Siemens\WinCC\Redundancy
  RestartTimeoutSec = 600   (REG_DWORD, decimal, seconds)

The same redundancy logic also writes entries to the file C:\Program Files (x86)\Siemens\Automation\WinCC\Diagnose\Redundancy.LOG. The two key messages to look for are:

• Partner lost - the Standby has not received a heartbeat within the partner watchdog interval (default 5 s).
• Master/Standby switchover completed - the Standby has taken over as the new Master.

After verifying this is the intended HA behavior, no further corrective action is needed - the reboot is doing exactly what the configuration specifies. However, document the failover event, the time to recover, and whether any operator action was required; this data feeds the PCS 7 availability KPI calculation defined in the V9.0 and V10.0 HA manuals.

UPS Shutdown Signal and Application Shutdown Sequence

Most PCS 7 OS Servers are deployed with an APC, Eaton, Vertiv, or Siemens SITOP UPS providing conditioned power and graceful shutdown on battery exhaustion. The shutdown software (APC PowerChute Business Edition, Eaton Intelligent Power Manager, SITOP UPS1600 with SIMATIC PCS 7 UPS-Connect) issues a Windows shutdown /r /t 0 command once battery capacity falls below the configured threshold. After AC power is restored, the server powers on if the BIOS "Restore on AC Power Loss" is set to "Power On" or "Last State".

Symptoms of UPS-induced reboots:

• Event ID 1074 in System log with User = NT AUTHORITY\SYSTEM and Process = C:\Program Files\APC\PowerChute Business Edition\agent.exe (or equivalent from the vendor)
• Multiple restarts clustered during a known power event in the building
• UPS software event log showing "Low battery shutdown initiated"
• UPS management card (AP9641, Network-MS) reporting Battery Discharged events at the reboot timestamp

Investigation steps:

1. Open the UPS management console (PowerChute, IPM, etc.) and review the last 30 days of events.
2. Cross-reference the building's BMS or facility log for the timestamp.
3. Check the UPS battery health; replacement interval is typically 3-5 years at 25 °C. Use the UPS self-test or vendor battery management to read the runtime vs. design runtime.
4. Confirm the shutdown threshold is set long enough for the OS to safely close WinCC archives (recommend ≥ 5 minutes runtime on battery before initiating shutdown).
5. Verify the UPS signaling cable is USB or RS-232 (not Ethernet-managed only) - if only network-managed, the shutdown command requires the agent service to be running.

To prevent the UPS from issuing a shutdown, while accepting the protection it provides, change the action from "Shutdown" to "Hibernate" or extend the runtime threshold. APC PowerChute configuration path: Configuration > Shutdown > Shutdown Criteria > Battery Capacity. For Eaton Intelligent Power Manager: Devices > Settings > Actions > On Battery Low.

Windows Update Forced Reboot Behavior

Windows Update can restart a server even when the Group Policy "No auto-restart with logged-on users" is enabled, because the WinCC Runtime services (and similar long-running processes) do not block the maintenance window indefinitely. Microsoft documents this behavior in the Windows as a service update framework. Reference: Microsoft Learn: Restart behavior for Windows updates. Field reports also confirm that even with the policy enabled and the Windows Update service stopped, automatic updates can re-enable themselves and force a restart. Reference: Microsoft Q&A: Turn off automatic reboot with updates.

Detection signals in Event Viewer:

Recommended settings for a PCS 7 OS Server:

1. Open gpedit.msc (or GPO for centralized policy on a managed domain).
2. Navigate to Computer Configuration > Administrative Templates > Windows Components > Windows Update > Manage end user experience.
3. Enable No auto-restart with logged on users: Configured.
4. Configure Active Hours to span the operator shift, e.g. 06:00 - 22:00 (max 18 hours per Microsoft policy).
5. Set Scheduled install time to a known maintenance window outside production, e.g. Sunday 03:00.
6. Use WSUS and approve updates explicitly; do not allow automatic download from Microsoft Update.
7. For PCS 7 OS Servers, prefer the SUS/Hotfix DVD set approved by Siemens (see compatibility matrix in the PCS 7 release notes for the supported Windows Server / SQL Server / .NET Framework combinations per PCS 7 version).

BIOS Power-On After AC Loss Settings

If AC power is removed and restored, server BIOS behavior is controlled by the "After Power Loss" or "AC Power Recovery" setting. Three options exist:

Detection in Event Viewer:

Event ID 6005  - "The Event log service was started"   (cold boot signature)
Event ID 6008  - "The previous system shutdown at <time> was unexpected"
Event ID 6009  - "The operating system started at <time>"

A 6008 followed immediately by 6005/6009 with no intervening activity is the signature of a power-loss event, not an OS-initiated shutdown.

To verify and change the BIOS setting on common PCS 7 server platforms:

1. Dell PowerEdge (R740, R650, R540): Reboot and press F2. Navigate to System BIOS > Integrated Devices > Power Restore or System BIOS > Miscellaneous Settings > AC Power Recovery. Select "On", "Off", or "Last". Reference: Dell: Automatic restarting of server after a power failure.
2. HPE ProLiant (DL360, DL380 Gen10): Press F9 for System Utilities > System Configuration > BIOS/Platform Configuration > Power Management > AC Power Recovery.
3. Siemens SIMATIC IPC347G / IPC677D / IPC1047E / IPC647G: Press F2 or DEL during POST. Navigate to Power > After Power Failure. On SIMATIC IPC stations with BMC (IPCMon / DiagMonitor), the setting is also exposed through the DiagMonitor management interface and can be set remotely.

For SIMATIC IPC stations, the BIOS and the BMC have independent power-on schedules. After configuring BIOS, also verify that the BMC's scheduled wake-up events do not conflict with operator shift hours.

Hardware Diagnostics: Power Supply and Thermal Monitoring

A failing PSU or overheating component can cause abrupt shutdowns that appear as reboots. The PSU protection circuits trip when input voltage drops below tolerance or when internal temperature exceeds 110 °C, cutting output and forcing the server off. When the fault clears, the server may reboot if BIOS is set to "Power On" or "Last State".

Diagnostic steps using the platform-specific out-of-band management:

1. Open Server Hardware Management:
   • Dell iDRAC 9 > Hardware > PSU Status, Temperature Log
   • HPE iLO 5 > Information > System Health > Power & Thermal
   • Siemens SIMATIC IPC DiagMonitor (IPCMon) > System Status > Power & Temperature
2. Check for events flagged as Warning or Critical:
   • Inlet air temperature > 35 °C sustained
   • CPU Tcase > 85 °C (Intel Xeon nominal threshold for thermal shutdown)
   • PSU output voltage outside ±5% of nominal (12 V, 5 V, 3.3 V rails)
3. Inspect physical hardware:
   • Heatsink dust accumulation; clean with ESD-safe vacuum every 6 months
   • Fan RPM deviations (all fans should report nominal speed ±10%); replace a fan at first sign of bearing noise
   • Capacitor bulging or leakage on the motherboard
   • ECC memory errors in iDRAC/iLO event log (correctable > 100/hour or any uncorrectable = 1 is the threshold for replacement)
4. Review Event ID 47 in System log for MCE (Machine Check Exception) records; these are CPU-reported hardware errors and correlate strongly with impending CPU or memory failure.

If a redundant PSU is installed and one fails, the server continues on the remaining PSU but may reboot if the load exceeds the single-PSU rating during a transitory spike (e.g., disk rebuild on a RAID 5 array, where the load can briefly double). Verify both PSUs are present, healthy, and of equal rating. Hot-plug a replacement PSU immediately on any predictive-failure warning.

For SIMATIC IPC stations in control cabinets, also verify the cabinet cooling: a 19" cabinet with closed front and rear doors needs active ventilation of at least 200 m³/h for an IPC647G-class workstation. Many PCS 7 cabinets ship with a thermostat-controlled fan top - verify the fan is operating and the thermostat is set to 30 °C (not 35 °C or higher).

Service Failure Restart Behavior

The Windows Service Control Manager (SCM) can be configured to restart a service on failure. PCS 7 components such as WinCC RT, CCAgent, SQL Server (WinCC), SIMATIC WinCC Connectivity Pack, and S7DOS rely on the SCM recovery options. If a service crashes three times within a short interval, the SCM may be configured to reboot the server to clear a stuck state.

The registry key controlling SCM recovery is:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\CrashControl
  AutoReboot       = 1   (REG_DWORD; set 0 to stop after BSOD)
  CrashDumpEnabled = 1   (REG_DWORD; 1=complete, 2=kernel, 3=small)
  DumpFile         = %SystemRoot%\MEMORY.DMP
  LogEvent         = 1   (REG_DWORD; write to System log)
  SendAlert        = 1   (REG_DWORD; admin alert on stop error)

When AutoReboot = 1, the server reboots automatically after a stop error (BSOD). To preserve the crash dump for analysis, also set:

1. CrashDumpEnabled = 1 (complete memory dump) or 3 (small memory dump only).
2. Ensure the page file on the system drive is large enough: RAM size + 257 MB minimum, but at least 16 GB on a typical PCS 7 OS Server with 32 GB RAM.
3. Ensure %SystemRoot%\MEMORY.DMP has write permission for the SYSTEM account and that the volume has at least 1.5x physical RAM free space.

To detect service-induced reboots, search the System log for Service Control Manager events in the minutes before the reboot:

Get-WinEvent -FilterHashtable @{LogName='System';ProviderName='Service Control Manager';Id=7034,7043} -MaxEvents 50 |
  Where-Object {$_.TimeCreated -gt (Get-Date).AddDays(-7)} |
  Select-Object TimeCreated,Message |
  Format-List

Common WinCC services that may trigger SCM recovery actions when they fail:

• CCAlgRlServer - alarm logging runtime
• CCArchiveServer - process value archiving
• CCReportServer - report runtime
• S7OPCSRV - OPC server for S7 communication
• SNMP_TRAP_SERVICE - SNMP trap forwarding
• SQLSERVERAGENT - SQL Server agent for the WinCC database
• SIMATIC WinCC Connectivity Pack

If a specific service is repeatedly crashing, capture the application Event ID 1000 entries for that service and check the WinCC Diagnose log in C:\Program Files (x86)\Siemens\Automation\WinCC\Diagnose. The Diagnose folder rotates daily; archive the previous 30 days to a network share so historical failures can be reconstructed.

Third-Party Software Interference: Antivirus, Backup, and Remote Management

Software that intercepts file I/O, registry writes, or network packets can destabilize the WinCC Runtime and trigger BSOD or service crashes. The most common offenders in PCS 7 deployments are:

• Antivirus real-time scanning: Real-time file system filters from Trend Micro, McAfee, Sophos, and even default-configured Windows Defender can produce BSOD 0xC2 BAD_POOL_CALLER when intercepting WinCC archive writes. Exclude the following directories and processes from real-time scanning:
  • C:\Program Files (x86)\Siemens\Automation\WinCC\
  • C:\Program Files (x86)\Siemens\Automation\PCS7\
  • The WinCC project directories (typically D:\WinCCProjects\ or C:\Program Files (x86)\Siemens\Automation\WinCC\WinCCProjects\)
  • C:\Windows\System32\wbem\ (WMI repository, used by WinCC for tag subscription)
  • SQL Server data and log directories for the WinCC instance
  • Processes: CCAlgRlServer.exe, CCArchiveServer.exe, S7OPCSRV.exe, S7DOS.exe, sqlservr.exe, SQLAGENT.exe
• Backup agents: Veeam, Acronis, NetBackup, and Windows Server Backup use VSS writers that conflict with SQL Server VSS writer used by WinCC archives. Schedule backups to a maintenance window and verify the backup software's VSS writer list includes the SQL Server writer before each run.
• Remote management tools: TeamViewer, AnyDesk, LogMeIn, and similar tools can be configured to install updates that reboot the host. Disable auto-update on these tools.
• System Center / Endpoint Manager agents: SCCM client push-install and Windows Defender ATP can stage reboots. Configure the SCCM client to require user approval for reboots, and use maintenance windows aligned with the operator shift.

Detection: review installed programs and services. Any product installed in the last 90 days is a suspect until excluded.

Step-by-Step Diagnostic Procedure

Execute the steps in order. Each step produces a verifiable result before moving to the next. All commands are run from an elevated PowerShell prompt on the affected OS Server.

1. Capture state immediately: Run msinfo32 /report C:\diag\msinfo.txt and export the System log: wevtutil epl System C:\diag\system.evtx. Also export Application, Setup, and WinCC Diagnose logs.
2. Identify reboot signature: Look for the Event ID sequence

   6008 (unexpected shutdown) -> 6005 (boot) -> 6009 (startup)

   vs. 1074 (planned shutdown by user/process) -> 6005 -> 6009. The former indicates a fault; the latter is intentional.
3. Check HA failover log: In WinCC Explorer > Tools > WinCC Diagnose, look for redundancy events with the timestamp of the reboot. If a redundancy event precedes the reboot and the OS popup was visible, this is the documented Chapter 7.13 behavior and is not a fault.
4. Check Windows Update log: Run Get-WindowsUpdateLog and search for "Reboot" or "shutdown" in the merged log (default location C:\Windows\Logs\WindowsUpdate\).
5. Check UPS software log: Open APC PowerChute / Eaton IPM / SITOP UPS-Connect console and inspect recent shutdown events. Cross-reference facility BMS.
6. Check hardware health: iDRAC / iLO / IPCMon logs for PSU, thermal, and memory errors. Note Event ID 47 (MCE) entries.
7. Check driver and BSOD history: !analyze -v on the latest minidump if present.
8. Check SCM service recovery: Look for repeated 7034/7043 events before the reboot.
9. Check scheduled tasks: schtasks /query /fo LIST /v for tasks scheduled to run at the reboot time. Pay attention to \Microsoft\Windows\UpdateOrchestrator\ tasks.
10. Check third-party software: Antivirus, backup agents (Veeam, Acronis, NetBackup), and remote management tools (TeamViewer, AnyDesk, Kaseya) can schedule or trigger reboots. Review Add/Remove Programs and Services for any product installed within 90 days of the first reboot.
11. Correlate with facility events: Building power, HVAC, fire-system tests, and lightning storms all correlate. Ask the facility manager for the BMS log of the same time window.

Verification Checklist and Preventive Maintenance

After applying corrective action, confirm the fix by:

1. Running a controlled failover test during the next maintenance window: stop the WinCC Runtime service on the Master and observe whether the Standby takes over and the Master reboots per configuration. Record the failover time (target < 30 s) and the availability window covered.
2. Performing a controlled UPS test: pull the AC input and verify the server remains on battery for the configured runtime, then issues a shutdown, then powers on after AC is restored. Verify no archive corruption in C:\WinCC\ArchiveManager.
3. Performing a Windows Update staging test in a lab OS Server before deploying to production. Apply the patch, reboot, and verify WinCC Runtime auto-starts and re-establishes redundancy within the partner watchdog interval.
4. Capturing a fresh baseline of the System log after each change so that the next reboot is unambiguously attributable.
5. Updating the PCS 7 OS Server hardware health baseline (PSU wattage, inlet temperature, fan RPM) in the CMMS so drift is detectable.

Preventive maintenance schedule for PCS 7 OS Servers:

FAQ