Resolving WinCC Redundancy Error 1003109 ALGRT Master Failover on Multi-NIC Servers

WinCC server redundancy is a standard availability architecture for SCADA systems where two parallel servers monitor each other over the terminal bus and provide equivalent redundancy to clients. When this pairing breaks because one partner is "stuck" as master or the standby server refuses to take over, the root cause is almost always in the underlying network plumbing rather than in the WinCC project itself. The diagnostic entry 1003109 ALGRT:[Server1]: The connection to the master was interrupted is one of the most common signatures of a misconfigured Simatic Shell binding on a multi-NIC host.

This reference documents the failure mode, the multi-NIC root cause, the exact Simatic Shell remediation steps, and the verification tests needed to certify a healthy redundant pair.

1. Problem Overview

A WinCC V7.x (and compatible RT Professional) redundant pair was configured on Windows Server 2008 R2 64-bit with the following observed behavior:

• Server1 was commissioned as the default master; Server2 was configured as standby.
• After initial startup, both servers reported healthy redundancy state and clients attached to Server1 as expected.
• When Server1 was powered off, all clients failed over to Server2 and Server2 correctly took the master role.
• When Server1 was restarted and re-joined, Server2 retained the master role; Server1 entered a degraded state and clients continued to be served from Server2.
• When Server2 was subsequently stopped, Server1 failed to take over the master role. Clients connected to Server2 stopped receiving data, while Server1 continued to run without announcing itself as master.
• After Server2 was powered back on, Server2 remained master and the synchronization state between the two servers became unstable.
• The internal alarm subsystem on Server1 stopped delivering new alarms after the first failover event.

Despite both projects being duplicated with the WinCC Project Duplicator and being confirmed identical, the redundancy state machine never recovered to its default master assignment. The issue was ultimately traced to the host network configuration, not to the WinCC project itself.

2. Diagnostic Output and Fault Code 1003109

The Siemens WinCC Diagnostics viewer (Siemens/WinCC/Diagnose) reported the following entry:

Event ID : 1003109
Source   : ALGRT
Server   : [Server1]
Message  : The connection to the master was interrupted.

The 1003109 entry is generated by the ALGRT subsystem when its TCP session to the partner server's alarm-logging replication endpoint times out or is refused. In a healthy pair, ALGRT events are silently mirrored every cycle. When 1003109 appears repeatedly with no corresponding connection re-established event, the terminal bus session itself is broken.

Related event IDs you should collect alongside 1003109 when triaging this fault:

2.1 How to Collect a Diagnostic Package

1. On the affected server, open the WinCC Explorer.
2. Right-click the project name and choose Diagnostics.
3. Click Save diagnostic files; WinCC will package the *.log, *.dmp, and CCAlgRtLog.txt into a zipped folder under <ProjectPath>\Diagnostics\<timestamp>.zip.
4. Inspect CCAlgRtLog.txt for repeated 1003109 entries and confirm the partner IP/hostname the ALGRT subsystem was attempting to reach.

3. Root Cause: Multi-NIC Hosts and the Simatic Shell Terminal Bus

The redundant servers in this incident each had four network interface cards. This is common in industrial server builds:

• NIC 1 — plant/terminal bus (WinCC partner communication)
• NIC 2 — corporate/IT network
• NIC 3 — iSCSI / dedicated storage network
• NIC 4 — management/iLO/iDRAC

WinCC uses the Simatic Shell (the Windows Shell namespace extension installed by SIMATIC products) to publish and resolve partner servers over the terminal bus. Simatic Shell relies on Microsoft COM/DCOM and the host's network binding order to determine which local IP address is announced to remote servers and which local interface is used to initiate outbound connections.

When a host has four NICs and the binding order is not explicitly constrained, Windows will:

1. Advertise Simatic Shell on a non-terminal-bus interface (e.g., the iSCSI NIC).
2. Listen for partner connections on the wrong interface IP.
3. Resolve the partner's hostname to an IP on a different subnet than the one the partner is actually broadcasting on.

The visible effect is exactly what was reported:

• Initial startup "works" because the first server happens to bind to the correct interface.
• Failover does not converge because the standby server has been listening on the wrong NIC and has never actually established a session with the partner's ALGRT port.
• Default-master assignment is never honored because Simatic Shell cannot complete the master-elimination handshake on the terminal bus.
• Alarms stop appearing on the "degraded" server because ALGRT silently drops its replication channel.

4. How WinCC Redundancy Works

According to the official Siemens documentation "How Redundancy Works (RT Professional)", the redundancy model relies on the following architectural principles:

• Equivalent redundancy: should one server fail, the partner server is always available to take over the complete runtime scope with no operator intervention.
• Terminal-bus coupling: the two redundant servers communicate via the terminal bus to monitor each other's availability in a continuous heartbeat cycle.
• Master / Standby roles: only the master actively serves clients and writes to process archives; the standby runs in hot-standby mode and mirrors the master's runtime state.
• Automatic failover: when the master stops responding to heartbeats beyond a configurable threshold, the standby promotes itself to master and clients reconnect.

Each of these steps assumes a single, reliable, bidirectional channel between the two servers. When Simatic Shell is misconfigured on a multi-NIC host, the channel is fragmented and the state machine cannot make a deterministic decision.

5. Prerequisites for Remediation

Before touching the WinCC project, complete the following host-level prerequisites. Skipping any of these leaves the configuration in a fragile state and risks recurrence after the next OS patch cycle.

1. Network topology inventory. Document every NIC on both redundant servers: MAC, IPv4/IPv6, subnet, gateway, DNS registration, and intended role (terminal bus, plant LAN, iSCSI, management).
2. Dedicated terminal-bus subnet. Both servers must share at least one NIC on the same subnet that is reserved exclusively for partner communication. No client traffic, no iSCSI, no IT.
3. Static addressing. Configure the terminal-bus NICs with static IPv4 addresses and matching subnet masks on both servers. Do not use DHCP on the terminal bus.
4. Symmetric binding order. The terminal-bus NIC must be first in the Windows network binding order on both servers.
5. Hostname resolution. Each server must be able to resolve the other's terminal-bus hostname to the terminal-bus IP. Add entries to C:\Windows\System32\drivers\etc\hosts on both servers if DNS does not have the records.
6. Windows Firewall rules. Open the ports used by WinCC Redundancy (default: TCP 1433-style SQL, 80/443 for web, plus dynamic ports used by ALGRT and TagLoggingRT) on the terminal-bus profile only.
7. Same Windows account credentials for the WinCC service login on both servers, with equivalent local administrator rights.
8. Project Duplicator verified as the deployment method, or a documented manual-comparison checklist completed (redundancy settings, time sync, user archive, alarm routing).

6. Step-by-Step Resolution: Configuring Simatic Shell for Multi-NIC Hosts

The corrective procedure below restores the deterministic single-channel terminal bus that WinCC redundancy requires.

Step 6.1 — Confirm the Terminal Bus Subnet

Open Control Panel → Network and Sharing Center → Change adapter settings. Identify the NIC connected to the partner server. Note:

• IPv4 address (e.g., 10.20.30.11)
• Subnet mask (e.g., 255.255.255.0)
• Preferred DNS suffix (must match between both servers)

Step 6.2 — Constrain the Network Binding Order

1. Press Win+R, type ncpa.cpl, press Enter.
2. Press Alt to expose the menu bar, choose Advanced → Advanced Settings.
3. In the Adapters and Bindings tab, expand Connections.
4. Move the terminal-bus NIC to the top of the list. The OS will use the topmost bound adapter for Simatic Shell announcements.
5. Click OK and reboot.

Step 6.3 — Restrict Simatic Shell to the Terminal Bus NIC

1. Open the registry editor (regedit) on each redundant server.
2. Navigate to HKEY_LOCAL_MACHINE\SOFTWARE\SIEMENS\AUTSW\WinCC\Redundancy (or the equivalent path for the installed WinCC version, e.g., ...\WinCC\V7.x\Redundancy).
3. Create or edit the value TerminalBusInterface (REG_SZ) and set it to the IPv4 address of the terminal-bus NIC, e.g. 10.20.30.11.
4. Repeat for the partner server using its own terminal-bus IP.
5. Reboot both servers.

Step 6.4 — Configure Simatic Shell for the Right Subnet

1. On each server, open Windows Explorer and navigate to Simatic Shell (it appears as a top-level folder under Computer).
2. Right-click Simatic Shell and choose Settings.
3. In the dialog, set the PG/PC interface to the terminal-bus NIC (named e.g. Intel I350-T4 — TerminalBus), not any.
4. Confirm that the partner server is now listed under Simatic Shell → [Server2_PartnerName] with the correct terminal-bus IP.
5. Repeat on the partner server, pointing to the first server.

Step 6.5 — Verify the COM/DCOM Endpoint

1. Open dcomcnfg.
2. Navigate to Component Services → Computers → My Computer → DCOM Config.
3. Find the WinCC redundancy-related application (e.g., CCRedundancyAgent or WinCC Redundancy Service).
4. Right-click → Properties → Endpoints.
5. Add a static endpoint using the terminal-bus IP and the WinCC redundancy port range (typically 5000-5010 + dynamically negotiated). Remove Any IP bindings.

Step 6.6 — Rebuild the Project with the Project Duplicator

1. On the master server (Server1), perform a full project backup.
2. Open WinCC Explorer → Tools → Project Duplicator.
3. Set Server1 as source, Server2 as target.
4. Confirm that redundancy settings, time-synchronization, user administration, and archive configurations are all selected for transfer.
5. Execute the duplicate.

7. Verification Procedure

After completing the remediation, the redundancy pair must be exercised end-to-end before being returned to production. The verification sequence below is intentionally aggressive — a real multi-NIC misconfiguration can survive one or two failover cycles.

7.1 Passive State Verification

1. Start Server1; confirm it reports Master.
2. Start Server2; confirm it reports Standby and the partner state shows Connected.
3. In WinCC Diagnostics, confirm no 1003109 entries appear in CCAlgRtLog.txt during a 15-minute soak.
4. Verify the WinCC redundancy internal tags:
   • @RM_MASTER = 1 on Server1, 0 on Server2
   • @RM_MASTER_NAME resolves to Server1
   • @RM_PARTNER_STATE = 1 (connected) on both

7.2 Failover Verification — Master Down

1. On Server1, gracefully stop the WinCC service.
2. Within the configured monitoring time (default 25 s), Server2 should promote itself to Master.
3. Confirm that connected clients reconnect automatically.
4. Confirm that new process values are being archived to Server2's local SQL database.

7.3 Failover Verification — Master Returns

1. Restart Server1.
2. If Server1 is the default master and the option Return to default master on restart is enabled, Server1 should reclaim the master role.
3. If the option is not enabled, Server2 retains master — this is expected behavior, not a fault.

7.4 Forced Failover Verification — Standby Down

1. With Server1 as master, stop Server2.
2. Confirm that Server1 continues to run as master without errors (it should not attempt to redirect clients to a non-existent partner).
3. Restart Server2; confirm it rejoins as standby.

7.5 Alarm Logging Verification

1. Trigger an alarm on Server1.
2. Confirm it appears in the alarm window on both Server1 (if visible) and on Server2's mirrored alarm view.
3. Trigger an alarm on Server2; confirm it appears on Server1.
4. Confirm no 1003109 entries are logged.

7.6 Verification Checklist

8. Troubleshooting Matrix

9. Best Practices for Multi-NIC WinCC Hosts

1. Reserve a dedicated subnet and NIC for the terminal bus. Never combine terminal-bus traffic with iSCSI, plant LAN, or IT traffic.
2. Disable NetBIOS over TCP/IP on all non-terminal-bus NICs. This prevents Simatic Shell from advertising on the wrong interface.
3. Disable Register this connection's address in DNS on non-terminal-bus NICs.
4. Use hosts entries for partner hostname resolution in addition to DNS, to guarantee deterministic resolution regardless of DNS state.
5. Document the binding order in the project's commissioning report and include a screenshot of ncpa.cpl → Advanced → Advanced Settings.
6. Mirror the binding order on both redundant servers; asymmetry is the leading cause of "works on Server1, not on Server2" behavior.
7. After every Windows update cycle, re-verify the binding order — KB patches occasionally reset the adapter metric.
8. Use the WinCC Project Duplicator exclusively for redundancy-pair deployment. Manual configuration drifts and silently breaks redundancy.
9. Configure time synchronization (NTP or Windows Time) before activating redundancy. Skew > 5 s can cause heartbeat misinterpretation.
10. Keep diagnostic archives for at least one full failover cycle after commissioning. The 1003109 entries you do not see are the proof of a healthy pair.

10. Operating-Systems Note: Windows Server 2008 R2

The original incident occurred on Windows Server 2008 R2. This OS remains supported by many legacy WinCC installations but has reached mainstream end-of-support; if the platform is still in production, the following considerations apply:

• Apply the latest available servicing updates before commissioning the redundancy pair.
• Confirm that the installed WinCC version is on the manufacturer's compatibility list for Windows Server 2008 R2 (typically WinCC V7.0 SP3 through V7.4 SP1).
• If migrating to Windows Server 2016 or 2019, re-test the full failover matrix in §7 — DCOM endpoint behavior changed between Windows Server versions and the registry paths described in §6.3 may differ.
• DCOM default authentication level was tightened in Windows Server 2016 and later. If the pair is migrated, set Packet Privacy (RPC_C_AUTHN_LEVEL_PKT_PRIVACY) explicitly on the WinCC application IDs.

11. Diagnostic Quick Reference

12. Summary

The "one partner is always master" symptom in a WinCC redundant pair with multi-NIC hosts is almost never a WinCC project problem. It is a host-network-binding problem that the WinCC layer exposes as event 1003109 ALGRT. The correction is not in the WinCC project but in the OS:

• Dedicate one NIC and one subnet to the terminal bus.
• Pin Simatic Shell to that NIC in its settings.
• Set the registry value TerminalBusInterface to that NIC's IP.
• Constrain DCOM endpoints to the same IP.
• Re-deploy with Project Duplicator after the host is corrected.
• Verify with the §7 matrix before returning the pair to production.

With these steps, the standby server will once again be a true standby, the default master assignment will be honored, and the 1003109 ALGRT entries will disappear from the diagnostic log.

For deeper architectural background, see the official Siemens documentation on how redundancy works in WinCC RT Professional.