Troubleshooting WinCC Tag Value Mixing Between Multiple RTUs

1. Problem Overview

In WinCC V7.0 SP3 Update 1 client-server configurations with TeleControl Server Basic (TCSB) and OPC DA, tag values from different Remote Terminal Units (RTUs) can appear mixed in WinCC Explorer and on HMI client pictures even when the underlying OPC server returns correct values. The symptom is observable on both the WinCC Server (redundant pair) and every connected WinCC Client, but not on third-party OPC clients such as Matrikon OPC Explorer or Kepware OPC Quick Client. This is a telecontrol-side protocol/buffering issue rather than a WinCC runtime display issue, and the diagnosis path must start at the TCSB process image and the GPRS link to the S7-200 station.

The defect is reproducible on a single WinCC Server configuration and on a redundant pair. It is independent of the IPC 547D hardware platform, the i7 processor generation, and the Windows Server 2008 R2 OS. The defect is also independent of the @RM_MASTER redundancy system tag and is not a redundancy synchronization issue. The defect is tied to the telecontrol channel and the S7-200 + MD720-3 + TCSB combination.

2. Affected System Architecture and Versions

The defect has been reproduced on the following combination. All part numbers and version identifiers below are the exact components observed in the field.

WinCC V7.0 SP3 Update 1 corresponds to the V7.0 service pack branch that introduced the OPC DA 3.0 client and the redundancy system tags. The TCSB V7.0 SP3 driver is the S7-200 + MD720-3 specific driver that talks to the S7-200 CPU using the SINAUT ST1 protocol encapsulated in TCP/IP over GPRS. The driver registers itself as an OPC DA 2.0 server with the ProgID TeleControlServerBasic.TCSB on the OPC server host.

3. Symptoms and Diagnostic Indicators

The failure mode has a reproducible timeline that can be used as a regression signature in any bench test:

1. Open a process picture monitoring RTU_5. All I/O fields show the correct values of RTU_5 for a period of 1 to 5 minutes.
2. When a tag belonging to a different RTU (RTU_7) changes its value, the picture monitoring RTU_5 displays the new value of RTU_7 in the field that should show RTU_5's value. The crossing is automatic, the value of RTU_5 is overwritten in the WinCC image without any operator action.
3. Closing and re-opening the picture on RTU_5 restores the correct value for RTU_5, until the next write on RTU_7 occurs.
4. In WinCC Explorer on the WinCC Server, the tag of RTU_5 already shows the wrong value of RTU_7. The error is therefore located in the WinCC process image, not in the display script on the client.
5. When the same tag is read with Matrikon OPC Explorer or Kepware OPC Quick Client on the same machine, or on the OPC server machine, the value of RTU_5 is always correct.

Tag-logging experiments confirm the failure mode at a finer grain:

• Tags displayed on the start picture of the WinCC client, with an update cycle of 10 seconds on every I/O field, show a clear tag-value swap pattern overnight. Two tags may even share the same numeric value at a given instant when the underlying process values are different.
• If the same tags are removed from the picture and only tag-logging is active, the archived values in the SQL database are correct. The defect is therefore tied to picture-driven read requests and the WinCC image buffer, not to the logging path.
• A dedicated WinCC client project that displays only the three affected tags and a trend window reproduces the same swap pattern, eliminating graphics-board or driver issues as a cause.

The absence of errors in the OPC log file of the WinCC Diagnose folder is a key observation. The WinCC Diagnose folder typically contains opc.log, opcerror.log, and wincc.log. A clean opc.log rules out a transport-level OPC DA error and confirms the defect is in the upper layer, the process image.

4. Root Cause Analysis

The combination of evidence - correct OPC values, wrong WinCC image values, swap pattern, picture-driven manifestation - points to the WinCC Telecontrol channel buffer rather than the OPC DA transport. TCSB maintains an internal process image keyed by the S7-200 station number and the configured tag address. When the WinCC runtime requests a tag, it looks up the tag in the WinCC tag database (configured in WinCC Explorer under "Tag Management" -> "OPC" channel), but it uses the OPC item handle returned by TCSB. The handle is composed of the station index and the address.

In TCSB V7.0 SP3 with the S7-200 + MD720-3 driver, the station index is derived from the source IP or the GPRS-allocated IP at the time of the very first polling handshake. The station index is then re-used as a cache key for every tag of that station. When the GPRS link drops and re-establishes - which is normal on a public APN - TCSB may assign the same station index to a different physical S7-200 if the GPRS IP pool reuses the address. The cache is then corrupted: the next read of tag RTU5.WaterLevel returns the value that was last written to RTU7.WaterLevel because both share the same station index in the cache.

The reason why the third-party OPC clients do not show the error is that Matrikon OPC Explorer and Kepware OPC Quick Client open a fresh subscription on every browse, and they do not maintain a process image keyed by station index. They read the live value through the OPC item handle, which TCSB recomputes on the fly. The WinCC runtime, by contrast, opens the OPC subscription once at picture activation and reuses the process image for the lifetime of the picture or the runtime.

The reason why removing the tags from the picture makes the error disappear is that the tag-logging path uses a different subscription mechanism: tag-logging requests the raw OPC value at the configured logging cycle (typically 1 s - 1 min) and writes it directly to the SQL database without going through the picture process image. The picture process image is the one that holds the corrupted cache key.

The cross-reference between tags is observable because the S7-200 + MD720-3 stack has a fixed tag map per station: every station has the same tag names (REAL2128 for water level, REAL2214 for another analog, etc.), only the station number differs. When two stations share the same cache key, the tag with the same name in the WinCC database is overwritten by the value of the other station. The overlay is invisible to the operator because the tag name is the same in the picture.

5. WinCC Redundancy Tags and OPC Channel Behavior

On a redundant WinCC server pair, the swap pattern can also be confused with a redundancy synchronization issue. The WinCC redundancy system tags @RM_MASTER and @RM_MASTER_NAME are the runtime variables used to drive the master/standby role and the client-side failover, as documented in the official WinCC Redundancy RT Professional reference.

The defect described in this article is independent of the redundancy role. The swap pattern is reproduced on a single-server configuration as well. The relevant check is therefore to confirm that @RM_MASTER_NAME is the expected server computer name during the swap window, which rules out a client accidentally talking to the wrong physical server during failover. A typical diagnostic script in WinCC VBS:

' VBS in WinCC - log master name when swap is observed
Dim sMaster
sMaster = HMIRuntime.Tags("@RM_MASTER_NAME").Read
HMIRuntime.Trace "Swap observed, master = " & sMaster & " t = " & Now

If the master name is the expected server during the swap, the failure is not a redundancy issue. If the master name flips during the swap, the redundancy is masking a telecontrol issue and must be addressed first.

For the OPC channel itself, WinCC V7.0 SP3 Update 1 uses the OPC DA 2.0 / 3.0 client. The relevant configuration items are:

The Quality code supervision setting is critical. When the GPRS link is re-establishing, TCSB may report the tag quality as Good (0xC0) for a brief period even though the value is stale. The default WinCC quality code filter accepts Good (0xC0) and Good Local Override (0xD0) only. Enabling explicit quality code supervision on every telecontrol tag ensures that Good-but-Stale (0x40) or Uncertain (0x40) values are not displayed. The operator sees a quality flag instead of a wrong value.

6. TCSB vs. SINAUT MICRO SC: Protocol Layer Comparison

The defect is specific to the S7-200 + MD720-3 + TCSB combination. SINAUT MICRO SC, the predecessor product, uses a different transport and station identification. The relevant differences are summarised below.

Because SINAUT MICRO SC keys the station identity on a configured number that is part of the SINAUT project file, two stations cannot share the same key even if the GPRS IP is reused. The S7-200 + MD720-3 + SINAUT MICRO SC stack is therefore immune to the cache corruption described above and is the documented fallback path for legacy installations.

The SINAUT MICRO SC project file is generated by the SINAUT MICRO SC configuration tool, and every station receives a unique 16-bit station number at commissioning time. The station number is written into the MD720-3 modem's non-volatile memory and is sent in every SINAUT ST1 frame. TCSB, by contrast, derives the station identity from the GPRS-allocated IP because TCSB was designed for a multi-vendor IP-based RTU fleet and the S7-200 + MD720-3 driver was an adapter, not a native TCSB path.

7. Hardware Migration: MD720-3 to CP 1242-7 with S7-1200

For new deployments, Siemens has retired the S7-200 + MD720-3 combination in favour of the S7-1200 + CP 1242-7 stack. The CP 1242-7 (article number 6GK7242-7KX30-0XE0) is a GPRS communications processor that plugs into the left side of an S7-1200 CPU and registers with TCSB using a project-engineered station number. The cache key is no longer the source IP, so the swap defect does not occur. The migration plan is:

1. Replace the S7-200 CPU with a S7-1200 CPU of equivalent I/O count. For example, a CPU 1214C DC/DC/DC (6ES7214-1AG40-0XB0) covers most S7-200 CPU 224 replacement scenarios. The S7-200 program must be ported to the S7-1200 using the SIMATIC S7-200 to S7-1200 migration tool, which converts the S7-200 STL/LAD to S7-1200 SCL/GRAPH where automatic.
2. Add a CP 1242-7 (6GK7242-7KX30-0XE0) to the S7-1200 rack. Configure the GPRS APN, the TCSB partner IP, and the TCSB partner port in the CP's web interface.
3. In TIA Portal V13 SP1 or later, configure the CP 1242-7 as a TCSB partner. The station number is configured explicitly in the CP and is the one TCSB uses to key the cache.
4. In the WinCC V7.0 SP3 Update 1 project, re-create the OPC tags against the new TCSB partner name. Existing tag names can be retained; only the connection name needs to be updated.
5. Re-test the picture-driven scenario with a 2-second update cycle. The swap pattern must be absent.

The S7-200 program port is non-trivial. The S7-200 uses a different memory model (V memory, M memory, T/C blocks) than the S7-1200 (data blocks, instance DBs, IEC timers/counters). A 1:1 port is rarely possible. A typical porting effort is 2-4 hours per station for a straightforward ladder program, and 1-2 days per station for a program with custom PPI communication, USS drives, or Modbus RTU master/slave.

The CP 1242-7 firmware version must be at least V2.1.7 for TCSB V7.0 SP3 compatibility. The firmware is updated via the TIA Portal online functions. The CP 1242-7 V2.1.7 firmware release notes state that the GPRS partner table is stored in the CP's non-volatile memory and is preserved across power cycles, eliminating the IP-reuse risk at the source.

8. Immediate Workaround: Reverting to SINAUT MICRO SC

If the migration to CP 1242-7 cannot be completed in the planned downtime, the documented temporary fix is to revert the telecontrol stack on the affected S7-200 stations from TCSB to SINAUT MICRO SC. The procedure is:

1. Install the SINAUT MICRO SC software on the Windows Server 2008 R2 host that previously hosted the WinCC Server. License the required number of stations (50-100 in this case).
2. Reconfigure the MD720-3 modem at each S7-200 to use the SINAUT MICRO SC firmware and to dial the SINAUT MICRO SC partner. The SINAUT MICRO SC configuration tool generates the dial string and the station number.
3. Re-create the WinCC tags against the SINAUT channel driver. The WinCC Tag Management channel for SINAUT MICRO SC is "SIMATIC S7 Protocol Suite" -> "SINAUT ST7" or the dedicated "SINAUT MICRO SC" OPC server. The tag names can be retained, only the channel and connection change.
4. Validate the picture-driven scenario with a 2-second update cycle. The swap pattern must be absent.

This is a temporary fix. The S7-200 + MD720-3 + SINAUT MICRO SC stack is in the Siemens product lifecycle phase "phase-out announced" and a hardware migration to S7-1200 + CP 1242-7 must be planned within the next maintenance window.

The reversion step can be performed station-by-station without a global outage. The SINAUT MICRO SC partner is configured to listen on a different TCP port than the TCSB partner, so the two stacks can co-exist during the transition. The S7-200 program is unaffected by the reversion because the S7-200 application is on the CPU side and the telecontrol protocol is on the MD720-3 side; the CPU only exchanges process data with the MD720-3 over the serial PPI interface.

9. Permanent Fix: Structure Addressing in TCSB

For installations that must remain on TCSB, the documented permanent fix is to switch the tag mapping from a flat tag list to a structured addressing scheme. In structured addressing, every RTU's tag set is grouped under a WinCC structure type, and the structure instance is bound to the station number. The TCSB driver re-evaluates the station number on every read, which eliminates the cache poisoning.

Procedure:

1. In WinCC Explorer, open the tag management and create a new structure type for each RTU tag set. The structure type must contain every tag of the RTU (water level, pump status, alarm bits, etc.). The structure type is a template and does not carry a station number.
2. For every RTU, create a structure instance of the structure type. Set the "Station number" property of the structure instance to the explicit TCSB station number configured in the S7-200 + MD720-3 project. The station number is the SINAUT ST1 station number written into the MD720-3 at commissioning time.
3. Replace every flat tag in the picture and in the tag-logging configuration with a reference to the structure instance member. The I/O field must use the dot notation: RTU5.WaterLevel instead of RTU5_WaterLevel. WinCC resolves the structure instance to the OPC item handle at runtime.
4. Re-test the picture-driven scenario with a 2-second update cycle. The swap pattern must be absent.

The structured addressing path requires that the TCSB driver on the OPC server is at least V7.0 SP3 Update 2. Earlier V7.0 SP3 drivers do not support per-instance station number binding and the swap persists. Validate the driver version in the Windows registry under HKLM\SOFTWARE\Siemens\TeleControlServerBasic\Version before re-creating the structure types.

The structure type can be exported from one WinCC project and imported into another WinCC project via the WinCC Configuration Studio. For 50-100 stations, the structure instance creation is best scripted with the WinCC ODK (OLE Development Kit) or the WinCC Configuration Studio's built-in bulk-edit feature. A typical bulk-edit script creates 100 structure instances from a CSV file in under 60 seconds.

10. Verification and Commissioning Procedure

After applying any of the three fixes above, run the following commissioning procedure on a non-production bench rig before the production cutover. The rig must contain at least three S7-200 stations (or three S7-1200 + CP 1242-7 stations for the migration path) and must force a GPRS IP re-assignment at least once during the test.

1. Bring up the WinCC Server and the WinCC Client on the production OS images. Open the WinCC Explorer on both the server and the client. Confirm the @RM_MASTER and @RM_MASTER_NAME tags are populated and the master name matches the expected server computer name.
2. Open the start picture of the WinCC client with the three RTU tags in I/O fields. Set the update cycle on every I/O field to 2 seconds.
3. Force a GPRS link re-establishment on at least one of the three stations by cycling the MD720-3 power. The MD720-3 will request a new IP from the APN. The new IP may or may not match the previous IP depending on the APN pool.
4. Observe the I/O fields for at least 30 minutes. None of the three I/O fields should ever show a value that does not belong to its own station.
5. Open Matrikon OPC Explorer on the WinCC Server and on the OPC server. Read the three tags. All three values must match the I/O fields on the picture.
6. Open the tag-logging database and confirm the archived values for the three tags are continuous and never swap.
7. Force a WinCC Server failover by stopping the WinCC Server on the master. The redundant server takes the master role. Confirm @RM_MASTER_NAME switches to the new master and the I/O fields on the picture continue to show the correct values without any swap.

If the test fails on step 4, the cache corruption is still present. The most common root cause in that case is a partial migration: some stations are still on the old driver, some on the new one. Check the Windows registry on the OPC server host for the exact driver version of every station entry.

The bench rig should also exercise the GPRS link under load. A typical load test pushes 100 tags per station at a 1-second cycle for 24 hours and forces a minimum of 10 GPRS link re-establishments. The 24-hour soak test catches defects that only manifest after the GPRS IP pool has cycled through every address in the subnet.

11. Troubleshooting Matrix

12. Monitoring and Long-Term Diagnostics

For a fleet of 50-100 stations, a permanent monitoring setup is recommended to catch the defect early. The recommended setup is:

1. Configure the WinCC Diagnostic Server to archive the OPC quality code of every telecontrol tag at a 1-second cycle. The OPC quality code is archived as a separate tag using the WinCC quality code extractor.
2. Build a WinCC trend view that overlays the OPC quality code of three adjacent stations. A quality code drop on all three stations at the same instant indicates a GPRS outage, which is the trigger event for the cache corruption.
3. Build a WinCC alarm message that fires when the OPC quality code of a tag is Good (0xC0) for more than 60 seconds but the value has not changed by more than 0.1% of the engineering range. This is a proxy for a "frozen" tag, which is a common symptom of the cache corruption.
4. Archive the @RM_MASTER_NAME tag at a 1-second cycle. A flip-flop of @RM_MASTER_NAME more than 5 times per hour indicates a redundancy instability and may be related to the telecontrol issue.

The monitoring setup costs approximately 200 MB of SQL database per day for 100 stations at a 1-second cycle. The SQL database retention is set to 30 days by default, which is sufficient to investigate a defect that manifests overnight.

13. Disaster Recovery and Rollback

If the structure addressing fix or the SINAUT MICRO SC reversion introduces a regression, the rollback procedure is:

1. Stop the WinCC Runtime on the affected server. Do not stop the WinCC Server service, only the runtime project.
2. In WinCC Explorer, restore the WinCC project from the last known-good backup. The backup is typically created by the WinCC Project Duplicator and stored on a network share.
3. Revert the TCSB driver to the previous version using the Windows Control Panel -> Programs and Features. The previous version installer is kept in C:\Siemens\Automation\WinCC\Setup.
4. Restart the WinCC Runtime. Verify the swap pattern is absent in the bench rig before restoring production.

The full rollback can be completed in under 30 minutes. The bench rig is a mandatory pre-condition: never roll back a telecontrol change in production without a bench-rig validation.

14. Frequently Asked Questions