PCS7 V6.0 SP1 WinCC Disappearance: SQL Error 9003 Recovery

Problem Overview

On a SIMATIC PCS 7 V6.0 SP1 multiproject (Engineering Station, ES, running Windows Server 2000 in the reported case), all WinCC OS applications under the SIMATIC Manager tree (PC Station components) suddenly disappear. The project contains multiple AS stations (S7-400 controllers) and multiple PC stations, all of which previously showed their WinCC OS sub-objects. After the failure, double-clicking a PC station returns an error dialog and the OS hierarchy under that station is empty.

The reported root indicators in this failure mode are:

• SQL Server service (instance MSSQL$WINCC) is stopped and cannot be restarted from the Services console. Pressing Start in the Services snap-in shows a brief Starting… state, then reverts to Stopped.
• SQL error log shows Error 9003, Severity 20, State 1: "Cannot recover the master database. Exiting."
• One or more SQL databases (often a backup copy created by a third-party tool such as VersionDoc) enter Suspect mode (icon shown grey instead of the normal yellowish green in SQL Enterprise Manager / Management Studio).

PCS 7 stores all OS project data, picture hierarchies, tag databases and alarm logging inside Microsoft SQL Server. When the SQL Server instance cannot bring its master database online, WinCC components — even though their .pdl and .db files may still be intact on disk — are no longer linkable inside SIMATIC Manager and are filtered out as a defensive measure.

Root Cause Analysis

The master database is the system catalog of SQL Server: it tracks every other database on the instance, the physical file locations, login accounts, configuration values, and the recovery state of in-flight transactions. SQL Server cannot start any user database, including the WinCC project databases, without a consistent master. Error 9003 is logged when the recovery process for master fails, typically because:

1. Physical corruption of master.mdf or mastlog.ldf (bad block, truncated file, disk full, anti-virus quarantine, abrupt power loss).
2. Presence of a Suspect user database that prevents instance-wide recovery. In the reported field case, the VersionDoc daily backup tool created a temporary database that was never dropped and was left in Suspect state.
3. Third-party software (archiving, antivirus, backup agents, PCS 7 add-ons) holding an exclusive handle on the .mdf/.ldf files while SQL Server tries to attach them, producing torn writes and a corrupted boot page.

When SIMATIC Manager cannot obtain a valid SQL handle, the OS sub-tree of every PC station in the multiproject collapses. This is intentional: SIMATIC Manager hides OS objects whose backing database is unreachable rather than presenting a half-functional project that would corrupt further edits.

Diagnostic Decision Path

Recovery Procedure (Without Reinstallation)

Use this procedure when the project file tree on disk is intact, a recent good backup of the master database is not available, and at least one of the user databases (typically CC_…_…) is in Suspect state but the master.mdf itself has a known good counterpart.

Prerequisites

• Local administrator rights on the ES.
• The PCS 7 V6.0 SP1 installation DVD set (for the SQL Server tools) or the original SQL Server 2000 installation media used to install MSSQL$WINCC.
• A working backup of the entire project directory (*.s7p project path, typically D:\PCS7\Project\<ProjectName>) copied to external media before any destructive action.

Step-by-Step

1. Stop all SIMATIC-related services that may hold database handles:
   

   net stop "S7DOS"
   net stop "CCAgent"
   net stop "SQLAgent$WINCC"

2. Open Start > Programs > Microsoft SQL Server > Enterprise Manager (V6.0 ships SQL Server 2000 client tools). If only Service Manager is installed, install the full client tools from the SQL Server 2000 CD using sqlstp.exe.
3. In Enterprise Manager, expand Microsoft SQL Servers > SQL Server Group > (local) (Windows NT) > WINCC. The instance name WINCC matches the named instance created by the PCS 7 setup.
4. Stop the SQL Server service. Right-click the server > Stop. Verify the Service Manager tray icon reports a red Stopped indicator.
5. Identify the data directory. The default for a named instance on a PCS 7 V6.0 install is C:\MSSQL$WINCC\Data. Confirm by looking at the startup parameters via Enterprise Manager > right-click server > Properties > General > Startup parameters. Typical values: -dC:\MSSQL$WINCC\Data\master.mdf and -eC:\MSSQL$WINCC\Log\ERRORLOG.
6. List the contents of the data directory and identify the suspect files:
   dir C:\MSSQL$WINCC\Data\*.mdf C:\MSSQL$WINCC\Data\*.ldf
   You should see master.mdf, mastlog.ldf, model.mdf, modellog.ldf, tempdb.mdf, templog.ldf, msdb.mdf, msdblog.ldf, plus the PCS 7 project databases such as CC_PCS7_01_08_14_18_31.mdf.
7. Move the suspect .mdf and .ldf of any Suspect database to a holding folder outside the data directory, e.g.:
   

   mkdir C:\MSSQL$WINCC\Data\_quarantine
   move C:\MSSQL$WINCC\Data\CC_PCS7_01_08_14_18_31.mdf C:\MSSQL$WINCC\Data\_quarantine\
   move C:\MSSQL$WINCC\Data\CC_PCS7_01_08_14_18_31_log.LDF C:\MSSQL$WINCC\Data\_quarantine\

   
   Do not delete yet. The files are still referenced by SIMATIC Manager and will be re-attached on the next project compile.
8. If master.mdf is also corrupt and you have a known good copy from a backup or from another ES that was running the same SP, replace it (see warning below):
   

   copy D:\Backup\master_good.mdf C:\MSSQL$WINCC\Data\master.mdf /Y
   copy D:\Backup\mastlog_good.ldf C:\MSSQL$WINCC\Data\mastlog.ldf /Y

   Warning: Replacing master.mdf with a copy from a different ES will desynchronize instance metadata (logins, server-level settings, linked server definitions). Only do this when no SQL backup exists and the alternative is full reinstallation.
9. Restart the SQL Server service from Service Manager. Watch the ERRORLOG with a tail:
   type C:\MSSQL$WINCC\Log\ERRORLOG | more
   Confirm the line SQL Server is now ready for client connections and the absence of any further 9003 / 9001 entries.
10. Re-open SIMATIC Manager, open the multiproject, and check every PC station. The OS application sub-tree should reappear within a few seconds as the connection to MSSQL$WINCC is re-established.
11. Open each OS in WinCC Explorer and trigger a full project compile (OS Project Editor > Close > Generate / Update). This re-attaches the project database files that were quarantined in step 7.

Recovery Procedure (Reinstalling SQL Server and WinCC)

If the master database is unrecoverable, no good backup exists, and SIMATIC Manager still shows no OS stations, the supported path is to rebuild the SQL instance and the WinCC installation. This was the final fix applied in the field case that produced the source log: "I fix problem with uninstall/install SQL and WinCC."

1. Back up the entire D:\PCS7\Project tree to external media. Also copy C:\MSSQL$WINCC\Data\*.mdf and *.ldf for every project database, in case the rebuild can be performed by re-attaching the files rather than restoring from project backup.
2. From Control Panel > Add/Remove Programs, uninstall Microsoft SQL Server Desktop Engine (WINCC) and WinCC in that order. Do not uninstall PCS 7 itself.
3. Delete (or rename) the leftover data directory C:\MSSQL$WINCC.
4. Reinstall the SQL Server instance using the PCS 7 V6.0 SP1 setup, choosing the Named Instance: WINCC and the same data paths as before.
5. Reinstall WinCC from the PCS 7 V6.0 SP1 setup.
6. Start SIMATIC Manager, open the project, and let the OS project editor recreate the project database files. Where prompted, point the re-creation at the original *.mdf/*.ldf copies that were backed up in step 1.
7. Recompile every OS and verify the alarm and tag databases are consistent by opening WinCC Explorer > Tag Management and Alarm Logging.

Verification

After either recovery path, perform the following checks before declaring the system healthy:

Prevention and Hardening

Several concrete measures reduce the likelihood of recurrence once the system is recovered:

• Audit third-party tools that touch the SQL instance. In the field case, the VersionDoc backup utility was creating a daily temp database that was not being dropped on backup completion. Configure the tool to remove its temporary database, or schedule a daily SQL Agent job to drop any database that has been in Suspect state for more than one hour:
  

  EXEC sp_executesql N'IF DB_ID(''VersionDoc_Temp'') IS NOT NULL
    BEGIN TRY
      ALTER DATABASE VersionDoc_Temp SET SINGLE_USER WITH ROLLBACK IMMEDIATE;
      DROP DATABASE VersionDoc_Temp;
    END TRY
    BEGIN CATCH
      PRINT ''Could not drop VersionDoc_Temp'';
    END CATCH';

• Exclude the SQL data and log directories from antivirus on-access scanning. Real-time scanning of *.mdf and *.ldf is a known contributor to torn writes. Add C:\MSSQL$WINCC\Data and C:\MSSQL$WINCC\Log to the AV exclusion list.
• Schedule a daily SQL backup of the system databases in addition to the PCS 7 project backup:
  

  BACKUP DATABASE [master] TO DISK = N'D:\SQLBackup\master_full.bak' WITH INIT;
  BACKUP DATABASE [msdb] TO DISK = N'D:\SQLBackup\msdb_full.bak' WITH INIT;

  
  A recent master backup is the single most important asset for avoiding a full reinstallation in the next failure.
• Move from Windows Server 2000 / SQL Server 2000. The PCS 7 V6.0 platform is itself long out of mainstream support. Plan a migration to at least PCS 7 V8.x or V9.x on a supported Windows Server / SQL Server pair; the disappearance symptom is much less common on SQL Server 2008 R2 and later because of improved recovery semantics and CHECKDB integration.
• UPS-protect the ES and configure Windows for Shut down: On battery, 5% rather than hibernate. Abrupt power loss is a frequent root cause of torn master pages.
• Set SQL Service startup to Automatic (Delayed Start) so the service starts after disk subsystems have stabilized after a reboot.

Frequently Asked Questions