Resolving SIMOTION D445 Task Overflow SHUTDOWN Triggered by Large OPC Profile / Cam Data

1. Problem Description

A SIMOTION D445 motion controller equipped with multiple SINAMICS drive axes stops with a hard SHUTDOWN when a third-party SCADA pushes a large profile or cam data set over an OPC connection (Ethernet). The runtime stops on the controller and the SIMOTION diagnostic buffer shows:

• Tolerated time overflow in the task
• STOP by execution system, cause: Timeout

The error is reproducible but intermittent: it does not occur on every transmission. Sending a moderate-size cam (a few hundred points) is fine, but pushing a multi-kilobyte cam buffer over OPC UA/DA while the controller is also executing the IPO task repeatedly pushes the affected task past its watchdog. The integrator has applied the latest Hotfix pack (SP5 HF15) but the symptom persists.

2. Affected Hardware, Firmware, and Software

The issue is not hardware-defective: the diagnostic buffer almost always pinpoints one of the cyclic tasks running past its TimeOut multiplier. The hardware watchdog (CPU level) and the SIMOTION task watchdogs (system level) are independent paths and both can force a SHUTDOWN.

3. How SIMOTION Tasks Are Scheduled

SIMOTION D controllers run an execution system composed of several fixed priority tasks. A task overflow in any of them can propagate to STOP/SHUTDOWN:

Two watchdogs protect each task: the configurable TimeOut and the system TaskTime (per-cycle time). If a single cycle exceeds TimeOut × Factor (default 3× to 5×), the SIMOTION execution system logs Tolerated time overflow in the task and, after the configured number of tolerated overflows, raises STOP by execution system. The hard SHUTDOWN is what the field sees.

4. Root Cause

The dominant root cause observed in D445 systems exchanging large cam/profile data over OPC is a wrong task assignment for the receive path. The OPC subscription / data pump logic is typically placed in the BackgroundTask while the consuming application code (cam activation, axis motion, profile interpretation) is placed in the IPO-synchronous task. Both tasks overlap when a bulk OPC write is processed, producing:

1. The BackgroundTask absorbs tens of milliseconds parsing/decomposing the incoming cam (memory copy, normalization, checksum).
2. The IPO-synchronous task consumes the cam and simultaneously computes IPO outputs, forcing both to run concurrently.
3. The sum exceeds the IPO task's TimeOut factor → Tolerated time overflow → SHUTDOWN.

Secondary contributors commonly identified in the diagnostic buffer:

• Loop structures in the BackgroundTask (e.g., WHILE loops until empty buffer) that never yield.
• OPC server sending bursts faster than the controller can drain (no flow control / acks).
• Cam sizes exceeding 16 KB per axis when multiple axes are updated simultaneously.
• Motion commands issued without nextCommand := IMMEDIATELY, blocking the IPO task.
• Heap fragmentation on the global data block after long uptimes.

5. Identifying the Offending Task

5.1 Reading the Diagnostic Buffer

Connect SCOUT to the D445 over Ethernet (Industrial Ethernet via TCP/UDP ports 102, 161, 34964). Navigate to:

Project → D445 → Diagnostics → Diagnostic Buffer

Open the entry STOP by execution system, cause: Timeout and switch to the Extended Info tab. The overflow task name is shown there as the active task ID at the moment of the stop (e.g., IPOSynchronousTask_1). Field experience shows the IPO-synchronous task is by far the most common offender in the profile-cam scenario.

5.2 Task Runtime Trace

Enable task runtime tracing without stopping the machine:

1. In SCOUT, right-click the D445 node → Target system → Commissioning → Task Trace.
2. Select IPOSynchronousTask_1 and BackgroundTask simultaneously.
3. Capture for at least one minute at the same SCADA burst that triggers SHUTDOWN.
4. Read the maximum cycle time per task (column t_max). The IPO task will show t_max approaching or exceeding the TimeOut value.

5.3 CPU Utilization Bars

The CPU load panel of SCOUT (Target system → CPU utilization) shows IPO load percentages. Sustained values above 85% on the IPO bar during the bulk OPC write confirm CPU saturation on the IPO task.

6. Solution

The fixes below are ordered by least invasive to most invasive; apply them sequentially and verify at each step.

6.1 Solution A – Move Bulk Data Reception to a MotionTask

Create a dedicated MotionTask (e.g., MotionTask_2) to receive, validate, and stage cam data. The BackgroundTask should only hand off the raw buffer; the MotionTask does the parse, normalization, and copy into the cam runtime block.

Project tree:

• Configuration → Task configuration → MotionTask_2 → set cycle time to IPO * 4 (e.g., 16 ms on a 4 ms IPO).
• Watchdog → TimeOut = 500 ms, ToleratedOverflows = 10.

ST pseudocode (BackgroundTask — minimal, non-blocking):

// BackgroundTask – only enqueue
IF bOpcDataReady THEN
    bOpcDataReady := FALSE;
    // copy raw buffer pointer into staging queue (no parsing here)
    StagingQueue[writeIdx] := pOpcCamBuffer;
    writeIdx := (writeIdx + 1) MOD 16;
END_IF;r>

ST pseudocode (MotionTask_2 — full processing):

// MotionTask_2 – all heavy work runs here
IF StagingQueue[readIdx] <> 0 THEN
    pBuf := StagingQueue[readIdx];
    StagingQueue[readIdx] := 0;
    parseCamPoints(pBuf, iPointCount);   // bulk math
    validateChecksum(pBuf);
    memcpy(gCamRuntime[iAxis], pBuf, iPointCount * 16);
    // Trigger axis cam activation
    _enableAxis := TRUE;
END_IF;r

6.2 Solution B – Increase IPO TimeOut and Tolerated Overflows

If the architecture must keep heavy work on the IPO task (legacy code, vendor constraint), widen the IPO watchdog temporarily:

6.3 Solution C – Use nextCommand := IMMEDIATELY on Motion Commands

If the IPO task issues motion commands synchronously (e.g., _cam_enable(...) with the default command issuance mode), they block until accepted. Add immediate linking:

// Inside IPO – example axis cam activation
_ax1.axisType := VIRTUAL;
_ax1.setpointAxis := TRUE;

// Issue motion command with IMMEDIATELY linking
ret := _cam_enable(
    axis := _ax1,
    cam := gCamRuntime[_ax1],
    masterSetpoint := 0,
    slaveSetpoint := 0,
    nextCommand := IMMEDIATELY
);r

Every _* motion command (e.g., _pos, _enable, _cam_enable, _stop) on the D445 exposes the nextCommand parameter. Using IMMEDIATELY releases the IPO scheduling slot at once.

6.4 Solution D – Remove Loop Constructs from BackgroundTask

Replace any WHILE / FOR loops inside the BackgroundTask with state machines. A loop that runs while the OPC buffer is being read can hold the BackgroundTask for hundreds of milliseconds while the IPO task starves. Pattern:

CASE eParseState OF
    STATE_IDLE:
        IF bOpcDataReady THEN
            eParseState := STATE_PARSE;
            iParseCursor := 0;
        END_IF;
    STATE_PARSE:
        // process one chunk per call (max ~256 points per pass)
        parseChunk(pOpcCamBuffer, iParseCursor);
        IF iParseCursor >= iPointCount THEN
            eParseState := STATE_VALIDATE;
        END_IF;
    STATE_VALIDATE:
        validateChecksum(pOpcCamBuffer);
        eParseState := STATE_PUBLISH;
    STATE_PUBLISH:
        memcpy(gCamRuntime[iAxis], pOpcCamBuffer, iPointCount * 16);
        eParseState := STATE_IDLE;
        bOpcDataReady := FALSE;
END_CASE;r

6.5 Solution E – Throttle the OPC Server

On the SCADA side, configure the OPC UA/DA write rate so that no more than 2–4 KB is written to the controller per IPO cycle. Add an explicit DataChangeFilter (OPC UA) or an item-level update rate (OPC DA) such that profile data is sent in incremental segments. Use the SIMOTION SIMOTION_variable_write interface or a CFC block that acks each chunk.

7. Firmware Bug Notice

On SIMOTION V5.4 and V5.5 (pre-SPE), a bug in the OPC-UA server extension module (CPSU) caused the BackgroundTask to lock for up to 800 ms when receiving certain binary structured types. The fix is bundled in:

Even after applying the latest Hotfix, the architecture-level fix (Solution A) is still mandatory because the OPC bug only addresses the worst-case lock duration, not the architectural problem of moving heavy data in the IPO task.

8. Verification Procedure

1. With Solutions A–D applied and the controller in OPERATE, replay the largest cam upload via the same OPC test tool used in production.
2. Monitor the diagnostic buffer for 30 minutes. No STOP by execution system entries should appear.
3. Confirm IPOSynchronousTask_1 t_max stays below the configured TimeOut (use SCOUT Task Trace).
4. Verify each axis cam activation via the SCOUT axis control panel: cam engages, axis follows, no jitter on the trace.
5. Stress test: trigger 10 consecutive cam uploads in 30 seconds. The controller must remain in RUN.
6. Repeat on the redundant D445 if your topology has two controllers running in parallel.

9. Performance Comparison of Solutions

10. Best Practices Checklist

• Never execute I/O heavy, data-heavy, or memory-copy-heavy code inside the IPO or servo-synchronous tasks.
• Use dedicated MotionTasks for cam/profile loading. Place the MotionTask cycle at 4×–8× IPO.
• Keep the BackgroundTask for non-real-time housekeeping only — log rotation, OPC housekeeping, alarm dispatch.
• Always use nextCommand := IMMEDIATELY on every motion command in the IPO path.
• Configure ToleratedOverflows ≥ 5 on every task to absorb one-time bursts.
• Set the SIMOTION _executeSystemStop reaction so that a single task overflow does NOT escalate to SHUTDOWN immediately; instead, log and degrade.
• Verify cam point counts and RAM headroom before deployment. A 16 KB cam with 1024 points uses 16 KB × 2 (double buffer) per axis.
• Keep at least one free MotionTask slot for emergency data loading.
• Document the TimeOut and Factor values for each task inside the project documentation so that future integrators do not reset them to default.
• Plan remote diagnostics: enable the SIMOTION web server and OPC-UA diagnostic namespace so engineers off-site can pull the diagnostic buffer without a SCOUT direct connection.

11. Quick Parameter Reference

12. References

• SIMOTION D4x5 firmware update V5.4 SP1 HF7
• SIMOTION D4x5 firmware update V5.5 SP2 HF4
• SIMOTION D4x5 Product Manual (Function Manual)
• SIMOTION SCOUT TIA Programming and Operating Manual
• SIMOTION Motion Control – Programming Reference (MCC, _cam_enable)
• SIMOTION OPC UA Server – Configuration Manual

FAQ