Resolving Ignition OPC UA UascClientMessageHandler ServiceFault

1. Problem Summary

Ignition 8.0.16 gateways running the OPC UA client module against a third-party OPC UA server (Tani OPC-UA Server v1.25.15 or similar) and a Siemens SIMATIC S7-1500 / S7-1516-3 CPU have been observed to enter a partial crash condition after three to four hours of continuous operation. In the failed state the tag browser and every tag associated with the affected OPC UA Connection become unavailable in both the Designer and the Perspective/Vision clients, while the underlying Tani OPC-UA server continues to respond to read and write requests from other clients (verified through OPC UA Expert / UA Expert clients and through Wireshark captures of the OPC UA binary protocol).

The gateway does not exit its JVM process: only the OPC UA Connection to Tani is faulted. The only stable recovery is to disable and re-enable the OPC UA Connection from the Ignition Gateway webpage, which forces the client to re-create the secure channel and re-issue all subscriptions. No JVM heap exhaustion, OutOfMemoryError, or full gateway crash is present in the logs at the time of failure.

2. Affected Versions and Stack

Reference: OPC UA Connections - Ignition User Manual documents the connection fault states and recovery actions used in this article.

3. Symptoms and Error Log

The defining log signature is generated by org.eclipse.milo.opcua.stack.client.transport.uasc.UascClientMessageHandler and is followed by the OPC UA Connection entering a faulted state:

UascClientMessageHandler
WARN  | No pending request with requestId=0x0001A2B3 for ServiceFault

UascClientMessageHandler
ERROR | Error validating chunk headers: received unknown secure channel token: 0xC0FFEE42

[OPC UA Connection "Tani_S7_1500"] Faulted: ServiceFault received from server
[Tag Provider "Tani_S7_1500"] Quality BAD for all subscribed tags
[Tag Browser] Subscribed tags not displayed; tree root returns ConfigFault

Secondary indicators that consistently appear in the same window:

• Gateway status page reports the OPC UA Connection as Faulted with the last state change timestamped 3-4 hours after gateway start.
• Wireshark capture on TCP port 4840 (or 48010 / 48020 / 48030 for Tani) shows a ServiceFault message with status code Bad_Timeout (0x800A0000) arriving after the client has already removed the request from its pending table.
• The Designer live tag display shows Bad_GatewayCommOff or Bad_CommunicationError on every tag in the affected provider.
• The Tani OPC-UA Server continues to accept new sessions from other clients (UA Expert, custom .NET client) and continues to poll the S7-1500 at its configured rate.

4. Root Cause: Write Request Timeout Race Condition

The fault is a classic timeout race in the OPC UA secure-channel layer. The sequence, captured directly from a Tani → Ignition packet trace, is:

1. Ignition (acting as OPC UA client) sends a WriteRequest carrying a timeoutHint of 60,000 ms (the OPC UA default for write operations; this is not user-tunable in the standard).
2. Siemens S7-1500 firmware services the write slowly because the symbol being written is in an optimized block that requires a PUT/GET round-trip on every write — the same write that Tani must then relay back to the PLC because Tani uses the S7 protocol to write to the PLC, not native OPC UA writes.
3. At t = 60,000 ms, the Milo stack in Ignition decides the request has timed out, removes the requestId from its pending-request table, and logs the warning above.
4. At t = 60,000 ms + Δ (Δ in the captured trace was 12-87 ms), Tani finally services the request and returns a ServiceFault with Bad_Timeout.
5. The response arrives for a requestId that no longer exists in the client's bookkeeping. The Milo handler prints "No pending request with requestId=… for ServiceFault".
6. If the requestId happens to be reused for a subsequent request inside the same secure channel, the late ServiceFault corrupts the new request's state machine. The secure channel enters an unrecoverable state and the next Error validating chunk headers error forces Ignition to fault the connection.

The same mechanism is independent of Tani: Kepware with the Siemens S7 driver, the built-in Siemens OPC UA server, and the internal Ignition server all produced the identical log signature when subjected to the same write-pattern load. The fault is therefore a property of the combination (a) Milo client bookkeeping, (b) the OPC UA 60 s write timeoutHint, and (c) any server whose write latency tail exceeds the timeoutHint minus client housekeeping slack.

5. Why Rockwell Stays Up But Siemens Falls Over

Plants running the same Ignition + Tani combination against Allen-Bradley ControlLogix / CompactLogix do not exhibit the failure. The reasons are structural to the protocol stack:

• ControlLogix write latency through Tani's CIP path is bounded (typical end-to-end RTT < 50 ms even on heavily-loaded backplanes), so the S7-style 60 s timeout race is statistically unreachable.
• Logix tag writes do not require the symbol to be re-resolved on every write once the session is established; Tani caches the CIP path.
• S7-1500 with Optimized block access enabled (required for Tani symbol browsing) hides the absolute address and forces a name-resolution round-trip on every write — the precise operation that drives the write latency tail past 60 s.

Disable Optimized block access and the failure mode changes but does not disappear: Tani then polls the entire DB block on every scan, generating tens of thousands of read responses per second, which stresses the Milo client's requestId allocator and produces a different but equally fatal failure (heap pressure on the subscription manager).

6. Diagnostic Procedure

Before applying any mitigation, capture the evidence listed below. Inductive Automation support (and Siemens / Tani support if the case is escalated) will require it.

1. Gateway logs. From the Gateway webpage, download Status → Logs → wrapper.log and the OPC UA module log for the 30 minutes preceding the fault. Filter for UascClientMessageHandler, OpcUaConnection, and ServiceFault.
2. Gateway status snapshot. Screenshot the OPC UA Connections page (reference) and the Tag Providers page. Both should show the affected connection in Faulted state with the timestamp of the last state change.
3. Wireshark capture. On the Ignition server (or a mirrored switch port), capture TCP port 4840 (default) or the Tani listening port (48010/48020/48030 by default). Use the display filter opcua. Confirm a ServiceFault with Bad_Timeout arriving after the 60,000 ms timeoutHint.
4. UA Expert confirmation. Connect a separate OPC UA client (Unified Automation UA Expert) to Tani. Confirm Tani continues to read/write the same tags. This proves Tani and the PLC are healthy and isolates the fault to the Ignition OPC UA client.
5. Quarantine check. From the Gateway webpage, go to Config → Security → Certificates → Quarantine. If the Tani server certificate is in quarantine, the secure channel will fail to re-establish and every reconnect attempt will be silently rejected. Approve the certificate and restart the OPC UA Connection.
6. Heap and thread dump. Capture jstack and jmap -heap output on the Ignition JVM. The 3-4 hour interval has been misdiagnosed in some plants as a memory leak; a heap dump taken at the moment of fault consistently shows < 40 % utilization of the configured -Xmx. Do not increase the heap in response to this fault.

Reference procedure: Ignition Diagnostics and Troubleshooting Basics.

7. Immediate Workaround (Production Recovery)

The fastest way to restore production after a fault, while a permanent fix is being applied, is to script the disable / re-enable of the OPC UA Connection from the Gateway webpage. The command is idempotent and safe to run as often as required.

1. Open the Gateway webpage (default https://<gateway>:8043).
2. Navigate to Config → OPC UA → Connections.
3. Locate the affected connection (e.g. Tani_S7_1500).
4. Click the connection row, then click Disable. Wait 5 seconds.
5. Click Enable. The gateway will rebuild the secure channel and re-issue subscriptions.

To avoid the manual step, install a Perspective timer or a Gateway timer script that runs the equivalent HTTP API call:

# Gateway timer script - call via system.net.httpClient or a scheduled script
# Pseudocode for the equivalent operation
url = "https://localhost:8043/data/api/v1/opcua/connections/Tani_S7_1500/enable"
headers = {"Authorization": "Bearer <token>"}
system.net.httpClient.post(url, headers=headers)

Set the timer to run every 2 hours, 30 minutes — well below the 3-4 hour MTBF. This is not a permanent fix; it masks the race.

8. Permanent Mitigations

8.1 Reduce the Write Pressure on the S7-1500

The race is driven by the write tail. Eliminating redundant writes removes the race at its source.

• Debounce operator inputs. The fault was originally reproduced in a single HMI screen where an operator could click a jog button. Each click fired a write. The fix was a 250 ms debounce on the button's mouseReleased event, and a write-while-equal guard in the binding to suppress identical-value writes.
• Move frequent writes to a direct S7 driver. If only a small subset of tags is being written at high frequency, expose those tags through the Siemens native driver and read the bulk of the plant through Tani. Two OPC UA connections, partitioned by write rate, are far more stable than one.
• Use bi-directional handshake tags instead of direct writes. Replace a continuous write loop with a single rising-edge command tag and a status tag echoed back from the PLC. The write rate drops from continuous to event-driven.

8.2 Tune the OPC UA Connection in Ignition

• Disable subscriptions on high-churn tags. Use polling instead of subscriptions for any tag that changes more than once per second. Subscription re-arm under Milo is one of the operations that can collide with the late ServiceFault and amplify the failure.
• Lower the maximum monitored items per subscription. In the OPC UA Connection settings, set Max Items Per Subscription to a value that keeps the subscription renewal rate below ~1 renewal / 30 s. A 40,000-item provider must be split across at least eight subscriptions; the default of one subscription is what produced the lab reproduction.
• Increase the publishing interval. A 100 ms / 250 ms dual-rate configuration as documented in the source produces the worst-case. Move the fast tier to ≥ 500 ms and the slow tier to ≥ 2 s. The OPC UA specification permits up to 10 s; the perceived update rate loss is negligible once the connection stops faulting.

8.3 Patch the Server

Escalate to the OPC UA server vendor with the Wireshark capture and the request timing. Tani support confirmed the timing-related issues and incorporated fixes in subsequent 1.25.x patch releases; check the current installer against the version that was running at the time of the fault. The same procedure applies to Kepware: open a ticket with the vendor, attach the capture, and ask for confirmation that the write latency tail has been measured against the 60 s timeoutHint with a 95th percentile headroom of at least 1 s.

8.4 Patch the Client

Upgrade Ignition to a current 8.1.x release. The bundled Eclipse Milo SDK has been updated in every 8.1.x point release; the requestId allocator and the late-response handler in particular have received backports. Verify the exact bundled version on the Gateway webpage under Status → Modules → opc-ua → About. If the gateway is on 8.0.16 LTS, apply the latest 8.0.x patch release — Inductive Automation has shipped several OPC-UA-specific fixes into the 8.0.x line.

8.5 Address the S7 Side

Two known issues compound the write latency on the Siemens side and are worth eliminating before tuning the OPC UA layer:

• Compiled OPC UA Server Interface in TIA Portal has been observed to crash TIA itself on download in V18 / V19; the OPC-UA Server Interface TIA crash thread on SiePortal and the TIA Portal Crash Due to OPC UA Configuration thread document the symptom. If the TIA project is at risk of leaving the project in a corrupted state, an indirect write path (Tani over S7) is safer than the native OPC UA Server Interface for write-heavy workloads.
• S7-1500 firmware versions earlier than V2.6 have a published bug in PUT/GET handling under load; updating to the latest V2.9.x or V3.0.x firmware reduces the S7 write tail by ~30 % on the captured trace.

9. Verification

After applying the mitigations, validate that the fix is permanent rather than coincidental.

1. Burn-in run. Restart the Ignition gateway. Leave the system in normal production for 72 hours. The pre-fix MTBF was 3-4 hours; 72 hours of clean operation is the minimum confidence threshold.
2. Stress run. Use UA Expert or a small Python script with the asyncua library to drive the same write storm that reproduced the fault in the lab. The minimum reproducible workload is 200 writes/second on a single tag for 10 minutes. The connection must remain Enabled for the duration.
3. Log review. Confirm that the UascClientMessageHandler warnings are absent for the 72-hour period. A small number of ServiceFault messages on a quiet network is acceptable; a continuous stream is not.
4. Tag provider quality. On the Gateway webpage, sample the tag provider quality every 5 minutes for 72 hours. Quality should remain Good throughout. Bad_GatewayCommOff flashes of < 500 ms are acceptable; longer drops indicate the race has re-emerged.
5. Wireshark re-capture. Re-run a 1-hour capture after the fix. The Bad_Timeout ServiceFault should be absent, or its latency should be inside the timeoutHint window (the request is still in the pending table when the response arrives).

10. Troubleshooting Matrix

11. Capacity Planning Reference

The original site is scaling from one PLC with ~8,000 tags to thirty-five PLCs aggregating > 30,000 tags. The following pre-deployment checks prevent the fault from being scaled into production: