OPC UA Server High CPU: Fix ClearChangeMasks/QueueValue

A 5-minute profile of an OPC UA server with 200 subscribed tags puts ClearChangeMasks at 36.59%, OnMonitoredNodeChange at 36.53% and QueueValue at 36.49% of CPU. Those three numbers are one call chain, not three separate costs. The server application writes a value, marks the node changed, and the stack pushes a copy into every monitored item queue watching that node. The load scales with writes per second, not with tag count alone. Scaling to 2000+ tags multiplies it by ten.

Where does the CPU go between the application write and the client queue?

The data path on the server side is:

1. Application code assigns a new value to a variable node.
2. The application calls ClearChangeMasks on that node, which raises a change notification.
3. OnMonitoredNodeChange runs once per monitored item attached to the node.
4. QueueValue copies the value, status and timestamps into that item's notification queue.
5. The publish cycle drains the queues into a Publish response toward the client (UaExpert here).

The three profiler percentages are nearly identical because each function is inclusive of the one it calls. The leaf, QueueValue, carries almost all the cost. Time is spent on queue allocation, value copies and locking. Nothing in this chain checks whether the value actually changed. Every ClearChangeMasks call becomes a queued notification.

Estimate the call rate with this formula (assumption: one monitored item per tag, every tag written every cycle):

QueueValue calls/s = tags x (1000 / write_cycle_ms) x monitoring_clients
200 tags,  100 ms cycle, 1 client = 2,000 calls/s
2000 tags, 100 ms cycle, 1 client = 20,000 calls/s

Check before moving on: confirm the profiler shows the same three frames in one stack, with QueueValue as the deepest. If a different frame sits under QueueValue, the cost is in that frame (allocation, lock contention), and the fixes below still apply because they cut the call count.

How do you measure write rate against real change rate?

Add two counters in the node manager write routine: writesAttempted and valuesChanged.

Check: the ratio is known per tag group before any configuration edit. Without it, config changes are guesses.

How do you stop unchanged values from reaching the monitored item queue?

Push only changes into the stack. Compare the new value with the current node value and call ClearChangeMasks only when they differ. Apply a source-side deadband to floating-point tags so instrument noise does not register as a change.

if (!newValue.Equals(variable.Value))   // arrays: element-wise compare
{
    variable.Value = newValue;
    variable.Timestamp = DateTime.UtcNow;
    variable.ClearChangeMasks(SystemContext, false);
}

Update the whole tag batch inside one lock scope, then raise the change masks. Do not take and release the node manager lock per tag. Set the status code and source timestamp from the field data, not from the write moment, so a client can tell a stale value from a fresh one.

Check: re-run the counters. QueueValue call rate must now track valuesChanged, not writesAttempted.

Which client and server settings cut notification volume?

The client requests the publishing interval, sampling interval, queue size and data change filter. The server clamps those requests to the limits in the config file. Read the supplied config against the request path:

On the client side, set the monitoring mode to report data changes with a deadband on analog items, a publishing interval no faster than the operator needs, and a small queue with discard-oldest.

Check: read back the revised values the server returns in CreateMonitoredItems and CreateSubscription responses. The revised publishing and sampling intervals must match what you intended; if not, the config clamp is overriding the request.

How do client-to-server writes fit into the 2000+ tag scenario?

Client writes enter through the Write service, not the change-notification path. Send one Write request carrying an array of WriteValue entries for a tag group instead of one request per tag. Each request costs a secure-channel decrypt/verify, a session lookup and a response encode, so batching removes per-request overhead. Apply the same change gate to values coming from the client: an unchanged write that still calls ClearChangeMasks fans out to every other subscriber of that node.

Check: with the client writing a 2000-item batch, confirm from the counters that only the changed nodes raise change notifications on the server.

What proves the fix under the 2000+ tag load?

1. Record the baseline: 200 tags, 5-minute profile, ClearChangeMasks 36.59%, OnMonitoredNodeChange 36.53%, QueueValue 36.49%.
2. Apply the change gate, deadband, revised sampling groups and client queue settings, then repeat the identical 5-minute profile at 200 tags. The three frames must drop well below baseline.
3. Compare QueueValue calls per second against valuesChanged per second from the counters. They must match.
4. Scale to 2000+ tags with realistic change behavior and profile again. Record process CPU and the share held by the three frames.
5. Run a worst-case step where every tag changes each cycle. This gives the real ceiling of the server; if CPU is still unacceptable there, cut the fast subscription's item count or lengthen its publishing interval.
6. At the client, confirm no BadTooManyPublishRequests-style backlog or queue-overflow status bits appear on items, and that item timestamps advance at the expected rate.