Troubleshooting Ignition Gateway High CPU and Memory Usage

Which layer is actually saturated: the OS, the JVM, or the heap?

Follow the number before you chase it. On an Ignition 8.1.44 gateway running on Ubuntu 22.04.5 LTS, a CPU or memory figure passes through three layers. Linux accounts for the java process. The JVM reports its own view through the platform MX beans. Ignition republishes those same MX bean values as [System]Gateway/Performance/... tags, including [System]Gateway/Performance/CPU Usage, through the tag subsystem. Each layer answers a different question, so read the lowest one first.

CPU and memory often climb together because of garbage collection. When used heap approaches the max, the collector runs back-to-back cycles, reclaims little, and burns cores doing it. A gateway in that state shows high CPU, but the root cause is whatever is holding memory. When CPU is high and the heap sawtooth stays healthy, code or I/O is doing real work, and the heap is not the cause.

Check: locate the process and watch it at thread granularity.

ps -o pid,rss,vsz,pcpu,etime,cmd -C java
top -H -p <pid>
free -h

Then compare the heap trend on the gateway Status page. Use this decision to choose a path:

• If the floor of the heap sawtooth keeps rising, or used heap sits near max, follow the memory path. You need a heap dump.
• If the heap is flat and CPU is pegged, follow the CPU path. Thread dumps are enough.

Which capture method fits a production gateway?

A spike that has already passed leaves nothing behind. Gateway logs do not record thread stacks, and a thread dump or heap dump only shows the instant it was taken. After the fact, with no preparation, there is nothing to analyze. You have to arm a capture mechanism before the next event.

Automatic thread dumps help with pure performance problems. When high CPU comes from memory exhaustion, thread dumps mostly show the collector's victims. Without a heap dump, that class of problem is nearly impossible to solve.

To use the built-in option:

1. Confirm the gateway version on the Status overview. 8.1.44 is newer than 8.1.13, so the feature is present.
2. Open Config > Gateway Settings. Find the automatic thread dump properties and read their thresholds and output location in the Gateway Settings property reference of the Ignition User Manual.
3. Save, and note where the dumps are written.

For a heap-dump-capable Diagnostics Bundle, look up the startup flag on the Status page of the Ignition User Manual. Add it to the gateway launch configuration, then schedule the restart.

Check: the settings page shows automatic thread dumps enabled. If you chose the flag route, the gateway has restarted since you added the flag.

How do I build the MX bean watchdog script?

The watchdog reads heap usage and process CPU load directly from the JVM every second. When either value crosses a threshold, it writes a thread dump and, optionally, a heap dump. You can deploy it without restarting the gateway, which matters on a production site.

First, create four memory tags so you can tune the watchdog live without editing code:

Next, add a project library script named diagWatch in the project that hosts your gateway events:

from java.lang.management import ManagementFactory
from com.sun.management import HotSpotDiagnosticMXBean
from com.sun.management import OperatingSystemMXBean as SunOsMXBean

logger = system.util.getLogger(system.util.getProjectName() + '.diagWatch')

_mbs  = ManagementFactory.getPlatformMBeanServer()
_diag = ManagementFactory.newPlatformMXBeanProxy(_mbs, 'com.sun.management:type=HotSpotDiagnostic', HotSpotDiagnosticMXBean)
_os   = ManagementFactory.newPlatformMXBeanProxy(_mbs, 'java.lang:type=OperatingSystem', SunOsMXBean)
_mem  = ManagementFactory.getMemoryMXBean()

BASE = '[default]Diagnostics/'
TAGS = [BASE + 'DumpInhibitTS', BASE + 'CpuThreshold', BASE + 'HeapThreshold', BASE + 'HeapDumpEnable']
OUT  = '/usr/share/ignition/diag_dump_'

def heapFraction():
    u = _mem.getHeapMemoryUsage()
    if u.max > 0:
        return 1.0 * u.used / u.max
    return 0.0

def cpuFraction():
    v = _os.getProcessCpuLoad()   # 0.0-1.0, negative if unavailable
    return v if v >= 0 else 0.0

def check():
    now = system.date.now()
    inhibit, cpuTh, heapTh, heapEnable = [q.value for q in system.tag.readBlocking(TAGS)]
    if inhibit is not None and not now.after(inhibit):
        return
    cpu, heap = cpuFraction(), heapFraction()
    if cpu < cpuTh and heap < heapTh:
        return
    system.tag.writeBlocking([TAGS[0]], [system.date.addSeconds(now, 30)])
    fn = OUT + system.date.format(now, 'yyyyMMdd_HHmmss')
    system.file.writeFile(fn + '.threads', system.util.threadDump())
    logger.warnf('Diag dump cpu=%.2f heap=%.2f file=%s', cpu, heap, fn)
    if heapEnable and heap >= heapTh:
        _diag.dumpHeap(fn + '.hprof', False)

The design choices work as follows:

• The 30-second inhibit window stops the watchdog from writing a dump every second while the condition persists.
• The thread dump is written before the heap dump. If the heap dump fails or stalls badly, you still have the stacks.
• The timestamped file name prevents collisions. The JVM refuses to overwrite an existing .hprof file.

Check: save the project and confirm the gateway log shows no script compile errors for diagWatch. Confirm the four tags read with good quality.

How do I wire the timer event, and why not a tag change script?

The [System]Gateway/Performance/CPU Usage tag gets its value from the same MX bean the script reads. A tag change event on that tag adds the tag subsystem, and its overhead, between the JVM and your trigger. That is the subsystem you might be diagnosing. Read the bean directly from a timer instead.

1. Open Gateway Events > Timer and add a new timer.
2. Choose dedicated threading so a busy shared timer thread cannot delay or starve the watchdog.
3. Set the script body to diagWatch.check().
4. Save the project.

Check: force a trigger without affecting production.

1. Leave HeapDumpEnable at false.
2. Write 0.01 to CpuThreshold.
3. Restore CpuThreshold to 0.95.

If no file appears, check that the gateway service user can write to /usr/share/ignition. If it cannot, move OUT to a directory it owns.

How should the heap dump branch be gated?

Taking a heap dump is extremely heavyweight. The JVM stops the world while it walks the whole heap and writes it to disk. During that stall, the gateway stops servicing drivers, clients, and scripts. Treat HeapDumpEnable like exploratory surgery: set it true only while you are actively tracking a memory problem, and set it false as soon as you have the file.

A heap dump file is roughly the size of the used heap. Before you enable heap dumps, run df -h /usr/share/ignition and confirm there is free space for at least two dumps at the configured max heap. Two dumps taken minutes apart are what expose a leak.

Check: HeapDumpEnable is true only during the tracking window. Disk headroom exceeds twice the max heap. Operations knows a stall of several seconds is possible when the threshold trips.

How do I read the thread dumps from a CPU event?

A single thread dump shows where every thread was at one instant. What you are looking for is persistence: the same thread RUNNABLELine the dumps up side by side and match threads by name.

Check: you can name one thread, or one group of threads, that is RUNNABLE in the same code in at least two consecutive dumps, and that code maps to a specific project resource or driver.

How do I read the heap dump from a memory event?

Copy the .hprof off the gateway before analysis. Opening it needs RAM comparable to the dump size, and you do not want to spend the gateway's memory on it. Load it into a JVM heap analyzer and work from retained size, not shallow size. The object that dominates retained memory is what keeps everything else alive.

1. Open the dominator view and sort by retained size.
2. Expand the top entries until you reach a collection (list, map, queue) or cache that holds a large share of the heap.
3. Trace its path to GC roots. That path names the owner, for example a project library module, a subsystem cache, or a session.
4. Open the second dump and compare. The collection that grew between dumps is the leak. Collections that stayed large but constant are working sets that need a larger heap, not a code fix.

Project library scripts persist between timer executions. A module-level list or dictionary that is appended to and never pruned is a common self-inflicted leak on gateways that run scripts.

Check: you can point to one growing structure and the code path that feeds it. Or you have shown that retained memory is stable, which means the heap is too small for the working set.

How do I verify the fix end to end?

1. Deploy the code or configuration change you traced from the dumps.
2. Set HeapDumpEnable to false. Leave the watchdog running thread-dump-only, with CpuThreshold and HeapThreshold at 0.95.
3. Watch the heap trend on the Status page through at least one full production cycle, including shift changes, report runs, and peak client counts. The sawtooth floor should return to the same level after each major collection, not ratchet upward.
4. Re-run top -H -p <pid> at peak load. No single thread should hold a core continuously.
5. Move the existing .hprof files off the gateway into controlled storage, or delete them. They contain in-memory data from the gateway.
6. At the end of the cycle, run ls -lh /usr/share/ignition/diag_dump_*. No file should carry a timestamp newer than the deployment. That empty list is the pass condition.

FAQ