Fixing Ignition Gateway JVM Hang and Wrapper Restarts

The restart is not a crash. A native wrapper process launches the JVM as a child, then exchanges periodic ping messages with a responder thread inside it over a local socket. When a reply does not arrive inside the ping timeout, the wrapper logs JVM appears hung: Timed out waiting for signal from JVM. Restarting JVM. and kills the process. The gateway was alive the whole time. It just could not answer.

So the question is never "why did Ignition crash." It is: which hop in the ping path stopped moving, and for how long. Layer one first — the physical host and its scheduler — then the JVM.

Where does the heartbeat stop?

Four hops carry the signal, and each one leaves a different fingerprint in wrapper.log.

That last line is the branch point. A garbage collection pause inside the JVM freezes Java threads only; the native wrapper keeps running and keeps counting. When the wrapper reports that it received no CPU for 14 seconds, the whole guest was descheduled. The stall is below Java.

Check 1: Did the wrapper itself lose CPU?

Grep the wrapper log for has not received any CPU time. Two outcomes:

1. Present. The guest froze. Go to Check 3 (hypervisor) before touching any Java setting. Nothing you change in ignition.conf fixes a VM that is not being scheduled.
2. Absent, only clock-drift warnings and hangs. The JVM stopped answering while the OS kept running. Go to Check 2 (heap and GC).

Both signatures appear in this record, which means both paths are live and both need clearing. A stall past 32 seconds is in the same monitoring set. Nothing inside a healthy gateway pauses for 32 seconds.

Check 2: Is the heap the stall, or the symptom?

Read the two heap entries in ignition.conf and the machine's physical RAM:

wrapper.java.initmemory=32768   # MB, current
wrapper.java.maxmemory=65536    # MB, current
# machine RAM: 100 GB

A 32 GB initial and 64 GB maximum is a split configuration, and it fails in production for two reasons. First, the JVM expands the heap on demand, so the OS must commit another 32 GB at the worst possible moment — under peak tag load — and the JVM dies if that memory is not available. Java does not hand pages back to the OS in any way you can rely on, so make it claim what it needs at startup, where the failure is loud and immediate. Second, the collector's stop-the-world work scales with the live set. A full collection on a 64 GB heap can run for tens of seconds, and that is exactly the duration that trips the wrapper ping timeout.

Decide the value from the memory trend on Status > Performance, not from the RAM the machine happens to have:

Large MQTT and native tag counts drive allocation, historian writes, and store-and-forward object churn. Those show up as collection frequency. A bigger ceiling lengthens each pause and does nothing for frequency.

Check 3: Is the hypervisor over-committed?

Pull these from the host, per-VM, over the same window as the restarts. Guest-side counters will not show any of it — inside the guest, stolen time simply looks like time that never happened.

Never over-commit host resources beneath a production gateway: allocate only what the host physically has, minus a reserve for the hypervisor itself. Inductive Automation publishes a VM configuration white paper covering these settings — obtain it from Inductive Automation and check the VM against it item by item, including the case where a support review has already passed on the obvious settings.

Check 4: What else blocks the JVM for seconds at a time?

On Windows Server, real-time antivirus scanning of the Ignition directories is the recurring offender. The internal database, the tag history store, and the wrapper logs are written constantly; a scanner in the write path adds latency to every one of them, and a scheduled full scan produces an I/O storm long enough to hang the ping responder. Exclude the Ignition install directory, the data directory, and the Java executable from real-time scanning, and move scheduled scans out of production hours.

Storage latency is the same failure with a different cause. Read guest disk latency for the volume holding the Ignition data directory during a stall window. Sustained latency in the hundreds of milliseconds will stall internal database writes on its own.

One thing not to do: raising the wrapper ping timeout. It suppresses the restart, not the stall. The gateway is unresponsive to clients, drivers, and MQTT for the entire pause either way — a longer timeout only converts a 50-second detection into a longer silent outage.

How do you apply the fix and prove it held?

1. Set the two heap entries equal in ignition.conf, at the size chosen from the memory trend. If the working floor sits well under 32 GB, use wrapper.java.initmemory=32768 and wrapper.java.maxmemory=32768 — a smaller heap collects faster. If the gateway genuinely needs the larger ceiling, set both to 65536 and confirm the remaining ~36 GB covers the OS and every other service on the machine.
2. Add unified GC logging to the additional JVM parameters block before restarting: -Xlog:gc*,gc+pause on a Java 11 or newer JRE, -XX:+PrintGCDetails on Java 8. Without it, the next stall is unattributable.
3. Restart the gateway once in a maintenance window. Confirm the process commits the full heap at startup rather than growing into it.
4. Set a full memory reservation for the VM and disable ballooning for it.
5. Resize vCPUs to fit inside one physical socket and clear host over-commit.
6. Apply antivirus exclusions for the Ignition install and data directories and the Java executable; reschedule full scans.
7. Move snapshot and backup windows off production hours, or switch to a method that does not stun the guest.

Verify over a full production week that includes one backup cycle: zero JVM appears hung entries and zero has not received any CPU time entries in the wrapper log; zero ClockDriftDetector warnings, since the detector only logs above the 1000 ms allowance; GC log maximum pause under one second with no full collection in the tens of seconds; host CPU ready near zero and ballooned/swapped bytes at zero for the guest. Then read the gateway uptime counter on the status page and confirm it has passed the previous mean time between restarts without resetting.

FAQ