Frequent restart of the Ignition services

JayR · July 9, 2025, 7:52pm

Has anyone else experienced frequent Ignition gateway restarts?
We’re running a large number of MQTT and Ignition tags, and our server is hosted in a virtual machine (VM).

Just before the most recent restart, I reviewed the log files and noticed the following (see below). I'm wondering if this could be related.

Any insights or similar experiences would be greatly appreciated.

Thanks in advance!

Log 1

INFO | jvm 4 | 2025/07/09 10:12:58 | W [ClockDriftDetector] [10:12:58.230]: Clock drift, degraded performance, or pause-the-world detected.
INFO | jvm 4 | 2025/07/09 10:12:58 | Max allowed deviation=1000ms, actual deviation=2223ms
STATUS | wrapper | 2025/07/09 10:13:48 | JVM appears hung: Timed out waiting for signal from JVM. Restarting JVM.
STATUS | wrapper | 2025/07/09 10:14:00 | JVM exited after being requested to terminate.
STATUS | wrapper | 2025/07/09 10:14:05 | Reloading Wrapper configuration...
STATUS | wrapper | 2025/07/09 10:14:05 | Launching a JVM...
INFO | jvm 5 | 2025/07/09 10:14:06 | WrapperManager: Initializing...
INFO | jvm 5 | 2025/07/09 10:14:07 | Scheduling logfile maintenance. Last maintenance: Wed Dec 31 19:00:00 EST 1969, events since then: 0

Log 2

tgarrison · July 9, 2025, 8:02pm

Never had that issue... What version? And what OS?

kvane · July 9, 2025, 8:13pm

You are describing performance issues which can be hard to pinpoint without actually seeing the system.

One common reason for the Ignition JVM restarting like this is due to allocated server memory reaching the limit. The reset can occur from the Ignition Server running out of available memory, and would be clearly visible in the Status > Performance page.

If the overall server itself has no memory issues, and there is nothing abnormal on the performance page, consider contacting support to dive deeper into the problem.

Normal "Sharktooth" Memory Trend:

pturmel · July 9, 2025, 10:24pm

As noted, memory exhaustion is most likely. It is vital on production systems that the memory setting for initial heap size match the max allowance (in ignition.conf). Java doesn't give memory back to the OS, and will crash if it can't get the max allowance when needed, so you must make it claim what it needs up front.

If that doesn't leave enough RAM for the OS and any other services, boost your hardware allocation.

JayR · July 9, 2025, 11:21pm

We’ve allocated 64 GB of memory to the Ignition Gateway, so I’m not entirely convinced it’s a memory issue.

We’ve already reached out to support and are currently waiting for a response, as the system is still under monitoring.

You can also clearly see from the image below that the server has been restarting frequently, with visible gaps in operation.

pturmel · July 9, 2025, 11:39pm

One of your events was over 32 seconds long. That's an eternity, and typically happens when using poorly-configured VMs. (If using a VM, have you followed IA's VM configuration white paper?)

Or you have clock warps of that magnitude.

How much actual RAM does that machine (or VM) have?

What are the values in ignition.conf for initial and max heap?

nminchin · July 9, 2025, 11:41pm

Is your VM resourcing set correctly? i.e. are you dedicating your resources - CPU and memory - to the Ignition VM? Not doing this can result in poor Ignition performance, with pause the world events a common result. Also, host resources should never be over-committed; only give as many resources to VMs as are available by the system, excluding an allowance for the host itself.

How many CPU cores and processors are allocated to the VM? If you have multiple processors, it will be more performant allocating cores from a single processor rather than across multiple processors

w_tc1 · July 10, 2025, 1:40am

Is the OS windows server? Do you enable the windows defender service which may cause the issue.

JayR · July 10, 2025, 1:03pm

The support has already reviewed the VM configuration and didn’t identify any misconfigured settings.

As for the Ignition configuration, we’ve set the initial heap size to 32,768 MB and the maximum heap size to 65,536 MB, while the machine itself has 100 GB of RAM. I’ll be checking the allocated CPU cores next to rule out any further resource-related issues.

I will also review the response from w_tc1 regarding the same issue to see if there are any relevant insights.

Leor_Fishman · July 10, 2025, 5:14pm

funny story about this
a decade ago, i was working on some ML tasks within java, and i set the max heap size a little high....and java grabbed so much memory that windows couldn't even run core kernel programs like 'shut down the computer' or 'task manager'. Had to unplug the tower!