Ignition server memory increasing

Hi to All,

Gatewey that is in production stopped working. On the client PC they could see an Error message “Error loading resources: Requesting component defaults time out”. The users had to restart the gateway PC.
After restarting and investigating the log files I found out that the message “Clock drift, degraded performance, or pause-the-world detected. Max allowed deviation=1000ms, actual deviation=4554ms” (with different actual deviation) was repeating for almost 20 hours.

Config file:

Gateway Host is running in VM(Nutanix) below you can see the chart about memory usage for the last month. This chart is a little bit unclear, doesn’t show the amount of memory.

I have no insight into setting the memory by the host for Gatewey VM. I was told it was assigned I think 16GB. When gatewey was restarted the first time Host of the gatewey VM was showing the VM using more than 50% of the assigned memory.
System info from Gatewey Pc showing next

.

After restarting Pc and let getaway run again. I notice (from gatewey page) that used memory growing per day around 200mb.

Is this memory liking, or garbage collector, or something that I wrote wrong?

Thanks!

That is a memory leak, almost certainly. Could be something in your scripting. You definitely should get support involved if you aren’t comfortable profiling your gateway.

Phil,

Thank you very much for your answer. I opened a ticket, but it usually takes time to get some response. I do not have a problem to profiling the gatewey. I have a problem analyzing results. I do not have experience with java, memory leak, garbage collector…
How can scripting lead to memory leaks?

Thanks!

Firing off async threads without a means to kill off. A single jython async thread stuck in an infinite wait or infinite loop will hold the entire old interpreter in memory after a scripting restart.

Opening python sockets without try-finally guards to close.

Attaching python listeners to java infrastructure with no way to remove/retire.

Caching python objects without control over lifetimes.

All of the above are prone to happen regularly as projects are edited and scripts are restarted (per project in the gateway).

1 Like

Phil,

Thank you very much for your answer and all your effort.
I do not use any of mentioned in my scripts that I know. I have pages with a lot of objects and movements of some objects. I am working a lot with the dataset. Also, I am using your view expression, which is great.
A lot of XY charts in the carousel object that is in autoplay mode. Lots of scripts lot of happening, but all are based on basic python scripting.

I’d probably start profiling memory usage, then. I’d start with the jmap tool’s histogram mode for per-class memory usage. Unfortunately, taking a profile snapshot isn’t instant–your gateway will freeze for a couple seconds. Do this multiple times, a day apart each, and look if there is a a particular class or collection of classes responsible for ~ 200meg extra per day. If so, take a full snapshot and look where those classes are being allocated. (Or turn it over to IA support at that point.)