We have been experiencing memory usage problems with our Ignition installation.
On average our system consumes about 365MB / Day of alloted memory. Our system is small and only has 4699 tags.
The primary provider for these is MQTT. The rest are OPC tags either through a remote KEPWare server or the Ignition UA server.
The trend of memory usage shows a cycling sawtooth pattern superimposed on the gradualling increasing usage so it’s at least doing some garbage collection.
I really don’t know where to begin as far as trying to figure out what is causing this.
As I have other responsibilities in my job, I don’t have much time to babysith this thing so I’ve been restarting the server to buy time. Any thoughts would be greatly appreciated.
I start receiving clock drift warnings. I’ve only every accidentally let it get rreally close and system performance degraded rapidly. So, let me get this straight. If I allot 8GB max to Ignition, it will continue to use that up until it hits the ceiling and then clean up? Wouldn’t that cause a restart?
Short clock drift warnings with high memory indicate that the CPU is struggling to keep up with the GC demands; pauses are becoming long enough to notice because the CPU can’t get through the heap it has to process fast enough.
Reduce the heap size, increase the CPU resources, or tune the GC to prefer shorter pause times (trading infrequent large CPU spikes for more consistent generally higher usage).
On a production system, you should always set initial memory and max allowed to the same values. Ignition is going to grab it all anyways--it is best to do it up front. (Java doesn't give memory back to the OS after it claims it. It just cycles between free and occupied within Java.)
Apologies for not providing more info up front. This system is running on Windows Server 2016 in a VM environment. It has 32GB of RAM. we are currently running on Ignition version 8.1.18.
I’m relatively new at this so I’m not sure on how to increase CPU resources. By reducing the heap, I assume you mean decrease the allotment in the Ignition config file.
I will go ahead and set the min and max the same though.
Has your VM environment allocated the memory exclusively to Ignition? With an exclusive allocation for CPU cores?
Ignition normally has very erratic CPU usage with a fair amount of idle time. Hypervisors see the idle time and give it to other VMs if not allocated exclusively. Ignition needs that idle time to keep latencies down.
Not sure about whether it’s exclusive, but I’ll have to run that one down with our IT provider to make sure.
I seem to remember you saying something about that in another thread I saw as I was doing research on this.
Eww! That's ominous. I'm allergic to production systems running on other people's infrastructure.
You may have also seen recommendations from IA staff to not use Virtual Machines. In my experience, VMs are fine if given exclusive resources. That, unfortunately, is not what IT departments/providers typically deliver.
I understand, if I had my way I would have already made this a local server, however, our Verizon and AT&T APN’s come directly to the data center to ignition and are not routed on the network. I may actually have to look into getting this done.
In the mean time, I have set the min and max memory the same as recommended and restarted ignition. Hopefully the coming battle with our provider won’t be too bad.
If your SCADA software is launching a lot of new threads, scripts or you are generating a lot of database queries, you may also see the memory usage creeping up like this.
I have included some links below showing how to get to the relevant diagnostics pages.
If this is the case, you will need to look at making your scripts/queries more efficient, and you should also see improved CPU performance as a result.
My question maybe isn't related super tight to original one, but is there a way to check what takes up memory in Ignition? For example out 4gb say 1GB on tags, 1GB on active Vision projects, 50MB on running scripts etc.?
Also I have understood from previous posts that memory leak on script with top up the memory easily, any suggestions how to narrow down potentially leaking ones?
The heap memory usage in a JVM is not categorized conveniently, so you need some knowledge of Ignition's architecture to dig around, but heap dumps (via jmap or jcmd) or heap monitoring (like in VisualVM) are the techniques to use.
There's a great deal of "noise" in a normal sawtooth pattern, so I usually use a multiple heap dumps over time, taken at the bottom of a sawtooth, to try to zero in on what is growing.
I'm not that familiar with architecture, but server was rebooted last afternoon and initially after reboot only half of memory was in use without big fluctuations. Things have changed just after 19:00, suddenly usage started to creep up again to maximum.
The scripts I found to be started around that time are using system.tag.writeBlocking and system.tag.readBlocking in a most basic way. I suspect there's also something else happening during the same time, but I don't see it.
One of the scripts running at that time. Two others have 3 lines of "system.tag.writeBlocking", each scheduled to start at every 7:00 at 19:00. I was also wondering if at that time someone could call a report request with report doing something odd.
