Ignition Memory Usage

We have been experiencing memory usage problems with our Ignition installation.
On average our system consumes about 365MB / Day of alloted memory. Our system is small and only has 4699 tags.


The primary provider for these is MQTT. The rest are OPC tags either through a remote KEPWare server or the Ignition UA server.
The trend of memory usage shows a cycling sawtooth pattern superimposed on the gradualling increasing usage so it’s at least doing some garbage collection.

I really don’t know where to begin as far as trying to figure out what is causing this.
As I have other responsibilities in my job, I don’t have much time to babysith this thing so I’ve been restarting the server to buy time. Any thoughts would be greatly appreciated.

What’s the actual problem?

That looks like a totally normal memory trend to me.

2 Likes

(Concur with Paul. Looks pretty normal.) What happens when the usage nears the max?

Java won’t do major garbage collection until it needs to. If you’ve told it that 8GB is acceptable, it will use it.

I start receiving clock drift warnings. I’ve only every accidentally let it get rreally close and system performance degraded rapidly. So, let me get this straight. If I allot 8GB max to Ignition, it will continue to use that up until it hits the ceiling and then clean up? Wouldn’t that cause a restart?

Short clock drift warnings with high memory indicate that the CPU is struggling to keep up with the GC demands; pauses are becoming long enough to notice because the CPU can’t get through the heap it has to process fast enough.
Reduce the heap size, increase the CPU resources, or tune the GC to prefer shorter pause times (trading infrequent large CPU spikes for more consistent generally higher usage).

1 Like

Also - is 8GB how much is available on this entire computer? Definitely don’t give Ignition all of your system’s memory.

1 Like

On a production system, you should always set initial memory and max allowed to the same values. Ignition is going to grab it all anyways–it is best to do it up front. (Java doesn’t give memory back to the OS after it claims it. It just cycles between free and occupied within Java.)

Apologies for not providing more info up front. This system is running on Windows Server 2016 in a VM environment. It has 32GB of RAM. we are currently running on Ignition version 8.1.18.

I’m relatively new at this so I’m not sure on how to increase CPU resources. By reducing the heap, I assume you mean decrease the allotment in the Ignition config file.

I will go ahead and set the min and max the same though.

1 Like

Are those 2 big drops in your graph GC events or are they when you restarted Ignition?

Has your VM environment allocated the memory exclusively to Ignition? With an exclusive allocation for CPU cores?

Ignition normally has very erratic CPU usage with a fair amount of idle time. Hypervisors see the idle time and give it to other VMs if not allocated exclusively. Ignition needs that idle time to keep latencies down.

Not sure about whether it’s exclusive, but I’ll have to run that one down with our IT provider to make sure.
I seem to remember you saying something about that in another thread I saw as I was doing research on this.

Eww! That’s ominous. I’m allergic to production systems running on other people’s infrastructure.

You may have also seen recommendations from IA staff to not use Virtual Machines. In my experience, VMs are fine if given exclusive resources. That, unfortunately, is not what IT departments/providers typically deliver.

1 Like

I understand, if I had my way I would have already made this a local server, however, our Verizon and AT&T APN’s come directly to the data center to ignition and are not routed on the network. I may actually have to look into getting this done.

In the mean time, I have set the min and max memory the same as recommended and restarted ignition. Hopefully the coming battle with our provider won’t be too bad.

If your SCADA software is launching a lot of new threads, scripts or you are generating a lot of database queries, you may also see the memory usage creeping up like this.
I have included some links below showing how to get to the relevant diagnostics pages.
If this is the case, you will need to look at making your scripts/queries more efficient, and you should also see improved CPU performance as a result.

https://docs.inductiveautomation.com/display/DOC81/Diagnostics+-+Threads
https://docs.inductiveautomation.com/display/DOC81/Diagnostics+-+Running+Scripts
https://docs.inductiveautomation.com/display/DOC81/Connections+-+Databases