We have a couple of questions around memory on our Ignition Gateway. This server is running Windows Server 2016, 16gb ram and Ignition 8.1 only (our historian database is not installed on this server). Appreciate any thoughts.
- We have noticed cyclical memory spikes but are unsure as to the cause. How might we better go about pinpointing the issue? See the Memory, Scripts & Threads images below.
- As mentioned, this server has 16gb of ram, of which we have allocated to Ignition (ignition.conf) as below. Is there a best practice in order to determine the best init/max values based on total amount of system ram?
Cylical ‘sawtooth’ memory usage like that is perfectly normal. The GC will allow things to “pile up”, then collect in one pass, freeing a lot of memory at once. Abnormal memory graphs will be immediately obvious as both a lot more chaotic, and also frequently ‘pegging’ against the maximum memory. Underallocated or underprovisioned systems will also likely get lots of ‘Clock Drift’ errors, as the garbage collector forces the entire JVM to pause.
In other words - I don’t think you have any actual problem, unless you’re experiencing some other side effect.
Great! We actually have ran into ‘Clock Drift’ several times. Would allocating more memory to Ignition address this?
Not necessarily. You can provide tuning options to the JVM to tell the garbage collector to run more aggressively (generally meaning you’ll trade CPU pressure for RAM pressure) - simply adding more memory is not a panacea. If you do want to go that route, I’d start with googling the G1 garbage collector and the tuning options it exposes, and making sure you understand them thoroughly before making any changes to your system.
I recommend setting G1GC’s target pause to 100ms. Or a bit less if you have beefy hardware. Your sawtooth graph is otherwise about as perfect as you might get. I recommend setting max memory such that the trough of the graph is in the 30% to 50% range, and the peaks are in the 50% to 80% range. I also recommend setting min==max. Any long-running process is going to allocate the max allowed, so there’s no point not doing it at startup.
Thanks Phil. Just so I’m clear, I have included portions of our current ignition.conf, as well as with potential changes. Assuming the calculations say ~8gb of memory is ‘right’ for us, is the rest accurate?
Looks like what I would do. (:
Consider using the gclog options to monitor your GC pause durations. (Makes a separate log file just for GC traffic.)
Ok. Is that as simple as, say, this?
I’ve never used a wildcard like that. Mine looks like this:
The WRAPPER_TIME makes it start a new file every startup. IIRC, this logger doesn’t append.
Got it. So, holistically, our params could look something like:
Or, as we are on Windows, this might be more accurate:
Ain’t that the truth.
Thanks guys. Good stuff - super helpful!