Recently we've started getting a couple clock drift events a day, and they appear to happen when the non-heap memory use drops from 1.4 GB down to about 700 MB. Any idea what could suddenly be causing this to happen?
That suggests garbage collection is stopping the world long enough to trigger a clock drift event, which generally suggests your CPU isn't able to manage memory fast enough to keep pauses in check. What Ignition version are you using, and have you entered any configuration arguments in the ignition.conf file?
8.1.36 -- We've been running that version for a while now, and this only started happening the last few days.
I haven't tinkered with the .conf file, but I don't know if my predecessors did.
The default for GC pause target has changed for various versions of java, but I've never found the default to be short enough. Consider adding something like this to your ignition.conf
file:
wrapper.java.additional.2=-XX:MaxGCPauseMillis=100
It isn't a hard maximum for GC, but it will cause the JVM to work a bit harder all the time to keep pauses in that range.
(Replace the .2
with a non-conflicting integer.)
This will generally keep GC from triggering the 1000ms threshold for the clock drift detector.
(You can still get those for other CPU overload situations.)