Follow up here in case anyone else runs across this problem.
JVM Hanging seems to be a resource based issue.
Kevin was on the right track suspecting Antivirus. But I was a little surprised that it turned out to be Microsoft Windows Defender. Since I have installed many gateways and never run into this issue. I typically leave Defender to “run out of the box”.
With Inductive support;
- We examined the Windows event logs at the time of one of the gateway crashes we could see that Windows Defender had been scheduled to run around the same time.
- Issuing a “Quick Scan” on Defender we were able to see the Gateway “restarting” multiple times. Defender appeared to get “stuck” on a file or location which froze the entire operating system ( i.e. File explorer would not navigate, Gateway web page was stuck ).
Further investigation
- I was able to issue an exception ( do not include in scan in Defender setup ) on a couple of Windows Folders ( system32 , SysWOW64 ) and make it through the quick scan without causing a gateway restart.
- I ran Defender Full scan and was able to cause the Gateway to “crash” despite those exceptions.
- Defender does not have a very verbose system of reporting it’s execution. So it is hard to pinpoint the file or issue.
For further clarity on how the JVM wrapper operates on Java freezing see;
https://wrapper.tanukisoftware.com/doc/english/qna-freeze.html
https://wrapper.tanukisoftware.com/doc/english/prop-jvm-terminate-timeout.html
Excerpt
- The Java Service Wrapper periodically pings the JVM process (every 5 seconds by default) and waits for a response. If the response fails to arrive within a configured period of time (30 seconds by default), then the Wrapper will determine that it is frozen. The Wrapper also takes into account things like overall system load to make sure that false positives are kept to a minimum.
Not sure if these are the defaults with Ignition…
wrapper.ping.timeout=30
wrapper.ping.interval=5
wrapper.jvm_terminate.timeout=10
At this point in time I have no choice but to disable Windows Defender. I have temporaily installed Kaspersky, done a full scan which resulted in no viruses and the gateway ticking along no problem during a full scan.