Gateway Stopped - JVM appears hung

Ignition
1

Version: 8.1.1 (b2020120808)

I have had 2 complete stops of the Gateway in the last 2 days. Wondering if someone can point me in the right direction.

Stop #2

STATUS | wrapper | 2020/12/17 14:51:16 | JVM appears hung: Timed out waiting for signal from JVM. Restarting JVM.
ERROR | wrapper | 2020/12/17 14:51:31 | Timed out waiting for the OS to forcibly terminate the JVM process, unable to continue.
ERROR | wrapper | 2020/12/17 14:51:31 | The scheduled restart of the JVM has been cancelled.
STATUS | wrapper | 2020/12/17 14:51:32 | <-- Wrapper Stopped

Stop #1 ( Not sure it stopped here but a similar message )

STATUS | wrapper | 2020/12/16 17:03:29 | JVM appears hung: Timed out waiting for signal from JVM. Restarting JVM.
ERROR | wrapper | 2020/12/16 17:03:44 | Timed out waiting for the OS to forcibly terminate the JVM process, unable to continue.
ERROR | wrapper | 2020/12/16 17:03:44 | The scheduled restart of the JVM has been cancelled.
STATUS | wrapper | 2020/12/16 17:03:45 | <-- Wrapper Stopped
WARN | wrapper | 2020/12/16 17:08:09 | The “wrapper.java.command” property was redefined on line #46 of configuration file: E:\Ignition Install\data\ignition.conf
WARN | wrapper | 2020/12/16 17:08:09 | Old Value wrapper.java.command=lib/runtime/jre-win/bin/java
WARN | wrapper | 2020/12/16 17:08:09 | New Value wrapper.java.command=%JAVA_HOME%/bin/java
STATUS | wrapper | 2020/12/16 17:08:09 | --> Wrapper Started as Service
STATUS | wrapper | 2020/12/16 17:08:10 | Java Service Wrapper Standard Edition 64-bit 3.5.42
STATUS | wrapper | 2020/12/16 17:08:10 | Copyright © 1999-2020 Tanuki Software, Ltd. All Rights Reserved.
STATUS | wrapper | 2020/12/16 17:08:10 | http://wrapper.tanukisoftware.com
STATUS | wrapper | 2020/12/16 17:08:10 | Licensed to Inductive Automation for Inductive Automation
STATUS | wrapper | 2020/12/16 17:08:10 |
STATUS | wrapper | 2020/12/16 17:08:11 | Launching a JVM


2

What are system resources looking like? Enough free memory?

3

Hi. Thanks for the response. It appears to have crashed again this morning. Same symptoms. I did double up on the memory yesterday. Here is a snapshot.

image
image1368×562 39.6 KB

This is a clean gateway with 1 or 2 clients.

Perhaps I should open a ticket up on this.

  • STATUS | wrapper | 2020/12/18 10:26:13 | JVM appears hung: Timed out waiting for signal from JVM. Restarting JVM.
  • ERROR | wrapper | 2020/12/18 10:26:29 | Timed out waiting for the OS to forcibly terminate the JVM process, unable to continue.
  • ERROR | wrapper | 2020/12/18 10:26:29 | The scheduled restart of the JVM has been cancelled.
  • STATUS | wrapper | 2020/12/18 10:26:29 | ← Wrapper Stopped
4

One additional piece on information. There is a Modbus tag to our Boiler that is not connecting properly as we try to establish comms etc. Perhaps the root of the problem ? From wrapper just before the crashes;

INFO | jvm 1 | 2020/12/17 14:50:08 | W [d.M.ReadHoldingRegistersRequest] [19:50:08]: Request failed. FailureType==DISCONNECTED device-name=Boiler
INFO | jvm 1 | 2020/12/17 14:50:08 | java.lang.Exception: Driver is disconnected.

INFO | jvm 1 | 2020/12/18 10:25:38 | E [d.M.ReadInputRegistersRequest ] [15:25:38]: Received response with ExceptionCode: 0x0B (GatewayTargetDeviceFailToRespond). device-name=Boiler1
INFO | jvm 1 | 2020/12/18 10:25:39 | E [d.M.ReadInputRegistersRequest ] [15:25:39]: Received response with ExceptionCode: 0x0B (GatewayTargetDeviceFailToRespond). device-name=Boiler1

5

Can the following be added to ignition.conf and adjusted ?

wrapper.jvm_terminate.timeout=10

image
image759×705 69.6 KB

6

Yes, it can be. But you may be trying to paper over a real problem here.

I doubt it’s related to your device error messages, but it’s certainly something you should call support to see if they can help you figure out. It’s usually some kind of run-away alarm status or history queries (too many, or for too large a range), but it could be anything really.

edit: oh, we’ve also seen it be external like an AV scan or Windows volume shadow copy / backup, but those tend to present at the same time over and over.

1 Like
7

Follow up here in case anyone else runs across this problem.

JVM Hanging seems to be a resource based issue.

Kevin was on the right track suspecting Antivirus. But I was a little surprised that it turned out to be Microsoft Windows Defender. Since I have installed many gateways and never run into this issue. I typically leave Defender to “run out of the box”.

With Inductive support;

  • We examined the Windows event logs at the time of one of the gateway crashes we could see that Windows Defender had been scheduled to run around the same time.
  • Issuing a “Quick Scan” on Defender we were able to see the Gateway “restarting” multiple times. Defender appeared to get “stuck” on a file or location which froze the entire operating system ( i.e. File explorer would not navigate, Gateway web page was stuck ).

Further investigation

  • I was able to issue an exception ( do not include in scan in Defender setup ) on a couple of Windows Folders ( system32 , SysWOW64 ) and make it through the quick scan without causing a gateway restart.
  • I ran Defender Full scan and was able to cause the Gateway to “crash” despite those exceptions.
  • Defender does not have a very verbose system of reporting it’s execution. So it is hard to pinpoint the file or issue.

For further clarity on how the JVM wrapper operates on Java freezing see;

https://wrapper.tanukisoftware.com/doc/english/qna-freeze.html
https://wrapper.tanukisoftware.com/doc/english/prop-jvm-terminate-timeout.html

Excerpt

Not sure if these are the defaults with Ignition

wrapper.ping.timeout=30
wrapper.ping.interval=5
wrapper.jvm_terminate.timeout=10

At this point in time I have no choice but to disable Windows Defender. I have temporaily installed Kaspersky, done a full scan which resulted in no viruses and the gateway ticking along no problem during a full scan.

2 Likes