# Gateway Stopped - JVM appears hung

**URL:** <https://forum.inductiveautomation.com/t/gateway-stopped-jvm-appears-hung/41632>\
**Category:** Ignition\
**Tags:** ignition81\
**Created:** [December 17, 2020, 9:50pm UTC](https://forum.inductiveautomation.com/t/gateway-stopped-jvm-appears-hung/41632 "2020-12-17T21:50:25Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![epsmitas](https://avatars.discourse-cdn.com/v4/letter/e/4af34b/32.png) [@epsmitas](https://forum.inductiveautomation.com/u/epsmitas)\
**Post date:** [December 17, 2020, 9:50pm UTC](https://forum.inductiveautomation.com/t/gateway-stopped-jvm-appears-hung/41632/1 "2020-12-17T21:50:25Z")

</div>

Version: **8.1.1 (b2020120808)**

I have had 2 complete stops of the Gateway in the last 2 days. Wondering if someone can point me in the right direction.

Stop #2

STATUS | wrapper | 2020/12/17 14:51:16 | JVM appears hung: Timed out waiting for signal from JVM. Restarting JVM.  
ERROR | wrapper | 2020/12/17 14:51:31 | Timed out waiting for the OS to forcibly terminate the JVM process, unable to continue.  
ERROR | wrapper | 2020/12/17 14:51:31 | The scheduled restart of the JVM has been cancelled.  
STATUS | wrapper | 2020/12/17 14:51:32 | \<-- Wrapper Stopped

Stop #1 ( Not sure it stopped here but a similar message )

STATUS | wrapper | 2020/12/16 17:03:29 | JVM appears hung: Timed out waiting for signal from JVM. Restarting JVM.  
ERROR | wrapper | 2020/12/16 17:03:44 | Timed out waiting for the OS to forcibly terminate the JVM process, unable to continue.  
ERROR | wrapper | 2020/12/16 17:03:44 | The scheduled restart of the JVM has been cancelled.  
STATUS | wrapper | 2020/12/16 17:03:45 | \<-- Wrapper Stopped  
WARN | wrapper | 2020/12/16 17:08:09 | The “wrapper.java.command” property was redefined on line #46 of configuration file: E:\Ignition Install\data\ignition.conf  
WARN | wrapper | 2020/12/16 17:08:09 | Old Value wrapper.java.command=lib/runtime/jre-win/bin/java  
WARN | wrapper | 2020/12/16 17:08:09 | New Value wrapper.java.command=%JAVA\_HOME%/bin/java  
STATUS | wrapper | 2020/12/16 17:08:09 | --\> Wrapper Started as Service  
STATUS | wrapper | 2020/12/16 17:08:10 | Java Service Wrapper Standard Edition 64-bit 3.5.42  
STATUS | wrapper | 2020/12/16 17:08:10 | Copyright © 1999-2020 Tanuki Software, Ltd. All Rights Reserved.  
STATUS | wrapper | 2020/12/16 17:08:10 | [http://wrapper.tanukisoftware.com](http://wrapper.tanukisoftware.com)  
STATUS | wrapper | 2020/12/16 17:08:10 | Licensed to Inductive Automation for Inductive Automation  
STATUS | wrapper | 2020/12/16 17:08:10 |  
STATUS | wrapper | 2020/12/16 17:08:11 | Launching a JVM…

---

<div class="post-metadata">

**Author:** ![Kevin.Collins](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/kevin.collins/32/13497_2.png) [@Kevin.Collins](https://forum.inductiveautomation.com/u/Kevin.Collins)\
**Post date:** [December 18, 2020, 3:41am UTC](https://forum.inductiveautomation.com/t/gateway-stopped-jvm-appears-hung/41632/2 "2020-12-18T03:41:53Z")

</div>

What are system resources looking like? Enough free memory?

---

<div class="post-metadata">

**Author:** ![epsmitas](https://avatars.discourse-cdn.com/v4/letter/e/4af34b/32.png) [@epsmitas](https://forum.inductiveautomation.com/u/epsmitas)\
**Post date:** [December 18, 2020, 6:30pm UTC](https://forum.inductiveautomation.com/t/gateway-stopped-jvm-appears-hung/41632/3 "2020-12-18T18:30:39Z")

</div>

Hi. Thanks for the response. It appears to have crashed again this morning. Same symptoms. I did double up on the memory yesterday. Here is a snapshot.

 ![image](https://us1.discourse-cdn.com/inductiveautomation/original/3X/c/7/c7b2092007a6b115f332a9b88fce9930ad86fcbc.png)

This is a clean gateway with 1 or 2 clients.

Perhaps I should open a ticket up on this.

- STATUS | wrapper | 2020/12/18 10:26:13 | JVM appears hung: Timed out waiting for signal from JVM. Restarting JVM.
- ERROR | wrapper | 2020/12/18 10:26:29 | Timed out waiting for the OS to forcibly terminate the JVM process, unable to continue.
- ERROR | wrapper | 2020/12/18 10:26:29 | The scheduled restart of the JVM has been cancelled.
- STATUS | wrapper | 2020/12/18 10:26:29 | ← Wrapper Stopped

---

<div class="post-metadata">

**Author:** ![epsmitas](https://avatars.discourse-cdn.com/v4/letter/e/4af34b/32.png) [@epsmitas](https://forum.inductiveautomation.com/u/epsmitas)\
**Post date:** [December 18, 2020, 6:44pm UTC](https://forum.inductiveautomation.com/t/gateway-stopped-jvm-appears-hung/41632/4 "2020-12-18T18:44:44Z")

</div>

One additional piece on information. There is a Modbus tag to our Boiler that is not connecting properly as we try to establish comms etc. Perhaps the root of the problem ? From wrapper just before the crashes;

INFO | jvm 1 | 2020/12/17 14:50:08 | W [d.M.ReadHoldingRegistersRequest] [19:50:08]: Request failed. FailureType==DISCONNECTED device-name=Boiler  
INFO | jvm 1 | 2020/12/17 14:50:08 | java.lang.Exception: Driver is disconnected.

INFO | jvm 1 | 2020/12/18 10:25:38 | E [d.M.ReadInputRegistersRequest] [15:25:38]: Received response with ExceptionCode: 0x0B (GatewayTargetDeviceFailToRespond). device-name=Boiler1  
INFO | jvm 1 | 2020/12/18 10:25:39 | E [d.M.ReadInputRegistersRequest] [15:25:39]: Received response with ExceptionCode: 0x0B (GatewayTargetDeviceFailToRespond). device-name=Boiler1

---

<div class="post-metadata">

**Author:** ![epsmitas](https://avatars.discourse-cdn.com/v4/letter/e/4af34b/32.png) [@epsmitas](https://forum.inductiveautomation.com/u/epsmitas)\
**Post date:** [December 18, 2020, 7:33pm UTC](https://forum.inductiveautomation.com/t/gateway-stopped-jvm-appears-hung/41632/5 "2020-12-18T19:33:30Z")

</div>

Can the following be added to ignition.conf and adjusted ?

wrapper.jvm\_terminate.timeout=10

 ![image](https://us1.discourse-cdn.com/inductiveautomation/original/3X/c/8/c85a9a70fbb75d47ec16546f78b0745272d05dc4.png)

---

<div class="post-metadata">

**Author:** ![Kevin.Herron](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/kevin.herron/32/7259_2.png) [@Kevin.Herron](https://forum.inductiveautomation.com/u/Kevin.Herron)\
**Post date:** [December 18, 2020, 7:45pm UTC](https://forum.inductiveautomation.com/t/gateway-stopped-jvm-appears-hung/41632/6 "2020-12-18T19:45:34Z")

</div>

Yes, it can be. But you may be trying to paper over a real problem here.

I doubt it’s related to your device error messages, but it’s certainly something you should call support to see if they can help you figure out. It’s usually some kind of run-away alarm status or history queries (too many, or for too large a range), but it could be anything really.

edit: oh, we’ve also seen it be external like an AV scan or Windows volume shadow copy / backup, but those tend to present at the same time over and over.

---

<div class="post-metadata">

**Author:** ![epsmitas](https://avatars.discourse-cdn.com/v4/letter/e/4af34b/32.png) [@epsmitas](https://forum.inductiveautomation.com/u/epsmitas)\
**Post date:** [December 19, 2020, 8:40pm UTC](https://forum.inductiveautomation.com/t/gateway-stopped-jvm-appears-hung/41632/7 "2020-12-19T20:40:47Z")

</div>

Follow up here in case anyone else runs across this problem.

JVM Hanging seems to be a resource based issue.

Kevin was on the right track suspecting Antivirus. But I was a little surprised that it turned out to be Microsoft Windows Defender. Since I have installed many gateways and never run into this issue. I typically leave Defender to “run out of the box”.

With Inductive support;

- We examined the Windows event logs at the time of one of the gateway crashes we could see that Windows Defender had been scheduled to run around the same time.
- Issuing a “Quick Scan” on Defender we were able to see the Gateway **“restarting”** multiple times. Defender appeared to get **“stuck”** on a file or location which froze the entire operating system ( i.e. File explorer would not navigate, Gateway web page was stuck ).

Further investigation

- I was able to issue an exception ( do not include in scan in Defender setup ) on a couple of Windows Folders ( system32 , SysWOW64 ) and make it through the quick scan without causing a gateway restart.
- I ran Defender Full scan and was able to cause the Gateway to **“crash”** despite those exceptions.
- Defender does not have a very verbose system of reporting it’s execution. So it is hard to pinpoint the file or issue.

For further clarity on how the JVM wrapper operates on Java freezing see;

[https://wrapper.tanukisoftware.com/doc/english/qna-freeze.html](https://wrapper.tanukisoftware.com/doc/english/qna-freeze.html)  
[https://wrapper.tanukisoftware.com/doc/english/prop-jvm-terminate-timeout.html](https://wrapper.tanukisoftware.com/doc/english/prop-jvm-terminate-timeout.html)

**Excerpt**

- _The Java Service Wrapper [periodically pings the JVM process (every 5 seconds by default)](https://wrapper.tanukisoftware.com/doc/english/prop-ping-interval.html) and waits for a response. If the response fails to arrive [within a configured period of time (30 seconds by default)](https://wrapper.tanukisoftware.com/doc/english/prop-ping-timeout.html), then the Wrapper will determine that it is frozen. The Wrapper also takes into account things like overall system load to make sure that false positives are kept to a minimum._

Not sure if these are the defaults with Ignition…  
wrapper.ping.timeout=30  
wrapper.ping.interval=5  
wrapper.jvm\_terminate.timeout=10

At this point in time I have no choice but to disable Windows Defender. I have temporaily installed Kaspersky, done a full scan which resulted in no viruses and the gateway ticking along no problem during a full scan.