Gateway Restarts Automatically

Ignition v: 8.1.1
Windows 10, 2016 Standard

Hello there,

We are facing this issue since last week that out Ignition production Gateway restarts automatically. Please see below today’s log details pulled after it was auto restarted…

INFO   | jvm 4    | 2021/09/13 12:29:38 | [38154.932s][warning][os,thread] Attempt to allocate stack guard pages failed.
INFO   | jvm 4    | 2021/09/13 12:29:38 | OpenJDK 64-Bit Server VM warning: Attempt to unguard stack red zone failed.
INFO   | jvm 4    | 2021/09/13 12:29:38 | An unrecoverable stack overflow has occurred.
INFO   | jvm 4    | 2021/09/13 12:29:38 | #
INFO   | jvm 4    | 2021/09/13 12:29:38 | # A fatal error has been detected by the Java Runtime Environment:
INFO   | jvm 4    | 2021/09/13 12:29:38 | #
INFO   | jvm 4    | 2021/09/13 12:29:38 | #  EXCEPTION_STACK_OVERFLOW (0xc00000fd) at pc=0x00007ffba800b223, pid=48856, tid=53232
INFO   | jvm 4    | 2021/09/13 12:29:38 | #
INFO   | jvm 4    | 2021/09/13 12:29:38 | # JRE version: OpenJDK Runtime Environment (Zulu11.39+16-SA) (11.0.7+10) (build 11.0.7+10-LTS)
INFO   | jvm 4    | 2021/09/13 12:29:38 | # Java VM: OpenJDK 64-Bit Server VM (11.0.7+10-LTS, mixed mode, tiered, compressed oops, g1 gc, windows-amd64)
INFO   | jvm 4    | 2021/09/13 12:29:38 | # Problematic frame:
INFO   | jvm 4    | 2021/09/13 12:29:38 | # V  [jvm.dll+0x35b223]
INFO   | jvm 4    | 2021/09/13 12:29:38 | #
INFO   | jvm 4    | 2021/09/13 12:29:38 | # Core dump will be written. Default location: C:\Program Files\Inductive Automation\Ignition\hs_err_pid48856.mdmp
INFO   | jvm 4    | 2021/09/13 12:29:38 | #
INFO   | jvm 4    | 2021/09/13 12:29:38 | # An error report file with more information is saved as:
INFO   | jvm 4    | 2021/09/13 12:29:38 | # C:\Program Files\Inductive Automation\Ignition\hs_err_pid48856.log
INFO   | jvm 4    | 2021/09/13 12:29:39 | [38156.088s][warning][os,thread] Failed to start thread - _beginthreadex failed (EINVAL) for attributes: stacksize: default, flags: CREATE_SUSPENDED STACK_SIZE_PARAM_IS.
INFO   | jvm 4    | 2021/09/13 12:29:39 | W [d.S.B.TimeoutDaemon           ] [06:59:37]: ScheduledRequest[com.inductiveautomation.xopc.drivers.siemens.requests.ReadRequest@13d33303] request with key "14061" failed due to timeout. device-name=PWV_S7_300
INFO   | jvm 4    | 2021/09/13 12:29:39 | W [d.S.ReadRequest               ] [06:59:37]: Request failed due to TIMEOUT device-name=PWV_S7_300
INFO   | jvm 4    | 2021/09/13 12:29:39 | W [d.S.B.TimeoutDaemon           ] [06:59:37]: ScheduledRequest[com.inductiveautomation.xopc.drivers.siemens.requests.ReadRequest@1261fc02] request with key "17597" failed due to timeout. device-name=Vehicle_Siement_PLC_300
INFO   | jvm 4    | 2021/09/13 12:29:39 | W [d.S.ReadRequest               ] [06:59:37]: Request failed due to TIMEOUT device-name=Vehicle_Siement_PLC_300
INFO   | jvm 4    | 2021/09/13 12:29:39 | W [d.S.B.TimeoutDaemon           ] [06:59:37]: ScheduledRequest[com.inductiveautomation.xopc.drivers.siemens.requests.ReadRequest@c92c658] request with key "1144" failed due to timeout. device-name=Engine_TC_1-8_PLC
INFO   | jvm 4    | 2021/09/13 12:29:39 | W [d.S.ReadRequest               ] [06:59:37]: Request failed due to TIMEOUT device-name=Engine_TC_1-8_PLC
INFO   | jvm 4    | 2021/09/13 12:29:39 | W [d.S.B.TimeoutDaemon           ] [06:59:37]: ScheduledRequest[com.inductiveautomation.xopc.drivers.siemens.requests.ReadRequest@4bdd1b56] request with key "14199" failed due to timeout. device-name=PW_Transmission_CutLine_STN1N2_PLC
INFO   | jvm 4    | 2021/09/13 12:29:39 | W [d.S.ReadRequest               ] [06:59:37]: Request failed due to TIMEOUT device-name=PW_Transmission_CutLine_STN1N2_PLC
INFO   | jvm 4    | 2021/09/13 12:29:47 | [38164.691s][warning][os,thread] Failed to start thread - _beginthreadex failed (EINVAL) for attributes: stacksize: default, flags: CREATE_SUSPENDED STACK_SIZE_PARAM_IS.
INFO   | jvm 4    | 2021/09/13 12:29:50 | [38166.791s][warning][os,thread] Failed to start thread - _beginthreadex failed (EINVAL) for attributes: stacksize: default, flags: CREATE_SUSPENDED STACK_SIZE_PARAM_IS.
INFO   | jvm 4    | 2021/09/13 12:29:50 | [38167.011s][warning][os,thread] Failed to start thread - _beginthreadex failed (EINVAL) for attributes: stacksize: default, flags: CREATE_SUSPENDED STACK_SIZE_PARAM_IS.
ERROR  | wrapper  | 2021/09/13 12:30:16 | The JVM process terminated due to an uncaught exception: EXCEPTION_ACCESS_VIOLATION (0xc0000005)
ERROR  | wrapper  | 2021/09/13 12:30:16 | JVM exited unexpectedly.
STATUS | wrapper  | 2021/09/13 12:30:21 | Reloading Wrapper configuration...
STATUS | wrapper  | 2021/09/13 12:30:22 | Launching a JVM...
INFO   | jvm 5    | 2021/09/13 12:30:22 | WrapperManager: Initializing...

The Error it specifies is as below:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  EXCEPTION_STACK_OVERFLOW (0xc00000fd) at pc=0x00007ffba800b223, pid=48856, tid=53232
#
# JRE version: OpenJDK Runtime Environment (Zulu11.39+16-SA) (11.0.7+10) (build 11.0.7+10-LTS)
# Java VM: OpenJDK 64-Bit Server VM (11.0.7+10-LTS, mixed mode, tiered, compressed oops, g1 gc, windows-amd64)
# Problematic frame:
# V  [jvm.dll+0x35b223]
#
# Core dump will be written. Default location: C:\Program Files\Inductive Automation\Ignition\hs_err_pid48856.mdmp
#

#   http://www.azulsystems.com/support/
#

]

Please suggest

What does your memory usage graph look like?

Here it is:

Well, that eliminates a memory leak as a possible culprit.

Does the gateway server have ECC memory? Any chance the motherboard/RAM/CPU are dying?

Yes last week the team informed us that Gateway was showing “High Memory” warning in red on status —> Performance page.

Get the crash dump(s) (C:\Program Files\Inductive Automation\Ignition\hs_err_pid48856.log or similar) and call support and provide these.

In the meantime, you can try upgrading Ignition, which should include an newer JDK version than you have with 8.1.1 and see if it fixes it.

This is what mentioned in the hs_err_pid48856

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  EXCEPTION_STACK_OVERFLOW (0xc00000fd) at pc=0x00007ffba800b223, pid=48856, tid=53232
#
# JRE version: OpenJDK Runtime Environment (Zulu11.39+16-SA) (11.0.7+10) (build 11.0.7+10-LTS)
# Java VM: OpenJDK 64-Bit Server VM (11.0.7+10-LTS, mixed mode, tiered, compressed oops, g1 gc, windows-amd64)
# Problematic frame:
# V  [jvm.dll+0x35b223]
#
# Core dump will be written. Default location: C:\Program Files\Inductive Automation\Ignition\hs_err_pid48856.mdmp
#

#   http://www.azulsystems.com/support/
#

]

I also wanted to know:
When we open Ignition Designer on some PC, does it use the memory available on gateway server or not?

No, Vision clients and designers are independent instances of the Java virtual machine with their own memory pool. More clients, designers, or Perspective sessions will create more ‘load’ on the gateway, but not directly consume system resources.

You should still contact support about this. Also, Phil’s suggestion of a hardware failure is plausible at this point.

Okay thanks for the update