Ignition gateway was crashed/restarted but showing up since last 2 months on the server

Ignition
1

Ignition gateway was crashed/restarted but when I checked the services on Linux server, it was showing up since last 2 months 24 days.

Ignition version: 8.0.13 Hosted on RHEL 7

[USER@MYSERVER savedi]# systemctl status Ignition-Gateway.service
● Ignition-Gateway.service - Ignition-Gateway
Loaded: loaded (/etc/systemd/system/Ignition-Gateway.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2023-11-13 07:26:17 EST; 2 months 24 days ago
Process: 1027 ExecStart=/apps/local/bin/ignition/ignition.sh start sysd (code=exited, status=0/SUCCESS)
Main PID: 1245 (ignition-gatewa)
CGroup: /system.slice/Ignition-Gateway.service
├─1245 /apps/local/bin/ignition/./ignition-gateway /apps/local/bin/ignition/data/ignition.conf wrapper.syslog.ident=Ignition-Gateway wrapper.pi...
└─5204 /apps/local/bin/ignition/lib/runtime/jre-nix/bin/java -Ddata.dir=/apps/local/bin/ignition/data -Xms1024m -Xmx8192m -Djava.library.path=/...

Gateway | CLIENT_Linux

Version: 8.0.13 (b2020060815)
License: partial
Uptime: a day

Errors in log file -

NFO | jvm 3 | 2024/02/05 16:55:57 | I [p.ClientSession ] [21:55:57]: Socket connected to session. pageId=24739de3 session-project=DanaOEE
INFO | jvm 3 | 2024/02/05 16:57:57 | #
INFO | jvm 3 | 2024/02/05 16:57:57 | # A fatal error has been detected by the Java Runtime Environment:
INFO | jvm 3 | 2024/02/05 16:57:57 | #
INFO | jvm 3 | 2024/02/05 16:57:57 | # SIGSEGV (0xb) at pc=0x00007fb61140520c, pid=29567, tid=29579
INFO | jvm 3 | 2024/02/05 16:57:57 | #
INFO | jvm 3 | 2024/02/05 16:57:57 | # JRE version: OpenJDK Runtime Environment (Zulu11.37+18-SA) (11.0.6+10) (build 11.0.6+10-LTS)
INFO | jvm 3 | 2024/02/05 16:57:57 | # Java VM: OpenJDK 64-Bit Server VM (11.0.6+10-LTS, mixed mode, tiered, compressed oops, g1 gc, linux-amd64)
INFO | jvm 3 | 2024/02/05 16:57:57 | # Problematic frame:
INFO | jvm 3 | 2024/02/05 16:57:57 | # V [libjvm.so+0xbe820c] compute_tree_cost(SwitchRange*, SwitchRange*, float)+0x21c
INFO | jvm 3 | 2024/02/05 16:57:57 | #
INFO | jvm 3 | 2024/02/05 16:57:57 | # No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
INFO | jvm 3 | 2024/02/05 16:57:57 | #
INFO | jvm 3 | 2024/02/05 16:57:57 | # An error report file with more information is saved as:
INFO | jvm 3 | 2024/02/05 16:57:57 | # /apps/local/bin/ignition/hs_err_pid29567.log
INFO | jvm 3 | 2024/02/05 16:57:57 | #
INFO | jvm 3 | 2024/02/05 16:57:57 | # Compiler replay data is saved as:
INFO | jvm 3 | 2024/02/05 16:57:57 | # /apps/local/bin/ignition/replay_pid29567.log
INFO | jvm 3 | 2024/02/05 16:57:57 | #
INFO | jvm 3 | 2024/02/05 16:57:57 | # If you would like to submit a bug report, please visit:
INFO | jvm 3 | 2024/02/05 16:57:57 | # Customer Support for Platform Core & Platform Prime | Azul
INFO | jvm 3 | 2024/02/05 16:57:57 | #
STATUS | wrapper | 2024/02/05 16:57:57 | JVM received a signal UNKNOWN (6).
STATUS | wrapper | 2024/02/05 16:57:57 | JVM process is gone.
ERROR | wrapper | 2024/02/05 16:57:57 | JVM exited unexpectedly.
STATUS | wrapper | 2024/02/05 16:58:01 | Reloading Wrapper configuration...
STATUS | wrapper | 2024/02/05 16:58:02 | JVM process is gone.
STATUS | wrapper | 2024/02/05 16:58:02 | Launching a JVM...
INFO | jvm 4 | 2024/02/05 16:58:02 | WrapperManager: Initializing...

Not sure what happened here
wrapper.log (4.5 MB)

3

It isn't actually unknown. It is SIGABRT, or "abort". The most common reason in Linux is that the process was selected by the OOM killer, for using too much memory in a situation where the OS was critically low on RAM.

You need to investigate your memory settings in ignition.conf and compare to the specs of the machine. When the machine runs nothing but the Ignition gateway, your max in ignition.conf should be a few gigabytes less than the machine's RAM size.

4

Ignition maxmemory - 8GB (out of 16GB)

5

Are you running anything else on that machine? (Your database, perhaps?) If so, make sure those things are also configured so the OS never runs out of RAM.

BTW:

Both of those items are old and buggy and unsupported.

6

Ignition has a Java Service Wrapper that will automatically restart the JVM (unless there are frequent crashes within a 5 minute [default] time period). That is why the service itself shows as still running, while Ignition itself restarted.

I've never seen an OOM kill occur via SIGABRT. This seems like something else, though I'll admit I'm not sure under what condition that signal is going to be sent (by systemd?), so I'm happy to be wrong.

7

Configurable, IIRC. Abort is the friendly "Please die now", quickly followed by the non-ignorable SIGKILL.

I've never run RHEL/CentOS 7, fwiw.

8

Only Ignition.

9

Look for suspicious log entries via journalctl at the times the restarts occur in the wrapper log.