We just went live with an Ignition 8.3.4 project and the gateway is restarting randomly.
The project is migrated from 7.9, but the server machine is all freshly installed.
Linux OS, gateway is configured to use 6GB of ram, the machine has 32GB. (It doesn't reach any limit)
In the gateway logs, seems like a normal shutdown and then restart. Digging a little deeper, it seems like systemctl is shutting down the gateway and then restarting it.
When running sudo journalctl -u ignition.serviceI get this:
After running for 13 hours, shutdown, restart, shutdown and restart again.
There are other events like this with the system working correctly for 6 to 30 hs.
What can I check?
Look in the wrapper log. If the OS triggered this, there will be a message in there about trapping the TERM signal. Given the resources described, I suspect your application needs more that 6GB, and java itself is choking on memory exhaustion. That also produces relevant messages in the wrapper log.
(Or your application has a slow memory leak...)
Thanks for the tip. Here is the TERM signal in the wrapper log:
So, if I understand correctly, is the OS.
Never say never with the memory leak, but seems unlikely to me.
In the first post the log shows that when the service shuts down memory peaked at 3GB, so its well below the limit. In fact it shuts down almost inmediatly after with peak memory of 1.6GB.
Also, the wrapper log is very descriptive and I can see that the system was idling for a few hours before this restart event.
In the image I captured the second shutdown, and it shows no message between the normal startup messages and the TERM trapped.
What else can be checked? Something from the OS side?
EDIT:
Looking closer at the log, it seems it received the TERM signal even before it finished booting ??
Not sure. Typically the kernel's OOM killer is responsible, and would show in dmesg, but that seems unlikely.
Try looking at all of the journalctl output at that point in time, ±60 seconds.
Consider opening a support ticket to get more ideas from IA--I haven't seen this combination.
In the journalctl output, there are only the startup/shutdown messages. When the gateway runs for a few days without problem, there are no messages in between.
dmesg | grep ignition gives no results
I will contact support. Thanks anyway
