Gateway Appears to Have Restarted on Its Own

Ignition
1

I discovered an issue yesterday where our Ignition instance failed over to the backup Ignition server. Upon investigation, it appears from the logs that the gateway service spontaneously restarted.

The first error in the logs was:

Unable to read "gateway.metroKeystoreAlias" gateway.xml property. Assuming metro keystore alias is: metro-key

image
image1217×777 70.2 KB

About 50 seconds later, the master server failed over to the backup.

image
image1220×705 46.9 KB

It appears that some of these errors could happen if someone was trying to install (or possibly update) Ignition, but we are not aware of anyone trying to update the server yesterday (Sunday morning).

Would anyone have any idea how I can find out more precisely what may have happened?

2

I am including another screen shot that shows that 2 seconds after the first error, the gateway service was starting.

image
image1223×847 71.5 KB

3

Check the wrapper.log file(s) - they have more information that can’t make it into the application logs you get on the web interface. Also, consider getting in contact with support (support.inductiveautomation.com) to start an actual official ticket to investigate this.

4

I second both of Paul’s points above.

One thing to check as well would be the uptime of the server that hosts the Master Ignition Gateway. Given your first screenshot we have Startup messages occurring without Shutdown messages. The uptime on the server could help rule out if the server itself went down at that time.

5

Thanks! I think I missed getting the wrapper logs from the time of the event, but I’ll keep that in mind for the future.

Thank you to @kvane also!

6

You are removing logs this fast? Are you sure they still arent there but now just have wrapper.log.1 ?