Gateway fails to start with MultiException[java.io.IOException: Invalid keystore format

Getting this error when starting up a gateway and it stops right after… Not sure where to go with this one!

INFO   | jvm 1    | 2022/06/09 10:54:28 | WrapperSimpleApp: 
INFO   | jvm 1    | 2022/06/09 10:54:28 | WrapperSimpleApp Error: Encountered an error running main:
INFO   | jvm 1    | 2022/06/09 10:54:28 | WrapperSimpleApp Error: MultiException[java.io.IOException: Invalid keystore format, java.io.IOException: Invalid keystore format]
INFO   | jvm 1    | 2022/06/09 10:54:28 | WrapperSimpleApp Error: 	at org.eclipse.jetty.server.Server.doStart(Server.java:347)
INFO   | jvm 1    | 2022/06/09 10:54:28 | WrapperSimpleApp Error: 	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
INFO   | jvm 1    | 2022/06/09 10:54:28 | WrapperSimpleApp Error: 	at com.inductiveautomation.catapult.Jetpad.start(Jetpad.java:370)
INFO   | jvm 1    | 2022/06/09 10:54:28 | WrapperSimpleApp Error: 	at com.inductiveautomation.catapult.Catapult.start(Catapult.java:140)
INFO   | jvm 1    | 2022/06/09 10:54:28 | WrapperSimpleApp Error: 	at com.inductiveautomation.catapult.Catapult.main(Catapult.java:62)
INFO   | jvm 1    | 2022/06/09 10:54:28 | WrapperSimpleApp Error: 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
INFO   | jvm 1    | 2022/06/09 10:54:28 | WrapperSimpleApp Error: 	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
INFO   | jvm 1    | 2022/06/09 10:54:28 | WrapperSimpleApp Error: 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
INFO   | jvm 1    | 2022/06/09 10:54:28 | WrapperSimpleApp Error: 	at java.lang.reflect.Method.invoke(Unknown Source)
INFO   | jvm 1    | 2022/06/09 10:54:28 | WrapperSimpleApp Error: 	at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:349)
INFO   | jvm 1    | 2022/06/09 10:54:28 | WrapperSimpleApp Error: 	at java.lang.Thread.run(Unknown Source)
INFO   | jvm 1    | 2022/06/09 10:54:28 | WrapperSimpleApp Error: 	Suppressed: java.io.IOException: Invalid keystore format
INFO   | jvm 1    | 2022/06/09 10:54:28 | WrapperSimpleApp Error: 		at sun.security.provider.JavaKeyStore.engineLoad(Unknown Source)
INFO   | jvm 1    | 2022/06/09 10:54:28 | WrapperSimpleApp Error: 		at sun.security.provider.JavaKeyStore$JKS.engineLoad(Unknown Source)
INFO   | jvm 1    | 2022/06/09 10:54:28 | WrapperSimpleApp Error: 		at sun.security.provider.KeyStoreDelegator.engineLoad(Unknown Source)
INFO   | jvm 1    | 2022/06/09 10:54:28 | WrapperSimpleApp Error: 		at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(Unknown Source)
INFO   | jvm 1    | 2022/06/09 10:54:28 | WrapperSimpleApp Error: 		at java.security.KeyStore.load(Unknown Source)
INFO   | jvm 1    | 2022/06/09 10:54:28 | WrapperSimpleApp Error: 		at org.eclipse.jetty.util.security.CertificateUtils.getKeyStore(CertificateUtils.java:52)
INFO   | jvm 1    | 2022/06/09 10:54:28 | WrapperSimpleApp Error: 		at org.eclipse.jetty.util.ssl.SslContextFactory.loadKeyStore(SslContextFactory.java:1016)
INFO   | jvm 1    | 2022/06/09 10:54:28 | WrapperSimpleApp Error: 		at org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:332)
INFO   | jvm 1    | 2022/06/09 10:54:28 | WrapperSimpleApp Error: 		at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
INFO   | jvm 1    | 2022/06/09 10:54:28 | WrapperSimpleApp Error: 		at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
INFO   | jvm 1    | 2022/06/09 10:54:28 | WrapperSimpleApp Error: 		at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
INFO   | jvm 1    | 2022/06/09 10:54:28 | WrapperSimpleApp Error: 		at org.eclipse.jetty.server.SslConnectionFactory.doStart(SslConnectionFactory.java:64)
INFO   | jvm 1    | 2022/06/09 10:54:28 | WrapperSimpleApp Error: 		at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
INFO   | jvm 1    | 2022/06/09 10:54:28 | WrapperSimpleApp Error: 		at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
INFO   | jvm 1    | 2022/06/09 10:54:28 | WrapperSimpleApp Error: 		at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
INFO   | jvm 1    | 2022/06/09 10:54:28 | WrapperSimpleApp Error: 		at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:260)
INFO   | jvm 1    | 2022/06/09 10:54:28 | WrapperSimpleApp Error: 		at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:81)
INFO   | jvm 1    | 2022/06/09 10:54:28 | WrapperSimpleApp Error: 		at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:244)
INFO   | jvm 1    | 2022/06/09 10:54:28 | WrapperSimpleApp Error: 		at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
INFO   | jvm 1    | 2022/06/09 10:54:28 | WrapperSimpleApp Error: 		at org.eclipse.jetty.server.Server.doStart(Server.java:384)
INFO   | jvm 1    | 2022/06/09 10:54:28 | WrapperSimpleApp Error: 		... 10 more
INFO   | jvm 1    | 2022/06/09 10:54:28 | WrapperSimpleApp Error: Caused by: java.io.IOException: Invalid keystore format
INFO   | jvm 1    | 2022/06/09 10:54:28 | WrapperSimpleApp Error: 	at sun.security.provider.JavaKeyStore.engineLoad(Unknown Source)
INFO   | jvm 1    | 2022/06/09 10:54:28 | WrapperSimpleApp Error: 	at sun.security.provider.JavaKeyStore$JKS.engineLoad(Unknown Source)
INFO   | jvm 1    | 2022/06/09 10:54:28 | WrapperSimpleApp Error: 	at sun.security.provider.KeyStoreDelegator.engineLoad(Unknown Source)
INFO   | jvm 1    | 2022/06/09 10:54:28 | WrapperSimpleApp Error: 	at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(Unknown Source)
INFO   | jvm 1    | 2022/06/09 10:54:28 | WrapperSimpleApp Error: 	at java.security.KeyStore.load(Unknown Source)
INFO   | jvm 1    | 2022/06/09 10:54:28 | WrapperSimpleApp Error: 	at org.eclipse.jetty.util.security.CertificateUtils.getKeyStore(CertificateUtils.java:52)
INFO   | jvm 1    | 2022/06/09 10:54:28 | WrapperSimpleApp Error: 	at org.eclipse.jetty.util.ssl.SslContextFactory.loadKeyStore(SslContextFactory.java:1016)
INFO   | jvm 1    | 2022/06/09 10:54:28 | WrapperSimpleApp Error: 	at org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:332)
INFO   | jvm 1    | 2022/06/09 10:54:28 | WrapperSimpleApp Error: 	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
INFO   | jvm 1    | 2022/06/09 10:54:28 | WrapperSimpleApp Error: 	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
INFO   | jvm 1    | 2022/06/09 10:54:28 | WrapperSimpleApp Error: 	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
INFO   | jvm 1    | 2022/06/09 10:54:28 | WrapperSimpleApp Error: 	at org.eclipse.jetty.server.SslConnectionFactory.doStart(SslConnectionFactory.java:64)
INFO   | jvm 1    | 2022/06/09 10:54:28 | WrapperSimpleApp Error: 	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
INFO   | jvm 1    | 2022/06/09 10:54:28 | WrapperSimpleApp Error: 	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
INFO   | jvm 1    | 2022/06/09 10:54:28 | WrapperSimpleApp Error: 	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
INFO   | jvm 1    | 2022/06/09 10:54:28 | WrapperSimpleApp Error: 	at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:260)
INFO   | jvm 1    | 2022/06/09 10:54:28 | WrapperSimpleApp Error: 	at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:81)
INFO   | jvm 1    | 2022/06/09 10:54:28 | WrapperSimpleApp Error: 	at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:244)
INFO   | jvm 1    | 2022/06/09 10:54:28 | WrapperSimpleApp Error: 	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
INFO   | jvm 1    | 2022/06/09 10:54:28 | WrapperSimpleApp Error: 	at org.eclipse.jetty.server.Server.doStart(Server.java:384)
INFO   | jvm 1    | 2022/06/09 10:54:28 | WrapperSimpleApp Error: 	... 10 more
INFO   | jvm 1    | 2022/06/09 10:54:28 | I [g.InternalDatabaseManager     ] [01:24:28]: Starting up... 
INFO   | jvm 1    | 2022/06/09 10:54:28 | I [P.InternalDatabase            ] [01:24:28]: Looking for existing internal database "config.idb"... 
INFO   | jvm 1    | 2022/06/09 10:54:28 | I [P.InternalDatabase            ] [01:24:28]: ... found existing. 
INFO   | jvm 1    | 2022/06/09 10:54:28 | I [P.InternalDatabase            ] [01:24:28]: internal database "config.idb" started up successfully. 
INFO   | jvm 1    | 2022/06/09 10:54:28 | I [P.InternalDatabase            ] [01:24:28]: Registering autobackup task [owner=Internal DB Autobackup, name=internal database "config.idb"] 
INFO   | jvm 1    | 2022/06/09 10:54:28 | I [g.ModuleManager               ] [01:24:28]: Loading modules.... 
INFO   | jvm 1    | 2022/06/09 10:54:29 | I [A.Notification                ] [01:24:29]: Alert notification system created. 
STATUS | wrapper  | 2022/06/09 10:54:31 | <-- Wrapper Stopped

Turns out this prop was empty in the ignition.conf file… no idea how it got wiped… I wonder what else has mysteriously vanished. My VM did get a windows update

wrapper.java.command=

The Windows JAVA_HOME environment variable is missing as well which is also very strange.

Did you recently upgrade Ignition? or was this just after reboot of routine Windows updates?

I haven’t made any changes whatsoever to Ignition in this VM - I haven’t touched the VM for a few weeks and it was running last time it was opened. So not really sure what happened