Gateway hanging at starting after startup

dgillis · September 15, 2020, 4:12pm

Hi there,

I’m using Ignition 8.0.16 on a win 10 VM in Virtualbox

I start the ignition service through the command line however the web interface is stuck perpetually at “Ignition Gateway is Starting”.

wrapper.log shows the following:

INFO   | jvm 1    | 2020/09/15 09:07:31 | java.lang.InternalError: java.security.NoSuchAlgorithmException: Error constructing implementation (algorithm: Default, provider: SunJSSE, class: sun.security.ssl.SSLContextImpl$DefaultSSLContext)
INFO   | jvm 1    | 2020/09/15 09:07:31 | 	at java.net.http/jdk.internal.net.http.HttpClientImpl.<init>(Unknown Source)
INFO   | jvm 1    | 2020/09/15 09:07:31 | 	at java.net.http/jdk.internal.net.http.HttpClientImpl.create(Unknown Source)
INFO   | jvm 1    | 2020/09/15 09:07:31 | 	at java.net.http/jdk.internal.net.http.HttpClientBuilderImpl.build(Unknown Source)
INFO   | jvm 1    | 2020/09/15 09:07:31 | 	at com.inductiveautomation.ignition.gateway.licensing.LicenseManagerImpl.createLeasedActivationStrategy(LicenseManagerImpl.java:154)
INFO   | jvm 1    | 2020/09/15 09:07:31 | 	at com.inductiveautomation.ignition.gateway.licensing.LicenseManagerImpl.<init>(LicenseManagerImpl.java:118)
INFO   | jvm 1    | 2020/09/15 09:07:31 | 	at com.inductiveautomation.ignition.gateway.modules.ModuleManagerImpl.init(ModuleManagerImpl.java:357)
INFO   | jvm 1    | 2020/09/15 09:07:31 | 	at com.inductiveautomation.ignition.gateway.IgnitionGateway.startupInternal(IgnitionGateway.java:851)
INFO   | jvm 1    | 2020/09/15 09:07:31 | 	at com.inductiveautomation.ignition.gateway.redundancy.RedundancyManagerImpl.startup(RedundancyManagerImpl.java:293)
INFO   | jvm 1    | 2020/09/15 09:07:31 | 	at com.inductiveautomation.ignition.gateway.IgnitionGateway.initRedundancy(IgnitionGateway.java:710)
INFO   | jvm 1    | 2020/09/15 09:07:31 | 	at com.inductiveautomation.ignition.gateway.IgnitionGateway.lambda$initInternal$0(IgnitionGateway.java:644)
INFO   | jvm 1    | 2020/09/15 09:07:31 | 	at com.inductiveautomation.ignition.common.execution.impl.BasicExecutionEngine$ThrowableCatchingRunnable.run(BasicExecutionEngine.java:518)
INFO   | jvm 1    | 2020/09/15 09:07:31 | 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
INFO   | jvm 1    | 2020/09/15 09:07:31 | 	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
INFO   | jvm 1    | 2020/09/15 09:07:31 | 	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
INFO   | jvm 1    | 2020/09/15 09:07:31 | 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
INFO   | jvm 1    | 2020/09/15 09:07:31 | 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
INFO   | jvm 1    | 2020/09/15 09:07:31 | 	at java.base/java.lang.Thread.run(Unknown Source)
INFO   | jvm 1    | 2020/09/15 09:07:31 | Caused by: java.security.NoSuchAlgorithmException: Error constructing implementation (algorithm: Default, provider: SunJSSE, class: sun.security.ssl.SSLContextImpl$DefaultSSLContext)
INFO   | jvm 1    | 2020/09/15 09:07:31 | 	at java.base/java.security.Provider$Service.newInstance(Unknown Source)
INFO   | jvm 1    | 2020/09/15 09:07:31 | 	at java.base/sun.security.jca.GetInstance.getInstance(Unknown Source)
INFO   | jvm 1    | 2020/09/15 09:07:31 | 	at java.base/sun.security.jca.GetInstance.getInstance(Unknown Source)
INFO   | jvm 1    | 2020/09/15 09:07:31 | 	at java.base/javax.net.ssl.SSLContext.getInstance(Unknown Source)
INFO   | jvm 1    | 2020/09/15 09:07:31 | 	at java.base/javax.net.ssl.SSLContext.getDefault(Unknown Source)
INFO   | jvm 1    | 2020/09/15 09:07:31 | 	... 17 common frames omitted
INFO   | jvm 1    | 2020/09/15 09:07:31 | Caused by: java.security.KeyStoreException: problem accessing trust store
INFO   | jvm 1    | 2020/09/15 09:07:31 | 	at java.base/sun.security.ssl.TrustManagerFactoryImpl.engineInit(Unknown Source)
INFO   | jvm 1    | 2020/09/15 09:07:31 | 	at java.base/javax.net.ssl.TrustManagerFactory.init(Unknown Source)
INFO   | jvm 1    | 2020/09/15 09:07:31 | 	at java.base/sun.security.ssl.SSLContextImpl$DefaultManagersHolder.getTrustManagers(Unknown Source)
INFO   | jvm 1    | 2020/09/15 09:07:31 | 	at java.base/sun.security.ssl.SSLContextImpl$DefaultManagersHolder.<clinit>(Unknown Source)
INFO   | jvm 1    | 2020/09/15 09:07:31 | 	at java.base/sun.security.ssl.SSLContextImpl$DefaultSSLContext.<init>(Unknown Source)
INFO   | jvm 1    | 2020/09/15 09:07:31 | 	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
INFO   | jvm 1    | 2020/09/15 09:07:31 | 	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
INFO   | jvm 1    | 2020/09/15 09:07:31 | 	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
INFO   | jvm 1    | 2020/09/15 09:07:31 | 	at java.base/java.lang.reflect.Constructor.newInstance(Unknown Source)
INFO   | jvm 1    | 2020/09/15 09:07:31 | 	at java.base/java.security.Provider.newInstanceUtil(Unknown Source)
INFO   | jvm 1    | 2020/09/15 09:07:31 | 	... 22 common frames omitted
INFO   | jvm 1    | 2020/09/15 09:07:31 | Caused by: java.io.IOException: toDerInputStream rejects tag type 0
INFO   | jvm 1    | 2020/09/15 09:07:31 | 	at java.base/sun.security.util.DerValue.toDerInputStream(Unknown Source)
INFO   | jvm 1    | 2020/09/15 09:07:31 | 	at java.base/sun.security.pkcs12.PKCS12KeyStore.engineLoad(Unknown Source)
INFO   | jvm 1    | 2020/09/15 09:07:31 | 	at java.base/sun.security.util.KeyStoreDelegator.engineLoad(Unknown Source)
INFO   | jvm 1    | 2020/09/15 09:07:31 | 	at java.base/java.security.KeyStore.load(Unknown Source)
INFO   | jvm 1    | 2020/09/15 09:07:31 | 	at java.base/sun.security.ssl.TrustStoreManager$TrustAnchorManager.loadKeyStore(Unknown Source)
INFO   | jvm 1    | 2020/09/15 09:07:31 | 	at java.base/sun.security.ssl.TrustStoreManager$TrustAnchorManager.getTrustedCerts(Unknown Source)
INFO   | jvm 1    | 2020/09/15 09:07:31 | 	at java.base/sun.security.ssl.TrustStoreManager.getTrustedCerts(Unknown Source)
INFO   | jvm 1    | 2020/09/15 09:07:31 | 	... 32 common frames omitted

Does anyone know why this is happening or how to fix it?

Thanks in advance.

code_skin · September 15, 2020, 4:16pm

I'm not sure what's happening, it seems SSL certificate related but you can check out this post to see if you can fix your issue (note this post is client while yours is gateway):

dgillis · September 15, 2020, 4:17pm

Thanks @code_skin

I have tried wiping out the cache already with no results. It’s doing the same thing unfortunately.

Kevin.Herron · September 15, 2020, 4:19pm

Try this:

Stop the gateway, delete the runtime folder at $IGNITION_INSTALL_DIR/lib/runtime/jre-win, start the gateway.

dgillis · September 15, 2020, 4:31pm

Thanks @Kevin.Herron.

After doing this it seems ignition can’t find the path to java:

INFO   | jvm 1    | 2020/09/15 09:25:05 | 09:25:05,402 |-INFO in ch.qos.logback.classic.AsyncAppender[DBAsync] - Queue flush finished successfully within timeout.
STATUS | wrapper  | 2020/09/15 09:25:05 | <-- Wrapper Stopped
STATUS | wrapper  | 2020/09/15 09:25:52 | --> Wrapper Started as Service
STATUS | wrapper  | 2020/09/15 09:25:52 | Java Service Wrapper Standard Edition 64-bit 3.5.42
STATUS | wrapper  | 2020/09/15 09:25:52 |   Copyright (C) 1999-2020 Tanuki Software, Ltd. All Rights Reserved.
STATUS | wrapper  | 2020/09/15 09:25:52 |     http://wrapper.tanukisoftware.com
STATUS | wrapper  | 2020/09/15 09:25:52 |   Licensed to Inductive Automation for Inductive Automation
STATUS | wrapper  | 2020/09/15 09:25:52 | 
FATAL  | wrapper  | 2020/09/15 09:25:52 | Unable to execute Java command.  The system cannot find the file specified. (0x2)
FATAL  | wrapper  | 2020/09/15 09:25:52 |     "lib\runtime\jre-win\bin\java" -version
ADVICE | wrapper  | 2020/09/15 09:25:52 | 
ADVICE | wrapper  | 2020/09/15 09:25:52 | --------------------------------------------------------------------
ADVICE | wrapper  | 2020/09/15 09:25:52 | Advice:
ADVICE | wrapper  | 2020/09/15 09:25:52 | Usually when the Wrapper fails to start the JVM process, it is
ADVICE | wrapper  | 2020/09/15 09:25:52 | because of a problem with the value of the configured Java command.
ADVICE | wrapper  | 2020/09/15 09:25:52 | Currently:
ADVICE | wrapper  | 2020/09/15 09:25:52 | wrapper.java.command=lib\runtime\jre-win\bin\java
ADVICE | wrapper  | 2020/09/15 09:25:52 | Please make sure that the PATH or any other referenced environment
ADVICE | wrapper  | 2020/09/15 09:25:52 | variables are correctly defined for the current environment.
ADVICE | wrapper  | 2020/09/15 09:25:52 | --------------------------------------------------------------------
ADVICE | wrapper  | 2020/09/15 09:25:52 | 
FATAL  | wrapper  | 2020/09/15 09:25:52 | Failed to resolve the version of Java.
STATUS | wrapper  | 2020/09/15 09:25:52 | <-- Wrapper Stopped
STATUS | wrapper  | 2020/09/15 09:27:25 | --> Wrapper Started as Service
STATUS | wrapper  | 2020/09/15 09:27:25 | Java Service Wrapper Standard Edition 64-bit 3.5.42
STATUS | wrapper  | 2020/09/15 09:27:25 |   Copyright (C) 1999-2020 Tanuki Software, Ltd. All Rights Reserved.
STATUS | wrapper  | 2020/09/15 09:27:25 |     http://wrapper.tanukisoftware.com
STATUS | wrapper  | 2020/09/15 09:27:25 |   Licensed to Inductive Automation for Inductive Automation
STATUS | wrapper  | 2020/09/15 09:27:25 | 
FATAL  | wrapper  | 2020/09/15 09:27:25 | Unable to execute Java command.  The system cannot find the file specified. (0x2)
FATAL  | wrapper  | 2020/09/15 09:27:25 |     "lib\runtime\jre-win\bin\java" -version
ADVICE | wrapper  | 2020/09/15 09:27:25 | 
ADVICE | wrapper  | 2020/09/15 09:27:25 | --------------------------------------------------------------------
ADVICE | wrapper  | 2020/09/15 09:27:25 | Advice:
ADVICE | wrapper  | 2020/09/15 09:27:25 | Usually when the Wrapper fails to start the JVM process, it is
ADVICE | wrapper  | 2020/09/15 09:27:25 | because of a problem with the value of the configured Java command.
ADVICE | wrapper  | 2020/09/15 09:27:25 | Currently:
ADVICE | wrapper  | 2020/09/15 09:27:25 | wrapper.java.command=lib\runtime\jre-win\bin\java
ADVICE | wrapper  | 2020/09/15 09:27:25 | Please make sure that the PATH or any other referenced environment
ADVICE | wrapper  | 2020/09/15 09:27:25 | variables are correctly defined for the current environment.
ADVICE | wrapper  | 2020/09/15 09:27:25 | --------------------------------------------------------------------
ADVICE | wrapper  | 2020/09/15 09:27:25 | 
FATAL  | wrapper  | 2020/09/15 09:27:25 | Failed to resolve the version of Java.
STATUS | wrapper  | 2020/09/15 09:27:26 | <-- Wrapper Stopped
STATUS | wrapper  | 2020/09/15 09:27:44 | --> Wrapper Started as Service
STATUS | wrapper  | 2020/09/15 09:27:44 | Java Service Wrapper Standard Edition 64-bit 3.5.42
STATUS | wrapper  | 2020/09/15 09:27:44 |   Copyright (C) 1999-2020 Tanuki Software, Ltd. All Rights Reserved.
STATUS | wrapper  | 2020/09/15 09:27:44 |     http://wrapper.tanukisoftware.com
STATUS | wrapper  | 2020/09/15 09:27:44 |   Licensed to Inductive Automation for Inductive Automation
STATUS | wrapper  | 2020/09/15 09:27:44 | 
FATAL  | wrapper  | 2020/09/15 09:27:45 | Unable to execute Java command.  The system cannot find the file specified. (0x2)
FATAL  | wrapper  | 2020/09/15 09:27:45 |     "lib\runtime\jre-win\bin\java" -version
ADVICE | wrapper  | 2020/09/15 09:27:45 | 
ADVICE | wrapper  | 2020/09/15 09:27:45 | --------------------------------------------------------------------
ADVICE | wrapper  | 2020/09/15 09:27:45 | Advice:
ADVICE | wrapper  | 2020/09/15 09:27:45 | Usually when the Wrapper fails to start the JVM process, it is
ADVICE | wrapper  | 2020/09/15 09:27:45 | because of a problem with the value of the configured Java command.
ADVICE | wrapper  | 2020/09/15 09:27:45 | Currently:
ADVICE | wrapper  | 2020/09/15 09:27:45 | wrapper.java.command=lib\runtime\jre-win\bin\java
ADVICE | wrapper  | 2020/09/15 09:27:45 | Please make sure that the PATH or any other referenced environment
ADVICE | wrapper  | 2020/09/15 09:27:45 | variables are correctly defined for the current environment.
ADVICE | wrapper  | 2020/09/15 09:27:45 | --------------------------------------------------------------------
ADVICE | wrapper  | 2020/09/15 09:27:45 | 
FATAL  | wrapper  | 2020/09/15 09:27:45 | Failed to resolve the version of Java.
STATUS | wrapper  | 2020/09/15 09:27:45 | <-- Wrapper Stopped

Kevin.Herron · September 15, 2020, 4:43pm

Is there still a jre-win.zip file in that runtime dir?

dgillis · September 15, 2020, 4:44pm

@Kevin.Herron yes there is.

Kevin.Herron · September 15, 2020, 4:44pm

Hmm. Well try unzipping it into a folder called jre-win. I thought it should happen automatically when the gateway starts up though…

franklepkowski · September 15, 2020, 5:04pm

What sort of memory are you allocating to the gateway? Is there sufficient RAM the accommodate this amount and support the OS properly?

Frank

leonard.smit · November 3, 2020, 5:20am

I had the exact issue happening when trying to recover from a power failure on the plant. After we restored power and got the systems up and running again, we encountered the exact same issue in the wrapper logs when trying the start the Gateway.

Any conclusion on why this might be happening?
Lenny

dgillis · November 3, 2020, 5:33am

I actually ended up reinstalling and restoring due to time constraints. Unfortunate. I would have liked to figure out the reason for this.

robert.zapalski · March 25, 2021, 5:18pm

seeing this issue today after we blue screened the desktop hosting our Edge Gateway. Any resolutions?

Kevin.Herron · March 25, 2021, 5:22pm

Same as mentioned here or in this thread: Gate way doesn't start after upgrade from 8.0.16 to 8.1

Delete some files and start it up again. We’re not sure why hard shutdowns are corrupting this file so easily.

nskoog · June 1, 2021, 1:52pm

We experienced the same issue after a power loss. We are on version 8.1.2, has there been any fix to this in newer releases or is this still an open issue? I was unable to find anything regarding this issue in the release notes.

Kevin.Herron · June 1, 2021, 2:35pm

A potential fix went into 8.1.6.

It's under this release note:

Improved gateway supplemental import mechanism and added additional logging to show certificates which are imported into the gateway's default trust store.

I don't think we called out the problem explicitly because we're still not able to reliably reproduce or verify the exact problem is fixed, but there were some changes made that hopefully fix the issue.

nskoog · June 1, 2021, 2:56pm

Thanks for the response!