Gateway is Faulted

Hello, I am going through the lessons on inductive university, so I am constantly restoring backup gateways and having to restart my gateway. I have had many issues where the gateway will not restart (gateway control utility says “Gateway is faulted”).

I found this old thread (http://ec2-204-236-130-104.us-west-1.compute.amazonaws.com/forum/viewtopic.php?f=72&t=12719) which suggested deleting the windows temporary files and that seemed to kindof work for a few gateway restarts (sometimes I would restart the gateway and it would fault, then I restart it again immediately after and it would work), but now I can’t get my gateway to restart at all.

Below is what I think are the relevant portions of my wrapper log file. Also, I am running ignition version 7.7.3 (64-bit), Java version 8 update 31 (64-bit), and windows 7.

STATUS | wrapper | 2015/03/02 13:21:13 | --> Wrapper Started as Service STATUS | wrapper | 2015/03/02 13:21:13 | Java Service Wrapper Standard Edition 64-bit 3.5.25 STATUS | wrapper | 2015/03/02 13:21:13 | Copyright (C) 1999-2014 Tanuki Software, Ltd. All Rights Reserved. STATUS | wrapper | 2015/03/02 13:21:13 | http://wrapper.tanukisoftware.com STATUS | wrapper | 2015/03/02 13:21:13 | Licensed to Inductive Automation for Ignition Gateway STATUS | wrapper | 2015/03/02 13:21:13 | STATUS | wrapper | 2015/03/02 13:21:14 | Launching a JVM... INFO | jvm 1 | 2015/03/02 13:21:14 | WrapperManager: Initializing... INFO | jvm 1 | 2015/03/02 13:21:14 | 2015-03-02 13:21:14.834:INFO:oejs.Server:WrapperSimpleAppMain: jetty-9.1.z-SNAPSHOT INFO | jvm 1 | 2015/03/02 13:21:15 | 2015-03-02 13:21:15.287:INFO:oejw.StandardDescriptorProcessor:WrapperSimpleAppMain: NO JSP Support for /main, did not find org.apache.jasper.servlet.JspServlet INFO | jvm 1 | 2015/03/02 13:21:15 | INFO [SRClassLoader ] [13:21:15,427]: SRClassLoader init INFO | jvm 1 | 2015/03/02 13:21:15 | INFO [Application ] [13:21:15,906]: [WicketFilter] init: Wicket core library initializer INFO | jvm 1 | 2015/03/02 13:21:15 | INFO [RequestListenerInterface ] [13:21:15,906]: registered listener interface [RequestListenerInterface name=IBehaviorListener, method=public abstract void org.apache.wicket.behavior.IBehaviorListener.onRequest()] INFO | jvm 1 | 2015/03/02 13:21:15 | INFO [RequestListenerInterface ] [13:21:15,906]: registered listener interface [RequestListenerInterface name=IFormSubmitListener, method=public abstract void org.apache.wicket.markup.html.form.IFormSubmitListener.onFormSubmitted()] INFO | jvm 1 | 2015/03/02 13:21:15 | INFO [RequestListenerInterface ] [13:21:15,906]: registered listener interface [RequestListenerInterface name=ILinkListener, method=public abstract void org.apache.wicket.markup.html.link.ILinkListener.onLinkClicked()] INFO | jvm 1 | 2015/03/02 13:21:15 | INFO [RequestListenerInterface ] [13:21:15,906]: registered listener interface [RequestListenerInterface name=IOnChangeListener, method=public abstract void org.apache.wicket.markup.html.form.IOnChangeListener.onSelectionChanged()] INFO | jvm 1 | 2015/03/02 13:21:15 | INFO [RequestListenerInterface ] [13:21:15,906]: registered listener interface [RequestListenerInterface name=IRedirectListener, method=public abstract void org.apache.wicket.IRedirectListener.onRedirect()] INFO | jvm 1 | 2015/03/02 13:21:15 | INFO [RequestListenerInterface ] [13:21:15,906]: registered listener interface [RequestListenerInterface name=IResourceListener, method=public abstract void org.apache.wicket.IResourceListener.onResourceRequested()] INFO | jvm 1 | 2015/03/02 13:21:15 | INFO [Application ] [13:21:15,906]: [WicketFilter] init: Wicket extensions initializer INFO | jvm 1 | 2015/03/02 13:21:16 | INFO [SRContext ] [13:21:15,907]: Ignition["/main", state=STOPPED] ContextState = STARTING INFO | jvm 1 | 2015/03/02 13:21:16 | INFO [SRContext ] [13:21:15,928]: Starting Ignition 7.7.3 (b2015022512) INFO | jvm 1 | 2015/03/02 13:21:16 | INFO [SRContext ] [13:21:15,928]: Reloading JDBC classloader. INFO | jvm 1 | 2015/03/02 13:21:16 | INFO [SecureRandomProvider ] [13:21:16,032]: Creating SecureRandom object... INFO | jvm 1 | 2015/03/02 13:21:16 | INFO [SecureRandomProvider ] [13:21:16,032]: ... SecureRandom Created. INFO | jvm 1 | 2015/03/02 13:21:16 | INFO [SecureRandomProvider ] [13:21:16,032]: Secure random seed generated in 1ms INFO | jvm 1 | 2015/03/02 13:21:16 | INFO [WebApplication ] [13:21:16,110]: [WicketFilter] Started Wicket version 6.1.1 in DEPLOYMENT mode INFO | jvm 1 | 2015/03/02 13:21:16 | 2015-03-02 13:21:16.110:INFO:oejsh.ContextHandler:WrapperSimpleAppMain: Started o.e.j.w.WebAppContext@39b6a016{/main,file:/C:/Program%20Files/Inductive%20Automation/Ignition/webserver/webapps/main/,AVAILABLE} INFO | jvm 1 | 2015/03/02 13:21:16 | INFO [StateMonitoring ] [13:21:16,141]: The redundant node state has changed. Current state: [RedundancyState[role=Independent, activity level=Active, project state=Good, history level=Full] INFO | jvm 1 | 2015/03/02 13:21:16 | 2015-03-02 13:21:16.172:INFO:oejs.ServerConnector:WrapperSimpleAppMain: Started ServerConnector@53ace9c7{HTTP/1.1}{0.0.0.0:8088} INFO | jvm 1 | 2015/03/02 13:21:16 | 2015-03-02 13:21:16.360:INFO:oejs.ServerConnector:WrapperSimpleAppMain: Started ServerConnector@222da59{SSL-HTTP/1.1}{0.0.0.0:8043} INFO | jvm 1 | 2015/03/02 13:21:20 | WARN [ModuleManager ] [13:21:20,400]: Could not read license from HASP key. HASP login() failed. Last error = 7. INFO | jvm 1 | 2015/03/02 13:21:20 | INFO [ModuleManager ] [13:21:20,400]: Loading modules.... INFO | jvm 1 | 2015/03/02 13:21:23 | INFO [JarFile ] [13:21:23,194]: Packed jar file "xopc-client.jar" length=3kb, time=63ms INFO | jvm 1 | 2015/03/02 13:21:23 | INFO [JarFile ] [13:21:23,209]: Packed jar file "xopc-common.jar" length=1kb, time=0ms INFO | jvm 1 | 2015/03/02 13:21:23 | INFO [JarFile ] [13:21:23,381]: Packed jar file "vis-common.jar" length=3kb, time=31ms INFO | jvm 1 | 2015/03/02 13:21:23 | INFO [JarFile ] [13:21:23,631]: Packed jar file "sql-common.jar" length=24kb, time=203ms INFO | jvm 1 | 2015/03/02 13:21:23 | INFO [JarFile ] [13:21:23,740]: Packed jar file "opccom-client.jar" length=2kb, time=16ms INFO | jvm 1 | 2015/03/02 13:21:23 | INFO [JarFile ] [13:21:23,755]: Packed jar file "opccom-common.jar" length=3kb, time=0ms INFO | jvm 1 | 2015/03/02 13:21:24 | INFO [JarFile ] [13:21:24,067]: Packed jar file "mobile-common.jar" length=4kb, time=15ms INFO | jvm 1 | 2015/03/02 13:21:24 | INFO [LocalDBManagerImpl ] [13:21:24,114]: Starting up... INFO | jvm 1 | 2015/03/02 13:21:24 | INFO [LocalDBManagerImpl ] [13:21:24,114]: Looking for existing internal database "settings"... INFO | jvm 1 | 2015/03/02 13:21:24 | INFO [ENGINE ] [13:21:24,473]: dataFileCache open start INFO | jvm 1 | 2015/03/02 13:21:24 | INFO [ENGINE ] [13:21:24,489]: dataFileCache open end INFO | jvm 1 | 2015/03/02 13:21:24 | INFO [ENGINE ] [13:21:24,629]: checkpointClose start INFO | jvm 1 | 2015/03/02 13:21:24 | INFO [ENGINE ] [13:21:24,660]: dataFileCache commit start INFO | jvm 1 | 2015/03/02 13:21:24 | INFO [ENGINE ] [13:21:24,691]: checkpointClose end INFO | jvm 1 | 2015/03/02 13:21:24 | INFO [LocalDBManagerImpl ] [13:21:24,691]: ... found existing. INFO | jvm 1 | 2015/03/02 13:21:24 | INFO [ENGINE ] [13:21:24,863]: Checkpoint start INFO | jvm 1 | 2015/03/02 13:21:24 | INFO [ENGINE ] [13:21:24,863]: checkpointClose start INFO | jvm 1 | 2015/03/02 13:21:24 | INFO [ENGINE ] [13:21:24,863]: dataFileCache commit start INFO | jvm 1 | 2015/03/02 13:21:24 | INFO [ENGINE ] [13:21:24,894]: checkpointClose end INFO | jvm 1 | 2015/03/02 13:21:24 | INFO [ENGINE ] [13:21:24,894]: Checkpoint end INFO | jvm 1 | 2015/03/02 13:21:24 | INFO [LocalDBManagerImpl ] [13:21:24,910]: internal database "settings" started up successfully. INFO | jvm 1 | 2015/03/02 13:21:24 | INFO [LocalDBManagerImpl ] [13:21:24,910]: Registering autobackup task [owner=Internal DB Autobackup, name=internal database "settings"] INFO | jvm 1 | 2015/03/02 13:21:25 | INFO [Notification ] [13:21:24,957]: Alert notification system created. INFO | jvm 1 | 2015/03/02 13:21:26 | *sys-package-mgr*: processing new jar, 'C:\Program Files\Inductive Automation\Ignition\lib\wrapper.jar' INFO | jvm 1 | 2015/03/02 13:21:26 | *sys-package-mgr*: processing new jar, 'C:\Program Files\Inductive Automation\Ignition\lib\catapult.jar' INFO | jvm 1 | 2015/03/02 13:21:26 | *sys-package-mgr*: processing new jar, 'C:\Program Files\Java\jre1.8.0_31\lib\resources.jar' INFO | jvm 1 | 2015/03/02 13:21:26 | *sys-package-mgr*: processing new jar, 'C:\Program Files\Java\jre1.8.0_31\lib\rt.jar' INFO | jvm 1 | 2015/03/02 13:21:26 | *sys-package-mgr*: processing new jar, 'C:\Program Files\Java\jre1.8.0_31\lib\jsse.jar' INFO | jvm 1 | 2015/03/02 13:21:26 | *sys-package-mgr*: processing new jar, 'C:\Program Files\Java\jre1.8.0_31\lib\jce.jar' INFO | jvm 1 | 2015/03/02 13:21:26 | *sys-package-mgr*: processing new jar, 'C:\Program Files\Java\jre1.8.0_31\lib\charsets.jar' INFO | jvm 1 | 2015/03/02 13:21:26 | *sys-package-mgr*: processing new jar, 'C:\Program Files\Java\jre1.8.0_31\lib\jfr.jar' INFO | jvm 1 | 2015/03/02 13:21:26 | *sys-package-mgr*: processing new jar, 'C:\Program Files\Java\jre1.8.0_31\lib\ext\access-bridge-64.jar' INFO | jvm 1 | 2015/03/02 13:21:26 | *sys-package-mgr*: processing new jar, 'C:\Program Files\Java\jre1.8.0_31\lib\ext\cldrdata.jar' INFO | jvm 1 | 2015/03/02 13:21:27 | *sys-package-mgr*: processing new jar, 'C:\Program Files\Java\jre1.8.0_31\lib\ext\dnsns.jar' INFO | jvm 1 | 2015/03/02 13:21:27 | *sys-package-mgr*: processing new jar, 'C:\Program Files\Java\jre1.8.0_31\lib\ext\jaccess.jar' INFO | jvm 1 | 2015/03/02 13:21:27 | *sys-package-mgr*: processing new jar, 'C:\Program Files\Java\jre1.8.0_31\lib\ext\jfxrt.jar' INFO | jvm 1 | 2015/03/02 13:21:27 | *sys-package-mgr*: processing new jar, 'C:\Program Files\Java\jre1.8.0_31\lib\ext\localedata.jar' INFO | jvm 1 | 2015/03/02 13:21:27 | *sys-package-mgr*: processing new jar, 'C:\Program Files\Java\jre1.8.0_31\lib\ext\nashorn.jar' INFO | jvm 1 | 2015/03/02 13:21:27 | *sys-package-mgr*: processing new jar, 'C:\Program Files\Java\jre1.8.0_31\lib\ext\sunec.jar' INFO | jvm 1 | 2015/03/02 13:21:27 | *sys-package-mgr*: processing new jar, 'C:\Program Files\Java\jre1.8.0_31\lib\ext\sunjce_provider.jar' INFO | jvm 1 | 2015/03/02 13:21:27 | *sys-package-mgr*: processing new jar, 'C:\Program Files\Java\jre1.8.0_31\lib\ext\sunmscapi.jar' INFO | jvm 1 | 2015/03/02 13:21:27 | *sys-package-mgr*: processing new jar, 'C:\Program Files\Java\jre1.8.0_31\lib\ext\sunpkcs11.jar' INFO | jvm 1 | 2015/03/02 13:21:27 | *sys-package-mgr*: processing new jar, 'C:\Program Files\Java\jre1.8.0_31\lib\ext\zipfs.jar' INFO | jvm 1 | 2015/03/02 13:21:28 | INFO [ModuleManager ] [13:21:28,014]: Trial time reset. Time remaining = 7101. Notify modules = false. INFO | jvm 1 | 2015/03/02 13:21:28 | INFO [ModuleManager ] [13:21:28,014]: Setting up modules INFO | jvm 1 | 2015/03/02 13:21:28 | INFO [ModuleHook ] [13:21:28,295]: JNI Library: C:\Windows\TEMP\jniwrap-3.8.33650901840710059349.Build.3.8.31027919970\jniwrap64.dll INFO | jvm 1 | 2015/03/02 13:21:28 | ERROR [SRContext ] [13:21:28,326]: Error during context startup. INFO | jvm 1 | 2015/03/02 13:21:28 | java.lang.ExceptionInInitializerError INFO | jvm 1 | 2015/03/02 13:21:28 | at com.jniwrapper.win32.system.Kernel32.isUnicode(Kernel32.java:77) INFO | jvm 1 | 2015/03/02 13:21:28 | at com.jniwrapper.win32.FunctionName.<clinit>(FunctionName.java:24) INFO | jvm 1 | 2015/03/02 13:21:28 | at com.jniwrapper.win32.MessageLoopThread.<clinit>(MessageLoopThread.java:24) INFO | jvm 1 | 2015/03/02 13:21:28 | at com.inductiveautomation.opccom.ModuleHook.setup(ModuleHook.java:176) INFO | jvm 1 | 2015/03/02 13:21:28 | at com.inductiveautomation.ignition.gateway.modules.ModuleManagerImpl$LoadedModule.setup(ModuleManagerImpl.java:2187) INFO | jvm 1 | 2015/03/02 13:21:28 | at com.inductiveautomation.ignition.gateway.modules.ModuleManagerImpl.setup(ModuleManagerImpl.java:263) INFO | jvm 1 | 2015/03/02 13:21:28 | at com.inductiveautomation.ignition.gateway.SRContext.startupInternal(SRContext.java:1166) INFO | jvm 1 | 2015/03/02 13:21:28 | at com.inductiveautomation.ignition.gateway.redundancy.RedundancyManagerImpl.startup(RedundancyManagerImpl.java:192) INFO | jvm 1 | 2015/03/02 13:21:28 | at com.inductiveautomation.ignition.gateway.SRContext.asyncInit(SRContext.java:913) INFO | jvm 1 | 2015/03/02 13:21:28 | at com.inductiveautomation.ignition.gateway.SRContext.access$200(SRContext.java:246) INFO | jvm 1 | 2015/03/02 13:21:28 | at com.inductiveautomation.ignition.gateway.SRContext$2.run(SRContext.java:647) INFO | jvm 1 | 2015/03/02 13:21:28 | at com.inductiveautomation.ignition.common.execution.impl.BasicExecutionEngine$ThrowableCatchingRunnable.run(BasicExecutionEngine.java:537) INFO | jvm 1 | 2015/03/02 13:21:28 | at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) INFO | jvm 1 | 2015/03/02 13:21:28 | at java.util.concurrent.FutureTask.run(Unknown Source) INFO | jvm 1 | 2015/03/02 13:21:28 | at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source) INFO | jvm 1 | 2015/03/02 13:21:28 | at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) INFO | jvm 1 | 2015/03/02 13:21:28 | at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) INFO | jvm 1 | 2015/03/02 13:21:28 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) INFO | jvm 1 | 2015/03/02 13:21:28 | at java.lang.Thread.run(Unknown Source) INFO | jvm 1 | 2015/03/02 13:21:28 | Caused by: com.jniwrapper.JNIWrapperException: java.io.IOException: Access is denied INFO | jvm 1 | 2015/03/02 13:21:28 | Caused by: java.io.IOException: Access is denied INFO | jvm 1 | 2015/03/02 13:21:28 | INFO [SRContext ] [13:21:28,326]: Ignition["/main", state=STARTING] ContextState = FAULTED

This is strange, but this line:

INFO   | jvm 1    | 2015/03/02 13:21:28 | INFO  [ModuleHook                    ] [13:21:28,295]: JNI Library: C:\Windows\TEMP\jniwrap-3.8.33650901840710059349.Build.3.8.31027919970\jniwrap64.dll

and these lines:

INFO | jvm 1 | 2015/03/02 13:21:28 | Caused by: com.jniwrapper.JNIWrapperException: java.io.IOException: Access is denied INFO | jvm 1 | 2015/03/02 13:21:28 | Caused by: java.io.IOException: Access is denied

Seems to indicate the problem.

For some reason the context of the windows user that’s is running Ignition (generally ‘Local System’) has no access to a native library (jniwrap64.dll).

I’m not sure, but you can try this:

With the Ignition Service shutdown, try to locate and delete this folder: C:\Windows\TEMP\jniwrap-3.8.33650901840710059349.Build.3.8.31027919970\

Then restart the Ignition Service and check if works.

Hope this helps you.

Regards,

[quote=“pdibenedetto”]This is strange, but this line:

INFO   | jvm 1    | 2015/03/02 13:21:28 | INFO  [ModuleHook                    ] [13:21:28,295]: JNI Library: C:\Windows\TEMP\jniwrap-3.8.33650901840710059349.Build.3.8.31027919970\jniwrap64.dll

and these lines:

INFO | jvm 1 | 2015/03/02 13:21:28 | Caused by: com.jniwrapper.JNIWrapperException: java.io.IOException: Access is denied INFO | jvm 1 | 2015/03/02 13:21:28 | Caused by: java.io.IOException: Access is denied

Seems to indicate the problem.

For some reason the context of the windows user that’s is running Ignition (generally ‘Local System’) has no access to a native library (jniwrap64.dll).

I’m not sure, but you can try this:

With the Ignition Service shutdown, try to locate and delete this folder: C:\Windows\TEMP\jniwrap-3.8.33650901840710059349.Build.3.8.31027919970\

Then restart the Ignition Service and check if works.

Hope this helps you.

Regards,[/quote]

Like I mentioned, I found an old thread with those instructions (delete the temp files) and they work, kind of. I tried them a few times and it would take multiple attempts to make it work. I made this thread after one day trying that multiple times and it didn’t work. Then an hour or so after posting I tried again and the gateway started successfully.

So now its running but I will have to do more gateway restores to finish the ignition training and I doubt it will restart reliably. Also, more importantly, if I ever run into this issue at a customers site I’d like to have a real solution.