Gateway Service not starting on mac

mpervaiz · September 4, 2020, 4:26pm

I am using Ignition Gateway 8.0.16, and the gateway is not starting. I have tried to restart multiple times, restarted my computer and still nothing. Originally I had verfsion 8.0.15 installed and that was working fine.

OS: macOS Catalina
Java version: 14.0.2

gwcmd logs (last few lines) are below:

Thread [webserver-43] id=43, (TIMED_WAITING)
waiting for: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@490bce07
java.base@11.0.7/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.7/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
java.base@11.0.7/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:309)
app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:379)
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
java.base@11.0.7/java.lang.Thread.run(Unknown Source)
Thread [webserver-44-acceptor-0@541e3c81-ServerConnector@631ee61d{HTTP/1.1,[http/1.1]}{0.0.0.0:8088}] id=44, (RUNNABLE) (native)
owns synchronizer: java.util.concurrent.locks.ReentrantLock$NonfairSync@41e2c4c9
java.base@11.0.7/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
java.base@11.0.7/sun.nio.ch.ServerSocketChannelImpl.accept(Unknown Source)
java.base@11.0.7/sun.nio.ch.ServerSocketChannelImpl.accept(Unknown Source)
app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:385)
app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:701)
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
java.base@11.0.7/java.lang.Thread.run(Unknown Source)
Thread [webserver-54] id=54, (TIMED_WAITING)
waiting for: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5c606a77
java.base@11.0.7/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.7/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
java.base@11.0.7/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:309)
app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:379)
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
java.base@11.0.7/java.lang.Thread.run(Unknown Source)
Thread [webserver-56] id=56, (RUNNABLE)
owns monitor: sun.nio.ch.Util$2@71501517
owns monitor: sun.nio.ch.KQueueSelectorImpl@18409315
java.base@11.0.7/sun.nio.ch.KQueue.poll(Native Method)
java.base@11.0.7/sun.nio.ch.KQueueSelectorImpl.doSelect(Unknown Source)
java.base@11.0.7/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
java.base@11.0.7/sun.nio.ch.SelectorImpl.select(Unknown Source)
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:472)
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:409)
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:360)
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:184)
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:388)
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
java.base@11.0.7/java.lang.Thread.run(Unknown Source)
Thread [webserver-60] id=60, (RUNNABLE) (native)
owns monitor: sun.nio.ch.Util$2@2421f394
owns monitor: sun.nio.ch.KQueueSelectorImpl@73b2af55
java.base@11.0.7/sun.nio.ch.KQueue.poll(Native Method)
java.base@11.0.7/sun.nio.ch.KQueueSelectorImpl.doSelect(Unknown Source)
java.base@11.0.7/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
java.base@11.0.7/sun.nio.ch.SelectorImpl.select(Unknown Source)
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:472)
app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:409)
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:360)
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:184)
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:388)
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
java.base@11.0.7/java.lang.Thread.run(Unknown Source)
Daemon Thread [Wrapper-Connection] id=16, (RUNNABLE) (native)
java.base@11.0.7/java.net.SocketInputStream.socketRead0(Native Method)
java.base@11.0.7/java.net.SocketInputStream.socketRead(Unknown Source)
java.base@11.0.7/java.net.SocketInputStream.read(Unknown Source)
java.base@11.0.7/java.net.SocketInputStream.read(Unknown Source)
java.base@11.0.7/java.net.SocketInputStream.read(Unknown Source)
java.base@11.0.7/java.io.DataInputStream.readByte(Unknown Source)
app//org.tanukisoftware.wrapper.WrapperManager.handleBackend(WrapperManager.java:5812)
app//org.tanukisoftware.wrapper.WrapperManager.run(WrapperManager.java:6233)
java.base@11.0.7/java.lang.Thread.run(Unknown Source)
Daemon Thread [Wrapper-Control-Event-Monitor] id=14, (TIMED_WAITING)
java.base@11.0.7/java.lang.Thread.sleep(Native Method)
app//org.tanukisoftware.wrapper.WrapperManager$3.run(WrapperManager.java:1073)
Thread [WrapperSimpleAppMain] id=19, (WAITING)
waiting for: java.lang.Object@1bb357eb
java.base@11.0.7/java.lang.Object.wait(Native Method)
java.base@11.0.7/java.lang.Object.wait(Unknown Source)
app//org.eclipse.jetty.util.thread.QueuedThreadPool.join(QueuedThreadPool.java:567)
app//org.eclipse.jetty.server.Server.join(Server.java:550)
app//com.inductiveautomation.catapult.IgnitionServer.start(IgnitionServer.java:903)
app//com.inductiveautomation.ignition.gateway.web.WebResourceManagerImpl.start(WebResourceManagerImpl.java:319)
app//com.inductiveautomation.ignition.gateway.IgnitionGateway.start(IgnitionGateway.java:538)
app//com.inductiveautomation.ignition.gateway.IgnitionGateway.main(IgnitionGateway.java:275)
app//com.inductiveautomation.catapult.Catapult.main(Catapult.java:8)
java.base@11.0.7/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
java.base@11.0.7/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
java.base@11.0.7/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
java.base@11.0.7/java.lang.reflect.Method.invoke(Unknown Source)
app//org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:349)
java.base@11.0.7/java.lang.Thread.run(Unknown Source)

PGriffith · September 4, 2020, 4:29pm

Anything in the wrapper.log file?

mpervaiz · September 4, 2020, 4:39pm

Here is the link to the wrapper.log

Kevin.Herron · September 4, 2020, 4:43pm

Can you upload a full thread dump as well? Run ./gwcmd.sh -t > dump.txt after it gets to that point in the logs. If that doesn’t work you can find the process id using jps and then get a dump using jstack.

That “cpx-profiler” module is the last thing to log during startup… seems a bit suspect to me.

mpervaiz · September 4, 2020, 4:56pm

log gwcmd logs are here.

cpx-profiler is a module i wrote…

pturmel · September 4, 2020, 4:58pm

Does the gateway start without your module? That's kind of important.

mpervaiz · September 4, 2020, 4:59pm

I dont know. is there way to uninstall it from command line (i don’t have access to UI because it just keeps showing the message “Gateway is starting”)

pturmel · September 4, 2020, 5:00pm

Just move your module file out of the user-lib/modules folder, wherever that ends up on a Mac.

Kevin.Herron · September 4, 2020, 5:00pm

Your module is sleeping during startup() and preventing the gateway from starting:

Thread [gateway-shared-exec-engine-2] id=39, (TIMED_WAITING)
	owns synchronizer: java.util.concurrent.ThreadPoolExecutor$Worker@52d6a851
	java.base@11.0.7/java.lang.Thread.sleep(Native Method)
	net.cpxinfra.cpxtools.GatewayHook.startup(GatewayHook.java:37)
	app//com.inductiveautomation.ignition.gateway.modules.ModuleManagerImpl$LoadedModule.startup(ModuleManagerImpl.java:2433)
	app//com.inductiveautomation.ignition.gateway.modules.ModuleManagerImpl.startup(ModuleManagerImpl.java:411)
	app//com.inductiveautomation.ignition.gateway.IgnitionGateway.startupInternal(IgnitionGateway.java:1146)
	app//com.inductiveautomation.ignition.gateway.redundancy.RedundancyManagerImpl.startup(RedundancyManagerImpl.java:293)
	app//com.inductiveautomation.ignition.gateway.IgnitionGateway.initRedundancy(IgnitionGateway.java:710)
	app//com.inductiveautomation.ignition.gateway.IgnitionGateway.lambda$initInternal$0(IgnitionGateway.java:644)
	app//com.inductiveautomation.ignition.gateway.IgnitionGateway$$Lambda$344/0x00000008003d4440.run(Unknown Source)
	app//com.inductiveautomation.ignition.common.execution.impl.BasicExecutionEngine$ThrowableCatchingRunnable.run(BasicExecutionEngine.java:518)
	java.base@11.0.7/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	java.base@11.0.7/java.util.concurrent.FutureTask.run(Unknown Source)
	java.base@11.0.7/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
	java.base@11.0.7/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	java.base@11.0.7/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	java.base@11.0.7/java.lang.Thread.run(Unknown Source)

mpervaiz · September 4, 2020, 5:06pm

OH YOU GUYS ARE AMAZING… Thank you. I am removed the module and everything is fine now.