[Bug?] [OPC Connections] Connection appeared out of nowhere at 9am EST

joe.friedrich · February 15, 2019, 3:16pm

After confirming with my team, we have come to the conclusion that an OPC Connection called Production magically appeared on our Ignition Gateway Server - 8.0.0-beta0 (b2019021502).
This occurred at 9am EST.

We first noticed it after our Log was getting spammed with [Route : Error handling route.] log entries.
Then we checked out Status > OPC Connections and saw the FAULTED connection.
There is no OPC Connection named Production in Config > OPC Client > OPC Connections.

Error from Connection:

java.lang.Exception: OPC server connection is faulted. Type “Production_ServerType” is not a known server type.
at com.inductiveautomation.ignition.gateway.opc.OpcConnectionManagerImpl.newOpcConnection(OpcConnectionManagerImpl.java:158)
at com.inductiveautomation.ignition.gateway.opc.OpcConnectionManagerImpl.addAllServers(OpcConnectionManagerImpl.java:134)
at com.inductiveautomation.ignition.gateway.opc.OpcConnectionManagerImpl.startup(OpcConnectionManagerImpl.java:83)
at com.inductiveautomation.ignition.gateway.IgnitionGateway.startupInternal(IgnitionGateway.java:1026)
at com.inductiveautomation.ignition.gateway.redundancy.RedundancyManagerImpl.startup(RedundancyManagerImpl.java:275)
at com.inductiveautomation.ignition.gateway.IgnitionGateway.initRedundancy(IgnitionGateway.java:657)
at com.inductiveautomation.ignition.gateway.IgnitionGateway.lambda$initInternal$0(IgnitionGateway.java:595)
at com.inductiveautomation.ignition.common.execution.impl.BasicExecutionEngine$ThrowableCatchingRunnable.run(BasicExecutionEngine.java:518)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)

8.0.0-beta0 (b2019021502)
Azul Systems, Inc. 11.0.2

Error from Logs:

org.eclipse.jetty.io.RuntimeIOException: org.eclipse.jetty.io.EofException

at org.eclipse.jetty.server.ResponseWriter.isOpen(ResponseWriter.java:133)

at org.eclipse.jetty.server.ResponseWriter.flush(ResponseWriter.java:146)

at com.inductiveautomation.ignition.gateway.dataroutes.Route.service(Route.java:286)

at com.inductiveautomation.ignition.gateway.dataroutes.RouteGroupImpl.service(RouteGroupImpl.java:49)

at com.inductiveautomation.ignition.gateway.dataroutes.DataServlet.service(DataServlet.java:87)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)

at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:852)

at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:535)

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)

at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)

at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)

at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)

at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)

at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)

at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)

at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)

at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)

at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)

at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)

at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)

at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:61)

at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)

at org.eclipse.jetty.server.Server.handle(Server.java:530)

at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347)

at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:256)

at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)

at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)

at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247)

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)

at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382)

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)

at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)

at java.base/java.lang.Thread.run(Unknown Source)

Caused by: org.eclipse.jetty.io.EofException: null

at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:292)

at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:429)

at org.eclipse.jetty.io.WriteFlusher.completeWrite(WriteFlusher.java:384)

at org.eclipse.jetty.io.ChannelEndPoint$3.run(ChannelEndPoint.java:139)

… 7 common frames omitted

Caused by: java.io.IOException: An existing connection was forcibly closed by the remote host

at java.base/sun.nio.ch.SocketDispatcher.write0(Native Method)

at java.base/sun.nio.ch.SocketDispatcher.write(Unknown Source)

at java.base/sun.nio.ch.IOUtil.writeFromNativeBuffer(Unknown Source)

at java.base/sun.nio.ch.IOUtil.write(Unknown Source)

at java.base/sun.nio.ch.SocketChannelImpl.write(Unknown Source)

at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:270)

[edit: fixed formatting, added picture]

Kevin.Herron · February 15, 2019, 3:25pm

Not magic, one of the OEE/Downtime modules adds that connection. It looks like you don’t have those installed any more (not sure they are updated for Ignition 8 yet), hence the error.

joe.friedrich · February 15, 2019, 3:46pm

Thanks! We uninstalled the faulted Sepasoft and CirrusLink modules.
Then restarted the Ignition services.
The connection is still there, but it is no longer spamming logs.

martinel3746 · July 13, 2021, 4:39pm

@Kevin.Herron, How do I get rid of the faulted Production OPC Connection on 8.1.3.
I don’t see a way to remove it - to my knowledge, neither Sepasoft or CirrusLink were ever installed on this server.

Kevin.Herron · July 13, 2021, 4:59pm

You definitely had a Sepasoft module installed at some point.

You’ll have to install it and then delete the connection or call support and they can run some delete queries against your internal DB to manually remove the entry.

martinel3746 · July 13, 2021, 5:21pm

Install and Uninstall fixed it - Thank you!!

WPhan · August 22, 2022, 12:19pm

Installed the Production module, then went to OPC connections to delete the “Production” connection. Uninstalled the module and it got rid of the error.