[Bug?] [OPC Connections] Connection appeared out of nowhere at 9am EST

After confirming with my team, we have come to the conclusion that an OPC Connection called Production magically appeared on our Ignition Gateway Server - 8.0.0-beta0 (b2019021502).
This occurred at 9am EST.

We first noticed it after our Log was getting spammed with [Route : Error handling route.] log entries.
Then we checked out Status > OPC Connections and saw the FAULTED connection.
There is no OPC Connection named Production in Config > OPC Client > OPC Connections.


Error from Connection:

java.lang.Exception: OPC server connection is faulted. Type “Production_ServerType” is not a known server type.
at com.inductiveautomation.ignition.gateway.opc.OpcConnectionManagerImpl.newOpcConnection(OpcConnectionManagerImpl.java:158)
at com.inductiveautomation.ignition.gateway.opc.OpcConnectionManagerImpl.addAllServers(OpcConnectionManagerImpl.java:134)
at com.inductiveautomation.ignition.gateway.opc.OpcConnectionManagerImpl.startup(OpcConnectionManagerImpl.java:83)
at com.inductiveautomation.ignition.gateway.IgnitionGateway.startupInternal(IgnitionGateway.java:1026)
at com.inductiveautomation.ignition.gateway.redundancy.RedundancyManagerImpl.startup(RedundancyManagerImpl.java:275)
at com.inductiveautomation.ignition.gateway.IgnitionGateway.initRedundancy(IgnitionGateway.java:657)
at com.inductiveautomation.ignition.gateway.IgnitionGateway.lambda$initInternal$0(IgnitionGateway.java:595)
at com.inductiveautomation.ignition.common.execution.impl.BasicExecutionEngine$ThrowableCatchingRunnable.run(BasicExecutionEngine.java:518)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)

8.0.0-beta0 (b2019021502)
Azul Systems, Inc. 11.0.2


Error from Logs:

org.eclipse.jetty.io.RuntimeIOException: org.eclipse.jetty.io.EofException

at org.eclipse.jetty.server.ResponseWriter.isOpen(ResponseWriter.java:133)

at org.eclipse.jetty.server.ResponseWriter.flush(ResponseWriter.java:146)

at com.inductiveautomation.ignition.gateway.dataroutes.Route.service(Route.java:286)

at com.inductiveautomation.ignition.gateway.dataroutes.RouteGroupImpl.service(RouteGroupImpl.java:49)

at com.inductiveautomation.ignition.gateway.dataroutes.DataServlet.service(DataServlet.java:87)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)

at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:852)

at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:535)

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)

at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)

at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)

at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)

at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)

at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)

at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)

at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)

at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)

at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)

at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)

at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)

at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:61)

at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)

at org.eclipse.jetty.server.Server.handle(Server.java:530)

at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347)

at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:256)

at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)

at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)

at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247)

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)

at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382)

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)

at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)

at java.base/java.lang.Thread.run(Unknown Source)

Caused by: org.eclipse.jetty.io.EofException: null

at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:292)

at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:429)

at org.eclipse.jetty.io.WriteFlusher.completeWrite(WriteFlusher.java:384)

at org.eclipse.jetty.io.ChannelEndPoint$3.run(ChannelEndPoint.java:139)

… 7 common frames omitted

Caused by: java.io.IOException: An existing connection was forcibly closed by the remote host

at java.base/sun.nio.ch.SocketDispatcher.write0(Native Method)

at java.base/sun.nio.ch.SocketDispatcher.write(Unknown Source)

at java.base/sun.nio.ch.IOUtil.writeFromNativeBuffer(Unknown Source)

at java.base/sun.nio.ch.IOUtil.write(Unknown Source)

at java.base/sun.nio.ch.IOUtil.write(Unknown Source)

at java.base/sun.nio.ch.SocketChannelImpl.write(Unknown Source)

at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:270)

[edit: fixed formatting, added picture]

Not magic, one of the OEE/Downtime modules adds that connection. It looks like you don’t have those installed any more (not sure they are updated for Ignition 8 yet), hence the error.

1 Like

Thanks! We uninstalled the faulted Sepasoft and CirrusLink modules.
Then restarted the Ignition services.
The connection is still there, but it is no longer spamming logs.

@Kevin.Herron, How do I get rid of the faulted Production OPC Connection on 8.1.3.
I don’t see a way to remove it - to my knowledge, neither Sepasoft or CirrusLink were ever installed on this server.

You definitely had a Sepasoft module installed at some point.

You’ll have to install it and then delete the connection or call support and they can run some delete queries against your internal DB to manually remove the entry.

1 Like

Install and Uninstall fixed it - Thank you!!