Kepserver Connection Reset

Ignition
1

Kepsever EX 6.15 and Ignition 8.1.33 Vision
Kepserver Connection reset at more or less regular intervals ( 5 sec).
I have never seen this before.

java.net.SocketException: Connection reset

at java.base/sun.nio.ch.SocketChannelImpl.throwConnectionReset(Unknown Source)

at java.base/sun.nio.ch.SocketChannelImpl.read(Unknown Source)

at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:254)

at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1132)

at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:357)

at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:151)

at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)

at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)

at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)

at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)

at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)

at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)

at java.base/java.lang.Thread.run(Unknown Source)

Where to look into at first?

2

That's the kind of thing I'd expect from a "deep packet inspection" firewall. It sees a little bit of traffic, analyzes it, and kills it off.

3

I have turn off the Firewall for trial.
Got the same logs (different numbers):

java.net.SocketException: Connection reset

at java.base/sun.nio.ch.SocketChannelImpl.throwConnectionReset(Unknown Source)

at java.base/sun.nio.ch.SocketChannelImpl.read(Unknown Source)

at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:254)

at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1132)

at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:357)

at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:151)

at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)

at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)

at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)

at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)

at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)

at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)

at java.base/java.lang.Thread.run(Unknown Source)

4

I don't know of any operating system that has a deep packet inspection firewall, much less one you can turn off. Do you mean you got your IT group to ensure the path between your gateway and your OPC server has no such firewall?

You will likely need to perform a packet capture to see what else might be going on, but connection resets come from outside Ignition.

5

Is Kepware running on a different computer? Did you turn it off on that one?

This does look like a network or firewall issue.

6

Ignition and Kepserver run on the same machine.

7

Yes, I used IT department help. I guess it needs deeper packets investigation.

8

I collected Wireshark captures, however I cannot figure out what is the reason Kepserver to alterate its state "Faulted" to "Connected" to Ignition.
Any opinion?
Ignition_Kepserver_Wireshark_Capture.pcapng (5.0 MB)

9

Is Kepware crashing or restarting?

10

If it was crashing, would it be able to reconnect? I do not really how to distinguish "Faulted" from crashing in this case.

11

"Faulted" is just a term Ignition uses for a connection that was lost.

In the capture I can see you successfully connect, a couple seconds of traffic occurs (not sure what it is, you have security for the connection enabled), then Kepware abruptly closes the connection. The next few reconnect attempts fail immediately with TCP RST, then it eventually succeeds and this pattern repeats.

12

Yes, this is exactly what happens.
In this moment, all machines are offline (it is 3:43 PM here, Sunday, no shifts), just one machine that uses Codesys 2.3 driver in Kepserver is online. The traffic you have seen, Kevin, is from that machine.
Somethig important: if I use Quick Client in Kepserver, that machine never crashes, I was monitoring that all the morning, so Kepserver to the machine connection is not interrupted.

13

I think you need to get in touch with Kepware support. I don't really have any idea what's going on and it doesn't look like a problem on the Ignition side.

14

I did, couple of times. Issue has not been resolved the last 2 years.

15

Does Kepware not have its own logging action? Are there no clues there?

16

@pturmel, I have tons of emails. Kepware assume it is Microsoft issue.
Another assumption is related to Codesys 2.3 driver, if those 12 machines we have, connected by this driver, are disabled, the connection is stable and not interrupted.
The PLCs are ELAU C400, kind of legacy already. Codesys V3 cannot be used for that PLC.

17

Ah, yeah, that sounds like a Codesys problem. Sorry.

18

The problem starts after the weekend, after the machines have been stopped for more 24 hours. During normal week the connection is fine. So the question is why such connection cannot be re-established?

19

Sounds like a Codesys problem. :man_shrugging:

1 Like
20

I am thinking of restarting Kepserver Runtime service on timer.

1 Like