# Server keeps dropping modbus connections to all plc's **URL:** \ **Category:** Ignition\ **Tags:** ignition78\ **Created:** [April 25, 2020, 4:42am UTC](https://forum.inductiveautomation.com/t/server-keeps-dropping-modbus-connections-to-all-plcs/34216 "2020-04-25T04:42:57Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![codersmurf](https://avatars.discourse-cdn.com/v4/letter/c/df788c/32.png) [@codersmurf](https://forum.inductiveautomation.com/u/codersmurf)\ **Post date:** [April 25, 2020, 4:42am UTC](https://forum.inductiveautomation.com/t/server-keeps-dropping-modbus-connections-to-all-plcs/34216/1 "2020-04-25T04:42:57Z")
So it just started today. Ignition can’t keep a connection to a single plc in the field. Keeps getting a bunch of these: INFO | jvm 1 | 2020/04/24 23:41:14 | ERROR [SocketIODelegate] [22:41:14,503]: [hostname=,port=503] Socket connection closed, DriverState was Connected. INFO | jvm 1 | 2020/04/24 23:41:14 | java.io.IOException: End of stream reached. INFO | jvm 1 | 2020/04/24 23:41:14 | at com.inductiveautomation.iosession.socket.AsyncSocketIOSession.run(AsyncSocketIOSession.java:74) INFO | jvm 1 | 2020/04/24 23:41:14 | at java.lang.Thread.run(Thread.java:745) Any thoughts, we are dead in the water. Thanks, ---
**Author:** ![pturmel](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pturmel/32/3295_2.png) [@pturmel](https://forum.inductiveautomation.com/u/pturmel)\ **Post date:** [April 25, 2020, 2:54pm UTC](https://forum.inductiveautomation.com/t/server-keeps-dropping-modbus-connections-to-all-plcs/34216/2 "2020-04-25T14:54:35Z")
> [@codersmurf](#): > > End of stream reached. That means something outside Ignition closed the connection. It is common for Modbus TCP devices to have a 10-second inactivity timeout. You might need to have at least one tag polled faster than your devices' timeouts to keep connections alive. ---
**Author:** ![codersmurf](https://avatars.discourse-cdn.com/v4/letter/c/df788c/32.png) [@codersmurf](https://forum.inductiveautomation.com/u/codersmurf)\ **Post date:** [April 25, 2020, 3:22pm UTC](https://forum.inductiveautomation.com/t/server-keeps-dropping-modbus-connections-to-all-plcs/34216/3 "2020-04-25T15:22:00Z")
Do you still see that as a possibility if it’s been working for years and all of a sudden every plc, over a hundred, stopped working simultaneously? ---
**Author:** ![drojas](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/drojas/32/3403_2.png) [@drojas](https://forum.inductiveautomation.com/u/drojas)\ **Post date:** [April 25, 2020, 5:13pm UTC](https://forum.inductiveautomation.com/t/server-keeps-dropping-modbus-connections-to-all-plcs/34216/4 "2020-04-25T17:13:26Z")
Have you tried restarting the Modbus driver? Also, sometimes just editing the device, then hit save helps re-establish and maintain connection. You said all of a sudden every PLC stopped working…is everything on a wired network or wireless? ---
**Author:** ![codersmurf](https://avatars.discourse-cdn.com/v4/letter/c/df788c/32.png) [@codersmurf](https://forum.inductiveautomation.com/u/codersmurf)\ **Post date:** [April 25, 2020, 5:15pm UTC](https://forum.inductiveautomation.com/t/server-keeps-dropping-modbus-connections-to-all-plcs/34216/5 "2020-04-25T17:15:10Z")
I’ve restarted the server a couple of times. The servers are wired, the plc’s are cell. Been that way for years. I can still hit the modem the plc’s are attached to and the configs on modems look good. ---
**Author:** ![pturmel](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pturmel/32/3295_2.png) [@pturmel](https://forum.inductiveautomation.com/u/pturmel)\ **Post date:** [April 25, 2020, 9:58pm UTC](https://forum.inductiveautomation.com/t/server-keeps-dropping-modbus-connections-to-all-plcs/34216/6 "2020-04-25T21:58:15Z")
> [@codersmurf](#): > > it’s been working for years Well, no. That suggests a change somewhere. OS change? OS Updates? OS Security Policy change? Network topology change? Network infrastructure OS updates? When something goes from working to non-working, troubleshooting 101 is "what else changed at that time?" Wireshark captures from a network node along the path from Ignition to the devices would be helpful, I suspect. ---
**Author:** ![codersmurf](https://avatars.discourse-cdn.com/v4/letter/c/df788c/32.png) [@codersmurf](https://forum.inductiveautomation.com/u/codersmurf)\ **Post date:** [April 25, 2020, 10:27pm UTC](https://forum.inductiveautomation.com/t/server-keeps-dropping-modbus-connections-to-all-plcs/34216/7 "2020-04-25T22:27:14Z")
True, but the only change was about a week before the problem there was an os update. This did jack up exim, which I just fixed an hour ago, couldn’t’ receive emails as root. Still can’t get the emails through sendgrid, but at least they aren’t sitting in a queue forever on the server with exim. I’ll try and get a \*nix version capture and see what it says. ---
**Author:** ![pturmel](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pturmel/32/3295_2.png) [@pturmel](https://forum.inductiveautomation.com/u/pturmel)\ **Post date:** [April 25, 2020, 10:51pm UTC](https://forum.inductiveautomation.com/t/server-keeps-dropping-modbus-connections-to-all-plcs/34216/8 "2020-04-25T22:51:53Z")
Anyone smart enough to run exim should be able to figure this out… (: { Full disclosure: I run exim for both home and work domains. I may be biased. } ---
**Author:** ![codersmurf](https://avatars.discourse-cdn.com/v4/letter/c/df788c/32.png) [@codersmurf](https://forum.inductiveautomation.com/u/codersmurf)\ **Post date:** [April 25, 2020, 11:01pm UTC](https://forum.inductiveautomation.com/t/server-keeps-dropping-modbus-connections-to-all-plcs/34216/9 "2020-04-25T23:01:09Z")
Hahaha, that’s what I figured:). So I ran wire shark, nothing but keep alives, nothing real suspicious going on ``` 1 0.000000000 → TCP 55 502 → 51974 [ACK] Seq=1 Ack=1 Win=512 Len=1 2 0.000024649 → TCP 54 51974 → 502 [ACK] Seq=1 Ack=2 Win=28400 Len=0 3 10.619541752 → TCP 55 [TCP Keep-Alive] 502 → 51974 [ACK] Seq=1 Ack=1 Win=512 Len=1 4 10.619566099 → TCP 54 [TCP Keep-Alive ACK] 51974 → 502 [ACK] Seq=1 Ack=2 Win=28400 Len=0 5 21.268434244 → TCP 55 [TCP Keep-Alive] 502 → 51974 [ACK] Seq=1 Ack=1 Win=512 Len=1 6 21.268469819 → TCP 54 [TCP Keep-Alive ACK] 51974 → 502 [ACK] Seq=1 Ack=2 Win=28400 Len=0 7 31.440785541 → TCP 55 [TCP Keep-Alive] 502 → 51974 [ACK] Seq=1 Ack=1 Win=512 Len=1 8 31.440835644 → TCP 54 [TCP Keep-Alive ACK] 51974 → 502 [ACK] Seq=1 Ack=2 Win=28400 Len=0 9 42.040864997 → TCP 55 [TCP Keep-Alive] 502 → 51974 [ACK] Seq=1 Ack=1 Win=512 Len=1 ``` 10 42.040899699 → TCP 54 [TCP Keep-Alive ACK] 51974 → 502 [ACK] Seq=1 Ack=2 Win=28400 Len=0 ---
**Author:** ![drojas](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/drojas/32/3403_2.png) [@drojas](https://forum.inductiveautomation.com/u/drojas)\ **Post date:** [April 26, 2020, 2:12am UTC](https://forum.inductiveautomation.com/t/server-keeps-dropping-modbus-connections-to-all-plcs/34216/10 "2020-04-26T02:12:41Z")
Not familiar with the linux side of things, but have you tried to just manually send out a Modbus read command to one of the PLCs using another PC? Possibly through the same network as the Ignition server. Just to prove whether it is a network connection related issue or Ignition itself. You could try Modbus Master or similar tool. PLCs are on cellular…I’m assuming through VPN. Keep Alives are going through, so I’m assuming other commands should work as well. ![image](https://us1.discourse-cdn.com/inductiveautomation/original/2X/b/b65218c01c05907083d48f8fef85840d9c961632.png) ---
**Author:** ![codersmurf](https://avatars.discourse-cdn.com/v4/letter/c/df788c/32.png) [@codersmurf](https://forum.inductiveautomation.com/u/codersmurf)\ **Post date:** [April 26, 2020, 3:38am UTC](https://forum.inductiveautomation.com/t/server-keeps-dropping-modbus-connections-to-all-plcs/34216/11 "2020-04-26T03:38:00Z")
modpoll returns the expected data ---
**Author:** ![codersmurf](https://avatars.discourse-cdn.com/v4/letter/c/df788c/32.png) [@codersmurf](https://forum.inductiveautomation.com/u/codersmurf)\ **Post date:** [April 26, 2020, 10:04pm UTC](https://forum.inductiveautomation.com/t/server-keeps-dropping-modbus-connections-to-all-plcs/34216/12 "2020-04-26T22:04:33Z")
I’m seeing a whole lot of RST ACK coming in, any thoughts? Thanks ---
**Author:** ![pturmel](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pturmel/32/3295_2.png) [@pturmel](https://forum.inductiveautomation.com/u/pturmel)\ **Post date:** [April 27, 2020, 12:16am UTC](https://forum.inductiveautomation.com/t/server-keeps-dropping-modbus-connections-to-all-plcs/34216/13 "2020-04-27T00:16:34Z")
From which direction? ~~Those should be preceded by a RST from the side that is deliberately closing the connection.~~ Actually, that is a combo packet where the sender is acknowledging the prior transmission from the peer, and also notifying the peer to close the connection. A deliberate connection close. More here: > ---
**Author:** ![pturmel](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pturmel/32/3295_2.png) [@pturmel](https://forum.inductiveautomation.com/u/pturmel)\ **Post date:** [April 27, 2020, 12:19am UTC](https://forum.inductiveautomation.com/t/server-keeps-dropping-modbus-connections-to-all-plcs/34216/14 "2020-04-27T00:19:27Z")
> [@codersmurf](#): > > nothing but keep alives, nothing real suspicious going on Having nothing but keep-alives is suspicious. Where are your regular reads? ---
**Author:** ![pturmel](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pturmel/32/3295_2.png) [@pturmel](https://forum.inductiveautomation.com/u/pturmel)\ **Post date:** [April 27, 2020, 12:32am UTC](https://forum.inductiveautomation.com/t/server-keeps-dropping-modbus-connections-to-all-plcs/34216/15 "2020-04-27T00:32:53Z")
Did something change recently that reduced your poll rate on these PLCs ? ---
**Author:** ![codersmurf](https://avatars.discourse-cdn.com/v4/letter/c/df788c/32.png) [@codersmurf](https://forum.inductiveautomation.com/u/codersmurf)\ **Post date:** [April 27, 2020, 1:41am UTC](https://forum.inductiveautomation.com/t/server-keeps-dropping-modbus-connections-to-all-plcs/34216/16 "2020-04-27T01:41:45Z")
Nothing changed on poll rates, so it seems like ignition is closing the connections then it’s it’s getting an ack from the plc. What would cause that? ---
**Author:** ![pturmel](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pturmel/32/3295_2.png) [@pturmel](https://forum.inductiveautomation.com/u/pturmel)\ **Post date:** [April 27, 2020, 1:42am UTC](https://forum.inductiveautomation.com/t/server-keeps-dropping-modbus-connections-to-all-plcs/34216/17 "2020-04-27T01:42:39Z")
Which end is sending the RST,ACK packets? That is the end that is closing the connection. ---
**Author:** ![codersmurf](https://avatars.discourse-cdn.com/v4/letter/c/df788c/32.png) [@codersmurf](https://forum.inductiveautomation.com/u/codersmurf)\ **Post date:** [April 27, 2020, 1:42am UTC](https://forum.inductiveautomation.com/t/server-keeps-dropping-modbus-connections-to-all-plcs/34216/18 "2020-04-27T01:42:47Z")
Scratch that, the plc’s are sending the acknowledgement. Currently have the poll rate at 15sec, always been that way, tried at 10 with same issue. ---
**Author:** ![pturmel](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pturmel/32/3295_2.png) [@pturmel](https://forum.inductiveautomation.com/u/pturmel)\ **Post date:** [April 27, 2020, 1:43am UTC](https://forum.inductiveautomation.com/t/server-keeps-dropping-modbus-connections-to-all-plcs/34216/19 "2020-04-27T01:43:17Z")
Try at nine seconds. ---
**Author:** ![pturmel](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pturmel/32/3295_2.png) [@pturmel](https://forum.inductiveautomation.com/u/pturmel)\ **Post date:** [April 27, 2020, 1:46am UTC](https://forum.inductiveautomation.com/t/server-keeps-dropping-modbus-connections-to-all-plcs/34216/20 "2020-04-27T01:46:22Z")
Have your PLCs gotten a firmware upgrade? Or have a configurable idle timeout that some other team is responsible for (and changed recently)? [Next page](https://forum.inductiveautomation.com/t/server-keeps-dropping-modbus-connections-to-all-plcs/34216.md?page=2)