Tcp spurious retransmission

Brian · January 30, 2020, 9:16pm

Ignition 7.9.6

I’m having some trouble with a device disconnecting/reconnecting periodically (around every 10 minutes) from ignition. The device is an AutomationDirect BRX PLC. I’m connected via an ethernet connection to the built in ModbusTCP server. The server is set to allow 4 concurrent connections, with a 60 second inactivity timeout.

In ignition, I have some modbus registers and coils being read on a 5 second scanclass. I also have a laptop on the same network running the programming software viewing live values in the PLC. The laptop connection hasn’t missed a beat all day, no retrys, no errors in hours, so I’m assuming the network itself is fine. When ignition is connected, the device is showing about 1% utilization, with requests being handled in about 28 ms, so I don’t think the device itself is slammed. Scan time for the PLC is hovering around 8 ms.

I installed tshark on the server (headless ubuntu) and was watching the traffic to that PLC. Here’s a snapshot of the screen (the server is the .2 address and the PLC is the .106 address):

 2409 1619.276564058 192.168.150.2 â 192.168.150.106 Modbus/TCP 66    Query: Trans: 24397; Unit:   0, Func:   3: Read Holding Registers
 2411 1619.301431968 192.168.150.106 â 192.168.150.2 Modbus/TCP 157 Response: Trans: 24397; Unit:   0, Func:   3: Read Holding Registers
 2413 1619.312400295 192.168.150.2 â 192.168.150.106 Modbus/TCP 66    Query: Trans: 24398; Unit:   0, Func:   1: Read Coils
 2415 1619.338403811 192.168.150.106 â 192.168.150.2 Modbus/TCP 66 Response: Trans: 24398; Unit:   0, Func:   1: Read Coils
 2417 1624.276085460 192.168.150.2 â 192.168.150.106 Modbus/TCP 66    Query: Trans: 24399; Unit:   0, Func:   3: Read Holding Registers
 2419 1624.305256597 192.168.150.106 â 192.168.150.2 Modbus/TCP 157 Response: Trans: 24399; Unit:   0, Func:   3: Read Holding Registers
 2421 1624.312849650 192.168.150.2 â 192.168.150.106 Modbus/TCP 66    Query: Trans: 24400; Unit:   0, Func:   1: Read Coils
 2423 1624.342299284 192.168.150.106 â 192.168.150.2 Modbus/TCP 66 Response: Trans: 24400; Unit:   0, Func:   1: Read Coils
 2425 1629.276893446 192.168.150.2 â 192.168.150.106 Modbus/TCP 66    Query: Trans: 24401; Unit:   0, Func:   3: Read Holding Registers
 2427 1629.303503206 192.168.150.106 â 192.168.150.2 Modbus/TCP 157 Response: Trans: 24401; Unit:   0, Func:   3: Read Holding Registers
 2429 1629.312768626 192.168.150.2 â 192.168.150.106 Modbus/TCP 66    Query: Trans: 24402; Unit:   0, Func:   1: Read Coils
 2431 1629.340483222 192.168.150.106 â 192.168.150.2 Modbus/TCP 66 Response: Trans: 24402; Unit:   0, Func:   1: Read Coils
 2433 1634.276147301 192.168.150.2 â 192.168.150.106 Modbus/TCP 66    Query: Trans: 24403; Unit:   0, Func:   3: Read Holding Registers
 2435 1634.305894353 192.168.150.106 â 192.168.150.2 Modbus/TCP 157 Response: Trans: 24403; Unit:   0, Func:   3: Read Holding Registers
 2436 1634.481769896 192.168.150.2 â 192.168.150.106 Modbus/TCP 66 [TCP Spurious Retransmission]    Query: Trans: 24403; Unit:   0, Func:   3: Read Holding Registers
 2438 1634.897776210 192.168.150.2 â 192.168.150.106 Modbus/TCP 66 [TCP Spurious Retransmission]    Query: Trans: 24403; Unit:   0, Func:   3: Read Holding Registers
 2441 1635.729782478 192.168.150.2 â 192.168.150.106 Modbus/TCP 66 [TCP Spurious Retransmission]    Query: Trans: 24403; Unit:   0, Func:   3: Read Holding Registers
 2445 1637.397779658 192.168.150.2 â 192.168.150.106 Modbus/TCP 66 [TCP Spurious Retransmission]    Query: Trans: 24403; Unit:   0, Func:   3: Read Holding Registers
 2450 1640.729782619 192.168.150.2 â 192.168.150.106 Modbus/TCP 66 [TCP Spurious Retransmission]    Query: Trans: 24403; Unit:   0, Func:   3: Read Holding Registers
 2459 1647.401780399 192.168.150.2 â 192.168.150.106 Modbus/TCP 66 [TCP Spurious Retransmission]    Query: Trans: 24403; Unit:   0, Func:   3: Read Holding Registers
 2468 1665.490579240 192.168.150.2 â 192.168.150.106 Modbus/TCP 66    Query: Trans: 24410; Unit:   0, Func:   3: Read Holding Registers
 2470 1665.516158202 192.168.150.106 â 192.168.150.2 Modbus/TCP 157 Response: Trans: 24410; Unit:   0, Func:   3: Read Holding Registers
 2472 1665.518580265 192.168.150.2 â 192.168.150.106 Modbus/TCP 66    Query: Trans: 24411; Unit:   0, Func:   1: Read Coils
 2474 1665.553150475 192.168.150.106 â 192.168.150.2 Modbus/TCP 66 Response: Trans: 24411; Unit:   0, Func:   1: Read Coils
 2475 1665.553497298 192.168.150.2 â 192.168.150.106 Modbus/TCP 66    Query: Trans: 24412; Unit:   0, Func:   3: Read Holding Registers
 2477 1665.590212416 192.168.150.106 â 192.168.150.2 Modbus/TCP 157 Response: Trans: 24412; Unit:   0, Func:   3: Read Holding Registers
 2478 1665.590579613 192.168.150.2 â 192.168.150.106 Modbus/TCP 66    Query: Trans: 24413; Unit:   0, Func:   1: Read Coils
 2480 1665.626107681 192.168.150.106 â 192.168.150.2 Modbus/TCP 66 Response: Trans: 24413; Unit:   0, Func:   1: Read Coils

I believe the TCP Spurious retransmissions from ignition to the PLC are causing the problem. I’m at a loss as to how to troubleshoot further. Any suggestions?

Brian

Kevin.Herron · January 30, 2020, 9:20pm

Can you upload the packet capture somewhere? What happens in the Ignition logs at the time of the TCP retransmissions?

Brian · January 30, 2020, 9:23pm

I was just viewing the header information on screen. Let me see if I can find the ‘write to file’ incantation for the command…

I’ll grab some info from the logs too.

Brian · January 30, 2020, 9:42pm

Here’s the section of the wrapper.log file during a disconnect. Working on getting the capture file off the server. It’s small (45k). Not sure if I can attach it here. PM me your email and I can send it that way.

I’ll split the wrapper.log file into two posts, due to length.

INFO   | jvm 1    | 2020/01/30 15:33:21 | W [d.M.B.TimeoutDaemon           ] [21:33:21]: ScheduledRequest[com.inductiveautomation.xopc.drivers.modbus2.requests.ReadHoldingRegistersRequest@c613cfb] request with key "25141" failed due to timeout.
INFO   | jvm 1    | 2020/01/30 15:33:21 | W [d.M.ReadHoldingRegistersRequest] [21:33:21]: Request failed. FailureType==TIMEOUT
INFO   | jvm 1    | 2020/01/30 15:33:21 | java.lang.Exception: Request failed by TimeoutDaemon due to timeout: ScheduledRequest[com.inductiveautomation.xopc.drivers.modbus2.requests.ReadHoldingRegistersRequest@c613cfb]
INFO   | jvm 1    | 2020/01/30 15:33:21 |       at com.inductiveautomation.xopc.driver.api.BasicRequestCycle$TimeoutDaemon.failRequests(BasicRequestCycle.java:374)
INFO   | jvm 1    | 2020/01/30 15:33:21 |       at com.inductiveautomation.xopc.driver.api.BasicRequestCycle$TimeoutDaemon.run(BasicRequestCycle.java:327)
INFO   | jvm 1    | 2020/01/30 15:33:21 |       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
INFO   | jvm 1    | 2020/01/30 15:33:21 |       at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
INFO   | jvm 1    | 2020/01/30 15:33:21 |       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
INFO   | jvm 1    | 2020/01/30 15:33:21 |       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
INFO   | jvm 1    | 2020/01/30 15:33:21 |       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
INFO   | jvm 1    | 2020/01/30 15:33:21 |       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
INFO   | jvm 1    | 2020/01/30 15:33:21 |       at java.lang.Thread.run(Thread.java:748)
INFO   | jvm 1    | 2020/01/30 15:33:25 | W [d.M.B.TimeoutDaemon           ] [21:33:25]: ScheduledRequest[com.inductiveautomation.xopc.drivers.modbus2.requests.ReadCoilsRequest@362a19c8] request with key "25142" failed due to timeout.
INFO   | jvm 1    | 2020/01/30 15:33:25 | W [d.M.ReadCoilsRequest          ] [21:33:25]: Request failed. FailureType==TIMEOUT
INFO   | jvm 1    | 2020/01/30 15:33:25 | java.lang.Exception: Request failed by TimeoutDaemon due to timeout: ScheduledRequest[com.inductiveautomation.xopc.drivers.modbus2.requests.ReadCoilsRequest@362a19c8]
INFO   | jvm 1    | 2020/01/30 15:33:25 |       at com.inductiveautomation.xopc.driver.api.BasicRequestCycle$TimeoutDaemon.failRequests(BasicRequestCycle.java:374)
INFO   | jvm 1    | 2020/01/30 15:33:25 |       at com.inductiveautomation.xopc.driver.api.BasicRequestCycle$TimeoutDaemon.run(BasicRequestCycle.java:327)
INFO   | jvm 1    | 2020/01/30 15:33:25 |       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
INFO   | jvm 1    | 2020/01/30 15:33:25 |       at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
INFO   | jvm 1    | 2020/01/30 15:33:25 |       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
INFO   | jvm 1    | 2020/01/30 15:33:25 |       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
INFO   | jvm 1    | 2020/01/30 15:33:25 |       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
INFO   | jvm 1    | 2020/01/30 15:33:25 |       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
INFO   | jvm 1    | 2020/01/30 15:33:25 |       at java.lang.Thread.run(Thread.java:748)
INFO   | jvm 1    | 2020/01/30 15:33:29 | W [d.M.B.TimeoutDaemon           ] [21:33:29]: ScheduledRequest[com.inductiveautomation.xopc.drivers.modbus2.requests.ReadHoldingRegistersRequest@c613cfb] request with key "25143" failed due to timeout.
INFO   | jvm 1    | 2020/01/30 15:33:29 | W [d.M.ReadHoldingRegistersRequest] [21:33:29]: Request failed. FailureType==TIMEOUT
INFO   | jvm 1    | 2020/01/30 15:33:29 | java.lang.Exception: Request failed by TimeoutDaemon due to timeout: ScheduledRequest[com.inductiveautomation.xopc.drivers.modbus2.requests.ReadHoldingRegistersRequest@c613cfb]
INFO   | jvm 1    | 2020/01/30 15:33:29 |       at com.inductiveautomation.xopc.driver.api.BasicRequestCycle$TimeoutDaemon.failRequests(BasicRequestCycle.java:374)
INFO   | jvm 1    | 2020/01/30 15:33:29 |       at com.inductiveautomation.xopc.driver.api.BasicRequestCycle$TimeoutDaemon.run(BasicRequestCycle.java:327)
INFO   | jvm 1    | 2020/01/30 15:33:29 |       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
INFO   | jvm 1    | 2020/01/30 15:33:29 |       at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
INFO   | jvm 1    | 2020/01/30 15:33:29 |       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
INFO   | jvm 1    | 2020/01/30 15:33:29 |       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
INFO   | jvm 1    | 2020/01/30 15:33:29 |       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
INFO   | jvm 1    | 2020/01/30 15:33:29 |       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
INFO   | jvm 1    | 2020/01/30 15:33:29 |       at java.lang.Thread.run(Thread.java:748)
INFO   | jvm 1    | 2020/01/30 15:33:33 | E [d.M.ReadHoldingRegistersRequest] [21:33:33]: Received response with ExceptionCode: 0x08 (MemoryParityError).
INFO   | jvm 1    | 2020/01/30 15:33:33 | W [d.M.B.TimeoutDaemon           ] [21:33:33]: ScheduledRequest[com.inductiveautomation.xopc.drivers.modbus2.requests.ReadCoilsRequest@362a19c8] request with key "25144" failed due to timeout.
INFO   | jvm 1    | 2020/01/30 15:33:33 | W [d.M.ReadCoilsRequest          ] [21:33:33]: Request failed. FailureType==TIMEOUT
INFO   | jvm 1    | 2020/01/30 15:33:33 | java.lang.Exception: Request failed by TimeoutDaemon due to timeout: ScheduledRequest[com.inductiveautomation.xopc.drivers.modbus2.requests.ReadCoilsRequest@362a19c8]
INFO   | jvm 1    | 2020/01/30 15:33:33 |       at com.inductiveautomation.xopc.driver.api.BasicRequestCycle$TimeoutDaemon.failRequests(BasicRequestCycle.java:374)
INFO   | jvm 1    | 2020/01/30 15:33:33 |       at com.inductiveautomation.xopc.driver.api.BasicRequestCycle$TimeoutDaemon.run(BasicRequestCycle.java:327)
INFO   | jvm 1    | 2020/01/30 15:33:33 |       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
INFO   | jvm 1    | 2020/01/30 15:33:33 |       at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
INFO   | jvm 1    | 2020/01/30 15:33:33 |       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
INFO   | jvm 1    | 2020/01/30 15:33:33 |       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
INFO   | jvm 1    | 2020/01/30 15:33:33 |       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
INFO   | jvm 1    | 2020/01/30 15:33:33 |       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
INFO   | jvm 1    | 2020/01/30 15:33:33 |       at java.lang.Thread.run(Thread.java:748)
INFO   | jvm 1    | 2020/01/30 15:33:37 | W [d.M.B.TimeoutDaemon           ] [21:33:37]: ScheduledRequest[com.inductiveautomation.xopc.drivers.modbus2.requests.ReadHoldingRegistersRequest@c613cfb] request with key "25145" failed due to timeout.
INFO   | jvm 1    | 2020/01/30 15:33:37 | W [d.M.ReadHoldingRegistersRequest] [21:33:37]: Request failed. FailureType==TIMEOUT
INFO   | jvm 1    | 2020/01/30 15:33:37 | java.lang.Exception: Request failed by TimeoutDaemon due to timeout: ScheduledRequest[com.inductiveautomation.xopc.drivers.modbus2.requests.ReadHoldingRegistersRequest@c613cfb]
INFO   | jvm 1    | 2020/01/30 15:33:37 |       at com.inductiveautomation.xopc.driver.api.BasicRequestCycle$TimeoutDaemon.failRequests(BasicRequestCycle.java:374)
INFO   | jvm 1    | 2020/01/30 15:33:37 |       at com.inductiveautomation.xopc.driver.api.BasicRequestCycle$TimeoutDaemon.run(BasicRequestCycle.java:327)
INFO   | jvm 1    | 2020/01/30 15:33:37 |       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
INFO   | jvm 1    | 2020/01/30 15:33:37 |       at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
INFO   | jvm 1    | 2020/01/30 15:33:37 |       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
INFO   | jvm 1    | 2020/01/30 15:33:37 |       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
INFO   | jvm 1    | 2020/01/30 15:33:37 |       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
INFO   | jvm 1    | 2020/01/30 15:33:37 |       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
INFO   | jvm 1    | 2020/01/30 15:33:37 |       at java.lang.Thread.run(Thread.java:748)
INFO   | jvm 1    | 2020/01/30 15:33:41 | W [d.M.B.TimeoutDaemon           ] [21:33:41]: ScheduledRequest[com.inductiveautomation.xopc.drivers.modbus2.requests.ReadCoilsRequest@362a19c8] request with key "25146" failed due to timeout.
INFO   | jvm 1    | 2020/01/30 15:33:41 | W [d.M.ReadCoilsRequest          ] [21:33:41]: Request failed. FailureType==TIMEOUT
INFO   | jvm 1    | 2020/01/30 15:33:41 | java.lang.Exception: Request failed by TimeoutDaemon due to timeout: ScheduledRequest[com.inductiveautomation.xopc.drivers.modbus2.requests.ReadCoilsRequest@362a19c8]
INFO   | jvm 1    | 2020/01/30 15:33:41 |       at com.inductiveautomation.xopc.driver.api.BasicRequestCycle$TimeoutDaemon.failRequests(BasicRequestCycle.java:374)
INFO   | jvm 1    | 2020/01/30 15:33:41 |       at com.inductiveautomation.xopc.driver.api.BasicRequestCycle$TimeoutDaemon.run(BasicRequestCycle.java:327)
INFO   | jvm 1    | 2020/01/30 15:33:41 |       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
INFO   | jvm 1    | 2020/01/30 15:33:41 |       at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
INFO   | jvm 1    | 2020/01/30 15:33:41 |       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
INFO   | jvm 1    | 2020/01/30 15:33:41 |       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
INFO   | jvm 1    | 2020/01/30 15:33:41 |       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
INFO   | jvm 1    | 2020/01/30 15:33:41 |       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
INFO   | jvm 1    | 2020/01/30 15:33:41 |       at java.lang.Thread.run(Thread.java:748)
INFO   | jvm 1    | 2020/01/30 15:33:45 | W [d.M.B.TimeoutDaemon           ] [21:33:45]: ScheduledRequest[com.inductiveautomation.xopc.drivers.modbus2.requests.ReadHoldingRegistersRequest@c613cfb] request with key "25147" failed due to timeout.
INFO   | jvm 1    | 2020/01/30 15:33:45 | W [d.M.ReadHoldingRegistersRequest] [21:33:45]: Request failed. FailureType==TIMEOUT
INFO   | jvm 1    | 2020/01/30 15:33:45 | java.lang.Exception: Request failed by TimeoutDaemon due to timeout: ScheduledRequest[com.inductiveautomation.xopc.drivers.modbus2.requests.ReadHoldingRegistersRequest@c613cfb]
INFO   | jvm 1    | 2020/01/30 15:33:45 |       at com.inductiveautomation.xopc.driver.api.BasicRequestCycle$TimeoutDaemon.failRequests(BasicRequestCycle.java:374)
INFO   | jvm 1    | 2020/01/30 15:33:45 |       at com.inductiveautomation.xopc.driver.api.BasicRequestCycle$TimeoutDaemon.run(BasicRequestCycle.java:327)
INFO   | jvm 1    | 2020/01/30 15:33:45 |       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
INFO   | jvm 1    | 2020/01/30 15:33:45 |       at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
INFO   | jvm 1    | 2020/01/30 15:33:45 |       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
INFO   | jvm 1    | 2020/01/30 15:33:45 |       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
INFO   | jvm 1    | 2020/01/30 15:33:45 |       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
INFO   | jvm 1    | 2020/01/30 15:33:45 |       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
INFO   | jvm 1    | 2020/01/30 15:33:45 |       at java.lang.Thread.run(Thread.java:748)
INFO   | jvm 1    | 2020/01/30 15:33:49 | W [d.M.B.TimeoutDaemon           ] [21:33:49]: ScheduledRequest[com.inductiveautomation.xopc.drivers.modbus2.requests.ReadCoilsRequest@362a19c8] request with key "25148" failed due to timeout.
INFO   | jvm 1    | 2020/01/30 15:33:49 | W [d.M.ReadCoilsRequest          ] [21:33:49]: Request failed. FailureType==TIMEOUT
INFO   | jvm 1    | 2020/01/30 15:33:49 | java.lang.Exception: Request failed by TimeoutDaemon due to timeout: ScheduledRequest[com.inductiveautomation.xopc.drivers.modbus2.requests.ReadCoilsRequest@362a19c8]
INFO   | jvm 1    | 2020/01/30 15:33:49 |       at com.inductiveautomation.xopc.driver.api.BasicRequestCycle$TimeoutDaemon.failRequests(BasicRequestCycle.java:374)
INFO   | jvm 1    | 2020/01/30 15:33:49 |       at com.inductiveautomation.xopc.driver.api.BasicRequestCycle$TimeoutDaemon.run(BasicRequestCycle.java:327)
INFO   | jvm 1    | 2020/01/30 15:33:49 |       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
INFO   | jvm 1    | 2020/01/30 15:33:49 |       at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
INFO   | jvm 1    | 2020/01/30 15:33:49 |       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
INFO   | jvm 1    | 2020/01/30 15:33:49 |       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
INFO   | jvm 1    | 2020/01/30 15:33:49 |       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
INFO   | jvm 1    | 2020/01/30 15:33:49 |       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
INFO   | jvm 1    | 2020/01/30 15:33:49 |       at java.lang.Thread.run(Thread.java:748)
INFO   | jvm 1    | 2020/01/30 15:33:52 | W [d.M.B.TimeoutDaemon           ] [21:33:52]: ScheduledRequest[com.inductiveautomation.xopc.drivers.modbus2.requests.ReadHoldingRegistersRequest@c613cfb] request with key "25149" failed due to timeout.
INFO   | jvm 1    | 2020/01/30 15:33:52 | W [d.M.ReadHoldingRegistersRequest] [21:33:52]: Request failed. FailureType==TIMEOUT
INFO   | jvm 1    | 2020/01/30 15:33:52 | java.lang.Exception: Request failed by TimeoutDaemon due to timeout: ScheduledRequest[com.inductiveautomation.xopc.drivers.modbus2.requests.ReadHoldingRegistersRequest@c613cfb]
INFO   | jvm 1    | 2020/01/30 15:33:52 |       at com.inductiveautomation.xopc.driver.api.BasicRequestCycle$TimeoutDaemon.failRequests(BasicRequestCycle.java:374)
INFO   | jvm 1    | 2020/01/30 15:33:52 |       at com.inductiveautomation.xopc.driver.api.BasicRequestCycle$TimeoutDaemon.run(BasicRequestCycle.java:327)
INFO   | jvm 1    | 2020/01/30 15:33:52 |       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
INFO   | jvm 1    | 2020/01/30 15:33:52 |       at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
INFO   | jvm 1    | 2020/01/30 15:33:52 |       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
INFO   | jvm 1    | 2020/01/30 15:33:52 |       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
INFO   | jvm 1    | 2020/01/30 15:33:52 |       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
INFO   | jvm 1    | 2020/01/30 15:33:52 |       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
INFO   | jvm 1    | 2020/01/30 15:33:52 |       at java.lang.Thread.run(Thread.java:748)
INFO   | jvm 1    | 2020/01/30 15:33:55 | W [d.M.B.TimeoutDaemon           ] [21:33:55]: ScheduledRequest[com.inductiveautomation.xopc.drivers.modbus2.requests.ReadCoilsRequest@362a19c8] request with key "25150" failed due to timeout.
INFO   | jvm 1    | 2020/01/30 15:33:55 | W [d.M.ReadCoilsRequest          ] [21:33:55]: Request failed. FailureType==TIMEOUT
INFO   | jvm 1    | 2020/01/30 15:33:55 | java.lang.Exception: Request failed by TimeoutDaemon due to timeout: ScheduledRequest[com.inductiveautomation.xopc.drivers.modbus2.requests.ReadCoilsRequest@362a19c8]
INFO   | jvm 1    | 2020/01/30 15:33:55 |       at com.inductiveautomation.xopc.driver.api.BasicRequestCycle$TimeoutDaemon.failRequests(BasicRequestCycle.java:374)
INFO   | jvm 1    | 2020/01/30 15:33:55 |       at com.inductiveautomation.xopc.driver.api.BasicRequestCycle$TimeoutDaemon.run(BasicRequestCycle.java:327)
INFO   | jvm 1    | 2020/01/30 15:33:55 |       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
INFO   | jvm 1    | 2020/01/30 15:33:55 |       at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
INFO   | jvm 1    | 2020/01/30 15:33:55 |       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
INFO   | jvm 1    | 2020/01/30 15:33:55 |       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
INFO   | jvm 1    | 2020/01/30 15:33:55 |       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
INFO   | jvm 1    | 2020/01/30 15:33:55 |       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
INFO   | jvm 1    | 2020/01/30 15:33:55 |       at java.lang.Thread.run(Thread.java:748)
INFO   | jvm 1    | 2020/01/30 15:33:59 | W [d.M.B.TimeoutDaemon           ] [21:33:59]: ScheduledRequest[com.inductiveautomation.xopc.drivers.modbus2.requests.ReadHoldingRegistersRequest@c613cfb] request with key "25151" failed due to timeout.
INFO   | jvm 1    | 2020/01/30 15:33:59 | W [d.M.ReadHoldingRegistersRequest] [21:33:59]: Request failed. FailureType==TIMEOUT
INFO   | jvm 1    | 2020/01/30 15:33:59 | java.lang.Exception: Request failed by TimeoutDaemon due to timeout: ScheduledRequest[com.inductiveautomation.xopc.drivers.modbus2.requests.ReadHoldingRegistersRequest@c613cfb]
INFO   | jvm 1    | 2020/01/30 15:33:59 |       at com.inductiveautomation.xopc.driver.api.BasicRequestCycle$TimeoutDaemon.failRequests(BasicRequestCycle.java:374)
INFO   | jvm 1    | 2020/01/30 15:33:59 |       at com.inductiveautomation.xopc.driver.api.BasicRequestCycle$TimeoutDaemon.run(BasicRequestCycle.java:327)
INFO   | jvm 1    | 2020/01/30 15:33:59 |       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
INFO   | jvm 1    | 2020/01/30 15:33:59 |       at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
INFO   | jvm 1    | 2020/01/30 15:33:59 |       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
INFO   | jvm 1    | 2020/01/30 15:33:59 |       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
INFO   | jvm 1    | 2020/01/30 15:33:59 |       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
INFO   | jvm 1    | 2020/01/30 15:33:59 |       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
INFO   | jvm 1    | 2020/01/30 15:33:59 |       at java.lang.Thread.run(Thread.java:748)
INFO   | jvm 1    | 2020/01/30 15:34:03 | W [d.M.B.TimeoutDaemon           ] [21:34:03]: ScheduledRequest[com.inductiveautomation.xopc.drivers.modbus2.requests.ReadCoilsRequest@362a19c8] request with key "25152" failed due to timeout.
INFO   | jvm 1    | 2020/01/30 15:34:03 | W [d.M.ReadCoilsRequest          ] [21:34:03]: Request failed. FailureType==TIMEOUT
INFO   | jvm 1    | 2020/01/30 15:34:03 | java.lang.Exception: Request failed by TimeoutDaemon due to timeout: ScheduledRequest[com.inductiveautomation.xopc.drivers.modbus2.requests.ReadCoilsRequest@362a19c8]
INFO   | jvm 1    | 2020/01/30 15:34:03 |       at com.inductiveautomation.xopc.driver.api.BasicRequestCycle$TimeoutDaemon.failRequests(BasicRequestCycle.java:374)
INFO   | jvm 1    | 2020/01/30 15:34:03 |       at com.inductiveautomation.xopc.driver.api.BasicRequestCycle$TimeoutDaemon.run(BasicRequestCycle.java:327)
INFO   | jvm 1    | 2020/01/30 15:34:03 |       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
INFO   | jvm 1    | 2020/01/30 15:34:03 |       at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
INFO   | jvm 1    | 2020/01/30 15:34:03 |       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
INFO   | jvm 1    | 2020/01/30 15:34:03 |       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
INFO   | jvm 1    | 2020/01/30 15:34:03 |       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
INFO   | jvm 1    | 2020/01/30 15:34:03 |       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

Brian · January 30, 2020, 9:43pm

Second half of wrapper.log:

INFO   | jvm 1    | 2020/01/30 15:34:03 |       at java.lang.Thread.run(Thread.java:748)
INFO   | jvm 1    | 2020/01/30 15:34:07 | W [d.M.B.TimeoutDaemon           ] [21:34:07]: ScheduledRequest[com.inductiveautomation.xopc.drivers.modbus2.requests.ReadHoldingRegistersRequest@c613cfb] request with key "25153" failed due to timeout.
INFO   | jvm 1    | 2020/01/30 15:34:07 | W [d.M.ReadHoldingRegistersRequest] [21:34:07]: Request failed. FailureType==TIMEOUT
INFO   | jvm 1    | 2020/01/30 15:34:07 | java.lang.Exception: Request failed by TimeoutDaemon due to timeout: ScheduledRequest[com.inductiveautomation.xopc.drivers.modbus2.requests.ReadHoldingRegistersRequest@c613cfb]
INFO   | jvm 1    | 2020/01/30 15:34:07 |       at com.inductiveautomation.xopc.driver.api.BasicRequestCycle$TimeoutDaemon.failRequests(BasicRequestCycle.java:374)
INFO   | jvm 1    | 2020/01/30 15:34:07 |       at com.inductiveautomation.xopc.driver.api.BasicRequestCycle$TimeoutDaemon.run(BasicRequestCycle.java:327)
INFO   | jvm 1    | 2020/01/30 15:34:07 |       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
INFO   | jvm 1    | 2020/01/30 15:34:07 |       at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
INFO   | jvm 1    | 2020/01/30 15:34:07 |       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
INFO   | jvm 1    | 2020/01/30 15:34:07 |       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
INFO   | jvm 1    | 2020/01/30 15:34:07 |       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
INFO   | jvm 1    | 2020/01/30 15:34:07 |       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
INFO   | jvm 1    | 2020/01/30 15:34:07 |       at java.lang.Thread.run(Thread.java:748)
INFO   | jvm 1    | 2020/01/30 15:34:11 | W [d.M.B.TimeoutDaemon           ] [21:34:11]: ScheduledRequest[com.inductiveautomation.xopc.drivers.modbus2.requests.ReadCoilsRequest@362a19c8] request with key "25154" failed due to timeout.
INFO   | jvm 1    | 2020/01/30 15:34:11 | W [d.M.ReadCoilsRequest          ] [21:34:11]: Request failed. FailureType==TIMEOUT
INFO   | jvm 1    | 2020/01/30 15:34:11 | java.lang.Exception: Request failed by TimeoutDaemon due to timeout: ScheduledRequest[com.inductiveautomation.xopc.drivers.modbus2.requests.ReadCoilsRequest@362a19c8]
INFO   | jvm 1    | 2020/01/30 15:34:11 |       at com.inductiveautomation.xopc.driver.api.BasicRequestCycle$TimeoutDaemon.failRequests(BasicRequestCycle.java:374)
INFO   | jvm 1    | 2020/01/30 15:34:11 |       at com.inductiveautomation.xopc.driver.api.BasicRequestCycle$TimeoutDaemon.run(BasicRequestCycle.java:327)
INFO   | jvm 1    | 2020/01/30 15:34:11 |       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
INFO   | jvm 1    | 2020/01/30 15:34:11 |       at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
INFO   | jvm 1    | 2020/01/30 15:34:11 |       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
INFO   | jvm 1    | 2020/01/30 15:34:11 |       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
INFO   | jvm 1    | 2020/01/30 15:34:11 |       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
INFO   | jvm 1    | 2020/01/30 15:34:11 |       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
INFO   | jvm 1    | 2020/01/30 15:34:11 |       at java.lang.Thread.run(Thread.java:748)
INFO   | jvm 1    | 2020/01/30 15:34:12 | E [d.M.ReadHoldingRegistersRequest] [21:34:12]: Received response with ExceptionCode: 0x08 (MemoryParityError).
INFO   | jvm 1    | 2020/01/30 15:34:15 | W [d.M.B.TimeoutDaemon           ] [21:34:15]: ScheduledRequest[com.inductiveautomation.xopc.drivers.modbus2.requests.ReadHoldingRegistersRequest@c613cfb] request with key "25155" failed due to timeout.
INFO   | jvm 1    | 2020/01/30 15:34:15 | W [d.M.ReadHoldingRegistersRequest] [21:34:15]: Request failed. FailureType==TIMEOUT
INFO   | jvm 1    | 2020/01/30 15:34:15 | java.lang.Exception: Request failed by TimeoutDaemon due to timeout: ScheduledRequest[com.inductiveautomation.xopc.drivers.modbus2.requests.ReadHoldingRegistersRequest@c613cfb]
INFO   | jvm 1    | 2020/01/30 15:34:15 |       at com.inductiveautomation.xopc.driver.api.BasicRequestCycle$TimeoutDaemon.failRequests(BasicRequestCycle.java:374)
INFO   | jvm 1    | 2020/01/30 15:34:15 |       at com.inductiveautomation.xopc.driver.api.BasicRequestCycle$TimeoutDaemon.run(BasicRequestCycle.java:327)
INFO   | jvm 1    | 2020/01/30 15:34:15 |       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
INFO   | jvm 1    | 2020/01/30 15:34:15 |       at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
INFO   | jvm 1    | 2020/01/30 15:34:15 |       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
INFO   | jvm 1    | 2020/01/30 15:34:15 |       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
INFO   | jvm 1    | 2020/01/30 15:34:15 |       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
INFO   | jvm 1    | 2020/01/30 15:34:15 |       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
INFO   | jvm 1    | 2020/01/30 15:34:15 |       at java.lang.Thread.run(Thread.java:748)
INFO   | jvm 1    | 2020/01/30 15:34:19 | W [d.M.B.TimeoutDaemon           ] [21:34:19]: ScheduledRequest[com.inductiveautomation.xopc.drivers.modbus2.requests.ReadCoilsRequest@362a19c8] request with key "25156" failed due to timeout.
INFO   | jvm 1    | 2020/01/30 15:34:19 | W [d.M.ReadCoilsRequest          ] [21:34:19]: Request failed. FailureType==TIMEOUT
INFO   | jvm 1    | 2020/01/30 15:34:19 | java.lang.Exception: Request failed by TimeoutDaemon due to timeout: ScheduledRequest[com.inductiveautomation.xopc.drivers.modbus2.requests.ReadCoilsRequest@362a19c8]
INFO   | jvm 1    | 2020/01/30 15:34:19 |       at com.inductiveautomation.xopc.driver.api.BasicRequestCycle$TimeoutDaemon.failRequests(BasicRequestCycle.java:374)
INFO   | jvm 1    | 2020/01/30 15:34:19 |       at com.inductiveautomation.xopc.driver.api.BasicRequestCycle$TimeoutDaemon.run(BasicRequestCycle.java:327)
INFO   | jvm 1    | 2020/01/30 15:34:19 |       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
INFO   | jvm 1    | 2020/01/30 15:34:19 |       at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
INFO   | jvm 1    | 2020/01/30 15:34:19 |       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
INFO   | jvm 1    | 2020/01/30 15:34:19 |       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
INFO   | jvm 1    | 2020/01/30 15:34:19 |       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
INFO   | jvm 1    | 2020/01/30 15:34:19 |       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
INFO   | jvm 1    | 2020/01/30 15:34:19 |       at java.lang.Thread.run(Thread.java:748)
INFO   | jvm 1    | 2020/01/30 15:34:23 | W [d.M.B.TimeoutDaemon           ] [21:34:23]: ScheduledRequest[com.inductiveautomation.xopc.drivers.modbus2.requests.ReadHoldingRegistersRequest@c613cfb] request with key "25157" failed due to timeout.
INFO   | jvm 1    | 2020/01/30 15:34:23 | W [d.M.ReadHoldingRegistersRequest] [21:34:23]: Request failed. FailureType==TIMEOUT
INFO   | jvm 1    | 2020/01/30 15:34:23 | java.lang.Exception: Request failed by TimeoutDaemon due to timeout: ScheduledRequest[com.inductiveautomation.xopc.drivers.modbus2.requests.ReadHoldingRegistersRequest@c613cfb]
INFO   | jvm 1    | 2020/01/30 15:34:23 |       at com.inductiveautomation.xopc.driver.api.BasicRequestCycle$TimeoutDaemon.failRequests(BasicRequestCycle.java:374)
INFO   | jvm 1    | 2020/01/30 15:34:23 |       at com.inductiveautomation.xopc.driver.api.BasicRequestCycle$TimeoutDaemon.run(BasicRequestCycle.java:327)
INFO   | jvm 1    | 2020/01/30 15:34:23 |       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
INFO   | jvm 1    | 2020/01/30 15:34:23 |       at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
INFO   | jvm 1    | 2020/01/30 15:34:23 |       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
INFO   | jvm 1    | 2020/01/30 15:34:23 |       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
INFO   | jvm 1    | 2020/01/30 15:34:23 |       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
INFO   | jvm 1    | 2020/01/30 15:34:23 |       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
INFO   | jvm 1    | 2020/01/30 15:34:23 |       at java.lang.Thread.run(Thread.java:748)
INFO   | jvm 1    | 2020/01/30 15:34:27 | W [d.M.B.TimeoutDaemon           ] [21:34:27]: ScheduledRequest[com.inductiveautomation.xopc.drivers.modbus2.requests.ReadCoilsRequest@362a19c8] request with key "25158" failed due to timeout.
INFO   | jvm 1    | 2020/01/30 15:34:27 | W [d.M.ReadCoilsRequest          ] [21:34:27]: Request failed. FailureType==TIMEOUT
INFO   | jvm 1    | 2020/01/30 15:34:27 | java.lang.Exception: Request failed by TimeoutDaemon due to timeout: ScheduledRequest[com.inductiveautomation.xopc.drivers.modbus2.requests.ReadCoilsRequest@362a19c8]
INFO   | jvm 1    | 2020/01/30 15:34:27 |       at com.inductiveautomation.xopc.driver.api.BasicRequestCycle$TimeoutDaemon.failRequests(BasicRequestCycle.java:374)
INFO   | jvm 1    | 2020/01/30 15:34:27 |       at com.inductiveautomation.xopc.driver.api.BasicRequestCycle$TimeoutDaemon.run(BasicRequestCycle.java:327)
INFO   | jvm 1    | 2020/01/30 15:34:27 |       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
INFO   | jvm 1    | 2020/01/30 15:34:27 |       at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
INFO   | jvm 1    | 2020/01/30 15:34:27 |       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
INFO   | jvm 1    | 2020/01/30 15:34:27 |       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
INFO   | jvm 1    | 2020/01/30 15:34:27 |       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
INFO   | jvm 1    | 2020/01/30 15:34:27 |       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
INFO   | jvm 1    | 2020/01/30 15:34:27 |       at java.lang.Thread.run(Thread.java:748)
INFO   | jvm 1    | 2020/01/30 15:34:31 | W [d.M.B.TimeoutDaemon           ] [21:34:31]: ScheduledRequest[com.inductiveautomation.xopc.drivers.modbus2.requests.ReadHoldingRegistersRequest@c613cfb] request with key "25159" failed due to timeout.
INFO   | jvm 1    | 2020/01/30 15:34:31 | W [d.M.ReadHoldingRegistersRequest] [21:34:31]: Request failed. FailureType==TIMEOUT
INFO   | jvm 1    | 2020/01/30 15:34:31 | java.lang.Exception: Request failed by TimeoutDaemon due to timeout: ScheduledRequest[com.inductiveautomation.xopc.drivers.modbus2.requests.ReadHoldingRegistersRequest@c613cfb]
INFO   | jvm 1    | 2020/01/30 15:34:31 |       at com.inductiveautomation.xopc.driver.api.BasicRequestCycle$TimeoutDaemon.failRequests(BasicRequestCycle.java:374)
INFO   | jvm 1    | 2020/01/30 15:34:31 |       at com.inductiveautomation.xopc.driver.api.BasicRequestCycle$TimeoutDaemon.run(BasicRequestCycle.java:327)
INFO   | jvm 1    | 2020/01/30 15:34:31 |       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
INFO   | jvm 1    | 2020/01/30 15:34:31 |       at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
INFO   | jvm 1    | 2020/01/30 15:34:31 |       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
INFO   | jvm 1    | 2020/01/30 15:34:31 |       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
INFO   | jvm 1    | 2020/01/30 15:34:31 |       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
INFO   | jvm 1    | 2020/01/30 15:34:31 |       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
INFO   | jvm 1    | 2020/01/30 15:34:31 |       at java.lang.Thread.run(Thread.java:748)
INFO   | jvm 1    | 2020/01/30 15:34:35 | W [d.M.B.TimeoutDaemon           ] [21:34:35]: ScheduledRequest[com.inductiveautomation.xopc.drivers.modbus2.requests.ReadCoilsRequest@362a19c8] request with key "25160" failed due to timeout.
INFO   | jvm 1    | 2020/01/30 15:34:35 | W [d.M.ReadCoilsRequest          ] [21:34:35]: Request failed. FailureType==TIMEOUT
INFO   | jvm 1    | 2020/01/30 15:34:35 | java.lang.Exception: Request failed by TimeoutDaemon due to timeout: ScheduledRequest[com.inductiveautomation.xopc.drivers.modbus2.requests.ReadCoilsRequest@362a19c8]
INFO   | jvm 1    | 2020/01/30 15:34:35 |       at com.inductiveautomation.xopc.driver.api.BasicRequestCycle$TimeoutDaemon.failRequests(BasicRequestCycle.java:374)
INFO   | jvm 1    | 2020/01/30 15:34:35 |       at com.inductiveautomation.xopc.driver.api.BasicRequestCycle$TimeoutDaemon.run(BasicRequestCycle.java:327)
INFO   | jvm 1    | 2020/01/30 15:34:35 |       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
INFO   | jvm 1    | 2020/01/30 15:34:35 |       at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
INFO   | jvm 1    | 2020/01/30 15:34:35 |       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
INFO   | jvm 1    | 2020/01/30 15:34:35 |       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
INFO   | jvm 1    | 2020/01/30 15:34:35 |       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
INFO   | jvm 1    | 2020/01/30 15:34:35 |       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
INFO   | jvm 1    | 2020/01/30 15:34:35 |       at java.lang.Thread.run(Thread.java:748)
INFO   | jvm 1    | 2020/01/30 15:34:39 | W [d.M.B.TimeoutDaemon           ] [21:34:39]: ScheduledRequest[com.inductiveautomation.xopc.drivers.modbus2.requests.ReadHoldingRegistersRequest@c613cfb] request with key "25161" failed due to timeout.
INFO   | jvm 1    | 2020/01/30 15:34:39 | W [d.M.ReadHoldingRegistersRequest] [21:34:39]: Request failed. FailureType==TIMEOUT
INFO   | jvm 1    | 2020/01/30 15:34:39 | java.lang.Exception: Request failed by TimeoutDaemon due to timeout: ScheduledRequest[com.inductiveautomation.xopc.drivers.modbus2.requests.ReadHoldingRegistersRequest@c613cfb]
INFO   | jvm 1    | 2020/01/30 15:34:39 |       at com.inductiveautomation.xopc.driver.api.BasicRequestCycle$TimeoutDaemon.failRequests(BasicRequestCycle.java:374)
INFO   | jvm 1    | 2020/01/30 15:34:39 |       at com.inductiveautomation.xopc.driver.api.BasicRequestCycle$TimeoutDaemon.run(BasicRequestCycle.java:327)
INFO   | jvm 1    | 2020/01/30 15:34:39 |       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
INFO   | jvm 1    | 2020/01/30 15:34:39 |       at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
INFO   | jvm 1    | 2020/01/30 15:34:39 |       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
INFO   | jvm 1    | 2020/01/30 15:34:39 |       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
INFO   | jvm 1    | 2020/01/30 15:34:39 |       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
INFO   | jvm 1    | 2020/01/30 15:34:39 |       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
INFO   | jvm 1    | 2020/01/30 15:34:39 |       at java.lang.Thread.run(Thread.java:748)
INFO   | jvm 1    | 2020/01/30 15:34:43 | W [d.M.B.TimeoutDaemon           ] [21:34:43]: ScheduledRequest[com.inductiveautomation.xopc.drivers.modbus2.requests.ReadCoilsRequest@362a19c8] request with key "25162" failed due to timeout.
INFO   | jvm 1    | 2020/01/30 15:34:43 | W [d.M.ReadCoilsRequest          ] [21:34:43]: Request failed. FailureType==TIMEOUT
INFO   | jvm 1    | 2020/01/30 15:34:43 | java.lang.Exception: Request failed by TimeoutDaemon due to timeout: ScheduledRequest[com.inductiveautomation.xopc.drivers.modbus2.requests.ReadCoilsRequest@362a19c8]
INFO   | jvm 1    | 2020/01/30 15:34:43 |       at com.inductiveautomation.xopc.driver.api.BasicRequestCycle$TimeoutDaemon.failRequests(BasicRequestCycle.java:374)
INFO   | jvm 1    | 2020/01/30 15:34:43 |       at com.inductiveautomation.xopc.driver.api.BasicRequestCycle$TimeoutDaemon.run(BasicRequestCycle.java:327)
INFO   | jvm 1    | 2020/01/30 15:34:43 |       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
INFO   | jvm 1    | 2020/01/30 15:34:43 |       at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
INFO   | jvm 1    | 2020/01/30 15:34:43 |       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
INFO   | jvm 1    | 2020/01/30 15:34:43 |       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
INFO   | jvm 1    | 2020/01/30 15:34:43 |       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
INFO   | jvm 1    | 2020/01/30 15:34:43 |       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
INFO   | jvm 1    | 2020/01/30 15:34:43 |       at java.lang.Thread.run(Thread.java:748)
INFO   | jvm 1    | 2020/01/30 15:34:47 | W [d.M.B.TimeoutDaemon           ] [21:34:47]: ScheduledRequest[com.inductiveautomation.xopc.drivers.modbus2.requests.ReadHoldingRegistersRequest@c613cfb] request with key "25163" failed due to timeout.
INFO   | jvm 1    | 2020/01/30 15:34:47 | W [d.M.ReadHoldingRegistersRequest] [21:34:47]: Request failed. FailureType==TIMEOUT
INFO   | jvm 1    | 2020/01/30 15:34:47 | java.lang.Exception: Request failed by TimeoutDaemon due to timeout: ScheduledRequest[com.inductiveautomation.xopc.drivers.modbus2.requests.ReadHoldingRegistersRequest@c613cfb]
INFO   | jvm 1    | 2020/01/30 15:34:47 |       at com.inductiveautomation.xopc.driver.api.BasicRequestCycle$TimeoutDaemon.failRequests(BasicRequestCycle.java:374)
INFO   | jvm 1    | 2020/01/30 15:34:47 |       at com.inductiveautomation.xopc.driver.api.BasicRequestCycle$TimeoutDaemon.run(BasicRequestCycle.java:327)
INFO   | jvm 1    | 2020/01/30 15:34:47 |       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
INFO   | jvm 1    | 2020/01/30 15:34:47 |       at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
INFO   | jvm 1    | 2020/01/30 15:34:47 |       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
INFO   | jvm 1    | 2020/01/30 15:34:47 |       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
INFO   | jvm 1    | 2020/01/30 15:34:47 |       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
INFO   | jvm 1    | 2020/01/30 15:34:47 |       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
INFO   | jvm 1    | 2020/01/30 15:34:47 |       at java.lang.Thread.run(Thread.java:748)
INFO   | jvm 1    | 2020/01/30 15:34:51 | W [d.M.B.TimeoutDaemon           ] [21:34:51]: ScheduledRequest[com.inductiveautomation.xopc.drivers.modbus2.requests.ReadCoilsRequest@362a19c8] request with key "25164" failed due to timeout.
INFO   | jvm 1    | 2020/01/30 15:34:51 | W [d.M.ReadCoilsRequest          ] [21:34:51]: Request failed. FailureType==TIMEOUT
INFO   | jvm 1    | 2020/01/30 15:34:51 | java.lang.Exception: Request failed by TimeoutDaemon due to timeout: ScheduledRequest[com.inductiveautomation.xopc.drivers.modbus2.requests.ReadCoilsRequest@362a19c8]
INFO   | jvm 1    | 2020/01/30 15:34:51 |       at com.inductiveautomation.xopc.driver.api.BasicRequestCycle$TimeoutDaemon.failRequests(BasicRequestCycle.java:374)
INFO   | jvm 1    | 2020/01/30 15:34:51 |       at com.inductiveautomation.xopc.driver.api.BasicRequestCycle$TimeoutDaemon.run(BasicRequestCycle.java:327)
INFO   | jvm 1    | 2020/01/30 15:34:51 |       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
INFO   | jvm 1    | 2020/01/30 15:34:51 |       at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
INFO   | jvm 1    | 2020/01/30 15:34:51 |       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
INFO   | jvm 1    | 2020/01/30 15:34:51 |       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
INFO   | jvm 1    | 2020/01/30 15:34:51 |       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
INFO   | jvm 1    | 2020/01/30 15:34:51 |       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
INFO   | jvm 1    | 2020/01/30 15:34:51 |       at java.lang.Thread.run(Thread.java:748)
INFO   | jvm 1    | 2020/01/30 15:34:55 | W [d.M.B.TimeoutDaemon           ] [21:34:55]: ScheduledRequest[com.inductiveautomation.xopc.drivers.modbus2.requests.ReadHoldingRegistersRequest@c613cfb] request with key "25165" failed due to timeout.
INFO   | jvm 1    | 2020/01/30 15:34:55 | W [d.M.ReadHoldingRegistersRequest] [21:34:55]: Request failed. FailureType==TIMEOUT
INFO   | jvm 1    | 2020/01/30 15:34:55 | java.lang.Exception: Request failed by TimeoutDaemon due to timeout: ScheduledRequest[com.inductiveautomation.xopc.drivers.modbus2.requests.ReadHoldingRegistersRequest@c613cfb]
INFO   | jvm 1    | 2020/01/30 15:34:55 |       at com.inductiveautomation.xopc.driver.api.BasicRequestCycle$TimeoutDaemon.failRequests(BasicRequestCycle.java:374)
INFO   | jvm 1    | 2020/01/30 15:34:55 |       at com.inductiveautomation.xopc.driver.api.BasicRequestCycle$TimeoutDaemon.run(BasicRequestCycle.java:327)
INFO   | jvm 1    | 2020/01/30 15:34:55 |       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
INFO   | jvm 1    | 2020/01/30 15:34:55 |       at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
INFO   | jvm 1    | 2020/01/30 15:34:55 |       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
INFO   | jvm 1    | 2020/01/30 15:34:55 |       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
INFO   | jvm 1    | 2020/01/30 15:34:55 |       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
INFO   | jvm 1    | 2020/01/30 15:34:55 |       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
INFO   | jvm 1    | 2020/01/30 15:34:55 |       at java.lang.Thread.run(Thread.java:748)
INFO   | jvm 1    | 2020/01/30 15:34:56 | E [d.M.ReadHoldingRegistersRequest] [21:34:56]: Received response with ExceptionCode: 0x08 (MemoryParityError).
INFO   | jvm 1    | 2020/01/30 15:34:59 | W [d.M.B.TimeoutDaemon           ] [21:34:59]: ScheduledRequest[com.inductiveautomation.xopc.drivers.modbus2.requests.ReadCoilsRequest@362a19c8] request with key "25166" failed due to timeout.
INFO   | jvm 1    | 2020/01/30 15:34:59 | W [d.M.ReadCoilsRequest          ] [21:34:59]: Request failed. FailureType==TIMEOUT
INFO   | jvm 1    | 2020/01/30 15:34:59 | java.lang.Exception: Request failed by TimeoutDaemon due to timeout: ScheduledRequest[com.inductiveautomation.xopc.drivers.modbus2.requests.ReadCoilsRequest@362a19c8]
INFO   | jvm 1    | 2020/01/30 15:34:59 |       at com.inductiveautomation.xopc.driver.api.BasicRequestCycle$TimeoutDaemon.failRequests(BasicRequestCycle.java:374)
INFO   | jvm 1    | 2020/01/30 15:34:59 |       at com.inductiveautomation.xopc.driver.api.BasicRequestCycle$TimeoutDaemon.run(BasicRequestCycle.java:327)
INFO   | jvm 1    | 2020/01/30 15:34:59 |       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
INFO   | jvm 1    | 2020/01/30 15:34:59 |       at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
INFO   | jvm 1    | 2020/01/30 15:34:59 |       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
INFO   | jvm 1    | 2020/01/30 15:34:59 |       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
INFO   | jvm 1    | 2020/01/30 15:34:59 |       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
INFO   | jvm 1    | 2020/01/30 15:34:59 |       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
INFO   | jvm 1    | 2020/01/30 15:34:59 |       at java.lang.Thread.run(Thread.java:748)
INFO   | jvm 1    | 2020/01/30 15:35:02 | E [d.M.SocketIODelegate          ] [21:35:02]: Socket connection closed, DriverState was Connected.
INFO   | jvm 1    | 2020/01/30 15:35:02 | java.net.SocketException: Connection reset
INFO   | jvm 1    | 2020/01/30 15:35:02 |       at java.net.SocketInputStream.read(SocketInputStream.java:210)
INFO   | jvm 1    | 2020/01/30 15:35:02 |       at java.net.SocketInputStream.read(SocketInputStream.java:141)
INFO   | jvm 1    | 2020/01/30 15:35:02 |       at java.net.SocketInputStream.read(SocketInputStream.java:127)
INFO   | jvm 1    | 2020/01/30 15:35:02 |       at com.inductiveautomation.iosession.socket.AsyncSocketIOSession.run(AsyncSocketIOSession.java:71)
INFO   | jvm 1    | 2020/01/30 15:35:02 |       at java.lang.Thread.run(Thread.java:748)
INFO   | jvm 1    | 2020/01/30 15:35:02 | W [d.M.ReadHoldingRegistersRequest] [21:35:02]: Request failed. FailureType==DISCONNECTED
INFO   | jvm 1    | 2020/01/30 15:35:02 | java.lang.Exception: Driver is disconnected.
INFO   | jvm 1    | 2020/01/30 15:35:02 |       at com.inductiveautomation.xopc.driver.api.AbstractDriver$RequestSchedule.cancelCurrentSchedule(AbstractDriver.java:1377)
INFO   | jvm 1    | 2020/01/30 15:35:02 |       at com.inductiveautomation.xopc.driver.api.AbstractDriver.notifyConnectionLost(AbstractDriver.java:717)
INFO   | jvm 1    | 2020/01/30 15:35:02 |       at com.inductiveautomation.xopc.driver.api.AbstractIODelegatingDriver.notifyConnectionLost(AbstractIODelegatingDriver.java:64)
INFO   | jvm 1    | 2020/01/30 15:35:02 |       at com.inductiveautomation.xopc.driver.api.SocketIODelegate$DriverIOEventHandler.connectionLost(SocketIODelegate.java:196)
INFO   | jvm 1    | 2020/01/30 15:35:02 |       at com.inductiveautomation.iosession.socket.AsyncSocketIOSession.reportExceptionAndStopRunning(AsyncSocketIOSession.java:146)
INFO   | jvm 1    | 2020/01/30 15:35:02 |       at com.inductiveautomation.iosession.socket.AsyncSocketIOSession.run(AsyncSocketIOSession.java:87)
INFO   | jvm 1    | 2020/01/30 15:35:02 |       at java.lang.Thread.run(Thread.java:748)
INFO   | jvm 1    | 2020/01/30 15:35:02 | W [d.M.ReadCoilsRequest          ] [21:35:02]: Request failed. FailureType==DISCONNECTED
INFO   | jvm 1    | 2020/01/30 15:35:02 | java.lang.Exception: Driver is disconnected.
INFO   | jvm 1    | 2020/01/30 15:35:02 |       at com.inductiveautomation.xopc.driver.api.AbstractDriver$RequestSchedule.cancelCurrentSchedule(AbstractDriver.java:1377)
INFO   | jvm 1    | 2020/01/30 15:35:02 |       at com.inductiveautomation.xopc.driver.api.AbstractDriver.notifyConnectionLost(AbstractDriver.java:717)
INFO   | jvm 1    | 2020/01/30 15:35:02 |       at com.inductiveautomation.xopc.driver.api.AbstractIODelegatingDriver.notifyConnectionLost(AbstractIODelegatingDriver.java:64)
INFO   | jvm 1    | 2020/01/30 15:35:02 |       at com.inductiveautomation.xopc.driver.api.SocketIODelegate$DriverIOEventHandler.connectionLost(SocketIODelegate.java:196)
INFO   | jvm 1    | 2020/01/30 15:35:02 |       at com.inductiveautomation.iosession.socket.AsyncSocketIOSession.reportExceptionAndStopRunning(AsyncSocketIOSession.java:146)
INFO   | jvm 1    | 2020/01/30 15:35:02 |       at com.inductiveautomation.iosession.socket.AsyncSocketIOSession.run(AsyncSocketIOSession.java:87)
INFO   | jvm 1    | 2020/01/30 15:35:02 |       at java.lang.Thread.run(Thread.java:748)
INFO   | jvm 1    | 2020/01/30 15:35:02 | W [d.M.ReadHoldingRegistersRequest] [21:35:02]: Request failed. FailureType==DISCONNECTED
INFO   | jvm 1    | 2020/01/30 15:35:02 | java.lang.Exception: RequestCycle stopped.
INFO   | jvm 1    | 2020/01/30 15:35:02 |       at com.inductiveautomation.xopc.driver.api.BasicRequestCycle.shutdown(BasicRequestCycle.java:281)
INFO   | jvm 1    | 2020/01/30 15:35:02 |       at com.inductiveautomation.xopc.driver.api.AbstractDriver.createNewRequestCycle(AbstractDriver.java:736)
INFO   | jvm 1    | 2020/01/30 15:35:02 |       at com.inductiveautomation.xopc.driver.api.AbstractDriver.notifyConnectionLost(AbstractDriver.java:718)
INFO   | jvm 1    | 2020/01/30 15:35:02 |       at com.inductiveautomation.xopc.driver.api.AbstractIODelegatingDriver.notifyConnectionLost(AbstractIODelegatingDriver.java:64)
INFO   | jvm 1    | 2020/01/30 15:35:02 |       at com.inductiveautomation.xopc.driver.api.SocketIODelegate$DriverIOEventHandler.connectionLost(SocketIODelegate.java:196)
INFO   | jvm 1    | 2020/01/30 15:35:02 |       at com.inductiveautomation.iosession.socket.AsyncSocketIOSession.reportExceptionAndStopRunning(AsyncSocketIOSession.java:146)
INFO   | jvm 1    | 2020/01/30 15:35:02 |       at com.inductiveautomation.iosession.socket.AsyncSocketIOSession.run(AsyncSocketIOSession.java:87)
INFO   | jvm 1    | 2020/01/30 15:35:02 |       at java.lang.Thread.run(Thread.java:748)
INFO   | jvm 1    | 2020/01/30 15:35:02 | W [d.M.ReadCoilsRequest          ] [21:35:02]: Request failed. FailureType==DISCONNECTED
INFO   | jvm 1    | 2020/01/30 15:35:02 | java.lang.Exception: RequestCycle stopped.
INFO   | jvm 1    | 2020/01/30 15:35:02 |       at com.inductiveautomation.xopc.driver.api.BasicRequestCycle.run(BasicRequestCycle.java:152)
INFO   | jvm 1    | 2020/01/30 15:35:02 |       at java.lang.Thread.run(Thread.java:748)

Kevin.Herron · January 30, 2020, 9:51pm

I sent you a message with a dropbox link you can upload to.

Do you have your Modbus devices configured with “Reconnect After Consecutive Timeouts” setting enabled? Probably want to turn that off if you do.

Brian · January 30, 2020, 9:56pm

Uploaded the files. The timestamp in the wrapper.log excerpt that I posted corresponds to the capture file. Let me know if you need something more accurate for timing.

I shut that off yesterday when I was starting to troubleshoot this problem.

Kevin.Herron · January 30, 2020, 10:06pm

Well… sorry to string you along here, I was just really interested to see the packet capture for this.

Unfortunately all of what’s happening with these retransmissions is going on in the TCP stack of your OS and there’s not really anything I can do from up in Ignition.

Is this a VM or real hardware? Have you tried a different ethernet cable or using a different port on the switch?

edit: whatever is happening, while we do see a response for key 25141 as the last thing in Wireshark, it doesn’t actually get delivered to Ignition’s Modbus driver, which is where the timeouts begin.

edit2: I can see it’s VMware from the MAC addresses in the capture. Bleh.

Brian · January 30, 2020, 10:09pm

No problem, I’ll take whatever help I can get

This is a VM. I have about a dozen other devices connected over ethernet that aren’t showing any problems. I have another BRX that gets more traffic than this one. I can monitor that connection for a while, but I don’t see any visible problems on the displays.

I’m assuming that the ethernet cable going to that particular PLC is fine based on my laptop programming software connection.

Brian · January 30, 2020, 10:11pm

I’ll see what I can dig up on spurious retransmissions in this situation. At least you’re pointing me in the right direction, I appreciate it

chi · February 1, 2020, 6:53pm

@Kevin.Herron It this possibly a problem with the modbus driver not handling the exception code?

Kevin.Herron · February 1, 2020, 6:55pm

No, it does handle the exception code (by logging it). There’s something happening at the TCP layer here.

Wireshark shows the response being delivered (at some level), but then the TCP stack in the OS doesn’t think it arrived, and starts sending out the “spurious” retries. This is all way lower level than Ignition or Java.

chi · February 1, 2020, 7:05pm

@Brian Are you running tshark on the host or on the guest OS? If it is running on the host, i may have a similar problem. Modbus answer packets get lost on the way from the host to the guest VM.
In this case, tshark running on the guest VM would not show the last answer packet before the retransmits start.

Brian · February 1, 2020, 7:52pm

@chi Actually, the ReadHoldingRegistersRequest … ExceptionCode: 0x08 (MemoryParityError) line is unrelated to my timeout problem. I’ve had those error frequently on the network on other devices for years, but no big problems. I’d track the device down, but there’s not enough of an error message to figure out what is causing it.

I have an idea submission from a few years ago requesting better error info. Looks like something with the device name was possibly fixed (not sure if it’s in the version I’m running, I’m behind the times…) Unfortunately, a bunch of the likely suspects are behind a modbus gateway, so having the device name will only narrow it down to something like 11 out of 20something devices. Not overly helpful.

Brian · February 1, 2020, 7:54pm

I’m running tshark on the guest os. The host is ESXi 6.5 or 6.7, I don’t remember the version without checking.

The guest os is Ubuntu server (headless) 16.04.

Brian · February 1, 2020, 8:27pm

The problem is just plain weird. Yesterday, one of the things I did was to create a second device under ignition and read the same tag range as my main connection (two holding register tags and two coil tags to define the range that is read on the other device). I was polling that one every second instead of every 5 seconds. After I got that running, it was smooth sailing for 2 hours and 20 minutes. Immediately prior to making those changes, the main connection was dropping every 11ish minutes. It takes a while for those connections to come back too, sometimes multiple minutes. I think that’s the most painful part of this whole thing.

Anyway, I thought I’d disable the second connection and see what happened. As soon as I clicked save, the main connection dropped once again, and started dropping every 11ish minutes again too. So, I thought, I’ll just reenable the second connection for now to keep things happy. Well, now it’s dumping the connection somewhat randomly. Might be up for a couple hours, might be back to 11 minutes. Wouldn’t be so bad if it came back faster.

I’ll keep poking around to see if there’s a buffer size setting or some random thing that I’m starting to bump into on either ubuntu or esxi. Still not entirely ruling out a flaky switch between the server and the device, but I’m having difficulties accepting that as a possibility with a second good connection traveling the same path. Might try moving the device IP address to see if that does anything.