I'm running a Gateway on version 8.1.44, connecting to an Automation Direct BRX DM1E-36 series PLC via Modbus TCP. I have ~1500 tags configured on 1000 and 5000 ms poll rates. The BRX is running firmware version 2.9.9.
The modbus server on the BRX is configured for a timeout of 60s with 8 allowed sessions. Ignition is configured for 2500ms timeout. The BRX average scan time is ~12ms with Ignition side response times of ~60ms.
Fairly consistently, Ignition's connection to the device will time out. Some times it takes 2, 3, 5, 6, or 8 hours for this to occur. I was able to get a wireshark capture from the vm during one of the time outs.
We have a second BRX D1ME-36 device that experiences similar dropouts but on a much longer timeline (12+hours or even days) that is running an almost identical setup but with only 15 tags on a 1000 ms poll rate.
Capture Overview Snippet
21 0.631707 IGN.addr BRX.addr Modbus/TCP 66 Query: Trans: 13865; Unit: 0, Func: 4: Read Input Registers
22 0.642210 BRX.addr IGN.addr TCP 60 502 → 65343 [ACK] Seq=1345 Ack=73 Win=8180 Len=0
23 0.692383 BRX.addr IGN.addr Modbus/TCP 311 Response: Trans: 13865; Unit: 0, Func: 4: Read Input Registers
24 0.692547 IGN.addr BRX.addr Modbus/TCP 66 Query: Trans: 13866; Unit: 0, Func: 2: Read Discrete Inputs
25 0.703593 BRX.addr IGN.addr TCP 60 502 → 65343 [ACK] Seq=1602 Ack=85 Win=8180 Len=0
26 0.750362 BRX.addr IGN.addr Modbus/TCP 91 Response: Trans: 13866; Unit: 0, Func: 2: Read Discrete Inputs
27 0.750549 IGN.addr BRX.addr Modbus/TCP 66 Query: Trans: 13867; Unit: 0, Func: 3: Read Holding Registers
28 0.762763 BRX.addr IGN.addr TCP 60 502 → 65343 [ACK] Seq=1639 Ack=97 Win=8180 Len=0
29 0.811725 BRX.addr IGN.addr Modbus/TCP 155 Response: Trans: 13867; Unit: 0, Func: 3: Read Holding Registers
30 0.811832 IGN.addr BRX.addr Modbus/TCP 66 Query: Trans: 13868; Unit: 0, Func: 3: Read Holding Registers
31 0.822298 BRX.addr IGN.addr TCP 60 502 → 65343 [ACK] Seq=1740 Ack=109 Win=8180 Len=0
32 4.292852 IGN.addr BRX.addr Modbus/TCP 66 Query: Trans: 13869; Unit: 0, Func: 1: Read Coils
33 4.298118 BRX.addr IGN.addr TCP 60 [TCP Previous segment not captured] 502 → 65343 [ACK] Seq=1997 Ack=121 Win=8180 Len=0
34 7.296778 IGN.addr BRX.addr Modbus/TCP 66 Query: Trans: 13870; Unit: 0, Func: 3: Read Holding Registers
35 7.301364 BRX.addr IGN.addr TCP 60 502 → 65343 [ACK] Seq=1997 Ack=133 Win=8168 Len=0
36 10.300639 IGN.addr BRX.addr Modbus/TCP 66 Query: Trans: 13871; Unit: 0, Func: 4: Read Input Registers
37 10.306736 BRX.addr IGN.addr TCP 60 502 → 65343 [ACK] Seq=1997 Ack=145 Win=8156 Len=0
38 13.305799 IGN.addr BRX.addr TCP 54 65343 → 502 [FIN, ACK] Seq=145 Ack=1740 Win=63020 Len=0
39 13.307861 IGN.addr BRX.addr TCP 66 58309 → 502 [SYN, ECE, CWR] Seq=0 Win=64240 Len=0 MSS=1460 WS=256 SACK_PERM
40 13.308506 BRX.addr IGN.addr TCP 60 502 → 65343 [ACK] Seq=1997 Ack=146 Win=8156 Len=0
41 13.309119 BRX.addr IGN.addr TCP 60 502 → 58309 [SYN, ACK] Seq=0 Ack=1 Win=8192 Len=0 MSS=1458
42 13.309145 IGN.addr BRX.addr TCP 54 58309 → 502 [ACK] Seq=1 Ack=1 Win=64240 Len=0
43 15.572253 IGN.addr BRX.addr Modbus/TCP 66 Query: Trans: 13872; Unit: 0, Func: 4: Read Input Registers
44 15.574175 BRX.addr IGN.addr TCP 60 502 → 58309 [ACK] Seq=1 Ack=13 Win=8180 Len=0
45 15.610644 BRX.addr IGN.addr Modbus/TCP 311 Response: Trans: 13872; Unit: 0, Func: 4: Read Input Registers
46 15.610839 IGN.addr BRX.addr Modbus/TCP 66 Query: Trans: 13873; Unit: 0, Func: 3: Read Holding Registers
47 15.621816 BRX.addr IGN.addr TCP 60 502 → 58309 [ACK] Seq=258 Ack=25 Win=8180 Len=0
48 15.669334 BRX.addr IGN.addr Modbus/TCP 243 Response: Trans: 13873; Unit: 0, Func: 3: Read Holding Registers
49 15.669499 IGN.addr BRX.addr Modbus/TCP 66 Query: Trans: 13874; Unit: 0, Func: 3: Read Holding Registers
50 15.680483 BRX.addr IGN.addr TCP 60 502 → 58309 [ACK] Seq=447 Ack=37 Win=8180 Len=0
51 15.728451 BRX.addr IGN.addr Modbus/TCP 311 Response: Trans: 13874; Unit: 0, Func: 3: Read Holding Registers
52 15.728629 IGN.addr BRX.addr Modbus/TCP 66 Query: Trans: 13875; Unit: 0, Func: 3: Read Holding Registers
53 15.739494 BRX.addr IGN.addr TCP 60 502 → 58309 [ACK] Seq=704 Ack=49 Win=8180 Len=0
Wrapper.log snippet
INFO | jvm 2 | 2025/05/01 15:35:27 | W [d.M.B.B.TimeoutDaemon ] [15:35:27.244]: ScheduledRequest[com.inductiveautomation.xopc.drivers.modbus2.requests.ReadHoldingRegistersRequest@5192a815] request with key "13868" failed due to timeout. device-name= BRX
INFO | jvm 2 | 2025/05/01 15:35:27 | W [d.M.ReadHoldingRegistersRequest] [15:35:27.244]: Request failed. FailureType==TIMEOUT device-name= BRX
INFO | jvm 2 | 2025/05/01 15:35:27 | java.lang.Exception: Request failed by TimeoutDaemon due to timeout: ScheduledRequest[com.inductiveautomation.xopc.drivers.modbus2.requests.ReadHoldingRegistersRequest@5192a815]
INFO | jvm 2 | 2025/05/01 15:35:27 | at com.inductiveautomation.xopc.driver.api.BasicRequestCycle$TimeoutDaemon.failRequests(BasicRequestCycle.java:376)
INFO | jvm 2 | 2025/05/01 15:35:27 | at com.inductiveautomation.xopc.driver.api.BasicRequestCycle$TimeoutDaemon.run(BasicRequestCycle.java:328)
INFO | jvm 2 | 2025/05/01 15:35:27 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
INFO | jvm 2 | 2025/05/01 15:35:27 | at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source)
INFO | jvm 2 | 2025/05/01 15:35:27 | at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
INFO | jvm 2 | 2025/05/01 15:35:27 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
INFO | jvm 2 | 2025/05/01 15:35:27 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
INFO | jvm 2 | 2025/05/01 15:35:27 | at java.base/java.lang.Thread.run(Unknown Source)
DEBUG | wrapperp | 2025/05/01 15:35:29 | Send a packet PING : ping 01db5781
INFO | jvm 2 | 2025/05/01 15:35:29 | WrapperManager Debug: Received a packet PING : ping 01db5781
INFO | jvm 2 | 2025/05/01 15:35:29 | WrapperManager Debug: Send a packet PING : ping 01db5781
DEBUG | wrapperp | 2025/05/01 15:35:29 | read a packet PING : ping 01db5781
INFO | jvm 2 | 2025/05/01 15:35:30 | W [d.M.B.B.TimeoutDaemon ] [15:35:30.248]: ScheduledRequest[com.inductiveautomation.xopc.drivers.modbus2.requests.ReadCoilsRequest@2b22cca5] request with key "13869" failed due to timeout. device-name= BRX
INFO | jvm 2 | 2025/05/01 15:35:30 | W [d.M.ReadCoilsRequest ] [15:35:30.248]: Request failed. FailureType==TIMEOUT device-name= BRX
INFO | jvm 2 | 2025/05/01 15:35:30 | java.lang.Exception: Request failed by TimeoutDaemon due to timeout: ScheduledRequest[com.inductiveautomation.xopc.drivers.modbus2.requests.ReadCoilsRequest@2b22cca5]
INFO | jvm 2 | 2025/05/01 15:35:30 | at com.inductiveautomation.xopc.driver.api.BasicRequestCycle$TimeoutDaemon.failRequests(BasicRequestCycle.java:376)
INFO | jvm 2 | 2025/05/01 15:35:30 | at com.inductiveautomation.xopc.driver.api.BasicRequestCycle$TimeoutDaemon.run(BasicRequestCycle.java:328)
INFO | jvm 2 | 2025/05/01 15:35:30 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
INFO | jvm 2 | 2025/05/01 15:35:30 | at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source)
INFO | jvm 2 | 2025/05/01 15:35:30 | at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
INFO | jvm 2 | 2025/05/01 15:35:30 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
INFO | jvm 2 | 2025/05/01 15:35:30 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
INFO | jvm 2 | 2025/05/01 15:35:30 | at java.base/java.lang.Thread.run(Unknown Source)
DEBUG | wrapper | 2025/05/01 15:35:32 | SERVICE_CONTROL_INTERROGATE
DEBUG | wrapper | 2025/05/01 15:35:32 | SERVICE_CONTROL_INTERROGATE
DEBUG | wrapperp | 2025/05/01 15:35:32 | Send a packet SERVICE_CONTROL_CODE : 4
DEBUG | wrapperp | 2025/05/01 15:35:32 | Send a packet SERVICE_CONTROL_CODE : 4
INFO | jvm 2 | 2025/05/01 15:35:32 | WrapperManager Debug: Received a packet SERVICE_CONTROL_CODE : 4
INFO | jvm 2 | 2025/05/01 15:35:32 | WrapperManager Debug: ServiceControlCode from Wrapper with code 4
INFO | jvm 2 | 2025/05/01 15:35:32 | WrapperManager Debug: Received a packet SERVICE_CONTROL_CODE : 4
INFO | jvm 2 | 2025/05/01 15:35:32 | WrapperManager Debug: ServiceControlCode from Wrapper with code 4
INFO | jvm 2 | 2025/05/01 15:35:33 | W [d.M.B.B.TimeoutDaemon ] [15:35:33.252]: ScheduledRequest[com.inductiveautomation.xopc.drivers.modbus2.requests.ReadHoldingRegistersRequest@25e21faf] request with key "13870" failed due to timeout. device-name= BRX
INFO | jvm 2 | 2025/05/01 15:35:33 | W [d.M.ReadHoldingRegistersRequest] [15:35:33.252]: Request failed. FailureType==TIMEOUT device-name= BRX
INFO | jvm 2 | 2025/05/01 15:35:33 | java.lang.Exception: Request failed by TimeoutDaemon due to timeout: ScheduledRequest[com.inductiveautomation.xopc.drivers.modbus2.requests.ReadHoldingRegistersRequest@25e21faf]
INFO | jvm 2 | 2025/05/01 15:35:33 | at com.inductiveautomation.xopc.driver.api.BasicRequestCycle$TimeoutDaemon.failRequests(BasicRequestCycle.java:376)
INFO | jvm 2 | 2025/05/01 15:35:33 | at com.inductiveautomation.xopc.driver.api.BasicRequestCycle$TimeoutDaemon.run(BasicRequestCycle.java:328)
INFO | jvm 2 | 2025/05/01 15:35:33 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
INFO | jvm 2 | 2025/05/01 15:35:33 | at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source)
INFO | jvm 2 | 2025/05/01 15:35:33 | at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
INFO | jvm 2 | 2025/05/01 15:35:33 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
INFO | jvm 2 | 2025/05/01 15:35:33 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
INFO | jvm 2 | 2025/05/01 15:35:33 | at java.base/java.lang.Thread.run(Unknown Source)
DEBUG | wrapperp | 2025/05/01 15:35:34 | Send a packet PING : ping 01db57b3
INFO | jvm 2 | 2025/05/01 15:35:34 | WrapperManager Debug: Received a packet PING : ping 01db57b3
INFO | jvm 2 | 2025/05/01 15:35:34 | WrapperManager Debug: Send a packet PING : ping 01db57b3
DEBUG | wrapperp | 2025/05/01 15:35:34 | read a packet PING : ping 01db57b3
INFO | jvm 2 | 2025/05/01 15:35:36 | W [d.M.B.B.TimeoutDaemon ] [15:35:36.257]: ScheduledRequest[com.inductiveautomation.xopc.drivers.modbus2.requests.ReadInputRegistersRequest@29b6f4c1] request with key "13871" failed due to timeout. device-name= BRX
INFO | jvm 2 | 2025/05/01 15:35:36 | W [d.M.ReadInputRegistersRequest ] [15:35:36.257]: Request failed. FailureType==TIMEOUT device-name= BRX
INFO | jvm 2 | 2025/05/01 15:35:36 | java.lang.Exception: Request failed by TimeoutDaemon due to timeout: ScheduledRequest[com.inductiveautomation.xopc.drivers.modbus2.requests.ReadInputRegistersRequest@29b6f4c1]
INFO | jvm 2 | 2025/05/01 15:35:36 | at com.inductiveautomation.xopc.driver.api.BasicRequestCycle$TimeoutDaemon.failRequests(BasicRequestCycle.java:376)
INFO | jvm 2 | 2025/05/01 15:35:36 | at com.inductiveautomation.xopc.driver.api.BasicRequestCycle$TimeoutDaemon.run(BasicRequestCycle.java:328)
INFO | jvm 2 | 2025/05/01 15:35:36 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
INFO | jvm 2 | 2025/05/01 15:35:36 | at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source)
INFO | jvm 2 | 2025/05/01 15:35:36 | at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
INFO | jvm 2 | 2025/05/01 15:35:36 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
INFO | jvm 2 | 2025/05/01 15:35:36 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
INFO | jvm 2 | 2025/05/01 15:35:36 | at java.base/java.lang.Thread.run(Unknown Source)
What is strange is that for the requests that time out, the BRX sends a TCP ACK to the modbus poll request as normal, but then never seems to send the Modbus TCP response after the ack, compared to the normal Modbus Query -> TCP Ack-> Modbus Response format that I see during the normal communications. (Which you can also see leading up to the disconnect in the snippet)
Following the connection reset sent from Ignition the device begins transmitting normally.
From what I can tell this doesn't appear to be an Ignition specifc issue, its either the BRX side or something in between, but I don't know what to look at/for at this time. The BRX logs don't show any errors or warnings regarding the modbus server. Any pointers would be appreciated.