Continuous desconnections in Modbus RTU devices

rcortes · October 23, 2020, 3:16pm

Hi guys,

I’m in the middle of a migration from Matrikon OPC to the Ignition OPC built-in the Gateway and I’m having connectivity problems with some of the devices.

All the devices uses Modbus RTU to communicate with the master, and a media converter (Moxa NPort) to convert the RS-485 to Ethernet. This is how Matrikon has been working since the beginning, and how I have to implement it on Ignition.

I’ve already migrate several devices, but some of them has been a real challenge. One of the devices, an Ingeteam Inverter, has been real difficult, showing me several time out and driver setting to stale in the console.

In the following console logs, the [CT1_MBGW1a] is the Moxa converter used to communicate with the Inverser via Modbus RTU

(D)	11:20:38	DriverVariableNode[CT1_MBGW1a]	Setting [CT1_MBGW1a]1.IR15@4500ms to stale. Last update was 172935ms ago. Max age is 90000.0ms.
(D)	11:20:38	DriverVariableNode[CT1_MBGW1a]	Setting [CT1_MBGW1a]1.IR14@4500ms to stale. Last update was 172935ms ago. Max age is 90000.0ms.
(D)	11:20:38	DriverVariableNode[CT1_MBGW1a]	Setting [CT1_MBGW1a]1.IR16@4500ms to stale. Last update was 172935ms ago. Max age is 90000.0ms.
(D)	11:20:38	DriverVariableNode[CT1_MBGW1a]	Setting [CT1_MBGW1a]1.IRI1@4500ms to stale. Last update was 174304ms ago. Max age is 90000.0ms.
(D)	11:20:38	drivers.ModbusOverTCPDriver[CT1_MBGW1a].RequestQueue	Low count: 5
(D)	11:20:38	drivers.ModbusOverTCPDriver[CT1_MBGW1a].RequestQueue	High count: 0
(D)	11:20:38	drivers.ModbusOverTCPDriver[CT1_MBGW1a].RequestQueue	Critical count: 0

WARN	11:21:52	drivers.ModbusOverTCPDriver[CT1_MBGW1a].BasicRequestCycle.TimeoutDaemon	ScheduledRequest[com.inductiveautomation.xopc.drivers.modbus2.requests.ReadInputRegistersRequest@c3b4c09] request with key "java.lang.Object@306b4275" failed due to timeout.
(D)	11:21:52	drivers.ModbusOverTCPDriver[CT1_MBGW1a].RequestSchedule	Canceling 7 ScheduledRequests.
(D)	11:21:52	drivers.ModbusOverTCPDriver[CT1_MBGW1a]	Disconnecting -> Disconnected
(D)	11:21:52	drivers.ModbusOverTCPDriver[CT1_MBGW1a]	Connected -> Disconnecting
(D)	11:21:52	drivers.ModbusOverTCPDriver[CT1_MBGW1a]	Reconnecting...
WARN	11:21:52	drivers.ModbusOverTCPDriver[CT1_MBGW1a].ReadInputRegistersRequest	Request failed. FailureType==TIMEOUT
 	
java.lang.Exception: Request failed by TimeoutDaemon due to timeout: ScheduledRequest[com.inductiveautomation.xopc.drivers.modbus2.requests.ReadInputRegistersRequest@c3b4c09]
at com.inductiveautomation.xopc.driver.api.BasicRequestCycle$TimeoutDaemon.failRequests(BasicRequestCycle.java:352)
at com.inductiveautomation.xopc.driver.api.BasicRequestCycle$TimeoutDaemon.run(BasicRequestCycle.java:305)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)


WARN	11:21:52	drivers.ModbusOverTCPDriver[CT1_MBGW1a].ReadHoldingRegistersRequest	Request failed. FailureType==DISCONNECTED
 	
java.lang.Exception: RequestCycle stopped.
at com.inductiveautomation.xopc.driver.api.BasicRequestCycle.run(BasicRequestCycle.java:135)
at java.lang.Thread.run(Unknown Source)
(D)	11:21:52	drivers.ModbusOverTCPDriver[CT1_MBGW1a]	Connecting -> Connected
(D)	11:21:52	drivers.ModbusOverTCPDriver[CT1_MBGW1a]	Releasing connect permit.
(D)	11:21:52	drivers.ModbusOverTCPDriver[CT1_MBGW1a].SocketIODelegate	[hostname=192.168.2.1,port=40001] TCP connection successfully opened.
(D)	11:21:52	drivers.ModbusOverTCPDriver[CT1_MBGW1a].SocketIODelegate	[hostname=192.168.2.1,port=40001] Opening TCP connection...
(D)	11:21:52	drivers.ModbusOverTCPDriver[CT1_MBGW1a]	Disconnected -> Connecting
(D)	11:21:52	drivers.ModbusOverTCPDriver[CT1_MBGW1a].BasicRequestCycle	Shutdown complete.
WARN	11:21:52	drivers.ModbusOverTCPDriver[CT1_MBGW1a].ReadInputRegistersRequest	Request failed. FailureType==DISCONNECTED
 	
java.lang.Exception: RequestCycle stopped.
at com.inductiveautomation.xopc.driver.api.BasicRequestCycle.shutdown(BasicRequestCycle.java:263)
at com.inductiveautomation.xopc.driver.api.AbstractDriver.createNewRequestCycle(AbstractDriver.java:652)
at com.inductiveautomation.xopc.driver.api.AbstractDriver.notifyDisconnectDone(AbstractDriver.java:618)
at com.inductiveautomation.xopc.driver.api.AbstractIODelegatingDriver.notifyDisconnectDone(AbstractIODelegatingDriver.java:71)
at com.inductiveautomation.xopc.driver.api.AbstractIODelegatingDriver.disconnect(AbstractIODelegatingDriver.java:43)
at com.inductiveautomation.xopc.driver.api.AbstractIODelegatingDriver.reconnect(AbstractIODelegatingDriver.java:93)
at com.inductiveautomation.xopc.drivers.modbus2.AbstractModbusDriver.access$100(AbstractModbusDriver.java:89)
at com.inductiveautomation.xopc.drivers.modbus2.AbstractModbusDriver$2.notifyCommunicationTimeout(AbstractModbusDriver.java:150)
at com.inductiveautomation.xopc.drivers.modbus2.requests.AbstractModbusRequest.fail(AbstractModbusRequest.java:173)
at com.inductiveautomation.xopc.drivers.modbus2.requests.AbstractModbusReadRequest.fail(AbstractModbusReadRequest.java:101)
at com.inductiveautomation.xopc.driver.api.ScheduledRequest.fail(ScheduledRequest.java:113)
at com.inductiveautomation.xopc.driver.api.BasicRequestCycle$TimeoutDaemon.failRequests(BasicRequestCycle.java:349)
at com.inductiveautomation.xopc.driver.api.BasicRequestCycle$TimeoutDaemon.run(BasicRequestCycle.java:305)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

I’ve been trying modifying the device parameters in the Ignition Gateway, the OPC Server parameters, the scan clases, but nothing seems to fix it. Even I install the last minor update (7.7.7 to 7.7.10) but it’s the same. But in Matrikon OPC works perfectly.

I hope you could help me. Thanks

pturmel · October 23, 2020, 3:36pm

Are you sure you are turning off access from Matrikon when turning on access from Ignition? RTU can only have one master.

rcortes · October 23, 2020, 3:44pm

Yes, I have disabled the tag in ignition that reads the Matrikon, and the object in Matrikon to the specific modbus device and the whole moxa gateway.

Funny thing is that actually the Modbus RTU device has the Matrikon and another controller as Master simultaneously and works with no problems

jay · October 23, 2020, 5:38pm

The Moxa NPort is a serial device server, not a gateway like the Moxa MB series. I am surprised you can make reliable concurrent connections that way. Have you tried making Ignition the only master that is active?

rcortes · October 23, 2020, 5:59pm

Is not that simple, since the other master is the process controller. I know Ignition should be polling data from the controller, but this is how was done initially, and if it works on Matrikon, why shouldn’t work on Ignition?

pturmel · October 23, 2020, 6:06pm

Is the matrikon also going through this Moxa Nport? If it is using a real serial port, it might be listening for silence to stick its requests in between the controller’s requests. This is very much a spec violation, so you shouldn’t expect it to work in general.

rcortes · October 23, 2020, 7:44pm

The whole architecture remain the same, the only change is the data being polling from Ignition istead from Matrikon.

Also, if I try polling data with Modbus Poll, it read the device without any problems, so Ignition can’t handle Modbus RTU?

allnet · October 23, 2020, 7:59pm

Actually could be the device unable to handle higher polling rate. Check that.
And you say Modbus RTU. Try with both ModbusRTU over TCP and Modbus TCP.
Is Span Gaps unchecked?

rcortes · October 23, 2020, 8:22pm

But the polling rate shouldn’t be defined by the scan classe of the tag? In that case, I have a Direct Read scan classe with a slow rate of 10000 ms (The stale timeout is 20s, and the timeout of the device in the Gateway is 30s). And yes, the span gaps is unchecked and the Max Registers Per Request is big enough to read all the data.

I didn’t try the Modbus TCP device because I’m using a Moxa NPort only to convert the serial interface to an Ethernet one, then should be a Modbus RTU over TCP device. Anyway, I tried the Modbus TCP and don’t even connect to the device.

allnet · October 23, 2020, 8:33pm

10s should be way more than enough.
I have seen this behavior when polling rate + length of data bigger than how fast the device can answer.
Or when trying to read a non-existing address. Is zero based enabled?
Try with only 1 tag, some higher address you are sure it exists.

rcortes · October 23, 2020, 8:58pm

I’m trying with another device, but the same architecture, through a Moxa Nport. I only have enable one tag and reads a value once every idk 30 seconds or more. I also tried the zero based config, but in both cases the address 1 should exist. Matrikon is configured to poll the slave ID 1.

allnet · October 23, 2020, 9:16pm

Hm. I see in OPC item path []11.HR1. If it is address one maybe should be 1.HR1

rcortes · October 23, 2020, 9:19pm

Sorry, may bad. The slave ID is 11, memory address 1 (1 to 20 actually, but in the screencap I’m only reading the first one)

allnet · October 23, 2020, 9:36pm

Dunno then. Delete all tags, create mapping for 2 holding registers, let’s say 2 and 3 and check with quick client.

cajaz · November 2, 2020, 5:14pm

rcortes, I had some issues when connecting to our generator with a moxa mb3180. I saw in the documentation that there was a maximum number of addresses that could be polled at a given time, so as soon as I added over x number of tags I’d lose all tags. What I had to do was to break up different tags into different scan classes so that I didn’t overwhelm the generator CPU. Not sure if it’s related to your issue, but hopefully it might help. Good Luck.

rcortes · November 2, 2020, 10:33pm

Maybe there is the solution. I remember when I started creating the data types, I tried with a few tag for testing and I could read with no problems. I’ll try that way