OPC UA Modbus TCP stuck on Connecting

I have a Modbus TCP connection to a device. This connection gets stuck on ‘Connecting’ when ever there is a power failure and power is restored.

I suspect it might be because the device that I am connecting to from Ignition, takes a little longer to start up, and by then the MBTCP connection goes into this stuck state.

What do I need to do to effectively troubleshoot this (logs?) or is there a setting to delay the connection to this device?

Currently, when the Modbus TCP connection is stuck in Connecting, I can edit and immediately resave this connection, and thereafter the status changes to connected.

I hope these power failures dont happen to often…

anyways you could try disabling and the re-enabling a connection with system.opc.setServerEnabled - Ignition User Manual 8.1 - Ignition Documentation
maybe that fixes the problem

Restarting the device would be more appropriate, using system.device.restart() or a pair of system.device.setDeviceEnabled(). (False, then True, for older versions that don't have the .restart() function.)

2 Likes

We had an identical bug a few days ago. For us the issue was that we had a redundant gateway in warm mode that was racing the primary gateway to establish a connection first. When one connected, the socket would be blocked and the other would be unable to connect. Changing it to cold standby fixed the problem.

My main concern is why it’s getting stuck in this Connecting state and how do I stop it from doing this.

Power loss is not often, but if it does occur, I would hope my devices would automatically connect to my Ignition Gateway without having to manually restart or reconnect any connections.

@BenKearney Noted. Interesting problem. My setup is simple - this is a single Edge Gateway connecting to one device.

Sorry to resurrect an old thread but I am having the exact same problem as imraan:

We have an Opto22 Groov RIO running Ignition Edge 8.1. The edge 8.1 Modbus driver connects to the opto22 modbus tcp server on 127.0.0.1 and after a power cycle the edge modbus driver gets stuck at “connecting” and simply going into the configuration, not changing anything, and then pressing save reconnects it.

Was wondering if anyone else had a solution to this problem? It’s getting annoying having to essentially manually reconnect each device after a power failure.

Thanks!

Can you take a thread dump while it’s stuck in “Connecting” after a reboot? Do you have a support ticket open already where you can upload this thread dump and the logs?

Is this something you can reproduce every time?

Hi Kevin, Thanks for getting back to me. I don’t have a support ticket open but I can open one if that makes it easier for you. We can probably identify an ideal time to power cycle the unit and grab a thread dump and logs too. I would need some instructions on how to capture the thread dump and logs though.

Thanks!

Both of those can be exported from the Status area of the Ignition Gateway.

If you call in and open a support ticket then if needed you can get a support engineer to walk you through it when you’re ready to power cycle, assuming it’s during business hours.

https://docs.inductiveautomation.com/display/DOC81/Diagnostics+-+Logs
https://docs.inductiveautomation.com/display/DOC81/Diagnostics+-+Threads

Ticket has been created - #43533

thread dump, screen shots and logs have also been updated.

Just confirmed again that when the modbus driver is stuck in “connecting” mode, going into the driver settings and clicking save (without actually changing anything) forces a restart of the modbus driver and then it connects fine.

I suppose i could write a script to periodically restart the modbus driver as suggested by pturmel but I think that’s a little bit hacky.

Thanks, we’ve had a couple cases recently where some variant of the Modbus driver exhibits the same behavior on startup, where it seems stuck “Connecting” until later edit/saved.

The only common thread so far is Ignition Edge on low-power or under-specced hardware. I’m not yet sure if it actually has anything to do with a Modbus driver or if that’s just a likely coincidence because it’s Edge.

interesting! I would definitely agree that ignition edge on the opto22 Groov RIO is maybe pushing the limits of the hardware. It takes 5-6 minutes for the ignition web interface to load after a power cycle. Probably good feedback for the opto22 team.

Worst case scenario we can disable edge on the device and connect from our main ignition gw which is running on a proper server computer.

Quick update for those who may run across this issue in the future:

I have been told the fix will be in the 8.1.17 release. The downside here is that ignition is not separately upgradable on the Opto22 Groov RIO devices - the only way it can be updated is with a firmware update and this process appears to be lagging due to the QA testing that OPTO22 on firmware releases. As I write this the current Opto22 firmware comes with 8.1.11 of ignition edge installed.

Using a gateway script to detect the module stuck in the connecting state and then forcing a restart is not possible on Ignition Edge unless you also have a compute module license which is an additional cost already above the $800 for the edge license.

It may be possible to install the modbus module from 8.1.17 onto 8.1.11 in an attempt to fix this issue but at this point I’m not willing to test it and possibly make things worse.

We’ll probably end up disabling Edge on the Opto22 Groov RIO device and just connect to it directly from our main ignition gateway using a modbus connection.

As it stands today, I cannot recommend using the Ignition Edge platform on the Groov RIO. I don’t think this is ready for a production environment.

Just want to clarify if anybody else comes across this particular issue - the fix is actually in the OPC UA module, not the Modbus module. I think the current nightly build of the OPC UA module requires Ignition 8.1.15, though, and in an upcoming nightly release the OPC UA module and all drivers will be synced/locked to 8.1.17.

It looks like I am dead in the water then - the Groov RIO module latest firmware is packaged with Ignition Edge 8.1.11 so it sounds like I can’t take the nightly build of the OPC UA module (with the associated bug fix) and install into 8.1.11 right?

No, I’m afraid that won’t work.

Have you talked with Opto22 about a timeline on a firmware upgrade or if it’s user-upgradable?

I’m pretty sure we have one of these in our office, and that there’s shell access to it, so that would make me think doing an upgrade yourself is possible…

If an industrial device that needs/requires firmware isn’t field upgradable then I would take that as a huge red flag.

In the last year I’ve had to deal with installed Emerson devices where we had to ask the question of “what firmware version are you on?” and get the local field engineer to play update bingo to see if various issues were solved by later versions of firmware.

Apparently there is a free license you can install to get shell access: https://developer.opto22.com/epicdev/SSH/

I’ve contacted opto22 again regarding firmware update timelines. I am not the biggest fan of upgrading via the shell as opto22 will not provide any support once you have this license unless you factory reset the unit.