MQTT Issues in Ignition

Hi all,

I have a problem with using MQTT between two Ignition gateways. The edge gateway is supposed to send tag data via MQTT to a central gateway, but we are having thread deadlocking issues at the central gateway level. No idea what’s causing it, but I’m assuming it’s coming from the MQTT Engine as it is receiving data. The error shows:

Deadlock Detected: Two or more threads are stuck waiting for each other’s locks.

And error log in file:

I’m unsure if the tags that appear in the engine can/should be edited, and maybe editing caused the problem. It corrects itself with a gateway and computer restart, a gateway restart doesn’t seem to be enough to fix the problem, at least within 15 minutes. Any ideas?

Additionally, I was under the impression that only one MQTT Distributor was needed for this architecture on the central gateway (with MQTT Transmitter on the edge gateway, and MQTT Engine at the central gateway), but it refuses to work with only one Distributor, and two seem to be needed, even with changing the host IDs in MQTT Transmission / Engine settings. Are two actually needed?

Hey Roger, i’m also testing out MQTT and i also am getting that “Blocked” State, and eventual lock. In my test I have an edge server with only MQTT Transmission and its sending to the Ignition gateway which is running the MQTT engine and MQTT Distributor. It runs fine for a while then i see the values not updating and noticed the errors/lock. I should add that to have values for the MQTT transmission to send to the Engine/Distributor, I added the simulator device to the OPC-UA server on the edge, then created the folder structure needed for sparkplug model, then i created opctags.

As far as restarting the gateway. I noticed that when i stopped it through the utility, it keeps trying to shutdown the gateway. I looked at the Ignition Gateway in Services, and saw that it was still running. So i just shut it down from there and then started it back up through the utility.

1 Like

Hi,

We have very similar, if not the same, architectures! I’m also running a remote simulator via KepwareEX which is sending data through OPC UA to the edge server, so i’m getting data at that Ignition installation just fine, but while being sent via MQTT it just crashes the central/remote gateway. Editing tags on the central gateway, or having anything in expired trial mode seems to cause it for me, but it’s very disconcerting how fickle it can be. I got in touch with Cirrus Link about this and we’ll probably figure out what the issue is - i’ll post an update here if we find the problem.

Good to know that you can shut it off in Services - I didn’t consider that!

Can one of you guys get a thread dump while this is happening? I’m sure Cirrus Link will ask for one as well.

Hi Kevin,

I will definitely do that if I can replicate the issue!

Do let me know what they come up with. I’ve contacted support also, but haven’t heard a response back yet.

Hi everyone,

I spoke with Cirrus Link today and part of my issue was with my architecture being incorrect initially, but after correcting it, deleting an extraneous UDT in the central gateway’s MQTT Engine tag provider caused another thread deadlocking. I got the Ignition thread dump if anyone is interested in taking a look at it (I sent it to Cirrus Link as well for them to look at).
thread_dump.txt (94.3 KB)

I have error logs as well, but the thread dump may be more interesting for everyone.

Best,
Roger

A post was split to a new topic: Is there a way to read MQTT JSON payload into Ignition tags without cirrus link driver?

Hi guys, I don't really see a concrete solutions or a reason why the deadlock was caused.
We are getting the same error, which looks like it's being triggered by MQTT Transmission.

Did any of you ever find out why this occured in your cases? What are the general reasons for a deadlock? Would a deadlock occur just because you have to few server resources?

Deadlock is almost always a programming bug and needs to be resolved by upgrading after it has been fixed. If you're running into a deadlock then provide the thread dump and Ignition/module versions. If you're on older versions then expect to upgrade first.

Here is the Cirrus Link/Ignition version compatibility chart.

Thanks for both your replies. @justinedwards.jle it is compatible but thanks for the chart.

@Kevin.Herron we are on 8.1.18, so I guess we could first do an upgrade and check if that works. If it doesn't, I guess we should log a support ticket to see what's going on.

1 Like

Also note that Cirrus Link has their own forum and support infrastructure. That's where any support ticket should start.