I have a problem with using MQTT between two Ignition gateways. The edge gateway is supposed to send tag data via MQTT to a central gateway, but we are having thread deadlocking issues at the central gateway level. No idea what’s causing it, but I’m assuming it’s coming from the MQTT Engine as it is receiving data. The error shows:
Deadlock Detected: Two or more threads are stuck waiting for each other’s locks.
I’m unsure if the tags that appear in the engine can/should be edited, and maybe editing caused the problem. It corrects itself with a gateway and computer restart, a gateway restart doesn’t seem to be enough to fix the problem, at least within 15 minutes. Any ideas?
Additionally, I was under the impression that only one MQTT Distributor was needed for this architecture on the central gateway (with MQTT Transmitter on the edge gateway, and MQTT Engine at the central gateway), but it refuses to work with only one Distributor, and two seem to be needed, even with changing the host IDs in MQTT Transmission / Engine settings. Are two actually needed?
Hey Roger, i’m also testing out MQTT and i also am getting that “Blocked” State, and eventual lock. In my test I have an edge server with only MQTT Transmission and its sending to the Ignition gateway which is running the MQTT engine and MQTT Distributor. It runs fine for a while then i see the values not updating and noticed the errors/lock. I should add that to have values for the MQTT transmission to send to the Engine/Distributor, I added the simulator device to the OPC-UA server on the edge, then created the folder structure needed for sparkplug model, then i created opctags.
As far as restarting the gateway. I noticed that when i stopped it through the utility, it keeps trying to shutdown the gateway. I looked at the Ignition Gateway in Services, and saw that it was still running. So i just shut it down from there and then started it back up through the utility.
We have very similar, if not the same, architectures! I’m also running a remote simulator via KepwareEX which is sending data through OPC UA to the edge server, so i’m getting data at that Ignition installation just fine, but while being sent via MQTT it just crashes the central/remote gateway. Editing tags on the central gateway, or having anything in expired trial mode seems to cause it for me, but it’s very disconcerting how fickle it can be. I got in touch with Cirrus Link about this and we’ll probably figure out what the issue is - i’ll post an update here if we find the problem.
Good to know that you can shut it off in Services - I didn’t consider that!
I spoke with Cirrus Link today and part of my issue was with my architecture being incorrect initially, but after correcting it, deleting an extraneous UDT in the central gateway’s MQTT Engine tag provider caused another thread deadlocking. I got the Ignition thread dump if anyone is interested in taking a look at it (I sent it to Cirrus Link as well for them to look at). thread_dump.txt (94.3 KB)
I have error logs as well, but the thread dump may be more interesting for everyone.
Hi guys, I don't really see a concrete solutions or a reason why the deadlock was caused.
We are getting the same error, which looks like it's being triggered by MQTT Transmission.
Did any of you ever find out why this occured in your cases? What are the general reasons for a deadlock? Would a deadlock occur just because you have to few server resources?
Deadlock is almost always a programming bug and needs to be resolved by upgrading after it has been fixed. If you're running into a deadlock then provide the thread dump and Ignition/module versions. If you're on older versions then expect to upgrade first.
Thanks for both your replies. @justinedwards.jle it is compatible but thanks for the chart.
@Kevin.Herron we are on 8.1.18, so I guess we could first do an upgrade and check if that works. If it doesn't, I guess we should log a support ticket to see what's going on.
