MQTT Engine Connect/Disconnect Loop

Ignition
1

Hi all,

We recently started facing an issue on 8.3.6 where our MQTT Engine (CirrusLink 5.0.3) was stuck in a connect/disconnect loop. Disabling and Enabling the engine only restored connection at the very beginning of the issue for a few minutes. Now it doesn't restore it at all.

We also upgraded to 8.3.8 and CirrusLink 5.0.4 to no avail. Here are the key logs:

image
image1619×479 56.1 KB

Any one have any ideas?

Edit: I'll add we started facing this on 7/26 and are continuing to see it at the time of writing

Edit 2: We're seeing this on 2/8 servers, and these happen to be our only two servers on 8.3

2

Check for oddball problems like server disk space.

3

All good in that department. I edited my post to add that we're seeing it on two of our servers that are our only servers on 8.3

4

I haven't seen a report like yours, so it is probably time to open a support ticket with Cirrus Link.

5

Yeah, I've sent this issue to our internal enterprise team that handles that comm. stream, wanted to see if there was anything to try in the meantime

6

Might try a packet capture to see what's going on too. It's saying it's having trouble publishing the host state so it almost seems like a permissions problem on the broker, but a packet capture may reveal more.

7

Worth ruling out while you wait on the ticket: check the client ID on both 8.3 gateways.

Same client ID on two clients gives you exactly this loop. Broker accepts one, kicks the other off, that one reconnects and kicks the first off. Never settles. Fits that it's only the two 8.3 boxes and none of the other six, especially if they came off the same gateway backup. Change it on one and see if both go quiet.

On the host state message, check the broker ACL for retained publishes specifically. STATE goes out retained and a lot of default ACLs allow spBv1.0/# but not retained. Broker log usually shows the reject even when the Engine side just says disconnected.