Hello folks,
I just opened the ‘Queue Management’ tab on the ‘Gateway Network’ page, and immediately started scratching my head. Here’s why:
1938 queues ! And they’re all there, it’s not just a name !
I checked 4 projects that are configured in the same way: a remote tag provider on a field gateway, and a front on a server somewhere. For all of them, the same thing: no extra queues on the front gateway, and a varying amount (from 194 to 1938) on the field gateway.
Can someone tell me what’s going on ? Where are they coming from, how can I find what’s generating them ?
In the 10 minutes since I posted this, one new queue was added to at least 2 out of 4 projects.
I now have 1939 queues instead of 1938 for the biggest offender.
Just to try and help narrow the scope a little, do you have any third-party modules installed? Also, what version?
Hello Kevin,
On the field gateways: 8.1.3 and 8.1.4
On the front gateway (no queue multiplication there): 8.10
modules:
From what I can see in the code, every queue creation is correctly accompanied by a name, which results in the other “normal-looking” queues you typically see. The _queue_<incrementing id> is a fallback name when no name property is supplied on a creation request. You might try turning on DEBUG against metro.queues.queue logger to see if you can get some insight on what types of messages are being put into those.
Thanks,
I can’t tell if there’s anything interesting in the logs…
They look like this:
edit: I thought this might be interesting:
In the Outgoing Queues:
In the Outgoing Task Statistics:
It looks like these queues are being created to facilitate remote alarm provider subscriptions. I was able to recreate this scenario locally between a 8.1.3 and 8.1.10 pair of gateways (with remote tag provider configured on 8.1.10 side against 8.1.3 source provider and Subscribed mode for alarming).
Tracing this down further, it looks like this behavior was addressed in 8.1.10. Confirmed by upgrading my test bench:
It looks like as of 8.1.10+, all Alarm Event Publishing is done through this named queue now.
2 Likes
Alright, thanks a lot !
Do you think this has any impact on anything besides the gigantic listing of queues on the Queue Management tab ?
I guess it’s time to upgrade the field gateways…
Are any of these other queues processing messages? (i.e. is their total count increasing). I imagine that something is just not deregistering/destroying those old queues when the subscription is recreated–not sure at the moment why that is happening. There might be some memory impact for those lingering queues. There look to be some good performance improvements in the way the queue is now implemented that should help for really busy alarming scenarios.
The one in the outgoing task statistics appears to be processing things:
It’s queue_1, its total did increase (getting close to 1,000,000 !), and I have no idea what it does. Given its name, it might have something to do with alarm updates, though I don’t recognize this particular name.
Only queue_1 appears in the outgoing task statistics, but only the last one (queue_1941 as of now) appears in the outgoing queues. The total for queue_1941 may or may not be still increasing, I’ll have to check back later.
The total number of queues on the queue management page keeps increasing: it’s up to 1941, 4 more than when I first posted this thread.
Not-entirely-unrelated note: Sorting by total completely fails. It sorts alpha-numerically, which doesn’t help.
I’ve been monitoring things a bit more closely, and I believe the queue_1 entry in the outgoing statistic tasks gets incremented by 1 when a new queue is created (and/or used, maybe).
I had a new queue_1942 in the queues list, and noticed queue_1’s total had been incremented by 1, and I witnessed the same thing right now when queue_1943 appeared.
Both 1942 and 1943 totals were 1.
I’m note sure what information is relevant, so I’m adding everything I see.
You will see messages in those queues to communicate changes in alarm states to the remote tag providers (the queue is for sending those updates). If those subscriptions are being recreated (perhaps due to bad network quality, etc), the queues should be destroyed and recreated (with an incremented index number). Additionally, you might have individual outgoing queues (at the field gateway) for each subscriber (i.e. remote tag provider reaching into this gateway).
Why would _queue_1_ keep being incremented then ?
It's now up to _queue_1948_, so that's 6 more queues, and _queue_1_'s total has been incremented by 6. I guess what I'm trying to figure out what's going on there.
It's not just out of curiosity, I actually encountered this queue issue while investigating something else (suspiciously high traffic on the field gateways).
There should be only one.
Thanks again for your help !
