I have a customer running version 8.0.12 that notified me of an issue where they were only receiving alarms via SMS, but not voice or pager (email). I logged into the gateway and checked the pipeline status and there are 20-30 alarms in here that seem to be stuck in the notification block with the status of throttled. I clicked cancel but they did not clear out. I switched rosters, did a few other things and had it calling my phone in which I was able to acknowledge the alarm. I also injected a fake alarm in to the pipeline via the gateway which worked, and was able to click cancel for each of these on that screen but they still won’t clear out. I am not sure if this is why the voice or pager notifications were not working.
Go to the Alarm pipeline block and show the configuration there. How do you have the blocks set up as well as what settings do you have on the notification block? What’s the settings for delay between messages as well as do you have consolidation enabled and what are the timeouts?
The only notification block that was working was the top one that just texts the on call operators phone
I have consolidation disabled currently. It was enabled originally.
These are the setting for the pipeline itself
Does that alarm flicker off and on? Is there any delay time before sending it to the pipeline (in the alarm props) to effectively debounce it?
It wasn’t just one alarm, there were others throughout those 5 days. They are debounced in the PLC.
I don’t know if you solved this, but I just spoke to a guy from IA and he mentioned that you can see what the pipeline is doing from the thread stack trace. So if you get a thread dump and look at it in Kindling, you might be able to work out where it’s stuck. Filter the dump by “alarm-notification” and you should see results in there.
But a throttled alarm pipeline is generally caused by something delaying it from exiting, like:
- alarm consolidation
- long-running scripts in the pipeline
- loops in the pipeline that haven’t exited yet
These are probably the overlap indicating the issue. If the second notification block isn’t working then it’s just endlessly stuck.