In our implementation we use notification pipelines for almost all alarms when they go active and when they clear. We also have started dynamically enabling/disabling sets of alarms. We have observed that when an alarm transitions from not enabled to enabled an event hits the notification pipeline. Either an "Active" event or "Clear" event depending on how the alarm is evaluated after it is enabled. This has resulted in SCADA getting a lot of bad attention because some condition changes which enables a bunch of alarms and each one sends out a "Clear" notification (potentially hundreds or clear notifications in a few seconds) even though none of the alarms were "Active".
Additionally these "Clear" events do not show in the Perspective Alarm Status table. They do appear in the Alarm Journal Table ("Enable" and "Clear" events with matching Event IDs).
So my primary question is how can I effectively and efficiently block a notification for an alarm that just transitioned from not enabled to enabled and is "Clear".
I'm looking at evaluating the alarm event object in the notification pipeline but it's not clear yet how I can detect this specific scenario in order to cancel the notification. Its complicated by the fact that the alarm event object does include "ActiveData" event though there was no "Active" event (seemingly associated with the "Disable" event timestamp-wise but I'm not sure if that is consistent).
@PGriffith I'm not sure if Ignition devs would consider this a bug but it seems to be undesirable behavior for any use case I can dream up... sending a Clear event to the notification pipeline when an alarm is enabled even though there was no "Active" alarm event/state that was cleared.
1 Like
Did you go into your alarm settings and enable 'Notify Initial Events'?
"Notify Initial Events" is set False, are you suggesting it should be set True (I think not)? With it set False we should be getting no notification for first evaluation after the alarm is enabled, but we certainly are.
Ideally we would want notification for an alarm that evaluates as "active" after it is enabled, but no notification for an alarm that evaluates as "clear" after it is enabled.
To meet my ideal I suspect we should maybe have "Notify Initial Events" set True but then have some means to block the initial clear events.
I have now tried "Notify Initial Events" both True and False and the initial event after the alarm is enabled still hits the notification pipeline either way. I'm assuming I don't have to restart the gateway for that setting to take effect.
I would formally contact support at this point. I vaguely remember seeing a bug ticket for something to do with alarm notifications ~recently in 8.1, but I don't remember the details and couldn't find it in a quick search. If it's not something we're aware of, support is your best bet to get it properly triaged and fixed. I don't work on the alarming system.
Quick update here. I tried to create a minimal setup to reproduce this issue on a clean/fresh dockerized Ignition install and could not. In the "clean" setup the "Notify Initial Events" settings more or less behaved as-advertised; when "False", after an alarm transitioned from not enabled to enabled initial state active/clear alarm events were not sent to the pipeline; when "True", after an alarm transitioned from not enabled to enabled an initial active state caused an event to be sent to the pipeline but an initial clear state did not... that is basically what I think is normal/desirable behavior... so now I need to sort out why my larger/older gateway/project does not behave the same way.
It seems that the documented behavior of the "Notify Initial Events" setting is not entirely accurate.
Notify Initial Events:
Default is false. If false, active alarms caused by the "initial state" (that is, the first value checked after being created, or after the enabled state changes) won't be sent to the notification system. This means if you add an alarm to a Tag, a notification won't be immediately sent when the new state is created.
Through much trial and observation I finally realized what actually happens is that when the "Notify Initial Events" is set False, the alarm event is still created and is still passed to the pipeline but it is pre-acknowledged. Default settings for new notification pipelines includes "Acknowledged" as a dropout condition. So in a pipeline with "Acknowledged" still checked as a dropout condition the alarm event would hit the pipeline but immediately drop out. For notification pipelines where "Acknowledged" was unchecked as a dropout condition the initial state alarm notifications would hit the pipeline and not drop out regardless of the "Notify Initial Events" setting.
We had unchecked "Acknowledge" as a drop out condition on our notification pipelines very early in our development to keep ack notifications as a option (managed by setting a pipeline for the acknowledged state in the alarm configuration, or not).... this explains so many headaches we found alternate work-arounds for, on gateway restarts we got soooo many notifications that we had to add an expression to our pipelines to drop all notifications until the gateway had been running for a few minutes (by checking uptime).
In actuality the "Notify Initial Events" really just tells the system whether to auto-acknowledge initial event states. The current documentation/description is misleading (incomplete) and should not suggest it does more that that. @PGriffith can we influence changes to the documentation and setting description on the gateway pages?
TL;DR solution was to check "Acknowledge" as a dropout condition on each notification pipeline. The "Notify Initial Events" setting does not prevent events being sent to the notification system it just pre-acknowledges initial events (contrary to documentation). If you have your notification pipeline set to dropout on "Acknowledge" events it will also drop out the pre-acknowledged initial state events, if not it wont drop out the initial state events.
Only works as a general solution if you have no need to notify on acknowledge events.
There is an issue here. The new alarm event created due to the enabled state change is not being processed as a base, initial event, but instead a cleared, system-acked event from a Clear transition. This is only happening due to how alarms retain information prior to being disabled. The Notify Initial Events flag is still intended to exclude events such as this.
I'll get a ticket filed. Thanks for bringing this to our attention.
2 Likes
Thanks, please keep us updated.