Consolidated alarms, calculated rosters, and custom messages! Oh my!

We have a complicated alarm pipelines setup. First up, we’re using calculated rosters. On top of that, each user can get distinct messages based on their timezone or other factors using this method. Now, we want to add a rate limiter, and consolidation looks like a good possibility, but I’m unsure if that’s usable after all the other customization.

I’ve not found any detailed discussion of how alarm consolidation does it’s thing. The documentation only shows an example using a simple repeated message, which would not be compatible with my customized messages. Is there any way that consolidated messages can use custom messages similar to the CustomEmailMessage, CustomSmsMessage, and CustomEmailSubject properties?

Beyond that, will the consolidation system be confused by how we split alarm events and use distinct calculated rosters on each? Does consolidation aggregate messages based on the same recipient SMS/email or the alarm source tag or some other key?

Is there perhaps some other simpler method to get alarm notification rate limiting? I was looking at adding delays on the alarm configs, but then the users would receive no alarm at all if a tag crossed back and forth over a threshold within those time limits. We need our system to send at least one message in those cases, so I don’t think those delays would work for us.

2 Likes

Okay. Thought-experiment.

I could set this up in Ignition to try it, but I’m hoping someone here might know directly and save me a few hours of work.

  • Let’s say I have two alarms X and Y
  • Consolidation is enabled on the notification block
  • All of the following alarm events happen in quick succession, within all consolidation timers
  • Alarm X triggers, loops around the pipeline multiple times and sends:
    • a notification with “First” in the CustomSmsMessage property to SMS user A
    • a notification with “Second” in the CustomSmsMessage property to SMS user B
    • a notification with “Third” in the CustomSmsMessage property to SMS user C
  • Alarm Y triggers, loops around the pipeline multiple times and sends:
    • a notification with “Fourth” in the CustomSmsMessage property to SMS user B
    • a notification with “Fifth” in the CustomSmsMessage property to SMS user C
    • a notification with “Sixth” in the CustomSmsMessage property to SMS user D
  • Alarm X triggers again, loops around the pipeline once and sends:
    • a notification with “Seventh” in the CustomSmsMessage property to SMS user A
    • (notice it sent nothing to user B or C this time)

Now, when things settle down, how many alarms are sent to who? Which get the CustomSmsMessage and which get the consolidated message?

In my perfect world I would hope for:

  • A consolidated message to user A containing the sub-messages “First” and “Seventh”
  • A consolidated message to user B containing the sub-messages “Second” and “Fourth”
  • A consolidated message to user C containing the sub-messages “Third” and “Fifth”
  • A normal message to user D containing “Sixth”

In practice, I expect something like:

  • A consolidated message to user A using the generic consolidated template instead of “First” and “Seventh”
  • A consolidated message to user B using the generic consolidated template instead of “Second” and “Fourth”
  • A consolidated message to user C using the generic consolidated template instead of “Third” and “Fifth”
  • A normal message to user D containing “Sixth”

Does anyone reading this know what actually gets sent?

Alright, I put together a simplified version of my looping pipeline. Here’s an export: sandbox_2021-07-09_0159.zip (3.8 KB)

If an alarm named “X” or “Y” hits that pipeline, it will trigger in a manner similar to my thought experiment above. The notification block is in “test mode” so it won’t send any emails/SMS as is. You will probably need to switch notification profiles unless you have some with the same names as mine…

It did worse than I expected. With a single “X active” alarm event and nothing more, the looped variations of that one event to all users become a generic consolidated message to just one of them that lists multiple occurrences of that one event:

INFO   | jvm 1    | 2021/07/09 01:50:27 | I [a.N.EmailNotificationProfile  ] [01:50:27]: [hostname=<null>,port=25] THIS PROFILE IS RUNNING IN TEST MODE. The following email WOULD have been sent:    
INFO   | jvm 1    | 2021/07/09 01:50:27 | To: [bob@localhost]                                                                                                                                         
INFO   | jvm 1    | 2021/07/09 01:50:27 | Subject: Ignition Alarm Notification                                                                                                                
INFO   | jvm 1    | 2021/07/09 01:50:27 | Message: 2 alarm events have occurred:                                                                                                                      
INFO   | jvm 1    | 2021/07/09 01:50:27 | Alarm "X" at "" transitioned to Active at 01:50:12                                                                                                          
INFO   | jvm 1    | 2021/07/09 01:50:27 | Alarm "X" at "" transitioned to Active at 01:50:12                                                                                                          
INFO   | jvm 1    | 2021/07/09 01:50:27 |  alarm-pipeline=consolidation_test, project=sandbox, alarm-notification-profile=alarm-email-profile, alarm-source=default/sandbox/tagX/X, alarm-name=X      

EDIT: Yuck. It seems to completely ignore the targets of the calculated roster as well. I modified the test so alarm X notifies users A and B, and alarm Y notifies user C, then triggered X and Y back to back. The end result is a consolidated message to just one user containing two lines for alarm X and one line for alarm Y.