We have a complicated alarm pipelines setup. First up, we’re using calculated rosters. On top of that, each user can get distinct messages based on their timezone or other factors using this method. Now, we want to add a rate limiter, and consolidation looks like a good possibility, but I’m unsure if that’s usable after all the other customization.
I’ve not found any detailed discussion of how alarm consolidation does it’s thing. The documentation only shows an example using a simple repeated message, which would not be compatible with my customized messages. Is there any way that consolidated messages can use custom messages similar to the CustomEmailMessage, CustomSmsMessage, and CustomEmailSubject properties?
Beyond that, will the consolidation system be confused by how we split alarm events and use distinct calculated rosters on each? Does consolidation aggregate messages based on the same recipient SMS/email or the alarm source tag or some other key?
Is there perhaps some other simpler method to get alarm notification rate limiting? I was looking at adding delays on the alarm configs, but then the users would receive no alarm at all if a tag crossed back and forth over a threshold within those time limits. We need our system to send at least one message in those cases, so I don’t think those delays would work for us.
2 Likes
Okay. Thought-experiment.
I could set this up in Ignition to try it, but I’m hoping someone here might know directly and save me a few hours of work.
- Let’s say I have two alarms X and Y
- Consolidation is enabled on the notification block
- All of the following alarm events happen in quick succession, within all consolidation timers
- Alarm X triggers, loops around the pipeline multiple times and sends:
- a notification with “First” in the CustomSmsMessage property to SMS user A
- a notification with “Second” in the CustomSmsMessage property to SMS user B
- a notification with “Third” in the CustomSmsMessage property to SMS user C
- Alarm Y triggers, loops around the pipeline multiple times and sends:
- a notification with “Fourth” in the CustomSmsMessage property to SMS user B
- a notification with “Fifth” in the CustomSmsMessage property to SMS user C
- a notification with “Sixth” in the CustomSmsMessage property to SMS user D
- Alarm X triggers again, loops around the pipeline once and sends:
- a notification with “Seventh” in the CustomSmsMessage property to SMS user A
- (notice it sent nothing to user B or C this time)
Now, when things settle down, how many alarms are sent to who? Which get the CustomSmsMessage and which get the consolidated message?
In my perfect world I would hope for:
- A consolidated message to user A containing the sub-messages “First” and “Seventh”
- A consolidated message to user B containing the sub-messages “Second” and “Fourth”
- A consolidated message to user C containing the sub-messages “Third” and “Fifth”
- A normal message to user D containing “Sixth”
In practice, I expect something like:
- A consolidated message to user A using the generic consolidated template instead of “First” and “Seventh”
- A consolidated message to user B using the generic consolidated template instead of “Second” and “Fourth”
- A consolidated message to user C using the generic consolidated template instead of “Third” and “Fifth”
- A normal message to user D containing “Sixth”
Does anyone reading this know what actually gets sent?
Alright, I put together a simplified version of my looping pipeline. Here’s an export: sandbox_2021-07-09_0159.zip (3.8 KB)
If an alarm named “X” or “Y” hits that pipeline, it will trigger in a manner similar to my thought experiment above. The notification block is in “test mode” so it won’t send any emails/SMS as is. You will probably need to switch notification profiles unless you have some with the same names as mine…
It did worse than I expected. With a single “X active” alarm event and nothing more, the looped variations of that one event to all users become a generic consolidated message to just one of them that lists multiple occurrences of that one event:
INFO | jvm 1 | 2021/07/09 01:50:27 | I [a.N.EmailNotificationProfile ] [01:50:27]: [hostname=<null>,port=25] THIS PROFILE IS RUNNING IN TEST MODE. The following email WOULD have been sent:
INFO | jvm 1 | 2021/07/09 01:50:27 | To: [bob@localhost]
INFO | jvm 1 | 2021/07/09 01:50:27 | Subject: Ignition Alarm Notification
INFO | jvm 1 | 2021/07/09 01:50:27 | Message: 2 alarm events have occurred:
INFO | jvm 1 | 2021/07/09 01:50:27 | Alarm "X" at "" transitioned to Active at 01:50:12
INFO | jvm 1 | 2021/07/09 01:50:27 | Alarm "X" at "" transitioned to Active at 01:50:12
INFO | jvm 1 | 2021/07/09 01:50:27 | alarm-pipeline=consolidation_test, project=sandbox, alarm-notification-profile=alarm-email-profile, alarm-source=default/sandbox/tagX/X, alarm-name=X
EDIT: Yuck. It seems to completely ignore the targets of the calculated roster as well. I modified the test so alarm X notifies users A and B, and alarm Y notifies user C, then triggered X and Y back to back. The end result is a consolidated message to just one user containing two lines for alarm X and one line for alarm Y.
I'm in the same kind of mess right now: Alarm pipeline: calculated roster and consolidation
Did you ever find interesting things and ways to manipulate rosters and consolidation ?
Have you found a solution yet?
I have not found a simple or easy solution.
I came to the conclusion that the only way to get what I want is going to be to do EVERYTHING in custom Python scripts. Meaning, the alarm pipelines would be stripped down to just a single line that passes the event off to Python. Then our Python code would have to maintain records of when notifications were sent to who, delay sending notifications in case of consolidation or rate limiting, and generate the SMS and emails directly using system.twilio.sendSms and system.net.sendEmail instead of an AlarmPipeline notification block.
The amount of work required pushed it down our priority queue and we still haven't attempted this yet.
1 Like
Same as Justin, couldn't find a way to make it work withing the pipeline's scope.
You'll have to implement your own system.
Just saw IGN-3256 on the nightly changelog that alters consolidation behavior...