Alarm Pipeline - Multiple Notification Methods Simultaneously & Escalation

Hello,

I am tasked with creating Alarm callout logic for a client using different notification methods simultaneously as well as escalation, but I am confused as to how to best achieve this.

The following is my client's desired "Functional Description" for alarm callouts:

1.) A Roster of users titled "Email_Always" will be emailed every time an alarm event goes active.

2.) When an alarm goes active, two users that are part of a Roster titled "Lead _Backup" will be called, texted, and emailed at the same time. They won't be able to acknowledge from any of these notification methods so they will have 5 minutes to log into Ignition and acknowledge from within the session or the alarm will call out to them again. If either the Lead or Backup acknowledges the alarm, the alarm will drop out of the alarm pipeline.

3.) If the alarm cycles through five attempts for both the Lead and the Backup without being acknowledged or cleared, the first user in a Contingency Roster will be called, texted, and emailed at the same time. If the first user of the Contingency Roster logs in and acknowledges the alarm within 5 minutes, the alarm will drop out of the alarm pipeline. If the first user doesn't acknowledge the alarm within 5 minutes, it will go to the second user.

4.) The second user will be called, texted, and emailed at the same time. If the second user logs in and acknowledges the alarm within 5 minutes, the alarm will drop out of the alarm pipeline. If the second user doesn't acknowledge the alarm within 5 minutes, it will go to the third user.

5.) The third user will be called, texted, and emailed at the same time. If the third user logs in and acknowledges the alarm within 5 minutes, the alarm will drop out of the alarm pipeline. If the third user doesn't acknowledge the alarm within 5 minutes, the entire call-out order will start again from step 1.

I've been experimenting with this for a couple of days, and along the way I've encountered a few wrinkles:

1.) Emails and SMS notifications take far less time to "complete" than do VoIP phone calls. So, notifying users via all three methods is a difficult thing to synchronize. For VoIP in particular, I have to contend with the Call Timeout & Answer Timeout Durations defined in the Gateway, as well as the delay between contacts on the notification block using VoIP. This is difficult because technically calls can take different amounts of time to "complete". For example, when a user answers and presses a key, if the Answer Duration is exceeded, or if the user presses the end call button when the system calls.

2.) Splitter blocks create instances of alarm events and I want to avoid multiplying nuisance alarms wherever possible.

In any case, I'm new to Ignition so any advice for putting something like this together would be greatly appreciated. I can attach screenshots and provide more details as needed. Cheers!

Oy! Your client is insane.

If I was on the receiving end of a system that simultaneously emailed, texted, and voice called me, I'd be hunting a new job that day.

Email is a medium that is highly reliable for eventual delivery. It should never be resent, unless the resend has new or additional information. Definitely do not resend, even with new info, in less than 30 minutes or so. (If any such emails are to mail servers not under your control, your sending server would likely be flagged as a spammer.)

Text is a medium that is very quick under normal circumstances, but can be entirely missed by the intended recipient. It is also very limited in message length. It is best used as a "alarms present--check your email" notice, with link to server embedded. Text processors are also very sensitive to high rate senders, and the spam flagging is even more aggressive. I recommend not sending texts until several minutes after the first email is sent.

Voice is quick with reliability immediately known, but the heaviest weight. While it can be used immediately for high priority alarms, and will likely not be flagged by the phone system as a spammer, It is also the most annoying.

All of the above should be deferred for several minutes if the intended recipient has a live user session. The user session should play sounds or flash a docked window or otherwise attract attention first.

IMNSHO.

3 Likes

Sometimes you have to site down with a client and do a reality check.

Really find out why they want it like that? Get to the root of their problem.

Offer technological insight on a level they understand. Go through a learning session and why it won't really work how they want.

That "Functional Description" was written by someone who is not ever going to be on the alarm roster. "Accidentally" Include them in your testing and see how quickly they modify their thinking.

5 Likes