I’ll call this a case study because that’s kind of what it looks like. There is no code (sorry - there isn’t much to see anyway), but rather a dialog of how we made it work. I hope this helps others build something, and gets some feedback to IA on how people are using it.
This on call scenario is for monitoring a remote pumping station that is critical in colder weather. It was recently upgraded to AB PLCs, integrated to Ignition and is connected via redundant fiber to where Ignition lives. I was very excited when Ignition came out with their voice notification module - it meant we could do away with an autodialer, and the numeric pager that got passed around (yes, pagers still exist). Myself and a local integrator spent about 3 days getting the alarming working how we wanted. I’m quite happy with the result.
Requirements
- Voice dial alarms (not all people on call have smart phones for email)
- Dial the person on call, dial the supervisor if they don’t acknowledge
- Always send the supervisor the alarm via email, and an email when it is acknowledged
- On call list changes weekly, but has no pattern
- Ability to change who is on call for a day or days easily, by potentially anyone
Considerations
We first looked at the built in rosters and schedules. If we used a single roster of all people on call and holiday schedules for those not on call, this mostly works. However, there is no option for odd schedules (they don’t repeat with any pattern), and swapping who is on call temporarily requires two changes and is therefore error prone. I also didn’t want to give people that much access to change them.
We then thought about having a roster that only had the on call person and the supervisor in it. Each week, they would update the roster, but again, this relies on human intervention. If there was the ability to edit the roster programmatically, we could have done this.
We looked at setting the alarm pipeline via a tag, and having different pipelines for each person, but there is no mechanism if an invalid pipeline is selected by some chance. This also necessitates a pipeline per person. Our solution avoids a pipeline per person, but not a roster per person.
The Solution
A screen was made with a calendar control that updated a database. The user can add an event and select a list of people on call, then give them a time period to be on call. There are checks to make sure two people cannot be on call at the same time. However, they can add an “alternate” to any time period which overrides the normal weekly schedule. Checks are done before the database is updated to make sure this is all consistent. The supervisor will typically fill this out for 2-3 months in advance.
A gateway script looks at the database at a timed interval and evaluates who should be on call at that moment, using the start/end times of the events. The event names are simply the name of the person. The resulting name of the person on call is written to a memory tag. If no one is on call, an alarm tag is written to and an email is sent to the supervisor.
The alarm tags that connect to the PLC have “associated data” added to the alarm condition, which is bound to the tag with the name of the person on call. When the alarm is triggered, this data gets put into the pipeline.
All alarms are directed to a main pipeline where they split to an email and phone pipeline (using jumps).
The email pipeline sends an immediate email the supervisor. Using an expression, it checks {isAcked} every 30 seconds for up to 30 minutes. If it is acknowledged, an additional email is sent so the supervisor knows the call was taken (unfortunately there is no way we could see to determine who acknowledged it). If it is not acked, we send an email saying it timed out so we don’t fill up the pipelines with junk (and they terminate after some time anyway).
Simultaneously, the voice pipeline feeds into a switch, looking at the associated data and determining who is on call. The outputs of the switch (one for each person) feed to notifications block for each person that notifies a roster containing just that person. If that fails, the supervisor is notified. If the switch doesn’t match a condition (someone screwed up the programming) the supervisor is called.
Lessons Learned (and features requested)
The pipeline editor is painfully slow with 15 notification blocks connected to a switch.
If we could programmatically edit the roster, we could avoid a lot of duplication on the rosters and notification blocks. In fact we could remove the associated data and the switch for the users.
If we could use the expression block to look at values outside the pipeline, we could avoid the associated data with each alarm. (ie look directly at the tag of who is on call). In our testing this didn’t work.
There is no way to set common dialing properties (dial 9 to get out), which made pulling the users from our active directly pointless. I created a database user source just for on call contacts so I could edit it in SQL if needed. At the end of this post, I just realized I could have used a generic database user and just changed their contact number on the fly…
On the positive, initially we weren’t sure if we could get it to work, but we found a way, so that’s a plus (yay Ignition).
Tom