Alerting/database setup via multiple gateways

Ignition
1

We are in the process of researching the feasibility of splitting our single gateway which monitors 3 sites into 3 separate gateways. This is so we can main local visibility to each site in the event that we run into an issue.

We have a main operations center which monitors the alerts for all 3 of our facilities, and also have individual operations centers at each site, so it is important that we maintain a single-pane view of the status of all of our facilitiese. Since the gateways can have realtime connections to multiple databases/sites, this isn’t an issue for the tags. The question is, how do we continue to grab alerts from each site? Currently, we have a single alert table which lists any new un-ACKed/active alert in the alert_log table. When we split, we will have 3 separate alert_log tables (one for each database/gateway).

I see that you can have multiple alert storage profiles in which you can choose the database to store to. In order to keep the alerts synced across all 3 facilities, we could use 3 alert profiles to accomplish this. This brings up the issue of having to acknowledge the alerts for all 3 tables. Rather than doing this, is there a way to aggregate the contents of multiple alert_log tables into one alert table component without having to write code to query all 3 tables? We still need to be able to acknowledge alerts occurring on equipment at any of our 3 facilities while being physically located at of the 3 facilities as well.

Sort of on a related note, we have some questions about alerting in an outage situation. We use the same database for our realtime & historical connections. This database is located on our SAN so we can get the best possible performance out of it, use features like replication/snapshotting, etc. During our split to the 3 facilities, we want to move our Ignition gateway server/Database server to a UPS that is separate from our other equipment to give us a bit of additional time. It is not feasible to move the SAN to this separate UPS. Do the alerts use the realtime or historical connection? If we would lose the SAN, would we still be able to receive alerts with both the realtime & historical databases being unavailable? Would it be best to just keep the historical database on the SAN, and the realtime database on the database server itself?

Thanks for the guidance!

2

Tough questions since you’ll run into pros and cons with different architectures (central versus distributed gateways). Get different opinions since you have so many factors. Here are some ideas that come to mind:

  1. Initially, and possibly in the long term, I would keep your alert logging on a central database that resides on your SAN. You shouldn’t lose alert data with connection losses since each of the gateways have the “store and forward” system. Downsides I can think of is that you may not have the most up to date historical alert info and that you won’t be able to see alert history from an isolated site.

  2. I lean toward keeping your alert status on a local database at each gateway.

  3. You can create a project or window at your operations center that reads alert status from databases at local and remote gateways. For “normal users” I would use a “retargeting” scheme so that they are aware that they are jumping between sites. This is safe to do - just consider what users should see when connectivity to a site is lost.

3a. You can consider a central “duplicate” alert setup that reaches out to remote sites. The advantage is that you have all of your data in one place. There are possibly far more disadvantages: you will be creating a bit more traffic, changes will need to be duplicated, status data is effectively duplicated, and you’ll need to decide which system to send and acknowledge alarms from. I don’t recommend going this route as it will most likely violate good design practice and lead to headaches.

  1. Direct alerting shouldn’t be dependent on status or history tables. This can be done centrally or split between sites. I recommend splitting it between sites. Just consider implications of connectivity in terms of viewing/acknowledging/receiving alarms. You could have an architecture where much more robust central connectivity outweighs the disadvantages.

I think there’s a lot of value in thinking about how each part could work split. It’ll then become more clear on which parts should be centralized versus distributed. You will likely end up with a hybrid scenario.