Architecture for Distributed Control Center

Hello All,

What would be the best architecture to follow for a central control center with 6 regional control centers? All of these control centers need to have a backup control center. The complete system has 14 control centers in total.

One option that came up was to put up a redundant pair of gateways in each of the regions. Would it be wise to have the primary server in one location and its redundant pair in another location?

Absolutely not.

1 Like

What would be the best way to implement the live/warm backup control center, with another pair of redundant servers ?

Do the servers in the alternate location need to be able to control remotely when both of the on-site servers are dead? If you can count on at least one of the local servers running, then you can simply use Remote Tag Providers and design projects that perform parallel control. (Distinct from redundant.)

Perhaps you can describe your required functionality in more detail?

Certainly, the primary objective of establishing a backup control center is to maintain control over assets in the event of a disaster occurring at the main control center, such as a lack of access due to road collapse or in the event of a flood.

All the assets/PLC have a redundant communication path to both these stations, the primary being a couple of FOC connections from different regions, with a failover to LTE (MPLS).

The requirement, in essence, is to have a replica of the main site at the backup center.

Parallel control. Not so much an Ignition redundancy setup.

I'd make sure each site (which you probably want redundant within the site) has a uniquely named realtime tag provider (not "default"), so that remote tag providers pointing into each site from the multi-site control centers can use those names. Then the UI projects can be shared across sites.

Phil, can you explain your answer? This architecture is quite common, especially for a lot of government projects I see.

Ignition redundancy is both latency sensitive and subject to split brain syndrome. If the comms goes down between sites, both servers will end up active and therefore numerous properties start to diverge.

So let me amend my answer to:

Ignition redundant servers must be installed on a common LAN with a robust network layout (preferably multi-path, like RTSP), such that any loss of comms also results in loss of comms to all clients and all devices and all databases. (Trunk port for multiple VLANs, or multiple trunk ports with RTSP and all VLANs on all ports.)

2 Likes

Thanks Phil, sorry I didn't read the OPs topic close enough (I just saw the word location). A location could be anything really, 2 feet away or 2 states away. The architectures I see are in different locations but close enough where they are connected via fiber.

If that comms layout makes it possible for both servers to talk to various parts of the system while also not being able to talk to each other, the resulting system could be unreliable when needed. A dedicated fiber between two physically nearby locations can typically satisfy the criteria, and have reasonable latency.

Anything going through the cloud or multiple service providers is unlikely to suffice.

The backup locations are ~30 to 40 km away. The entire ~400 stations are connected with dedicated single-mode fiber connections. And there is LTE failover, just in case there is any switch failure or fiber break. There are no multiple service providers or cloud in the picture.

So as I understand the best way to do this would be to have two pairs of GW, one at the main and another at the backup, and the GW with Modbus Client, polling all the PLCs (Modbus Masters).

Thanks Phil!

If you expect the on-site redundant gateways to be functional, just without humans available, you don't really need more gateways elsewhere. You just need a robust set of network paths to open clients on those gateways from the remote control centers.

I would not poll any modbus devices remotely. Or any other ordinary PLC protocol.

An external set of gateways for convenience working with many sites is advisable, but would not need to themselves be redundant.

You should get IA sales engineering involved to fine tune your plan based on your actual requirements.

1 Like