A client requires a system to be always up.
Assuming we have a robust network infrastructure.
What are the scenarios, obvious and not, where redundant gateway will takeover?
My understanding is that the gateway will fail over when the master gateway fails. The backup gateway monitors the status of the master gateway to determine this, but even a network disconnection between the master and backup will look like a failure to the backup, so keep this in mind.
What scenarios are you trying to protect against? Loss of connectivity to devices for instance will not fail over to the other server even if the backup can connect to the device when the master cannot.
1 Like
Protect the MES. 100% uptime of backend gateway connecting to PLCs and front end gateway facing multiple operators.
Lost of device connection does not have something to do with gateway, exactly why I ask, when does the master gateway fail over to backup gateway.
That's it. When communication from the inactive gateway to the active gateway is broken, the inactive gateway will promote itself (after a configurable delay--10 seconds by default).
If only the comm channel between them dies, you will have a split-brain situation (both gateways trying to run). It is important that the connections to devices and databases run over the same physical links as the gateway-to-gateway channel.
3 Likes
This is fundamentally impossible to achieve. Uptime is measured by number of 9's as a percentage availability. Eg five 9's / 99.999% availability means ~5mins downtime allowed in a year. Even the most sosphicated cluster of servers cannot be guaranteed to be 100% available
1 Like
Hi, We are facing a split brain situation, where both gateways are writing to the database and as a result we have duplicate records. Has anyone faced this before and any solutions to this? Thanks.
I've never had the issue, but make sure your gateways can see each other. Look at the redundancy status pages on the primary and backup. Are these gateways on the same network or are you trying to do redundancy across different VLANs or networks?
To add on to what @michael.flagler is saying. If the gateways can each see the database, but cannot see each other, then you will have a split brain.
We are trying to fix the network issue. It seems to be with the PORT getting blocked on which the gateways talk. Thus the gateways cant see each other but can communicate with the database. So i wanted to see if there is some solution in case this happens again, as this is the 3rd time. And when this happens in the 3rd shift at night there is tons of duplicate data.
The solution sounds like it's a network problem, not an Ignition problem. Ignition servers are doing their job and if they can't see the other server they have no way of knowing the other server is running, so they assume control. Fix your network problem and the split brain problem should go away.
2 Likes
Document the fix to your firewall problem so it doesn't recur. (Though there shouldn't be a firewall between them, because they should be co-located.)
I suspect they could be using a software firewall (Windows blocks those ports by default) and they just needed to open a port.
Ah, well, y'all know how I feel about that.
I mean, they could be using some distro of Linux and maybe IT locked them down with iptables or UFW, but my bet is they're running Windows and never opened the correct ports.
1 Like