Redundancy Backup with Database Faulted

Ran into a situation that left us stuck.
One of our sites had a hard drive failure on the Master Ignition Server. It transferred everything over to backup without issue. While working to troubleshoot, we had to reboot the backup server, and when it did Ignition had issues connecting to the database so it left it in the faulted condition. This sometimes happens as the database if Ignition doesn't see the database running right aways. The solution was to go into the Database Setting and click save. Ignition then retries the connection and everything is good.

In this case it was no possible, as only the backup server was online. The backup Ignition server does not allow you to save the settings, so nothing would trigger Ignition to recheck the database. We were close to restarting the Master server so we sped up that solution so we could force the check.

There should be some other method to get Ignition to recheck the database while the Master ignition server is down.

TLDR: Cannot find any way to have Ignition resolve database faulted if the Master Ignition server is unresponsive.

1 Like

A solution of auto re-enable of the faulted database connection using a timed script with system.db.getConnectionInfo and system.db.setDatasourceEnabled was deployed and seems to be working. You need access to the Master of course but once the script is in the project, the ignition failover server is able to re-connect to the database when it happens to fault on startup.