Tag Historian During Failover

General Discussion
1

I have a very unusual architecture that I am trying to get some more information on in some edge cases where recovering historical data is the ultimate goal. I know this is not an ideal setup, but any input would be appreciated.

The architecture:

  • Site A
    • Master Ignition Gateway
    • Primary SQL Server
  • Site B
    • Redundant Ignition Gateway
    • Secondary SQL Server

The database configuration:

  • Historian1 on Primary SQL Server (failover to Historian2)
  • Historian2 on Secondary SQL Server

The historian configuration:

  • Historian1 on Primary SQL Server
  • Historian2 on Secondary SQL Server

The tag configuration:

  • All historized tags are configured to Historian1

I am trying to figure out what exactly happens when the Historian1 database connection fails over to Historian2. From the testing I've done, I can see that if Historian2 schema is empty, Ignition will create all of the necessary tables in order to record the data. Once the connection is re-established to Historian1, any data on Historian2 is in there and only there.

My question is that if Historian1 fails over to Historian2 and tables are created automatically, is there a means of getting this data back since Ignition won't as part of failover? I understand the historical table mapping fairly well, and assuming there were no retired tags during failover, would it be safe to copy the rows in the failover sqlt_data table back in? If using a failover database, is store and forward typically disabled?

Thanks!

2

  1. Having actual redundant gateways on different sites is unsupported and unsupportable--you will get into "split-brain" situations. (Is this actual Ignition redundancy? Or something cobbled together with Independent gateways?)

  2. In general, database redundancy should be external to Ignition, using the DB brand's native redundancy/clustering/replication capabilites. This typically involves a floating IP address for the active DB, and Ignition neither knows nor cares what DB instance is behind that IP.

  3. Ignition redundancy has settings to control recovery and merging of history from a backup gateway when its master regains control. Depends on reliable detection of master/backup failure--that is, no possibility of split brain.

In your situation, you are entirely responsible for reconciling the history.

3

Thanks, Phil.

  1. I am using Ignition's redundancy for the Master/Backup gateways - that's to say they are licensed and configured as such.
  2. These are MySQL servers - not preferred, but I am working with what was given.
  3. Can you elaborate more on this? I don't remember seeing any configuration options to merge data, but maybe it does that automatically? Or is that dependent on having the DB's redundancy/clustering/replication in place?

-Kyle

4

See Config => System => Redundancy => Backup Node Settings => History Mode.

"Full" is appropriate when the DB is external to both gateways and/or is itself redundant with a single URL (floating IP or whatever).

"Partial" is appropriate when the DB is not expected to be available when the master is down.

There is no support for failover DB connections in lieu of proper external DB redundancy. IIRC, a failed-over DB connection stays connected to the secondary until forced back.

5

Thank you for elaborating on that, Phil.

From Database Considerations, hypothetically, if I have set up replication between these two databases and have chosen Partial History Mode, the gateway would use the failover database for any queries, but it will prevent data from going into the failover database. When the Primary database comes back, it will then insert the data into that database (which would then be replicated back into the Secondary).

  • Where is this data cached during failover - Ignition or SQL?
  • Is Store and Forward not recommended when using this?
  • Does Partial Mode only need to be applied to the Primary DB, Secondary DB, or both?
  • What does it mean by "prevent data from being inserted...in some cases"?

I have set up a couple VMs to test all this out.

Clustering Settings - History Mode

The history mode dictates how history will be treated when the node is not active. If partial, the data will be cached, and only forwarded when the master node is available. This mode can be used to prevent data from being inserted into a backup database in some cases. This setting can be found on the Redundancy page under the Config section of the Gateway.

6

Store and forward is involved by default with the historian, and is needed for "Partial" mode. (Which occurs only on the backup.)

Partial versus Full is a gateway setting, not a DB connection setting.

The insertion protection is for duplicate/overlapping data.

Your gateway pair should have one DB connection, which is diverted to the correct live DB by the DB clustering software, entirely out of Ignition's hands.

7

(post deleted by author)