Intermittent Redundancy Synchronization Loss During Large Configuration Changes – Expected Behavior or Performance Issue?

Ignition
1

Hello everyone,

I would like to ask about intermittent redundancy synchronization issues in an Ignition Master/Backup Gateway environment, particularly when making large configuration changes through Designer.

We are using Ignition for a large-scale data center EPMS (Electrical Power Monitoring System), where system availability and redundancy stability are important operational requirements.

We have observed that the redundancy connection occasionally becomes interrupted and subsequently recovers. We have also noticed that the Backup Gateway sometimes enters the "Syncing" state when a large number of Tag Group or Device configuration changes are made through Designer.

I would like to better understand whether this behavior is expected during configuration synchronization or whether it indicates a potential performance or communication issue.

1. System Environment

  • Application: Data Center EPMS

  • Architecture: Ignition Master/Backup Gateway Redundancy

  • Communication protocols: IEC 61850 (MMS Polling) and Modbus TCP

  • Database architecture: Separate database server

  • Engineering: Configuration changes performed through Ignition Designer

  • System scale: Large-scale deployment involving numerous electrical monitoring devices

2. Observed Behavior

We have noticed the following behavior during operation:

  1. The redundancy connection occasionally becomes interrupted and then reconnects.

  2. When modifying multiple Tag Groups or Device configurations through Designer, the Backup Gateway sometimes transitions into the "Syncing" state.

  3. After some time, synchronization appears to recover.

At this point, we have not confirmed whether the temporary synchronization interruption is directly related to configuration changes, Gateway resource utilization, or network communication.

3. Questions

Q1. Is it normal for the Backup Gateway to enter the "Syncing" state when a large number of configuration changes are made through Designer?

For example, when modifying multiple Device configurations or Tag Group settings, does Ignition automatically initiate a broader synchronization process between the Master and Backup Gateways?

Q2. What is the expected behavior when redundancy synchronization is temporarily interrupted?

Specifically:

  • Does the Backup Gateway continue operating with its last successfully synchronized configuration?

  • Can temporary synchronization loss affect the Backup Gateway's readiness to take over during a Master failure?

  • How can we distinguish normal configuration synchronization from an actual redundancy communication failure?

Q3. Are there recommended limits or best practices for making large configuration changes in a redundant environment?

For example:

  • Should Device and Tag Group configuration changes be performed in smaller batches?

  • Can frequent configuration changes cause additional synchronization overhead?

  • Is there a recommended method for minimizing redundancy synchronization interruptions during engineering work?

Q4. What diagnostic information should we collect to identify the root cause of intermittent synchronization loss?

Are there specific Gateway loggers, redundancy diagnostics, or performance metrics that IA recommends monitoring?

We would particularly like to understand whether CPU utilization, JVM memory, garbage collection, network latency, or synchronization workload could contribute to this behavior.

Q5. What are the recommended redundancy design and operational practices for large-scale EPMS systems?

We are also preparing a new system using Ignition 8.3.6 with Master/Backup redundancy and MS SQL Server.

We would appreciate any recommendations for preventing similar synchronization issues in the new environment.

4. Additional Considerations

Since our system uses IEC 61850 MMS Polling and Modbus TCP, we are also interested in understanding whether temporary redundancy synchronization loss could affect device communication behavior during an actual failover.

Additionally, are there any considerations regarding Tag Historian and Alarm Journal continuity when the Backup Gateway is temporarily not fully synchronized?

Our primary goal is to distinguish expected redundancy behavior from abnormal synchronization failures and establish a reliable operating procedure for large-scale Ignition deployments.

Any technical guidance, documentation, or recommendations from IA engineers and experienced users would be greatly appreciated.

Thank you!