Gateway Redundancy Providers System Events (Remote needs to do full pull)

We have noticed on the Redundancy page, repeatedly the events :

  • [alarm_system] Remote needs to do full pull. Reason: redundant provider on remote side has no data.

In the logger info we have also repeatedly warnings :

  • Partial sync operation failed: result was 'Bad_OutOfRange("Remote version has different UUID, or remote revision is too far behind")'. Full sync will be needed.
  • Partial sync operation failed: result was 'Bad_OutOfRange("Full pull already in progress")'. Full sync will be needed.
    The redundancy properties are stating 'Activity Level' : Active and 'Synchronisation Status : ' : Good

Are these logging something to be worried about?
Do we have to perform a 'Force Re-Sync'?

Information about the redundancy :

afbeelding

The Dispatched Updates/sec seems to be high, somtimes a warning is shown
afbeelding

Information about the system :

  • Ignition : 8.1.29
  • Tags : 744332
    afbeelding

afbeelding

Was there any point in time where you might have upgraded a module on the master while the backup wasn't synchronized? Having mismatched modules prevents synchronization. (Been there. Symptoms like yours.) Module installs while synchronized are propagated from master to backup.

No, the last time we dit a full update to 8.1.29 on both master and backup.
I've also performed a "Force Re-Sync" but it quickly again shows " [alarm_system] Remote needs to do full pull. Reason: redundant provider on remote side has no data."

I'm wondering if we have to much Dispached Updates / sec (at this moment 3,250.4)
Qued Updates is at 4,166.5
and Pending Updates sometimes low, sometimes > 8,000

That update queue sounds suspicious. Are your gateways not on the same LAN? (Putting a WAN between a master and backup is not supported, because the bandwidth and low latency requirements become a problem for anything but trivial setups.)

We are having the same issue with 3 systems with redundancy. I have been working with IA, but no answers yet. We are running version 8.1.32. We are not sure when this started. 2 of the systems have been running without the message for over a year. We noticed it on the system we added redundancy to a few weeks ago. Then we noticed the other 2 systems had the same message.
Hope to have an answer soon.

Those are two virtual machines from one of out customer. I will have to askt them if they are on the same lan

Hello everyone, do you have any updates?.

I'm having the same issue on Ignition 1.8.35 (the master is hosted on the cloud and the redundancy is on premisse). Also, I believe that this is related with other alarm times issues.

That is simply unsupportable. Don't do that.

Thank you for your fast response, I will work on that.
Do you know if there is any document that details why this is not recommended?, I would like to understand more about that, would you mind to explain to me a little bit more please. Since, it's an important change I need to support it before perform this change.

See this recent topic:

1 Like

More topics:

https://forum.inductiveautomation.com/search?q=split%20brain%20%40pturmel%20order%3Alatest