# Redundant Pair Backup Resync Repeating **URL:** \ **Category:** Ignition\ **Tags:** redundancy, ignition81\ **Created:** [November 2, 2022, 4:24pm UTC](https://forum.inductiveautomation.com/t/redundant-pair-backup-resync-repeating/66004 "2022-11-02T16:24:54Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![pturmel](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pturmel/32/3295_2.png) [@pturmel](https://forum.inductiveautomation.com/u/pturmel)\ **Post date:** [November 2, 2022, 4:24pm UTC](https://forum.inductiveautomation.com/t/redundant-pair-backup-resync-repeating/66004/1 "2022-11-02T16:24:54Z")
I have a newly upgraded redundant pair (v7.9.18 =\> v8.1.21) whose backup server won't synchronize. A snip from the log where it decides to do a full resync looks like this: ```auto INFO | jvm 3 | 2022/11/02 11:40:24 | I [R.S.BackupStateManager] [15:40:24]: Redundancy state changed: Role=Backup, Activity level=Warm, Project state=Good, History level=Full redundant-provider=system INFO | jvm 3 | 2022/11/02 11:40:24 | I [c.i.i.g.o.s.IgnitionOpcUaServer$OpcUaRedundancyListener] [15:40:24]: redundancyStateChanged: Role=Backup, Activity level=Warm, Project state=Good, History level=Full INFO | jvm 3 | 2022/11/02 11:41:05 | I [R.S.SyncManager] [15:41:04]: [internaldb] Will perform full pull. Reason: major change detected between nodes (sync provider UUIDs did not match). gan-remote-gateway-name=Ignition-801SCADA-Master, redundant-provider=internaldb INFO | jvm 3 | 2022/11/02 11:41:05 | I [R.S.SyncManager] [15:41:04]: Sync provider 'internaldb' requires full sync and will result in a restart. gan-remote-gateway-name=Ignition-801SCADA-Master, redundant-provider=internaldb INFO | jvm 3 | 2022/11/02 11:41:05 | I [R.S.BackupStateManager] [15:41:04]: Redundancy state changed: Role=Backup, Activity level=Warm, Project state=OutOfDate, History level=Full gan-remote-gateway-name=Ignition-801SCADA-Master, redundant-provider=system INFO | jvm 3 | 2022/11/02 11:41:05 | I [c.i.i.g.o.s.IgnitionOpcUaServer$OpcUaRedundancyListener] [15:41:04]: redundancyStateChanged: Role=Backup, Activity level=Warm, Project state=OutOfDate, History level=Full gan-remote-gateway-name=Ignition-801SCADA-Master INFO | jvm 3 | 2022/11/02 11:41:05 | I [R.S.BackupStateManager] [15:41:04]: Peer node information has been updated: RedundancyNode(address=172.16.70.8, httpAddresses=[http://172.26.70.8:8088], activityLevel=Active, projectState=OutOfDate) gan-remote-gateway-name=Ignition-801SCADA-Master, redundant-provider=system INFO | jvm 3 | 2022/11/02 11:41:12 | I [R.S.BackupStateManager] [15:41:12]: System restore file successfully downloaded and processed. System will now restart. gan-remote-gateway-name=Ignition-801SCADA-Master, redundant-provider=system INFO | jvm 3 | 2022/11/02 11:41:12 | I [IgnitionGateway] [15:41:12]: Restarting JVM gan-remote-gateway-name=Ignition-801SCADA-Master INFO | jvm 3 | 2022/11/02 11:41:13 | I [IgnitionGateway] [15:41:13]: Ignition[state=RUNNING] ContextState = STOPPINGAfter repeating five times, the wrapper kills the backup server's service. ``` I have ticket #63436 started, but I was wondering if anyone else has seen this (or similar). The wrapper kills off the backup gateway's service after it loops five times. ---
**Author:** ![paul-griffith](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/paul-griffith/32/37249_2.png) [@paul-griffith](https://forum.inductiveautomation.com/u/paul-griffith)\ **Post date:** [November 2, 2022, 4:36pm UTC](https://forum.inductiveautomation.com/t/redundant-pair-backup-resync-repeating/66004/2 "2022-11-02T16:36:10Z")
How large is the state-transfer .gwbk in the data dir on the backup? If you open it, does it look well-formed? ---
**Author:** ![pturmel](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pturmel/32/3295_2.png) [@pturmel](https://forum.inductiveautomation.com/u/pturmel)\ **Post date:** [November 2, 2022, 4:44pm UTC](https://forum.inductiveautomation.com/t/redundant-pair-backup-resync-repeating/66004/3 "2022-11-02T16:44:28Z")
213 MB. Ark opens it fine. An actual gateway backup from the master is ~ 77 MB. ---
**Author:** ![pturmel](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pturmel/32/3295_2.png) [@pturmel](https://forum.inductiveautomation.com/u/pturmel)\ **Post date:** [November 2, 2022, 4:49pm UTC](https://forum.inductiveautomation.com/t/redundant-pair-backup-resync-repeating/66004/4 "2022-11-02T16:49:22Z")
Opening the db\_backup\_sqlite.idb file is fine, and passes an integrity check in my SQLite DB Browser. ---
**Author:** ![paul-griffith](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/paul-griffith/32/37249_2.png) [@paul-griffith](https://forum.inductiveautomation.com/u/paul-griffith)\ **Post date:** [November 2, 2022, 4:52pm UTC](https://forum.inductiveautomation.com/t/redundant-pair-backup-resync-repeating/66004/5 "2022-11-02T16:52:09Z")
Can you manually restore the state transfer via gwcmd (might have to make the master independent first)? 213 mB is moderately hefty, but I would expect some errors in the logs if you're hitting a timeout or something. I just know there have been some problems in the past with larger backups syncing properly. ---
**Author:** ![pturmel](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pturmel/32/3295_2.png) [@pturmel](https://forum.inductiveautomation.com/u/pturmel)\ **Post date:** [November 2, 2022, 4:54pm UTC](https://forum.inductiveautomation.com/t/redundant-pair-backup-resync-repeating/66004/6 "2022-11-02T16:54:08Z")
Would it be worth trying with the real backup from the Master? Or are there things in the transfer backup that are needed? ---
**Author:** ![paul-griffith](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/paul-griffith/32/37249_2.png) [@paul-griffith](https://forum.inductiveautomation.com/u/paul-griffith)\ **Post date:** [November 2, 2022, 4:56pm UTC](https://forum.inductiveautomation.com/t/redundant-pair-backup-resync-repeating/66004/7 "2022-11-02T16:56:18Z")
Modules, apparently? But if they're the same between the two gateways, a restore from the master is probably fine. ---
**Author:** ![pturmel](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pturmel/32/3295_2.png) [@pturmel](https://forum.inductiveautomation.com/u/pturmel)\ **Post date:** [November 2, 2022, 4:58pm UTC](https://forum.inductiveautomation.com/t/redundant-pair-backup-resync-repeating/66004/8 "2022-11-02T16:58:58Z")
I can't conveniently make the server independent. It doesn't have internet access for pushing the license back to IA. Or can I move the `.ipl` files out of the folder? ---
**Author:** ![paul-griffith](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/paul-griffith/32/37249_2.png) [@paul-griffith](https://forum.inductiveautomation.com/u/paul-griffith)\ **Post date:** [November 2, 2022, 5:00pm UTC](https://forum.inductiveautomation.com/t/redundant-pair-backup-resync-repeating/66004/9 "2022-11-02T17:00:09Z")
Probably? I would expect trial or possibly emergency activation mode to suffice to get things in the right place. ---
**Author:** ![pturmel](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pturmel/32/3295_2.png) [@pturmel](https://forum.inductiveautomation.com/u/pturmel)\ **Post date:** [November 2, 2022, 5:13pm UTC](https://forum.inductiveautomation.com/t/redundant-pair-backup-resync-repeating/66004/10 "2022-11-02T17:13:31Z")
I was able to make it independent with the license files temporarily renamed, and it picked the licenses back up after I renamed them back. ---
**Author:** ![pturmel](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pturmel/32/3295_2.png) [@pturmel](https://forum.inductiveautomation.com/u/pturmel)\ **Post date:** [November 2, 2022, 5:13pm UTC](https://forum.inductiveautomation.com/t/redundant-pair-backup-resync-repeating/66004/11 "2022-11-02T17:13:54Z")
And it isn't syncing. Looping again. ): ---
**Author:** ![paul-griffith](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/paul-griffith/32/37249_2.png) [@paul-griffith](https://forum.inductiveautomation.com/u/paul-griffith)\ **Post date:** [November 2, 2022, 5:23pm UTC](https://forum.inductiveautomation.com/t/redundant-pair-backup-resync-repeating/66004/12 "2022-11-02T17:23:28Z")
Hmm. Maybe a jstack thread dump while it's in the loop would be helpful? ---
**Author:** ![pturmel](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pturmel/32/3295_2.png) [@pturmel](https://forum.inductiveautomation.com/u/pturmel)\ **Post date:** [November 2, 2022, 5:27pm UTC](https://forum.inductiveautomation.com/t/redundant-pair-backup-resync-repeating/66004/13 "2022-11-02T17:27:15Z")
> [@pturmel](#): > > `major change detected between nodes (sync provider UUIDs did not match)` I think this is the key. Care to grep the source for me? ---
**Author:** ![paul-griffith](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/paul-griffith/32/37249_2.png) [@paul-griffith](https://forum.inductiveautomation.com/u/paul-griffith)\ **Post date:** [November 2, 2022, 5:37pm UTC](https://forum.inductiveautomation.com/t/redundant-pair-backup-resync-repeating/66004/14 "2022-11-02T17:37:23Z")
Check the `redundancy.systemstateuid` in `redundancy.xml` on both sides? Or, less likely, the `"SystemUID"` column in `SYSPROPS` in the idb? ---
**Author:** ![pturmel](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pturmel/32/3295_2.png) [@pturmel](https://forum.inductiveautomation.com/u/pturmel)\ **Post date:** [November 2, 2022, 5:44pm UTC](https://forum.inductiveautomation.com/t/redundant-pair-backup-resync-repeating/66004/15 "2022-11-02T17:44:32Z")
I think we have a winner. In the master: ```auto e47f3e14-c3be-4056-89e0-44e779a56aa9 ``` In the backup: ```auto 9b0a31a0-acb0-46c5-aaf7-bc78fcea5c2b ``` Testing..... ---
**Author:** ![pturmel](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pturmel/32/3295_2.png) [@pturmel](https://forum.inductiveautomation.com/u/pturmel)\ **Post date:** [November 2, 2022, 5:59pm UTC](https://forum.inductiveautomation.com/t/redundant-pair-backup-resync-repeating/66004/16 "2022-11-02T17:59:16Z")
And not. These UUIDs are changing on each loop, and are different on master vs. backup. The value in the SYSPROPS table matches the value in redundancy.xml. ---
**Author:** ![pturmel](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pturmel/32/3295_2.png) [@pturmel](https://forum.inductiveautomation.com/u/pturmel)\ **Post date:** [November 2, 2022, 7:05pm UTC](https://forum.inductiveautomation.com/t/redundant-pair-backup-resync-repeating/66004/17 "2022-11-02T19:05:35Z")
FWIW, this is fixed. Support found this in my wrapper log: ```auto INFO | jvm 2 | 2022/11/02 11:38:22 | E [c.i.i.g.r.RedundantPersistenceInterfaceImpl$RunTask] [15:38:22]: Error applying redundant update. INFO | jvm 2 | 2022/11/02 11:38:22 | simpleorm.dataset.SRecordInstance$BrokenOptimisticLockException: Broken Optimistic Lock [DeviceSettingsRecord 67 Dirty0] ``` I disabled that device. Something re-enabled it instantly. I deleted that device. Backup server stopped resyncing. /: I have since found a script doing this, wildly, as a side effect of slight behavior difference in `objectScript`. Oy! ---
**Author:** ![djgilson](https://avatars.discourse-cdn.com/v4/letter/d/f14d63/32.png) [@djgilson](https://forum.inductiveautomation.com/u/djgilson)\ **Post date:** [December 12, 2023, 7:22pm UTC](https://forum.inductiveautomation.com/t/redundant-pair-backup-resync-repeating/66004/18 "2023-12-12T19:22:27Z")
Can you tell us what service was running that you deleted? From your screen shot I am not sure. I am having the same issue on 3 sets of redundant servers. Thanks! ---
**Author:** ![pturmel](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pturmel/32/3295_2.png) [@pturmel](https://forum.inductiveautomation.com/u/pturmel)\ **Post date:** [December 12, 2023, 7:46pm UTC](https://forum.inductiveautomation.com/t/redundant-pair-backup-resync-repeating/66004/19 "2023-12-12T19:46:59Z")
I didn't delete a service. I deleted a device connection under the OPC Server configuration. The root cause was a wild gateway script that was re-enabling a broken device--causing a "flood" of redundancy updates for which the backup could never "catch up". ---
**Author:** ![Cole\_Townsend](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/cole_townsend/32/53149_2.png) [@Cole\_Townsend](https://forum.inductiveautomation.com/u/Cole_Townsend)\ **Post date:** [October 18, 2024, 2:57pm UTC](https://forum.inductiveautomation.com/t/redundant-pair-backup-resync-repeating/66004/20 "2024-10-18T14:57:53Z")
I had a similar issue today but with a different device/lock. ![image](https://us1.discourse-cdn.com/inductiveautomation/original/3X/f/b/fb24223873a2c3cba14c79a5e119a9ecc75736c3.png) Do you know if there is a reference somewhere to decipher what IncomingConnection 411 refers to? My first guess is a connection to the gateway network. Is this reference stored in the local database? [Next page](https://forum.inductiveautomation.com/t/redundant-pair-backup-resync-repeating/66004.md?page=2)