We upgraded our redundant pair from 7.5 to 7.7 and now are running into issues. I have a question about how our upgrade was preformed. The company that did it for us took down the Master and converted the Backup to the Master role and then upgraded it. It was done this way we were told in case something went wrong we could just go back to 7.5 using the other Master server which was off line.
The upgrade seemed to go OK with the backup (now new Master) running 7.7 and then we went to upgrade the old Master. Due to our environment we wanted the old Master to stay the Master so the current Master was brought down, the old one brought up and upgraded to 7.7 and then the new Master was brought up and converted back to a backup role.
This seemed to work OK for about a day or two and then things started really going south. We can no longer have our Master on line because when we do all the clients run extremely slow (takes about 45 to 60 seconds for screens to change, sometimes longer) and when we try to launch Designer it shows the same effect (45 to 60 seconds to go between screens). This was not the case before we did the upgrade. Yes we did have some slowness at times but nothing along these lines, and we were able to use Designer just fine (no slowness there at all) prior to the upgrade.
Reading some posts on how to upgrade redundant pairs it seems like this was not the way to do it. However now I need to know how to get our redundant pair working again. Would it be smarter to make the current Backup server the Master and then rebuild the downed Master as the Backup, or could we rebuild the currently downed Master as a new Master? If the latter idea is doable what are the correct steps to take so that we don’t mess up our environment altogether.
Thanks
If you take the backup machine down, does the client and designer performance improve? i.e. does the system go back to normal with just the master gateway running? Also, what version of 7.7 are you running currently?
If your system returns to normal with the backup offline then I would wipe Ignition off of that server and start fresh. But the backup server settings in and let the two servers sync up.
Sorry, just saw that this post was here. I honestly can’t remember all the things we have tested trying to find a root cause of this. I’m double checking on this and if not we’ll test that out today and update accordingly.
Thanks
We did test shutting down the backup server completely and just having the master up and running. The issues still remain. In fact we have also noticed that our nightly backups went from a small size of about 45 MB to now over 500 MB for no reason, we haven’t been able to change anything. Luckily we have a good gateway backup from prior to the upgrade, we may have to start over again with this upgrade process following Travis’ post from here: support.inductiveautomation.com … ition-pair
This issue has been tracked back to the Internal Database Issue that was found after upgrading to version 7.7.5. The driver for the internal database was upgraded and a bug found that causes the internal backup to grow continually. This along with the weird upgrade path was causing a sync issue with our redundant pair. After using the database shrink tool, we were able to set up the redundant pair again and allow them to sync.
Please note that if you use the database shrink tool you may have to schedule a maintenance window to run it multiple times. We have found about every three weeks we need to run the tool to get the backups of the internal database back down in size as it grows a few hundred MBs in size during that time window.