Unexpectedly large, corrupted gateway backup file (~34GB)

I encountered an issue where the scheduled automatic backup created a backup file of 34.6GB. The expected backup fiels size is ~10MB. The backup file could not be opened as a zip, as it was reported to be corrupted.

This in turn caused issues when the git backup script ran and tried to push this large file to the repo.
Additionally, this file filled up the storage on our temporary test server, and project changes were lost when attemping to save work the following day.

I’m wondering if there is anyway to diagnose the root cause of this issue?

Any help would be greatly appreciated.

Can you look in the Ignition installation folder and determine if there are any really large files? I’d be interested in the data/db/config.idb and also if there is something that is growing under data/projects (as this path is collected within a gateway backup). Also, what version of Ignition is this?

Thanks for the reply Kevin,

The data/db/config.idb files is 4.6MB and the entire data/projects directory is 43MB (including the object files in the .git folder)

I am storing the automatic backups in /data/projects/gatewayBackups directory. I was initially wary that this might cause a loop during the backup creation, but read that the backup file will be ignored by Ignition.

Don't do that, it is why your backup file grows unbounded.

EDIT: I just tried setting this up on a fresh gateway container and sure enough, the first backup (of an otherwise fresh gateway) just keeps growing (I stopped it once it reached >1GB). I think you'll be just fine if you move that folder out of there (I'd recommend something outside of the data/ folder altogether) and adjust your scheduled backups accordingly.

Cheers Kevin,

I thought that was probably it.

I actually followed this guide to set up the git

and it states

" We recommend storing the Gateway backup to the following location:

<IGNITION INSTALL DIRECTORY>/data/projects/
Ignition-backup-XXXXXXXX-XXXX.gwbk

The Gateway backup will be ignored by Ignition."

Would be good if someone from IA could update this info to avoid this issue for others. @CMallonee

I have no control or say in what gets placed on the website, so I’m not sure why I’m being pinged here.

My Apologies,

do you know who the best person to refer this to is?

Apparently it’s the Marketing team. I’ve reached out to them with a recommendation to remove that suggestion, but I have no control whether or not they act on it.

To be fair, I think there’s a crucial distinction between the advice on the website and what you did.
By creating an intermediate folder gatewayBackups/, you’ve created something that looks enough like a project named gatewayBackups that the overall backup system is trying to helpfully include it. If you directly store scheduled backups at the root, then they won’t be detected as projects and thus recursively included in your backups.

EDIT: Although, the plot thickens - this may still not be quite working right. We’re working internally to figure out the best practice, and will get the document updated accordingly.

5 Likes

Thank you for the update Paul.

I’ll keep an eye out for what you come up with :slight_smile:

The gist of it is that there is an issue here, in that the backup process is too lenient with what it includes inside the projects/ directory. If you want to use a nested folder for organization, then you can name it with a leading ., i.e. projects/.scheduledBackups/, and the folder will thus be ignored by both the project system and the backup system. Or, you can (manually) rename your .gwbk files into .bak files (or append the .bak suffix, i.e. backup_xxx.gwbk.bak - .bak files are blacklisted by the backup process. The latter is a strictly manual solution, though - the scheduled backup system doesn’t let you change filenames.

1 Like

I opted for the .scheduledBackups and it works as expected.

Thank you for looking into it.

1 Like

I had the same problem, but I didn’t create a separate folder, I just put them in /data/projects/Ignition-backup-XXXXXXXX-XXXX.gwbk as advised by the deployment guide. Slowly ate up my whole drive, 40GB. After I moved it to .scheduledBackups, problem went away. This is 8.0.12. Will have an 8.1.17 version going soon, I will employ the same technique there.

Speaking from customer point of view, I really like that you have developers and QA folks on the site, most forums like this consist only of marketing people, and you can’t ever get to the engineering realities of anything in those situations. However, there needs to be a marketing person that is checking this site, and/or responsible to respond as to when/how/process of changing documentation would be.

Thanks

2 Likes

The training team do have a presence here, @Paul.Scott being one person I know of.

Definitely!! It's one of the many differences between a great company and one just there to gouge their customers of money without actually providing anything useful for it, and providing sub-par software from the 90's

1 Like

Odd, there were some changes that were supposed to be pushed out to that article. I'll check in with the team. Thanks for speaking up @ken.maze

Edit: looks like the changes were just pushed. For posterity, the article now suggests creating hidden folders (for example. .folder) for backups and images.

2 Likes

I see it, thanks everyone!