This is admittedly a weird corner case, but if your projects directory is managed by git and you try to restore a gateway backup then the restore fails halfway. Some gateway properties are updated, but not all. It doesn’t toss up any immediate red flags, and you have to go look in the server log to see the issue. If you’re not paying enough attention, you may think the restore worked when it actually left things broken.
Since the projects directory in our backup were coming from another commit in the same git repository, the workaround I used was this:
- temporarily move
.git out of the projects directory
- do the gateway restore
- shutdown the gateway
- move
.git back into the projects directory
- do
git checkout and git reset to get back to the place I wanted to be
- restart the gateway
Like I said, this is a weird corner case since the primary purpose of using git is to not need to backup/restore the whole gateway or import/export the projects directory. But we wanted to re-synchronize all the tags and other gateway settings between two machines and bumped into this along the way.
How does it fail? What do you mean by that? An exception in the logs and a half resorted state? Can you share the logs or error message?
Exceptions in the logs and half restored.
wrapper_snippet.log (55.0 KB)
Thanks. Not really sure why git has anything to do with this, but we’ll see.
That snippet of logs may have been from my 2nd attempt to restore the backup. I had noticed some settings missing after the 1st attempt (from gateway interface) and tried a second time from the command line. I hadn’t touched the projects directory manually by this point. After the second failure, I found the “Restore failed: Failed to delete the data/projects directory. This may result in incorrect project overwrites and merges” warning and decided to try removing the “.git” directory. Then the 3rd attempt worked.
This might be a strange ownership issue? I have a service account setup for Ignition to run under, and the git repo can only be modified by that account. I was doing the restore using sudo gwcmd from the CLI.
What version are you using?
If .git is inside data/projects, then the restore process certainly will fail to delete the folder. Is there any reason you cannot track at the data level in the hierarchy, possibly with additional entries in .gitignore?
I was running 8.0.12 when this happened.
I suppose I could move the git management up a directory level, but I don’t think there is any benefit in managing other files at that level, so I’d just have to add ignores for everything other than projects. The only benefit from what I know would be avoiding this one issue that doesn’t come up very often in my workflow.
I should actually take time to sit and look around before saying stuff… There are definitely some configuration files up in the /var/lib/ignition/data directory that could possibly benefit from being in git. I need to compare the versions of those files between some of my servers and get a sense of which ones might be different.
And if I do decide to make this change, I’d like to do it without losing the git history on the projects so far, which means some git magic to rearrange directories and move the repo. (I’ve done similar before, just so infrequently I can’t remember the easy way to do it.) … And hopefully without interrupting the production server for too long, since it will need to match the new setup …