[BUG-14227] Restoring gateway backup fails data and temp are on different Linux file systems

Ignition
1

I tried to restore a gateway backup on an Ignition 8.0.2 server running on Linux but got the error

com.inductiveautomation.ignition.gateway.localdb.DBStartupException: Error restoring internal database from backup.
jvm 2    |      at com.inductiveautomation.ignition.gateway.localdb.LocalDBManagerImpl.setup(LocalDBManagerImpl.java:162)
jvm 2    |      at com.inductiveautomation.ignition.gateway.IgnitionGateway.startupInternal(IgnitionGateway.java:820)
jvm 2    |      at com.inductiveautomation.ignition.gateway.redundancy.RedundancyManagerImpl.startup(RedundancyManagerImpl.java:272)
jvm 2    |      at com.inductiveautomation.ignition.gateway.IgnitionGateway.initRedundancy(IgnitionGateway.java:664)
jvm 2    |      at com.inductiveautomation.ignition.gateway.IgnitionGateway.lambda$initInternal$0(IgnitionGateway.java:602)
jvm 2    |      at com.inductiveautomation.ignition.common.execution.impl.BasicExecutionEngine$ThrowableCatchingRunnable.run(BasicExecutionEngine.java:518)

After restarting Ignition I got another error:

jvm 2    | com.inductiveautomation.ignition.gateway.localdb.DBStartupException: Error restoring internal database from backup.
jvm 2    |     at com.inductiveautomation.ignition.gateway.localdb.LocalDBManagerImpl.setup(LocalDBManagerImpl.java:162)
jvm 2    |     at com.inductiveautomation.ignition.gateway.IgnitionGateway.startupInternal(IgnitionGateway.java:820)
jvm 2    |     at com.inductiveautomation.ignition.gateway.redundancy.RedundancyManagerImpl.startup(RedundancyManagerImpl.java:272)
jvm 2    |     at com.inductiveautomation.ignition.gateway.IgnitionGateway.initRedundancy(IgnitionGateway.java:664)
jvm 2    |     at com.inductiveautomation.ignition.gateway.IgnitionGateway.lambda$initInternal$0(IgnitionGateway.java:602)
jvm 2    |     at com.inductiveautomation.ignition.common.execution.impl.BasicExecutionEngine$ThrowableCatchingRunnable.run(BasicExecutionEngine.java:518)
jvm 2    |     at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
jvm 2    |     at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
jvm 2    |     at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
jvm 2    |     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
jvm 2    |     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
jvm 2    |     at java.base/java.lang.Thread.run(Unknown Source)
jvm 2    | Caused by: java.io.IOException: Unable to move file '/usr/local/share/ignition/temp/temp_3605037950089348593db.idb' to '/var/lib/ignition/data/db/config.idb'
jvm 2    |     at com.inductiveautomation.ignition.gateway.localdb.sqlite.SQLiteDBManager.restoreFromDBBackup(SQLiteDBManager.java:436)
jvm 2    |     at com.inductiveautomation.ignition.gateway.localdb.LocalDBManagerImpl.restoreFromGatewayBackup(LocalDBManagerImpl.java:685)
jvm 2    |     at com.inductiveautomation.ignition.gateway.localdb.LocalDBManagerImpl.setup(LocalDBManagerImpl.java:160)
jvm 2    |     ... 11 common frames omitted

This second error made it a little more clear what was going on. I looked at the file permissions to see if it was a permissions issue, but both directories had correct file/group ownership and permissions.
It occurred to me that it could be because I had my data and temp directories on different file systems (i.e. different physical or logical disks or storage mechanisms).

After some more experimentation, I found that if I created a symbolic link so that the temp and data directories were on the same file system then the restore would succeed.

I’m not exactly sure what Ignition is doing behind the scenes to transfer the .idb file from the temp directory to the data directory, but ideally it would allow transferring across file system boundaries.

I understand this is probably a narrow use-case, but I thought I’d report this issue in case other users hit it.

1 Like
2

Thanks. While this will be difficult for QA to repro and test, it looks like an easy fix…

3

This is a bit old, but I am experiencing the same thing with a docker container running 7.8.12 and trying to restore a gateway backup, same error and all.

How exactly did you do this part to resolve your problem? Newer to linux here.

Thanks,
Keith G.

4

This issue had been addressed in the kcollins/ignition:8.0.2 docker image a while back but was never back ported to the 7.9.12 image, until this evening. It works around this issue for the moment by moving the temp directory into the data directory. When a Docker volume is mounted against the data directory, it creates the scenario with different file systems that this bug describes. Should be good to go with a fresh docker pull now.

1 Like
5

Any chance of pushing this back to the 7.8.5 image? If not, how about a starter course on docker compose and I will create a new 7.8.5 image with the fix?

6

It looks like this commit is the one where I finally removed the 7.7 and 7.8 areas from my repo (and thusly, this preceding one still has the 7.7 and 7.8 folders)… It shouldn’t be too hard to apply the same fix (creating a symbolic link to route the temp directory to within the data directory) to the Dockerfile there and build an image…

EDIT: … and this is where even I forget sometimes that you can actually just do your own Dockerfile with a FROM kcollins/ignition:7.8.5 at the top and add in the symlink that you need… That is the joy of Docker after all, being able to just build atop a pre-existing image! :smiley:

EDIT 2: Create a folder, place the contents below into a file called Dockerfile:

FROM kcollins/ignition:7.8.5

USER root

RUN mv /var/lib/ignition/temp /var/lib/ignition/data/ && \
    ln -s /var/lib/ignition/data/temp /var/lib/ignition/temp

USER ${IGNITION_INSTALL_USERNAME}

… and then build it with docker build . -t kcollins/ignition:7.8.5a. Now containers (from your new kcollins/ignition:7.8.5a image) you start attached to a data volume will route temp assets to a folder located within that data volume and workaround this bug (just like the newer images). Keep in mind that if you’ve got existing data volumes, you’ll need to mkdir /var/lib/ignition/data/temp; the entrypoint script in my other images help “bring forward” older images with the workaround automatically. Not a big deal though if you’re able to just start with a fresh volume and go from there.

1 Like
7

Thanks Kevin! I actually pulled some of your Dockerfiles from git, made a couple mods, and built a 7.8.3 container.

1 Like