Lost all my work in a docker instance

mcarritt1 · March 10, 2021, 3:12pm

I installed and ran a docker container (using kitematic on Ubuntu) with ignition 8.1, and worked for most of the day on a project. I enabled the “always restart” option for the container, and restarted it and it brought me to the “select which version of ignition to install” page. Is everything gone?

kcollins1 · March 10, 2021, 3:36pm

Sorry to hear this.. There are a few fundamentals to understand when working with Docker. When you launch a container, an isolated filesystem is created (based on the image) and changes made live only within the container. As long as the container is not removed, your data will still be within that container. Setting "always restart" shouldn't cause the container to be removed, but rather restart automatically upon failure.

I'll also mention that it is typical that containers are treated as somewhat ephemeral in nature and data persistence for stateful containers (like an Ignition Gateway) is done through the use of volumes. I always recommend using named volumes with your containers if you're doing anything but very temporary work.

First things first, can you share the output of:

docker ps -a

This will print out a listing of all containers, including stopped ones
docker inspect <container_name>

Use the output from the container listing above to determine which container name should be used for <container_name>.

mcarritt1 · March 10, 2021, 3:49pm

docker inspect 091f47c25b78
[
{
"Id": "091f47c25b78aac194ca3cb00498c07142f3875eba6e6bb5dfa823d9f504a30e",
"Created": "2021-03-10T14:59:26.685170436Z",
"Path": "./docker-entrypoint.sh",
"Args": ,
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 243348,
"ExitCode": 0,
"Error": "",
"StartedAt": "2021-03-10T15:03:33.828275667Z",
"FinishedAt": "2021-03-10T15:02:28.01941004Z",
"Health": {
"Status": "healthy",
"FailingStreak": 0,
"Log": [
{
"Start": "2021-03-10T16:46:58.827255077+01:00",
"End": "2021-03-10T16:47:02.277058395+01:00",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2021-03-10T16:47:12.280975486+01:00",
"End": "2021-03-10T16:47:15.732238344+01:00",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2021-03-10T16:47:25.735808104+01:00",
"End": "2021-03-10T16:47:29.215416294+01:00",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2021-03-10T16:47:39.219164679+01:00",
"End": "2021-03-10T16:47:42.671150692+01:00",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2021-03-10T16:47:52.673858277+01:00",
"End": "2021-03-10T16:47:56.096972823+01:00",
"ExitCode": 0,
"Output": ""
}
]
}
},
"Image": "sha256:bc1c518f2958c5a3e0d1f42f9a200329254c38a24123cefc5ffdd957d7e6e182",
"ResolvConfPath": "/var/lib/docker/containers/091f47c25b78aac194ca3cb00498c07142f3875eba6e6bb5dfa823d9f504a30e/resolv.conf",
"HostnamePath": "/var/lib/docker/containers/091f47c25b78aac194ca3cb00498c07142f3875eba6e6bb5dfa823d9f504a30e/hostname",
"HostsPath": "/var/lib/docker/containers/091f47c25b78aac194ca3cb00498c07142f3875eba6e6bb5dfa823d9f504a30e/hosts",
"LogPath": "/var/lib/docker/containers/091f47c25b78aac194ca3cb00498c07142f3875eba6e6bb5dfa823d9f504a30e/091f47c25b78aac194ca3cb00498c07142f3875eba6e6bb5dfa823d9f504a30e-json.log",
"Name": "/ignition-pathfinder2e",
"RestartCount": 0,
"Driver": "overlay2",
"Platform": "linux",
"MountLabel": "",
"ProcessLabel": "",
"AppArmorProfile": "docker-default",
"ExecIDs": null,
"HostConfig": {
"Binds": null,
"ContainerIDFile": "",
"LogConfig": {
"Type": "json-file",
"Config": {}
},
"NetworkMode": "default",
"PortBindings": {
"62541/tcp": [
{
"HostIp": "",
"HostPort": "32768"
}
],
"8000/tcp": [
{
"HostIp": "",
"HostPort": "32774"
}
],
"8043/tcp": [
{
"HostIp": "",
"HostPort": "32773"
}
],
"8060/tcp": [
{
"HostIp": "",
"HostPort": "32771"
}
],
"8088/tcp": [
{
"HostIp": "",
"HostPort": "32700"
}
]
},
"RestartPolicy": {
"Name": "no",
"MaximumRetryCount": 0
},
"AutoRemove": false,
"VolumeDriver": "",
"VolumesFrom": null,
"CapAdd": null,
"CapDrop": null,
"Capabilities": null,
"Dns": ,
"DnsOptions": ,
"DnsSearch": ,
"ExtraHosts": null,
"GroupAdd": null,
"IpcMode": "private",
"Cgroup": "",
"Links": null,
"OomScoreAdj": 0,
"PidMode": "",
"Privileged": false,
"PublishAllPorts": true,
"ReadonlyRootfs": false,
"SecurityOpt": null,
"UTSMode": "",
"UsernsMode": "",
"ShmSize": 67108864,
"Runtime": "runc",
"ConsoleSize": [
0,
0
],
"Isolation": "",
"CpuShares": 0,
"Memory": 0,
"NanoCpus": 0,
"CgroupParent": "",
"BlkioWeight": 0,
"BlkioWeightDevice": null,
"BlkioDeviceReadBps": null,
"BlkioDeviceWriteBps": null,
"BlkioDeviceReadIOps": null,
"BlkioDeviceWriteIOps": null,
"CpuPeriod": 0,
"CpuQuota": 0,
"CpuRealtimePeriod": 0,
"CpuRealtimeRuntime": 0,
"CpusetCpus": "",
"CpusetMems": "",
"Devices": null,
"DeviceCgroupRules": null,
"DeviceRequests": null,
"KernelMemory": 0,
"KernelMemoryTCP": 0,
"MemoryReservation": 0,
"MemorySwap": 0,
"MemorySwappiness": null,
"OomKillDisable": false,
"PidsLimit": null,
"Ulimits": null,
"CpuCount": 0,
"CpuPercent": 0,
"IOMaximumIOps": 0,
"IOMaximumBandwidth": 0,
"MaskedPaths": [
"/proc/asound",
"/proc/acpi",
"/proc/kcore",
"/proc/keys",
"/proc/latency_stats",
"/proc/timer_list",
"/proc/timer_stats",
"/proc/sched_debug",
"/proc/scsi",
"/sys/firmware"
],
"ReadonlyPaths": [
"/proc/bus",
"/proc/fs",
"/proc/irq",
"/proc/sys",
"/proc/sysrq-trigger"
]
},
"GraphDriver": {
"Data": {
"LowerDir": "/var/lib/docker/overlay2/e842b356f9aa52a03774d0756f76142efb49f37c053fd2a37862087823ffe276-init/diff:/var/lib/docker/overlay2/03e46375df70de59f9ccbc5fe01fdc482abc6452cea4e29e1b0dbfc757f284bd/diff:/var/lib/docker/overlay2/04d8d89b6782cfe96cbd325a0f446707b79ba4a6b15d8dba9daee92f2eae0b6d/diff:/var/lib/docker/overlay2/55589d3bd2da5b8edcf1a6d5d2272bbf1dbfd42e4d8db4b69a6df66942c15616/diff:/var/lib/docker/overlay2/e2b0c8f882d3b63541fd94a2cb4ec23e636a270b5711fcacabcb09afeedc93ce/diff:/var/lib/docker/overlay2/8a7c05a0fafb8fba35760bd1df1a5c4f8834ea47d87d83f871bab6a71664e318/diff:/var/lib/docker/overlay2/34393b809fa96d4c71a2b500f9ef47c095cd469d3d38cc7ed774bfac9b6d9f5a/diff:/var/lib/docker/overlay2/3cf190b8ef03d1062f5a3cba6f5d2a6bfcd53315391799f95398ba0335db4695/diff:/var/lib/docker/overlay2/db6e6fd7d84254d57cefdc9ea707789dab13578e4a649a14311f65e81b89dc21/diff:/var/lib/docker/overlay2/1054c16941520d40311b2ec7df11a10b403c0aade6df060fff0067d3f25c278e/diff:/var/lib/docker/overlay2/917174a5bc3a3f135d62e459ebca49659502dc2ac1362a3216c7d9d3a8eea6be/diff:/var/lib/docker/overlay2/c0b148c421222f7264e8c78234f3e4302ad1866867d5bfcb789a12529c0a0eec/diff:/var/lib/docker/overlay2/6972ff8d84979e00095f7c4bc76bf91b6b2378af2387e93aad8f94559aab6124/diff",
"MergedDir": "/var/lib/docker/overlay2/e842b356f9aa52a03774d0756f76142efb49f37c053fd2a37862087823ffe276/merged",
"UpperDir": "/var/lib/docker/overlay2/e842b356f9aa52a03774d0756f76142efb49f37c053fd2a37862087823ffe276/diff",
"WorkDir": "/var/lib/docker/overlay2/e842b356f9aa52a03774d0756f76142efb49f37c053fd2a37862087823ffe276/work"
},
"Name": "overlay2"
},
"Mounts": ,
"Config": {
"Hostname": "3b8368131490",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"ExposedPorts": {
"62541/tcp": {},
"8000/tcp": {},
"8043/tcp": {},
"8060/tcp": {},
"8088/tcp": {}
},
"Tty": true,
"OpenStdin": true,
"StdinOnce": false,
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"LANG=en_US.UTF-8",
"LANGUAGE=en_US",
"LC_ALL=en_US.UTF-8"
],
"Cmd": null,
"Healthcheck": {
"Test": [
"CMD-SHELL",
"test $(./gwcmd.sh --info | grep -c 'RUNNING') -ge 1"
],
"Interval": 10000000000,
"Timeout": 10000000000,
"Retries": 10
},
"Image": "inductiveautomation/ignition:latest",
"Volumes": null,
"WorkingDir": "/usr/local/bin/ignition",
"Entrypoint": [
"./docker-entrypoint.sh"
],
"OnBuild": null,
"Labels": {
"io.portainer.accesscontrol.teams": "Development/QA"
}
},
"NetworkSettings": {
"Bridge": "",
"SandboxID": "6b9e3061d5c303a7e74b17b76655a9e5d2c2e16b30fac6904a26e787204d212b",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": {
"62541/tcp": [
{
"HostIp": "0.0.0.0",
"HostPort": "32768"
}
],
"8000/tcp": [
{
"HostIp": "0.0.0.0",
"HostPort": "32774"
}
],
"8043/tcp": [
{
"HostIp": "0.0.0.0",
"HostPort": "32773"
}
],
"8060/tcp": [
{
"HostIp": "0.0.0.0",
"HostPort": "32771"
}
],
"8088/tcp": [
{
"HostIp": "0.0.0.0",
"HostPort": "32700"
}
]
},
"SandboxKey": "/var/run/docker/netns/6b9e3061d5c3",
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "cd0f846e2f549f83675198f9f9fadb24672cf4b222e2dac8b84aea040dba0fa3",
"Gateway": "172.17.0.1",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "172.17.0.4",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"MacAddress": "02:42:ac:11:00:04",
"Networks": {
"bridge": {
"IPAMConfig": null,
"Links": null,
"Aliases": ,
"NetworkID": "1ed67a2e0a52c0073e37682d66c90ea7abc635cbc2c405a26807bb2b0902be92",
"EndpointID": "cd0f846e2f549f83675198f9f9fadb24672cf4b222e2dac8b84aea040dba0fa3",
"Gateway": "172.17.0.1",
"IPAddress": "172.17.0.4",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "02:42:ac:11:00:04",
"DriverOpts": null
}
}
}
}
]

kcollins1 · March 10, 2021, 4:07pm

Okay, I see here that this container doesn’t seem to have RestartPolicy set the way the original container you described, I see in your inspect above:

“RestartPolicy”: {
    “Name”: “no”,
    “MaximumRetryCount”: 0
},

… where I’d expect to see something like "Name": "always", like:

"RestartPolicy": {
    "Name": "always",
    "MaximumRetryCount": 0
},

Another thing that I think is key to understand about containers is that you cannot change the configuration of a container, it is static. So if you make a config change in Kitematic and “Save”, it will happily remove your previous container and create a new one with the updated configuration.

I think the GUI tools for Docker can be very helpful, but I typically recommend them only after you’ve gotten the basics of the Docker CLI down. Once you know how things work and are ready to possibly accelerate your workflows, then get the GUI helpers out since you’ll understand what they’re making faster for you.

Unfortunately, in this particular case, I think you’ve probably lost the state of that original container.

mcarritt1 · March 10, 2021, 4:15pm

bummer. thanks for the quick and insightful response though.

kcollins1 · March 10, 2021, 4:23pm

You’re welcome, sorry again for the lost work. I hadn’t touched Kitematic for quite some time (spooled it up here to help). It seems very limited (I couldn’t even find a way to specify the launch configuration for a new container, ). Honestly I’d look into Portainer if you’re wanting a good GUI for managing your Docker host[s], though my recommendation for getting comfortable with the Docker CLI and docker-compose still stands…

Good luck and happy coding (it’s easier the second time through, right? ).

mcarritt1 · March 10, 2021, 4:25pm

yeah I think they may have stopped working on it as well. I do have portainer set up, i’ll be sure to just use that in the future instead.

jimender2 · March 12, 2021, 3:21am

Portainer is a really really helpful feature. It has some shortcomings (like only supporting compose 2.something instead of 3) but it is a good gui.

kcollins1 · March 12, 2021, 3:41pm

Thankfully, as of Portainer CE 2.1.1, it does support newer Compose versions!

jimender2 · March 12, 2021, 11:03pm

I think I updated my version of Portainer the day before they updated it.

ken.maze · May 29, 2021, 5:18am

So it seems like it would be a good idea to use docker volume inspect to findo ut where your volume files are being stored and back them up occasionally, especially if you’;re fooling around with your container. You could restore it easily it seems into the new container by finding out where its volume is, and putting the file tree back there.

Or is it read-only when looked at from the host context?

jimender2 · June 1, 2021, 10:49am

@ken.maze The docker volumes (at least on windows) are stored in C:\Users\Public\Documents\Hyper-V\Virtual hard disks
You would probably take a backup of whatever disk it is. The one thing that I would want to point out is that at some point it would probably just be easier to use a bind mount (think windows folder shared directly with docker) and just backup the data that way. The downside to this (only exists on windows version of docker) is that permissions may get screwy (I normally have to run the docker container as a root user) and bind mounts can be pretty slow compared to the files in the volumes. But for dev work, the speed is pretty fine.

kcollins1 · June 1, 2021, 2:13pm

The benefits of volumes discussed above are definitely clear. However, it is usually a good idea to have proper backup strategies in place for your development. The backup method I prefer for most scenarios is utilize a bind-mount into the container as a target for the scheduled gateway backups within Ignition. By providing a bind-mount of -v /path/to/backups:/backups combined with configuring scheduled backups (Config->System->Gateway Settings in the Gateway webpage), you can easily have those backups piped back to your host system.

If you need to backup the volume itself, usually you can just use another disposable container using the --volumes-from argument to enable tar’ing up the volume contents. It is a bit of an advanced exercise, and I’d only really recommend this technique if you need to move volumes from one system to another. Thankfully, Ignition has a great scheduled backup function built-in, and it helps guarantee write consistency that might be problematic with the volume export (if the container is still running).

Hope this helps!

Kevin C.

ken.maze · June 1, 2021, 5:40pm

Thanks everyone, this is a big help! That’s a great idea about a separate bind mount for backups