I have an Ignition deployment running in Azure Kubernetes Service. The deployment worked from an Ignition application point of view. As part of the standard operating procedures, there is a requirement to perform backups/restores. I have a couple questions about this:
Is the Backup/Restore process expected to function normally while running in kubernetes?
Do you have any guidance on how to make sure the image/deployment is kept up to date so that no data is lost in the event of pod/pvc corruption?
I attempted a backup from an already running gateway (VM Based) and tried to perform the restore in the Kubernetes environment and it caused the pod to go into CrashLoopBackoff and continually restart (The application would restart so we could login; then it would crash)
Backup and Restore should work like in other containerized run contexts. A typical strategy would be to mount a PVC (could be an Azure Files endpoint) into the pod and configure scheduled backups within Ignition to target it.
If you're going into CrashLoopBackoff after a restore, the first step would be to get the events from the pod, perhaps logs also. There might be something incorrect in your K8s resource configurations. Are you using StatefulSet for the Ignition gateway?
Sorry for the delay - I am attaching the pod logs. Also, I'm confirming that we are not using a StatefulSet for the Ignition Gateway ignition_pod_log.txt (20.9 KB)
By the look of that log, it appears to me that you're launching another pod against the same underlying PV, causing contention (with corruption a likely result). Ignition never expects another Ignition to be trying to interact with the same files. While there is now ReadWriteOncePod access mode (GA in K8s 1.29), the best way to avoid this is to leverage the guarantees afforded by StatefulSet. If you've not taken a peek at my K8s examples Github repo, it might provide some useful insights into some patterns for setting up your target architecture: