Kubernetes PersistentVolume - Backup /data

[hodgins.e]

I was trying to see how Ignition is in the Kubernetes world. I’m simply just trying to backup the path /usr/local/bin/ignition/data. But I can not get it to work. If I remove the mount path the container will at least run. If I leave it as is the pod runs but Ignition never starts up. Wondering if there’s a command that needs to be run or something?

Here’s my deployment:

apiVersion: apps/v1

kind: Deployment

metadata:

    name: ignition-depl

spec:

    replicas: 1

    selector:

        matchLabels:

            component: ignition

    template:

        metadata:

            labels:

                component: ignition

        spec:

            volumes:

                - name: ignition-storage

                  persistentVolumeClaim:

                    claimName: ignition-pv-claim

            containers:

                - name: ignition

                  image: inductiveautomation/ignition:nightly 

                  ports:  

                    - containerPort: 8088

                  volumeMounts:

                    - name: ignition-storage                      

                      mountPath: /usr/local/bin/ignition/data

And the Persistent volume claim

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
    name: ignition-pv-claim
spec:
    accessModes:
        - ReadWriteOnce
    resources:
        requests:
            storage: 2Gi

LOGS

Creating init.properties file
Creating gateway.xml
cp: cannot stat 'data/gateway.xml_clean': No such file or directory
Adding gateway.publicAddress.autoDetect=false to gateway.xml
failed to load external entity "data/gateway.xml"
Adding gateway.publicAddress.address=0.0.0.0 to gateway.xml
failed to load external entity "data/gateway.xml"
Adding gateway.publicAddress.httpPort=8088 to gateway.xml
failed to load external entity "data/gateway.xml"
Adding gateway.publicAddress.httpsPort=8043 to gateway.xml
failed to load external entity "data/gateway.xml"
Enabling arg pass through
Disabling PID file check
Starting Ignition gateway
Following wrapper.log

On container (nothing in the data folder really)

[kcollins1]

Right now I think this will likely not work [on the official image at this time], for some of the reasons I describe in my unofficial image docs at ALTERNATIVE: How to persist Gateway data. You might have a go at subbing for kcollins/ignition:8.1 and pointing your volume target at /data. Let me know if that works out for ya…

[pturmel]

Nice badge on your avatar, @kcollins1 !

[hodgins.e]

well that’s awesome, that did work out for me. Had some issues initially but once I restarted my local environment it worked. Thanks @kcollins1 !

[mazhar]

@hodgins.e Do you mind sharing your k8 deployment def with the workaround. We are facing the same issue.

[hodgins.e]

hey @mazhar no problem. Just use my deployment that I included in the my question but change the containers config to this:

            containers:
                - name: ignition
                  image: kcollins/ignition:8.1
                  ports:  
                    - containerPort: 8088
                  volumeMounts:
                    - name: ignition-storage                      
                      mountPath: /data     

I am also using a nginx-ingress controller (https://kubernetes.github.io/ingress-nginx/deploy/)

my clusterIp service for Ignition is this:

apiVersion: v1
kind: Service
metadata:
    name: ignition-cluster-ip-service
spec:
    type: ClusterIP
    selector:
        component: ignition
    ports:
        - port: 8088
          targetPort: 8088

Hopefully that helps!

[mazhar]

Thanks for sharing. Official image worked for me too with this workaround.

[Kevin.Collins]

@mazhar, do keep in mind that with the official image there is no special handling for mounts at /data and thusly, your gateway state will not be persisted.

[mazhar]

@Kevin.Collins Understood. ‘kcollins/ignition’ apparently needs initialization unlike official image, @hodgins.e can you share how you accomplished the initialization?

[mazhar]

@hodgins.e

kcollins/ignition:8.1

never mind, "kcollins/ignition:8.1" worked too, thanks for the help

[kcollins1]

Just augmenting this thread for those that find it (since I had a couple questions on this topic, recently)… In K8s, you can leverage initContainers to seed the data volume prior to initial launch of the Gateway container [in the pod].

Here is an example deployment that leverages an init container to seed the data volume with files from the image:

# Ignition K8s Example Deployment
---
# Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ignition-deployment
spec:
  selector:
    matchLabels:
      app: ignition-app
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: ignition-app
    spec:
      initContainers:
      - name: seed-volume
        image: inductiveautomation/ignition:8.1.17
        resources:
          limits:
            memory: "256Mi"
            cpu: "1000m"
        command:
        - sh
        - -c
        - >
          if [ ! -f /data/.ignition-seed-complete ]; then
            touch /data/.ignition-seed-complete ;
            cp -dpR /usr/local/bin/ignition/data/* /data/ ;
          fi
        volumeMounts:
        - mountPath: /data
          name: ignition-data
      containers:
      - name: ignition
        image: inductiveautomation/ignition:8.1.17
        resources:
          limits:
            memory: "2048Mi"
            cpu: "1000m"
        args:
        - -n
        - Ignition-k8s
        - -m
        - "2048"
        env:
        - name: ACCEPT_IGNITION_EULA
          value: "Y"
        - name: GATEWAY_ADMIN_PASSWORD_FILE
          value: /run/secrets/gateway-admin/password
        - name: IGNITION_EDITION
          value: standard
        ports:
        - name: ignition-web
          containerPort: 8088
        volumeMounts:
        - mountPath: /usr/local/bin/ignition/data
          name: ignition-data
        - mountPath: /run/secrets/gateway-admin
          name: gateway-admin
          readOnly: true
        readinessProbe:
          exec:
            command:
            - health-check.sh
            - -t
            - "3"
          initialDelaySeconds: 60
          periodSeconds: 10
          failureThreshold: 10
          timeoutSeconds: 3
      volumes:
      - name: ignition-data
        persistentVolumeClaim:
          claimName: ignition-pv-claim
      - name: gateway-admin
        secret:
          secretName: gateway-admin

---
# Secret (which you wouldn't normally just have hanging out here in your deployment yaml)
apiVersion: v1
kind: Secret
metadata:
  name: gateway-admin
type: Opaque
data:
  password: UEBzc3cwcmQ=
---
# Service
apiVersion: v1
kind: Service
metadata: 
  name: ignition-lb
spec:
  selector:
    app: ignition-app
  type: LoadBalancer
  ports:
    - name: http
      port: 8088
      targetPort: 8088
---
# PersistentVolume
apiVersion: v1
kind: PersistentVolume
metadata:
  name: ignition-pv-volume
  labels:
    type: local
spec:
  storageClassName: manual
  capacity:
    storage: 10Gi
  hostPath:
    path: "${PWD}/ignition-data"
  accessModes:  # ideally we want ReadWriteOncePod, but still in alpha?
    - ReadWriteOnce
---
# PersistentVolumeClaim
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ignition-pv-claim
spec:
  storageClassName: manual
  resources:
    requests:
      storage: 3Gi
  accessModes:
    - ReadWriteOnce

This is designed to be run from a folder on your host system with a fresh (and empty, initially) ignition-data subfolder. I use envsubst to substitute the current working directory. So you could deploy this with, for example:

$ kubectl create ns k8s-testing
$ kubectl config set-context --current --namespace=k8s-testing
$ envsubst < ignition-app.yaml | kubectl apply -f -

Keep in mind that this is just an example deployment that you can test locally, not a production-ready configuration.

Here’s a little video to check out with this too, in the event that moving pictures>pictures>words.
k8s-testing-2.mp4

[Sanat_Pattanaik]

Hello,
I tried to follow this approach. But /data folder gets created without write permission

So the touch and copy are failing.

[kcollins1]

I'll mention that at ICC 2023 this year, I gave a presentation on Ignition on K8s with some useful (and updated!) examples. See those here: GitHub - thirdgen88/icc2023-ignition-k8s: ICC 2023 Session - Deployment Patterns for Ignition on Kubernetes

@Sanat_Pattanaik

I tried to follow this approach. But /data folder gets created without write permission

What kind of backing storage are you using for your PVC's? Perhaps share the output of the command below for your cluster:

kubectl get storageclass

[Sanat_Pattanaik]

kubernetes.io/aws-ebs

[Sanat_Pattanaik]

[Sanat_Pattanaik]

Thank you for your response. After looking thru your github, I added the security context and it is working now.

[kcollins1]

Okay.. You shouldn't have any issue with EBS volumes. Feel free to share your YAML, but ultimately, I'd recommend looking at that GH repo I referenced above as a better baseline.

EDIT: wrote the above before I saw your most recent reply.. Glad you're up and running!

[Pratik_Kansagara]

@kcollins1

DIT: wrote the above before I

Do you have any example to deploy it in AKS with azure disk as a presist volume.

[kcollins1]

I don't have any AKS examples prepared, but you're on the right track targeting Azure Disk as the storage class for your PVC.