Docker all gateways notify "disk full"

Ignition
1

Hi all,

we’ve succesfully used Ignition in docker containers for the last couple of years but now we have to front this issue:
suddenly all containers have become corrupted and after recreating (just a few) all of them show the message Disk Full The disk that Ignition is installed on is nearly full. in the status page.

We’ve been working with 20 or more container instantiated and without a problem, now we use just 2 or 3 and every 2 or 3 days the ignition containers (and only them!) explode.

We use compose files like this:

version: '3.1'
services:
  gateway:
    container_name: "Ignition8221"
    restart: unless-stopped
    dns: 192.168.10.1
    image: kcollins/ignition:7.9.18  # You can change `latest` to a specific version, e.g. `8.0.5`
    ports:
      - 8221:8088
    stop_grace_period: 30s
    volumes:
      - .\gateway_backup.gwbk:/restore.gwbk
      - data8221:/var/lib/ignition
      - .\log:/var/log/ignition
    environment:
      GATEWAY_SYSTEM_NAME: "Ignition@8221"
      GATEWAY_ADMIN_PASSWORD: supersecretP@ssword
      GATEWAY_NETWORK_AUTOACCEPT_DELAY: 60  

volumes:
  data8221:
    external: True

networks:
  default:
    external: True
    name: ignition

The disk is actually empty and we use about half of the ram available (64G)… we can’t figure out what’s going on or where to look for the proper parameter to set in the docker configuration.
This is the docker engine setting json:

{
  "builder": {
    "gc": {
      "defaultKeepStorage": "20GB",
      "enabled": true
    }
  },
  "debug": false,
  "experimental": false,
  "features": {
    "buildkit": true
  },
  "insecure-registries": [],
  "registry-mirrors": []
}

thi is what docker says about disk usage:

> docker system df
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          8         4         3.881GB   2.049GB (52%)
Containers      6         6         312.2MB   0B (0%)
Local Volumes   5         5         3.478GB   0B (0%)
Build Cache     10        0         0B        0B

I did docker system prune -a and the message disappeared, but when I go to the status page I see this:

Process Id	           1
Operating System	   Linux | amd64
Java Version	       1.8.0_262-b10
Local Time	           11:27:20 AM
Available Disk Space   3gb / 251gb
Detected NICs	       172.26.0.2

Does it mean that the gateways occupy 248gb of my disk (which actually has 529gb free)?

Have anyone any suggestions?

Thank you in advance.

2

It will be helpful to know your environment (Host OS, Docker versions, etc), but I’m going to make some guesses based on context clues. It looks like your volume mount syntax hint at a Windows host system. It also looks like the Docker VM is pegged at the 250GB max size (it will never get bigger than this under Docker Desktop).

My guess as to where your disk space has gone is container logs. Container logs will continue to grow unbounded in their default configuration. Unfortunately, their disk space usage isn’t well reported. Take a look at the guidance below (if you’re NOT using Docker Desktop, the initial step for how to access the under-the-hood Linux VM may not apply):

https://asciinema.org/a/llcaXmJKupPF150BjA2BRAhVF

Ultimately, consider adding some supplemental logging configuration to your Compose definitions, like can be found here: Compose file version 3 reference | Docker Documentation

2021-12-21 at 9.20 AM
2021-12-21 at 9.20 AM989×261 19 KB

3 Likes
3

Another tip that I’ll mention here for posterity is that you can also modify the default logging configuration for all containers via the daemon.json config. With Docker Desktop, you can reach this configuration through the UI:

2021-12-21 at 11.14 AM
2021-12-21 at 11.14 AM1121×373 40.9 KB

If you do it at this level, you won’t have to remember to apply this at each container run configuration.

3 Likes
4

:sweat_smile: of course!
we are running docker desktop 4.3.1 (72247), engine 20.10.11, compose v2.2.1
on windows 10 Pro Workstation version 20H2 Os build 19042.804
we rely on WSL2

I used Justin Cormack guidance and I found a very large log file (240GB) but I couldn't remove it since the file system is read-only so, from explorer, I removed and recreated the file, then from J.C. container I reset the permissions on the file and restarted docker

Eventually I found these instructions and I was able to resize the disk and let docker know I removed the 250GB limit: now the Availabe Disk Space in Ignition Status page is 461gb / 491gb.

Thank you @kcollins1 :grinning: