# Docker all gateways notify "disk full"

**URL:** <https://forum.inductiveautomation.com/t/docker-all-gateways-notify-disk-full/54351>\
**Category:** Ignition\
**Created:** [December 21, 2021, 10:38am UTC](https://forum.inductiveautomation.com/t/docker-all-gateways-notify-disk-full/54351 "2021-12-21T10:38:45Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![abiesbiogas](https://avatars.discourse-cdn.com/v4/letter/a/919ad9/32.png) [@abiesbiogas](https://forum.inductiveautomation.com/u/abiesbiogas)\
**Post date:** [December 21, 2021, 10:38am UTC](https://forum.inductiveautomation.com/t/docker-all-gateways-notify-disk-full/54351/1 "2021-12-21T10:38:45Z")

</div>

Hi all,

we’ve succesfully used Ignition in docker containers for the last couple of years but now we have to front this issue:  
suddenly all containers have become corrupted and after recreating (just a few) all of them show the message **Disk Full The disk that Ignition is installed on is nearly full.** in the status page.

We’ve been working with 20 or more container instantiated and without a problem, now we use just 2 or 3 and every 2 or 3 days the ignition containers (and only them!) explode.

We use compose files like this:

```auto
version: '3.1'
services:
  gateway:
    container_name: "Ignition8221"
    restart: unless-stopped
    dns: 192.168.10.1
    image: kcollins/ignition:7.9.18 # You can change `latest` to a specific version, e.g. `8.0.5`
    ports:
      - 8221:8088
    stop_grace_period: 30s
    volumes:
      - .\gateway_backup.gwbk:/restore.gwbk
      - data8221:/var/lib/ignition
      - .\log:/var/log/ignition
    environment:
      GATEWAY_SYSTEM_NAME: "Ignition@8221"
      GATEWAY_ADMIN_PASSWORD: supersecretP@ssword
      GATEWAY_NETWORK_AUTOACCEPT_DELAY: 60  

volumes:
  data8221:
    external: True

networks:
  default:
    external: True
    name: ignition

```

The disk is actually empty and we use about half of the ram available (64G)… we can’t figure out what’s going on or where to look for the proper parameter to set in the docker configuration.  
This is the docker engine setting json:

```auto
{
  "builder": {
    "gc": {
      "defaultKeepStorage": "20GB",
      "enabled": true
    }
  },
  "debug": false,
  "experimental": false,
  "features": {
    "buildkit": true
  },
  "insecure-registries": [],
  "registry-mirrors": []
}

```

thi is what docker says about disk usage:

```auto
> docker system df
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 8 4 3.881GB 2.049GB (52%)
Containers 6 6 312.2MB 0B (0%)
Local Volumes 5 5 3.478GB 0B (0%)
Build Cache 10 0 0B 0B

```

I did docker system prune -a and the message disappeared, but when I go to the status page I see this:

```auto
Process Id 1
Operating System Linux | amd64
Java Version 1.8.0_262-b10
Local Time 11:27:20 AM
Available Disk Space 3gb / 251gb
Detected NICs 172.26.0.2

```

Does it mean that the gateways occupy 248gb of my disk (which actually has 529gb free)?

Have anyone any suggestions?

Thank you in advance.

---

<div class="post-metadata">

**Author:** ![kcollins](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/kcollins/32/19646_2.png) [@kcollins](https://forum.inductiveautomation.com/u/kcollins)\
**Post date:** [December 21, 2021, 3:25pm UTC](https://forum.inductiveautomation.com/t/docker-all-gateways-notify-disk-full/54351/2 "2021-12-21T15:25:09Z")

</div>

It will be helpful to know your environment (Host OS, Docker versions, etc), but I’m going to make some guesses based on context clues. It looks like your volume mount syntax hint at a Windows host system. It also looks like the Docker VM is pegged at the 250GB max size (it will never get bigger than this under Docker Desktop).

My guess as to where your disk space has gone is container logs. Container logs will continue to grow unbounded in their default configuration. Unfortunately, their disk space usage isn’t well reported. Take a look at the guidance below (if you’re NOT using Docker Desktop, the initial step for how to access the under-the-hood Linux VM may not apply):

[https://asciinema.org/a/llcaXmJKupPF150BjA2BRAhVF](https://asciinema.org/a/llcaXmJKupPF150BjA2BRAhVF)

Ultimately, consider adding some supplemental logging configuration to your Compose definitions, like can be found here: [Compose file version 3 reference | Docker Documentation](https://docs.docker.com/compose/compose-file/compose-file-v3/#logging)

 ![2021-12-21 at 9.20 AM](https://us1.discourse-cdn.com/inductiveautomation/original/3X/2/c/2c988345d681de674dd1d0bc348871017c2d711f.png)

---

<div class="post-metadata">

**Author:** ![kcollins](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/kcollins/32/19646_2.png) [@kcollins](https://forum.inductiveautomation.com/u/kcollins)\
**Post date:** [December 21, 2021, 5:17pm UTC](https://forum.inductiveautomation.com/t/docker-all-gateways-notify-disk-full/54351/3 "2021-12-21T17:17:36Z")

</div>

Another tip that I’ll mention here for posterity is that you can also [modify the default logging configuration](https://docs.docker.com/config/containers/logging/configure/#configure-the-default-logging-driver) for all containers via the `daemon.json` config. With Docker Desktop, you can reach this configuration through the UI:

 ![2021-12-21 at 11.14 AM](https://us1.discourse-cdn.com/inductiveautomation/original/3X/1/0/10bd2c93c31f21f3746d319ad67748bae3dd4aa4.png)

If you do it at this level, you won’t have to remember to apply this at each container run configuration.

---

<div class="post-metadata">

**Author:** ![abiesbiogas](https://avatars.discourse-cdn.com/v4/letter/a/919ad9/32.png) [@abiesbiogas](https://forum.inductiveautomation.com/u/abiesbiogas)\
**Post date:** [December 22, 2021, 10:05am UTC](https://forum.inductiveautomation.com/t/docker-all-gateways-notify-disk-full/54351/4 "2021-12-22T10:05:14Z")

</div>

> [@kcollins](#):
>
> It will be helpful to know your environment (Host OS, Docker versions, etc)

😅 of course!  
we are running docker desktop 4.3.1 (72247), engine 20.10.11, compose v2.2.1  
on windows 10 Pro Workstation version 20H2 Os build 19042.804  
we rely on WSL2

I used Justin Cormack guidance and I found a very large log file (240GB) but I couldn't remove it since the file system is read-only so, from explorer, I removed and recreated the file, then from J.C. container I reset the permissions on the file and restarted docker

Eventually I found [these instructions](https://docs.microsoft.com/en-us/windows/wsl/vhd-size) and I was able to resize the disk and let docker know I removed the 250GB limit: now the **Availabe Disk Space** in Ignition Status page is 461gb / 491gb.

Thank you @kcollins 😀