# Ignition Docker: Disk Full The disk that Ignition is installed on is nearly full **URL:** \ **Category:** Ignition\ **Tags:** docker, ignition81\ **Created:** [April 9, 2024, 3:49pm UTC](https://forum.inductiveautomation.com/t/ignition-docker-disk-full-the-disk-that-ignition-is-installed-on-is-nearly-full/88585 "2024-04-09T15:49:35Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![drewdin](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/drewdin/32/3363_2.png) [@drewdin](https://forum.inductiveautomation.com/u/drewdin)\ **Post date:** [April 9, 2024, 3:49pm UTC](https://forum.inductiveautomation.com/t/ignition-docker-disk-full-the-disk-that-ignition-is-installed-on-is-nearly-full/88585/1 "2024-04-09T15:49:35Z")
I followed the post below and I still cant clear my hard disk usage. > [@Docker all gateways notify "disk full"](https://forum.inductiveautomation.com/t/docker-all-gateways-notify-disk-full/54351): > > Hi all, we’ve succesfully used Ignition in docker containers for the last couple of years but now we have to front this issue: suddenly all containers have become corrupted and after recreating (just a few) all of them show the message Disk Full The disk that Ignition is installed on is nearly full. in the status page. We’ve been working with 20 or more container instantiated and without a problem, now we use just 2 or 3 and every 2 or 3 days the ignition containers (and only them!) explode. … My Docker instance says the disk is full, using all 15gb of hard drive space. I followed the post above and deleted the log file and rebooted docker but I'm still getting a disk full error. I'm running HighByte, Ignition, and mssql, all of the log files are tiny for each container. I am running Proxmox and have a dedicated ubuntu server docker vm. the overall HD size if 35gb. Any pointers on where to start looking? thanks! ---
**Author:** ![drewdin](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/drewdin/32/3363_2.png) [@drewdin](https://forum.inductiveautomation.com/u/drewdin)\ **Post date:** [April 10, 2024, 1:20pm UTC](https://forum.inductiveautomation.com/t/ignition-docker-disk-full-the-disk-that-ignition-is-installed-on-is-nearly-full/88585/2 "2024-04-10T13:20:18Z")
I was able to figure it out for anyone looking, I installed portainer and found about 14 1gb volumes that were not attached to anything. I removed them and I am in the clear. ---
**Author:** ![drewdin](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/drewdin/32/3363_2.png) [@drewdin](https://forum.inductiveautomation.com/u/drewdin)\ **Post date:** [April 28, 2024, 2:29am UTC](https://forum.inductiveautomation.com/t/ignition-docker-disk-full-the-disk-that-ignition-is-installed-on-is-nearly-full/88585/3 "2024-04-28T02:29:10Z")
Just to revisit this for anyone looking at this post. I originally thought it was the log files taking up all the space, then I thought it was a 10GB container limit until I found out it was a VM disc space issue. I needed to resize my vm disc space and enlarge the logical volume to match and everything was happy. ---
**Author:** ![kcollins](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/kcollins/32/19646_2.png) [@kcollins](https://forum.inductiveautomation.com/u/kcollins)\ **Post date:** [April 29, 2024, 1:53pm UTC](https://forum.inductiveautomation.com/t/ignition-docker-disk-full-the-disk-that-ignition-is-installed-on-is-nearly-full/88585/4 "2024-04-29T13:53:47Z")
Did you install Docker Engine directly on this Ubuntu machine? Or did you install something like Docker Desktop for Linux? Docker should simply use whatever storage is available on the host when installed directly on Linux. ---
**Author:** ![drewdin](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/drewdin/32/3363_2.png) [@drewdin](https://forum.inductiveautomation.com/u/drewdin)\ **Post date:** [April 29, 2024, 2:50pm UTC](https://forum.inductiveautomation.com/t/ignition-docker-disk-full-the-disk-that-ignition-is-installed-on-is-nearly-full/88585/5 "2024-04-29T14:50:20Z")
Its a Proxmox Ubuntu server 22.04 VM, I then installed the docker engine directly onto the vm and its the only thing installed. I see what your are saying, if the vm was 30gb and I installed the docker engine, it should have used the max space available. I have to try to recreate my steps and see how it got mismatched. ---
**Author:** ![michael.flagler](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/michael.flagler/32/26272_2.png) [@michael.flagler](https://forum.inductiveautomation.com/u/michael.flagler)\ **Post date:** [April 29, 2024, 2:52pm UTC](https://forum.inductiveautomation.com/t/ignition-docker-disk-full-the-disk-that-ignition-is-installed-on-is-nearly-full/88585/6 "2024-04-29T14:52:10Z")
If I had to guess (if you used Ubuntu server) is that by default it doesn't allocate all available disk space to the partition, so if you're not paying attention when installing the OS, you'll get an undersized disk and have to go in and resize it later to use all the available disk space. This isn't specific to Docker, but to Ubuntu in general. ---
**Author:** ![kcollins](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/kcollins/32/19646_2.png) [@kcollins](https://forum.inductiveautomation.com/u/kcollins)\ **Post date:** [April 29, 2024, 2:52pm UTC](https://forum.inductiveautomation.com/t/ignition-docker-disk-full-the-disk-that-ignition-is-installed-on-is-nearly-full/88585/7 "2024-04-29T14:52:27Z")
Okay, it sounds like maybe you just needed to expand the partition/filesystem after expanding it in Proxmox? I was confused by the wording "expand docker" and what that would entail in your described context. EDIT: what @michael.flagler said.. 😃 ---
**Author:** ![drewdin](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/drewdin/32/3363_2.png) [@drewdin](https://forum.inductiveautomation.com/u/drewdin)\ **Post date:** [April 29, 2024, 2:58pm UTC](https://forum.inductiveautomation.com/t/ignition-docker-disk-full-the-disk-that-ignition-is-installed-on-is-nearly-full/88585/8 "2024-04-29T14:58:40Z")
There is a 10gb container limit in docker and I was going down that path until I realized it was the disc space issue with the VM. I was confusing container size with the undersized disc space in my vm. I'll update my reply. ---
**Author:** ![michel\_feria](https://avatars.discourse-cdn.com/v4/letter/m/ad7895/32.png) [@michel\_feria](https://forum.inductiveautomation.com/u/michel_feria)\ **Post date:** [July 22, 2025, 5:43pm UTC](https://forum.inductiveautomation.com/t/ignition-docker-disk-full-the-disk-that-ignition-is-installed-on-is-nearly-full/88585/9 "2025-07-22T17:43:07Z")
I'm having the same issue but it seems to be related to the volume type, I'm using named volume but docker is creating files in the folder /var/lib/docker/overlay2/ any idea why is creating those files? thanks ---
**Author:** ![kcollins](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/kcollins/32/19646_2.png) [@kcollins](https://forum.inductiveautomation.com/u/kcollins)\ **Post date:** [July 22, 2025, 6:53pm UTC](https://forum.inductiveautomation.com/t/ignition-docker-disk-full-the-disk-that-ignition-is-installed-on-is-nearly-full/88585/10 "2025-07-22T18:53:33Z")
There is some variation based on settings, but on a typical Docker Engine installation, you'll find volume data under `/var/lib/docker/volumes/...` and container/image layer files under `/var/lib/docker/overlay2/...` (find your storage driver using `docker info` and looking for "Storage driver" in the output). More than likely, you're accumulating files outside of the defined named volume (typically `/usr/local/bin/ignition/data`). More inspection is warranted, though perhaps the output of this might be helpful: ```bash # assuming an Ignition container docker exec du -d 2 -h ``` The above will give you a 2-level-deep summary of disk space usage under the Ignition installation folder. ---
**Author:** ![kcollins](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/kcollins/32/19646_2.png) [@kcollins](https://forum.inductiveautomation.com/u/kcollins)\ **Post date:** [July 22, 2025, 6:58pm UTC](https://forum.inductiveautomation.com/t/ignition-docker-disk-full-the-disk-that-ignition-is-installed-on-is-nearly-full/88585/11 "2025-07-22T18:58:13Z")
Another thing I'll leave here for other folks to find is that whenever I install Docker Engine on a Linux host, I make sure to put the following supplemental configuration (at a minimum) under `/etc/docker/daemon.json`: ```json { "log-driver": "local", "log-opts": { "max-size": "64m", "max-file": "16" } } ``` This makes sure that container logs don't eventually fill up the host disk. The default [logging driver configuration](https://docs.docker.com/engine/logging/configure/) does _NOT_ perform rotation and will accumulate forever. This can result in quite a surprise later if you've got long-running containers, especially when (at least the last time I checked) log file disk usage isn't accounted for in `docker system df`. Happy Containering! ---
**Author:** ![michel\_feria](https://avatars.discourse-cdn.com/v4/letter/m/ad7895/32.png) [@michel\_feria](https://forum.inductiveautomation.com/u/michel_feria)\ **Post date:** [July 22, 2025, 10:08pm UTC](https://forum.inductiveautomation.com/t/ignition-docker-disk-full-the-disk-that-ignition-is-installed-on-is-nearly-full/88585/12 "2025-07-22T22:08:36Z")
Hello and thanks very much for your prompt reply, it's much appreciated! so, yes I'm pretty sure that the issue is related to logs as we're using extensively those logs for debug our application 🙂 do you have any idea why the logs are outside the named container? or how to use a bind volume instead of named volume? is that possible? using du command, I got those files coming from overlay2 ![image](https://us1.discourse-cdn.com/inductiveautomation/original/3X/e/a/ea21fc235043733e119386c098b81a06c009f69c.png) thanks again ! kind regards, ---
**Author:** ![kcollins](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/kcollins/32/19646_2.png) [@kcollins](https://forum.inductiveautomation.com/u/kcollins)\ **Post date:** [July 22, 2025, 10:48pm UTC](https://forum.inductiveautomation.com/t/ignition-docker-disk-full-the-disk-that-ignition-is-installed-on-is-nearly-full/88585/13 "2025-07-22T22:48:51Z")
With our Docker image, we currently don't store the wrapper.log within the container, as the logs need to be emitted to stdout for collection by whatever logging driver Docker (or other container runtime) is using. This is to prevent duplicate storage of logs. The `du` output you posted seems to be from the host--I was originally asking for the output of `du -d 2 -h` from _within_ your Ignition container in order to determine if something there was writing a large volume of data outside of your named volume. If you're curious about your container logs, those end up under `/var/lib/docker/containers/[container id]/[container id]-json.log` (again, with the default JSON logging driver of a typical Docker Engine installation). It should be pretty obvious if one of those has grown very large. You might be able to truncate them while the container is running, but the best way is to at least stop the container, truncate the log, and start. Ideally, stop Docker Engine altogether, make the changes to `/etc/docker/daemon.json` to enable log file rotation/pruning and then start everything back up. As always, make sure you've got good backups of everything before making changes! ---
**Author:** ![michel\_feria](https://avatars.discourse-cdn.com/v4/letter/m/ad7895/32.png) [@michel\_feria](https://forum.inductiveautomation.com/u/michel_feria)\ **Post date:** [July 23, 2025, 12:34am UTC](https://forum.inductiveautomation.com/t/ignition-docker-disk-full-the-disk-that-ignition-is-installed-on-is-nearly-full/88585/14 "2025-07-23T00:34:45Z")
> [@kcollins](#): > > `docker exec du -d 2 -h` Hi, I ran this command: **docker exec RemoteServer du -d 2 -h | sort -hr** And I got this: ![image](https://us1.discourse-cdn.com/inductiveautomation/original/3X/5/6/565bfc00134277677496492df8279a50eb62486c.jpeg) I know what I have in the logs as I'm using to debug my functions in the gateway context (the script console is not giving me all that I need) so I know that I'm pouring the log and what I need is to have control on those logs. I cannot acces to any folder below /var/lib/docker so I don't know how to check the json.log on /containers sub-folder. Last thing, I put your suggested supplemental configuration under /etc/docker/daemon.json and it works as the driver changed from json-file to local but ...inside the gateway setting, I'm seeing the same 50k logs... what I understand is that the wrapper.log is outside the container, likely in the overlay2 storage, isn't it? so I really need to have access to this wrapper.log to trim it, I will continue to test the log rotation as well, thanks very much and have a good evening! MF ---
**Author:** ![kcollins](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/kcollins/32/19646_2.png) [@kcollins](https://forum.inductiveautomation.com/u/kcollins)\ **Post date:** [July 23, 2025, 1:59pm UTC](https://forum.inductiveautomation.com/t/ignition-docker-disk-full-the-disk-that-ignition-is-installed-on-is-nearly-full/88585/15 "2025-07-23T13:59:15Z")
> [@michel\_feria](#): > > Hi, I ran this command: **docker exec RemoteServer du -d 2 -h | sort -hr** This shows that your disk usage inside the container is very minimal. There are still logs within the `logs/system_logs.idb` SQLite DB that serves the web UI. If you're experiencing locking errors related to this system due to excessive log events, you may need to adjust the log maintenance settings inside `data/logback.xml`--the defaults are in a commented-out block within that file in the event you need to uncomment and adjust those settings. Typically, you might need to increase the `diskspaceCleanupEventCount` setting if the maintenance frequency is unable to keep pace with log growth. All of that said, at `11M`, your logs folder is not very large--I don't think you're having this issue. > [@michel\_feria](#): > > Last thing, I put your suggested supplemental configuration under /etc/docker/daemon.json and it works as the driver changed from json-file to local but ...inside the gateway setting, I'm seeing the same 50k logs... Putting this setting into place means that Docker will rotate the container logs (again, to be clear, these are the logs managed by Docker Engine and are not Ignition-specific). > [@michel\_feria](#): > > what I understand is that the wrapper.log is outside the container, likely in the overlay2 storage, isn't it? so I really need to have access to this wrapper.log to trim it, I will continue to test the log rotation as well, See my previous post on where those logs actually live if you want to inspect--you'll need to escalate to `root` (using sudo, for example) to access the under-hood Docker folders under `/var/lib/docker`. Outside of Docker Engine installs in rootless mode (which isn't the case here as you reference `/var/lib/docker`), if you can manage containers on the host, you have root-level access. Also, to clarify further, there **is no wrapper.log** file in the container and nothing to trim--`/usr/local/bin/ignition/logs/wrapper.log` is a symlink to `/dev/stdout` in our container image and thus emits logs directly to the container runtime (whose log rotation we've already discussed). ---
**Author:** ![michel\_feria](https://avatars.discourse-cdn.com/v4/letter/m/ad7895/32.png) [@michel\_feria](https://forum.inductiveautomation.com/u/michel_feria)\ **Post date:** [July 23, 2025, 2:55pm UTC](https://forum.inductiveautomation.com/t/ignition-docker-disk-full-the-disk-that-ignition-is-installed-on-is-nearly-full/88585/16 "2025-07-23T14:55:23Z")
> [@michel\_feria](#): > > du -d 2 -h | sort -hr Hi, thanks very much for your help, I understand now much better the ignition structure inside dockers and I think that the growing issue is fixed with the log rotation configuration. kind regards MF