Ignition Docker: Disk Full The disk that Ignition is installed on is nearly full

[drewdin]

I followed the post below and I still cant clear my hard disk usage.

Docker all gateways notify "disk full"

Hi all, we’ve succesfully used Ignition in docker containers for the last couple of years but now we have to front this issue: suddenly all containers have become corrupted and after recreating (just a few) all of them show the message Disk Full The disk that Ignition is installed on is nearly full. in the status page. We’ve been working with 20 or more container instantiated and without a problem, now we use just 2 or 3 and every 2 or 3 days the ignition containers (and only them!) explode. …

My Docker instance says the disk is full, using all 15gb of hard drive space. I followed the post above and deleted the log file and rebooted docker but I'm still getting a disk full error.

I'm running HighByte, Ignition, and mssql, all of the log files are tiny for each container.

I am running Proxmox and have a dedicated ubuntu server docker vm. the overall HD size if 35gb.

Any pointers on where to start looking? thanks!

[drewdin]

I was able to figure it out for anyone looking, I installed portainer and found about 14 1gb volumes that were not attached to anything. I removed them and I am in the clear.

[drewdin]

Just to revisit this for anyone looking at this post. I originally thought it was the log files taking up all the space, then I thought it was a 10GB container limit until I found out it was a VM disc space issue. I needed to resize my vm disc space and enlarge the logical volume to match and everything was happy.

[kcollins1]

Did you install Docker Engine directly on this Ubuntu machine? Or did you install something like Docker Desktop for Linux? Docker should simply use whatever storage is available on the host when installed directly on Linux.

[drewdin]

Its a Proxmox Ubuntu server 22.04 VM, I then installed the docker engine directly onto the vm and its the only thing installed.

I see what your are saying, if the vm was 30gb and I installed the docker engine, it should have used the max space available. I have to try to recreate my steps and see how it got mismatched.

[michael.flagler]

If I had to guess (if you used Ubuntu server) is that by default it doesn't allocate all available disk space to the partition, so if you're not paying attention when installing the OS, you'll get an undersized disk and have to go in and resize it later to use all the available disk space. This isn't specific to Docker, but to Ubuntu in general.

[kcollins1]

Okay, it sounds like maybe you just needed to expand the partition/filesystem after expanding it in Proxmox? I was confused by the wording "expand docker" and what that would entail in your described context.

EDIT: what @michael.flagler said..

[drewdin]

There is a 10gb container limit in docker and I was going down that path until I realized it was the disc space issue with the VM.

I was confusing container size with the undersized disc space in my vm. I'll update my reply.