I followed the post below and I still cant clear my hard disk usage.
My Docker instance says the disk is full, using all 15gb of hard drive space. I followed the post above and deleted the log file and rebooted docker but I'm still getting a disk full error.
I'm running HighByte, Ignition, and mssql, all of the log files are tiny for each container.
I am running Proxmox and have a dedicated ubuntu server docker vm. the overall HD size if 35gb.
Any pointers on where to start looking? thanks!
I was able to figure it out for anyone looking, I installed portainer and found about 14 1gb volumes that were not attached to anything. I removed them and I am in the clear.
Just to revisit this for anyone looking at this post. I originally thought it was the log files taking up all the space, then I thought it was a 10GB container limit until I found out it was a VM disc space issue. I needed to resize my vm disc space and enlarge the logical volume to match and everything was happy.
Did you install Docker Engine directly on this Ubuntu machine? Or did you install something like Docker Desktop for Linux? Docker should simply use whatever storage is available on the host when installed directly on Linux.
Its a Proxmox Ubuntu server 22.04 VM, I then installed the docker engine directly onto the vm and its the only thing installed.
I see what your are saying, if the vm was 30gb and I installed the docker engine, it should have used the max space available. I have to try to recreate my steps and see how it got mismatched.
If I had to guess (if you used Ubuntu server) is that by default it doesn't allocate all available disk space to the partition, so if you're not paying attention when installing the OS, you'll get an undersized disk and have to go in and resize it later to use all the available disk space. This isn't specific to Docker, but to Ubuntu in general.
Okay, it sounds like maybe you just needed to expand the partition/filesystem after expanding it in Proxmox? I was confused by the wording "expand docker" and what that would entail in your described context.
EDIT: what @michael.flagler said.. 
There is a 10gb container limit in docker and I was going down that path until I realized it was the disc space issue with the VM.
I was confusing container size with the undersized disc space in my vm. I'll update my reply.
I'm having the same issue but it seems to be related to the volume type, I'm using named volume but docker is creating files in the folder /var/lib/docker/overlay2/
any idea why is creating those files?
thanks
There is some variation based on settings, but on a typical Docker Engine installation, you'll find volume data under /var/lib/docker/volumes/... and container/image layer files under /var/lib/docker/overlay2/... (find your storage driver using docker info and looking for "Storage driver" in the output).
More than likely, you're accumulating files outside of the defined named volume (typically /usr/local/bin/ignition/data). More inspection is warranted, though perhaps the output of this might be helpful:
# assuming an Ignition container
docker exec <container name> du -d 2 -h
The above will give you a 2-level-deep summary of disk space usage under the Ignition installation folder.
Another thing I'll leave here for other folks to find is that whenever I install Docker Engine on a Linux host, I make sure to put the following supplemental configuration (at a minimum) under /etc/docker/daemon.json:
{
"log-driver": "local",
"log-opts": {
"max-size": "64m",
"max-file": "16"
}
}
This makes sure that container logs don't eventually fill up the host disk. The default logging driver configuration does NOT perform rotation and will accumulate forever. This can result in quite a surprise later if you've got long-running containers, especially when (at least the last time I checked) log file disk usage isn't accounted for in docker system df.
Happy Containering!
Hello and thanks very much for your prompt reply, it's much appreciated!
so, yes I'm pretty sure that the issue is related to logs as we're using extensively those logs for debug our application 
do you have any idea why the logs are outside the named container? or how to use a bind volume instead of named volume? is that possible?
using du command, I got those files coming from overlay2
kind regards,
With our Docker image, we currently don't store the wrapper.log within the container, as the logs need to be emitted to stdout for collection by whatever logging driver Docker (or other container runtime) is using. This is to prevent duplicate storage of logs.
The du output you posted seems to be from the host--I was originally asking for the output of du -d 2 -h from within your Ignition container in order to determine if something there was writing a large volume of data outside of your named volume.
If you're curious about your container logs, those end up under /var/lib/docker/containers/[container id]/[container id]-json.log (again, with the default JSON logging driver of a typical Docker Engine installation). It should be pretty obvious if one of those has grown very large. You might be able to truncate them while the container is running, but the best way is to at least stop the container, truncate the log, and start. Ideally, stop Docker Engine altogether, make the changes to /etc/docker/daemon.json to enable log file rotation/pruning and then start everything back up. As always, make sure you've got good backups of everything before making changes!
Hi, I ran this command: docker exec RemoteServer du -d 2 -h | sort -hr
And I got this:
I know what I have in the logs as I'm using to debug my functions in the gateway context (the script console is not giving me all that I need) so I know that I'm pouring the log and what I need is to have control on those logs. I cannot acces to any folder below /var/lib/docker so I don't know how to check the json.log on /containers sub-folder.
Last thing, I put your suggested supplemental configuration under /etc/docker/daemon.json and it works as the driver changed from json-file to local but ...inside the gateway setting, I'm seeing the same 50k logs...
what I understand is that the wrapper.log is outside the container, likely in the overlay2 storage, isn't it? so I really need to have access to this wrapper.log to trim it, I will continue to test the log rotation as well,
thanks very much and have a good evening!
MF
This shows that your disk usage inside the container is very minimal. There are still logs within the logs/system_logs.idb SQLite DB that serves the web UI. If you're experiencing locking errors related to this system due to excessive log events, you may need to adjust the log maintenance settings inside data/logback.xml--the defaults are in a commented-out block within that file in the event you need to uncomment and adjust those settings. Typically, you might need to increase the diskspaceCleanupEventCount setting if the maintenance frequency is unable to keep pace with log growth. All of that said, at 11M, your logs folder is not very large--I don't think you're having this issue.
Putting this setting into place means that Docker will rotate the container logs (again, to be clear, these are the logs managed by Docker Engine and are not Ignition-specific).
See my previous post on where those logs actually live if you want to inspect--you'll need to escalate to root (using sudo, for example) to access the under-hood Docker folders under /var/lib/docker. Outside of Docker Engine installs in rootless mode (which isn't the case here as you reference /var/lib/docker), if you can manage containers on the host, you have root-level access. Also, to clarify further, there is no wrapper.log file in the container and nothing to trim--/usr/local/bin/ignition/logs/wrapper.log is a symlink to /dev/stdout in our container image and thus emits logs directly to the container runtime (whose log rotation we've already discussed).
Hi, thanks very much for your help, I understand now much better the ignition structure inside dockers and I think that the growing issue is fixed with the log rotation configuration.
kind regards
MF
