Last Wednesday around 04:00, I woke up to a “Disk Usage Critical” mail from the monitoring system running on my own VPS. At first I thought “what now?” but, with experience from previous VPS overload and OOM scenarios, I tried to stay calm. When I tried to SSH in, I saw the system was struggling — for a while sshd couldn’t even accept connections.
This time the culprit, as I suspected, wasn’t a database bloat or a misconfigured cache. The log files produced by 13+ Docker containers had silently and sneakily eaten my disk. When df -h showed 100% disk usage, I dove into a deep investigation to find the source.
Why Does the Disk Fill Up? The Sneakiness of Docker Logs
Docker’s default json-file logging driver simply appends logs to a separate JSON file per container. While that’s fine for small, short-lived containers, for long-running services that produce a lot of output it becomes a real problem over time. My Astro sites, the AI generation pipeline, and other side products (hesapciyiz.com, islistesi.com) constantly produce logs.
The more a container logs, the bigger that log file grows. By default Docker doesn’t auto-rotate or auto-clean these log files. Which means: if you don’t intervene, log files can grow forever and completely fill the disk. This reminded me of the _work/_temp directory fill-up I had on my GitHub Actions runner — there too, temporary files inflated my disk.
Symptoms and the First Response
The first symptom of disk-full was, of course, the alert from the monitor. Then while trying to SSH in I had delays and even refused connections. The system’s overall performance dropped, my websites slowed or wouldn’t open at all.
The first thing I did was check overall disk state with df -h. Then I tried identifying large directories with du -sh /*. I quickly saw that /var/lib/docker/containers was abnormally large. Inside that directory, the *-json.log files under each container ID were taking up gigabytes. For example, the log file of one of my Next.js apps had reached 15 GB on its own!
For the emergency, I emptied the largest log files with cat /dev/null > /var/lib/docker/containers/<container-id>/<container-id>-json.log. That clears the contents instead of deleting the file, freeing the disk while preventing running containers from losing their file handles. After that the system relaxed and went back to normal. But I knew this was just a “patch” and I needed a permanent fix.
The Permanent Fix: Docker Log Rotation Strategies
The most effective way to prevent these disk fires is to use Docker’s built-in log rotation. For the json-file driver, Docker exposes options called log-opts that let you limit log file size and count.
Global Settings via daemon.json
If you want to apply the same log rotation rules to all containers, you can update the Docker daemon configuration via the daemon.json file. It’s typically at /etc/docker/daemon.json.
You set log-driver to json-file and configure max-size and max-file under log-opts. max-size is the maximum size a log file can reach (e.g. “10m” = 10 MB), and max-file is the maximum number of log files to keep.
{
"log-driver": "json-file",
"log-opts": {
"max-size": "10m",
"max-file": "3"
}
}This config will prevent any log file from exceeding 10 MB and keep at most 3 log files. Old log files will be deleted automatically when a new one is created. After applying these changes you’ll need to restart the Docker service: sudo systemctl restart docker.
The trade-off here is that it affects all containers and requires restarting the Docker daemon. If you’re doing this in a production environment, be careful to avoid downtime. On my own VPS, I prefer to do this at night when traffic is lowest.
Per-Container Settings via docker-compose.yml
If you want different log rotation rules per container — or you don’t want to make a global change in daemon.json — you can use logging settings per service in your docker-compose.yml. That gives me more flexible control, since some services produce fewer logs while others (like the AI pipeline) produce far more.
In the example below, log rotation settings are defined for the my-app service:
version: '3.8'
services:
my-app:
image: my-image:latest
ports:
- "80:80"
logging:
driver: "json-file"
options:
max-size: "5m"
max-file: "5"
another-app:
image: another-image:latest
ports:
- "81:81"
logging:
driver: "json-file"
options:
max-size: "20m"
max-file: "2"In this example, each my-app log file will be at most 5 MB and at most 5 files will be kept, while for another-app the limits are 20 MB and 2 files. With this method, the settings take effect when you re-create the relevant service via docker-compose up -d. It’s a more controlled approach compared to the global Docker daemon restart.
Maintenance and Monitoring After Log Rotation
The work doesn’t end after configuring log rotation. Existing log files won’t be affected by these settings. So even if you apply the configuration above, the existing huge log files won’t shrink. So after the initial setup, you may need to manually delete or empty the old, large log files.
To verify that the settings are correctly applied, you can use docker inspect <container-id>. In the output, check the LogOpts section to see that max-size and max-file are properly set.
In my own pipeline, I have a small shell script that regularly checks the df -h output. If disk usage exceeds a threshold (e.g. 80%), it sends me an alert mail. That way I can detect potential disk-fill issues early. It’s part of my preflight resource guard pattern. Tracking container sizes with docker ps -s can also be useful.
When choosing max-size and max-file, be careful. Very small values can cause logs to rotate too often and increase disk I/O. Very low max-file values can limit how far back you can investigate during an incident. It’s a trade-off between disk space and debuggability. For me, 10 MB and 3-5 files is usually enough.
Alternative Log Drivers and When to Use Them
While the json-file driver is enough for most small and mid-sized self-hosted projects, sometimes you need more advanced solutions. Especially in large-scale and distributed systems, collecting logs in a central system and analyzing them is much more efficient.
Docker supports different log drivers like syslog, journald, gelf, fluentd, awslogs. These drivers can send logs directly to a syslog server, the systemd journal, Graylog, Fluentd or AWS CloudWatch. On my own VPS, json-file with proper rotation is enough for now. But while working at a large Turkish e-commerce site or at a bank’s internal platform in the past, I remember pushing logs to Elasticsearch via fluentd and analyzing them on Kibana. These solutions are essential in environments with thousands of containers.
But for someone like me managing 13+ containers on their own server, building an additional log-collection infrastructure is both extra cost and management load. So the log-opts options the json-file driver provides offer a pragmatic and sufficient solution.
Conclusion
Docker logs silently filling the disk is a problem I’ve faced many times — and it’s confirmed by experience. Especially when I have these “disk fires” on my own VPS, I rediscover how critical automatic log rotation is. Setting the right max-size and max-file values is one of the simplest and most effective ways to prevent these issues.
This problem doesn’t only cause disk-fill — it also drops overall system performance and can render critical services unusable. So configuring log rotation when you set up your Docker containers will save you many headaches later.
Have you had a similar disk-fill adventure? Which log rotation strategies do you use? I’d love to hear in the comments. Maybe in a future post I’ll explain how I make the data inside these log files more meaningful with simple shell scripts, or how I monitor the logs of my AI-driven content pipeline.