The Docker Volume That Accumulated 200GB of Logs (And Killed the Server)

"No disk space left."

I SSH'd into our production server. df -h showed 0% available.

The server had been running for 6 months without issues. What changed?

Nothing. That was the problem.

The Setup

We ran a Node.js app inside a Docker container:

FROM node:18
WORKDIR /app
COPY . .
RUN npm ci
CMD ["node", "app.js"]

The app wrote logs to /app/logs/app.log.

We started the container with:

docker run -d --name myapp myapp:latest

No volume mount. No log rotation. No size limits.

The Hidden Accumulation

Docker creates anonymous volumes for any directory that writes data inside the container, unless you explicitly mount something there.

/app/logs became an anonymous volume.

Every day, the app wrote ~1GB of logs.

After 6 months: ~180GB.

But Docker doesn't clean up anonymous volumes automatically. They persist even after the container stops.

Our server had a 250GB disk. The logs took 180GB. The rest was the OS, Docker images, and other containers.

When the disk filled up:

• The app couldn't write logs (but kept retrying, consuming CPU)
• The database container couldn't write WAL files
• SSH became sluggish
• The entire server became unusable

The Investigation

# Check disk usage
df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1      250G  250G   0G  100% /

# Find large files
du -sh /var/lib/docker/volumes/* | sort -h
180G    /var/lib/docker/volumes/3f8a9b2c.../_data

# Look inside
sudo ls -la /var/lib/docker/volumes/3f8a9b2c.../_data
-rw-r--r-- 1 root root 180G app.log

That anonymous volume had 180GB of logs.

The Emergency Fix

1. Stop the container

docker stop myapp

2. Remove the anonymous volume (DANGER: data loss)

# List volumes
docker volume ls

# Remove the specific volume
docker volume rm 3f8a9b2c...

3. Free up space

# Prune everything
docker system prune -a --volumes

4. Restart with proper logging

docker run -d   --name myapp   --log-opt max-size=10m   --log-opt max-file=3   --mount type=bind,source=/var/log/myapp,target=/app/logs   myapp:latest

The Proper Fix

1. Use Docker logging driver with limits

# daemon.json (global)
{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3",
    "compress": "true"
  }
}

2. Never write logs to a file inside container – write to stdout

// app.js - use console.log instead of file transport
const logger = winston.createLogger({
  transports: [
    new winston.transports.Console({
      format: winston.format.json()
    })
  ]
});

Then Docker captures stdout and applies log rotation.

3. Use a dedicated log volume with rotation

# Create a volume with log rotation using a sidecar container
docker run -d   --name log-rotator   -v myapp-logs:/logs   docker.io/ubuntu:latest   sh -c "while true; do logrotate /etc/logrotate.conf; sleep 3600; done"

4. Set up log shipping to external system

# Use fluentd or vector to ship logs to S3/ELK
docker run -d   --log-driver=fluentd   --log-opt fluentd-address=localhost:24224   myapp:latest

The Monitoring

We added disk usage alerts:

#!/bin/bash
# check-disk.sh
USAGE=$(df / | tail -1 | awk '{print $5}' | sed 's/%//')
if [ $USAGE -gt 85 ]; then
  curl -X POST -H 'Content-type: application/json'     --data "{"text":"Disk usage at $ {USAGE}% on $(hostname)"}"     $SLACK_WEBHOOK
fi

The Prevention Checklist

• [ ] Containers write logs to stdout, not files.
• [ ] Docker daemon has log rotation configured.
• [ ] Anonymous volumes are avoided (always name volumes).
• [ ] Disk usage monitoring is in place.
• [ ] 'docker system prune' runs weekly in cron.

Commands to Audit Volumes

List all volumes with size

docker run --rm -v /var/run/docker.sock:/var/run/docker.sock   alpine/du:latest /var/lib/docker/volumes

Find unused volumes

docker volume ls -qf dangling=true

Inspect volume contents

docker run --rm -v my_volume:/data alpine ls -la /data

What I Learned

• Anonymous volumes are persistent and grow forever.
• Docker's default logging has no limits unless configured.
• Writing logs to files inside containers is an anti-pattern.
• 'docker system prune -a --volumes' is your friend in emergencies.

That 200GB of logs cost us 2 hours of downtime and a very angry customer. Now every container writes to stdout and we ship logs to Datadog. Disk alerts saved us twice since then.