Skip to main content

Docker on Linux log maintenance and disk space recovery

Logs and unused images are the leading cause of disk space consumption on Docker deployments. This article recovers space now and limits future growth. (If Elasticsearch indices are consuming the disk, that usage is valid and expected.)

Make sure Informer is not over-logging​

From the Informer UI:

  • Informer 5.5.x and higher: set all Log Levels to Error under Administration > Log > Settings, then Apply.
  • Informer 5.4.x and below: turn off all System Logging under Administration > Log > Settings, then Save.

Purge logs from disk​

Remove the Docker JSON and PM2 logs from the host. This is safe to run while Informer is online:

sudo find /var/lib/docker/ -type f -name "*.log" -delete

Purge unused Docker components​

Clean up unused images, containers, and volumes (a good follow-up after an upgrade). Informer must be online for this:

sudo docker system prune -a --volumes

This removes all stopped containers, networks not used by a container, volumes not used by a container, images with no associated container, and the build cache.

Prevent future log growth​

There are two log sources on Docker: the PM2 logs (redundant for Informer) and the Docker JSON logs.

Stop PM2 from writing logs to disk​

Edit docker-compose.yml and add -o /dev/null -e /dev/null to the pm2 start in each service's entrypoint (Informer API, Query, Schedules, and the load balancer). For example:

informer-api:
  entrypoint: "/bin/sh -c 'node dockerpreboot.js && pm2 start -o /dev/null -e /dev/null index.js -i 2 --name informer --no-daemon'"
i5-load-balancer:
  entrypoint: "/bin/sh -c 'pm2 start -o /dev/null -e /dev/null index.js -i 1 --name i5-load-balancer --no-daemon'"

Rotate the Docker JSON logs​

Add a logging block to each Informer and load-balancer service to cap log file size:

logging:
  driver: "json-file"
  options:
    max-file: "5"
    max-size: "10m"

Apply the changes​

Save docker-compose.yml, then recreate the affected containers:

docker-compose stop
docker-compose rm informer-api
docker-compose rm informer-query
docker-compose rm informer-schedules
docker-compose rm i5-load-balancer
docker-compose up -d

Space is recovered and future growth is limited. As a strongly recommended follow-up, make sure the Elasticsearch data is mapped to the host file system: see Elasticsearch volume and memory (Docker).