Gather metrics for Docker img filling up

[tone]

Nearly everyday my docker image size (20gb total) goes from ~60% usage to 70%+ usage and the alarms start going off (70%). The image size continues to fluctuate (69%, 72%, etc.) so a ton of alarms go off. I believe the "Container" and "Writeable" are growing quickly due to Plex traffic but am not sure that is the only problem and want to understand all my applications a bit more. I am not interested in increasing the docker image size or changing when the alarms go off, I want to understand which applications are consuming the space and why.

 

I have analyzed all my active docker containers and added a max log setting where applicable, when the log became large (>50mb), so I don't believe runaway logs are the problem:

--log-opt max-size=50m --log-opt max-file=1

I know I can get a current snapshot of my docker usage by clicking the "Container Size" button on the /Docker page but that is only a current snapshot and it isn't fast enough or historical. I was thinking about collecting this data and placing it into a time series DB (InfluxDB or Prometheus) and graphing it (Grafana).

 

I am most familiar with Prometheus but am not sure how to collect the data from:

/mnt/user/system/docker/docker.img

 

Any tips? Is this the best way to get this information/solve this problem? Has this been done before?

Thanks!