[Solved] docker.img was 82% full / 159GB used — dangling anonymous volumes, Prometheus mapping error, and a strange Btrfs space recovery

[Solved] docker.img was 82% full / 159GB used — dangling anonymous volumes, Prometheus mapping error, and a strange Btrfs space recovery

I wanted to document this in case it helps someone else chasing a mysteriously full docker.img.

I had a 195 GiB Docker image using the Btrfs storage driver. It had grown to roughly:

Filesystem Size Used Avail Use%

/dev/loop3 195G 159G 35G 82%

btrfs filesystem usage -T /var/lib/docker agreed that this wasn't merely an Unraid GUI/accounting issue. Btrfs itself reported roughly 158 GiB used, including about 156 GiB of data.

The final result, after troubleshooting and cleanup, was:

Filesystem Size Used Avail Use%

/dev/loop3 195G 34G 160G 18%

So this recovered roughly 125 GB inside docker.img without rebuilding Docker.

What I checked first

I initially suspected the usual things:

- runaway Docker logs

- a container writing large amounts of data into its writable layer

- bad path mappings

- old/dangling Docker images

- build cache

- orphaned Btrfs Docker subvolumes

None of those explained it.

Logs were small and capped. Build cache was zero. No live container had a giant writable layer.

I also compared the Btrfs Docker layer directories against Docker's layer metadata. There were:

385 image layers

33 container mount layers

33 init layers

451 Docker references total

451 physical Btrfs subvolumes

There were no visible physical subvolumes that Docker didn't know about, and no Docker references with a missing physical subvolume.

Interestingly, summing Docker's layerdb/.../size files gave only about 29.4 GiB of logical image-layer diffs, despite Btrfs reporting ~156 GiB of data in use.

The first real clue: Docker volumes

docker system df -v showed a large number of volumes with LINKS 0.

I enumerated only volumes that Docker itself considered dangling and found 85 anonymous dangling volumes, containing about 13.21 GiB of visible data.

Most had this label:

com.docker.volume.anonymous

Before deleting anything, I inspected their contents.

Most of the large ones turned out to be old Prometheus TSDB databases. They contained the expected Prometheus structures:

wal

chunks_head

queries.active

TSDB block directories

There were also a few unrelated old temporary-data volumes.

Why Prometheus kept making them

This turned out to be an error in my Unraid Prometheus template.

I had:

Host:

/mnt/user/appdata/prometheus/data

Container:

/prometheus/data

But the Prometheus image declares its persistent volume at:

/prometheus

So my bind mount was one directory too deep.

Docker was therefore still creating an anonymous volume for /prometheus, and Prometheus was putting the actual TSDB there.

When the container was recreated or updated, old anonymous Prometheus volumes could be left behind.

The correct mapping is:

/mnt/user/appdata/prometheus/data -> /prometheus

Cleanup

I did not run a blind docker system prune or delete the entire volume directory.

I generated a list containing only volumes that were both:

dangling=true

and:

label=com.docker.volume.anonymous

I then re-checked each volume immediately before passing it to docker volume rm.

There were 85 candidates containing 13.21 GiB of visible files.

After removing them, something surprising happened:

Before: ~159 GiB used

After: ~40 GiB used

That was a roughly 119 GiB reduction, despite the deleted volumes themselves containing only ~13 GiB according to du.

I immediately checked that I hadn't destroyed the Docker installation.

Everything was still present:

Containers: 33

Images: 33

Btrfs subvolumes: 451

All running containers retained their previous state.

Btrfs then reported only about 39.5 GiB actually used.

The Btrfs part is still an open question

I want to be careful here because I don't have enough evidence to claim exactly what happened internally.

The 13.21 GiB of anonymous-volume files did not necessarily account directly for the additional ~100 GiB that Btrfs released.

One plausible explanation is that there was already a backlog of deleted/shared Btrfs extents or snapshot cleanup, and the filesystem activity during this process coincided with the cleaner/transaction machinery finally reclaiming that space.

Btrfs subvolume deletion/cleanup can be asynchronous, so that explanation is technically plausible.

However, I did not capture the relevant state before the large drop, so I can't prove it.

After the cleanup I ran:

btrfs subvolume list -d /var/lib/docker

and it was empty.

That tells me there were no deleted subvolumes still awaiting cleanup at that point, but it doesn't tell me whether such a backlog existed immediately before the 119 GiB reclamation.

So I would characterize this as:

Confirmed: deleting the 85 dangling anonymous volumes coincided with Btrfs releasing ~119 GiB.

Confirmed: Docker's image/container graph remained intact.

Confirmed: Prometheus had been accumulating anonymous TSDB volumes because of the incorrect /prometheus/data mapping.

Not confirmed: exactly which Btrfs references/extents accounted for the additional ~100 GiB released.

Fixing Prometheus without losing its current database

The active Prometheus anonymous volume contained about 1.1 GiB.

I stopped Prometheus, copied that current database to the proper appdata location, and then changed the Unraid mapping:

docker stop prometheus

# SRC was the Mountpoint reported by:

docker volume inspect <current-prometheus-volume>

DST=/mnt/user/appdata/prometheus/data

mkdir -p "$DST"

rsync -aHAX --numeric-ids "$SRC"/ "$DST"/

Then I changed the template to:

/mnt/user/appdata/prometheus/data -> /prometheus

instead of:

/mnt/user/appdata/prometheus/data -> /prometheus/data

After recreating the container:

docker inspect prometheus \

-f '{{range .Mounts}}{{printf "%-6s %-55s -> %s\n" .Type .Source .Destination}}{{end}}'

showed:

bind /mnt/user/appdata/prometheus/etc -> /etc/prometheus

bind /mnt/user/appdata/prometheus/data -> /prometheus

There was no longer an anonymous volume mounted at /prometheus.

Prometheus started normally, found its existing TSDB blocks healthy, replayed the WAL successfully, started the TSDB, loaded its configuration, and reported that it was ready for requests.

I then verified that the old anonymous Prometheus volume was dangling before removing that specific volume.

Final state

Current Docker accounting:

TYPE TOTAL ACTIVE SIZE

Images 33 32 31.58GB

Containers 33 26 3.476GB

Local Volumes 3 2 ~6kB

Build Cache 0 0 0B

And:

/dev/loop3 195G 34G 160G 18%

The two remaining anonymous volumes are active, empty volumes for applications that deliberately use ephemeral paths. There is also one tiny named volume of only a few KB.

Things I'd recommend checking before recreating docker.img

If anyone finds their Docker image inexplicably huge, these were the most useful commands for me:

docker system df -v

df -h /var/lib/docker

btrfs filesystem usage -T /var/lib/docker

docker volume ls

docker volume ls -qf dangling=true

btrfs subvolume list -d /var/lib/docker

And I would strongly recommend inspecting dangling volumes before pruning them. LINKS 0 means no current container references the volume; it doesn't mean the volume contains nothing important.

In my case, rebuilding docker.img or changing storage drivers would have treated the symptom without finding the actual Prometheus configuration mistake.

The system is now sitting normally at about 18% Docker-image usage, so I'm leaving Btrfs alone and monitoring it.

I'd be interested if anyone with deeper Btrfs knowledge can explain the 13 GiB of deleted visible data coinciding with ~119 GiB of actual Btrfs reclamation. That's the remaining part I can't definitively explain.