"no space left on device." - subvolumes fill up entire cache drive

[MagicLike]

I just experienced a very annoying issue:

My cache drive is filled up, with docker btrfs subvolumes, from earlier Docker containers.

I can't even install new Docker containers / reinstall them.

Is there a way of knowing which is used and which not, so I can delete and use my cache again?

[JorgeB]

Please post the diagnostics.

[MagicLike]

22 minutes ago, JorgeB said:

Please post the diagnostics.

I am not able to get the diagnostics.

When I try it over the GUI, it freezes and when I try it over ssh or the normal syslog, I get the following error:

Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 20480 bytes) in /usr/local/emhttp/plugins/dynamix/include/Syslog.php on line 18

 

[JorgeB]

Rebooting should fix that, then grab diags after array start.

[trurl]

What do you get from command line with this? Before rebooting please

df -h

 

[MagicLike]

1 minute ago, trurl said:

What do you get from command line with this?

df -h

 

Filesystem      Size  Used Avail Use% Mounted on
rootfs          7.8G  1.3G  6.5G  17% /
tmpfs            32M  764K   32M   3% /run
/dev/sda1       988M  545M  443M  56% /boot
overlay         7.8G  1.3G  6.5G  17% /lib/firmware
overlay         7.8G  1.3G  6.5G  17% /lib/modules
devtmpfs        8.0M     0  8.0M   0% /dev
tmpfs           7.9G  168K  7.9G   1% /dev/shm
cgroup_root     8.0M     0  8.0M   0% /sys/fs/cgroup
tmpfs           128M   82M   47M  64% /var/log
tmpfs           1.0M     0  1.0M   0% /mnt/disks
tmpfs           1.0M     0  1.0M   0% /mnt/remotes
tmpfs           1.0M     0  1.0M   0% /mnt/rootshare
/dev/md1        932G  870G   61G  94% /mnt/disk1
/dev/sdb1        58G   57G     0 100% /mnt/cache
shfs            932G  870G   61G  94% /mnt/user0
shfs            932G  870G   61G  94% /mnt/user
/dev/loop2      1.0G  4.1M  905M   1% /etc/libvirt
tmpfs           1.6G     0  1.6G   0% /run/user/0

 

[trurl]

Is that before rebooting?

 

I thought perhaps you had filled rootfs, but looks like cache is the only thing full.

[MagicLike]

1 minute ago, trurl said:

Is that before rebooting?

 

I thought perhaps you had filled rootfs, but looks like cache is the only thing full.

Yes, this is what I wrote in my initial message:

1 hour ago, MagicLike said:

My cache drive is filled up, with docker btrfs subvolumes, from earlier Docker containers.

 

Edit: I already faced the issue with filling up rootfs

Edited August 1, 2022 by MagicLike

[MagicLike]

18 minutes ago, JorgeB said:

Rebooting should fix that, then grab diags after array start.

I grabbed the diagnosis now, and from my perspective, nothing changed in my problem:

How do I identify old - now unused - Docker btrfs subvolumes, so I can clear some space on my cache drive?

By the way, the Docker container aren't starting anymore:

Warning: stream_socket_client(): unable to connect to unix:///var/run/docker.sock (Connection refused) in /usr/local/emhttp/plugins/dynamix.docker.manager/include/DockerClient.php on line 710
Couldn't create socket: [111] Connection refused
Warning: Invalid argument supplied for foreach() in /usr/local/emhttp/plugins/dynamix.docker.manager/include/DockerClient.php on line 896

 

diagnostics.zip

[JorgeB]

Diags are incomplete, shares are missing, I wanted to see if there was something else on cache.

[MagicLike]

1 minute ago, JorgeB said:

Diags are incomplete, shares are missing, I wanted to see if there was something else on cache.

Oops, I stripped the diagnostics a bit down...

The cache is only used by the appdata & system shares, if that answers your question.

[JorgeB]

Make sure disk shares are enabled, then click on cache and "compute" to see what exactly is using the space.

[MagicLike]

1 minute ago, JorgeB said:

Make sure disk shares are enabled, then click on cache and "compute" to see what exactly is using the space.

I already know, what uses my diskspace:

1 hour ago, MagicLike said:

My cache drive is filled up, with docker btrfs subvolumes, from earlier Docker containers.

I just want to know, how i can identify, if they are used or not, so I can delete those unused...

[trurl]

On Docker page, turn on Advanced View, delete the orphans

[trurl]

Also, click Container Size and post a screenshot

[MagicLike]

11 hours ago, trurl said:

On Docker page, turn on Advanced View, delete the orphans

I am not able to open the Docker page, because I restartet and now the cache is compeltely full and I get this error message:

Warning: stream_socket_client(): unable to connect to unix:///var/run/docker.sock (Connection refused) in /usr/local/emhttp/plugins/dynamix.docker.manager/include/DockerClient.php on line 710
Couldn't create socket: [111] Connection refused
Warning: Invalid argument supplied for foreach() in /usr/local/emhttp/plugins/dynamix.docker.manager/include/DockerClient.php on line 896