Initially, I thought I was dealing with a full docker image but now I'm wondering if it's possible drive failure? I just know things are complaining they cant write to disk because it's full but nothing appears full. 

veyron-diagnostics-20171024-2055.zip

Edited October 25, 20178 yr by BrandonG777

Cache filesystem is fully allocated and that can result in out of space errors, see here:

 

 

 

Ok, so it looks like all of my space is being eaten up by Plex cache files. Like 2 or 3 GB is eating up approx. 100GB! Is there something I should be doing to avoid this? I'm nearly finished offloading these cache drives to my main array but is reformatting my cache drives going to achieve anything if I put the Plex stuff back?

The allocation problem is related to btrfs, you don't need to format, just balance the cache.

 

At the time of the diagnostics you had 105GiB used, 129GiB free, but all free space is allocated, resulting in the out of space errors.

Thank you. Back up and running. 

 

btrfs fi show /mnt/cache
Label: none  uuid: 1e328593-96c1-4e6c-9e25-7d0db9eb359d
        Total devices 2 FS bytes used 75.64GiB
        devid    1 size 232.89GiB used 88.03GiB path /dev/sdc1
        devid    2 size 232.89GiB used 88.03GiB path /dev/sdb1

btrfs fi df /mnt/cache  
Data, RAID1: total=86.00GiB, used=75.38GiB
System, RAID1: total=32.00MiB, used=16.00KiB
Metadata, RAID1: total=2.00GiB, used=272.55MiB
GlobalReserve, single: total=74.64MiB, used=0.00B

Looking good now, you may want to run a weekly balance to prevent this for happening again in the future, something like:

 

btrfs balance start -dusage=75 /mnt/cache

You can use the user scripts plugin to schedule it.

Approx. 12 hours later...

root@Veyron:~# btrfs fi show /mnt/cache
Label: none  uuid: 1e328593-96c1-4e6c-9e25-7d0db9eb359d
        Total devices 2 FS bytes used 75.51GiB
        devid    1 size 232.89GiB used 118.03GiB path /dev/sdc1
        devid    2 size 232.89GiB used 118.03GiB path /dev/sdb1

root@Veyron:~# btrfs fi df /mnt/cache 
Data, RAID1: total=116.00GiB, used=75.21GiB
System, RAID1: total=32.00MiB, used=48.00KiB
Metadata, RAID1: total=2.00GiB, used=304.41MiB
GlobalReserve, single: total=99.08MiB, used=0.00B
root@Veyron:~# 

root@Veyron:~# btrfs balance start -dusage=75 /mnt/cache
Done, had to relocate 43 out of 119 chunks

root@Veyron:~# btrfs fi show /mnt/cache
Label: none  uuid: 1e328593-96c1-4e6c-9e25-7d0db9eb359d
        Total devices 2 FS bytes used 75.50GiB
        devid    1 size 232.89GiB used 78.03GiB path /dev/sdc1
        devid    2 size 232.89GiB used 78.03GiB path /dev/sdb1
root@Veyron:~# btrfs fi df /mnt/cache 
Data, RAID1: total=76.00GiB, used=75.20GiB
System, RAID1: total=32.00MiB, used=16.00KiB
Metadata, RAID1: total=2.00GiB, used=300.88MiB
GlobalReserve, single: total=96.36MiB, used=0.00B
root@Veyron:~# 

 

I'm thinking I need to do that nightly.  The system does get used though.

 

It's normal if the cache is regularly filled and emptied, weekly balance should still be enough, and this behavior should stop or at least improve considerably once we get to kernel 4.14

Thank you again, Sir. I installed the user script plugin awhile back but never did anything else with it. Now that I've rebuilt my docker containers and lost my cron jobs,  I see this plugin is a better approach.