Cache drive reporting full, when its not

[JoshTheBlack]

Unraid 6.9.2

 

My server recently started sending me notifications about the cache drive utilization being over 90%. Its a 1TB samsung Evo SSD that is fairly new.  Purchased 4/15/21 brand new and installed a few days later.  Originally part of a raid-0 mirror with another drive that failed.  Recently (a week or so ago) wiped and set up as single.

 

Go to check on it.  The free space keeps dropping, fairly slowly but consistently.  I ran the mover to get all (non appdata/system) files off the cache disk.  It moved a fair amount of stuff, but didn't free up any space.  

 

Using the file browser in unraid, only the appdata and system folders remain on the disk. 

 

Running the du -h /mnt/cache command shows that 310g are being used.

 

df -h /mnt/cache or df -h /dev/sdb1 show 932g size 869g used 63g avail 94% use.

 

I am currently running a scrub, its at 5.54% with no errors found.

Also running a balance with 66% left.

 

I'm stumped!  Diagnostics attached.

rocinante-diagnostics-20210820-1137.zip

[JorgeB]

2 minutes ago, JoshTheBlack said:

du -h /mnt/cache command shows that 310g are being used.

du isn't reliable with btrfs, GUI will show the correct used/free space, note that if you have vdisks they will grow if not trimmed, see here for Windows VMs.

[JoshTheBlack]

VMs are not enabled. 

[JorgeB]

This is the current usage:

 

    Used:             859.18GiB
    Free (estimated):          71.58GiB    (min: 71.58GiB)

 

Something is using that space, balance won't change anything, if you don't know what it is start moving data to the array until you find out.

 

 

[JoshTheBlack]

This is weird.  I changed appdata from cache: prefer to cache: yes, then invoked the mover.  It's been running close to an hour so far.

 

As you can see in the pictures, my usage has returned to normal in the first image.  The second is my shares with usage broken down by drive.  It transferred 14.6 GB from Cache to Disk 8, and managed to drop used from 859GB to 308GB.

 

Could it be something to do with sector size on the cache?  My plex metadata is in appdata, so far it has transferred 18621 files in the plex folder share, totalling 7.2g. 

14.6G total across 36265 files.  Maybe they are taking up lots of space despite being small?

 

Then again, probably not.  The cache drive portion of appdata still has 539131 files.  I really don't understand what caused this problem in the first place.  If moving 14G cleared up 500G of space, something just isn't right.

 

EDIT:  The pool drives are GPT 4kb-aligned.  The cache is MBR: 1MiB-aligned.

Edited August 20, 2021 by JoshTheBlack

[JorgeB]

12 hours ago, JoshTheBlack said:

Could it be something to do with sector size on the cache? 

Sector size is the same, just the alignment is different, could be filesystem related, i.e., xfs might be more efficient with very small files, but don't remember this coming up before, and certainly not by this much:

12 hours ago, JoshTheBlack said:

It transferred 14.6 GB from Cache to Disk 8, and managed to drop used from 859GB to 308GB.

See if moving the data back to cache takes up all the space again.

[JoshTheBlack]

Was planning on doing that today, but before I could it's back to 70% usage again!  New screenshots of my cache and pool.  There is NOTHING on the cache drive that is not in the pool.  

 

I don't know if it's relevant, but there are some BTRFS errors in the disk log I attached.  Errors are from Aug 9.  Lines 24-26 and 38-40.  That was probably from when I removed a cache drive and setup as single, then reformatted.

 

I'm currently moving all data off the cache drive with the intention of reformatting again, probably with xfs.  

disklog.txt

[JoshTheBlack]

Reformatted to xfs and transferred data back to the cache drive.  All is well now.

[JorgeB]

16 hours ago, JoshTheBlack said:

but there are some BTRFS errors in the disk log I attached

Those aren't errors, it's because of the balance.