[v6.10rc4] Storage calculations on Main tab doesn't take account for sparse files

GUI will show the actual used space, except for some pool configurations, like a pool with an odd number of devices in RAID1, also du isn't reliable with btrfs, which I assume is what the pool is using, see if this helps get the space back:

https://forums.unraid.net/topic/51703-vm-faq/?do=findComment&comment=557606

 

6 hours ago, JorgeB said:

GUI will show the actual used space, except for some pool configurations, like a pool with an odd number of devices in RAID1, also du isn't reliable with btrfs, which I assume is what the pool is using, see if this helps get the space back:

https://forums.unraid.net/topic/51703-vm-faq/?do=findComment&comment=557606

 

I originally followed this guide to get it working a couple years ago. The image is sparse and using a virtual scsi controlled. Windows properly picks up the drives as thin provisioned and trims them. The Unraid trims the cache.

 

That’s all working properly. It’s just the size reported in the gui is accounting for the full size of the img, not the sparse used space.

10 minutes ago, nickp85 said:

It’s just the size reported in the gui is accounting for the full size of the img, not the sparse used space.

Unlikely, GUI should be reporting what btrfs is reporting being used, but please post the diagnostics to confirm.

3 hours ago, JorgeB said:

please post the diagnostics to confirm.

Diagnostics attached

nicknas2-diagnostics-20220123-1417.zip

 

Even the QDirstat docker is showing the full allocation vs used size.

Edited January 23, 2022 by nickp85

found some btrfs commands too that appears to show correct file usage and others I thought could be useful. (I changed the directory name to Windows 11 because I upgrade a couple months ago and my OCD got me today). Today GUI says I am using 177GB of space.

btrfs filesystem du /mnt/cache/domains/Windows\ 11/
     Total   Exclusive  Set shared  Filename
  92.36GiB    92.36GiB           -  /mnt/cache/domains/Windows 11/vdisk1.img

 

btrfs filesystem du -s /mnt/cache/
     Total   Exclusive  Set shared  Filename
 118.10GiB    94.99GiB    23.11GiB  /mnt/cache/

 

btrfs filesystem usage /mnt/cache/
Overall:
    Device size:                 953.88GiB
    Device allocated:            332.06GiB
    Device unallocated:          621.81GiB
    Device missing:                  0.00B
    Used:                        329.66GiB
    Free (estimated):            311.50GiB      (min: 311.50GiB)
    Free (statfs, df):           311.50GiB
    Data ratio:                       2.00
    Metadata ratio:                   2.00
    Global reserve:              264.11MiB      (used: 0.00B)
    Multiple profiles:                  no

Data,RAID1: Size:165.00GiB, Used:164.41GiB (99.64%)
   /dev/nvme0n1p1        165.00GiB
   /dev/nvme1n1p1        165.00GiB

Metadata,RAID1: Size:1.00GiB, Used:428.06MiB (41.80%)
   /dev/nvme0n1p1          1.00GiB
   /dev/nvme1n1p1          1.00GiB

System,RAID1: Size:32.00MiB, Used:48.00KiB (0.15%)
   /dev/nvme0n1p1         32.00MiB
   /dev/nvme1n1p1         32.00MiB

Unallocated:
   /dev/nvme0n1p1        310.91GiB
   /dev/nvme1n1p1        310.91GiB

Edited January 23, 2022 by nickp85

Not surprisingly GUI is showing the correct usage, btrfs reports around 165GiB used (or 177GB), and that is the actual used space, like mentioned du isn't reliable with btrfs.

Changed Status to Closed

Changed Priority to Other

That’s still not right, it’s too high for the data that’s on cache. Simply doesn’t add up

It's what btrfs is reporting, and I bet that it's correct, but if it isn't it's not an Unraid problem, at most could be a btrfs issue, you'd need to report it for example in the btrfs mailing list or their IRC channel.

Changed Status to Open

I wanted to bring this issue back open for additional info. I am now on v6.10rc4 and because of read error/corrupt correction lines in my cache disk log (usually happens about once a year and the nvme drives don't have any problems), I used mover to move everything off cache then delete the cache and reformat then finally use mover to put it all back. I have 3 shares that only use cache so I followed these steps.

1. Disabled Docker and VM
2. Set the shares to "Yes"
3. Ran mover and let it complete
4. Verified cache was empty
5. Deleted cache and recreated
6. Set shares to "Prefer"
7. Ran mover and let it complete
8. Change shares back to "Only"
9. Enabled Docker and VM

 

Prior to the move, cache showed 165GB used, after moving back, 125GB used. All my docker containers started normally and my Windows 11 VM booted fine so I'm fairly sure I did not lose any data. Somehow rebuilding the cache pool freed up 40GB according to the GUI.

 

I'm attaching my diagnostics from after I took all of these steps to rebuild my cache.

 

nicknas2-diagnostics-20220326-2357.zip

Pretty sure the mover now supports sparse files, so files like vdisks and docker image are "re-sparsified".

9 hours ago, JorgeB said:

Pretty sure the mover now supports sparse files, so files like vdisks and docker image are "re-sparsified".

Yup, it does, but the files were already sparse. Good to know this method will at least correct the issue