September 28, 20241 yr Ongoing consumption of /mnt/cache, but I can't figure out where the space is going. The numbers do not add up. root@laffy:/mnt/cache# du -sbh * 11K T_Media 138G appdata 28M plex-app-media 248M smokeping root@laffy:/mnt/cache# du -sh * 12K T_Media 118G appdata 28M plex-app-media 249M smokeping root@laffy:/mnt/cache# df -h /mnt/cache Filesystem Size Used Avail Use% Mounted on /dev/sdd1 466G 317G 148G 69% /mnt/cache The du command says about 150-170GB is in use, but output from df says 317GB is in use. There are no hidden files in that mount point either. I'm completely baffled by what is going on here. Why do I have a 140GB discrepancy? laffy-diagnostics-20240927-1816.zip
September 28, 20241 yr Community Expert du doesn't work correctly with btrfs, df shows the write usage, you can also check it on the pool page: Free (estimated): 147.98GiB (min: 147.98GiB) Free (statfs, df): 147.98GiB
September 29, 20241 yr Author I’m away from computer right now on vacation. I will check further when I’m back. The utilization was still recognized system wide as applications that monitor it stopped believing there was no space available. Do I need to account for some kind of overhead? This is confusing.
September 29, 20241 yr Community Expert The space availed should be what the GUI reports, if you are seeing something different there may be other issues, post new diags after you get a no space issue.
October 7, 20241 yr Author I'm back home now and checking usage. The problem persists. Output from diskfree: root@laffy:/mnt/cache# df -h /mnt/cache Filesystem Size Used Avail Use% Mounted on /dev/sdd1 466G 330G 136G 71% /mnt/cache That's another 13GB gone in the past 9 days. WebUI: The growth the past 30 days: I can't wait for the system to reach 100%. Several applications are already configured to shutdown if /mnt/cache gets to a critical threshold to prevent other applications from crashing.
October 7, 20241 yr Author Attaching a fresh set of diagnostics. I understand what you are saying how du can misrepresent disk usage on btrfs. However, I'm unable to resole the ongoing problem with increasing disk utilization. There must be some method to find out how this space is getting allocated before it reaches 100%. I need a proactive solution. laffy-diagnostics-20241006-2151.zip
October 7, 20241 yr Author Some additional output after reading the man page for btrfs: root@laffy:/mnt/cache# btrfs filesystem du -s /mnt/cache/* Total Exclusive Set shared Filename 12.00KiB 12.00KiB 0.00B /mnt/cache/T_Media 115.60GiB 115.60GiB 4.00KiB /mnt/cache/appdata 43.97MiB 43.97MiB 0.00B /mnt/cache/plex-app-media 248.09MiB 248.09MiB 0.00B /mnt/cache/smokeping root@laffy:/mnt/cache# du -sbh * 11K T_Media 137G appdata 44M plex-app-media 248M smokeping root@laffy:/mnt/cache# btrfs filesystem df /mnt/cache Data, RAID1: total=463.73GiB, used=328.11GiB System, RAID1: total=32.00MiB, used=80.00KiB Metadata, RAID1: total=2.00GiB, used=457.98MiB GlobalReserve, single: total=229.95MiB, used=0.00B The numbers are still not adding up.
October 7, 20241 yr Community Expert df adds up perfectly with what the GUI shows, you just need to find what is using the space, do you have any vdisks on that filesystem?
October 7, 20241 yr Community Expert Given the share names it sounds like you'd be downloading new stuff? That's how cache's typically used, then you schedule the mover to move things to the array. Edited October 7, 20241 yr by Kilrah
October 7, 20241 yr Author 5 hours ago, Kilrah said: Given the share names it sounds like you'd be downloading new stuff? That's how cache's typically used, then you schedule the mover to move things to the array. Yes, I do have a few shares where data is stored first on cache and then moved daily to array. That process is running normally.
October 7, 20241 yr Author 6 hours ago, JorgeB said: df adds up perfectly with what the GUI shows, you just need to find what is using the space, do you have any vdisks on that filesystem? How does that add up? How does 115G = 328G used?
October 7, 20241 yr Community Expert 37 minutes ago, snowboardjoe said: How does that add up? How does 115G = 328G used? 330GIB = 354GB
October 7, 20241 yr Author You're missing my point. Btrfs tools show that only 135GB is in use. What is the df output hundreds of GB's off?
October 7, 20241 yr Community Expert 10 minutes ago, snowboardjoe said: Btrfs tools show that only 135GB is in use. Where do you see that? btrfs also reports the same used: 14 hours ago, snowboardjoe said: used=328.11GiB
October 7, 20241 yr Author This is what I have right now on my system. I'm focusing just on the /mnt/cache mount point here. root@laffy:/mnt/cache# btrfs filesystem df /mnt/cache Data, RAID1: total=463.73GiB, used=349.81GiB System, RAID1: total=32.00MiB, used=80.00KiB Metadata, RAID1: total=2.00GiB, used=460.05MiB GlobalReserve, single: total=224.84MiB, used=0.00B root@laffy:/mnt/cache# btrfs filesystem du -s /mnt/cache Total Exclusive Set shared Filename 136.58GiB 136.58GiB 4.00KiB /mnt/cache DF reports 349GiB used. DU reports 136GiB used. That's a massive variance. How do I find what is filling up /mnt/cache? When DF reaches 100% I'm dead in the water. That's what I'm trying to prevent. I'm 71% of the way there already and it keeps rising constantly. Normally I would use du to help find directories and subdirectories to isolate growth, but that is not working here as none of them show any significant usage or growth (nothing that would explain a 215GiB variance). What preventative measures can I take to to find and isolate what is eating up disk space before I'm out of space in /mnt/cache? Edited October 7, 20241 yr by snowboardjoe
October 7, 20241 yr Community Expert 24 minutes ago, snowboardjoe said: What preventative measures can I take to to find and isolate what is eating up disk space ncdu works fine for me.
October 8, 20241 yr Community Expert du doesn't provide correct results with btrfs, df, the GUI, and the btrfs stats will.
October 8, 20241 yr Author 3 hours ago, JorgeB said: du doesn't provide correct results with btrfs, df, the GUI, and the btrfs stats will. I'm fully aware of that. I've been managing Linux systems for decades. Restating this point is not helping the situation here. What would be helpful is for you to answer my direct questions above. If you can't then let someone els provide assistance.
October 8, 20241 yr Community Expert The easiest way to find would be move shares one by one to the array then see how much free space is freed, btrfs usage can grow with time for some files, especially for files like vdisks or image type files, if you have some, a defrag can help with those, if you are not using snapshots, but I would try the move first to see which share(s) is using more than you expect.
October 8, 20241 yr Author I also thought about making a copy of /mnt/cache into some scratch space and evaluating things as well. Seems more like an issue with brtfs and it's frustrating. I've had to troubleshoot problem like this for years, but only with more traditional filesystem types. This really eludes me. I have a lot of directories in /mnt/cache/appdata. I was hoping the du command under the suite of brtfs subcommands would help, but just confirm what du was already telling me.
October 9, 20241 yr Author This is interesting. I just inserted a spare drive into my system and formatted as xfs. I then rsync'd /mnt/cache to the new drive. Then I checked output from df for this: root@laffy:~# df -h /mnt/cache /mnt/disks/WD-WX21DA735946/ Filesystem Size Used Avail Use% Mounted on /dev/sdd1 466G 334G 132G 72% /mnt/cache /dev/sdc1 5.5T 177G 5.3T 4% /mnt/disks/WD-WX21DA735946 root@laffy:~# df -l /mnt/cache /mnt/disks/WD-WX21DA735946/ Filesystem 1K-blocks Used Available Use% Mounted on /dev/sdd1 488386552 349294220 137669236 72% /mnt/cache /dev/sdc1 5858435620 185293440 5673142180 4% /mnt/disks/WD-WX21DA735946 That's surprising to see the variation there between the filesystems both in terms of disk used and inodes used. Appears brtfs is causing a LOT of overhead for the same data. Not sure how I'm going to find this. I suspect it's the Plex data, but not sure yet.
October 9, 20241 yr Community Expert btrfs supports compression, if you've enabled that you'll get further discrepancies between ways of checking used/free space. Also things can create snapshots and volumes on btrfs that would take space beyond what the "currently visible" files take, like running Docker in "folder" mode.
October 10, 20241 yr Author Compression is off--never enabled. Even if it was enabled, I would not expect the raw data to double in size like it does now. Gong to check if there are any underlying snaphots. UPDATE: No subvolumes found. Going to try offloading directories to see what happens with that mount point. Edited October 10, 20241 yr by snowboardjoe
October 10, 20241 yr Community Expert Solution If you don't have snapshots you can try defrag, but I suspect that if you just copy the data somewhere else, and then copy back, it will use a lot less space, similar to what you are seeing with xfs, though it will likely grow again with time.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.