TLDR: If you use XFS this affects YOU. 100GB is lost on a 18TB drive. This is confirmed as reserved metadata space for a feature unusable by Unraid.  This is a known issue with the Unraid XFS format settings.
--
Are you guys seeing gigabyes of used space on an empty disk? This unused XFS reflink feature consumes a huge amount of disk space, ~70GB on a 14TB drive.

 

Demonstration: https://serverfault.com/questions/983907/newly-created-xfs-filesystem-shows-78-gb-used

Quote

# df -h loop/

Filesystem Size Used Avail Use% Mounted on

/dev/loop0 11T 78G 11T 1% /dev/shm/loop (reflink=1, crc=1)

/dev/loop0 11T 12G 11T 1% /dev/shm/loop (reflink=0, crc=1)

/dev/loop0 11T 33M 11T 1% /dev/shm/loop (reflink=0, crc=0)

 

You can check if this effects you:
  xfs_info /dev/md1p1

please post diagnostic.

I don't see this in testing on stable or beta 7.
did you use the unraid array to make the xfs partition?

2 hours ago, bmartino1 said:

did you use the unraid array to make the xfs partition?

Yes.

 

Quote

please post diagnostic. I don't see this in testing on stable or beta 7.

I'm on stable 6.12.14, I can start a fresh install if you'd like a diagnostic. Does your "xfs_info /dev/mdXXp1" on your box show reflink=1?

FYI I'm runing v7 RC2 atm...

/dev/nvme# as the actual device and xfs partiton not the raid array for /dev/md that is called vai mdadm and sofftware riad.



I'm gpt aline fomrated as well using a ssd/nvme:


may be HDD only ....

while i do see xfs reflink=1 i don't see any data used up:

 

In the context of XFS, the reflink=1 (or reflink1) option in the xfs_info output indicates that reflink (reference linking) is enabled on the file system.

 

What is Reflink in XFS?

Reflink is a feature in XFS that allows for data deduplication and copy-on-write (COW) functionality. It enables multiple files to share the same physical blocks of data on disk while appearing as separate files in the file system. When one of the files is modified, the modified data is written to a new location, leaving the original file unchanged (this is the copy-on-write behavior).

 

Key Benefits of Reflink:

Efficient Cloning:

Reflink makes it possible to create lightweight file copies (clones) without duplicating the data. This is faster and saves disk space.

Data Deduplication:

Identical data blocks can be shared between files, reducing redundant storage usage.

Snapshot-like Functionality:

Modifications to files do not overwrite shared data blocks, enabling snapshot-like behavior for individual files.

Improved Performance:

File cloning and deduplication operations are performed without the need for expensive data copying.

Example Use Cases:

Backups: Creating point-in-time backups or snapshots of files without consuming additional disk space.

Virtualization: Efficiently cloning virtual machine images.

Versioning: Storing multiple versions of large files efficiently.

Limitations:

Reflink is only available on XFS file systems formatted with the appropriate feature flags (it cannot be retroactively added to existing file systems).

Some tools or workflows may not fully support reflink-enabled XFS file systems.

 

I'd have you check xfs options like xfs_spaceman and check as again the /dev/md ### is a mdadm software and that could be the cache across disk ...

Edited December 29, 20241 yr by bmartino1
Data - Typo

Reflink support was added years ago (not long after XFS added it), and is generally of benefit... but if you want the space back the steps are shown in this post

 

Edited December 29, 20241 yr by tjb_altf4

... 🙄

go recaculte your xfs...

ls -lh file1 file2
du -h file1 file2

Use the xfs_metadump and xfs_mdrestore commands to capture and analyze the metadata of the file system for more advanced space usage analysis. These require deeper knowledge of XFS internals.

37 minutes ago, tjb_altf4 said:

Reflink support was added years ago (not long after XFS added it), and is generally of benefit... but if you want the space back the steps are shown in this post

 

OK, so this is a known issue. Looks like reformatting is required to reclaim the space.

 

I imagine Unraid is using the defaults. As there's no use case (that I'm aware of) for reflink/COW support this might be a bug.

1 hour ago, bmartino1 said:

while i do see xfs reflink=1 i don't see any data used up:

 

Thanks for your reply, I see you have the reflink "stolen" space. The missing space is subtracted from the "SIZE" column of your screen shot of your spinning drives.  It's wasted file system overhead.

Edited December 29, 20241 yr by cuimalo

46 minutes ago, bmartino1 said:

... 🙄

go recaculte your xfs...

ls -lh file1 file2
du -h file1 file2

Use the xfs_metadump and xfs_mdrestore commands to capture and analyze the metadata of the file system for more advanced space usage analysis. These require deeper knowledge of XFS internals.

Thanks for helping me chase this down. This is wasted filesystem overhead, they will not show with "du" or "ls".  I've cited a source in my original post.

I'm familiar with those XFS tools, how can they be applied here?

Edited December 29, 20241 yr by cuimalo

Without the diagnostic file, and based on the data provided, here are some tools and commands I’ve used on Debian/Arch Linux to work with disk data and identify disk blocks for various services. XFS has been my go-to file system, so this advice primarily pertains to it.

From what I can observe, it doesn’t appear that a significant chunk—like 70 GB—is being consumed on the 14 TB disk. Based on the information presented, only about 4 GB of the 4 TB appears to be cached data.

Commands like ls and du are useful for confirming file existence, but the actual data written to and read from a disk doesn’t necessarily align with the block usage on the disk itself. I can't definitively determine what was stored, is stored, or is currently being written to the disk. With XFS, partial remnants of previous data may remain due to optimizations for performance.

The other XFS-related commands I mentioned earlier are helpful for troubleshooting and analyzing disk usage in the absence of a diagnostic file. They aim to gather additional information and verify disk behavior.

I don’t doubt that 70 GB might be listed as in use, but I suspect it’s more likely tied to cache or metadata rather than actual user data on the disk. Running additional XFS commands to clear the cache may help address this.

It’s been a while since I primarily used XFS, as I now use ZFS. However, I have an Unraid test system with an XFS partition, and I wanted to share what I could to assist.

One final note: I use SSDs for storage, which may behave differently compared to HDDs, particularly due to trim operations and differences in disk architecture. These considerations might affect the observed behavior.

 

....

The xfs_metadump and xfs_mdrestore tools are valuable for diagnosing and analyzing XFS file systems, especially in complex cases involving issues like excessive disk usage or unexplained space consumption (e.g., 70GB on a 14TB disk). Here's how these tools can be helpful in this scenario:

 

Diagnosing the Issue with xfs_metadump

What it does: xfs_metadump creates a metadata dump of an XFS file system. This dump includes metadata information (like inode structures, directory layouts, and free space maps) but excludes actual file data for privacy and size efficiency.

How it helps:

Inspecting Reflinked Files: If XFS reflink is enabled, it allows multiple files to share the same physical blocks until they are modified. This can lead to confusion over disk usage. xfs_metadump can provide insight into which inodes or blocks are being referenced multiple times due to reflinks.

Identifying Metadata Overhead: The dump can reveal if excessive metadata or directory structures are contributing to the apparent high space usage.

Spotting Corrupt or Orphaned Inodes: It helps detect issues like orphaned or corrupt inodes, which might falsely report space usage.

 

Restoring and Analyzing with xfs_mdrestore

What it does: xfs_mdrestore can reconstruct a usable XFS file system from the xfs_metadump metadata file. This restored file system is placed on a different disk or image file, not impacting the original disk.

How it helps:

Simulating Repairs: You can run XFS repair tools (xfs_repair, xfs_check) on the restored file system in a non-destructive environment to understand what issues exist and how they affect space usage.

Testing Changes: If the problem is related to XFS-specific configurations (like reflink), you can test adjustments to these settings in the restored file system without risking data on the production disk.

 

*Since we are dealing with software, raid with unraid. and unraid Slackware is using a fuse os. this may be a side effect from fuse so reviewing its relationship:
 

In Relation to FUSE and Reflink

Reflink and Disk Usage: Reflink allows for efficient block sharing, but tools like du and df may not accurately reflect the actual data usage because they can't easily distinguish between shared and unique blocks. xfs_metadump can help pinpoint how many blocks are shared via reflinks.

FUSE Layer: If a FUSE-based application or file system (e.g., mergerFS, rclone) is interacting with the XFS disk, it may also introduce additional overhead or caching layers that inflate perceived usage. Metadata dumps could reveal if the FUSE layer is generating extra inodes or directories.

some side troubleshooting on dumping the block and inspecting / repairing example:
 

#create a dump
xfs_metadump /dev/sdX metadump.img
#restore for analysis
xfs_mdrestore metadump.img restored.img
#inspect / repair
mount -o loop,ro restored.img /mnt/test

--Run XFS-specific tools like xfs_repair or analyze block usage with xfs_bmap. But that getting into advanced xfs tools and CLI.

General troubleshooting, nothing more. It was more taking a deeper dive and trying to come up with a possible solution to your problem. 
https://eng.libretexts.org/Bookshelves/Computer_Science/Operating_Systems/Linux_-_The_Penguin_Marches_On_(McClanahan)/04%3A_Managing_Linux_Storage/4.11%3A_File_System_Management/4.11.02%3A_File_System_Management_xfs_Tools

 

1 hour ago, tjb_altf4 said:

Reflink support was added years ago (not long after XFS added it), and is generally of benefit... but if you want the space back the steps are shown in this post

 

This does appear to be a known bug / issue and is not fixed in unraid v7 rc2

Edited December 29, 20241 yr by bmartino1
Data - typo

12 minutes ago, bmartino1 said:

I suspect it’s more likely tied to cache or metadata rather than actual user data on the disk

Exactly! It's reserved metadata space for a feature unusable by Unraid. 

 

I respect the effort inherent in your response, this issue could affect the majority of users. I will set up a testing bed and get back to you. 

 

On ZFS: I'm hesitant to use it with Unraid as there are too many corner cases, though I may build out a "Perfect Media Server" in future.

Edited December 29, 20241 yr by cuimalo

This issue is WAY worse than I thought, 100GB of wasted space!!

 

 

This is a brand new blank 18TB Exos X20, model ST18000NM003D. Formatted through Unassigned Devices, then from the CLI without reflink.

 

Please tone it down on the attitude and stay on-topic from now on. I've hidden the posts in question.

8 minutes ago, Rysz said:

Please tone it down on the attitude and stay on-topic from now on. I've hidden the posts in question.

I don't follow, what's going on? Mod removed unrelated bad attitude posts. 

 

Thanks for the work on SnapRAID and mergerFS.

Edited December 29, 20241 yr by cuimalo

Bug report opened. 

Hey Unraid friends! I need your help. Please comment on the bug report so we can get this fixed.

I've turned the topic into a feature request for better visibility, seeing as this seems to be a design decision.

Please stop bumping and/or re-opening the bug report continuously if there's no new information/participation.

 

Thanks for your participation! Let's keep this as a bug until we get clarification. 

  

1 minute ago, cuimalo said:

>  a change in package defaults ... occurred in XFS tools 5.1.0.

Hey @Rysz, I think you missed this new information.

Can you confirm that this was an intentional design decision? It's doubtful as as other supported filesystems, like ReiserFS, do not support reflinks or COW.

 

I'm trying to understand why this is so important for you, block cloning is a very nice feature, I've seen multiple users using it with xfs over the last few years, to for example, make a quick backup of a vdisk before trying any changes, or make a quick VM clone, and other uses.

 

Would it make sense to remove that feature, that is available in most modern filesystems, just to recover like 0.5% of capacity?

 

If you really don't want to use reflinks, just format the disk with Unraid and then re-format manually using

 

mkfs.xfs -m crc=1,finobt=1,reflink=0 -f /dev/mdXp1

 

You only need to this once for any new disk, and it would remain without reflinks for ever, even if the disk was later upgraded to a larger one.

 

 

> mkfs.xfs ... reflink=0 

As in my initial post. 
 

> just to recover like 0.5% of capacity?

This is 10 TB on a 10 disk array with 20TB disks. Yes. 

 

> I'm trying to understand why this is so important for you

Unraid is an appliance, not a Linux distro. The point of an appliance is that it works as preconfigured, it should not be tampered with. I have simply and repeatedly asked confirmation that reflink=1 was an intentional design decision. If so disabling it may break functionality or future updates. Uptime = profit. Downtime = cost.

 

> block cloning is a very nice feature ... make a quick backup of a vdisk ... and other uses.

Running VMs on an array is not recommended by Unraid. 0.001% of users need this feature, but all will miss the disk space. 

On 1/7/2025 at 12:57 AM, JorgeB said:

AFAIK it was a choice so that reflinks can be used

Nope. It's a default. 

 

On 11/23/2019 at 9:34 AM, limetech said:

If another option, 'crc=1' is set (which we do set since feature has been available) then reflink=1 is the default.

Note that the above is incorrect as it was really a change to XFS tools in 5.1.0.

Edited January 10, 20251 yr by cuimalo

2 hours ago, cuimalo said:

Nope. It's a default. 

You asked if it was an "intentional design choice" and the reply was "... it was a choice"

10 hours ago, cuimalo said:

This is 10 TB on a 10 disk array with 20TB disks. Yes. 

0.5% of 200TB is 1TB, not 10.

 

10 hours ago, cuimalo said:

it should not be tampered with

Unraid is using the default XFS settings, and they have been the default for a long time, if you type just mkfs.xfs /dev/sdX, it will create a filesystem with reflinks enabled.

 

10 hours ago, cuimalo said:

Running VMs on an array is not recommended by Unraid

That is a good point, I was considering XFS in general, for array and pools, but an argument could be made for leaving reflinks enabled for pools and disable them for the array, probably not many users using reflinks in the array.

 

My only reservation is that, a new user, that may not know about reflinks and would want to use them in the future would need to backup the existing data and re-format the disk, while a user that formats a new disk, sees the overhead and doesn't like it, can find the issue with a quick forum search and re-format the still empty disk without reflinks.

 

One other interesting issue, I was testing exactly how much overhead is used by reflinks, and with the current kernel in 7.0.0, the crc feature uses much more overhead than reflinks, and you don't want to remove that, since they are important for data integrity and creating an XFS filesystem without them uses the deprecated v4 format, which will be unsupported in the future.

 

To get 0 overhead with XFS you need to disable the crc option:

 

 

 

 

The default, and what Unraid uses, has crc and relinks enabled:

 

 

 

Finally, with crc still enabled, but reflinks disabled

 

 

 

Used both a 1TB and 4TB disks, to show that the overhead is proportional to the disk capacity.

 

Since crc=0 is not really an option, I personally still think that it may not be worth disabling reflinks, even if it's just for the array devices, you just recover around 6GB per TB, but if other users would prefer that, please post in this thread and LT may consider changing it.

 

 

 

 

 

Excellent response, thank you! 

 

On 1/10/2025 at 4:59 AM, JorgeB said:

0.5% of 200TB is 1TB, not 10.

 

My mistake and bad math.

 

> with the current kernel in 7.0.0, the crc feature uses much more overhead than reflinks, 

That's interesting, perhaps the overhead for reflinks has changed. 

 

Edited January 15, 20251 yr by cuimalo

I found this post while trying to understand why my brand new 20TB drives, freshly formatted in Unraid 7 with XFS defaults, were starting off with 383GB allocated - much higher than the documented expectation of 0.7%.

 

I found the analysis here very useful, and tested formatting with reflinks disabled, and am now at 263GB allocated (confirming reflinks consumes 6GB per TB). I'm still not sure if I need reflinks or not - from the discussion here it seems I likely don't for my use case.

 

But what isn't answered here is why crc alone is roughly double the documented allocated size (for 20TB it seems I should be around 140GB). I wonder if perhaps there was an enhancement with XFS crc in Unraid 7, as I don't see many older posts showing anything close to this.

 

Has anyone else been able to figure this out?