I noticed that my Media share (to which Deluge downloads, and Sonarr/Radarr create hard links to corresponding folders for Plex to read in from) was roughly double the size I was expecting it to be when looking at unRAID Web GUI. I looked into it a bit more and it seems that hard links aren't working as expected, and instead are creating new inodes (despite appearing to point to the same file).

Example:

/mnt/user/Media/Test# touch foo
/mnt/user/Media/Test# ln foo bar
/mnt/user/Media/Test# ls -li
total 0
660391 -rw-rw-rw- 2 root root 0 Dec 22 09:15 bar
660390 -rw-rw-rw- 2 root root 0 Dec 22 09:15 foo

It was my understanding that hard links worked fine within a user share, but this appears to be creating new inodes. If I do the same thing on /mnt/diskX/... it works, so is something possibly misconfigured with my share?

Probably how the fuse filesystem works, whereas /mnt/diskX bypasses fuse and is handled directly from the kernel.  But, as you've said it does work, simply returns the wrong inode and disk usage.  @limetech would have to chime in for the explanation / if it's a bug

Edited December 22, 20187 yr by Squid

6 hours ago, Liam_Galt said:

I noticed that my Media share (to which Deluge downloads, and Sonarr/Radarr create hard links to corresponding folders for Plex to read in from) was roughly double the size I was expecting it to be when looking at unRAID Web GUI. I looked into it a bit more and it seems that hard links aren't working as expected, and instead are creating new inodes (despite appearing to point to the same file).

Example:

/mnt/user/Media/Test# touch foo
/mnt/user/Media/Test# ln foo bar
/mnt/user/Media/Test# ls -li
total 0
660391 -rw-rw-rw- 2 root root 0 Dec 22 09:15 bar
660390 -rw-rw-rw- 2 root root 0 Dec 22 09:15 foo

It was my understanding that hard links worked fine within a user share, but this appears to be creating new inodes. If I do the same thing on /mnt/diskX/... it works, so is something possibly misconfigured with my share?

 

Your 'ls -li' command is listing "pseudo" inode numbers in the fuse file system.  The two names indeed point to same file.  Example:

root@Test1:/mnt/user/test# echo "foo" >foo
root@Test1:/mnt/user/test# ln foo bar
root@Test1:/mnt/user/test# cat foo
foo
root@Test1:/mnt/user/test# cat bar
foo

 

I understand that the contents are the same, it's the storage calculation that is the concern. If I create a file and then create a hard link to it, the space appears to be calculated when viewed in both Shares and Main (for whatever disk the file and its link are on). Is this expected behavior? What happens if the calculations of these links exceeds my actual storage space?

5 hours ago, Liam_Galt said:

I understand that the contents are the same, it's the storage calculation that is the concern. If I create a file and then create a hard link to it, the space appears to be calculated when viewed in both Shares and Main (for whatever disk the file and its link are on). Is this expected behavior? What happens if the calculations of these links exceeds my actual storage space?

That should not be the case.  Are you sure you really have hard links?  You can look at the inode numbers on the /mnt/disk<N> where the files exist to verify.

I just did some testing and hard links appear to be working when created manually, it just seems to be with Sonarr that things aren't working as expected. Feel free to disregard anything here.

On 12/22/2018 at 12:39 PM, Liam_Galt said:

to which Deluge downloads, and Sonarr/Radarr create hard links to corresponding

Regardless if your downloads and your movies / tv shows are in the same user share, if you are passing through multiple paths to Sonarr / Radarr (one for downloads (/mnt/user/media/downloads and /mnt/user/media/movies) and then are attempting to hard link from /downloads to /movies it won't work because at that point you are attempting to traverse a hardlink over two different mount points (/downloads and /movies) which isn't allowed, even though physically they both exist within the same user share.

On 12/22/2018 at 6:04 PM, limetech said:

Your 'ls -li' command is listing "pseudo" inode numbers in the fuse file system.

Ah! I think this explains the hard link struggles I'm experiencing when attempting to de-dupe files using rmlint.

 

So, @limetech, do I understand correctly that:

1. If I make a hard link in a User Share (e.g. in /mnt/user/test), it will indeed make a hard link on the Device (e.g. in /mnt/disk3/test), but...
2. The shfs FUSE file system generates "pseudo" inode values, and therefore...
3. There's no way to detect hard links in User Shares--I can only detect them by examining the inodes of files on Device(s)?

3 hours ago, bland328 said:

Ah! I think this explains the hard link struggles I'm experiencing when attempting to de-dupe files using rmlint.

 

So, @limetech, do I understand correctly that:

1. If I make a hard link in a User Share (e.g. in /mnt/user/test), it will indeed make a hard link on the Device (e.g. in /mnt/disk3/test), but...
2. The shfs FUSE file system generates "pseudo" inode values, and therefore...
3. There's no way to detect hard links in User Shares--I can only detect them by examining the inodes of files on Device(s)?

All correct.

 

Here's the conundrum:  It is possible in 'shfs' to pass the inode number of the file on the actual device it's located on.  However, recall that inode numbers must be unique within a file system, and linux considers entirety of 'mnt/user/' a single file system.  It's possible there could be files on multiple disks that have the same inode number.  Back when hard link support was added, we added a change so that 'shfs' would form an 'st_ino' field that consisted of the actual inode number of the file along with the disk number - this made 'st_ino' unique.  BUT we started getting reports of all kinds of NFS-attached media players all of a sudden not seeing all the files within user shares.  After much debugging and actually looking at source code for one of these players, it turned out that our 'st_ino' field sometimes was greater than 32 bits (the field itself is 64 bits), this combined with a bug in the player NFS client code resulted in the issue we were seeing.  The compromise reached was to revert our change to pass the actual inode values.

22 hours ago, limetech said:

The compromise reached was to revert our change to pass the actual inode values.

That's painful, but makes perfect sense. NFS implementations that only consider 32 bits of a 64-bit value are why we can't have nice things

 

Thanks for the explanation!

Interesting breakdown in the cause of the issues for those who like tech details. I suppose it's not worth the development and quality assurance time to have it be a user configurable behavior?

On 12/28/2018 at 2:31 PM, limetech said:

All correct.

 

Here's the conundrum:  It is possible in 'shfs' to pass the inode number of the file on the actual device it's located on.  However, recall that inode numbers must be unique within a file system, and linux considers entirety of 'mnt/user/' a single file system.  It's possible there could be files on multiple disks that have the same inode number.  Back when hard link support was added, we added a change so that 'shfs' would form an 'st_ino' field that consisted of the actual inode number of the file along with the disk number - this made 'st_ino' unique.  BUT we started getting reports of all kinds of NFS-attached media players all of a sudden not seeing all the files within user shares.  After much debugging and actually looking at source code for one of these players, it turned out that our 'st_ino' field sometimes was greater than 32 bits (the field itself is 64 bits), this combined with a bug in the player NFS client code resulted in the issue we were seeing.  The compromise reached was to revert our change to pass the actual inode values.

 

Is there a way that this behavior could be added as a preference? I don't use any of these media players and would like to have consistent inode numbers within user shares rather than inode numbers generated incrementally.