Hard links not working as expected


Recommended Posts

I noticed that my Media share (to which Deluge downloads, and Sonarr/Radarr create hard links to corresponding folders for Plex to read in from) was roughly double the size I was expecting it to be when looking at unRAID Web GUI. I looked into it a bit more and it seems that hard links aren't working as expected, and instead are creating new inodes (despite appearing to point to the same file).

Example:

/mnt/user/Media/Test# touch foo
/mnt/user/Media/Test# ln foo bar
/mnt/user/Media/Test# ls -li
total 0
660391 -rw-rw-rw- 2 root root 0 Dec 22 09:15 bar
660390 -rw-rw-rw- 2 root root 0 Dec 22 09:15 foo

It was my understanding that hard links worked fine within a user share, but this appears to be creating new inodes. If I do the same thing on /mnt/diskX/... it works, so is something possibly misconfigured with my share?

Link to comment
6 hours ago, Liam_Galt said:

I noticed that my Media share (to which Deluge downloads, and Sonarr/Radarr create hard links to corresponding folders for Plex to read in from) was roughly double the size I was expecting it to be when looking at unRAID Web GUI. I looked into it a bit more and it seems that hard links aren't working as expected, and instead are creating new inodes (despite appearing to point to the same file).

Example:


/mnt/user/Media/Test# touch foo
/mnt/user/Media/Test# ln foo bar
/mnt/user/Media/Test# ls -li
total 0
660391 -rw-rw-rw- 2 root root 0 Dec 22 09:15 bar
660390 -rw-rw-rw- 2 root root 0 Dec 22 09:15 foo

It was my understanding that hard links worked fine within a user share, but this appears to be creating new inodes. If I do the same thing on /mnt/diskX/... it works, so is something possibly misconfigured with my share?

 

Your 'ls -li' command is listing "pseudo" inode numbers in the fuse file system.  The two names indeed point to same file.  Example:

root@Test1:/mnt/user/test# echo "foo" >foo
root@Test1:/mnt/user/test# ln foo bar
root@Test1:/mnt/user/test# cat foo
foo
root@Test1:/mnt/user/test# cat bar
foo

 

Link to comment

I understand that the contents are the same, it's the storage calculation that is the concern. If I create a file and then create a hard link to it, the space appears to be calculated when viewed in both Shares and Main (for whatever disk the file and its link are on). Is this expected behavior? What happens if the calculations of these links exceeds my actual storage space?

Link to comment
5 hours ago, Liam_Galt said:

I understand that the contents are the same, it's the storage calculation that is the concern. If I create a file and then create a hard link to it, the space appears to be calculated when viewed in both Shares and Main (for whatever disk the file and its link are on). Is this expected behavior? What happens if the calculations of these links exceeds my actual storage space?

That should not be the case.  Are you sure you really have hard links?  You can look at the inode numbers on the /mnt/disk<N> where the files exist to verify.

Link to comment
On 12/22/2018 at 12:39 PM, Liam_Galt said:

to which Deluge downloads, and Sonarr/Radarr create hard links to corresponding

Regardless if your downloads and your movies / tv shows are in the same user share, if you are passing through multiple paths to Sonarr / Radarr (one for downloads (/mnt/user/media/downloads and /mnt/user/media/movies) and then are attempting to hard link from /downloads to /movies it won't work because at that point you are attempting to traverse a hardlink over two different mount points (/downloads and /movies) which isn't allowed, even though physically they both exist within the same user share.

Link to comment
On 12/22/2018 at 6:04 PM, limetech said:

Your 'ls -li' command is listing "pseudo" inode numbers in the fuse file system.

Ah! I think this explains the hard link struggles I'm experiencing when attempting to de-dupe files using rmlint.

 

So, @limetech, do I understand correctly that:

  1. If I make a hard link in a User Share (e.g. in /mnt/user/test), it will indeed make a hard link on the Device (e.g. in /mnt/disk3/test), but...
  2. The shfs FUSE file system generates "pseudo" inode values, and therefore...
  3. There's no way to detect hard links in User Shares--I can only detect them by examining the inodes of files on Device(s)?
Link to comment
3 hours ago, bland328 said:

Ah! I think this explains the hard link struggles I'm experiencing when attempting to de-dupe files using rmlint.

 

So, @limetech, do I understand correctly that:

  1. If I make a hard link in a User Share (e.g. in /mnt/user/test), it will indeed make a hard link on the Device (e.g. in /mnt/disk3/test), but...
  2. The shfs FUSE file system generates "pseudo" inode values, and therefore...
  3. There's no way to detect hard links in User Shares--I can only detect them by examining the inodes of files on Device(s)?

All correct.

 

Here's the conundrum:  It is possible in 'shfs' to pass the inode number of the file on the actual device it's located on.  However, recall that inode numbers must be unique within a file system, and linux considers entirety of 'mnt/user/' a single file system.  It's possible there could be files on multiple disks that have the same inode number.  Back when hard link support was added, we added a change so that 'shfs' would form an 'st_ino' field that consisted of the actual inode number of the file along with the disk number - this made 'st_ino' unique.  BUT we started getting reports of all kinds of NFS-attached media players all of a sudden not seeing all the files within user shares.  After much debugging and actually looking at source code for one of these players, it turned out that our 'st_ino' field sometimes was greater than 32 bits (the field itself is 64 bits), this combined with a bug in the player NFS client code resulted in the issue we were seeing.  The compromise reached was to revert our change to pass the actual inode values.

Link to comment
  • 2 years later...
On 12/28/2018 at 2:31 PM, limetech said:

All correct.

 

Here's the conundrum:  It is possible in 'shfs' to pass the inode number of the file on the actual device it's located on.  However, recall that inode numbers must be unique within a file system, and linux considers entirety of 'mnt/user/' a single file system.  It's possible there could be files on multiple disks that have the same inode number.  Back when hard link support was added, we added a change so that 'shfs' would form an 'st_ino' field that consisted of the actual inode number of the file along with the disk number - this made 'st_ino' unique.  BUT we started getting reports of all kinds of NFS-attached media players all of a sudden not seeing all the files within user shares.  After much debugging and actually looking at source code for one of these players, it turned out that our 'st_ino' field sometimes was greater than 32 bits (the field itself is 64 bits), this combined with a bug in the player NFS client code resulted in the issue we were seeing.  The compromise reached was to revert our change to pass the actual inode values.

 

Is there a way that this behavior could be added as a preference? I don't use any of these media players and would like to have consistent inode numbers within user shares rather than inode numbers generated incrementally.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.