thatsthefrickenlightning

Sadly, I in turn know very little about Linux and rsync. I came from Windows just a week ago. I do know hardlinks, though. A hardlink is a sort of false copy. A file that points to the same data on the disk that another file is also pointing to. So you've got the same amount of data taking up space, but twice the amount of files. Or even more than twice, of course. What is an application for this? Well, I keep one hardlink in my torrent seeding folder and one in my plex library, for example. That way I can seed, but I also adhere to Plex's rules conventions regarding folder structures. Radarr and Sonarr have options to use hardlinks instead of copies. So what happens when you delete hardlink A, but not B? Then file A is gone, but the data is not. You've gained no hard disk space. If you delete B but not A? Same deal, one less file, no space gain. Delete both, you gain the extra free space. In my use case, my Plex library is all hardlinks. I can delete my entire library and not bat an eye. Because I'll always have the torrent folders where the data still resides. Conversely, if I wanted to, I could also delete my torrents folder and my data would live on in my Plex library. Hardlinks are super useful. Now here's my problem. As you know, moving with Unbalance is essentially 'copy to other drive and delete source'. But as I've explained, if you delete only one of two hardlinked files, you're not gaining any drive space. But you ARE losing drive space on the other drive because you're writing new data to it. This is how Unbalance's 'move' essentially becomes 'copy' when you're dealing with a hardlinked file. I'm no programmer, but I think I can already envision the solution here. There's a command that you can use to easily check if a file is hardlinked. In a terminal in the folder, type 'stat file.ext'. It'll tell you how many links the data has. 2 links = 2 files pointing to the same data. Second, the size will be exactly the same. I think modifying one of these files means modifying the other as well, but I haven't tried that. No need in my use case. I read-only. Filenames can differ. Aside from that, there are perhaps more ways to check for hardlinks. I know dupeGuru does, but idk how. Anyway, when Unbalance moves a file, it could check the file it's trying to move for hardlinks. And if it finds hardlinks, it could opt to move those along with it. So hardlink /user/mediaserver/torrents/file.ext gets moved to another disk, but hardlink /user/mediaserver/plex/film/file.ext gets moved as well. And of course just as with the former, the exact location, directory-wise, is preserved. Perhaps to simplify the process, in reality Unbalance should just create a new hardlink instead of trying to move an existing one, to avoid the risk of the process tripping up and creating a dupe instead of moving hardlinks. So to summarize, I think Unbalance needs to... Check if /disk1/mediaserver/torrents/file.ext has alter-egos in the form of hardlinks If yes (and in this case, it does, in the plex folder), remember the location (folder-wise, not disk-wise) and filename of that hardlink alter-ego Delete this alter-ego/hardlink Copy /disk1/mediaserver/torrents/file.ext to /disk2/mediaserver/torrents/file.ext, then delete /disk1/mediaserver/torrents/file.ext (this is Move as it already exists now) Recreate the hardlink deleted off disk1 on disk2, referencing the newly created file /disk2/mediaserver/torrents/file.ext That way, the pitfall of using Move but not freeing up space on the source disk is avoided. I hope this helps explain my situation. I'm sincerely grateful for this cool tool you've made and I hope you can implement an option that'll help me solve my problem. I'd love to hear what you think. Thanks. EDIT: to add, it appears the Mover respects and conserves hardlinks. I just tested this myself. Downloaded a torrent onto the cache drive, hardlink was automatically created on the cache drive, ran the mover, it moved the two files to disk1, and they are still hardlinks. Perhaps you could simply copy Mover's implementation of hardlink conservation?

Hi Djoss, thanks for the app, I can use a tool to combat dupes on my server! Question though, I've got the 'ignore hardlinks' option checked, but a scan still returns hardlinks in the results. Is this a known bug or am I missing something on my end? Edit: retested. dupeGuru marks a hardlink as a dupe, regardless of the option to ignore. E2: the documentation for dupeGuru says it checks if inodes are the same. Somehow, even though they're hardlinks, the inodes are different. I'm stumped for now.

I just ran into a bit of a snag with this plugin. When moving a hardlinked file with Gather, I'm essentially still copying the data, as it continues to exist on the source drive in form of the other (but same) hardlinked file elsewhere. Does Unbalance have an option to check if files are hardlinked, and if it finds that a file is indeed hardlinked, to opt to move all these linked files to the target drive? I can't find it in the web UI, so I fear this is not an option at this time. If it's added, perhaps this could be a regularly scheduled task so as to fix the Mover creating the same problem.

Alright. To be sure, are you talking about the enable_outgoing_tcp/udp or enable_incoming_tcp/udp?

Sounds like the same problem I described above. Would love a fix to this, as 90% of potential bandwidth is missing. It's the final piece to my server's puzzle.

Any news? I'm having the same problem, even on the QB docker. Edit: fixed this on qbittorrentvpn by switching to tcp only.

Hi Kaalis, did you ever get this fixed? Topping out at 2 MiB/s here.

Thank you for this! Fixed my hardlink problem without too much headache. Summing up for any lurkers/Googlers: I removed all volume mounts in the docker settings except for config aka appdata for both sonarr/radarr and my torrent client. Then I pointed /mnt/ to /mnt/ for both my torrent client docker and radarr/sonarr, and now hardlinks are automatically created succesfully. This can be verified with Krusader by opening a terminal in either your torrents folder or your episode/film folder and typing 'stat *filename*'. Among the output is the Links count, which should say 2, meaning the data is linked to by 2 files in total: a hardlink. Feel free to correct me if I got anything wrong there.