Quick High Water Question


Recommended Posts

Let's say I have two 8TB disks, disk 1 is 90% full and the other is empty.  If I move and rename a file stored on disk 1 stored at /mnt/user/share1/file.txt to /mnt/user/share2/new name.txt, will it move it quickly on disk 1 between shares and ignore the high water rule, or will it move and copy (slower) to disk 2, obeying the high water rule?

 

I'm getting into bottlenecks with renaming and moving downloads, and I'm wondering if my setup where I keep imports, movie_adults, movie_kids, tv_adults etc on different disks is causing problems i.e.  a new TV show is currently going /mnt/disk1/imports/show1.mkv to /mnt/disk2/TV shows/show_renamed.mkv which takes time to copy across, whereas mnt/disk1/imports/show1.mkv to /mnt/disk1/TV shows/show_renamed.mkv should be faster if unraid always moves a file on the same disk even if another disk has more space.

 

Thanks

Link to comment
2 hours ago, DZMM said:

If I move and rename a file stored on disk 1 stored at /mnt/user/share1/file.txt to /mnt/user/share2/new name.txt, will it move it quickly on disk 1 between shares and ignore the high water rule, or will it move and copy (slower) to disk 2, obeying the high water rule?

Depends how you do it.  If you move it via the network, then it will obey high water.  If you do it via command line, mc, krusader etc then it won't

Link to comment

Thanks.  I'm hoping the changes I've made to not exclude disks from shares, or at least matching them up, and also to turn hardlinks on so hopefully less copying, will improve my import speeds as my backlog is approaching a couple of days which is annoying when you know something is there, but plex can't see it yet.

Link to comment

Thanks.  I think I've got my flows setup the way I want and optimised:

 

nzbget: 

  1. downloads to /mnt/user/downloads on disk 4 - /mnt/downloads/ set to disks 4 only
  2. extracts to /mnt/user/import on disks 1-7 based on High Water choosing a disk.  99% of the time this shouldn't be on disk 4 as this has the least amount of space, so extraction should be faster as not occurring on same disk (I'm tempted to do downloads to an UD to ensure this is always the case)
  3. sonarr/radarr/lidarr/lazylibrarian move files to respective media shares.  HW rules should mean this is on the same disk that was chosen for the import, so just a rename and not a copy so fast

Deluge:

  1. downloads direct to /mnt/user/import on disks 1-7
  2. sonarr/radarr/lidarr/lazylibrarian copys/hardlinks files to respective media shares.  When hardlinks works should be fast as no copy

The bit I'm not sure about yet, is for some of my files the media share is a unionfs mount e.g. for kids_movies it's 

 

/mnt/user/movies_kids_upload=RW:/mnt/disks/rclone_vfs/movies_kids_gd=RO /mnt/disks/unionfs_movies_kids
unionfs -o cow,allow_other,direct_io,auto_cache,sync_read 

so sonarr/radarr etc are moving the file to /mnt/disks/unionfs_movies_kids i.e. not a /mnt/user location, although in reality the file behind the scenes is being written to /mnt/user/movies_kids_upload (rclone move job then moves it to google drive mounted at /mnt/disks/rclone_vfs/......). 

 

Do the dockers/unraid know this and still follow the HW rule i.e. the files are just renamed rather than copied?  I'm assuming hardlinks won't work as the docker will think /mnt/disks/ is on a different filesystem?

Link to comment
12 minutes ago, DZMM said:

Do the dockers/unraid know this and still follow the HW rule i.e. the files are just renamed rather than copied?  I'm assuming hardlinks won't work as the docker will think /mnt/disks/ is on a different filesystem?

Rather than spend forever trying to understand your workflow and inferring the host/container mappings, here's the rules in a nutshell

 

If you passthrough to the containers 2 mount points, one for downloads, and one for the final destination then:

  • Highwater / include / exclude rules will be honored
  • Hardlinks between downloads and final destinations will not work (because the mountpoints between the source and destination are different)

If you passthrough to the containers a single mount point, and reference downloads and the final destination from it then, (assuming everything in the container winds up referencing /mnt/user/)

 

  • Highwater / include / exclude rules will not be honored (files on the final destination will wind up being on the same disk that the download happened on)
  • Hardlinks between downloads and final destinations will work

My thoughts on hardlinks between a rclone mount and /mnt/user would be that even if you passthrough /mnt mapped to /mnt then it will not work at all as ultimately /mnt/disks and /mnt/user are different filesystems.

Edited by Squid
  • Like 1
Link to comment

Thanks that helped.  tweaking slightly to get desired result of nzb extraction onto different disk to the acutal download then a hardlink to the media folder, and torrents hardlinking for all, except for /mnt/disk rclone moves:

 

nzbget: /downloads mount point --> /mnt/user/import ---> /mnt/user/media

deluge:  /mnt/user/import ---> /mnt/user/media

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.