(sort-of read-cache) I Just setup rsync to copy certain directories to my 2nd Cache pool. Could use some tips on making this more manageable...

[relink]

literally over 90% of what causes my Array to be spun up is my kids watching the same hand full of movies and shows.

 

So I setup rsync to copy certain directories back to the cache drive, this way they will be on the cache AND the array. So when those shows or movies are watched my array wont need to spin up, but they are also still protected by the array.

 

In this example, the show I tested with is downloaded from Youtube using Tubesync. I want to be able to use the --delete option with rsync so that anything removed from the Array will also get removed from the cache the next time my rsync userscript is ran. However I also realized that if Tubesync grabs a new episode it will place it on the cache drive, and if my userscript runs before the download is finished wouldn't rsync delete the in progress download?

 

Current Rsync command:

rsync -av --progress --delete "/mnt/user/Video/Youtube/video/show" "/mnt/cache/Video/Youtube/video/"

 

I have a couple of solutions but they all have potential issues:

 

1.  just remove the --delete option. But then I would have to remember to manually remove the cache drive copy wouldn't I? or would unraid know to delete both?

 

2. Modify the mover script to exclude those directories and reverse my rsync command. But I really don't want to modify the mover or any "system" files. Im ok if the files get moved from cache to array and then copied back to cache.

 

3. Use "Mover Tuning" to only run my command after the mover finishes. This seems like it may work, but I'm still concerned about in-progress downloads? I'm sure the mover is smart enough not to move them, but is rsync smart enough not to delete them?

 

4. Create separate shares for these files. But I'd prefer not to, I might use this for other files too, and things could get confusing fast.

 

OR maybe I'm overthinking the whole thing and there is a better way to do this...

 

 

 

[JonathanM]

7 minutes ago, relink said:

"/mnt/user/Video/Youtube/video/show" "/mnt/cache/Video/Youtube/video/"

Those 2 paths are the same.

 

You will lose data.

 

Please read all of this thread.

 

[relink]

11 minutes ago, JonathanM said:

Those 2 paths are the same.

Technically yes, that was intentional. The sync only goes 1 way, from the share on the Array > Cache, and I was under the impression that If a file exists on the array, and it also exists in the exact same location on the cache drive that when the mover is ran it will simply fail to move those files and skip them. Correct me if Im wrong.

 

18 minutes ago, JonathanM said:

Please read all of this thread.

I have actually read that before. But in the description of the first post it's talking about "If a user attempts to copy files from a disk included in a user share, to the user share", which is technically not what I am trying to do. I am copying files from a user share to a disk that "technically" isn't directly part of the user share.

 

25 minutes ago, JonathanM said:

You will lose data.

I have tested this with one show so far, and it seems to have worked

 

If I am completely wrong in my thinking of how to accomplish this, then what would be the best way for me to remove the data from the cache disk without risking the data in the share on the array?

[relink]

 

 

[mgutt]

7 hours ago, relink said:

I have tested this with one show so far, and it seems to have worked

 

Do not execute it again. The source path must be /mnt/disk*/Video/Youtube/video/show

 

If you move from /mnt/user, you move from cache to cache as /mnt/user access files on the cache if they already exist there. And because of a bug in FUSE, this kills all your files:

https://github.com/libfuse/sshfs/issues/262

 

[relink]

6 hours ago, mgutt said:

If you move from /mnt/user, you move from cache to cache as /mnt/user access files on the cache if they already exist there. And because of a bug in FUSE, this kills all your files:

Got it, I hadn't considered it from that angle. I only ran it once so far and all my data is still intact. Luckily I had the foresight to only test it on a show that is easily replaced if needed.

 

At this point can I safely delete the copy on the cache disk and not affect the copy on the array?

I think ill take a bit easier of an approach to this for now, and use a spare SSD in unassigned devices for these movies/shows, copy them over, and then add a ".plexignore" file to the originals so Plex doesn't index the array copy and the SSD copy, only the cache copy.

[mgutt]

On 4/5/2022 at 12:56 PM, relink said:

At this point can I safely delete the copy on the cache disk and not affect the copy on the array?

Correct

 

On 4/5/2022 at 12:56 PM, relink said:

I think ill take a bit easier of an approach to this for now, and use a spare SSD in unassigned devices for these movies/shows, copy them over, and then add a ".plexignore" file to the originals so Plex doesn't index the array copy and the SSD copy, only the cache copy.

Create a new share with Cache "Prefer" and move recent shows and movies to it. After you added this share to Plex it automatically re-index the location of the media without deleting it or creating duplicates. Plex is really smart handling moved media.