Questions about Mover

[bkastner]

I am having issues with how Mover moves contents to the array, and I don't understand the behaviour. My server is on Unraid 6.11 rc4 for reference.

 

As background, I have a really large movie library and over the last couple of weeks I've been refreshing it to ensure I have the best 1080p version available vs whatever I may have grabbed 5-10 years ago. I have a ssd cache that my nzbget downloads uses and I am also using that drive as cache for my movies share, so when radarr finishes processing the movie it unders up under \movies on the ssd. I also have hardlinks enabled (though I don't know if this is any part of the issue).

 

I've started using trash guides to help focus on the versions I want, and have a much more detailed file naming convention than I did before. So, when I have a new version of a movie it doesn't overwrite the old file. It seems to delete the old file, and the new file is initially found in \movies on the cache drive as noted. 

 

This is all fine, however when mover kicks in to start copying the data to the array it behaves in a way I don't understand.

 

Even though I have the share configured for most-free mover seems to target a specific disk and just keep filling it rather than distributing across the array. Even though I have 12 data disks, it only seems to target 3 of them... which are now the most full and theoretically should never be picked. One of the drives is actually failed, shows as failed in the gui and I have it as an exclusion in the movies share, but mover will still copy to it for some reason. I did post about this a few weeks back and was told if a movie exists on a disk already then the new copy will be moved to that same disk under the existing folder, which makes sense, but this is not what's happening. Mover seems to pick a single disk (one of the 3 noted) and push everything to that one disk.

 

I had 3TB of data on my cache that I needed to move to the array today... Mover picked the most used disk and filled it to 100% capacity (even though I have 500GB free space as a minimum). Once it maxed that out it moved to one of the other 2 disks that get used.

 

I dont' know if once Mover starts using a disk it will move everything to that disk (rather than distribute based on most-free), but I would have expected it to at least respect the minimum free space setting, as well as the disk exclusion (ideally I'd hope Unraid would not write to a failed disk at all, but this may be asking too much).

 

I am attaching images of my movies share settings as well as the array (as well as diags). Mover is running right now, and it maxed out Disk10 (again, well beyond the minimum free disk space), and is now writing to disk5 (one of it's other favorites), even though there are other disks in the array with much more free space.

 

I understand the theory that if a movie exists it will use the existing folder, however I am refreshing 200-300 movies at a time, and there is no way they all resided on a single disk.

 

I'm really confused, and not sure how to fix.

 

As an aside, I did post somewhat about this earlier this month, but the disk in question failed, and I let this linger, and now have more information, so thought I'd start a new thread. I marked the other as solved, but it's really not.

 

Any insights would be appreciated. I usually think I understand how Unraid works, but then situations like this pop up that prove me wrong.

 

 

cydstorage-diagnostics-20220919-1930.zip

Edited September 19, 2022 by bkastner

[Frank1940]

2 hours ago, bkastner said:

As an aside, I did post somewhat about this earlier this month, but the disk in question failed, and I let this linger, and now have more information, so thought I'd start a new thread. I marked the other as solved, but it's really not.

I would be worried about disk 8 being disabled with all of the write activity that you are currently subjecting this system to. I looked at the SMART report for this disk and it might be good but I am not really that sure.  (Disks are always disable if there is a write failure and from that point on, the disk is emulated.  An Emulated disk can and will be written to.  So this requires that a physical disk ---new one or the old one, if good--- be rebuilt.)  If you have another failure, you could loss data on two drives.  Do you have another disk that you could rebuilt disk 8 onto?  In any case, I am going to ping @JorgeB as he is a real Guru about disk problems. 

[bkastner]

Thanks for the thoughts. I just received the replacement drive from WD today and am pre-clearing it as we speak. I also have 2 parity drives so trust I should be good even with another failure (though I would rather not confirm that theory).

 

I guess an emulated disk is still a 'disk' but would think it would be a safety feature to not write to an emulated disk, in case, as you mention, a second drive dies and you don't have 2 parity drives. You are increasing the amount of data at risk. However, I would have thought excluding disk8 from the share (for exactly that reason) would prevent the disk from being written to, but it hasn't. I see files written today, yesterday and the day before... even though it's been over a week since I excluded it after the failure.

 

Hopefully Jorge can provide some insights into why this is happening.

[itimpi]

@bkastner  you have a very restrictive value for the Split Level setting.   In the event of there being contention between the various settings for the share about which disk to select for a new file, then the Split Level is the one that wins.    This can force a file to be written to a specific drive regardless of the other settings such as allocation method or minimum free space.

 

it is also worth pointing out that if a file already exists then a file is always updated in situ on the same drive as where it already exists.

[Kilrah]

5 hours ago, bkastner said:

I guess an emulated disk is still a 'disk' but would think it would be a safety feature to not write to an emulated disk, in case, as you mention, a second drive dies and you don't have 2 parity drives.

Preventing writes to the emulated disk would be pretty much useless since if you only have one parity drive then another failed disk would need that first, failed disk to be working to rebuild, but it's not. Only solution would be to disable the array completely.

Edited September 20, 2022 by Kilrah

[JorgeB]

1 hour ago, itimpi said:

In the event of there being contention between the various settings for the share about which disk to select for a new file, then the Split Level is the one that wins.

This, most likely the problem.

[bkastner]

What are the recommendations for split level? I have a root movies share and tv share (i.e. not under \media or anything).  For movies, historically, there were always jpg files for the movie, actors, etc, so I had this set to ensure they all sat on a common disk. I could see where the restriction would cause issue with tv shows as each season will try to ensure it's on the same disk, which can  (and has) caused space issues. However, with movies, it seems to just be the movie file nowadays, and Plex finds all the metadata, so if it's a single file per movie, how is this too restrictive? I'm not sure I understand (though maybe my understanding of the split level set is incorrect).

 

As for files being replace in situ... that is what I'd expect, and some of my confusion. I will end up with 100-150 movies in the \movies cache folder that mover will need to move. As mentioned, it seems that the original file is deleted, and the only copy of the updated movie sits on cache.. however presumably the folder still exists on the array somewhere, and I would expect the new movie to be placed in the existing folder, which would distribute across the array (to some degree). The fact that it always pegs the same 3 disks and maxes them out rather than distributing across the array is not what I'd expect. There is no way those 150 movies all are from the same disk.. and even if they were by fluke, I've refreshed 4,500 movies at this point with the majority trying to write to one of three disks. 

[Kilrah]

58 minutes ago, bkastner said:

The fact that it always pegs the same 3 disks and maxes them out rather than distributing across the array is not what I'd expect. There is no way those 150 movies all are from the same disk..

 

If you have a "media" share with "Movies", "TV" etc subfolders and you set "split top level only" then once say the "Movies" folder is made on one disk everything that is going to be written to that folder or any subfolders will end up on that disk, you've disallowed unraid from creating a "Movies" folder on other disks and putting files there. Deeper level does the same further levels down. 

 

So as mentioned it's likely the split level that you've set disallows unraid from using the other drives for a new file placed at the folder depth it's being created at.

 

Hard to say much more without knowing how your files are organised in the shares, are the movies in subfolders etc

Edited September 20, 2022 by Kilrah

[trurl]

1 hour ago, bkastner said:

As for files being replace in situ... that is what I'd expect, and some of my confusion. I will end up with 100-150 movies in the \movies cache folder that mover will need to move.

Also note that if the file already exists on the array, it won't be cached, it will be replaced on the array.

[bkastner]

1 hour ago, Kilrah said:

 

If you have a "media" share with "Movies", "TV" etc subfolders and you set "split top level only" then once say the "Movies" folder is made on one disk everything that is going to be written to that folder or any subfolders will end up on that disk, you've disallowed unraid from creating a "Movies" folder on other disks and putting files there. Deeper level does the same further levels down. 

 

So as mentioned it's likely the split level that you've set disallows unraid from using the other drives for a new file placed at the folder depth it's being created at.

 

Hard to say much more without knowing how your files are organised in the shares, are the movies in subfolders etc

I don't use a media share... I have root shares for Movies & TV and they span all the disks. This is where I was confused about it being restrictive. I am not sure what change I am supposed to make to help this in this scenario

[bkastner]

1 hour ago, trurl said:

Also note that if the file already exists on the array, it won't be cached, it will be replaced on the array.

This is only if the file name is exactly the same though, correct? As mentioned in my original post there is a zero % chance of this. Up until this refresh project I just used movie.mkv, but now use movie, quality, audio setup, imdb link, release group, etc. Before this project I didn't use Radarr... I manually cleaned up movies, fixed metadata with Media Companion and then moves to Unraid. I am now trying to automate this and narrow my focus on finding the best quality version that meets my criteria.

 

So, the folder would pre-exist, but the movie file name doesn't as it seems to get deleted once the new file is in the movies folder on my cache drive. I would still sort of expect that if the folder exists on disk2 for example that the new movie would be put into the same folder.... but this doesn't look like is what's happening. 

 

As I think more about this I likely need to do a test with a specific movie and follow it through the process. Right now I've just been searching all movies in a given year, and letting the 300-400 movies update as a single batch.

[trurl]

10 minutes ago, bkastner said:

only if the file name is exactly the same

If any of the paths for the file is exactly the same. I say it that way because a user share path is ultimately a disk path. /mnt/disk1/share1/file1 is the same as /mnt/user/share1/file1, for example, but /mnt/user/share2/file1 is different.

 

I often say "mover won't move duplicates", but a better way to say that and more like the actual code is "mover doesn't replace files". This can happen if cache is readonly and so a file doesn't get removed from cache after being copied to the array. Can also happen if a file is created directly on cache.

[Kilrah]

29 minutes ago, bkastner said:

I don't use a media share... I have root shares for Movies & TV and they span all the disks. This is where I was confused about it being restrictive. I am not sure what change I am supposed to make to help this in this scenario

Probably relax the split setting as mentioned, but we don't know your subfolder structure so hard to comment.

[kizer]

Mine is as follows

Media/Movies/Movie-Title/Movie-File

Media/TV/Season/TV-File

 

Now I'm only running one Share called media and I have it set to split at the third level

 

I could only Imagine if your Not running the Media Folder you could get a way with 2 on TV which would allow the Seasons to be split all over the array, but would keep all files for each season together. 

 

Here's a Link to the Manual about it. 

https://wiki.unraid.net/Manual/Shares#Split_level

 

[bkastner]

I still tend to leave it at split level 1 for tv as I'd prefer to have everything related to a show in a single folder on a disk, but there is no real reason I suppose. My brain just likes things ordered like that.

 

1 hour ago, Kilrah said:

Probably relax the split setting as mentioned, but we don't know your subfolder structure so hard to comment.

 

Movies\Movie Name (year)\Movie Name is how everything is structured, and with metadata not downloading it's only a single file in a given movie folder, so there is really nothing to split beyond the first level.

[bkastner]

So, I haven't done the one off test yet, but here is the picture I posted yesterday of the drives, and then again from just now. Not absoutely everything uses those 3 disks, but it looks like the vast majority does.

 

Yesterday:

 

Today:

[Frank1940]

Here is a quick spread sheet of the 'Free' Storage:

 

It looks to me that you had activity on six of nine disks. 

 

Have you actually looked at the cache disk(s) to see what is there before Mover runs.   I suspect you may find that existing files are being overwritten as they are updated.

[bkastner]

35 minutes ago, Frank1940 said:

Here is a quick spread sheet of the 'Free' Storage:

 

It looks to me that you had activity on six of nine disks. 

 

Have you actually looked at the cache disk(s) to see what is there before Mover runs.   I suspect you may find that existing files are being overwritten as they are updated.

 

While I do see a number of disks have been written to, it's interested that I've gained almost half a TB on Disk4... presumably because the old version of a movie was deleted and the new version went to another disk (Other than mover I don't think I've touched any files). Though I am guessing, I don't know why else space would have gone up and not down.

 

I do look at the cache disk. I'm constantly monitoring it to see what's been completed, what's still showing incomplete, etc. So right now I have a 173 movies in the Movies folder on the cache drive. Because the file names are different I don't think it's overwritting. It would treat them as seperate files (I've had movie duplicates this way before by accident). 

[Kilrah]

19 minutes ago, bkastner said:

Because the file names are different I don't think it's overwritting.

The file name is different, but is it getting put in the same folder? As mentioned the folder counts, if it is it'll end up on the disk where the folder already exists if you've got the split level restricted.

Edited September 20, 2022 by Kilrah

[kizer]

1 hour ago, bkastner said:

I still tend to leave it at split level 1 for tv as I'd prefer to have everything related to a show in a single folder on a disk, but there is no real reason I suppose. My brain just likes things ordered like that.

 

 

Movies\Movie Name (year)\Movie Name is how everything is structured, and with metadata not downloading it's only a single file in a given movie folder, so there is really nothing to split beyond the first level.

 

That's just it. Since your asking it NOT to split beyond that everything is always going to the same drive. 

meaning if you tell it to store Movies on Disk 1 then it'll never go beyond disk 1. 

 

I used to be really bad at making sure all my seasons of a show are in the same Tv Folder. Now I'm just making sure all of the files from say Lucifer Season 1 remain together. I don't care about Season 2 or Season 3 as long as ALL episodes from a Season stay together.  When you drill down to TV or Movie shares it displays them all together, because unraid is doing all the magic in the back end. Plus Plex, Emby or whatever you use don't care either.   

[bkastner]

1 minute ago, kizer said:

 

That's just it. Since your asking it NOT to split beyond that everything is always going to the same drive. 

meaning if you tell it to store Movies on Disk 1 then it'll never go beyond disk 1. 

 

I used to be really bad at making sure all my seasons of a show are in the same Tv Folder. Now I'm just making sure all of the files from say Lucifer Season 1 remain together. I don't care about Season 2 or Season 3 as long as ALL episodes from a Season stay together.  When you drill down to TV or Movie shares it displays them all together, because unraid is doing all the magic in the back end. Plus Plex, Emby or whatever you use don't care either.   

 

 

So.. this server has been running for a decade with these exact same share settings, and there are Movies folders on each of the 12 data disks and this has never been an issue before. When manually copying to the Movies share the files can go to any disk. As a test I just disabled the cache on my movies share so it writes directly to the array. I picked a movie at random that was on disk8 and moved it to my local machine, and then moved it back... and it ended up on disk9

 

So, is it fair to say that share settings/split level are not the issue I am having right now?

 

 

[Frank1940]

@bkastner, please provide a chart like this one showing the folder layout that you are using and the value of the current split level that you have currently specified. 

 

 

(You can do a screen capture of this one (or the attachment)  and just mark it up if you want.)

[bkastner]

Here is the diagram as well as a screenshot from my Movies share. The TV share is set up identically.

 

 

[Frank1940]

OK --- now for how I think it works.  

 

Assume a setup a Movie Folder named    300_(1917)   and it is generated on disk 3.   Because the Split level is "Automatically split only the top level directory as required", that is only time this folder will generated on the array.  Any files that are put into this folder will automatically be written to that disk and folder.  This will happen regardless of the Allocation method or the "Minimum Free Space" setting!   (In fact, the write can actually fail if the file is larger than the available space available on the drive.)

 

Why does it work this way?  Let's go back to when this 'feature' first came into Unraid.  Many folks were ripping DVD's and the resulting video files were .VOB files which are limited to 1GB in size.  A feature-length movie would have from three to nine of these files which were played in sequence.  What was happening (remember this was back in the days when a 100GB disk was the breeding edge of the disk technology) was that these files would be split between two disks.  Suddenly, the movie would stop playing (sometimes in the middle of word) for several seconds until the next file could be accessed on the second disk.  The split level parameter was a Kludge solution to this problem. 

 

Unfortunately, this tool also fell victim to many people's OCD behavior pattern where they want organization for something that probably does not need organization.  (I, personally, completely fail to understand why people instill that all of the episodes of a single season of a TV show has to be on one disk--- that is what you use a season directory structure.  A delay of a few seconds to start the next episode should not detract from the enjoyment of a binge watcher.  Plus, if you use the "High Water" allocation scheme, they probably end up on the same disk anyway if you are using a modern high capacity hard disk.) 

[bkastner]

That all makes sense, and yes... it's OCD behaviour around the TV shows. I know it really doesn't matter, and should likely get over it.

 

Even with this explanation I don't understand why those 3 disks I mentioned are being targetted excessively... I understand and expect some data to be written there, but not like it has. Also, I have 500GB minimum free space required set on those disks... even if it's writing to movie folders there already I could see it using an extra 25-50GB or even 100GB, but disk 10 is down to 552MB of space and disk5 is 193GB of space... both well under that threshold. That still confuses me.