copy question for array/cache

[DotJun]

I was wondering if anyone could help me understand what unraid/mover will do in these cases. Let's say that I have a 3 disk array and split level is set for tv series so that the series can span across drives, but seasons and files contained will only go into 1 disk. Let's also say that I have chosen the setting to fill up 1 disk at a time.

 

1. (Not a copy/move question but I'm curious) If a series season is split across 3 disks so that season 1 is on disk 1 and so fourth, when I watch s01e01 will it spin up all the disks or just the disk that episode is located on (I'm sure the answer will be that it depends on how your client accesses these files e.g. plex/kodi/etc)?

 

2. If I'm copying a season directly to the array, cache disabled, and the disk that is being written to fills up before the transfer is finished, will I get an out of space error or will unraid finishing copying the files to the next disk? If the remaining files are copied to the next disk, will unraid balance that folder afterwards according to the split level rule set in place?

 

3. Same question as #2, but with cache enabled. For this question though, the folder being copied fits into the cache disks, but when mover activates will it know that the folder cannot fit into a disk and instead choose a disk that it will fit into?

 

4. A series season is on a disk that is now full. What happens if I overwrite some of the files in that season with files that are larger? Disk is full error or will it be placed on a different disk and if it does go to a different disk will unraid do a balance per the split level rule?

 

Thanks for the help guys! New to unraid so I'm not sure how these things work.

[itimpi]

You never mention the Minimum Free Space setting which can be relevant in selecting a drive  

 

As to the questions:

1. Difficult to answer definitively as it can depend on many factors (such as the disk controller), and what Unraid currently has in RAM at the time in terms of directory information.  In ideal conditions only the required drive is spun up.
2. It probably depends on the client.  You will definitely get an out-of-space error for the file that does not fit - whether the others then copy will depend on the client (but they should then go to another drive as long as the share settings allow another drive to be selected.
3. Mover works at the individual file level not the folder level.  At the point it selects a drive a file size of 0 is assumed (and thus the importance of the Minimum Free space setting).
4. If a file already exists then Unraid will always attempt to overwrite it in situ so you get an out-of-space error.   Not quite sure what you meant by Unraid balancing out, but it never moves a file from one array drive to another.

[trurl]

You would need Minimum Free larger than all the files you can't split since split level is going to force those to the same disk .

 

I never bother with split level 

[DotJun]

2 hours ago, itimpi said:

You never mention the Minimum Free Space setting which can be relevant in selecting a drive  

 

As to the questions:

1. Difficult to answer definitively as it can depend on many factors (such as the disk controller), and what Unraid currently has in RAM at the time in terms of directory information.  In ideal conditions only the required drive is spun up.
2. It probably depends on the client.  You will definitely get an out-of-space error for the file that does not fit - whether the others then copy will depend on the client (but they should then go to another drive as long as the share settings allow another drive to be selected.
3. Mover works at the individual file level not the folder level.  At the point it selects a drive a file size of 0 is assumed (and thus the importance of the Minimum Free space setting).
4. If a file already exists then Unraid will always attempt to overwrite it in situ so you get an out-of-space error.   Not quite sure what you meant by Unraid balancing out, but it never moves a file from one array drive to another.

I have min free space set at 75gb, but you pretty much answered all my questions by stating that unraid never moves files from one array disk to another. Due to this, would best practice be to move the whole folder to a different disk if the file I want to overwrite is located on a disk that is full already?

[itimpi]

1 hour ago, DotJun said:

Due to this, would best practice be to move the whole folder to a different disk if the file I want to overwrite is located on a disk that is full already?

Probably, although if you have 75GB set as Minimum Free Space then there is a good chance the larger file will still fit so no action is required.

[Frank1940]

I am wondering why are you concerned about keeping all the episodes of a single season on one disk.  I just timed the delay in loading a blue ISO from a spun-down disk and it was eight seconds! 

 

First, a bit of history, the split-level parameter was added many, many years ago.  Here is the issue that was being addressed.  DVD video files (.VOB) are restricted to being 1GB in size and it takes 4 to 7 of these files to make up a full length movie--- only the last file not being exactly 1GB.   If you saving DVDs in folder-and-file structure and these .VOB files get split up onto two disks, you could have a eight pause in the middle of a word!  NOW, that is a real problem!!!   Thus, the split level came into Unraid as a Kludge solution.  (I do mean Kludge as it can cause some very serious side issues as are being pointed out.)

 

It also happened at a time when there were still Unraid servers with HDs of less than 500GB. (One of mine started out with 250GB parity and I think I had 100Gb data drive.)  The probability of getting 1GB files split across two disks is much higher with those HD drive sizes than with one with a size of 8TB.

 

If you really want a trouble free Unraid life, go with the default Unraid settings with the exception of the 'Minimum free space:' one (mine is set to 50GB).  Don't worry about where files end up.  Don't worry about one disk being fuller than other disk.  These things are really quite travail.  Trying to 'fix' them can cause real problems if you don't do things properly.  (Believe me when I say this.  Every couple of months, someone will screw up a Linux command trying to correct an OCD problem and that causes real problems.)

 

If you really have an issue with a file being on a wrong disk---  address that problem when you identify it.  (As I said, you may not even realize that Unraid has split the files...)    Check to see what is causing the issue and look for a workable solution.  You can probably find the room to move files if you have the 50GB Min Free Space setting in place as you have two disks to work with.   (I would recommend use the Dynamix File Manager as it is designed to work with the Unraid array.)

Edited January 27, 2023 by Frank1940

[DotJun]

On 1/27/2023 at 7:28 AM, Frank1940 said:

I am wondering why are you concerned about keeping all the episodes of a single season on one disk.  I just timed the delay in loading a blue ISO from a spun-down disk and it was eight seconds! 

 

First, a bit of history, the split-level parameter was added many, many years ago.  Here is the issue that was being addressed.  DVD video files (.VOB) are restricted to being 1GB in size and it takes 4 to 7 of these files to make up a full length movie--- only the last file not being exactly 1GB.   If you saving DVDs in folder-and-file structure and these .VOB files get split up onto two disks, you could have a eight pause in the middle of a word!  NOW, that is a real problem!!!   Thus, the split level came into Unraid as a Kludge solution.  (I do mean Kludge as it can cause some very serious side issues as are being pointed out.)

 

It also happened at a time when there were still Unraid servers with HDs of less than 500GB. (One of mine started out with 250GB parity and I think I had 100Gb data drive.)  The probability of getting 1GB files split across two disks is much higher with those HD drive sizes than with one with a size of 8TB.

 

If you really want a trouble free Unraid life, go with the default Unraid settings with the exception of the 'Minimum free space:' one (mine is set to 50GB).  Don't worry about where files end up.  Don't worry about one disk being fuller than other disk.  These things are really quite travail.  Trying to 'fix' them can cause real problems if you don't do things properly.  (Believe me when I say this.  Every couple of months, someone will screw up a Linux command trying to correct an OCD problem and that causes real problems.)

 

If you really have an issue with a file being on a wrong disk---  address that problem when you identify it.  (As I said, you may not even realize that Unraid has split the files...)    Check to see what is causing the issue and look for a workable solution.  You can probably find the room to move files if you have the 50GB Min Free Space setting in place as you have two disks to work with.   (I would recommend use the Dynamix File Manager as it is designed to work with the Unraid array.)

 

The biggest reason that I want all associated files on a single disk is for spin up/down reasons. I don't see the need to spin up multiple disks when I'm watching a single season or having metadata accessed on a separate disk.

 

Here's an issue I just ran into:

1. Split level is set so that folder A can go on any disk, but folder B that resides under folder A has to stay on one disk.

2. Free space is set to 50gb. Disk X has 51gb of free space left and Disk Z has 1tb of free space left.

3. I copied a folder that contained 2 files into folder B (which means that all files in the folder that I am copying has to be put on the same disk).

4. The 2 files are both 50gb in size.

5. I ended up getting an out of free space error.

 

What I think happened is that during the copy the first file fit into Disk X just fine. Unraid then started to copy the second file into the same disk which had run out of space. It chose this same disk due to my split-level rule. Is this correct?

 

1. Is there no way for undraid to check the file sizes of the folder that I am copying into the array?

2. Does this mean that it would be best practice to copy 1 file at a time into the array when my disks are getting close to full?

3. Are there any plugins that can do this if unraid is not able?

4. If I had set the destination of my copy to cache and then have mover move them into the array would I have experienced the same issue?

5. Trying to get a grip on this as I've not had this problem using Windows + Stablebit Drivepool.

 

Thanks for the help guys! Outside of this quirk I am really enjoying unraid so far!

[Frank1940]

1 hour ago, DotJun said:

What I think happened is that during the copy the first file fit into Disk X just fine. Unraid then started to copy the second file into the same disk which had run out of space. It chose this same disk due to my split-level rule. Is this correct?  Yes

 

1. Is there no way for undraid to check the file sizes of the folder that I am copying into the array?  NO!

2. Does this mean that it would be best practice to copy 1 file at a time into the array when my disks are getting close to full? Yes, that is the only way.  But be careful as there is always the file overhead...

 

As I said earlier, the Split Level was a Kludge to address a specific problem.

 

1 hour ago, DotJun said:

5. Trying to get a grip on this as I've not had this problem using Windows + Stablebit Drivepool.

 

And exactly how did this combination address the issue of putting a file on a drive where it would not fit!  (Remember that it is a whole lot easier to handle an error situation where one is only dealing with one OS and no network protocols.) 

[itimpi]

1 hour ago, DotJun said:

It chose this same disk due to my split-level rule. Is this correct?

Yes.   Split Level over-rides all other selection criteria if there is any contention about which drive to use.   If you use restrictive Split Level settings then you need to monitor free space on drives a bit more carefully.

 

probably the best avoidance action is to set the warning levels for each drive to be sufficient to give you time to consider moving some files to another drive (dtive-to-dtive moves bypass User Share settings).

[itimpi]

1 hour ago, DotJun said:

Is there no way for undraid to check the file sizes of the folder that I am copying into the array?

The drive selection is done at a low level when the file is first created (and thus has 0 size) and that level has no knowledge of the final size of the file.   

[Kilrah]

2 hours ago, DotJun said:

2. Does this mean that it would be best practice to copy 1 file at a time into the array when my disks are getting close to full?

In general you always do, since doing 2 parallel copies to the same drive is way slower than queuing the 2.

[DotJun]

19 hours ago, Frank1940 said:

 

As I said earlier, the Split Level was a Kludge to address a specific problem.

 

And exactly how did this combination address the issue of putting a file on a drive where it would not fit!  (Remember that it is a whole lot easier to handle an error situation where one is only dealing with one OS and no network protocols.) 

1. Does this mean that split level is not recommended anymore?

2. Drivepool accomplished it via a balancing plugin that can move files in real-time which is why I also asked if there are plugins for unraid that perform the same function.

[DotJun]

19 hours ago, Kilrah said:

In general you always do, since doing 2 parallel copies to the same drive is way slower than queuing the 2.

What I meant by this was, instead of copying a whole folder with multiple files inside of it that would then copy sequentially, should I instead copy files over singly from that folder?

[Kilrah]

I guess that depends on how your OS behaves. AFAIK from Windows Explorer normally files are created sequentially so one won't be created before the previous one has finished writing, individual applications might behave differently though.

[trurl]

4 hours ago, DotJun said:

balancing plugin

unBALANCE on your Apps page.