[SOLVED] Confused on importing Files

[whitewraith]

Maybe I have read to much info that I cant sort it out in my head. I have a three disk array. I created a share called Plex Media info and using windows created in that share a Movies Folder and a TV Folder. I used Krusader to transfer Movies and TV shows into their respective "folders". Now that wrote all the data to one drive on the array and it is almost filled. I have another 5+ terabytes to add of movies and TV shows. If I add them to the same respective Movie and TV folders will the data that wont fit on the first drive just write to the next drive? Or will it fail at transferring the data due to lack of disk space. I read somewhere that you should not transfer to shares but to drives. When I transferred a couple of files to the third disk in the array it made the Movie folders into shares.  I am really confused and think I may have messed up out of the gate here. Help a Noob get better. 

 

[Gragorg]

It really depends on how the split level is setup on your user share.   Most people around here seem to setup one user share for Movies and another for TV shows.  If you click on the SHARES tab you can click on your user share and see what the allocation method and split level is.  Better yet post your diagnostics file.

Edited July 31, 2020 by Gragorg

[whitewraith]

8 minutes ago, Gragorg said:

It really depends on how the split level is setup on your user share.   Most people around here seem to setup one user share for Movies and another for TV shows.  If you click on the SHARES tab you can click on your user share and see what the split level is.  Better yet post your diagnostics file.

Here are both things you asked for:

, I have one share with one folder for Movies and one for TV.  Split only the top lvl directory as required.

mediahog-diagnostics-20200730-2027.zip

[Gragorg]

• Your Allocation Method is High-water which will try and fill one disk right up before moving to the next.  I use most-free which will take the disk with the most room left.  You can change it if you wish.  Here is some information on allocation methods.
• https://wiki.unraid.net/index.php/Un-Official_UnRAID_Manual#Allocation_method

[whitewraith]

8 minutes ago, Gragorg said:

• Your Allocation Method is High-water which will try and fill one disk right up before moving to the next.  I use most-free which will take the disk with the most room left.  You can change it if you wish.  Here is some information on allocation methods.
• https://wiki.unraid.net/index.php/Un-Official_UnRAID_Manual#Allocation_method

I changed it to most free but it still writes to the most full disk. I even said min free space is 500GB and it still hitting that dist. Do I have to reboot to have the changes take effect?

 

[Gragorg]

Also you will need to change your split level.  Right now you can only split the top directory.  If you add more episodes of a TV show they would want to go to the same disk.

 

https://wiki.unraid.net/Un-Official_UnRAID_Manual#Split_level

 

You probably want top three levels of directories.  If you have a seasons folder then your may want top four.

Edited July 31, 2020 by Gragorg

[whitewraith]

7 minutes ago, Gragorg said:

Also you will need to change your split level.  Right now you can only split the top directory.  If you add more episodes of a TV show they would want to go to the same disk.

 

https://wiki.unraid.net/Un-Official_UnRAID_Manual#Split_level

 

You probably want top three levels of directories

AMEN Brother/Sister!!!! It is working finally!!!!  Thanks so much. I settled on Fill-up min free space 500GB, and split top three.

 

[Gragorg]

Good to hear

 

[whitewraith]

Well this has led me to another question.  I am using Krusader to transfer my files. Krusader just filled up my second drive even with the corrections I made yesterday.  Of course Krusader said cant copy anymore to drive no space. When I am transferring files am I suppose to transfer only the amount each drive can hold, I thought that unraid would just use all the drives so therefore Krusader would just switch to the next drive in line with space?  This seems weird to me that it is so hard to copy files from mounted drive to unraid array. 

 

I think that part of my problem is I have TV and Movies in the same share. At this point how hard would it be to make a TV share and a Movie share? Can I just make the shares and point them to the proper folder? Then remove the all encompassing Plex Media share I used first. 

Edited July 31, 2020 by whitewraith

[trurl]

Krusader, rsync, and many other things begin the transfer by creating the empty destination folders first. Those will likely all go to one disk since those empty folders don't take much space. Then the actual files are copied to those already created folders, with the result that they all try to go to the disk the empty folders were created on.

 

Also, this is not correct:

17 hours ago, Gragorg said:

High-water which will try and fill one disk right up before moving to the next.

Filling one disk before moving to the next is the Fill Up allocation method.

 

The default (for good reasons) High Water allocation method is more complicated. It is a compromise between spreading the files to other disks "eventually" while not constantly switching between disks as might happen when using

17 hours ago, Gragorg said:

most-free which will take the disk with the most room left.

  

All of this is explained in the already linked wiki:

17 hours ago, Gragorg said:

https://wiki.unraid.net/index.php/Un-Official_UnRAID_Manual#Allocation_method

 

[whitewraith]

"Fill-Up

The fill-up allocation method simply attempts to fill each disk in order from the lowest numbered disk to the highest numbered disk. The fill-up allocation method must be used in conjunction with the minimum free space setting. Otherwise, unRAID will begin to give disk full errors and not allow any more transfers once the first disk gets close to being full"

 

So the minimum free space is what raid looks at before it attempts to write the folders and the data. So correct me if this is wrong. If I set the minimum free space to 500GB if there is 506GB, Unraid will attempt to write everything to that disk because unraid doesnt know the total size of what I asked it to copy.

[trurl]

Minimum Free is important regardless of what Allocation Method you use. Its importance is just more obvious when using Fill-Up.

 

Unraid has no way to know how large a file will become when it chooses a disk for it. You should set Minimum Free to larger than the largest file you expect to write to the share. If a disk has less than minimum, Unraid will choose another when it is choosing a disk for a new file.

 

But this isn't really what I was talking about with Krusader, rsync, etc. In those cases, the destination folders for the files are created in advance, and then the files are written to those folders. This is really about the way those applications work. As far as Krusader, etc. knows, the destination user share is all one space and it will try to put everything on that space, starting with precreating the destination folders and won't worry about running out of space until it does. But since those precreated empty folders don't take much space, they all get created on one disk.

 

Here is the way I handled this when I was I was using rsync to load my backup server. I saw that the folders had already been created all on one disk, and I knew all the files wouldn't fit on that one disk. I also saw that it was filling those folders in alphabetical order. I just started at the other end of the alphabet and began moving empty folders to other disks so that the files for those folders got written to those other disks.

 

Another way would be to transfer the data in smaller amounts, so that on later transfers the folders would get created on other disks.

 

Or you could just transfer directly to specific disks, but you would still have to be aware of how much a single disk could hold and not start a transfer that would overfill a disk.

[trurl]

Just now, trurl said:

Or you could just transfer directly to specific disks, but you would still have to be aware of how much a single disk could hold and not start a transfer that would overfill a disk.

When transferring files directly to specific disks, they still end up in user shares. The user shares are simply the aggregate of the top level folders on cache and array. The top level folder name is the same as the share name. So if you have a Media share, for example, you could have several disks each with a top level folder named Media, and all the files in all those Media folders are part of the Media share. Doesn't matter whether you transferred those files to the Media share, or you transferred them to the top level folder named Media on a particular disk, still part of the Media share.