I NEED HELP ...

[SiRMarlon]

So I am hoping someone can shine some light on storage allocation because my system is doing something weird. 1st off this is a brand new Unraid server running 6.7.2.

I currently have 3 drives in the array, one 8TB WD-Red, and two 4TB WD-Reds for a total of 16TB. I created one media share that utilizes the array. The Media share has two folders, Movies, and TV_Shows. 

 

Yesterday I began the process of copying over all my media from my old hard drives. I copied over one TV show folder which was about 2.8TB and I noticed that it began putting the data in one of the 4TB drives. I thought this was a bit odd as I figured it would use the 8TB as it's the 1st disc in the array. It copied over the data just fine (Using Krusader btw and I am going disk to share) and then I began coping over the next folder of TV Shows which was about 3.2TB. Now I had that media share setup as "High Water" "Split Top Level Directory as required" and I had 250GB as the minimum free space.

 

I thought high water fills up the drive about half way and then moves on to the next drive? Because it did not, it maxed out the 4TB drive and then my file copying stopped! I got an error from Krusader. It did not move any data to the next drive at all. I am not sure if I did anything wrong. I followed the instructions that Spaceinvader one has on his video in regards to moving data.

 

Anyway since it didn't work, and I was just copying and not moving data. I decided to pretty much reformat the drives and re-do the array with encryption. I recreated the Media share, again trying highwater and put 1TB as Minimum Free space. I did that and started copying over data again I've  noticed that this time it is actually copying data to the 8TB, but it's gone past the halfway mark and it's pretty much filling up the whole drive again. Am I not using the correct allocation method? Did I forget to enable something? Run a special plug in?

 

My current setup for the array. No parity as I am copying data. I will be using the 10TB as the parity and the two 4TB WD-Red I am currently pre-clearing will be added to the array once they are done.

 

https://ibb.co/wKq99x1

 

This is how I setup the share the 2nd time around... again using "High Water" and someone suggested I use "Split only the top two directory" 

 

https://ibb.co/gT5zrJr

 

so far this space issue is the only thing that has given me any grief. The rest of the server and my dockers are running like a freaking CHAMP!! I just need to get this data disc space thing figured out and I will be golden. 

 

Any insight or help you could provide would be great!

Edited December 14, 2019 by SiRMarlon

[JorgeB]

Split level overrides allocation method, set split level to "split any directory".

[itimpi]

2 hours ago, SiRMarlon said:

8TB WD-Red, and two 4TB WD-Reds

High-water allocation uses the size of the largest drive in deciding when to potentially switch drives.   In your case with a 8TB drive present the first switch point is 4TB.  When all drives get down to this the next switch point at 2TB is activated.   Since the 8TB drive has this much free space Unraid will continue to use it until it gets down to 2TB free, and when that happens it will start using the 4TB drives.  When all drives get down to 2TB free the next switch point of 1TB will be activated so Unraid will go back to using the 8TB drive.

 

There is a feature request raised requesting a new allocation method similar to high-water but based on percentage of drive used rather than basing it on the absolute size of the largest drive but whether this will ever get implemented I have no idea.

[trurl]

Since you are using Krusader I am guessing you are transferring from Unassigned disks instead of over the network. 

 

One problem that happens with Krusader and rsync and other applications, it will create all of the folders first, and they will all go on one disk since high water won't be reached. Then it copies the files into those folders that have already been created. 

 

One workaround is to do the transfer in smaller chunks. When it happened to me with rsync, I noticed what was happening and that it was going in alphabetical order, so I just started at the other end of the alphabet and began moving the empty folders to other disks before it started filling them. 

[SiRMarlon]

10 hours ago, johnnie.black said:

Split level overrides allocation method, set split level to "split any directory".

 

I would but I kinda of want to keep semblance of structure in place. 

 

8 hours ago, itimpi said:

High-water allocation uses the size of the largest drive in deciding when to potentially switch drives.   In your case with a 8TB drive present the first switch point is 4TB.  When all drives get down to this the next switch point at 2TB is activated.   Since the 8TB drive has this much free space Unraid will continue to use it until it gets down to 2TB free, and when that happens it will start using the 4TB drives.  When all drives get down to 2TB free the next switch point of 1TB will be activated so Unraid will go back to using the 8TB drive.

 

There is a feature request raised requesting a new allocation method similar to high-water but based on percentage of drive used rather than basing it on the absolute size of the largest drive but whether this will ever get implemented I have no idea.

 

Okay that makes completely sense, but for some reason my system is not doing any of that. But I think what Turl said makes sense as to why my system is doing what it's doing and it might be more because I am using Krusader. But yes a new allocation method base on percentage of the drive used would be great! 

 

6 hours ago, trurl said:

Since you are using Krusader I am guessing you are transferring from Unassigned disks instead of over the network. 

 

One problem that happens with Krusader and rsync and other applications, it will create all of the folders first, and they will all go on one disk since high water won't be reached. Then it copies the files into those folders that have already been created. 

 

One workaround is to do the transfer in smaller chunks. When it happened to me with rsync, I noticed what was happening and that it was going in alphabetical order, so I just started at the other end of the alphabet and began moving the empty folders to other disks before it started filling them. 

 

Yes I am going straight from unassigned devices to the share, and yes! I noticed that Krusader would first create ALL the folders of the data I am copying over which I thought was a little strange as I thought it would create the folders as it was moving the data over. 

 

Anyway the data I was copying over last night ended up filling the drive the 8TB drive and I got errors but it looks like we are on to something with what Trurl has mentioned. I think what I am going to do is rename the current share to TV Shows since all that media copied over successfully to the drive. I am going to created another share called Movies set the split levels that I want to use since the shows require a different split level then the movies and copy over data little by little see if that does anything. I will report back once I am done or if I run into issues again. 

 

Thanks again for all your help and insight! I really do appreciate it! 

[SiRMarlon]

I just want to report back on this issue. Looks like the best thing to do is copy the data over in chunks, by doing this I can see now that the data is being copied over to different drives now. So for anyone in the future who might run into this problem, just copy your data over little by little so you don't end up filling up your drive like I did. Thanks again to everyone who chimed in.