Changing share split level for disk leveling

[Fencer]

Hi, got my system up and it's looking promising. I did seem to make one pretty good blunder though. For my first few shares (comics and movies, both that go several folders deep) I had set the Allocation Method to High Water and the split level to Automatically split only the first 5 directories. Problem I have is that the way I did it, my 12TB disk filled up and now has 760 megs of free space left while 3 of the 6TB drives haven't even been touched. I'm guessing that it's because the split level overrode that 50% threshold thing that High Water is supposed to do?

 

My system has 2x 12TB drives and 7 6TB ones and the array has 1x 12TB disk and 7 6TB ones. I'm using Krusader to add the 30-odd terabytes of data from my mounted usb drives and speed is better than expected. Once I have the files moved to the array, I'll be adding parity with the 2nd 12TB disk and adding in the 1TB nvme drive in the system as my cache drive.

 

I reset the allocation method on those two folders to "automatically split only the top level directories" and am looking to delete 4 folders in the movie share (about 3.5TB) and then reupload them with the new split level and excluding the first drive. Would this achieve what I'm looking for? I really don't like the idea of running a disk at over 90% right from the start. If I'm reading things correctly, I need to get these files spread out over another disk or 3 before adding the parity drive.

 

Thoughts on whether I'm on the right track or way off?

Edited April 30, 2022 by Fencer
clarity and mistaken term

[itimpi]

1 minute ago, Fencer said:

I reset the allocation method on those two folders to "automatically split only the top level directories"

This could well make it worse   The smaller the number of folder levels you allow to be split the less effectively you get balancing across drives.   I am surprised your original value of 5 did not work - that suggests most of your paths have more than 5 folder levels !   Perhaps you should instead allow any number of levels to be split?  It could also be due to the fact that Krusader tends to allocate all folders before moving files which can end up letting the Split Level setting force files to an unwanted drive, and doing the files in smaller batches can help avoid this.

 

In terms of rectifying the current way the drives have been split I would think it faster to do a disk-to-disk move of folders from the 12TB disk to other drives.   Do not do disk-to-user-share or user-share-to-disk as this can lead to data loss unless you know exactly what you are doing.

[Fencer]

That sounds like a plan. I was looking for tools or directions on how to move folders to different disks but all the posts I found were from years ago and I don't know if they work on 6.9. I'm definitely open to moving them disk to disk. If you could point me in a direction I'll be happy to check it out.

 

The first 5TB of folders all went to the first drive and I expected that. I then added the movie folder about 10 genres at a time. The movies folder is broken down like: /movies/movie genre/movie series (if it has one)/movie title/files. The TV Shows folder was added afterwards about 15 series at a time and has a similar structure /TV Shows/series name/season/episode with a few of them probably going another folder in for something or another.

 

I'm seeing that the second half of the movies folder transfers have some files on disk 2 but it doesn't look like it started splitting them until it hit 90%.

Then when I added to the TV Shows share a number of them went on disk 1 as well, bringing it up to 11.2TB used and 760 megs free. It seems to have completely ignored the high water allocation altogether, as I thought it would hit about 6TB on drive 1 and start filling the other drives before coming back to drive 1. Only thing I can think of is that having a setting in the split level setting other than auto overrode the allocation method?

Edited April 30, 2022 by Fencer

[Fencer]

Averaging 160-185MB/s is definitely faster than I had expected, so that's a plus and with a fan blowing on my external, temps are all a nice 37-42 degrees C. Disks 2-3 stopped right at 3TB apiece and it's currently writing to disk 4. Seems like the only ornery one that didn't follow the allocation method is the first (bigger) disk. At it's current pace, it should hit 3TB on disk 4, then 5 and part of 6 in the next 10 hours. 

[itimpi]

10 minutes ago, Fencer said:

I was looking for tools or directions on how to move folders to different disks

If you are using Krusader then it should be easy enough to use that as long as the container volume mapping is such that Krusader has access to /mnt so it can show the disks that are at  /mnt/diskX at the Linux level on Unraid.

 

Not clear from your description why things did not work as expected the first time though as it sounds as if your Split Level setting of 5 was compatible with your paths.

[Fencer]

1 minute ago, itimpi said:

If you are using Krusader then it should be easy enough to use that as long as the container volume mapping is such that Krusader has access to /mnt so it can show the disks that are at  /mnt/diskX at the Linux level on Unraid.

 

Not clear from your description why things did not work as expected the first time though as it sounds as if your Split Level setting of 5 was compatible with your paths.

My guess is what you mentioned about Krusader pre-allocating the folders before copying the data over. I kind of suspected that when I went into the folder in Windows and all the folders were already there. I followed the Spaceinvader One tutorial on Krusader and it had me delete the original path and add paths 1_unraid_shares and 2_unraid_unassigned. I'll have to go dig a bit read up on how to move folders to disks rather than shares in it.

[JorgeB]

32 minutes ago, Fencer said:

Krusader pre-allocating the folders before copying the data over

Yep, it does that.

[Fencer]

Thank you very much for your insights! I'm off to bed and will be reading up a bit more on Krusader's usage for moving files to different disks when I get up. This stuff will still take another 7-8 hours to finish copying, lol. I'll know better for next time. Looking forward to getting this monster box out from under my desk and into it's proper place soon...