Split Levels for Shares

[AdrianF]

Hey Everyone,

About to start the migration (disk by disk) from my old SnapRaid build under W10 to Unraid on new drives.  I'll be installing the four main drives being used by Unraid and leaving the Parity Disk to install later (alongside the Cache SSD -- which is currently used to house W10)).  

 

That said, I have four main shares currently -- VIDEO, AUDIO, DOCUMENTS, SOFTWARE.  Inside each of these will exist different folders of different types -- some will be fully Blu-Ray images, others will have movies nested in their own folders (and concerts in their own), while others will have TV shows in them (which have season folders).

 

What I want to ensure is that documents end up entirely on one drive, software the same, audio and video can be split across drives but want to make sure that folders are on one given drive and not split across.

 

I want to make sure I get this right when I set it up so I don't have a mess once everything's copied (about 30TB of data).

 

Thanks in advance!

Adrian

 

[JorgeB]

Some split level examples here:

https://forums.lime-technology.com/topic/59589-solved-v635-advice-please-on-shares-settings/?do=findComment&comment=584558

[AdrianF]

Thanks johnnie.black.  Still not seeing the results I expected, unless I'm looking at this incorrectly. 

 

I've managed to get the system up and running on my Gigabyte mainboard (remind me NEVER to buy another one of those) but it's what I had running my existing i5-6500 Windows PC so I figured it'd be more than sufficient for this build.

 

Anyhow, I've managed to get the (4) new 10TB drives into the box leaving the Parity Drive offline.  The Cache Drive is connected but not being used.  I've installed Binhex-Krusader and Unassigned Devices to allow me to plug into the remaining SATA port each old drive so I can copy DATA over.

 

I've decided, even though it means more shares, to set up each folder as a share -- save for audio files where I'd want them in one or two places.

 

For Movies, I'm using "Automatically split only the top-level directory as required" -- this assumes that it would split after the MOVIES share/folder and each movie is in its own folder meaning that any files from the movie (bonus features, etc.) would remain on the same drive.  Included Disks are 1, 2 and 3.  Minimum Free Space is 0KB.

 

The issue is this.  With (4) 10TB drives, I would expect High-water to kick over to the second available drive just past 5TB since it's 50%.

 

When I copied using Krursader last night, I copied a 3.2TB folder off the old Snapraid drive part and parcel while asleep.  When I woke up, I noticed Drive (1) was just over 6TB used and Drive (2) and (3) were not touched except for what was copied to them in other shares.

 

So, I did a little digging and somebody said with the way Krusader works, it'll copy all of the folders to a given drive. So, I went back and deleted about 600GB of files from the share to bring it under 5TB and copied just past the 50% mark to find that it's still not working.  Copied two more movies over (definitely at 5.1TB now) and I'm still seeing the same thing.

 

Any help here is appreciated because something isn't right -- or am I misunderstanding the concept here?  

 

The other issue I know I'm going to run into is that TV shows are going to go past the highwater mark since I want all shows from a particular series ON one drive.  In that case, I also selected split on the Top Level as required.  

 

Thanks again!

[trurl]

18 minutes ago, AdrianF said:

Minimum Free Space is 0KB.

You should set Minimum to larger than the largest file you expect to write to the share. When Unraid chooses a disk to write, it has no way to know in advance how large the file will become. If a disk has less than Minimum, it will choose another disk.

 

21 minutes ago, AdrianF said:

I did a little digging and somebody said with the way Krusader works

Krusader, rsync, and some other applications write all the folders in advance. Since a bunch of empty folders likely will not reach Highwater, all the folders get written to the same disk then the files get written to those folders.

 

Post your diagnostics.

[AdrianF]

19 minutes ago, trurl said:

You should set Minimum to larger than the largest file you expect to write to the share. When Unraid chooses a disk to write, it has no way to know in advance how large the file will become. If a disk has less than Minimum, it will choose another disk.

 

Krusader, rsync, and some other applications write all the folders in advance. Since a bunch of empty folders likely will not reach Highwater, all the folders get written to the same disk then the files get written to those folders.

 

Post your diagnostics.

Fully get that Krusader writes the folders in advance which confuses the algorithm as it does not know how much data is being written, however, would it not understand that 100GB is being copied after it reaches the 50% threshold and bump to the second drive?  This is confusing me since I'm copying files AFTER High-water (50%) has been reached yet it still continues to copy to the same drive and not the next available? I actually stopped in the process to ensure that I could see it hit 4.99, 5.02 and then 5.09TB where it is right now.  It's stopped me moving the rest of my drives over since this isn't going as planned.

 

The problem I see is that I'm coming from DrivePool/SnapRaid on a PC where folders (and files) are spread across multiple drives.  What I don't want to see is me end up with a situation where it all copies to one drive where it shouldn't necessarily be and then I'm stuck.

Would this be a case of just copying as is, setting the minimum free space and then using ReBalance later on because of the mess I inherited as I copy files over?

 

[JorgeB]

19 minutes ago, AdrianF said:

however, would it not understand that 100GB is being copied after it reaches the 50% threshold and bump to the second drive?

If the empty folders are still there it will use them, split level overrides allocation method.

[AdrianF]

16 minutes ago, johnnie.black said:

If the empty folders are still there it will use them, split level overrides allocation method.

Thanks for the update, @johnnie.black.  So what I'm expecting to happen and what does happen are two different things in my case as the split level overrides the allocation method (which seems silly).  

 

In this case, and I know I'm straying off topic a little (lot) here, but what would the best option be to get the data to where it should be?  Is this where I look at ReBalance before I continue?  I don't want to end up with one drive that's maxed out (which is what it looks like will happen). 

[JorgeB]

1 minute ago, AdrianF said:

but what would the best option be to get the data to where it should be?

Delete the empty folders and don't use Krusader, you can use windows explorer, as long as using a recent Windows (8 or newer) it will copy the files locally.

[AdrianF]

Just now, johnnie.black said:

Delete the empty folders and don't use Krusader, you can use windows explorer, as long as using a recent Windows (8 or newer) it will copy the files locally.

Situation is this:

(4) brand new drives in the tower.  I've got one bay available to swap out drives.  Laptop is another room -- not connected via ethernet which is being used by one of us for work.  Only way I can do this is locally on the Unraid build via the UI.  Any suggestions as to the best practice in this case is appreciated.

[JorgeB]

7 minutes ago, AdrianF said:

you can use windows explorer, as long as using a recent Windows (8 or newer) it will copy the files locally.

 

[AdrianF]

2 minutes ago, johnnie.black said:

 

In other words, copy within the share itself?  The drive being copied from is NTFS. I'm guessing there's a way to see the unassigned device as a share?

[JorgeB]

Copy from the UD share to the Unraid share and the copy will be made locally on the server, it won't do the network round trip, as long as it's Win8/10.

 

 

[AdrianF]

26 minutes ago, johnnie.black said:

Copy from the UD share to the Unraid share and the copy will be made locally on the server, it won't do the network round trip, as long as it's Win8/10.

 

 

Almost got this -- and appreciate the help thus far.  I've enabled the share in the drive and auto-mounted it in Unassigned Devices. I'm simply not seeing in the windows share.  Am I missing something altogether here?  Thanks!

[trurl]

1 hour ago, AdrianF said:

split level overrides the allocation method

Split level would be meaningless if it didn't. 

[JorgeB]

1 minute ago, AdrianF said:

I've enabled the share in the drive and auto-mounted it in Unassigned Devices. I'm simply not seeing in the windows share.

It might take a few minutes to appear after enabling, if it takes long reboot Unraid and it should appear at array start.

[trurl]

1 minute ago, AdrianF said:

Almost got this -- and appreciate the help thus far.  I've enabled the share in the drive and auto-mounted it in Unassigned Devices. I'm simply not seeing in the windows share.  Am I missing something altogether here?  Thanks!

You have to share it in Unassigned Devices. 

[AdrianF]

5 minutes ago, trurl said:

You have to share it in Unassigned Devices. 

 

That's what I've got set up there... am I missing something else here?

*I just enabled Pass Thru and that seems to have resolved my issue.  I'm guessing that was the missing link?

Edited May 19, 2020 by AdrianF
Addding Pass-Thru update.

[JorgeB]

Read only is not needed (but should be OK) and you have two partitions, share only the larger one, but there might be an issue because of having both, you'd need to ask on the UD thread.

[AdrianF]

6 minutes ago, johnnie.black said:

Read only is not needed (but should be OK) and you have two partitions, share only the larger one, but there might be an issue because of having both, you'd need to ask on the UD thread.

I'm doing the Read Only as a precaution because I don't want to delete anything JUST IN CASE something goes wrong with the migration.

I just hit 5.01TB on the last file and it switched to the second drive while using W10 and the shares.  Looks like I'm finally in business to put the second drive in once the rest of Disk01 is finished copying.

Thanks again @johnnie.black and @trurl for your help with this.  First time using Unraid and feel very welcome in the forums here. This was one of the main reasons I went with Unraid over some of the other options.