Can someone telll me where my logic is wrong.

 

If i have a Folder structure on several drives identically as:

 

//Shareme/A/

//Shareme/B

...

//Shareme/Z/

 

and then i copy up data to say the "A" folder via the USER share, am i correct in saying that as long as any single drive has the physical space for the data I should not see the "Out of Space" warning.

 

The reason I ask is that I often do see this message and I have literally hundreds of gigs free accross a couple of the drives in the User share.

 

Do i have to explciity specify the disks in the GUI for each folder or does not having any set count as "Use all disks" that have the associated folder structure.

 

 

Sorry if i am not being clear.

 

I should add that i am only now getting used to always using User shares but stillc ant solely use them as i often get out of space messages and therefore copy the data onto the //disk-x/ drives manually.

I'm almost positive that this is a bug.  I'm still relatively new to unRAID and my only experience is with 4.2 Beta4 and i watched it this weekend copy to one disk until it was completely full when i had like 500+GB free on other drives. 

 

I think that what is happening is that the "high-water" test is only being performed when you create a new folder at the root level of the share. If you create a folder as a sub directory of the share it never wants to go to another drive. The only way i've been able to get around this is to copy directly to the disk share and restart the array.

 

I am currently copying about 300gb to the unraid array and right now it is fixated on Disk3 which does not have 300gb of space. From what i've seen this weekend i can all but guarantee that it will run that drive until it has 0 bytes free and error out instead of just moving to a different disk.

 

In playing with it yesterday i realized that if i went to the usershare and created a brand new root level folder that it then works properly. But if i add a subdirectory beneath an existing folder it will try to put it onto the full disk and it will fail.

 

Maybe Tom can verify that he is aware of this bug, or tell us that we are stupid users and don't know what the hell we're doing.

Well slap my ass and call me betty... it worked.  Hm. 

 

Now I'm confused. I have no idea why it wouldn't switch drives this weekend.  Maybe the problem was that the hard drives that i watned it to expand to weren't in the system when i created the share? Maybe it's because i went in and set my split level to 0 explicitly on this share. Hm. 

 

Well, i'm no help because it seems to be working now.

I think there is generally a bit of confusion over what should and shouldnt work and as such we dont really know whats the correct way to implement things.

 

When Limetech posts and clarifys i think all will be revealed as right now i have no idea if its a feature, bug or user error

Ok, yesterday i copied about 500GB to an unRAID user share and i believe that it worked as indented.

 

It set it's high water mark at half the size of my largest HD and then copied data until that mark was hit then it moved on to the next drive.  Once all drives had about that much space free the high water mark divided in half (to 75gb or so) and then all drives filled to that level.  Then again down to 35GB and to 16GB where it is currently. 

 

I have seen it not work but when it didn't work i had two different conditions. 1) the user share was limited to one drive when i first created it and i added another drive to unRAID after i made the share. I modified the share to add the new drive but it didn't seem to want to use it if i didn't create a new folder at the root of the share.  2) split level was not explicitly set to 0.

Nice documentation of the problem.  That should help Tom figure out what happened.

 

 

Bill

Whenever a new file or directory is created, it is always created in a parent directory.  It's the "level" of the parent directory in the hierarchy which determines if files/folders within that directory are "split" (ie, allocated) across multiple drives.

 

For example, consider this hierarchy:

 

//ShareName/Folder1/Folder2/Folder3

 

A split-level of 1, means any file/folder created in //ShareName or  //ShareName/Folder1 is a candidate for having it's disk allocated; but, any file/folder created in //ShareName/Folder1/Folder2 (and below) will be allocated to the same disk //ShareName/Folder1/Folder2 already exists on.

 

A split-level of 2 would mean that any file/folder created in //ShareName, //ShareName/Folder1, or //ShareName/Folder1/Folder2 is a candidate for having it's disk allocated, but any file/folder created in //ShareName/Folder1/Folder2/Folder3 (and below) will be allocated to the same disk as //ShareName/Folder1/Folder2/Folder3 already exists on.

 

A split-level of 0, means any file/folder created in //ShareName is a candidate for having it's disk allocated; but, any file/folder created in //ShareName/Folder1 (and below) will be allocated to the same disk //ShareName/Folder1 already exists on.

 

So it is possible to run out of space in a User share even though there is still lots of space available in the disks assigned to the share.

 

The best way to avoid this is to carefully design your user shares.

 

Here is what we are trying to accomplish with split-level.

 

We would like a directory-full of files to be kept together on the same disk.  This is necessary when the directory contains a set of video files which are played in series.  If the first few video files were stored on one disk, and the rest stored on another, you would see a pause in the video stream when the player hits a file on the second disk due to that disk needing to spin up first.

 

We're assuming most of the time disks are spun down, and spun up "on demand".  User can tolerate the initial "pause" when the first disk spins up, but would not enjoy 10-15 second pauses right in the middle of a movie.  Of course you can have all your disks spinning all the time, but we want to try and avoid that as well.

 

The split-level accomplishes this by directly specifying how deep in the file hierarchy we go before not allowing directories to be split between two or more disks.  This gives you some flexibility to set up say,

 

(method A - using split-level=1)

//movies/title1

//movies/title2

:

 

(method B - using split-level=2)

//movies/westerns/title1

//movies/westerns/title2

//movies/action/title1

:

 

So if you have a User share called 'Movies', you should have a strict and consistent folder hierarchy.  For example, you would probably not want to do this:

//movies/title1

//movies/title2

//movies/action/title3

 

Hope this helps.

 

PS- Actually we're looking into some alternatives.  For example, we're considering a "policy-bit" on a share which, if set, would do this: Any time a file in a share is accessed, it would spin up all disks that that file's parent directory exists on.  Not sure if we'll do this, but I bring it up to illustrate that we're still trying to think of ways to simplify this.

Thanks for the clear explanation. Now that i have read this i understand exactly what to do.

 

Any time a file in a share is accessed, it would spin up all disks that that file's parent directory exists on.  Not sure if we'll do this, but I bring it up to illustrate that we're still trying to think of ways to simplify this.

 

If you don't implement a policy like that, at least an option to do that would be quite useful

Van i just clarify a couple of other things, In this scenario with 2 disks:

 

 

disk1/Share/Folder/ <<0MB free

disk1/Share/ <<100GB free

 

and the associated User share

 

/Share/Folder/

 

unRAID doesnt have the ability to a a folder to the disk with free space so that you dont receive an out of space message.?

 

Is there ANY way to make unRAID do this as without that i still have to do alot of manual folder creation.

 

 

Secondly when adding folder to disk shares is the only way to have them show up in User shares to restart the array.?

Van i just clarify a couple of other things, In this scenario with 2 disks:

 

 

disk1/Share/Folder/ <<0MB free

disk1/Share/ <<100GB free

 

and the associated User share

 

/Share/Folder/

 

unRAID doesnt have the ability to a a folder to the disk with free space so that you dont receive an out of space message.?

 

Is there ANY way to make unRAID do this as without that i still have to do alot of manual folder creation.

 

Sorry, I can't figure out what you're asking 

 

If you have this folder:

disk1/Share/Folder

 

The User share is /Share not /Share/Folder.

 

Secondly when adding folder to disk shares is the only way to have them show up in User shares to restart the array.?

 

Yes that's true at the present time.  There are some subtle differences between accessing files via User share vs. via disk share.  In theory we can arrange it so any action via a disk share will show up immediately in a user share and vice versa, but it doesn't do this currently.

The second answer also answered the first one.

 

Essentially for the data to be balanced between drives the immediate folder the data is being added to (via the user share) has to exist on all disk shares. Even if the folder 2 up from the data file exists on all disk drives unless the immediate one is there as well it will not be balanced.

 

This explains what i am seeing. Thanks

 

 

Essentially for the data to be balanced between drives the immediate folder the data is being added to (via the user share) has to exist on all disk shares. Even if the folder 2 up from the data file exists on all disk drives unless the immediate one is there as well it will not be balanced.

 

Well that's not strictly true... It depends on what 'split-level' is set to for the share.

 

If you set split-level high, say 999, then all folders/files created will get spread out onto all the disks eligible for that share, even if the parent folders don't exist (they'll be created).

 

I know this is confusing.  A big problem with RAID systems in general is that they are confusing.  We are constantly striving to reduce the complexity.  I don't like the 'split-level' concept; but for the moment it solves a particular problem.  Please read this post:

http://lime-technology.com/forum/index.php?topic=1051.msg7284#msg7284

 

 

 

 

Ooooh!  So in order to get it to span drives as often as possible you want to set the split level higher.  I was setting it to zero assuming that that would essentially disable it but the best way to get split to disable is to set it to 999.  So my next question is what is the defautl? If we leave split level blank does it default to 0 or 999? If my understanding of the funtionality as explained above is accuate i'd want it to be 999.

The default is 0.

I know this is confusing.  A big problem with RAID systems in general is that they are confusing.  We are constantly striving to reduce the complexity.  I don't like the 'split-level' concept; but for the moment it solves a particular problem.

Here's an alternative suggestion:

 

\Movies\Action\Silent Hill

\Movies\Action\LOTR\Fellowship of the Ring

\Movies\Action\LOTR\The Two Towers

\Movies\Action\LOTR\Return of the King

 

In this case the current split level system doesn't work too well. So my idea is this: unRAID could allow splitting for every folder except the "deepest" one. So that would mean files in "Silent Hill" would be kept on one harddisk. Files in "Fellowship of the Ring" would be kept on one harddisk. But the 3 LOTR movies would be split to different harddisks. In the moment when the user adds a subfolder to "Silent Hill", the files in "Silent Hill" would be allowed to be split over multiple harddisks, but the files in the new subfolder wouldn't.

 

However, such an automatic algorithm might still sometimes not be what the user wants. So I'd suggest giving us more control over how it works on a folder-by-folder base. E.g. we could add a dummy 0 byte file "dontSplitThisFolder" to a folder to tell unRAID to not split the subfiles+subfolders in this folder. Such a "dontSplitThisFolder" marker together with the automatic algorithm I explained above would work well automatically in most cases and still give us users enough control to tune things to our needs. Another possibility would be to also allow a "dontSplitNextFolder" marker. We could place such a marker in the "Movies\Action" directory to make the following directory structure working:

 

\Movies\Action\Silent Hill

\Movies\Action\Silent Hill\Trailers

\Movies\Action\Silent Hill\Covers

\Movies\Action\LOTR1

\Movies\Action\LOTR1\Trailers

\Movies\Action\LOTR2

\Movies\Action\LOTR3

 

Alternatively instead of using such 0 byte markers, you could allow a "config.txt" or "config.ini" file, which would then control how splitting is handled. Such a txt/ini file would allow for more settings to be stored which might not even be unRAID specific. (E.g. such settings could be read by a home cinema control software).

 

What do you think?

In this case the current split level system doesn't work too well. So my idea is this: unRAID could allow splitting for every folder except the "deepest" one. So that would mean files in "Silent Hill" would be kept on one harddisk. Files in "Fellowship of the Ring" would be kept on one harddisk. But the 3 LOTR movies would be split to different harddisks. In the moment when the user adds a subfolder to "Silent Hill", the files in "Silent Hill" would be allowed to be split over multiple harddisks, but the files in the new subfolder wouldn't.

 

Hm, you might be on to something but I think that this may be simpler and give the same results.  What if there were a user share option so that unRAID would never split *files* across shares, only directories.  That way you could have your DVD folders in any type of hierarchy you want but the vobs would always be contained on a single drive.  That shouldn’t be too hard to implement, I wouldn’t think.

 

I don’t like the "marker" idea though. That seems like more trouble than it’s worth.  If you want to manage the folders with that level of granularity just access the drive shares when loading the media.  It’s not like this stuff changes very often.

 

What if there were a user share option so that unRAID would never split *files* across shares, only directories.  That way you could have your DVD folders in any type of hierarchy you want but the vobs would always be contained on a single drive.

So you mean if e.g. I create a new file in directory "\\tower\someShare\subFolder\readme.txt" unRAID would look through all harddisks and select the first harddisk that is found which has a "\\tower\someShare\subFolder" on it with at least one file in it. Right? That does sound like an interesting idea to me.

 

The only real problem I see is that if you have e.g. a folder named "Silent Hill" with subfolders for trailers and covers etc then these subfolders might be spread over multiple harddisk. So if you browse through the movies and want to watch a trailer and then check out the covers, different harddisks may need to spin up. Personally, I'd like to have control over that in some way. But then the most simple solution might be the best one, even if it has some (small) disadvantages.

 

I don’t like the "marker" idea though. That seems like more trouble than it’s worth.

Well, you wouldn't have to use it! It would be only used by people who absolutely want to have finest control...

 

If you want to manage the folders with that level of granularity just access the drive shares when loading the media.  It’s not like this stuff changes very often.

The problem right now is that user shares and drive shares don't reflect into each other. You have to reboot to make changes visible in the other type of share.