I don't think my system tries hard enough to keep folders together...

[NLS]

Well as the title says.

I have grouped stuff to disks.

I think I have properly set split level (usually one or two deep, depending on case) and also always use high-water.

Yet the system doesn't seem to try and keep things together.

For example there is a share disc_images which normally full resides on a disk and the disk has enough space (note it is not the emptiest disk).

I have set "top two" for split level, as right below disc_images I have 4-5 central folders and if ever space was not enough in this disk I would expect system to "overflow" to another disk for a folder BELOW disc_images.

Yet when I added some new data, not much, nothing that would need any kind of overflow... they were copied to disk 1, ignoring the fact there there is already the proper disk structure (and space enough) on another disk.

This happened to many shares I used. They all preferred disk 1 and I don't think any attempt to keep things together was made.

I don't want to make it "manual, do not split" as I do want it to split IF NEEDED. But not before is needed.

 

What am I doing wrong?

Maybe high-water is not the proper method and needed method is a (non implemented) "prefer existing directories" or something?

 

[Dissones4U]

On 11/24/2019 at 4:30 PM, NLS said:

This happened to many shares I used. They all preferred disk 1 and I don't think any attempt to keep things together was made.

 

My suspicion is that because you only have 2.8 TB across 7 disks your high water levels are set pretty low for each pass. I'm not sure that split level can be explained any better than this, it really depends on your file structure. I have most shares on L2 with a couple on L1 & L3 . Allocation method, including a good description of how high water works is here. There is an older discussion about split levels and opinions on how best to write to shares here, it's pretty old but I believe it's still relevant.

[NLS]

Where did I say how much space across how many disks?

The numbers are not correct.

 

[saarg]

1 hour ago, NLS said:

Where did I say how much space across how many disks?

The numbers are not correct.

 

The numbers are from your signature.

[NLS]

Well I don't see a signature, I don't even remember I had one and I cannot find a way to edit it (!?!?!?).

It is probably from my original unRAID install 12 years ago.

EDIT: Found a way to edit it and will do it later. Indeed 12 years ago... nice flashback.

 

No those data are not correct.

 

Thing is that high-sierra mode doesn't fit me anyway.

My disk 1 is now empty (because it happened I consolidated folders and this became empty), so it has the biggest space than any other disk.

But if I want to add 100GB of files in disk 4 where I have let say emulators share and I have 500GB empty on it, I want the system to select disk 4 for those new data, not disk 1. But if it fills up I want it to overflow to another disk.
Is there a mode to allow me to do just that?

 

Edited November 26, 2019 by NLS

[gberg]

1 hour ago, NLS said:

Well I don't see a signature, I don't even remember I had one and I cannot find a way to edit it (!?!?!?).

It is probably from my original unRAID install 12 years ago.

EDIT: Found a way to edit it and will do it later. Indeed 12 years ago... nice flashback.

 

No those data are not correct.

 

Thing is that high-sierra mode doesn't fit me anyway.

My disk 1 is now empty (because it happened I consolidated folders and this became empty), so it has the biggest space than any other disk.

But if I want to add 100GB of files in disk 4 where I have let say emulators share and I have 500GB empty on it, I want the system to select disk 4 for those new data, not disk 1. But if it fills up I want it to overflow to another disk.
Is there a mode to allow me to do just that?

 

Well, you can select which disk(s) each share should use, and if you want the disk(s) to be filled one after another you can use fill-up allocation method.

[trurl]

The thing missing from this discussion is the fact that Split Level has priority over Allocation Method, so regardless of Allocation Method and how full a disk is, Split Level will override all that and put files together that belong together.

 

Maybe you are just off one level when specifying Split Level.

[NLS]

I will return to this subject when new data move in (because now I have cleaned everything up and don't remember) and list settings for those specific shares, so we can have practical examples.

Probably next week.

Thanks.

[NLS]

OK I am back...

So these few days I updated (much) data related to Emulation.

So how things were and how they are now (with all disks involved) with the help of QDirStat...

 

Before:

disk 1 -> was empty

disk 3 -> ~1TB free

> generic (high-water, 1MB min, auto-split only top, use cache) - the folder resides only on this disk, held few data (12 files, less than 1GB)

> isos (high-water, 700MB min, auto-split any dir, use cache) - again these resided only on this disk (6 images, around 11GB)

disk 5 -> ~2.4TB free

> emulation (high-water, 100MB, auto-split top 3, use cache) - this disk holds part of my emulation folder with the following folders (note those subfolders are JUST on this disk unless mentioned otherwise)

>> DOWNLOADS - around 65GB in 1501 folders 

>> MISC - less than 1GB in 1598 files

>> MULTIMEDIA - ~290GB in 96190 files

disk 10 -> ~600GB free

> emulation - some more subfolders of my emulation folder reside here

>> ROMs - ~2.1TB in 2045183 files

>> TOOLS - ~2GB in 4741 files

disk 11 -> ~550GB free

> emulation - again some emulation subfolders here

>> EMULATORS - ~3TB in 449551 files

>> FRONT-ENDS - ~87GB in 138323 files

(I have no included/excluded disks set for any)

 

After:

disk 1 ->

> emulation ->

>> DOWNLOADS - 558MB in 34 files

>> EMULATORS - 6.5GB in 502 files

>> FRONT-ENDS  - 340MB in 1378 files

>> MISC - 1MB in 2 files

>> MULTIMEDIA - 464MB in 3 files

>> ROMs - 281GB in 801559 files

>> TOOLS - 80MB in 58 files

> generic - has one file I added to the folder

> isos - has 1 file (which is a new version of an image)

 

....so I guess the disks that already had the folders got ignored, although all of them had enough space to get the data themselves.

I don't think this works properly, OR system indeed needs an extra split mode.

 

Now I have to use unBALANCE fully manually...

 

[trurl]

6 hours ago, NLS said:

> generic (high-water, 1MB min, auto-split only top, use cache) - the folder resides only on this disk, held few data (12 files, less than 1GB)

> isos (high-water, 700MB min, auto-split any dir, use cache) - again these resided only on this disk (6 images, around 11GB)

Just wanted to pull these out to comment on the correct use of the Minimum Free setting.

 

You should make Minimum Free larger than the largest file you expect to write to the user share. Unraid has no way to know how large a file will become when it chooses a disk for it. If you have Minimum Free set too low, Unraid could choose a disk without enough free space and then writing the file will fail.

 

There is also a Minimum Free setting for cache in Global Share Settings that works similarly. If cache doesn't have enough free space Unraid will choose an array disk instead (for cache-yes and cache-prefer shares)

 

 

[NLS]

As far as I understand it (from a coding point of view also), this setting actually makes sure to keep that much space on target disk as minimum.

 

So for my "generic" share, unRAID wants to keep (just) 1MB free on target disk, so if we are over that number (of free space in target), it can freely choose that disk.

For my isos share, unRAID wants to keep 700MB (roughly the size of a CD) free, so if that disk has let say 670MB left free from last file write, it chooses another disk.

Correct?

 

Well, there was NO disk without enough free space when I was copying those files. All targets had enough space to fit the whole file transfer.

So system should keep them together.

 

To make a specific example for the shares you pinpointed, in "isos" I only wrote one file and that file is 1GB.

Share "isos" was on disk 3 that had 1TB free!

So based on the 700MB rule (I was way over 700MB free), it is should choose disk 3, not (ex-empty) disk 1.

 

ON THE OTHER HAND (and here maybe you hit a nail in the head), my cache is set to (default?) 2000000, around 2MB, which is rather low and pop-ups of cache full did occur. MAYBE, the system has a problem to choose proper disk destination in critical situations?

Because even if cache was full, it should choose the correct disk that already has the share AND has enough space.

 

Could this be the problem?

 

[itimpi]

It is important to realize that the size of a file is not taken into consideration when choosing a disk.   If the disk satisfies the Minimum Free Space criteria it can be chosen and the system start writing the file to the disk.   If the file does not fit you will subsequently get an error as the free space is exhausted.

[NLS]

I understand that. unRAID doesn't know file size before it is completely written.

But this is not related to my issue.

 

 

[itimpi]

2 hours ago, NLS said:

Because even if cache was full, it should choose the correct disk that already has the share AND has enough space.

Unraid checks the disks in ascending order.   Therefore if a lower numbered disk is allowed to be part of the share and has enough space and meets the Allocation Method and Split Level criteria that is where it is sent.      I believe this is the behaviour that you want to change with a new Allocation Method setting?  I think you want it to always first check any disk that already has the path for the file rather than simply in ascending order?

[trurl]

4 hours ago, NLS said:

Because even if cache was full, it should choose the correct disk that already has the share AND has enough space.

Only if the user share is cache-yes or cache-prefer. If the user share is cache-no it isn't going to be writing to cache anyway, and if it is cache-only it won't consider writing to an array disk.

[NLS]

Yes exactly (for both posts above).

 

Since I have already opened a thread as a feature request, I will post the actual process as I imagine it on that thread in a while...