Jump to content
We're Hiring! Full Stack Developer ×

[6.9.2] Allocation Method Not Working Correctly


Recommended Posts

I recently added a new drive to the array and assigned it to a specific share where all the other drives are almost full, but when I save files to the share, unRAID does NOT save the files to the drive I've intended it to (Most Free) and instead attempts to save them to the other drives until there's no more free space and I get errors (OS X).

 

I've tried setting the Allocation Method to High-Water but same situation: it doesn't always send files to the most free drive.

 

In the past, I've never had this issue when expanding an array and assigning new drives to a specific share.  This is the first time I've done so under 6.9.2, so I don't know if there's a new bug introduced with this version, or something else.

 

Sys log shows nothing out of the ordinary.

 

In the meantime, I'm now stuck moving files from the full drives to the new drive to regain the ability to save files to the share again without encountering errors which stops file transfers to the NAS.

 

Allocation Method.png

Drive Space.png

Edited by Auggie
Link to comment

Haven't looked at diagnostics yet, but noticed some things in your screenshots.

 

No good reason to set both included and excluded. Best if you don't and maybe doing so breaks some logic. It certainly makes it possible to have inconsistent lists. Include means ONLY those disks, Exclude means EXCEPT for those disks. Do either Include or Exclude, not both. There is no disk usage that you can't configure by only using either Include or Exclude.

 

Also, Most Free is the least efficient. Highwater is a good compromise between eventually using all disks without constantly switching between disks just because one disk briefly has more free space than another.

 

And, I notice you have no Minimum Free for the share. Unraid has no way to know how large a file will become when it chooses a disk for it. You must set Minimum Free to larger than the largest file you expect to write to the share. If a disk has less than Minimum, Unraid will choose another disk if other factors allow it. If Unraid is allowed to choose a disk (with zero minimum it can always choose the disk) and the file won't fit, the write will fail.

 

Cache (and other pools) also have a Minimum Free setting. If the pool has less than Minimum, Unraid will choose an array disk instead (overflow) for cache-yes and cache-prefer shares.

 

 

Link to comment

If you are overwriting files, a replacement file will go to the same disk the file was already on.

 

How exactly are you doing the file transfer? Sometimes an application will create empty folders before copying any files, and the folders might end up all on one disk since empty folders don't take much space. Then the files are copied to the folders that have already been created.

 

Link to comment
On 10/3/2021 at 2:03 PM, trurl said:

No good reason to set both included and excluded. Best if you don't and maybe doing so breaks some logic. It certainly makes it possible to have inconsistent lists. Include means ONLY those disks, Exclude means EXCEPT for those disks. Do either Include or Exclude, not both. There is no disk usage that you can't configure by only using either Include or Exclude.

 

I don't agree with this train of thought: I believe you can have both, so long as there is no conflict with disks in both exceptions, which I don't have any.

 

But just to prove this theory, I removed all disks under "exclude" and just tried copying files to the NAS and nope, it still wrote to one of the almost-full drives versus the newly added drive with the most free.

 

On 10/3/2021 at 2:03 PM, trurl said:

Also, Most Free is the least efficient. Highwater is a good compromise between eventually using all disks without constantly switching between disks just because one disk briefly has more free space than another.

 

For years I had everything set to Highwater (in fact, all my other user shares are still set to Highwater); it's only when unRAID kept filling up the other drives until they had zero (0) free space is when I switched to Most Free in hopes of forcing it to the new drive, but nope, it kept behaving unexpectedly.

 

On 10/3/2021 at 2:03 PM, trurl said:

And, I notice you have no Minimum Free for the share. Unraid has no way to know how large a file will become when it chooses a disk for it. You must set Minimum Free to larger than the largest file you expect to write to the share. If a disk has less than Minimum, Unraid will choose another disk if other factors allow it. If Unraid is allowed to choose a disk (with zero minimum it can always choose the disk) and the file won't fit, the write will fail.

 

IIRC correctly, I read somewhere, perhaps in an unRAID FAQ, to set Minimum Free to 0; I know when I started having this issue and bumped it to, say, 256GB, I couldn't move/copy any files to unRAID because it said there was no more free space, when in fact, the new drive had about 3TB free whilest the original drives were less than 80GB.

 

I again set minimum free to 128GB in this similar situation and it still wrote to the other drives with less space.

 

BTW, I don't use cache drives...

Link to comment
On 10/3/2021 at 2:23 PM, trurl said:

If you are overwriting files, a replacement file will go to the same disk the file was already on.

 

My experience is that this is not what actually happens: unRAID will still typically place the files/folder according to the allocation method, even if there is a same-named folder on one or more drives.

 

On 10/3/2021 at 2:23 PM, trurl said:

How exactly are you doing the file transfer? Sometimes an application will create empty folders before copying any files, and the folders might end up all on one disk since empty folders don't take much space. Then the files are copied to the folders that have already been created.

 

I'm moving files and folders from a Mac (presently running Big Sur) through the Finder and the user share mounted on the desktop.

 

I do notice a few empty folders here in there on one or more drives, while the actual contents may reside on different drive. Not often and not a deal breaker and so far that I can tell, not related to the issue I'm having because those situations with empty folders are not invovled with the current file operation causing my headaches.

Link to comment
47 minutes ago, Auggie said:

I don't agree with this train of thought: I believe you can have both, so long as there is no conflict with disks in both exceptions, which I don't have any.

 

But just to prove this theory, I removed all disks under "exclude" and just tried copying files to the NAS and nope, it still wrote to one of the almost-full drives versus the newly added drive with the most free.

Obviously, this test result just indicates that you have the problem whether or not you do both include and exclude, not that doing both include and exclude works.

 

Even if it would work the way you have it, why do it?

 

If you include a list of disks, all other disks are excluded. If you exclude a list of disks, all other disks are included.

 

If you add disks later without changing the lists, what happens?

 

If you do both, the include list will mean don't include the new disks, but the exclude list will mean don't exclude the new disks.

 

There are no good reasons to do both and some good reasons to not do both.

 

 

Link to comment
37 minutes ago, Auggie said:

My experience is that this is not what actually happens: unRAID will still typically place the files/folder according to the allocation method, even if there is a same-named folder on one or more drives.

I didn't say anything about whether there are folders with the same names on multiple disks. This is, after all, what user shares is all about.

 

What I said is that if the file already exists, and you replace the file, that replacement will go to the same disk the file is already on.

Link to comment
On 10/6/2021 at 6:33 AM, trurl said:

I didn't say anything about whether there are folders with the same names on multiple disks. This is, after all, what user shares is all about.

 

What I said is that if the file already exists, and you replace the file, that replacement will go to the same disk the file is already on.

 

I will reiterate, my experience does not indicate that that is what happens.  If what you state happens, then all contents with the same parent folder (e.g. "My Favorite Movie" with contents of "My Favorite Movie (1984)" and "My Favorite Movie (Bonus Disc) (1984)," will both be on the same disk as the parent "My Favorite Movie," including any other files, such as poster art and fanart.

 

But no sir, that's not what I see across two unRAID setups over the 11 years I've been using unRAID: individual files/folders can be split across multiple drives if copied/moved over in separate sessions.

 

What I do see is that during a move/copy to the NAS, unRAID will keep all filers and folders in the move in the same disk during the same operation.

Edited by Auggie
Link to comment
On 10/6/2021 at 6:27 AM, trurl said:

Obviously, this test result just indicates that you have the problem whether or not you do both include and exclude, not that doing both include and exclude works.

 

Even if it would work the way you have it, why do it?

 

 

Just to be redundant in forcing my expectations.  The developer of uNRAID apparently added these two options individually, separately, with no dependancies between them, meaning you can mix and match to your heart's content.


Which I've been doing now for 11 years with ZERO issues (of which this is not related to the issue I'm reporting on).

 

On 10/6/2021 at 6:27 AM, trurl said:

If you include a list of disks, all other disks are excluded. If you exclude a list of disks, all other disks are included.

 

That assumption is absolutely incorrect according to the unRAID's Unofficial Manual:

 

Quote

Included and Excluded disk(s)

The included disk(s) and excluded disk(s) parameters control which disks are allowed to be used by each user share. These parameters can be used separately or together to define the group of disks allowed for each user share. The disks are entered by disk number with a comma separating each disk, for example "disk2,disk5".

unRAID will first check the included disks(s) set and then the Excluded disk(s) set when deciding which disk to place a file on. Then, unRAID will use the split level and allocation method to pick a disk which is allowed to hold the file.

 

SECOND SENTENCE: These parameters can be used separately or together to define the group of disks allowed for each user share.

 

The design of which is obviously ambigious enough to cause debates, such as yours, as to the correct usage, but the above statement is crystal clear on the functioning of these two featues. I don't disagree that this is not a very well though-out presentation of these features, which is why I filled out both Include and Exclude to ensure it behaves according to my intentions.

Edited by Auggie
Link to comment

Split levels on shares also override include / exclude.

 

The point on include / exclude is that it makes no sense to specify them both regardless of it there is a conflict or not.   They are mutually exclusive by definition.

 

In the case if a conflict say 5 disks, include disks 1&2 exclude disk 3 it is ambiguous whether or not disk 4 and 5 receive files because the rules are in conflict with each other.

 

There are rules in place of what to do in a conflict but it makes the most sense to only do one or the other not both

Link to comment
On 10/10/2021 at 2:42 PM, Squid said:

Split levels on shares also override include / exclude.

 

The point on include / exclude is that it makes no sense to specify them both regardless of it there is a conflict or not.   They are mutually exclusive by definition.

 

In the case if a conflict say 5 disks, include disks 1&2 exclude disk 3 it is ambiguous whether or not disk 4 and 5 receive files because the rules are in conflict with each other.

 

There are rules in place of what to do in a conflict but it makes the most sense to only do one or the other not both

 

I do understand the concept of what you are postulating, but can you provide any documentation on the statement, "They are mutually exclusive by definition."

 

Where is that "definition?"  A legitimate question.  The quote I provided from the "unofficial" manual is explicity clear that we can set each setting independently and separately. So I'm genuinely interested in reading up on this "definition" you've referred to because it may not be working as you've stated. Remember, what we "think" of how it should be behave based on our individual paradigm of user interface design may not be the paradigm of the developers.

 

The problem with software development is that we as a developer can write code to do whatever it is we want, regardless of interface guidelines and whatnot.  I've been writing code since 1980: BASIC, Fortran, Assembly 6502, COBOL... and interface design can get real messy


I will say when I initially added that drive to the array I noticed that a new sub-folder was created from another User Share which I forgot to "exclude" from it's settings; since adding the new drive to the "exlude" list of the other User Share I have not seen any unwanted sub-folders created.

 

ANYWHO, an update to this situation: that recently "new" drive eventually filled to maximum capacity I wanted so I added another "new" drive.  So far, with all the settings I've had for many years (allocation method High-Water, both Include and Exclude disks entered, Automatically Split), it's now working as intended: All new file transfers goes to the "new" drive.

 

Of note: the first "new" drive was added to the "middle" of the array (disk 11) whereas the new "new" drive was added at the end (disk 24). Not sure if that has any bearing but so far, that's the only difference in setup.

Edited by Auggie
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...