Fill up disk - min free space setting is ignored


Recommended Posts

Hi all,

Im realatively new to unRAID. I have a share called Media and sits on 5 hard drives. I want to fill each disk gradually as I keep adding stuff, instead of the high-water setting. I just want to keep each of my movies and stuff contained rather than split on multiple disks. So I have set the share to use Fill up and set min free space 50Gb. I just noticed that my first disk has filled 100% and there is only 500mb free left out of 1TB. Is the min free space setting assumed for all drives total or can it be set for individual drives? Also if I want to switch back to High-water, would the data get re-distributed and free some space on disk 1? I read that it is not good to fill the disk more than 90%. Thanks for any guidance.

 

Link to comment

Hi all,

Im realatively new to unRAID. I have a share called Media and sits on 5 hard drives. I want to fill each disk gradually as I keep adding stuff, instead of the high-water setting. I just want to keep each of my movies and stuff contained rather than split on multiple disks. So I have set the share to use Fill up and set min free space 50Gb. I just noticed that my first disk has filled 100% and there is only 500mb free left out of 1TB. Is the min free space setting assumed for all drives total or can it be set for individual drives? Also if I want to switch back to High-water, would the data get re-distributed and free some space on disk 1? I read that it is not good to fill the disk more than 90%. Thanks for any guidance.

Unraid will never move existing data on its own, it only allocates newly written data according to the rules you set. If I read you correctly, I think you would be better served by adding your files to the disk you want using the disk shares, and just using the user shares to provide the all inclusive view of the entire share. Unless you understand and set the split level properly, unraid can't magically follow your wishes of "fill each disk gradually" and "keep each of my movies and stuff contained rather than split". The split level determines how many sub folders deep unraid will go before denying new files to go to a new disk. If you have it set too high, it will never automatically use the next disk, it will simply say no more space.
Link to comment

Hi all,

Im realatively new to unRAID. I have a share called Media and sits on 5 hard drives. I want to fill each disk gradually as I keep adding stuff, instead of the high-water setting. I just want to keep each of my movies and stuff contained rather than split on multiple disks. So I have set the share to use Fill up and set min free space 50Gb. I just noticed that my first disk has filled 100% and there is only 500mb free left out of 1TB. Is the min free space setting assumed for all drives total or can it be set for individual drives? Also if I want to switch back to High-water, would the data get re-distributed and free some space on disk 1? I read that it is not good to fill the disk more than 90%. Thanks for any guidance.

Unraid will never move existing data on its own, it only allocates newly written data according to the rules you set. If I read you correctly, I think you would be better served by adding your files to the disk you want using the disk shares, and just using the user shares to provide the all inclusive view of the entire share. Unless you understand and set the split level properly, unraid can't magically follow your wishes of "fill each disk gradually" and "keep each of my movies and stuff contained rather than split". The split level determines how many sub folders deep unraid will go before allowing new files to go to a new disk. If you have it set too high, it will never automatically use the next disk, it will simply say no more space.

 

Actually, you got the Split Level setting backwards.  See this section in the unofficial WIKI:

 

    http://lime-technology.com/wiki/index.php/Un-Official_UnRAID_Manual#Split_level

 

Don't feel bad as this Split Level setting is one of the most confusing settings in unRAID.  While it is logical, it is not really intuitive.

 

I suspect that the bmw-man has somehow miss-set the Split Level and that is probably his problem.

 

Link to comment

I think you have misunderstood the meaning of min free space.

 

If you have a 1TB disk and specified min free space of 100GB it does not mean that once the disk has reached 900GB it will stop copying to that disk. It means that it will not simply not copy any file larger than 100GB to that disk.

 

This is incorrect. The min free space setting does not limit file size. No new files will be written will be written to the disk once the min free space has been reached. In the example, the disk will no longer be used for new files once the disk has reached 900GB.

 

The min free space should be selected based on the largest file that you anticipate writing. However, this does not in any way limit the size of the files that may be written. Min free space setting is needed because unRAID has no idea how large any file will become during writing. If more than the min free space is available on the destination disk then the file will be created. If the file grows larger than the available space then the write will fail. If the min free space is set to larger than the largest file ever written then there should be no errors resulting from write failures to full disks as long as a second disk has available capacity greater than the min free space setting.

Link to comment

Thank you all, great pointers. I think I have my split levels set fine and I can see my Media folders the way i want, combined at level 2. I guess my misunderstanding is that with the Fill-up setting each drive will get filled gradually as I add stuff, until it reaches the min free space setting. I guess I will manually move some of the data from disk 1 to disk 3 and free up some space that way.

Link to comment

This is incorrect. The min free space setting does not limit file size. No new files will be written will be written to the disk once the min free space has been reached. In the example, the disk will no longer be used for new files once the disk has reached 900GB.

 

The min free space should be selected based on the largest file that you anticipate writing. However, this does not in any way limit the size of the files that may be written.

 

I'm sorry, but it is this part of the unraid documentation that made me think otherwise...

 

"unRAID typically receives data in this manner. First, unRAID receives the request to store a file, named for example "file.eg". At this time, unRAID has no idea how big "file.eg" is so unRAID will pick a spot to place "file.eg" and begin to store the file data as the data is transfered over the network. Now, this is important because unRAID may pick a storage disk that does not have enough space to store the complete "file.eg". unRAID doesn't know there is not enough space when it first places the file so unRAID will only find out the disk doesn't have enough space when the disk is full. At this point, the transfer will fail with a disk full error.

So, unRAID will write to a different disk if the minimum free space is set to a value larger than the biggest file size you will ever transfer. The recommended setting is 2 times the largest file size you will ever transfer."

 

The last part in bold implied that one sets the min free space to a number higher than what you would ever transfer irrespective of whether the available free space on the disk is larger or smaller than what is specified in the setting because unRAID seems to detect it (or rather not detect it), until the file is entirely copied (or not, due to lack of free space).

 

 

Link to comment

Actually, you got the Split Level setting backwards.

Yeah, right concept, wrong word. I modified my post to avoid further confusion. I get in a hurry and type half a sentence, have to step away to deal with another issue, come back and finish my thought. Sometimes my brain doesn't keep track of the thought all the way through.
Link to comment

Not really...

 

The way file copies over samba typically happens is a request for a 0 byte length file creation is made. Then the file is appended to in chunks until the file is fully written. You could have a 300 Gigs free, a min free space setting of 250 Gigs, and still end up with copying a file of 290 Gigs to the disk. Unraid has no way of preventing this once the initial file is placed on a disk. It can only control what disk it selects when the 0 byte file is created when using the user filesystem.

 

It all depends on how the client behaves, this would be your Windows clients and those programs or it could be your Unraid server and whatever plugins you have installed. When they behave as specified above, there's no way around it.

 

Sent from my Nexus 7 using Tapatalk 2

 

 

Link to comment

This is incorrect. The min free space setting does not limit file size. No new files will be written will be written to the disk once the min free space has been reached. In the example, the disk will no longer be used for new files once the disk has reached 900GB.

 

The min free space should be selected based on the largest file that you anticipate writing. However, this does not in any way limit the size of the files that may be written.

 

I'm sorry, but it is this part of the unraid documentation that made me think otherwise...

 

"unRAID typically receives data in this manner. First, unRAID receives the request to store a file, named for example "file.eg". At this time, unRAID has no idea how big "file.eg" is so unRAID will pick a spot to place "file.eg" and begin to store the file data as the data is transfered over the network. Now, this is important because unRAID may pick a storage disk that does not have enough space to store the complete "file.eg". unRAID doesn't know there is not enough space when it first places the file so unRAID will only find out the disk doesn't have enough space when the disk is full. At this point, the transfer will fail with a disk full error.

So, unRAID will write to a different disk if the minimum free space is set to a value larger than the biggest file size you will ever transfer. The recommended setting is 2 times the largest file size you will ever transfer."

 

The last part in bold implied that one sets the min free space to a number higher than what you would ever transfer irrespective of whether the available free space on the disk is larger or smaller than what is specified in the setting because unRAID seems to detect it (or rather not detect it), until the file is entirely copied (or not, due to lack of free space).

 

I don't understand how this implies that the min-free-space limits the size of the file that may be written. If min free space is set to 10G and the drive has 1T free. Any file up to 1T in size may written. Ignoring split-level, if the allocation method selects a disk the the disk has greater than min-free-space available a write will begin to the disk without regard to the size of the file. unRAID does not know the anticipated file size so it cannot make any decisions based on file size. Selecting a large enough min-free-space, i.e., double the anticipated max file size, should prevent write failures due to overwriting a disks capacity; although, split-level may cause full disk write failures due to its precedence.

Link to comment
On 3/14/2013 at 12:44 PM, dgaschk said:

The split level has precedence over min free space.

 

Yes, this is likely the problem and why the fill-up method can be difficult to use without manual intervention.

 

No File will be written to a disk once the free space on the disk is less than the min free space setting unless the split level is forcing the files to go to that drive.

 

As an example, if you use the split level to force each season of a TV Series to stay on a single disk and then start a new season on an almost full disk unRAID will attempt to write the whole season to that disk regardless of the min free setting. You'd be much better using the high water or most free allocation methods so the new season is started on a disk with more free space.

 

The min free space has nothing to do with limiting the file size being written.

Edited by lionelhutz
Link to comment
  • 1 year later...

I'm getting mover errors i believe to be related to split level setting forcing the files to certain disks...

 

rsync: write failed on "/mnt/user0/Series/The Mentalist/Season 06/The Mentalist - S06E02 - Black-Winged Redbird SDTV-thumb.jpg": No space left on device (28)
rsync error: error in file IO (code 11) at receiver.c(389) [receiver=3.1.0]
./Series/The Mentalist/Season 06/The Mentalist - S06E01 - The Desert Rose SDTV.mp4
>f.stpog... Series/The Mentalist/Season 06/The Mentalist - S06E01 - The Desert Rose SDTV.mp4
^Z
[1]+  Stopped                 mover

 

I have the Series share set to Split level 3 (So that new episodes can be copied to new disks)

 

My understanding is as follows:

Level 1 = User Share Series Shows can be spread over disks

Level 2 = True Blood Seasons can be spread over disks

Level 3 = Season 1  Episodes Can be spread over disks

 

Min free space set to 10000 (I forgot the unit, bytes?)

Allocation set to "Most Free"

 

I have a few hundred gigs free on the other drives, but Disk 1 keeps filling and resulting in this error.

 

Sorry if I'm not seeing something that should be clear to me.

Link to comment
  • 4 years later...

I have gone over the same rhetoric a number of time when it comes to disk utilization as to what it does and what it does not do, ad nauseam, simply because IT DOES NOT OPERATE THE WAY THAT IT SHOULD so everyone tries to figure out just what it is doing...

If it would operate in a USEFUL manner, there would not be a problem, BUT IT DOES NOT.

And, here is why - in this, I'm in complete agreement with the guy who first wrote as to what "minimum" SHOULD mean.

The idea is to "fill-up" the disk up to a "minimum" so as to allow for some remaining free space AS SPECIFIED BY THE MINIMUM.

Once the minimum has been reached (or exceeded in order to finish a file write), move to the next disk that has space available above the MINIMUM.

With an adequate "minimum" there should be no issue with failed writes of a file and their associated files or metadata.

(Without an adequate minimum, writing of a file or file folder with contents SHOULD STILL CONTINUE to the next disk, unless dis-allowed by the split-level*.)

To use it to specify the "minimum" that needs to be available AS A MINIMUM ABOVE the size of the file to be written IS USELESS!

If you wish to write a file to the disk (automated or otherwise) THEN YOU WANT THE FILE TO BE WRITTEN!

And who gives a shit about it's size, as long as there is sufficient disk space and the minimum has not (yet) been reached.

The likelihood of a write failure due to running out of space on a disk in this manner (with a proper "minimum") is just about negligible

AS COMPARED TO UTILIZING A "MINIMUM" IN THE MANNER SPECIFIED HERE!

Therefore, it should be coded to be used in a manner as WOULD BE ASSUMED BY ITS VERY NAME.

To do otherwise is pretty damned USELESS and CREATES PROBLEMS... (as exemplified by the INNUMERABLE times this issue is raised)

So, why does it not operate in a manner where IT WOULD BE OF USE (LimeTech)?

 

*I also understand why the split-level is then increased by users who are attempting to utilize their disks in such a manner in order to allow use of the next disk when a disk "fills-up".  (Which wouldn't have happened with the PROPER APPLICATION OF A MINIMUM SPECIFICATION.)

This "split-level" then creates a problem when the split does not coincide with the "logical" desire to keep ASSOCIATED FILES on the same disk.

i.e.  Due to disk utilization issues CREATED by the "minimum" not operating in a USEFUL manner, the split-level has to be increased so the writes occur to the next disk which has space.  What occurs then is that despite it being specified in the application (and desired) to keep associated files in the same folder as the media (movie, TV episode, song, or album, etc.), the ASSOCIATED METADATA ends-up on another disk altogether (in a folder of the same name)...

NOT WHAT IS DESIRED AT ALL – AND IT ALL STARTS WITH NOT BEING ABLE TO SPECIFY THE MINIMUM DISK SPACE (reserved) TO BEGIN WITH!

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.