Drive 3 full despite high water mark allocation, can't add files to User share


Recommended Posts

Hi Folks,

 

I've experienced an issue with a new unRaid setup wherein the 3rd (of 4) data drives has filled up despite allocation and minimum space setting that should prevent that.  I can no longer write to the array.  Changing allocation setting does not seem to take effect.

 

My setup:

unRaid 6.2.0.beta21

5 drive xfs array

8 tb parity drive, 240x2 cache

high water allocation, level 3 split

Pro license

 

I have a brand new installation of 6.2.0beta21, not an upgrade.  I have a single user share and no drive shares.  I've been copying data from a zfs array mounted using the zfs plugin on the same box.  The copy target is the user share path, not a direct drive path, not a drive share.  Once the data was moved off one 3x4tb zfs array, I destroyed the zfs array and pre-cleared each disk using beta pre-clear plugin, then added it to the array.  This worked fantastically for the first array, moving 12 TB to the 20 TB array (an initial 8tb drive, plus the 3 4tb drives).

 

When copying the second of the two arrays to this same user share, the 3rd of 4 data disks filled up.  I'm not sure why this would happen with high water mark allocation.  I did have a typo in my minimum space setup where i had meant 10 GB, but apparently entered 1 GB.  However, I would still expect the array to check that the file being copied is smaller than the minimum space.

 

At this point I cannot move anything to the array.  Resetting the share setting, or changing the allocation to least full did not change anything.  Any writes still go to drive 3 until it fills up and the copy fails.  Restarting the array, and rebooting the machine after changing share settings did not help.

 

I'm not sure what to do at this point.  I've searched the forums, read the docs, and done some googling, but can't seem to find the right answer.

 

I have tried the common problem plugin.  I fixed a few unrelated bits.  No change wrt the copy problem.

 

Diagnostics are attached.

 

Any suggestions would be most appreciated.  Can one recreate the share based on the data already on the disks?  Is there some other method to fix this?

 

Thanks!

Chris

tower-diagnostics-20160520-1441.zip

Link to comment

... I would still expect the array to check that the file being copied is smaller than the minimum space...

Could be split level as itimpi says, but thought I would clarify the way minimum free space actually works.

 

It doesn't know how large a file is going to be when it starts writing it. If the free space is more than the minimum it will write to the disk.

 

Suppose you set minimum to 10GB, and the disk has 11GB free. If you write a 10GB file it will write it to the disk, and then it won't try to write any more files to that disk because the free space is less than the minimum.

 

Suppose you set minimum to 10GB and the disk has 11GB free. If you try to write a 12GB file it will try to write because the free space is greater than the minimum, but it will fail when it runs out of space.

 

This is why you set minimum free space larger than the largest file you will write.

 

Link to comment

Thanks for the explanation, trurl, that definitely helps my understanding going forward.

 

The files were rsync'ing from /video/ (a zfs mount) to /mnt/user/Media/video (where Media is the user share).  I was doing this from the server console (due to the large amount of time to transfer TB's at a time).

 

Everything worked fine with disks 1 & 2 filling to 80% & 83% respectively.  It was not until it moved to disk 3 that it kept going.  i recognize that my 1 vs 10 GB typo caused much or all of my original issue  :o

 

Can I assume from the fact that the UI let's you, that changing share setting after it's built is an "ok" thing to do?

 

Thx...

Link to comment

Another thing I have found when rsyncing between two servers is that rsync will often create the folders before it writes the files. Depending on your split level, this can result in the destination drive being predetermined before the files are written. Once it has created the folder for a file, the split level can cause it to try to write files to that folder even after the drive is full. I don't know if there is any rsync option to prevent this behavior.

  • Like 1
Link to comment

You might have a good point about rsync, although I would hope to not have to worry about such things when copying to my array.  I'm also curious why it did not move to drive4 when it reached 80% like it did when going from drive1 -> drive2 and drive2 -> drive3.

 

So is there a known way to fix a scenario like mine?  I've seen that other users have had drive-filling issues.  I haven't seen a post with a solution yet, though I will continue searching.

 

Questions that come to mind about unRaid are:

 

- Is a user share just a "view" over the data such that one can recreate a share if including the same disks?

- Does unRaid have any smarts around "re-balancing", say when you add another disk, etc.?

- If I change the allocation to "Most Free", shouldn't the array start using the least full disk on the next copy?  So far this does not happen in my setup.

 

Thx!

Link to comment

unRAID never moves files once they have been written to the parity array. It will move cached User Share files from the cache disk to the parity array at the scheduled time. If you need to rebalance you will have to do it yourself. See the unBalance plugin.

 

Most free means the disk with the most free space. The meaning of "least full" seems a little ambiguous to me. Changing the setting will take effect for subsequent writes but will not move any files already written. And split level still takes precedence so if the folders where a write should happen according to split level are already on a disk it will continue to write to that disk.

 

A User Share is really just top level folder(s) on the parity array or cache. The folder(s) will have the same name as the share. Upper/lower case is significant so Movies is a different share than movies.

 

A User Share is the aggregate of all these top level folders named for the share. Conversely, any top level folder on the array or cache are part of a User Share named for the folder. If you create a top level folder it is a share and if you don't make settings for a share it has default settings.

 

The settings for a User Share are mostly about telling unRAID where to put files for the share when it writes them. Even if a disk is not included in the share for writing, it will still be included in the share for reading unless excluded from all User Shares in the Global Share Settings.

 

The unRAID webUI has a Help toggle in the upper right that will turn help on/off for all pages. Share settings are explained in the help though without as much detail about what happens behind the scenes.

 

 

Link to comment

Great info, thanks.  The info on the split level taking precedence, and still writing if in the middle of a level is a key point, thanks.

 

I should have stuck with "most free" as a term.  If I set the share to "most free", it does not in fact start copying to the empty disk4 on subsequent writes.  I guess that goes back to the split level precedence.

 

I will checkout the unBalance plugin.

 

I really appreciate the help, and the quick post turnaround!  8)

Link to comment
  • 3 years later...
On 5/21/2016 at 6:56 AM, trurl said:

Another thing I have found when rsyncing between two servers is that rsync will often create the folders before it writes the files. Depending on your split level, this can result in the destination drive being predetermined before the files are written. Once it has created the folder for a file, the split level can cause it to try to write files to that folder even after the drive is full. I don't know if there is any rsync option to prevent this behavior.

Thanks for posting this on the forums so I can find it years later! 

 

Ran into this today while migrating data to a new array ... VERY FRUSTRATING!  After investigating, I found that lots of empty folders in a user share hierarchy were created on one disk, while the other way basically empty, so unRAID kept trying to populate the nearly full disk since the folders were already there.

 

I hope unRAID in active use is better than in migration. It's not exactly been smooth sailing thus far.

Link to comment
8 hours ago, JesterEE said:

Thanks for posting this on the forums so I can find it years later! 

 

Ran into this today while migrating data to a new array ... VERY FRUSTRATING!  After investigating, I found that lots of empty folders in a user share hierarchy were created on one disk, while the other way basically empty, so unRAID kept trying to populate the nearly full disk since the folders were already there.

 

I hope unRAID in active use is better than in migration. It's not exactly been smooth sailing thus far.

I was copying my Unraid data to another Unraid server I had created for backup when it happened to me. I saw that rsync was going in alphabetical folder order when it began to actually write the files, so I just started at the other end moving the empty folders to other disks before it began to write to them.

Link to comment
  • 1 year later...
On 6/15/2019 at 10:02 PM, JesterEE said:

Thanks for posting this on the forums so I can find it years later! 

 

Ran into this today while migrating data to a new array ... VERY FRUSTRATING!  After investigating, I found that lots of empty folders in a user share hierarchy were created on one disk, while the other way basically empty, so unRAID kept trying to populate the nearly full disk since the folders were already there.

 

I hope unRAID in active use is better than in migration. It's not exactly been smooth sailing thus far.

 

Also ran across this myself while migrating a lot of data to a new array. A bit frustrating -- really slows the migration process -- but at least I understand what's happening now.

 

Thanks for the information!

 

  • Like 1
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.