[Plugin] unbalanced


Recommended Posts

20 hours ago, deaerator said:

After the update to 6.4 and setting up ssl certificate, I can no longer connect to Unbalance Web gui.

 

Yes, it works over http only right now.

 

If you want to access the UI, just connect to http:// instead of https://

 

It's in my to-do list to get it working on https.

Link to comment

a search didn't reveal anything.

 

Why have the 450MB reserve requirement?

 

I want to completely fill my full disks, so they can't be written to.  They are just for storage, so there's no point in me leaving space and having unRAID try to manage such a small amount of free space.

 

Sadly, I didn't realize this issue until I'd filled most disks, so most have only this amount of free space, and many of my files are larger than this, so it will not be easy to fill up these disks without much work of me finding small files and manually moving them.

 

Is there any way around this limit?  the program says it's hard limited, so I guess not, but thought I'd ask.

Link to comment

Hi JustinChase,

 

The main reason for the hard-limit is the issue of file size vs sector size.

 

If you have a 1 byte sized file it still occupies a full disk sector (it's still 4k these days right?) and these 4k bytes count towards the total available free space.

 

If you have A LOT of small sized files, you could have many sectors with "unused" free space.

 

So, I settled on a 450Mb limit, to account for this unused disk sector space.

 

Later on, via this forum, I also came to know that ReiserFS doesn't like mostly filled disks that much (unBALANCE was born when RFS was the norm).

 

I could have increased the hard limit, but since the min free space is configurable, I decided to leave it be.

 

I understand that XFS is better behaved with regards to fully populated disks, but people in the forum with knowledge on the matter still recommend not filling up a disk.

 

Hope this makes sense.

Link to comment

Greetings, I have attempting to use the Scatter/Move twice, and both times it fails out at the end of the 1st Source copy.  I've tried moving to 2 separate disks, both with plenty of free space afterwards, and it still fails.  The unbalance.log file only shows exit status 23: Partial transfer due to error, it doesn't say what file it ended on.

 

I was able to use the rsync command through ssh without issue, other than it leaves the source files.

Link to comment
On ‎1‎/‎19‎/‎2018 at 11:28 AM, jbrodriguez said:

I could have increased the hard limit, but since the min free space is configurable, I decided to leave it be.

 

I understand that XFS is better behaved with regards to fully populated disks, but people in the forum with knowledge on the matter still recommend not filling up a disk.

 

What would you suggest filling the available hard drives to?

I have my array set to fill approx. 85% of my drives....

 

Disk stats.jpg

Link to comment

Ah balls. After a failed transfer "exit status 23: Partial transfer due to error" I went to re-run the move and accidentally selected the incorrect destination disk. Rather than waiting an hour and a half for it to finish, and then having to figure out how best to distribute the files to the correct disk, is there any way to cancel the operation? I can't see one. 

Thanks.

Link to comment
  • 2 weeks later...

Can you give us the ability in the settings to set all the flags on our own? The -X flag to copy extended attributes is causing a ReiserFS error and thus causing a move to fail. I tried using my own flags but realized, after it failed again and looking at the logs, that putting custom flags in the settings page just adds those flags in addition to your standard flags. Please give an option for us to use only our custom flags.

 

It’s not a huge deal. I’m just running the rsync operation from the command line for now.

Edited by mrow
Link to comment
16 hours ago, mrow said:

Please give an option for us to use only our custom flags.

 

I used to have fully customizable flags, but the app really needs to "guard" itself especially with

  • -a: since it will be replaced with a -c, for checksum/validation
  • -R: because it needs to have the working directory set to the source disk at least

But I can see that -X isn't really a core flag and can be left on as customizable default (removable). I'll do this.

 

In any case, I'm looking into an alternative rsync invocation that I believe will bring execution parity with a command line invocation.

 

I'm still doing some research and testing, but it looks promising.

Link to comment
On 1/8/2016 at 3:40 PM, jbrodriguez said:

Description

unBALANCE helps you manage space in your unRAID array, via two different operating modes:

  • Scatter

Transfer data out of a disk, into one or more disks

  • Gather

Consolidate data from a user share into a single disk

Hey jbrodriguez, If you are open to thoughts/ideas on adding features, please consider adding another mode which will auto-magically 'defrag' (for lack of a better term) all the folders based on the starting depth. So it would be like using Gather Mode, but unBalance will take the time to reorder things across all the disks. For example, if a share is spread across 5 disks, it would find where the bulk of the data is and move the data from other drives to that drive. If there isn't enough space, it will figure out the next best place for it, or perhaps it would move other folders around to achieve the best 'defrag' (again, not the right word) option for the share at the starting depth the user selected.

 

Hope this makes sense, it does in my head. 9_9:D

Link to comment
1 hour ago, Joseph said:

Hey jbrodriguez, If you are open to thoughts/ideas on adding features, please consider adding another mode which will auto-magically 'defrag' (for lack of a better term) all the folders based on the starting depth. So it would be like using Gather Mode, but unBalance will take the time to reorder things across all the disks. For example, if a share is spread across 5 disks, it would find where the bulk of the data is and move the data from other drives to that drive. If there isn't enough space, it will figure out the next best place for it, or perhaps it would move other folders around to achieve the best 'defrag' (again, not the right word) option for the share at the starting depth the user selected.

 

Hope this makes sense, it does in my head. 9_9:D

Sounds like a sort of Split Level after the fact. Do you know how to use Split Level to prevent this "fragmentation" in the first place?

Link to comment
12 minutes ago, trurl said:

Sounds like a sort of Split Level after the fact. Do you know how to use Split Level to prevent this "fragmentation" in the first place?

I'm somewhat familiar with it, but perhaps not enough. As I understand, even with Split level, folders/files will become fragmented over time as things are deleted and space is freed. Based on the size of new data being moved to the array, there may not be enough room to store everything on a single disk; so at some point when the array starts to get full, Split level will then split it onto 2 or more disks.

 

You are correct that what I was proposing to jbrodriguez is like split level after the fact for situations such as this... or if someone didn't use split level in the first place (yours truly) and would like to get as much things 'unscrambled' as possible. The idea here is it would intelligently 'look' at the folder and depth selected by the user, and move even other data around as necessary to achieve the best fit possible for everything by minimizing splits as much as possible.

Link to comment
1 hour ago, Joseph said:

I'm somewhat familiar with it, but perhaps not enough. As I understand, even with Split level, folders/files will become fragmented over time as things are deleted and space is freed. Based on the size of new data being moved to the array, there may not be enough room to store everything on a single disk; so at some point when the array starts to get full, Split level will then split it onto 2 or more disks.

 

You are correct that what I was proposing to jbrodriguez is like split level after the fact for situations such as this... or if someone didn't use split level in the first place (yours truly) and would like to get as much things 'unscrambled' as possible. The idea here is it would intelligently 'look' at the folder and depth selected by the user, and move even other data around as necessary to achieve the best fit possible for everything by minimizing splits as much as possible.

 

Split Level takes precedence over other User Share settings, so I think even if a disk gets too full it is still going to try to keep things together according to Split Level and just let the write fail if there isn't room.

 

I think another way to accomplish this "after the fact" at the command line (or mc) would be to get your Split Level like you want it, then move things from user0 to cache and let mover put it back.

Link to comment
23 minutes ago, trurl said:

 

Split Level takes precedence over other User Share settings, so I think even if a disk gets too full it is still going to try to keep things together according to Split Level and just let the write fail if there isn't room.

I'll have to re-examine split level. Mostly, I need to rethink my shares strategy instead of dumping everything into one user share.

 

27 minutes ago, trurl said:

I think another way to accomplish this "after the fact" at the command line (or mc) would be to get your Split Level like you want it, then move things from user0 to cache and let mover put it back.

There's more things scrambled than not, but there are several ways to go about 'defraging' them for sure. It would just be nice to have a utility that could do that so it wasn't a manual process.

Link to comment
  • jbrodriguez changed the title to [Plugin] unbalanced

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.