[Plugin] unbalanced


Recommended Posts

Sorry just a quick question.. is it best practice to turn off dockers and vms when using this? just I am using it to encrypt a large existing array and as you can imagine it will take quiet some time. I have a few ppl who use the services regularly and am curious if having dockers running will cause any issues. Sorry if this is already answered I couldnt find it when i searched and googled it.

Link to comment

Hi,

 

Short Answer

Use unbalance exclusively. No other dockers/vms/plugins/mover touching the disks.

 

Long Answer

The bottom line is if you're moving content outside unbalance when it's running, the assumptions it made may no longer be valid.

That is, it thought you had 40Gb free on disk1, but a docker or mover dropped it to 35Gb.

Maybe it will still work, but it's very possible that it won't (out of space, source not found, etc)

If you know what you're doing, that you won't be touching the disks that unbalance is operating on, then keep running dockers/vms, but you want to be sure that doing so won't have side effects.

  • Upvote 1
Link to comment

Thanks for the info. I have shut off everything that might add or move files on the array. Plex will stay active but it shouldnt change any drive. 

 

One more question. As i said earlier im using this to encrypt my drives. When im moving files to empty drives I have a few drives that are 100% full, save for about 500mb. Im not sure if this is over filling the hard drives if i wanted to defrag etc. or when im done is there a way to balance the data over the entirety of the array?

 

Or is this totally fine to leave as is!

Link to comment

The plugin reserves a minimum of 512Mb as a hard limit, but you can go higher in the settings.

 

There are some forum members very knowledgeable on the hard disk subject matter and they prefer not to have disks "filled to the brim".

 

On the other hand, if you use your hard disk as "read-only" (fill it with contents, start with a new disk, repeat), it shouldn't really be a problem, and you skip the fragmentation you mentioned.

Link to comment

ah very good. I guess on the one hand the majority of the hard drives are going to be read only after the contents is added. Also i am formatting each drive so fragmentation shouldnt be an issue. Ill look into that hard limit when im done with the process. cheers for the response.

Link to comment
22 minutes ago, jbrodriguez said:

prefer not to have disks "filled to the brim".

I'm not very knowledgeable on the subject, but I do know that you can run into issues with file system repair and other nasties if there isn't enough slack space for the file system to do its thing. Just recently there was an update in the XFS format, where disks formatted after a certain date reserved a much larger percentage of the total file system size, so if you had a disk filled to the brim with the old format, and tried to do a disk to disk copy to a newly formatted disk of the same size, you would be out of luck as there isn't as much free space available on the new format.

 

I'd try to keep some portion free just to avoid headaches later. How much? Who knows. 512MB is probably safe, but I'd rather keep as much space free as my largest typical file. YMMV, IANAL, yada-yada, etc.

  • Upvote 2
Link to comment
...Just recently there was an update in the XFS format, where disks formatted after a certain date reserved a much larger percentage of the total file system size....

 

Thanks for presenting this scenario jonathanm.

 

I was surprised by how much overhead xfs had, the last time I formatted a disk.

 

I'm thinking about raising the hard limit to 1Gb, although one can always go above it in the settings, if so preferred.

 

 

Sent from my iPhone using Tapatalk

Link to comment
  • 5 weeks later...

Not sure if this has be addressed before, but I wanted to use unBALANCE today but the PLAN is showing moving everything from one drive to another, it's not splitting even though I have "most free" on my drives and shares.  These are new drives and the only difference I see is that they do not have the folders created.   Should I create my main share folders?  Not sure how to solve this problem.  Thanks!

Link to comment

Hi,

 

unBALANCE works at the disk level not share level, so it won't consider any share rules you set.

 

I'd guess you're doing a scatter?

 

The concept for scatter is to empy a disk by moving data to other disks you choose: you can choose one, some or all of them.

 

It's up to you.

 

Hope this helps.

  • Upvote 1
Link to comment
On 11/15/2018 at 5:21 AM, jbrodriguez said:

Hi,

 

unBALANCE works at the disk level not share level, so it won't consider any share rules you set.

 

I'd guess you're doing a scatter?

 

The concept for scatter is to empy a disk by moving data to other disks you choose: you can choose one, some or all of them.

 

It's up to you.

 

Hope this helps.

Well, I have all my data on 1 drive (from another unRAID server) and I wanted to "balance" the data through all the drives, is that possible?  Or could it be a future option?

Link to comment

Right, as BRiT mentioned, choosing more drives will scatter the data across disks, but if the data you're trying to move fits one of the disks, it will put it all there.

unbalance doesn't "balance"/fill disks evenly, that's more of an intrinsic feature of Unraid, by using share rules.


Sent from my iPhone using Tapatalk

  • Upvote 1
Link to comment

Also as stated earlier, there really isn't any benefit to spreading things out evenly among your disks.  It's kind of counter-productive and wastes more electricity by requiring more drives to be spun up more of the time.  You're really better off just letting the drives automatically fill up and move on to the next one as it sees fit.

Link to comment
  • 2 weeks later...

Dear all

 

if I understand correctly the "Docker Safe New Perms" is installed by the plugin "Fix Common Problems", so I post my problem here.

 

I have run Docker Safe New Perms on all disks except for the Cache disk (nothing to be moved).

Then I wanted to use unBalance to move the content of one disk to another disk. During the planning phase unBalance would complain that it found directories with improper permissions and I shall run "Docker Safe New Perms".

So how can this be since I did run "Docker Safe New Perms" in advance.

 

It turns out that unBalance considers existing Set-U-ID oder Set-G-ID bits as problematic, and "Docker Safe New Perms" does not clear out these bits.

 

I do not know whether these bits are actually a real problem for the unBalance procedure.

But if they are, then "Docker Safe New Perms" should remove those bits, e.g. by setting the permissions to 00777. (Note the leading 2 zeroes! Yes this is correct, with just one zero it will NOT clear those bits!)

 

This can be done manually as long as "Docker Safe New Perms" does not do it:

find /mnt/disk?/ -type d -and  -not -perm 0777 -exec chmod 00777 {} \;

 

However, if those bits do not create a problem with the unBalane procedure, the unBalance should not report them as an issue.

if you search like this, you will find the directories where SUID or SGID are set:

find /mnt/disk?/ -type d -and  -not -perm 0777

This is not what we want if we do not want to report those directories. Instead it should be used:

find /mnt/disk?/ -type d -and  -not -perm -0777

Note the "-" (minus) character in front of the 0777. Now additional bits being set will not be reported!

 

It would be great to get feedback from the author of unBalance whether these bits should be cleared or whether the should rather not be reported.

 

In the first case it would be good to get feedback from the author of the plugin "Fix Common Problems" whether clearing of the SUID SGID bit for directories can/will be implemented.

 

I am going to cross-link both postings in both support threads.
Fix Common Problems thread:

 

Link to comment
45 minutes ago, jus7incase said:

I am going to cross-link both postings in both support threads.

Please don't do this. Which thread should we respond to? Should we all be expected to read both threads to see what others have said in response?

 

In a more general case, you could have someone going to a lot of trouble to research a problem for you and write a long response in one thread, without being aware that someone else has already made a similar effort on the other thread. This is one reason why crossposting has been frowned upon on message boards since long before the World Wide Web.

 

In future, if you really feel you must make the post in one thread and take responses there, then just post a link in the other thread.

 

 

  • Like 1
Link to comment
Want to use this, especially now that a drive failed, but since I've installed this I can't uncheck the dry run option, I can only perform dry runs. Changed browser and computers and still can't uncheck it. what can I do to solve this?


I remember something like this happened before, not sure what was it.

The check box is disabled until there's enough input for the operation, make sure about that.

If you can try on Chrome, could you check the dev console for any errors?

Last option, send me the log (/boot/logs/unbalance.log), to take a look.


Sent from my iPhone using Tapatalk
Link to comment

I took a look at the code with regards to SUID/SGID.

 

It checks that permissions are "r--r--r--" or "rw-rw-rw-" (for files), "rwxrwxrwx" (for folders), so if SUID/SGID are set, it will report a file or folder permission issue.

 

Not sure how to go about this, but SUID/SGID don't seem too common, I think ?

Link to comment

I had SUID/SGID initially on my array as it was setup from before and I think one of my downloaders had it set.

 

You could use the following to get octal permissions and take the right-most 3 character of the first portion (not shown below):

 

stat -c "%a %n" *

 

Or more verbose to compare between Octal and Human Readable form:

 

stat -c "%a %A %n" *

 

 

Link to comment
  • 2 weeks later...

I have 2 issues

1. When I scatter, the scatter process failed/not complete. I can see the target drive free space is reduced which means that some files have been moved. Unfortunately, the source drive space is not increasing which means that the moved files are not being deleted.

2. Later when I run plan and wait for it, I notice an error message pop up and then vanished with no way of retrieving the error message back. This are the error message (after several time quickly captured it).    

There are some permission issues with the folders/files you want to transfer
    9 file(s)/folder(s) with an owner other than 'nobody'
    9 file(s)/folder(s) with a group other than 'users'
    9 folder(s) with a permission other than 'drwxrwxrwx'
    0 files(s) with a permission other than '-rw-rw-rw-' or '-r--r--r--'
    You can find more details about which files have issues in the log file (/boot/logs/unbalance.log)
    At this point, you can transfer the folders/files if you want, but be advised that it can cause errors in the operation
    You are STRONGLY suggested to install the Fix Common Problems plugin, then run the Docker Safe New Permissions command

 

How can I

1. Remove duplicate files generated by Unbalance?

 

By the way, can you make it the important message not vanished? If its important why hide it?

 

Thanks.

Link to comment
10 hours ago, publicENEMY said:

1. Remove duplicate files generated by Unbalance?

Move works by copying then deleting, but if a command fails, it doesn't delete the source by design, so your original files are intact.

 

You can check the operation "log" in the History tab and check which command ('rsync') failed and act accordingly (delete target folder manually, retry the operation, etc.)

 

Why did it fail ? What does the log say ?

 

10 hours ago, publicENEMY said:

By the way, can you make it the important message not vanished? If its important why hide it?

It's a fair point.

 

Maybe I could just leave it open, you can always click to delete it.

Link to comment

Can someone please help. I am having so many issues learning v6. I came from unrid v5 for 5 years without any issues...

 

I am encrypting drives so I am first using the unbalance plugin to move files from one drive to another, then exncrypt the empty one, etc...

 

For some reason everything has come to a crawl lately. I have disabled any parity and don't even have any parity drive assigned. I am getting max speeds of about 20MB/s! I just goan i5 8400 processor, LSI IBM m1015 controller. Never had issues with such slow speed before! Can someone help me out and let me know what's happening? I feel like this is slowing down my other dockers that I installed. I have 16 gigs of ram too. What's happening and why is everything so slow, even without parity!

Screen Shot 2018-12-25 at 12.18.35 AM.png

Edited by maxse
Link to comment
  • jbrodriguez changed the title to [Plugin] unbalanced

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.