[Plugin] unbalanced

[Darryl]

First time using Unbalance,

 

I have a number of files on a disk that are either hard linked elsewhere or could be hard links themselves.  Does Unbalance currently support hard links (I read the quote below, but couldn't see the outcome), and if so would the correct flags be -avPRXH or is -avPRX presumed by Unbalance and only -H is required?

 

Or is -aH not supported?  I couldn't `man rsync` on the UnRAID terminal for some reason, but MacOS `man rsync` says "-a, --archive               archive mode; same as -rlptgoD (no -H)".

 

And in a dry run test, the option syntax look ... odd 'rsync -avPR -XH --dry-run "data/media" "/mnt/disk6/"'

 

On 3/13/2023 at 11:52 PM, MammothJerk said:

on the hardlinks issue @jbrodriguez

 

 

Seems to be some logic that moves the subdirectories separately which is what is breaking the hardlinks between said subdirectories.

 

i.e if i have two hardlinked files
 

/disk9/media/movies/movie.mkv
/disk9/media/seeding/movie/movie.mkv

the /movies/ and /seeding/ subdirectories get moved separately and the hardlinks break.

 

 

 

Edited July 14, 2023 by Darryl
additional info

[crescentwire]

Hi @jbrodriguez, I've been using unBALANCE successfully for several weeks as I grow my disk capacity across my array.

 

I have a certificate applied to my unRAID server using LetsEncrypt, and every so often it's renewed automatically when it approaches the expiration date. I access my unRAID server using https://my-server.my-domain.com, so when I access unBALANCE, the full URL is https://my-server.my-domain.com:6738.

 

Recently, I renewed the certificate through LetsEncrypt. However, it seems that unBALANCE continues to use the previous cert (which is now revoked). Safari on macOS throws an error, stating this.

 

Here's the cert details when accessing unRAID (working normally):

 

 

And here's what I see when accessing unBALANCE (throws revocation error):

 

 

My guess is there's a reference to the certificate that unBALANCE uses--or at least how it inherits what's already there under unRAID. Is there a directory or config file I can check to make sure I'm also updating this when I renew my cert from LetsEncrypt?

 

Let me know if you'd like any other info. Thanks very much for your help on this in advance!

[Darryl]

On 3/13/2023 at 11:52 PM, MammothJerk said:

on the hardlinks issue @jbrodriguez

 

 

Seems to be some logic that moves the subdirectories separately which is what is breaking the hardlinks between said subdirectories.

 

i.e if i have two hardlinked files
 

/disk9/media/movies/movie.mkv
/disk9/media/seeding/movie/movie.mkv

the /movies/ and /seeding/ subdirectories get moved separately and the hardlinks break.

@MammothJerk cc: @jbrodriguez Do you have any more information or work around on this issue?

Doing my own rudimentary research and considering your hypothesis/finding.  If subdirectories are moved separately, all hard links to the files will not be in the data set: therefore I think this, from 'man rsync' is true (emphasis added).

Quote

 

Note that rsync can only detect hard links between files that are inside

the transfer set. If rsync updates a file that has extra hard-link connections to files outside the transfer, that linkage will be broken. If you are tempted to use the --inplace option to avoid this breakage, be very careful that you know how your files are being updated so that you are certain that no unintended changes happen due to lingering hard links (and see the --inplace option for more caveats).

 

So --inplace would need to used to mitigate the issue. but the following from 'man rsync' should be observed:

Quote

 

[--inplace] has several effects: (1) in-use binaries cannot be updated (either the

OS will prevent this from happening, or binaries that attempt to swap-in their data will misbehave or crash), (2) the file's data will be in an inconsistent state during the transfer, (3) a file's data may be left in an inconsistent state after the transfer if the transfer is interrupted or if an update fails, (4) a file that does not have write permissions can not be updated, and (5) the efficiency of rsync's delta-transfer algorithm may be reduced if some data in the destination file is overwritten before it can be copied to a position later in the file (one exception to this is if you combine this option with --backup, since rsync is smart enough to use the backup file as the basis file for the transfer).

WARNING: you should not use this option to update files that are being

accessed by others, so be careful when choosing to use this for a copy.

 

 

Any thoughts, or could you confirm my thinking?

[jbrodriguez]

1 hour ago, Darryl said:

Any thoughts, or could you confirm my thinking?

hi, i'm not that knowledgeable with hard links , but yes, unbalance uses the rsync "built-in" on unraid

 

my only comment would be to `man rsync` on unraid to make sure of the capabilities offered in the platform, if `man rsync` is done on macos some things (especially things like hard links) may be different

[jbrodriguez]

On 7/14/2023 at 8:11 AM, crescentwire said:

Is there a directory or config file I can check to make sure I'm also updating this when I renew my cert from LetsEncrypt?

hi, it looks for these 2 certificates, in this order

/boot/config/ssl/certs/certificate_bundle.pem
/boot/config/ssl/certs/<server name>_unraid_bundle.pem

 

[crescentwire]

20 minutes ago, jbrodriguez said:

hi, it looks for these 2 certificates, in this order

/boot/config/ssl/certs/certificate_bundle.pem
/boot/config/ssl/certs/<server name>_unraid_bundle.pem

 

 

Thank you, this is very helpful. This is the exact directory I write the new certs to. If there's a cert change, does unBALANCE reload this file during startup? Or is it kept the same as long as the filename is identical? (My filename remains the same during each renewal, but the file's MD5 hash and cert fingerprint is obviously different.)
 

Thanks very much!

[jbrodriguez]

10 minutes ago, crescentwire said:

does unBALANCE reload this file during startup

yes, only on startup, it's not monitoring that file, maybe restart unbalance from the unraid plugin page ? 

[MAM59]

Beeing a newby with unbalance I ran into several problems already. At first i thought it was my fault (wrong click?), but now that I check carefully before clicking, I am sure there is something wrong or at least "very strange" with the plugin..

 

Situation is quite simple:

Array has 4 drives, there are also 4 pools (one cache, the others for different purposes).

What I want to do is to free up the Array drives one by one, move the files to a free disk (one of the 4 is free) and then reformat the original drive with a different filesystem (and afterwards repeat the procedure until all drives are done).

 

So I start unBALANCE, click "scatter", select "disk4" as the source drive (and select all shares on it) deselect (hmm, why are automatically all other drives select as targets? does not make much sense, especially for the pool drives) all but "disk2" and click "plan".

It takes some time (a lot of time), then I am presented a list which I have to accept (looong list, nobody reads it!).

Hitting "move" begins the copy orgy.

 

Starts rather promising, but if I scroll down the job list to the end I finally find:

HUH? nobody ever asked for "disk3"!!! The target should be disk2 like in the two upper lines (and the hundreds before these).

Worst of it, "disk3" is already 97% full, it will be overrun when these copies will be done (I have to keep an eye on them and STOP manually when the get into the running queue).

 

This happened before already today, the last 30 jobs went to pool device "vm" instead of "disk1". Poor "vm" only has 2tb and i was bombed with mail every few minutes because it was filling up rapidly. And it would have been overfilled if I did not stop unbalance.

The free space calculation was done on base of "disk1" not "vm"!

(and the current example looks the same, those (there are dozens more lines for "disk3") jobs will never fit into the remaining 3% of disk3.

 

So the question is: why does the "plan" suddlenly changes target drive??? (btw: that share is even forbidden by configuration to reside on disk3)

And: why does it overload a drive ?

 

Update: I waited until 70% of the job was done, then stopped unraid (before it came to the wrong jobs). Re-added like before (disk4 -> disk2) and re-planned. This time all target jobs were planned correctly. So I assume, its a mass problem in the plugin. The calculations fail if there are plenty of folders to copy (i my case around 1000)

 

Edited July 18, 2023 by MAM59

[crescentwire]

On 7/15/2023 at 3:18 PM, jbrodriguez said:

yes, only on startup, it's not monitoring that file, maybe restart unbalance from the unraid plugin page ? 

 

Completely removing, then reinstalling the unBALANCE plugin fix the issue. Thank you!

[dunn]

Is this plugin even working with the current version? No matter what I do, I can't use it.

First, there is always the message about exclusive access to disk. I stopped everything and even disabled mover and Docker completely, reinstalled the app multiple times, restarted it, but that message will always be there. I guess I just have to ignore it, but why is it being displayed there? I have no idea whether this is just a general message or a notification for me.

Second, I'm unable to uncheck dry run. There was once a user having similar issue, but he got ignored. Then there was another guy who solved it by himself by playing with the custom rsync flags. I tried that too, but had no luck at all.

 

I'm going to use rsync now manually, since trying to solve this plugin is just a waste of time. But can anyone update this plugin to be working properly? It's the first thing everyone recommends.

[MAM59]

8 minutes ago, dunn said:

Is this plugin even working with the current version?

yes, but the usability is really "strange". And you need to keep an sharp eye open.

 

First of all: install "fix common problems" and run "Docker safe Permissions" from the tools menu. This will set up file permissions and ownerships to a common and safe state (and the plugin wont complain anymore).

 

Then select "scatter" and pick your source disk. Select your shares/folders to be transfered (warning, do not select too many at once, I guess there is a real bad bug in the list processing of the plugin, I often end up by transfering to the wrong target disk (even if deselected!))

 

Beware, all other disks and pools are instantly selected as targets (which is quite silly I think), so make sure you deselect everything but the real desired targets.

 

Now hit "Plan" an brace yourself.

This can take some time and you wont see much.

 

Start the copy with either "Move" or "Copy" (I've tested Move only so far).

BEWARE! Now you will see the job list what the plugin is about to do. It will be a long list, but scroll down copletely to the bottom!

Read the last entries! Does the target disk still is the one that you have wanted???

 

If not, STOP the move, start over with "Scatter" and select fewer folders/share next time.

Repeat until all jobs look good.

 

Let it run.

When finished, redo scatter until all folders are done someday.

 

I guess the plugin runs out of memory if the list gets too long and then produces "strange" results.

 

But if you pay attention and control everthing, it will work.

 

 

[jmbailey2000]

I used to use Unbalance a lot, but with the issues creeping up, I've gotten to the point where I just use MC to moves files between disks and haven't had any issues so far. To me, I feel a little better as I have more control over what goes where. Of course, I don't moves tons of files around that often, but I did have to clear off a disk I need to remove from the array and had great luck using MC.

[dunn]

16 minutes ago, MAM59 said:

yes, but the usability is really "strange". And you need to keep an sharp eye open.

 

First of all: install "fix common problems" and run "Docker safe Permissions" from the tools menu. This will set up file permissions and ownerships to a common and safe state (and the plugin wont complain anymore).

 

Then select "scatter" and pick your source disk. Select your shares/folders to be transfered (warning, do not select too many at once, I guess there is a real bad bug in the list processing of the plugin, I often end up by transfering to the wrong target disk (even if deselected!))

 

Beware, all other disks and pools are instantly selected as targets (which is quite silly I think), so make sure you deselect everything but the real desired targets.

 

Now hit "Plan" an brace yourself.

This can take some time and you wont see much.

 

Start the copy with either "Move" or "Copy" (I've tested Move only so far).

BEWARE! Now you will see the job list what the plugin is about to do. It will be a long list, but scroll down copletely to the bottom!

Read the last entries! Does the target disk still is the one that you have wanted???

 

If not, STOP the move, start over with "Scatter" and select fewer folders/share next time.

Repeat until all jobs look good.

 

Let it run.

When finished, redo scatter until all folders are done someday.

 

I guess the plugin runs out of memory if the list gets too long and then produces "strange" results.

 

But if you pay attention and control everthing, it will work.

 

 

 

Thanks but I did that already, only had the issue of not being able to uncheck the dry run. Now after doing another plan I was able to uncheck it suddenly. No idea what was different this time.

 

Honestly in this state this application probably does more harm than it helps. The UI is just horrible and you have no idea what's going on in the background of this app. It would be much better to have an article written for the documentation that explains how to use the terminal for such things. Instead we got this mess and people are still recommending it. 

 

The only thing this application is good for is generating the rsync commands. I now just copied them, mass edited and executed them in TMUX. I only needed it once so it gets deleted and I'll never touch it again. I would have not wasted hours on this application if I would have just used the terminal directly. 

[MAM59]

33 minutes ago, dunn said:

The only thing this application is good for is generating the rsync commands

And, it is running them in the background, not attached from any console or web session. Thats an important thing for me because the copies last long and you cannot close the session otherwise.

(its ok, if you have access to the console, but I am the lazy type)

[jbrodriguez]

2 hours ago, dunn said:

Is this plugin even working with the current version? No matter what I do, I can't use it.

i'm pretty sure the app won't work for some users

 

46 minutes ago, dunn said:

The UI is just horrible and you have no idea what's going on in the background of this app

it's an open source app, you can always provide a PR to suggest changes

 

48 minutes ago, dunn said:

The only thing this application is good for is generating the rsync commands

this is the whole purpose of this app

[dunn]

  

2 minutes ago, jbrodriguez said:

i'm pretty sure the app won't work for some users

 

it's an open source app, you can always provide a PR to suggest changes

 

this is the whole purpose of this app

 

Why isn't it deprecated if only some people can use it? There isn't even a warning in the CA.

 

I'm not a real developer, so I can't contribute to this, and honestly I wouldn't want to. I initially thought this application would be much more sophisticated. But now it looks to me like a simple Bash script could have done a better job. Hell, if you'd think about all the time and energy that went into making this a web application, you could have made a very user-friendly Bash script with dialogues and all the bells and whistles. You maybe even could have implemented it in the UI, so it could be called by a click on the web interface.

 

[MAM59]

2 minutes ago, dunn said:

Why isn't it deprecated if only some people can use it?

He did say "maybe". He just cannot be sure that it will work anywhere.

 

And yeah, the web app is confusing, but after some try & error (more tries and much more errors) you can get the idea and let it do a good job for you.

Its his desicion how he implemented it. You are free to create your own and maybe better stuff and offer it to the public. But unless you do, be silent and either accept and use it work, or leave it alone.

 

[Arndroid]

I have the plugin installed, and seem to be unable to get the web gui started:

Accessing the port number on the server ip regardless shows unbalanced is indeed not started.

Is this a common occurrence, there are no new logs in /boot/logs/unbalanced.log to be found

[FlyingTexan]

Is this plugin broken now days? I'm on 6.11.5, have done the docker permissions. I start it and do "Plan" only for it to tell me that it can't move /data because there's not enough room on the disk it's trying to write to.  Which is true, I don't have any one disk with enough space but was hoping to spread it among all the other disks.

[MAM59]

2 minutes ago, FlyingTexan said:

but was hoping to spread it among all the other disks.

you have to select more than one target disk to allow the spread

 

[FlyingTexan]

I'm on unraid 6.11.5 and would like to clear one of my 14TB drives to turn it to a second parity drive.  It appears unraid is trying to move everything to one particular drive and not spread it through the other drives.  I don't believe I'm doing anything wrong.

[MAM59]

It fills up the 1st drive, then switches over to the next. Scroll down the "to do" job list the plan has generated. I am sure that at the end you will find other target paths as on top of the list

 

[FlyingTexan]

5 minutes ago, MAM59 said:

It fills up the 1st drive, then switches over to the next. Scroll down the "to do" job list the plan has generated. I am sure that at the end you will find other target paths as on top of the list

 

 

 

I don't see anything about "to-do" but it says in the plan that it will not move /data and when I try to move anything it never does.  It seems like it's trying to target only one disk even though I have many selected.

Edited August 3, 2023 by FlyingTexan

[MAM59]

are there no subdirs below /data ?

There are options to group or split them together/apart up to a certain level below.

 

[FlyingTexan]

20 minutes ago, MAM59 said:

are there no subdirs below /data ?

There are options to group or split them together/apart up to a certain level below.

 

There is but what am I supposed to do with it? Isn't the whole point that I select what directories I'd like to move then I have them moved? If I have to go in and manually select groups of files to move one disk at a time then I don't see the point of the plugin. I also have some older 8TB drives that are 4.5yrs old if I go to remove them I'd hope I could do so before they really fail and by an automated process.