Jump to content
jbrodriguez

[Plug-In] unBALANCE

1457 posts in this topic Last Reply

Recommended Posts

1 hour ago, gfjardim said:

I've added a code to UD to export mounted devices information at /var/state/unassigned.devices/unassigned.devices.json .

 

Will be available on the next UD update, I hope.

Awesome ! That sounds very easy to consume, thanks !

Share this post


Link to post
14 minutes ago, jbrodriguez said:

Awesome ! That sounds very easy to consume, thanks !

Please note that every mounted device is exported there, even ISO or SMB shares. Since I didn't know your plans to integrate both plugins, I exported everything so you can choose which mountpoints you include in unBALANCE.

Share this post


Link to post

Is there a setting in unRAID to disable mover? Scheduler setting doesn't allow you to disable it, but it does allow you to set the day of the week. Is that the best method for preventing mover from running during an unBALANCE move?

Share this post


Link to post
7 minutes ago, RAM said:

Is there a setting in unRAID to disable mover? Scheduler setting doesn't allow you to disable it, but it does allow you to set the day of the week. Is that the best method for preventing mover from running during an unBALANCE move?

Mover only acts on shares where it is enabled. If the share is cache : no or cache : only, mover won't act on that share.

Share this post


Link to post

Thanks for a great plugin... question however, I'm using it to scatter data from a drive I wish to remove.  Using Copy mode just in case.  It completes, but says 980G of data transferred, yet the drive usage says 1.7TB used... how accurate is the transferred data number, is there compression being used?  Validating now to double check.

Share this post


Link to post
11 minutes ago, marcusone1 said:

says 980G of data transferred, yet the drive usage says 1.7TB used

Where exactly are you getting these numbers?

Share this post


Link to post
30 minutes ago, trurl said:

Where exactly are you getting these numbers?

At the top of unBalance, where it says "Completed", "Speed", "Transfered", "Total".

 

"Total" shows 965GB, then on unraid main page, Disk1 (the disk i'm moving data off of via scatter), "USED" says 1.72TB.

Share this post


Link to post
12 minutes ago, marcusone1 said:

At the top of unBalance, where it says "Completed", "Speed", "Transfered", "Total".

 

"Total" shows 965GB, then on unraid main page, Disk1 (the disk i'm moving data off of via scatter), "USED" says 1.72TB.

Have you checked the log?

Share this post


Link to post
1 minute ago, trurl said:

Have you checked the log?

Yes, nothing to indicate why.  Going through the folders that it synced, it skipped one for some reason.  I would guess because it only went 2 deep, and at that depth the folder it skipped would not fit on a single drive (700G, none of my drives have 700G free) so it appears to have just quietly skipped that folder :(

Share this post


Link to post

Hi, given a folder it checks its children and allocated to other disks according to the available space. I guess you can say it's two levels deep.

 

It does inform which folders it won't copy, both in the "console" when PLAN completes and it sends an email with the results of the plan, if you set notifications in the settings

Share this post


Link to post

I'm moving data from multiple drives to multiple free drives so I can slowly change my array to encrypted.

Right now I have disk 1 scattering to disk 8.

But at the same time I'd like to scatter disk 2 to disk 9 and so on.

I cant find a way to run unbalance more than once, and its going to take 6+ hours per drive, I'd love to do them at the same time if possible?

Share this post


Link to post

Hey, it's currently not possible, but if you're mostly copying disk to disk, maybe you can do parallel rsyncs from the command line.

 

Not sure if your parity disk will start barking at the work thrown its way :)

Share this post


Link to post

So, it only works via HTTP, but the link on the Unbalance settings page goes to HTTPS and you get an error. 

 

buzz

Share this post


Link to post
7 minutes ago, buzzra said:

So, it only works via HTTP, but the link on the Unbalance settings page goes to HTTPS and you get an error. 

 

buzz

Works for me

 image.png.c7eb3f169a1c2f5e15e99a2a5fad59b0.png

Share this post


Link to post
14 minutes ago, Squid said:

Works for me

 image.png.c7eb3f169a1c2f5e15e99a2a5fad59b0.png

That's wierd.  I get connection failed in both Chrome and Firefox with HTTPS. Something strange about my setup I'll have to figure out. 

 

Thanks

buzz

Share this post


Link to post
On 12/11/2019 at 5:44 PM, jbrodriguez said:

Hi, given a folder it checks its children and allocated to other disks according to the available space. I guess you can say it's two levels deep.

 

It does inform which folders it won't copy, both in the "console" when PLAN completes and it sends an email with the results of the plan, if you set notifications in the settings

The "console" is flooded with all kinds of things, and doesn't show it "apparently", perhaps it did, but not in any noticeable way one would expect when an Error is encountered.  Perhaps it can be highlighted or pop-out like it does for file permission errors are shown?

Share this post


Link to post
On 12/13/2019 at 5:37 PM, buzzra said:

That's wierd.  I get connection failed in both Chrome and Firefox with HTTPS. Something strange about my setup I'll have to figure out. 

 

Thanks

buzz

I'm noticing the same thing. I'm starting to wonder if this is due to when I setup Let's Encrypt. The plugin seems to have an expired cert and I can't seem to update it.

 

Share this post


Link to post

Love your work on UB!!!

 

I searched this thread but only saw one other comment about rsync -options. Would it be possible to let users change all the command line options for rsync, and not just the -X?
"By default, rsync is invoked with -avPRX flags. Note that the X flag is customizable, so you can remove it if needed."
Here is why I ask. the -P is problematic. I lost a 250GB file because of it. Here is the example.

I have widget.dat (250GB) on Disk 1 along with other files in the share. I start the move to Disk 2, most of the file move, but it starts moving the 250GB file, and I cancel the move, before being completely done (or power outage or whatever).  Now I have widget.dat (250GB) on Disk 1, and widget.dat (39GB) on Disk 2.

 

Now not looking at all my file sizes in the CLI, which most folks would not go. I use UB to move the files from Disk 2 back to Disk 1. My good widget.dat (250GB) on Disk 1 got overwritten by the partial ie bad widget.dat (39GB) from Disk 2. Good by file!!!!

 

IMHO If -P was not on I would not have had data loss.

 

 

Share this post


Link to post

Hi, thanks for the comments !

 

unBALANCE move works by copying then deleting, precisely for events such as your scenario (interrupted mid transfer).

 

So you didn't actually need to move back files to the source, just delete files from the destination.

 

Hope that makes sense.

Share this post


Link to post
6 hours ago, jbrodriguez said:

Hi, thanks for the comments !

 

unBALANCE move works by copying then deleting, precisely for events such as your scenario (interrupted mid transfer).

 

So you didn't actually need to move back files to the source, just delete files from the destination.

 

Hope that makes sense.

Yes that makes sense, thanks.

One can never interrupt a transfer with -P enabled, for fear of a partial file hanging out. The only option would be to not interrupt the transfer, or if the transfer is interrupted, to make sure you do the same transfer later successfully or like you said delete all the files that were copied thus far.

 

It would be great if it defaults stay as -avPRX, but you have made the -X customizable, it would be great if all the options were customizable. 

 

Not UnBALANCE related, but rsync related. Today I had a problem moving files with rsync, and it was failing because it could not change the group settings. As you know the -a includes the -g and -o options specifically -a includes -rlptgoD. So I changed my settings from -av to -rlptDv (same as -av but no g and o) and was able to complete my transfer. Sometimes there are reasons when someone needs to change the rsync options.

 

Thanks.

Edited by jbat66
typo

Share this post


Link to post

I get you.

 

I have those hard-coded arguments, because users can shoot themselves in the foot if they're not familiar with rsync.

 

Especially the -R is really needed.

 

I'll give it some more thought.

Share this post


Link to post
On 12/29/2019 at 5:07 PM, jbrodriguez said:

I get you.

 

I have those hard-coded arguments, because users can shoot themselves in the foot if they're not familiar with rsync.

 

Especially the -R is really needed.

 

I'll give it some more thought.

Thanks, and I really do like your Reset to Default button. Basically just do a warning "don't change these settings, unless you understand rsync, otherwise click the reset button" lol

Share this post


Link to post

Hello everyone,

so I've been copying files from one disk to another (disk1 to disk2) and I noted that some files missing so I'm thinking of doing another run.

But I was wondering if unBalance will skip duplicates, overwrite or make a second file?

Share this post


Link to post

Hi, unbalance runs rsync underneath, so it has the same semantics, it will skip files that are already in the target for example.

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.