[Plugin] unbalanced

[JonathanM]

43 minutes ago, trurl said:

Limited test release. Since nobody looks at sigs, and most don't have them even enabled since they are turned off by default, you are only the 2nd person to notice.

Trust me, I notice. No point in bringing it up though, it'll happen when it happens.

[jbrodriguez]

Just now, jonathanm said:

No point in bringing it up though, it'll happen when it happens

That is true.

[WhiteTowerBlackNode]

Hi,

 

I am using the Plugin to move data from Disk 6 to other disks using the Scatter command.

When I select Disk 6 as the source and Disks 14 to 18 as targets, it all gets moved to Disk 14 until its full, and then Unraid gives me warnings, as only 1GB or less are free on that disk.

 

Do I have to set something else to scatter data from one disk to several others or is that just the behaviour I have to live with?

 

Thanks for any pointers.

[almarma]

I have a suggestion for this awesome plugin: I've noticed that, before any operation there's a scan of each share, folder, disk or whatever will be moved to ensure a successful operation (which is great!), but in case of warnings or issues, it's very difficult to read them, because it's a popup that disappears by itself after a few seconds, and you have no time to read it all. I think that, in case of warnings or errors, this popup should be permanent with maybe a "Close" or "OK" button. In this way the user has time to read it all and do the corrections needed to continue. Here's a screenshot example of a warning I get after the scan. It also was the only way I found to read it till the end:

 

[jbrodriguez]

On 4/22/2019 at 4:33 AM, WhiteTowerBlackNode said:

Do I have to set something else to scatter data from one disk to several others or is that just the behaviour I have to live with?

It's unBALANCE (for now).

 

As soon as it finds a enough space, it will use it.

 

What you're asking for is something like the built-in share allocation does.

[jbrodriguez]

11 hours ago, almarma said:

I have a suggestion for this awesome plugin

Hi, thanks for the suggestion and the kind words !

 

You have a point, the disappearing warning is something I can change easily.

 

I also need to say that, the warning is basically a tally.

 

Imagine this path:

/mnt/disk1/tvshows/Billions/Season 01/Episode 1.mkv

 

Well, turns out each component (Billions, Season 01, Episode1.mkv) may have wrong permissions.

 

Each component is counted and that's what you see in the summary.

 

You can fix it by running the docker safe permissions script.

 

There's a path worth considering for the plugin:

 

- You could localize/restrict the docker safe script to just /mnt/disk1/tvshows/Billions (and descend recursively). The script would need a minor adjustment, but it's doable and would use a lot less time, than going through the whole disk.

 

In any case, what I'm trying to say is that in its current incarnation, the warning `pop-up` is mostly a heads up to run the docker safe permissions script.

 

You won't gather too much more info out of it.

Edited April 27, 2019 by jbrodriguez

[jbrodriguez]

So, I also want to thank all of you making donations.

 

I can't find a way to thank individually, so ... this is my thanks ! 🙌

[almarma]

2 minutes ago, jbrodriguez said:

In any case, what I'm trying to say is that in it's current incarnation, the warning `pop-up` is mostly a heads up to run the docker safe permissions script.

Thanks for your answer. I did, I run the docker safe permission script but the error kept popping so I'm copying instead of moving so I'm not left with half copied folders. Maybe the safest way to move data deleting it from the original source would be: copy everything, verify, and only then if 100% is copied and ok, then delete the source. Is is done like this?

[jbrodriguez]

5 minutes ago, almarma said:

copy everything, verify, and only then if 100% is copied and ok, then delete the source. Is is done like this?

Currently, no.

 

Move, right now, is: copy, check the folder actually exists on the target, delete the source.

 

Verify is a VERY expensive operation, since it compares checksums of each file on the source and the target.

 

You can do it with unbalance, by doing  a COPY, then going to the History screen and clicking on the Validate button.

 

[dgwharrison]

Recently I've been running into issues using unBALANCE. Most recently I got this error when trying to scatter files from one disk to another:

I: 2019/05/07 12:43:00 core.go:710: Command Started: (src: /mnt/disk7) rsync -avPR -X "media/TV Shows" "/mnt/disk9/"
W: 2019/05/07 12:43:00 shell.go:51: flag:(rsync: failed to set times on "/mnt/disk9/media": Operation not permitted (1)
)
I: 2019/05/07 12:43:00 core.go:185: Sending operation
W: 2019/05/07 12:43:01 core.go:763: command:end:error(exit status 13)
I: 2019/05/07 12:43:01 core.go:767: command:retcode(0):exitcode(13)
I: 2019/05/07 12:43:01 core.go:1028: Command Finished
I: 2019/05/07 12:43:01 core.go:1041: Current progress: 92.14% done ~ 0s left (5147219.92 MB/s)
W: 2019/05/07 12:43:01 core.go:960: skipping:deletion:(rsync command was flagged):(/mnt/disk9/media/TV Shows)

So I ran the following terminal command to do it for me:

rsync -av -X "/mnt/disk7/media/TV Shows" "/mnt/disk9/media/"

When that's finished, I'll obviously have two copies of the files in the source which I obviously don't want. Am I save to just press the 'rmsrc' button on the history page in unBALANCE? For example is unRAID going to know I've copied them over and the destination then automatically becomes the files you access via the user share? 

 

My question is because I'm not sure if unRAID achieves its share distribution via sim links or by indexing everything...? If the later, how does it resolve duplicates? If the former how to you update?

[itimpi]

11 minutes ago, dgwharrison said:

My question is because I'm not sure if unRAID achieves its share distribution via sim links or by indexing everything...? If the later, how does it resolve duplicates? If the former how to you update?

Neither!    Unraid uses a FUSe file system to give you its share view.    If a file exists on more than one disk then the one on the lowest number disk is the one shown and any others ignored.  

 

This can sometimes be confusing as if you attempt to delete such a duplicated file it appears to not have worked as the next instance of the file becomes visible as the one that is displayed.   This is especially confusing if the two copies are not identical as the file can appear to have changed.

Edited May 8, 2019 by itimpi

[dgwharrison]

19 hours ago, itimpi said:

Neither!    Unraid uses a FUSe file system to give you its share view.    If a file exists on more than one disk then the one on the lowest number disk is the one shown and any others ignored.  

 

This can sometimes be confusing as if you attempt to delete such a duplicated file it appears to not have worked as the next instance of the file becomes visible as the one that is displayed.   This is especially confusing if the two copies are not identical as the file can appear to have changed.

Ok, that makes sense. So basically if I delete the originals on disk 7 by any means the copies on disk 9 will become the authoritative copies so far as the user shares are concerned right?

[NewDisplayName]

Could there be a option in unbalance to delete duplicate files?

 

All of my duplicates were bc of using this addon (yeah my fault, i know), so it would be cool to have a easy solution for this?

[jbrodriguez]

On 5/7/2019 at 10:28 PM, dgwharrison said:

So I ran the following terminal command to do it for me:

rsync -av -X "/mnt/disk7/media/TV Shows" "/mnt/disk9/media/"

When that's finished, I'll obviously have two copies of the files in the source which I obviously don't want. Am I save to just press the 'rmsrc' button on the history page in unBALANCE?

You were actually running `basically` the same command twice.

 

The app copied the folder, but the underlying rsync failed to set some file's date/time.

 

The rsync command you ran should have shown the same warnings for the same file(s).

[jbrodriguez]

20 hours ago, nuhll said:

Could there be a option in unbalance to delete duplicate files?

That's the purpose of the `rmsrc` button.

 

The app can't automate that because it has to be a user decision.

[NewDisplayName]

On 5/9/2019 at 10:51 PM, jbrodriguez said:

That's the purpose of the `rmsrc` button.

 

The app can't automate that because it has to be a user decision.

No.

 

I mean a option or extra tab to scan for files with same name and or same size on all disks, and then a manual or automatic delete of every extra files.

 

Would be a nice addition.

[axipher]

I've made use of this quite a few times now for disk migrations.  Each time I forget that Mover and/or Plex can cause issues where UnBalance seems to freeze at a spot, but as soon as you change Scheduled Mover from Hourly to Daily and turn off Plex Docker, UnBalance happilly resumes.

 

Would it be possible to add some sort of warning that checks for Mover being enabled and/or Dockers that are known to have high disk usage prior to staring an UnBalanace session?

 

Even something simply like a checkbox that the user clicks to say "Yes I'm aware Docker and other high disk-read Apps may cause UnBalance to Slow to an unresponsive crawl, I would like to continue."

 

 

Otherwise still a handy plug-in to this day and appreciate the work on it.

[JonathanM]

On 5/11/2019 at 8:11 PM, nuhll said:

option or extra tab to scan for files with same name and or same size on all disks, and then a manual or automatic delete of every extra files.

Already available.

 

[geekypenguin]

Hope someone can help me with this issue, or at least point me in a direction to help work out what the issue might be.

So ive never used unbalance before but have had unraid for a few years, and im looking to use the gather function to move my cctv share onto one disk.

 

If I dont have all disk in the array spinning, then unbalance wont list any shares (dont know if its related or not).

My main issue, however, is once i select a share, and click next, I cant go any further. The next screen doesnt list any disks to gather to, the next and previous buttons dont function and the links across the top to settings, log etc are static text and not links. There is just a small coloured bars animation that gives the impression something is doing something, but even after waiting over an hour, still nothing.

 

Any ideas?

 

never mind, i removed the plugin and tried a fresh install and its working now, albeit very slowly.

Edited May 17, 2019 by geekypenguin

[jbrodriguez]

On 5/13/2019 at 2:35 PM, axipher said:

Would it be possible to add some sort of warning that checks for Mover being enabled and/or Dockers that are known to have high disk usage prior to staring an UnBalanace session?

I could check for mover, but other dockers would be difficult.

[jbrodriguez]

2 hours ago, geekypenguin said:

never mind, i removed the plugin and tried a fresh install and its working now, albeit very slowly.

The speed is impacted by a read/write on the same array (since every rsync command goes through parity), whereas writing from another workstation (or similar) doesn't have the same penalty, so you can even reach wire speeds (with a cache drive).

 

Another source of slow down could be disk health.

 

Even if SMART reports are fine, the drive could still be slow.

 

Check this docker

 

[axipher]

18 hours ago, jbrodriguez said:

I could check for mover, but other dockers would be difficult.

That would be great, even just checking to see if Docker's are enabled to give a warning to the user with a check box to proceed as a quick sanity check.

[manolodf]

I start the Unbalance but it perpetually shows as "Stopped" even when I enable it and apply.  

 

May 18 18:45:58 Tower ool www[31806]: /usr/local/emhttp/plugins/unbalance/scripts/start
May 18 18:45:58 Tower sudo: root : TTY=unknown ; PWD=/usr/local/emhttp ; USER=nobody ; COMMAND=/usr/bin/bash -c /usr/local/emhttp/plugins/unbalance/unbalance -port 6237

[d2dyno]

On 5/18/2019 at 6:47 PM, manolodf said:

I start the Unbalance but it perpetually shows as "Stopped" even when I enable it and apply.  

 

May 18 18:45:58 Tower ool www[31806]: /usr/local/emhttp/plugins/unbalance/scripts/start
May 18 18:45:58 Tower sudo: root : TTY=unknown ; PWD=/usr/local/emhttp ; USER=nobody ; COMMAND=/usr/bin/bash -c /usr/local/emhttp/plugins/unbalance/unbalance -port 6237

Same issue here on 6.7.0

[manolodf]

I am also on 6.7.0

 

Glad I am not alone, well not glad, but you know what I mean.  I tried uninstalling the plugin and re-installing it as well without any luck.  I am sure that its something minor but wanted to get it reported.