[Plugin] unbalanced


Recommended Posts

I've just had a similar problem, turns out it was a permissions issue, checking / changing the permissions on the files through the command line might allow it to do its thing, I ended up running the "new permissions" tool and then unbalance would move the files.

 

wgstarks, that error 13 is kind of strange, but historically Apple files have been problematic.

 

In any case, check the permissions issues as unJim mentions. Hopefully that solves it.

 

I'll give some attention to unBALANCE these days :)

 

Link to comment

I've just had a similar problem, turns out it was a permissions issue, checking / changing the permissions on the files through the command line might allow it to do its thing, I ended up running the "new permissions" tool and then unbalance would move the files.

wgstarks, that error 13 is kind of strange, but historically Apple files have been problematic.

 

In any case, check the permissions issues as unJim mentions. Hopefully that solves it.

 

I'll give some attention to unBALANCE these days :)

 

Thanks. Actually, I just reverted to running the same command from terminal.

Link to comment

Dumb question.. if I change the Included Disks from All Disks to Disk 20 on a share that is currently spread on Disks 1, 13, & 22.. Will just new data go to Disk 20 & the old data on Disks 1, 13 & 22 still be accessible?  It doesn't actually move all the data to Disk 20 does it?  Just talking about the settings of a Share.  Thank you for this plug-in, it helps me move stuff around.

Link to comment

Dumb question.. if I change the Included Disks from All Disks to Disk 20 on a share that is currently spread on Disks 1, 13, & 22.. Will just new data go to Disk 20 & the old data on Disks 1, 13 & 22 still be accessible?  It doesn't actually move all the data to Disk 20 does it?  Just talking about the settings of a Share.  Thank you for this plug-in, it helps me move stuff around.

In your example, only the data on disk 20 would be accessible via the user share. The data on the excluded disks would still be accessible via disk shares. No data would be moved.

Link to comment

Dumb question.. if I change the Included Disks from All Disks to Disk 20 on a share that is currently spread on Disks 1, 13, & 22.. Will just new data go to Disk 20 & the old data on Disks 1, 13 & 22 still be accessible?  It doesn't actually move all the data to Disk 20 does it?  Just talking about the settings of a Share.  Thank you for this plug-in, it helps me move stuff around.

In your example, only the data on disk 20 would be accessible via the user share. The data on the excluded disks would still be accessible via disk shares. No data would be moved.

The settings for a specific user share only affect future writes to the share. unRAID will still include any disks that have the share on them for any reads, unless a disk is excluded in Global Share Settings.
Link to comment

v1.8.0 is now available !!!

 

2016-11-11 - 1.8.0

- Show rsync error descriptions (instead of error codes)

- Check permissions on folders to be moved, when calculating. If issues arise, show a warning

- Honor RunAs user setting

- Add Show Log menu option

- UI Layout improvements

 

Just a note ... v1.7.0 was very successful, with almost twice as many downloads as the next best version.

 

Here's hoping 1.8.x is as successful :)

 

 

Link to comment

having a strange error with the latest version, I go to run a calculate operation and the server seems to just crash, there is no response on the webgui port and there is nothing in the logs, I don't know if you have any ideas?

 

I: 2016/11/13 21:39:17 unbalance.go:29: unBALANCE v1.8.0-617.8f6eba1 starting up ...
I: 2016/11/13 21:39:17 unbalance.go:39: Using config file /boot/config/plugins/unbalance/unbalance.conf ...
I: 2016/11/13 21:39:17 server.go:50: Starting service Server ...
I: 2016/11/13 21:39:17 server.go:67: Serving files from /usr/local/emhttp/plugins/unbalance
I: 2016/11/13 21:39:17 server.go:99: Server started listening on :6237
I: 2016/11/13 21:39:17 core.go:119: starting service Core ...
I: 2016/11/13 21:39:17 unbalance.go:49: Press Ctrl+C to stop ...
I: 2016/11/13 21:39:23 core.go:164: Sending config
I: 2016/11/13 21:39:23 unraid.go:391: Unraid Box Condition: &{NumDisks:5 NumProtected:0 Synced:2016-11-13 00:00:01 +0000 GMT SyncErrs:0 Resync:0 ResyncPos:0 State:STARTED Size:4498718736384 Free:1884667150336 NewFree:1884667150336}
I: 2016/11/13 21:39:23 unraid.go:392: Unraid Box SourceDiskName: 
I: 2016/11/13 21:39:23 unraid.go:393: Unraid Box BytesToMove: 0
I: 2016/11/13 21:39:23 unraid.go:404: Id(1); Name(disk1); Path(/mnt/disk1); Device(sdb); Type(Data); FsType(xfs); Free(338.7G); NewFree(338.7G); Size(1.4T); Serial(SAMSUNG_HD154UI_S1XWJ1LSB00488); Status(DISK_OK); Bin(<nil>)
I: 2016/11/13 21:39:23 unraid.go:404: Id(2); Name(disk2); Path(/mnt/disk2); Device(sdc); Type(Data); FsType(xfs); Free(334.3G); NewFree(334.3G); Size(931.1G); Serial(Hitachi_HDT721010SLA360_STF604MH0NH5AB); Status(DISK_OK); Bin(<nil>)
I: 2016/11/13 21:39:23 unraid.go:404: Id(3); Name(disk3); Path(/mnt/disk3); Device(sdd); Type(Data); FsType(xfs); Free(475G); NewFree(475G); Size(931.1G); Serial(Hitachi_HDT721010SLA360_STF604MH0K4RSB); Status(DISK_OK); Bin(<nil>)
I: 2016/11/13 21:39:23 unraid.go:404: Id(4); Name(disk4); Path(/mnt/disk4); Device(sdf); Type(Data); FsType(xfs); Free(607.2G); NewFree(607.2G); Size(931.1G); Serial(Hitachi_HDT721010SLA360_STF604MH0RBJ2B); Status(DISK_OK); Bin(<nil>)
I: 2016/11/13 21:39:27 core.go:274: Toggling dryRun from (false)
I: 2016/11/13 21:39:32 core.go:274: Toggling dryRun from (true)
I: 2016/11/13 21:39:59 core.go:346: Running calculate operation ...
I: 2016/11/13 21:39:59 core.go:380: _calc:Begin:srcDisk(/mnt/disk1); dstDisks(3)
I: 2016/11/13 21:39:59 core.go:383: _calc:elegibleDestDisk(/mnt/disk2)
I: 2016/11/13 21:39:59 core.go:383: _calc:elegibleDestDisk(/mnt/disk3)
I: 2016/11/13 21:39:59 core.go:383: _calc:elegibleDestDisk(/mnt/disk4)
I: 2016/11/13 21:39:59 core.go:648: perms:Scanning disk(/mnt/disk1):folder(/Media/Movies)
I: 2016/11/13 21:39:59 core.go:660: perms:Executing find "/mnt/disk1/Media/Movies/." -exec stat --format "%A|%U:%G|%F|%n" {} \;

Link to comment

having a strange error with the latest version, I go to run a calculate operation and the server seems to just crash, there is no response on the webgui port and there is nothing in the logs, I don't know if you have any ideas?

 

I: 2016/11/13 21:39:17 unbalance.go:29: unBALANCE v1.8.0-617.8f6eba1 starting up ...
I: 2016/11/13 21:39:17 unbalance.go:39: Using config file /boot/config/plugins/unbalance/unbalance.conf ...
I: 2016/11/13 21:39:17 server.go:50: Starting service Server ...
I: 2016/11/13 21:39:17 server.go:67: Serving files from /usr/local/emhttp/plugins/unbalance
I: 2016/11/13 21:39:17 server.go:99: Server started listening on :6237
I: 2016/11/13 21:39:17 core.go:119: starting service Core ...
I: 2016/11/13 21:39:17 unbalance.go:49: Press Ctrl+C to stop ...
I: 2016/11/13 21:39:23 core.go:164: Sending config
I: 2016/11/13 21:39:23 unraid.go:391: Unraid Box Condition: &{NumDisks:5 NumProtected:0 Synced:2016-11-13 00:00:01 +0000 GMT SyncErrs:0 Resync:0 ResyncPos:0 State:STARTED Size:4498718736384 Free:1884667150336 NewFree:1884667150336}
I: 2016/11/13 21:39:23 unraid.go:392: Unraid Box SourceDiskName: 
I: 2016/11/13 21:39:23 unraid.go:393: Unraid Box BytesToMove: 0
I: 2016/11/13 21:39:23 unraid.go:404: Id(1); Name(disk1); Path(/mnt/disk1); Device(sdb); Type(Data); FsType(xfs); Free(338.7G); NewFree(338.7G); Size(1.4T); Serial(SAMSUNG_HD154UI_S1XWJ1LSB00488); Status(DISK_OK); Bin(<nil>)
I: 2016/11/13 21:39:23 unraid.go:404: Id(2); Name(disk2); Path(/mnt/disk2); Device(sdc); Type(Data); FsType(xfs); Free(334.3G); NewFree(334.3G); Size(931.1G); Serial(Hitachi_HDT721010SLA360_STF604MH0NH5AB); Status(DISK_OK); Bin(<nil>)
I: 2016/11/13 21:39:23 unraid.go:404: Id(3); Name(disk3); Path(/mnt/disk3); Device(sdd); Type(Data); FsType(xfs); Free(475G); NewFree(475G); Size(931.1G); Serial(Hitachi_HDT721010SLA360_STF604MH0K4RSB); Status(DISK_OK); Bin(<nil>)
I: 2016/11/13 21:39:23 unraid.go:404: Id(4); Name(disk4); Path(/mnt/disk4); Device(sdf); Type(Data); FsType(xfs); Free(607.2G); NewFree(607.2G); Size(931.1G); Serial(Hitachi_HDT721010SLA360_STF604MH0RBJ2B); Status(DISK_OK); Bin(<nil>)
I: 2016/11/13 21:39:27 core.go:274: Toggling dryRun from (false)
I: 2016/11/13 21:39:32 core.go:274: Toggling dryRun from (true)
I: 2016/11/13 21:39:59 core.go:346: Running calculate operation ...
I: 2016/11/13 21:39:59 core.go:380: _calc:Begin:srcDisk(/mnt/disk1); dstDisks(3)
I: 2016/11/13 21:39:59 core.go:383: _calc:elegibleDestDisk(/mnt/disk2)
I: 2016/11/13 21:39:59 core.go:383: _calc:elegibleDestDisk(/mnt/disk3)
I: 2016/11/13 21:39:59 core.go:383: _calc:elegibleDestDisk(/mnt/disk4)
I: 2016/11/13 21:39:59 core.go:648: perms:Scanning disk(/mnt/disk1):folder(/Media/Movies)
I: 2016/11/13 21:39:59 core.go:660: perms:Executing find "/mnt/disk1/Media/Movies/." -exec stat --format "%A|%U:%G|%F|%n" {} \;

 

Hi Bizmo, I'd say it's crashing :)

 

Would you mind running that last line in the console, see if throws any error ? I'll check the code as well.

 

find "/mnt/disk1/Media/Movies/." -exec stat --format "%A|%U:%G|%F|%n" {} \;

Link to comment

Is there a way to gracefully terminate an unBALANCE job in progress?

 

Hi jnheinz, well gracefully isn't possible.

 

I'd do

 

killall rsync
killall unbalance

 

You'd be left with a partially copied folder.

 

The log will tell you which folders were copied and which one was in progress.

 

Then you'd have to do some manual tending after (delete the partially copied folder on the target disk maybe?)

Link to comment

Is there a way to gracefully terminate an unBALANCE job in progress?

 

Hi jnheinz, well gracefully isn't possible.

 

I'd do

 

killall rsync
killall unbalance

 

You'd be left with a partially copied folder.

 

The log will tell you which folders were copied and which one was in progress.

 

Then you'd have to do some manual tending after (delete the partially copied folder on the target disk maybe?)

 

Ok, thanks.  I will probably hold off.  Thank you for this plug-in, it has been very helpful.

 

Unrelated question, I'll lay out the scenario to see if this is something unBALANCE can handle.

 

I have a share called TV shows that is roughly 5TB in size, spread out amongst about 6 disks currently.  I have three 2TB disks that are empty that I would like to migrate these shows to.  However, I do want to observe the correct split level.. which is keeping the entire series together, so I have it set to split the top level only now.  I don't think it was set this way when it was originally copied, so the data has ended up everywhere.  In unBALANCE, If I go through each source disk with tvshows selected & select the 3 target disks.. Does it observe the split level?  Or will I need to split manually?  Am I better off moving them all out of the share & back in?  Looking for the most efficient approach to this.

Link to comment

 

Hi Bizmo, I'd say it's crashing :)

 

Would you mind running that last line in the console, see if throws any error ? I'll check the code as well.

 

find "/mnt/disk1/Media/Movies/." -exec stat --format "%A|%U:%G|%F|%n" {} \;

 

That just returns a list of the files in that folder in the console

 

E.g.

 

-rw-r--r--|nobody:users|regular file|/mnt/disk1/Media/Movies/./The Constant Gardener (HD).m4v
-rw-r--r--|nobody:users|regular file|/mnt/disk1/Media/Movies/./The Secret Life of Walter Mitty (HD).m4v
-rw-r--r--|nobody:users|regular file|/mnt/disk1/Media/Movies/./The Dark Knight Rises (HD).m4v
-rw-r--r--|nobody:users|regular file|/mnt/disk1/Media/Movies/./The Spy Who Loved Me (HD).m4v
-rw-r--r--|nobody:users|regular file|/mnt/disk1/Media/Movies/./The Devil Wears Prada (HD).m4v
-rw-r--r--|nobody:users|regular file|/mnt/disk1/Media/Movies/./The Five-Year Engagement.m4v
-rw-r--r--|nobody:users|regular file|/mnt/disk1/Media/Movies/./The Sword in the Stone.m4v
-rw-r--r--|nobody:users|regular file|/mnt/disk1/Media/Movies/./The Girl with the Dragon Tattoo.m4v
-rw-r--r--|nobody:users|regular file|/mnt/disk1/Media/Movies/./The Theory of Everything (1080p HD).m4v
-rw-r--r--|nobody:users|regular file|/mnt/disk1/Media/Movies/./The Grand Budapest Hotel (HD).m4v
-rw-r--r--|nobody:users|regular file|/mnt/disk1/Media/Movies/./Trainwreck (2015).mkv
-rw-r--r--|nobody:users|regular file|/mnt/disk1/Media/Movies/./The Great Gatsby (HD) 2.m4v
-rw-r--r--|nobody:users|regular file|/mnt/disk1/Media/Movies/./Who Framed Roger Rabbit.m4v
-rw-r--r--|nobody:users|regular file|/mnt/disk1/Media/Movies/./The Help (HD).m4v
-rw-r--r--|nobody:users|regular file|/mnt/disk1/Media/Movies/./The Lucky One (HD).m4v
-rw-r--r--|nobody:users|regular file|/mnt/disk1/Media/Movies/./The Hobbit The Desolation of Smaug (2013).mkv

 

EDIT: Interestingly it appears it works fine with other user shares but does report back this error

 

There are some permission issues with the folders/files you want to move
0 file(s)/folder(s) with an owner other than 'nobody'
0 file(s)/folder(s) with a group other than 'users'
0 folder(s) with a permission other than 'drwxrwxrwx'
18 files(s) with a permission other than '-rw-rw-rw-' or '-r--r--r--'
You can find more details about which files have issues in the log file (/boot/logs/unbalance.log)
At this point, you can move the folders/files if you want, but be advised that it can cause errors in the operation
You are STRONGLY suggested to install the Fix Common Problems plugin, then run the Docker Safe New Permissions command

 

I have downloaded this plugin and run the command but it doesn't change anything

Link to comment

v1.8.0 is now available !!!

 

2016-11-11 - 1.8.0

- Show rsync error descriptions (instead of error codes)

- Check permissions on folders to be moved, when calculating. If issues arise, show a warning

- Honor RunAs user setting

- Add Show Log menu option

- UI Layout improvements

 

Just a note ... v1.7.0 was very successful, with almost twice as many downloads as the next best version.

 

Here's hoping 1.8.x is as successful :)

 

Hi

 

First of all thanks for creating this plugin!

Second I want to try it out for the first time on my very first UnRAID installation, but according the log it would start except I can't get in the webUI...

 

I'm running UnRAID 6.2.4 (at the moment still in trial modus).

The plugin version of unBalanced is v1.8.0 and in the settings of unBalanced even if I set the option to enable the server to "Yes" still it won't work.

 

This is what I get in my log after enabling the plugin:

Nov 17 13:49:32 *** php: /usr/local/emhttp/plugins/unbalance/scripts/start
Nov 17 13:49:32 *** sudo:     root : TTY=unknown ; PWD=/usr/local/emhttp ; USER=nobody ; COMMAND=/bin/bash -c /usr/local/emhttp/plugins/unbalance/unbalance -port 6237
Nov 17 13:49:35 *** emhttp: cmd: /usr/local/emhttp/plugins/dynamix/scripts/tail_log syslog

 

Then I checked the permissions on the folder and files for unbalance and this is the output:

root@***:/usr/local/emhttp/plugins/unbalance# ls -al
total 7980
drwxr-xr-x  7  501 wheel     280 Nov 11 18:14 ./
drwxrwxrwx 36 root root      720 Nov 17 13:57 ../
-rw-r--r--  1  501 wheel   33487 Nov 11 18:14 CHANGES
-rw-r--r--  1  501 wheel     142 Nov 11 18:14 README.md
-rw-r--r--  1  501 wheel       6 Nov 11 18:14 VERSION
drwxr-xr-x  2  501 wheel      80 Nov 17 13:57 app/
drwxr-xr-x  2  501 wheel      80 Nov 17 13:57 event/
drwxr-xr-x  2  501 wheel      60 Nov 17 13:57 images/
drwxr-xr-x  2  501 wheel     240 Nov 17 13:57 img/
-rw-r--r--  1  501 wheel     666 Nov 11 18:14 index.html
drwxr-xr-x  2  501 wheel      80 Nov 17 13:57 scripts/
-rwxr-xr-x  1  501 wheel 7937102 Nov 11 18:14 unbalance*
-rw-r--r--  1  501 wheel    4030 Nov 11 18:14 unbalance.page
-rw-r--r--  1  501 wheel  179257 Nov 11 18:14 unbalance.png

 

To me that doesn't seem right, because user is set to "501" en group is set to "wheel".

Do I need and is it safe to chmod (755) and/or chown the unbalance plugin folder and all the files inside to a "correct/known" user and/or group?

 

The following I've already tried, but all result into the same log:


  • Tried a different port to start the plugin
     

  • Tried a different user to start the plugin
     

  • Tried uninstalling and installing the plugin
     

 

I would like to take a look into the possibilities of this plugin, because now it seems to be doing nothing even if it is enabled.

 

Edit:

What can I do to solve this issue and is this enough information to take a look into the problem or what information is needed to further look into this?

Link to comment

Check created issue on github for not starting issue:

https://github.com/jbrodriguez/unbalance/issues/1

 

maybe with more examples the author can pin point what is/can be wrong...

 

For me this issue is solved by doing the following:

If the plugin fails to start: "create a /boot/config/plugins/unbalance/unbalance.conf file (it should be fine empty)"

The unbalance.cfg file is another file, so don't think there is already a config file.  ???:P

Link to comment

...

 

Unrelated question, I'll lay out the scenario to see if this is something unBALANCE can handle.

 

I have a share called TV shows that is roughly 5TB in size, spread out amongst about 6 disks currently.  I have three 2TB disks that are empty that I would like to migrate these shows to.  However, I do want to observe the correct split level.. which is keeping the entire series together, so I have it set to split the top level only now.  I don't think it was set this way when it was originally copied, so the data has ended up everywhere.  In unBALANCE, If I go through each source disk with tvshows selected & select the 3 target disks.. Does it observe the split level?  Or will I need to split manually?  Am I better off moving them all out of the share & back in?  Looking for the most efficient approach to this.

 

Hi jnheinz, not sure what the most efficient approach could be, but I can tell you that it won't observe split levels since it writes directly to disks.

Link to comment

Would you mind running that last line in the console, see if throws any error ? I'll check the code as well.

 

find "/mnt/disk1/Media/Movies/." -exec stat --format "%A|%U:%G|%F|%n" {} \;

 

That just returns a list of the files in that folder in the console

 

...

 

There are some permission issues with the folders/files you want to move
0 file(s)/folder(s) with an owner other than 'nobody'
0 file(s)/folder(s) with a group other than 'users'
0 folder(s) with a permission other than 'drwxrwxrwx'
18 files(s) with a permission other than '-rw-rw-rw-' or '-r--r--r--'
You can find more details about which files have issues in the log file (/boot/logs/unbalance.log)
At this point, you can move the folders/files if you want, but be advised that it can cause errors in the operation
You are STRONGLY suggested to install the Fix Common Problems plugin, then run the Docker Safe New Permissions command

 

I have downloaded this plugin and run the command but it doesn't change anything

 

Bizmo, ok .. I thought that the first command was the one crashing.

 

The message is just a warning, since there are multiple instances where permission issues cause problems with rsync. The Docker Safe New Permissions resets permissions in the best possible way.

 

I had another idea ... when it's sitting idle as you mention .. can you do a

ps aux | grep unbalance

  and a

ps aux | grep find

... that would tell us if it's really running or not

Link to comment

@kziegler @AnyColorYouLike @ZataH as discussed on github, the app is exiting if it doesn't find /boot/config/plugins/unbalance/unbalance.conf

 

I found a bug in the template: it creates unbalance.cfg instead of unbalance.conf, so fixing that should get it working again :)

 

Thanks kziegler for opening the issue on github !

 

Thanks a lot to @tobytl on github, for actually finding the root cause, and to all of you for the patience while figuring this out.

Link to comment

Not sure if I am using the plugin wrong :)

Lets say I want to move all the appdata to my ssd cache.

Is it not posslbe to mark all the from to the 1 target?

Right now i have to do jobs for every single drive

It's possible I'm misunderstanding what you are trying to accomplish, but it you want to move your appdata share to cache all you have to do is go into the share settings and set cache disk to prefer. Then run the mover. Mover will relocate the appdata share to cache.

Link to comment

Not sure if I am using the plugin wrong :)

Lets say I want to move all the appdata to my ssd cache.

Is it not posslbe to mark all the from to the 1 target?

Right now i have to do jobs for every single drive

It's possible I'm misunderstanding what you are trying to accomplish, but it you want to move your appdata share to cache all you have to do is go into the share settings and set cache disk to prefer. Then run the mover. Mover will relocate the appdata share to cache.

 

Ofc :) Dont know why i didnt think about that. Thanks

Link to comment
  • jbrodriguez changed the title to [Plugin] unbalanced

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.