[Plugin] unbalanced

[jbrodriguez]

unBALANCE 2 is now available !

 

2017-01-13 - 2.0.0

unBALANCE 2 now displays a disk explorer in its main interface, in order to improve the experience of choosing which files/folders need to be moved off of a disk.

 

Also in this release:

- General improvements and bug fixes

 

The core operation is unchanged, but now the app is easier to operate.

[JustinChase]

unBALANCE 2 is now available !

 

2017-01-13 - 2.0.0

unBALANCE 2 now displays a disk explorer in its main interface, in order to improve the experience of choosing which files/folders need to be moved off of a disk.

 

Also in this release:

- General improvements and bug fixes

 

The core operation is unchanged, but now the app is easier to operate.

Great, I look forward to trying it. I was going to report that the current version wasn't working for me.

 

I am trying to move everything off of disk 3, and it appears I have plenty of room on all the other disks to clear disk 3, but unbalance is telling me there isn't enough room.

 

It feels like it's doing its math on the one remaining folder on disk 3, which contains 1.5 TB of movies, and no single other disk has this much room, but there is almost 2TB of free space on all the other disks, so it should all fit.

 

I'll update and try again when I get home, but wanted to report the issue anyway, in case it is a bug the hasn't gotten fixed already.

 

Also, thanks for changing the way it works, I think it will be far more intuitive with your changes.

 

Posted from my phone, forgive any spelling errors.

 

 

[jbrodriguez]

...

I'll update and try again when I get home, but wanted to report the issue anyway, in case it is a bug the hasn't gotten fixed already.

 

Also, thanks for changing the way it works, I think it will be far more intuitive with your changes.

 

Posted from my phone, forgive any spelling errors.

 

Thanks for your comments !

 

Let me know how it goes.

[cpthook]

Hello!  I tried the plugin for the first time but was unable to get the WebGUI to display the file transfer progress information and transfer speed.  The log window will update for a bit, but will time out shortly after.  When the GUI is refreshed, the log window is empty.  Are there any tricks or a particular browser I could use for the GUI to display properly?  This plugin is awesome, but I would feel more comfortable using it, if I could monitor the progress of the files being transferred.

 

 

[jbrodriguez]

Hello!  I tried the plugin for the first time but was unable to get the WebGUI to display the file transfer progress information and transfer speed.  The log window will update for a bit, but will time out shortly after.  When the GUI is refreshed, the log window is empty.  Are there any tricks or a particular browser I could use for the GUI to display properly?  This plugin is awesome, but I would feel more comfortable using it, if I could monitor the progress of the files being transferred.

 

Hi cpthook,

 

I generally use Chrome and it works fine for me

 

Ff you check the raw log (/boot/logs/unbalance.log), does it show any progress ?

[cpthook]

Ff you check the raw log (/boot/logs/unbalance.log), does it show any progress ?

 

Yes it does!  But I guess I am expecting "Real Time" statuses.  See attachment; using Chrome, statuses are now shown but  has been stuck in this state for about 2hours.  I'm noticing it only being refreshed after the transfer of an entire share is complete.  Should this be expected behavior or should the status be refreshed more often?  Currently transferring 1.5TB of data to clear a disk for format to xfs. 

[jbrodriguez]

Ff you check the raw log (/boot/logs/unbalance.log), does it show any progress ?

 

Yes it does!  But I guess I am expecting "Real Time" statuses.  See attachment; using Chrome, statuses are now shown but  has been stuck in this state for about 2hours.  I'm noticing it only being refreshed after the transfer of an entire share is complete.  Should this be expected behavior or should the status be refreshed more often?  Currently transferring 1.5TB of data to clear a disk for format to xfs.

 

Oh ok.

 

Currently, that's expected behavior.

 

It refreshes after each rsync command has completed.

 

I wrote about that when I first introduced the progress feature, but forgot to mention it in the updated github README.

 

The thing is rsync's progress is somewhat difficult to parse, so I just track the whole operation, not the individual files.

 

I'll eventually look into improving that

 

 

[ryanborstelmann]

unBALANCE 2 is now available !

 

2017-01-13 - 2.0.0

unBALANCE 2 now displays a disk explorer in its main interface, in order to improve the experience of choosing which files/folders need to be moved off of a disk.

 

Also in this release:

- General improvements and bug fixes

 

The core operation is unchanged, but now the app is easier to operate.

Great, I look forward to trying it. I was going to report that the current version wasn't working for me.

 

I am trying to move everything off of disk 3, and it appears I have plenty of room on all the other disks to clear disk 3, but unbalance is telling me there isn't enough room.

 

It feels like it's doing its math on the one remaining folder on disk 3, which contains 1.5 TB of movies, and no single other disk has this much room, but there is almost 2TB of free space on all the other disks, so it should all fit.

 

I'll update and try again when I get home, but wanted to report the issue anyway, in case it is a bug the hasn't gotten fixed already.

 

Also, thanks for changing the way it works, I think it will be far more intuitive with your changes.

 

Posted from my phone, forgive any spelling errors.

 

 

+1 here - trying to migrate about 600GB of data across ~5TB of available space on the array, and it reports that there isn't enough disk space. It's like it's not splitting the folder structure across the drives, it appears as though it's just trying to move the main folder and that's that.

 

Thoughts on how to fix this?

[jbrodriguez]

...

+1 here - trying to migrate about 600GB of data across ~5TB of available space on the array, and it reports that there isn't enough disk space. It's like it's not splitting the folder structure across the drives, it appears as though it's just trying to move the main folder and that's that.

 

Thoughts on how to fix this?

 

Hi,

 

I suggest that you select the sub-folders of the main folder.

 

Or whatever level of granularity you want to work with.

 

What the app does is (roughly):

Given a folder, get the sizes of the first-level folder/files

Allocate them across the target disk(s) (recreating the whole tree)

 

An example

 

If you have a folder structure such as this:

- films

-- bluray

--- Movie1

--- Movie2

...

--- Movie 999

-- dvd

 

and select films, it will try to allocate bluray (which might be in the TB range) and dvd.

 

Unless you have an empty target disk, most likely it won't be able to allocate the bluray folder.

 

If you select films/bluray, it will try to allocate the individual MovieX's, which will be roughly 50GB each, and that is very likely to succeed.

 

The app depends on the user providing the "work unit" to operate on, to put it in other words.

[derekos]

I am grateful for this plugin.  I just installed it and am giving it a test run moving all the files off of one drive to another.  If it works as intended, this may be much easier for me to manage converting 24 drives from ReiserFS to XFS. 

 

Use unbalance to empty a drive.  Re-format empty drive as XFS.  Rinse and repeat.

 

Does anyone know if there is a problem with this approach?

 

 

 

[jbrodriguez]

I am grateful for this plugin.  I just installed it and am giving it a test run moving all the files off of one drive to another.  If it works as intended, this may be much easier for me to manage converting 24 drives from ReiserFS to XFS. 

 

Use unbalance to empty a drive.  Re-format empty drive as XFS.  Rinse and repeat.

 

Does anyone know if there is a problem with this approach?

 

Should work

[jbrodriguez]

v2.1.0 is now available !

 

2017-02-03 - 2.1.0

 

This release implements real time move progress !

 

Progress is synchronized with rsync transfer updates.

 

Also in this release:

- Bug fixes and improvements

 

 

The default rsync flags are now: -avPRX (previously "avRX --partial")

 

The -P represents "--progress --partial", so this change is basically adding --progress, which is required to make rsync show progress as it transfer files.

[derekos]

Great work.  Looking forward to using it.  2.0x is been working great.  I've migrated two 4TB and two 3TB drives to XFS without an issue.

 

I do have a suggestion, rather than relying on that steamroller Fix Common Problems to reset permissions (slow, locks array up, etc).  Its much faster to take a scalpel to it:

 

Just change directory to the disk with the problems:  cd /mnt/diskXX

 

If ownership problems - chmod -R nobody:user

If file permission problems - find \! -perm 666 -type f -print0 | xargs -0 -I {} chmod 666 {}

If directory permission problems - find \! -perm 777 -type d -print0 | xargs -0 -I {} chmod 777 {}

 

These commands only replace file permissions as needed on the bad files/directory.  Takes seconds.

 

 

 

 

[itimpi]

Great work.  Looking forward to using it.  2.0x is been working great.  I've migrated two 4TB and two 3TB drives to XFS without an issue.

 

I do have a suggestion, rather than relying on that steamroller Fix Common Problems to reset permissions (slow, locks array up, etc).  Its much faster to take a scalpel to it:

 

Just change directory to the disk with the problems:  cd /mnt/diskXX

 

If ownership problems - chmod -R nobody:user

If file permission problems - find \! -perm 666 -type f -print0 | xargs -0 -I {} chmod 666 {}

If directory permission problems - find \! -perm 777 -type d -print0 | xargs -0 -I {} chmod 777 {}

 

These commands only replace file permissions as needed on the bad files/directory.  Takes seconds.

i would have thought it is even easier to do

newperms path-to-folder

to reset permissions for the folder and all its contents.    That will work at any level in unRAID. 

 

If the parameter is omitted then it does all drives and is equivalent to the running the 'New Permissions' option from the GUI.

[Squid]

If the parameter is omitted then it does all drives and is equivalent to the running the 'New Permissions' option from the GUI.

Do not do this if you have docker applications running.

[jbrodriguez]

If the parameter is omitted then it does all drives and is equivalent to the running the 'New Permissions' option from the GUI.

Do not do this if you have docker applications running.

 

Yes, I remember Squid mentioning that FCP's newperms is Docker-safe.

[derekos]

i would have thought it is even easier to do

newperms path-to-folder

to reset permissions for the folder and all its contents.    That will work at any level in unRAID. 

 

If the parameter is omitted then it does all drives and is equivalent to the running the 'New Permissions' option from the GUI.

 

I'm not sure how the newperms command works.  The ones I provided only change the directories/files with bad permissions, rather than resetting everything.    I was mainly trying to avoid running Fix Common Problems permissions fix because it hits everything everywhere and takes forever.

[Squid]

I was mainly trying to avoid running Fix Common Problems permissions fix because it hits everything everywhere and takes forever.

Because it's a replacement for the stock New Perms tool.  But if its taking forever, then it may be hitting your appdata (make sure you have your default appdata location within settings - docker set up correctly), or are running an old version which was hitting the appdata backup created with CA Appdata Backup.

 

But realistically, on a properly configured system there should never be a reason to have to run newperms ever...

[cpthook]

Ff you check the raw log (/boot/logs/unbalance.log), does it show any progress ?

 

Yes it does!  But I guess I am expecting "Real Time" statuses.  See attachment; using Chrome, statuses are now shown but  has been stuck in this state for about 2hours.  I'm noticing it only being refreshed after the transfer of an entire share is complete.  Should this be expected behavior or should the status be refreshed more often?  Currently transferring 1.5TB of data to clear a disk for format to xfs.

 

 

 

Oh ok.

 

Currently, that's expected behavior.

 

It refreshes after each rsync command has completed.

 

I wrote about that when I first introduced the progress feature, but forgot to mention it in the updated github README.

 

The thing is rsync's progress is somewhat difficult to parse, so I just track the whole operation, not the individual files.

 

I'll eventually look into improving that

 

 

Thank you so much for the update!  Used this morning to clear cache drive for formatting and it worked per my request above. 

[jbrodriguez]

...

Thank you so much for the update!  Used this morning to clear cache drive for formatting and it worked per my request above.

 

Thanks for the comment ! Glad to know it's working

[derekos]

if its taking forever, then it may be hitting your appdata (make sure you have your default appdata location within settings - docker set up correctly), or are running an old version which was hitting the appdata backup created with CA Appdata Backup.

 

But realistically, on a properly configured system there should never be a reason to have to run newperms ever...

 

Thanks @Squid.  I'll take a closer look.  You are almost certainly right, I bet that my docker appdata location is problematic.   

[trurl]

Used this morning to clear cache drive for formatting and it worked per my request above.

Modifying shares cache setting as needed then running mover is the usual way to clear the cache drive. See here.

[Madhouse]

First of all, thanks for a great plugin.

 

second: I think it would be nice to have an option to cancel a running move.

 

 

[kizer]

Ok if this has been answered somewhere I apologize. However I'm feeling rather dumb right now.

 

There are some permission issues with the folders/files you want to move

51 file(s)/folder(s) with an owner other than 'nobody'

0 file(s)/folder(s) with a group other than 'users'

0 folder(s) with a permission other than 'drwxrwxrwx'

0 files(s) with a permission other than '-rw-rw-rw-' or '-r--r--r--'

You can find more details about which files have issues in the log file (/boot/logs/unbalance.log)

At this point, you can move the folders/files if you want, but be advised that it can cause errors in the operation

You are STRONGLY suggested to install the Fix Common Problems plugin, then run the Docker Safe New Permissions command

 

I do have the Fix Common Problems plugin installed as well. What exactly am I trying to do here. Lol

I'm trying to move a <Movie> Share

 

I did figure out I have some files that are [nobody] and some are [cj]

cj is my machine that I normally upload files to the server from.

 

 

********Edit************

Went in and simply chown -R nobody * and it removed the errors, but when I run a dry run it seems to keep getting stuck on the same folder. Not sure why as of yet,but I have progress.

 

Folder name has a & I wonder if that's what freezing up the simulated move. I'll give it an edit and see if that fixes it.

Nope. Still wont pass this particular folder. I'll keep doing some testing and possibly just remove the folder all together and keep tinkering.

[Madhouse]

Ok if this has been answered somewhere I apologize. However I'm feeling rather dumb right now.

 

There are some permission issues with the folders/files you want to move

51 file(s)/folder(s) with an owner other than 'nobody'

0 file(s)/folder(s) with a group other than 'users'

0 folder(s) with a permission other than 'drwxrwxrwx'

0 files(s) with a permission other than '-rw-rw-rw-' or '-r--r--r--'

You can find more details about which files have issues in the log file (/boot/logs/unbalance.log)

At this point, you can move the folders/files if you want, but be advised that it can cause errors in the operation

You are STRONGLY suggested to install the Fix Common Problems plugin, then run the Docker Safe New Permissions command

 

I do have the Fix Common Problems plugin installed as well. What exactly am I trying to do here. Lol

I'm trying to move a <Movie> Share

 

I did figure out I have some files that are [nobody] and some are [cj]

cj is my machine that I normally upload files to the server from.

 

 

********Edit************

Went in and simply chown -R nobody * and it removed the errors, but when I run a dry run it seems to keep getting stuck on the same folder. Not sure why as of yet,but I have progress.

 

Folder name has a & I wonder if that's what freezing up the simulated move. I'll give it an edit and see if that fixes it.

Nope. Still wont pass this particular folder. I'll keep doing some testing and possibly just remove the folder all together and keep tinkering.

 

Found this a couple of posts back.

 

Just change directory to the disk with the problems:  cd /mnt/diskXX

 

If ownership problems - chmod -R nobody:user

If file permission problems - find \! -perm 666 -type f -print0 | xargs -0 -I {} chmod 666 {}

If directory permission problems - find \! -perm 777 -type d -print0 | xargs -0 -I {} chmod 777 {}