58 minutes ago, wreave said:

Here you go.

https://pastebin.com/jCB0kFRt

 

It looks clean as well.

 

So in your case it didn't move any data ?

 

I'll check a couple more things and get back to you.

 

 

v3.3.2 is available !

 

2017-08-20 - 3.3.2

 

This release fixes the issue with the wrong progress percentages.

 

Also fixes the issue with the source folder not being removed.

 

Finally, the plugin now checks for an update when starting.

This is enabled by default, but can be turned off in the settings.

 

@wreave I haven't found a reason why it wouldn't transfer data. If you want, let's pm to further troubleshoot.

 

I will check into deleting the top source folder (if empty) in an upcoming release.
 

Thanks for the update!  I'll check things over the next few days...really love the new Gather feature

On 20/08/2017 at 10:19 PM, jbrodriguez said:

v3.3.2 is available !

 

2017-08-20 - 3.3.2

 

This release fixes the issue with the wrong progress percentages.

 

Also fixes the issue with the source folder not being removed.

 

Finally, the plugin now checks for an update when starting.

This is enabled by default, but can be turned off in the settings.

 

@wreave I haven't found a reason why it wouldn't transfer data. If you want, let's pm to further troubleshoot.

 

I will check into deleting the top source folder (if empty) in an upcoming release.

 

 

Working for me now.

 

Wasn't deleting the files off the other disks on 3.3.0

 

 

Would be awesome if you could get rid of the empty folders save going into each disk when you have lots of them scattered around.

Edited August 22, 20178 yr by _george

I've had the web UI for unbalance stop working a few times. This is the last line in the log file when this happens.
 

W: 2017/08/22 06:37:08 server.go:322: closing socket (&{id: ws:0xc420182090 onMessage:0x6e1520 onClose:0x6e1570}): EOF

Disabling and re-enabling unbalance gets it back up. I'd be happy to provide additional logs the next time it happens if that would help.

 

Edit: The last time this happened (a few hours ago) it did not complete the move it was in the process of. So that indicates that maybe unbalance crashed?

Edited August 22, 20178 yr by wreave

On 8/22/2017 at 3:11 AM, _george said:

Would be awesome if you could get rid of the empty folders save going into each disk when you have lots of them scattered around.

Yes, I will definitely address this.

 

@wreave is doing "find <source folder> -type d -empty -delete".

 

I'm thinking about replicating it to delete the empy folder left behind.

 

Does that sound ok ? (if you're knowledgeable in linux command line, please chime in).

20 hours ago, wreave said:

This is the last line in the log file when this happens.

 

That message means that the websocket was closed, because either the browser or the tab showing unbalance was closed. It's mostly informational.

 

20 hours ago, wreave said:

So that indicates that maybe unbalance crashed?

 

Do you have full notifications enabled ? If the operation was interrupted, it would send an email stating so.

 

Otherwise, if it may have crashed.

 

Let me know.

2 hours ago, jbrodriguez said:

 

That message means that the websocket was closed, because either the browser or the tab showing unbalance was closed. It's mostly informational.

 

 

Do you have full notifications enabled ? If the operation was interrupted, it would send an email stating so.

 

Otherwise, if it may have crashed.

 

Let me know.

Looks like notifications were off, I'll turn them on and monitor.

Hi.

 

Have just installed unbalanced and in my opinion it seems to be a great utility!

 

 

Two questions!

1. I'm currently moving files from one drive to another. both normal sata drives. Is the speed supposed to be about 30MB/s?
2. When using the gather function, should i also change the setting for the share from using all drives to only the drive that i'm moving the files to? I thought that changing the settings for the share regarding which disks to use (and not) would do the same thing as this program but that are maybe only for new files added to the share!?

7 hours ago, wreave said:

Looks like notifications were off, I'll turn them on and monitor.

Happened again, no notification and nothing obvious in the logs. Enabling verbose logging and will see what that yields.

1 hour ago, mrvilla said:

Two questions!

 

Correct on both!

 

1. Parity slows down writes to array disks, and when the source is an array disk as well, you see the results. Turbo write MAY help, may not.

2. Only new files. Unraid will not move files from one array disk to another, only from array to cache and vice versa depending on cache:prefer or yes, and only inside a user share. Theoretically you could manipulate the cache settings and disk allocation for a share and use the mover to do what you are doing, but it would be kind of a pain depending on the size of your cache drive.

8 minutes ago, jonathanm said:

 

Correct on both!

 

1. Parity slows down writes to array disks, and when the source is an array disk as well, you see the results. Turbo write MAY help, may not.

2. Only new files. Unraid will not move files from one array disk to another, only from array to cache and vice versa depending on cache:prefer or yes, and only inside a user share. Theoretically you could manipulate the cache settings and disk allocation for a share and use the mover to do what you are doing, but it would be kind of a pain depending on the size of your cache drive.

 

Ah! Didn't remember the parity! 

 

Thanks for the explanation!:)

1 hour ago, wreave said:

Enabling verbose logging and will see what that yields.

 

If it happens again, in addition to the verbose logging, try starting the plugin from a tmux command line, to show when/if it crashes.

 

Something like 

$ tmux
$ killall unbalance
$ cd /usr/local/emhttp/plugins/unbalance
$ ./unbalance

 

6 hours ago, jbrodriguez said:

 

If it happens again, in addition to the verbose logging, try starting the plugin from a tmux command line, to show when/if it crashes.

 

Something like 

$ tmux
$ killall unbalance
$ cd /usr/local/emhttp/plugins/unbalance
$ ./unbalance

 

I finished up all the transfers I had without another crash. Hopefully, it was just my machine. The only possible correlation I ever saw was having the UI open in the browser and having my PC disconnect from the Internet.

 

Great plugin that really saved me some time!

Edited August 24, 20178 yr by wreave

Hi

 

Have an issue. I want to move the folder that contains all my files that I share on Nextcloud. It comes up with an message that some files and folders have the wrong permission. It tells my to run new safe permissions. I have done that but there are still folders that have an permission issue.

 

Any ideas?

14 minutes ago, mrvilla said:

Hi

 

Have an issue. I want to move the folder that contains all my files that I share on Nextcloud. It comes up with an message that some files and folders have the wrong permission. It tells my to run new safe permissions. I have done that but there are still folders that have an permission issue.

 

Any ideas?

Are you files stored in the nextcloud config folder? Also you might run the risk of nextcloud not seeing the data if you try and move it depending on your settings.

Edited August 24, 20178 yr by wreave

2 minutes ago, wreave said:

Are you files stored in the nextcloud config folder? Also you might run the risk of nextcloud not seeing the data if you try and move it depending on your settings.

 

It is stored in /mnt/user/NextcloudFiles

 

Are you serious? This mean that I can never change the filesystem on my drive.

I thought the shares worked like it made similar system on all disks so it didn't matter which drive the information is stored on.

3 minutes ago, mrvilla said:

 

It is stored in /mnt/user/NextcloudFiles

 

Are you serious? This mean that I can never change the filesystem on my drive.

I thought the shares worked like it made similar system on all disks so it didn't matter which drive the information is stored on.

So I think you'll be okay moving your existing data based on how you are setup. In regards to the permission issues, I am honestly not sure about that. Perhaps jbrodriguez will have more info.

 

Does it tell you what folder has the permission issues? Is it the entire share?

I marked every folder and files in the folder and the calculation went well. When i Mark the folder itself it comes up with permission issue on one folder. Apparently it is the NextcloudFiles folder.  

18 hours ago, mrvilla said:

Apparently it is the NextcloudFiles folder

 

So, you identified the folder.

 

The message is a general warning, the rsync command could fail due to permission issues, but it may work if the user under which unbalance runs has enough rights. It depends on your scenario really.

 

I don't know how nextcloud handles permissions and such. If you're running it off a docker, I understand the docker safe new perms won't touch those files.

I'm wondering if UnBalance can help me in this situation now, or perhaps this is in the roadmap for the future...

 

I'm wanting to migrate to use Split Levels on my User Shares which are presently just a big mess of multiple shares across all disks. I've found a manual approach that works to do this, but it's, well manual with some steps involved. Here's what I do that is working, just it's time consuming.

 

1. I first check that I have enough availbile free space on the array for the share that I want to migrate.
   1. Right now I'm ensuring I have >= free space than the share I'm trying to re-distribute, but if automated, this safety measure could be reduced to perhaps the >= the largest file in the source. I use the built in compute space feature on the shares page.
2. I rename the share that I want to re-distribute, example: "TV" gets rename to "TV_old"
   1. On the rename share I exclude all disks that have the free space that are part of my new share that will be created
3. I create a new share called "TV" and I set the appropriate 'split level' I want to obtain
   1. I  exclude all the disks that are part of "TV_Old"
4. Then I list all the disks that have data on the from "TV_Old" and I pick the first disk and do a move of that data to /mnt/user/TV which then allows UnRAID to redistributes the data across that share using Split Level
5. After that's complete, I move to the next drive and repeat until I'm done.

Many ways to make this more efficient using UnBALANCE.

• All of the above could be automated, also as a drive from the previous "old" share is emptied, it could be then removed from each of the shares 'excluded disks' list, making it possible to migrate with less free space available. But it's still kind of an in-elegant solution...
  • Plus to save anyone the effort, yes I'm aware of the warning that moving from a disk share to a user share is not a recommended. This is on my test server.
• Or it could take a much smarter approach which is possible in code, but way too much work to even consider manually.... identify the existing data on disks that meet the requirements of the new Split Level to be applied, and only move the data between disks that does not meet that requirement.
  • This is closest to the 'Gather' function that UnBALANCE does now, just with some more intelligence added to determine this.
  • It keeps the data transfers as disk to disk and avoids the risk of the user shares, if that is still a risk given how I mitigated it by using the 'excluded disk' option

 

 

 

6 hours ago, Lev said:

I'm wanting to migrate to use Split Levels on my User Shares which are presently just a big mess of multiple shares across all disks. I've found a manual approach that works to do this, but it's, well manual with some steps involved. Here's what I do that is working, just it's time consuming.

 

You can use consolid8, like so:

 

I've use it to consolidate both my Movies and TV servers some time ago, can't imagine the time it take to do it manually.

15 hours ago, Lev said:

I'm wanting to migrate to use Split Levels on my User Shares which are presently just a big mess of multiple shares across all disks

Probably I'm not fully getting what you mean, but isn't this the result of writing to user shares ? You eventually have multiple shares in across disks ? Unless you exclude shares from specific disks (don't remember if that's possible) ?

 

On the other hand, if what you want is to consolidate some share to a single disk, either consold8 (as mentioned by johnnie.black) or unBALANCE's GATHER operation will do the trick.

 

Not sure that's what you meant.

 

If it isn't, do you have a concrete example ?

 

I've revisited this plugin time to time and each time, I well "don't get it."

I'll explain the best that I can.  "Scatter"  -  this makes sense and seems like the "option?" that I would like to use.  I've added a hard drive to the array.  It's relatively empty (besides the newest few grabs, all TV eps.).

Upon accessing the GUI, I'm presented with my entire array, however, every disk (including the Cache, is there an option to ignore the cache?  I use my SSD cache only for dockers) has "to" selected and the newest emptiest drive has "from" selected.  This seems backwards from my perhaps flawed logic.  Then the GUI seems to fight against me (in an insistent manner, borderline annoyingly) when attempting to deselect the "to" check-boxes, insisting on keeping my emptiest (new) hard drive with its "from" check-boxed.  Ugg?

Is this a passive plugin or active?  Meaning, do I actually have to initiate any and all moves this thing will do?  Or is it active, constantly running and making adjustments as deemed necessary?  If the latter, then I certainly dont it to move files from my least filled to all the others that are much closer to being filled.

25 pages is a bit to read though, though I'm doing so now, however, in the meantime and in case those questions aren't answered anywhere in the thread, I figured I'd ask.  I appreciate the time and effort put into this, don't misread my seemingly critical questions.  They're not I assure you.

8 minutes ago, grandprix said:

is there an option to ignore the cache?

 

Someone requested to have the cache disk available, so I added it. If you don't want to work it with it, you can safely ignore it.

 

9 minutes ago, grandprix said:

every disk (including the Cache, is there an option to ignore the cache?  I use my SSD cache only for dockers) has "to" selected and the newest emptiest drive has "from" selected

 

That's strange. Every disk has the to field checked ? I have never heard of that. What version of the plugin are you running ?

 

The From disk checked is just a courtesy, the plugin selects the disk with the most free space, assuming that's the one you want to free, but you can always change it as needed.

 

12 minutes ago, grandprix said:

Then the GUI seems to fight against me (in an insistent manner, borderline annoyingly)

 

Not sure exactly what you're doing, but you should select the source disk first, then select/deselect any "to" disk.

You can only have one source disk in SCATTER, so it will deselect all other disks, when you select any disk as source.

Is this what you're referring to ?

 

14 minutes ago, grandprix said:

25 pages is a bit to read though, though I'm doing so now, however, in the meantime and in case those questions aren't answered anywhere in the thread

 

There's a "manual" here (and a quick start for the SCATTER function). I will put a link to it in the first post.

 

16 minutes ago, grandprix said:

I appreciate the time and effort put into this, don't misread my seemingly critical questions.  They're not I assure you.

 

Thanks and no problem, hopefully the plugin can do what you're looking for.