[Plugin] unbalanced

[GoChris]

Just trying out this plugin (tired the docker a while back). Working good now.

 

However when I tried it I originally had the "exit status 11". This was moving files to a brand new empty drive I had just added to the array.

 

I fixed it by manually creating a folder on the new drive via telnet/ssh. It took several seconds to create the new folder, which was one of the folders that will be moved from the old drive. Anyway, after I created the folder, I then tried the Move command again, and it's going just fine.

[GoChris]

Just trying out this plugin (tired the docker a while back). Working good now.

 

However when I tried it I originally had the "exit status 11". This was moving files to a brand new empty drive I had just added to the array.

 

I fixed it by manually creating a folder on the new drive via telnet/ssh. It took several seconds to create the new folder, which was one of the folders that will be moved from the old drive. Anyway, after I created the folder, I then tried the Move command again, and it's going just fine.

 

Actually, I need to put a caveat on this. It only worked to move the files for the folder I created. Do you have to create the top level folders onto each destination drive? I'd prefer it to move everything from the source disk to the target disk, regardless of what folders may or may not exist on target. In my case as stated, the target was a new empty drive. I'll now go create the rest of the top level folders and I expect the rest of the source disk files to move. Not ideal though.

 

Edit: Awesome plugin by the way, very polished, works and looks great. Just the one small issue with it not creating the top level folders.

[jbrodriguez]

Every time I try to use unBalance I get the same result-

The dry run seems fine, but when I try the actual run I get this:

Move command (rsync -avX --partial "/mnt/disk8/Media/XBMC" "/mnt/disk1/Media") was interrupted: exit status 13

No commands are being executed.

 

Hi wgstarks, as per rsync's manual

 

13     Errors with program diagnostics

 

So ... I'm not sure what that means

 

I think I will increase rsync's verbosity with an upcoming release (maybe make it optional), for these cases.

 

Can't tell for sure why it's not working for you right now.

[jbrodriguez]

Just trying out this plugin (tired the docker a while back). Working good now.

 

However when I tried it I originally had the "exit status 11". This was moving files to a brand new empty drive I had just added to the array.

 

I fixed it by manually creating a folder on the new drive via telnet/ssh. It took several seconds to create the new folder, which was one of the folders that will be moved from the old drive. Anyway, after I created the folder, I then tried the Move command again, and it's going just fine.

 

Actually, I need to put a caveat on this. It only worked to move the files for the folder I created. Do you have to create the top level folders onto each destination drive? I'd prefer it to move everything from the source disk to the target disk, regardless of what folders may or may not exist on target. In my case as stated, the target was a new empty drive. I'll now go create the rest of the top level folders and I expect the rest of the source disk files to move. Not ideal though.

 

Edit: Awesome plugin by the way, very polished, works and looks great. Just the one small issue with it not creating the top level folders.

 

Thanks for the comments GoChris !

 

rsync _should_ recreate the structure on the destination disk.

 

The folders you created, were the top-level shares from your source disk ?

[wgstarks]

Every time I try to use unBalance I get the same result-

The dry run seems fine, but when I try the actual run I get this:

Move command (rsync -avX --partial "/mnt/disk8/Media/XBMC" "/mnt/disk1/Media") was interrupted: exit status 13

No commands are being executed.

 

Hi wgstarks, as per rsync's manual

 

13     Errors with program diagnostics

 

So ... I'm not sure what that means

 

I think I will increase rsync's verbosity with an upcoming release (maybe make it optional), for these cases.

 

Can't tell for sure why it's not working for you right now.

I found one folder on the bad disk, that I was trying to empty, that has corrupted permissions. I can't access it so I'm guess that could be the reason for this.

[GoChris]

Just trying out this plugin (tired the docker a while back). Working good now.

 

However when I tried it I originally had the "exit status 11". This was moving files to a brand new empty drive I had just added to the array.

 

I fixed it by manually creating a folder on the new drive via telnet/ssh. It took several seconds to create the new folder, which was one of the folders that will be moved from the old drive. Anyway, after I created the folder, I then tried the Move command again, and it's going just fine.

 

Yes that is correct, the same top level folders that I selected on the settings page to include in the calculate/move.

Actually, I need to put a caveat on this. It only worked to move the files for the folder I created. Do you have to create the top level folders onto each destination drive? I'd prefer it to move everything from the source disk to the target disk, regardless of what folders may or may not exist on target. In my case as stated, the target was a new empty drive. I'll now go create the rest of the top level folders and I expect the rest of the source disk files to move. Not ideal though.

 

Edit: Awesome plugin by the way, very polished, works and looks great. Just the one small issue with it not creating the top level folders.

 

Thanks for the comments GoChris !

 

rsync _should_ recreate the structure on the destination disk.

 

The folders you created, were the top-level shares from your source disk ?

 

Weird, looks like my reply got lost, I'll edit it in here. Yes it was the top level share folders, the same folders I selected to move from the settings page.

[switchman]

Spaces in the directory structure can cause issues if you have not accounted for them in your code.  What is the path structure it want build?

[jbrodriguez]

... Weird, looks like my reply got lost, I'll edit it in here. Yes it was the top level share folders, the same folders I selected to move from the settings page.

 

Right, I think I saw a similar issue in the past.

 

@switchman, paths sent to rsync are quoted, so spaces shouldn't be an issue.

 

I'll dig a bit deeper to try to figure it out.

[switchman]

@jbrodriguez,

 

Ive seen in the past, you can quote the end directory, but when you nest it deeper, it would not create the path.  For example /Dir1/Dir2/"Dir 3" would create the entire path, but /Dir1/"Dir2/Dir 3" would not.

 

If /Dir1/Dir2 already existed, it would work.

 

Not sure if it was just the command string I was using or not.

[GoChris]

@jbrodriguez,

 

Ive seen in the past, you can quote the end directory, but when you nest it deeper, it would not create the path.  For example /Dir1/Dir2/"Dir 3" would create the entire path, but /Dir1/"Dir2/Dir 3" would not.

 

If /Dir1/Dir2 already existed, it would work.

 

Not sure if it was just the command string I was using or not.

 

I am finding this is causing an issue also. I'm moving another disk and after it's done there are still some folders and files left over. I'll run it again and it will show the command with an exit status 11. If I paste that command into the terminal it will run.

 

It fits the pattern of a sub folder with a space in it.

 

Edit: Only once in a while, most of my folders have spaces in them.

[Zonediver]

I am using this plugin (v1.4.0) since today and i am a little bit confused.

This plugin moves files always to only one disk. Is this normal?

The Discription says, it moves as long as there's enough space available.

But it stops always after one disk reach the limit - in my case 200GB - and it dont continue on the next selected disk.

Can someone explain this 

 

The Dev says in Post 1:

unBALANCE is an unRAID plugin that frees space from one of the disks in your array, by moving user shares/folders to other disks in the array, as long as there is space available - but it dosnt

 

EDIT: Got this message today during movement:

MOVE: Move command (rsync -avX --partial "/mnt/disk5/TV-Serien/Homeland" "/mnt/disk7/TV-Serien") was interrupted: exit status 13

 

It seems, there is an error 

The array has free space of 5455GB

 

EDIT 2: After copying a File from Disk 5 to Disk 10, the Plugin didnt remove the File from Disk 5 - lol???

I started the "next" copy process (because of exit status 13) and this Plugin copies now "the same file again" to Disk 4 - iiihihi 

It seems there is something wrong with the "space recognition" because the Name of the non deleted Folder on disk 5 was "007 - James Bond"

 

EDIT 3: After this "strange" operation, the plugin stopt working completely and gives the message "exit status 13" - no more copies anymore - lol 

[redbeard]

I too get the same error. I think my issue is related to the extended attributes of the files.

 

There are couple other threads abt extended attributes causing trouble out there. Example below.

http://lime-technology.com/forum/index.php?topic=28484.0

 

I was kind of wondering why not do the "--remove-source-files" flag for rsync when moving them over.

 

when I run

rsync -avX --partial --dry-run "/mnt/disk8/" "/mnt/disk9/"  from telnet

 

I get lots of errors

 

However when I run

rsync -av --partial --dry-run "/mnt/disk8/" "/mnt/disk9/"

 

No errors.

 

I'm thinking about just manually doing this one, with

rsync -rhut --progress --stats --remove-source-files "/mnt/disk8/" "/mnt/disk9/" 

 

any reason I shouldn't just use that command?

 

[Zonediver]

I decided to do this by hand now over:

 

rsync -avPX  /mnt/disk10/ /mnt/disk11/

 

After all is copied, i remove the files from the "old reiserfs disk" by hand

Works perfect 

It seems, the Dev is ooo because no reply since days...

[JonathanM]

I decided to do this by hand now over:

 

rsync -avPX  /mnt/disk10/ /mnt/disk11/

 

After all is copied, i remove the files from the "old reiserfs disk" by hand

Why take the time to remove the files on a drive you are about to format?

[Zonediver]

Its just a "secure" step to keep the parity clean. I saw, that the parity disk writes a lot during deleting the whole content of the reiserfs-disk.

I am not sure, if it effects the parity when i reformat the disk directly  and i dont want to try it 

I dont want to correct the parity after each disk-copy.

[itimpi]

Its just a "secure" step to keep the parity clean. I saw, that the parity disk writes a lot during deleting the whole content of the reiserfs-disk.

I am not sure, if it effects the parity when i reformat the disk directly  and i dont want to try it 

I dont want to correct the parity after each disk-copy.

as long as the array is started (or is in Maintenance mode) then all operation update the parity in real time to keep it consistent with the disks.  This is the same whether you are doing file deletes or a disk format.

[Zonediver]

But you cant format the disk during the array is online - so you must stop it - and then? What happens with the parity?

[JonathanM]

But you cant format the disk during the array is online - so you must stop it - and then? What happens with the parity?

You misunderstand. You must stop the array to change the desired format type of a specific disk, but you CAN NOT format the drive until you start the array. The array MUST be started to show the button that allows you to format the disk.

[Zonediver]

Hi jonathanm,

 

i see - i tried it and yes, there is no difference for the parity between deleting the files by hand or reformat the drive directly

Thanks for this correction - safes a lot of time 

[jbrodriguez]

...

This plugin moves files always to only one disk. Is this normal?

...

 

It should spread the folders over various disks, if space is available.

 

If you can send me the logs (pm if you want), I'll take a look.

[jbrodriguez]

I too get the same error. I think my issue is related to the extended attributes of the files.

 

There are couple other threads abt extended attributes causing trouble out there. Example below.

http://lime-technology.com/forum/index.php?topic=28484.0

 

I was kind of wondering why not do the "--remove-source-files" flag for rsync when moving them over.

 

when I run

rsync -avX --partial --dry-run "/mnt/disk8/" "/mnt/disk9/"  from telnet

 

I get lots of errors

 

However when I run

rsync -av --partial --dry-run "/mnt/disk8/" "/mnt/disk9/"

 

No errors.

 

I'm thinking about just manually doing this one, with

rsync -rhut --progress --stats --remove-source-files "/mnt/disk8/" "/mnt/disk9/" 

 

any reason I shouldn't just use that command?

 

Thanks redbeard, that's interesting. I'll check this in more detail.

 

I will increase verbosity level in the next release, hoping it provides additional details in the logs.

[Zonediver]

...

This plugin moves files always to only one disk. Is this normal?

...

 

It should spread the folders over various disks, if space is available.

 

If you can send me the logs (pm if you want), I'll take a look.

 

Hi jbrodriguez,

 

here is the requested Log 

unraid-diagnostics-20160417-1804.zip

[jbrodriguez]

here is the requested Log 

 

Actually, unbalance logs are separate from the diagnostics provided by unRAID.

 

They're located at /boot/logs/unbalance.log

[redbeard]

For me, the primary issue was caused by the extended attributes on OS X, specifically by those put in by the program 'Speed Download', which is no longer made (http://www.yazsoft.com/ - states out of business).

 

But I did think that rsync was patched to handle OS X's extended attributes.

[Zonediver]

here is the requested Log 

 

Actually, unbalance logs are separate from the diagnostics provided by unRAID.

 

They're located at /boot/logs/unbalance.log

 

Sorry mate - here is the correct one 

unbalance.zip