[Plugin] unbalanced

[Unqualified Spectator]

Hey Saxy - you said:

 

 

ie. in the logs it has:

rsync -avRX --partial "/Other/.AppleDB" /mnt/disk3/

 

but it should be:

 

rsync -avRX --partial "Other/.AppleDB" /mnt/disk3/

 

Did you try that rsync without the leading slash? Did that finish yours up?

[trurl]

... I'm just concerned about what Parity might think about that...

All writes to disks in the unRAID parity array update parity at the same time, so parity will be OK.

[jbrodriguez]

Ok, I published 1.7.0

 

It does the following:

 

- Fixes the leading slash issue mentioned by saxy.

- Resets the default rsync flags to the new default (-avRX --partial), if it's currently set to the old one (-avX --partial). If custom flags are set, it will not touch them.

- Applies the correct default flags in the settings ui.

 

Additionally, the full changelog is now hosted in Github.

 

In the first post, I'll keep the last three or so.

 

Sorry for any inconveniences, turns out rsync is somewhat finicky 

[jbrodriguez]

First off - this app is super useful - thanks for this.

....

Here is a summary of the status shown when I tried to reselect disk 1 and continue the MOVE:

 

MOVE: Started: Jun 9, 2016 06:44:00

MOVE: Ended: Jun 9, 2016 06:44:00

MOVE: Elapsed: 129ms

MOVE: Move command (rsync -avX --partial "/TV Shows/STAR TREK THE NEXT GENERATION" /mnt/disk3/) was interrupted: exit status 23

MOVE: These are the commands that were executed:

 

and the Error dialog at the top:

OPERATION FEEDBACK

 

    Move operation was interrupted. Check logs for details.

 

Thanks for the comments Unqualified Spectator !

 

The issue you mention is related to the leading slash, as mentioned by saxy.

 

Try the latest version, it should work fine.

 

Let me know how it goes.

[Unqualified Spectator]

Awesome! I 'll give the app an update and a new shot when I get home.

 

Yes, rsync is NOT for the uninitiated. That's why I didn't relish the idea of doing it manually - this app is nothing short of brilliant. 

 

Would it be possible to add a "log reader" button to the UI?  When I got the error initially, it took me a few minutes to remember to go to the Main menu and right click the folder on the flash device to go find my log (I don't go there very often!) . I think you could just steal the folder icon, code and all from Main and put it on your page? It might help others in the future, especially when things don't go quite as planned.

 

Really, truly great job. 

 

[saxy]

Hey Saxy - you said:

 

 

ie. in the logs it has:

rsync -avRX --partial "/Other/.AppleDB" /mnt/disk3/

 

but it should be:

 

rsync -avRX --partial "Other/.AppleDB" /mnt/disk3/

 

Did you try that rsync without the leading slash? Did that finish yours up?

Yep, fyi finished it off with manual rsync - all successfully...

 

And now after replacing a drive, and updating to 1.70 I'm successfully doing some more unBalancing

 

Sent from my Redmi Note 2 using Tapatalk

 

 

[naturalcarr]

;D

 

Just writing to say 1.7.0 is a success, everything is working perfectly.

[Unqualified Spectator]

Mine is so big it's still going....But it's working AWESOMELY! 

 

I should be complete with all moves by tomorrow morning...then, the reformatting and finally - load those suckers up with new content.

 

MUSH! 1.7.0  MUSH!

 

Great job - you need a "Buy me a Beer" button!

[Conson Droppa]

Can someone shine some kind of definition on an "exit 23" error.

 

I'm on the road at the moment, I just got my results email from unbalance and that was the message.

 

I'll post a log if needed in a bit.

 

 

Sent from my iPhone using Tapatalk

[trurl]

Can someone shine some kind of definition on an "exit 23" error.

 

I'm on the road at the moment, I just got my results email from unbalance and that was the message.

 

I'll post a log if needed in a bit.

 

 

Sent from my iPhone using Tapatalk

rsync exit code 23 - Partial transfer due to error

[saxy]

Can someone shine some kind of definition on an "exit 23" error.

 

I'm on the road at the moment, I just got my results email from unbalance and that was the message.

 

I'll post a log if needed in a bit.

 

 

Sent from my iPhone using Tapatalk

rsync exit code 23 - Partial transfer due to error

I found that I had permissions problems... You could try running the 'new permissions' function in unraid to see if that resolves it for you... Otherwise check the files/folders on source and target, see if anything is owned by root our any other user other than 'nobody'.

 

Sent from my Redmi Note 2

 

 

[Squid]

Since you're relying on rsync, probably wouldn't hurt to have something like this in your code so that it'll print the actual error message instead of just the code

 

function getRsyncReturnValue($returnValue) {
  $returnMessage[0] = "Success";
  $returnMessage[1] = "Syntax or usage error";
  $returnMessage[2] = "Protocol incompatibility";
  $returnMessage[3] = "Errors selecting input/output files, dirs";
  $returnMessage[4] = "Requested action not supported: an attempt was made to manipulate 64-bit files on a platform that cannot support them; or an option was specified that is supported by the client and not by the server.";
  $returnMessage[5] = "Error starting client-server protocol";
  $returnMessage[6] = "Daemon unable to append to log-file";
  $returnMessage[10] = "Error in socket I/O";
  $returnMessage[11] = "Error in file I/O";
  $returnMessage[12] = "Error in rsync protocol data stream";
  $returnMessage[13] = "Errors with program diagnostics";
  $returnMessage[14] = "Error in IPC code";
  $returnMessage[20] = "Received SIGUSR1 or SIGINT";
  $returnMessage[21] = "Some error returned by waitpid()";
  $returnMessage[22] = "Error allocating core memory buffers";
  $returnMessage[23] = "Partial transfer due to error";
  $returnMessage[24] = "Partial transfer due to vanished source files";
  $returnMessage[25] = "The --max-delete limit stopped deletions";
  $returnMessage[30] = "Timeout in data send/receive";
  $returnMessage[35] = "Timeout waiting for daemon connection";
  
  $return = $returnMessage[$returnValue];
  if ( ! $return ) {
    $return = "Unknown Error";
  }
  return $return;
}

[Drewster727]

Just wanted to say thanks for this plugin, it has made things much easier for me.

 

I'm curious if anyone has considered using unbalance to replace the mover operation? IMO, this would be a perfect replacement. The mover just moves data to one drive in a "dumb" way, if you could choose destination disks and min free space per disk (like unbalance), the mover would be WAY better. Just a thought!

 

-Drew

[saxy]

Hi All,

 

FYI I am getting some further status 23 errors now myself... looking into the logs, I'm getting the following error:

rsync: get_xattr_data: lgetxattr(""/mnt/disk1/Other/dev section 3.webm"","user.com.apple.quarantine",67) failed: Input/output error (5)

 

so now I need to figure out why there is an IO error trying to get the xattr data

[saxy]

Hi All,

 

FYI I am getting some further status 23 errors now myself... looking into the logs, I'm getting the following error:

rsync: get_xattr_data: lgetxattr(""/mnt/disk1/Other/dev section 3.webm"","user.com.apple.quarantine",67) failed: Input/output error (5)

 

so now I need to figure out why there is an IO error trying to get the xattr data

 

further to this, the syslog is telling me:

Jun 11 09:48:55 Unraid kernel: REISERFS warning (device md1): jdm-20002 reiserfs_xattr_get: Invalid hash for xattr (user.com.apple.quarantine) associated with [208735 300497 0x0 SD]

 

So... I'm not sure what is going on - is it that there is an IO issue, or that message about 'invalid hash for the xattr' means that it cannot read the attribute detail for some reason... maybe one option is to remove that attribute (as it is a Mac  OSX one only that I presumably dont really care about...)

[jbrodriguez]

Mine is so big it's still going....But it's working AWESOMELY! 

 

I should be complete with all moves by tomorrow morning...then, the reformatting and finally - load those suckers up with new content.

 

MUSH! 1.7.0  MUSH!

 

Great job - you need a "Buy me a Beer" button!

 

Thanks Unqualified Spectator !

 

I think I might go for that "Buy me a Beer" button .. after all, beer helps me code better 

[jbrodriguez]

Since you're relying on rsync, probably wouldn't hurt to have something like this in your code  ...

 

You're right, I'll do it.

[jbrodriguez]

Hi All,

 

FYI I am getting some further status 23 errors now myself... looking into the logs, I'm getting the following error:

rsync: get_xattr_data: lgetxattr(""/mnt/disk1/Other/dev section 3.webm"","user.com.apple.quarantine",67) failed: Input/output error (5)

 

so now I need to figure out why there is an IO error trying to get the xattr data

 

FWIW, I get some of those errors when backing up my mac to a zfs pool, so, in general, something to do with apple's filesystem.

[saxy]

Hi All,

 

FYI I am getting some further status 23 errors now myself... looking into the logs, I'm getting the following error:

rsync: get_xattr_data: lgetxattr(""/mnt/disk1/Other/dev section 3.webm"","user.com.apple.quarantine",67) failed: Input/output error (5)

 

so now I need to figure out why there is an IO error trying to get the xattr data

 

FWIW, I get some of those errors when backing up my mac to a zfs pool, so, in general, something to do with apple's filesystem.

 

 

Agreed.

Whats strange is that I can't quite recreate this issue... If I have some files with this Apple quarantine attribute set on my Mac, when I copy them to the server, get don't have the problem like these files did... So I'm not sure how they got this way.

 

Fyi I created a script to find any files with this extended attribute (with errors) and remove the extended attribute, however I had already fixed those files in the process of testing, so I can't fully test the script.

 

I also created a script just to find files with this attribute that have the error.

 

Both scripts attached if anyone is interested or this helps anyone else... Just need to change the file path in the first part of it to the path you want to search from (I had it set to /mnt/disk1).

 

http://cloud.tapatalk.com/s/57607fc95d023/scripts_apple_attrs.zip

 

 

Sent from my Redmi Note 2

 

 

[gfjardim]

Can someone shine some kind of definition on an "exit 23" error.

 

I'm on the road at the moment, I just got my results email from unbalance and that was the message.

 

I'll post a log if needed in a bit.

 

 

Sent from my iPhone using Tapatalk

rsync exit code 23 - Partial transfer due to error

I found that I had permissions problems... You could try running the 'new permissions' function in unraid to see if that resolves it for you... Otherwise check the files/folders on source and target, see if anything is owned by root our any other user other than 'nobody'.

 

Sent from my Redmi Note 2

 

I had this too.  Maybe, while calculating, the plugin could verify if all chosen files are owned by the chosen user + 06XX permission or are owned by another user and have 0X66 permission.

 

Another thing missing is the ability to stop the current operation.

[jbrodriguez]

...

I had this too.  Maybe, while calculating, the plugin could verify if all chosen files are owned by the chosen user + 06XX permission or are owned by another user and have 0X66 permission.

 

Another thing missing is the ability to stop the current operation.

 

Actually, it's currently logging the ownership+permissions during the calculation phase, but it's not acting upon it.

 

I'll look into implementing a warning if there's a mismatch in ownership or permissions, suggesting to run the 'new permission' function.

 

I also discovered that the plugin always runs as nobody, regardless of the settings 

 

I'll fix this in the next version.

 

With regards to stopping the move operation, I had thought about it.

 

What I don't like too much is that it could leave the folders in an inconsistent state (some files copied to a disk, rather than a whole folder,  duplicate folders, etc.)

 

The user would need to be very aware of this fact.

 

[Sean M.]

Just wanted to follow up on my previous issue but also to say thank you for a great tool!  I wouldn't have been able to format my older drives so easily without out.

 

To avoid any of the permissions issues, I recommend using the Install Fix Common Problems, and there is a replacement script called Docker Safe New Permissions.

 

From there it worked flawlessly.

[Squid]

 

 

To avoid any of the permissions issues, I just ran 'New Permissions' from the 'Tools' menu on all my drives (minus parity & cache) to assign them to the 'nobody' profile again.  From there it worked flawlessly.

Not a good idea to do if you have docker applications.  The New Perms script can mess up the apps' config files due to change the permissions on it.  Install Fix Common Problems, and there is a replacement script called Docker Safe New Permissions in the tools menu which avoids that issue

 

[Sean M.]

 

 

To avoid any of the permissions issues, I just ran 'New Permissions' from the 'Tools' menu on all my drives (minus parity & cache) to assign them to the 'nobody' profile again.  From there it worked flawlessly.

Not a good idea to do if you have docker applications.  The New Perms script can mess up the apps' config files due to change the permissions on it.  Install Fix Common Problems, and there is a replacement script called Docker Safe New Permissions in the tools menu which avoids that issue

 

Oh wow, thanks! Updating my post to recommend using that as well so no one makes the same mistake.

 

It might be worth adding to utilize that Install Common Problems application & permission fix to the OP as I know a few posts related to permissions errors have come up.

[jbrodriguez]

Thanks for the comments Sean M !

 

I've made some progress on the next version, which should include a warning when there are permission issues, during the calculation phase.

 

Based on Squid's comment, the suggestion will be to install Fix Common Problems plugin and run Docker Safe New Permissions.