[Plugin] unbalanced

[jbrodriguez]

Hi,

 

I'm making a change in v1.4.0, by replacing the diskmv script command with a straight rsync invocation.

 

The main reason for this change is to have better control over the executable being run in the server code (written in Go).

 

In very general terms, diskmv performed the following tasks:

- Invoke find to actually find all the files from a given root folder

- Check if the file is in use (by calling fuser)

- rsync the file to the destination disk

- delete the file from the source disk

 

Given that unBALANCE MUST be run as the sole process reading/writing to the disks, we can let go of the "check file in use" condition.

 

So, the new logic will be:

- rsync a folder from source to destination

- delete folder from source disk

 

I'll be using a slight variation of the rsync command shown in the "Convert from RFS to XFS" thread.

 

rsync -avX --partial "/mnt/disk18/films/bluray/Avatar (2009)" "/mnt/disk5/films/bluray"

 

As per the manual

-a  archive mode; equals -rlptgoD (no -H,-A,-X)

-v  increase verbosity

-X  preserve extended attributes

--partial keep partially transferred files

 

The only difference from the command in the conversion thread is that I've removed the --progress option. This is in order to keep the log a bit saner, but I will eventually revisit this and see if I can parse and display the progress output from rsync.

 

After the rsync has completed, the app will issue a remove command to delete the source folder.

[JonathanM]

Any chance of adding an optional automatic copy first - verify checksum - then remove workflow for paranoid people?

[danioj]

Any chance of adding an optional automatic copy first - verify checksum - then remove workflow for paranoid people?

 

Doesn't Rsync verify copies without the need for an additional switch anyway?

[JonathanM]

Any chance of adding an optional automatic copy first - verify checksum - then remove workflow for paranoid people?

 

Doesn't Rsync verify copies without the need for an additional switch anyway?

It doesn't flush the cache and reread the file from disk.

[jimbobulator]

I tried a dry-run (1.3.7) moving the contents of a old disk (disk2) onto an empty disk (disk3).  I selected all the shares in settings.  I was expecting nothing to change, being a dry-run, but to my surprise this morning, 700GB out about 2.1GB of data was moved from disk2 to disk3 during the dry run.  Shouldn't the dry-run leave everything where it is?  And weirder still, if it didn't do a dry-run, why only move some of the files?

 

Where can I find the log files for unbalance?

[jbrodriguez]

I tried a dry-run (1.3.7) moving the contents of a old disk (disk2) onto an empty disk (disk3).  I selected all the shares in settings.  I was expecting nothing to change, being a dry-run, but to my surprise this morning, 700GB out about 2.1GB of data was moved from disk2 to disk3 during the dry run.  Shouldn't the dry-run leave everything where it is?  And weirder still, if it didn't do a dry-run, why only move some of the files?

 

Where can I find the log files for unbalance?

 

Hi, right ... dry-run shouldn't move anything.

 

You can find the log file at /boot/logs/unbalance.log

[danioj]

I just tried 15 dry-runs. None of them moved anything.

[jbrodriguez]

Any chance of adding an optional automatic copy first - verify checksum - then remove workflow for paranoid people?

 

Doesn't Rsync verify copies without the need for an additional switch anyway?

It doesn't flush the cache and reread the file from disk.

 

So, it would be the second rsync command in the rfs to xfs thread ? rsync -rcvPX ?

[jimbobulator]

I tried a dry-run (1.3.7) moving the contents of a old disk (disk2) onto an empty disk (disk3).  I selected all the shares in settings.  I was expecting nothing to change, being a dry-run, but to my surprise this morning, 700GB out about 2.1GB of data was moved from disk2 to disk3 during the dry run.  Shouldn't the dry-run leave everything where it is?  And weirder still, if it didn't do a dry-run, why only move some of the files?

 

Where can I find the log files for unbalance?

 

Hi, right ... dry-run shouldn't move anything.

 

You can find the log file at /boot/logs/unbalance.log

 

I'll take a look at the logs tonight and see if I can see anything weird.

[BartDG]

...

Otherwise a VERY handy plugin!  Thanks so much for devoting your time to create this!

 

Thanks for your comments BartDG !

 

Not sure why it would do that either, probably the disks were spinning up or rsync was still starting up ?

No idea.  The disks indeed didn't spin up.  I waited for 10 minutes.  Then I even manually spun them all up, then that also didn't do much good.

Then I closed the menus, reopened them and tried again, this time without doing a dry run.  (didn't need it anyway since the (empty) target drive was larger than the source.  That worked without a problem.

 

Anyway, I'm very happy and your little app was a tremendous help!  Thanks again!

[JonathanM]

Any chance of adding an optional automatic copy first - verify checksum - then remove workflow for paranoid people?

 

Doesn't Rsync verify copies without the need for an additional switch anyway?

It doesn't flush the cache and reread the file from disk.

 

So, it would be the second rsync command in the rfs to xfs thread ? rsync -rcvPX ?

You can't do it with a single run, AFAIK. It would be three steps, like I posted. A copy, then check to see if everything is identical, then a delete.

[Dimtar]

Hey all.

 

This plugin doesn't work for me, can anyone tell me where this logs to?

EDIT:

Ok, so i found where it logs. I cleared the logs to start fresh and tried a move operation but it didn't work. No logs were created, this was showed on the web page.

"MOVE: Operation started

MOVE: rsync -avX --partial "/mnt/disk11/media/tv" "/mnt/disk13/media"

MOVE: sending incremental file list

MOVE: Started: Mar 18, 2016 00:33:50

MOVE: Ended: Mar 18, 2016 00:33:50

MOVE: Elapsed: 201ms

MOVE: Move command (rsync -avX --partial "/mnt/disk11/media/tv" "/mnt/disk13/media") was interrupted: exit status 11

MOVE: These are the commands that were executed:

MOVE: Operation Finished"

[jbrodriguez]

Hey all.

 

This plugin doesn't work for me, can anyone tell me where this logs to?

EDIT:

Ok, so i found where it logs. I cleared the logs to start fresh and tried a move operation but it didn't work. No logs were created, this was showed on the web page.

"MOVE: Operation started

MOVE: rsync -avX --partial "/mnt/disk11/media/tv" "/mnt/disk13/media"

MOVE: sending incremental file list

MOVE: Started: Mar 18, 2016 00:33:50

MOVE: Ended: Mar 18, 2016 00:33:50

MOVE: Elapsed: 201ms

MOVE: Move command (rsync -avX --partial "/mnt/disk11/media/tv" "/mnt/disk13/media") was interrupted: exit status 11

MOVE: These are the commands that were executed:

MOVE: Operation Finished"

 

Hi Dimtar,

 

Looking at rsync's error codes, it says:

 

11    Error in file I/O

 

So ... a question ... what's the smart status on disk11 in the main unRAID dashboard ?

 

I would have to believe that the disk has some kind of fault, for rsync to be throwing that error.

 

Please let me know if that's not the case.

[Dimtar]

[quote author=jbrodriguez link=topic=45352.msg456318#msg456318 date=1458338377

 

Hi Dimtar,

 

Looking at rsync's error codes, it says:

 

11    Error in file I/O

 

So ... a question ... what's the smart status on disk11 in the main unRAID dashboard ?

 

I would have to believe that the disk has some kind of fault, for rsync to be throwing that error.

 

Please let me know if that's not the case.

 

Hey there.

 

MOVE: Operation started
MOVE: rsync -avX --partial "/mnt/disk10/backup/Bvckup" "/mnt/disk12/backup"
MOVE: sending incremental file list
MOVE: Started: Mar 19, 2016 11:53:18
MOVE: Ended: Mar 19, 2016 11:53:18
MOVE: Elapsed: 24ms
MOVE: Move command (rsync -avX --partial "/mnt/disk10/backup/Bvckup" "/mnt/disk12/backup") was interrupted: exit status 11
MOVE: These are the commands that were executed:
MOVE: Operation Finished

 

This is with another drive, to my knowledge there is nothing wrong with these drives. The smart status on this drive and drive 11 are green on the basic checks.

[jbrodriguez]

[quote author=jbrodriguez link=topic=45352.msg456318#msg456318 date=1458338377

 

Hi Dimtar,

 

Looking at rsync's error codes, it says:

 

11    Error in file I/O

 

So ... a question ... what's the smart status on disk11 in the main unRAID dashboard ?

 

I would have to believe that the disk has some kind of fault, for rsync to be throwing that error.

 

Please let me know if that's not the case.

 

Hey there.

 

MOVE: Operation started
MOVE: rsync -avX --partial "/mnt/disk10/backup/Bvckup" "/mnt/disk12/backup"
MOVE: sending incremental file list
MOVE: Started: Mar 19, 2016 11:53:18
MOVE: Ended: Mar 19, 2016 11:53:18
MOVE: Elapsed: 24ms
MOVE: Move command (rsync -avX --partial "/mnt/disk10/backup/Bvckup" "/mnt/disk12/backup") was interrupted: exit status 11
MOVE: These are the commands that were executed:
MOVE: Operation Finished

 

This is with another drive, to my knowledge there is nothing wrong with these drives. The smart status on this drive and drive 11 are green on the basic checks.

 

Ok.

 

If you run it from the command line

 

rsync -avPX "/mnt/disk10/backup/Bvckup" "/mnt/disk12/backup"

 

what happens ?

[asdiasx]

Similar problem here... the dry run goes OK, but when moving got same "exit status 11"... 

 

 

[jbrodriguez]

Similar problem here... the dry run goes OK, but when moving got same "exit status 11"... 

 

Are you able to run the command manually (the rsync command that appears in the log), to check if you get the same error ?

 

[asdiasx]

I'll try this and post here the results... Tks!

[asdiasx]

Hi, I didn't try yet as a move process is working on a huge share...

Yes, I could get this to work with another share ( a newer one).

 

At this moment I suspect that there are problems to move shares with .Appledouble folders (hidden). By a quick research these folders are created by OSX  when dealing with no fully compatible fs... I cant remember if the files inside the shares went from a ntfs disk handled by OSX (its possible!).

 

There are other .Applexxx folders that I'm not sure causes the same problem...

 

I'll make a test deleting manually some folders and retrying the move command.

 

As there are a lot of these .apple folders, is there a way to make the command handle them?

 

Tks!

[Morpheus]

I seem to be unable to get to the Home page. It doesn't matter if I click on "Home" or "Settings" it stays on the page shown below. If I hover on the "Home" button, the url is shown as MYIP:6237:/#/ Is that correct? Suggestions?

https://www.dropbox.com/s/rf5xza7vh5xwpkq/unbalance.jpg?dl=0

 

It would seem that I can't figure out the tags either. 

[asdiasx]

An update, I did some commands to erase the .AppleDouble folders, but I think that what really solves the problem was run the permissions tool...

The first try I was using a new disk formated xfs as destination.

Now I am empting one drive at time and formating xfs (they're reiserfs) and migrating data from next disk to be formated.

I thougt that unraid format command should do a permission update but seems that this is not the case...

Anyway, it's working fine after run permission tool.

 

Regards

[jbrodriguez]

An update, I did some commands to erase the .AppleDouble folders, but I think that what really solves the problem was run the permissions tool...

The first try I was using a new disk formated xfs as destination.

Now I am empting one drive at time and formating xfs (they're reiserfs) and migrating data from next disk to be formated.

I thougt that unraid format command should do a permission update but seems that this is not the case...

Anyway, it's working fine after run permission tool.

 

Regards

 

Hi asdiasx, glad to know it's working for you now !

 

That permission stuff seems to be a bit of a pain 

 

 

 

[jbrodriguez]

I seem to be unable to get to the Home page. It doesn't matter if I click on "Home" or "Settings" it stays on the page shown below. If I hover on the "Home" button, the url is shown as MYIP:6237:/#/ Is that correct? Suggestions?

https://www.dropbox.com/s/rf5xza7vh5xwpkq/unbalance.jpg?dl=0

 

It would seem that I can't figure out the tags either. 

 

Hi Morpheus, are you related to the Morpheus from the Matrix movies Just kidding 

 

Try and select one of the shares .. I don't have the code at hand right now, but I check for first run of the app: if it's the first run, the app goes to the settings page.

 

I believe I'm checking if some share has been selected, to decide whether it's the first run or not

 

[Morpheus]

Hi Morpheus, are you related to the Morpheus from the Matrix movies Just kidding 

 

Try and select one of the shares .. I don't have the code at hand right now, but I check for first run of the app: if it's the first run, the app goes to the settings page.

 

I believe I'm checking if some share has been selected, to decide whether it's the first run or not

 

That worked, Thanks for the reply. 

[wgstarks]

Every time I try to use unBalance I get the same result-

The dry run seems fine, but when I try the actual run I get this:

Move command (rsync -avX --partial "/mnt/disk8/Media/XBMC" "/mnt/disk1/Media") was interrupted: exit status 13

No commands are being executed.