jbrodriguez Posted March 3, 2016 Author Share Posted March 3, 2016 Hi, I'm making a change in v1.4.0, by replacing the diskmv script command with a straight rsync invocation. The main reason for this change is to have better control over the executable being run in the server code (written in Go). In very general terms, diskmv performed the following tasks: - Invoke find to actually find all the files from a given root folder - Check if the file is in use (by calling fuser) - rsync the file to the destination disk - delete the file from the source disk Given that unBALANCE MUST be run as the sole process reading/writing to the disks, we can let go of the "check file in use" condition. So, the new logic will be: - rsync a folder from source to destination - delete folder from source disk I'll be using a slight variation of the rsync command shown in the "Convert from RFS to XFS" thread. rsync -avX --partial "/mnt/disk18/films/bluray/Avatar (2009)" "/mnt/disk5/films/bluray" As per the manual -a archive mode; equals -rlptgoD (no -H,-A,-X) -v increase verbosity -X preserve extended attributes --partial keep partially transferred files The only difference from the command in the conversion thread is that I've removed the --progress option. This is in order to keep the log a bit saner, but I will eventually revisit this and see if I can parse and display the progress output from rsync. After the rsync has completed, the app will issue a remove command to delete the source folder. Quote Link to comment
JonathanM Posted March 4, 2016 Share Posted March 4, 2016 Any chance of adding an optional automatic copy first - verify checksum - then remove workflow for paranoid people? Quote Link to comment
danioj Posted March 4, 2016 Share Posted March 4, 2016 Any chance of adding an optional automatic copy first - verify checksum - then remove workflow for paranoid people? Doesn't Rsync verify copies without the need for an additional switch anyway? Quote Link to comment
JonathanM Posted March 4, 2016 Share Posted March 4, 2016 Any chance of adding an optional automatic copy first - verify checksum - then remove workflow for paranoid people? Doesn't Rsync verify copies without the need for an additional switch anyway? It doesn't flush the cache and reread the file from disk. Quote Link to comment
jimbobulator Posted March 4, 2016 Share Posted March 4, 2016 I tried a dry-run (1.3.7) moving the contents of a old disk (disk2) onto an empty disk (disk3). I selected all the shares in settings. I was expecting nothing to change, being a dry-run, but to my surprise this morning, 700GB out about 2.1GB of data was moved from disk2 to disk3 during the dry run. Shouldn't the dry-run leave everything where it is? And weirder still, if it didn't do a dry-run, why only move some of the files? Where can I find the log files for unbalance? Quote Link to comment
jbrodriguez Posted March 4, 2016 Author Share Posted March 4, 2016 I tried a dry-run (1.3.7) moving the contents of a old disk (disk2) onto an empty disk (disk3). I selected all the shares in settings. I was expecting nothing to change, being a dry-run, but to my surprise this morning, 700GB out about 2.1GB of data was moved from disk2 to disk3 during the dry run. Shouldn't the dry-run leave everything where it is? And weirder still, if it didn't do a dry-run, why only move some of the files? Where can I find the log files for unbalance? Hi, right ... dry-run shouldn't move anything. You can find the log file at /boot/logs/unbalance.log Quote Link to comment
danioj Posted March 4, 2016 Share Posted March 4, 2016 I just tried 15 dry-runs. None of them moved anything. Quote Link to comment
jbrodriguez Posted March 4, 2016 Author Share Posted March 4, 2016 Any chance of adding an optional automatic copy first - verify checksum - then remove workflow for paranoid people? Doesn't Rsync verify copies without the need for an additional switch anyway? It doesn't flush the cache and reread the file from disk. So, it would be the second rsync command in the rfs to xfs thread ? rsync -rcvPX ? Quote Link to comment
jimbobulator Posted March 4, 2016 Share Posted March 4, 2016 I tried a dry-run (1.3.7) moving the contents of a old disk (disk2) onto an empty disk (disk3). I selected all the shares in settings. I was expecting nothing to change, being a dry-run, but to my surprise this morning, 700GB out about 2.1GB of data was moved from disk2 to disk3 during the dry run. Shouldn't the dry-run leave everything where it is? And weirder still, if it didn't do a dry-run, why only move some of the files? Where can I find the log files for unbalance? Hi, right ... dry-run shouldn't move anything. You can find the log file at /boot/logs/unbalance.log I'll take a look at the logs tonight and see if I can see anything weird. Quote Link to comment
BartDG Posted March 4, 2016 Share Posted March 4, 2016 ... Otherwise a VERY handy plugin! Thanks so much for devoting your time to create this! Thanks for your comments BartDG ! Not sure why it would do that either, probably the disks were spinning up or rsync was still starting up ? No idea. The disks indeed didn't spin up. I waited for 10 minutes. Then I even manually spun them all up, then that also didn't do much good. Then I closed the menus, reopened them and tried again, this time without doing a dry run. (didn't need it anyway since the (empty) target drive was larger than the source. That worked without a problem. Anyway, I'm very happy and your little app was a tremendous help! Thanks again! Quote Link to comment
JonathanM Posted March 4, 2016 Share Posted March 4, 2016 Any chance of adding an optional automatic copy first - verify checksum - then remove workflow for paranoid people? Doesn't Rsync verify copies without the need for an additional switch anyway? It doesn't flush the cache and reread the file from disk. So, it would be the second rsync command in the rfs to xfs thread ? rsync -rcvPX ? You can't do it with a single run, AFAIK. It would be three steps, like I posted. A copy, then check to see if everything is identical, then a delete. Quote Link to comment
Dimtar Posted March 17, 2016 Share Posted March 17, 2016 Hey all. This plugin doesn't work for me, can anyone tell me where this logs to? EDIT: Ok, so i found where it logs. I cleared the logs to start fresh and tried a move operation but it didn't work. No logs were created, this was showed on the web page. "MOVE: Operation started MOVE: rsync -avX --partial "/mnt/disk11/media/tv" "/mnt/disk13/media" MOVE: sending incremental file list MOVE: Started: Mar 18, 2016 00:33:50 MOVE: Ended: Mar 18, 2016 00:33:50 MOVE: Elapsed: 201ms MOVE: Move command (rsync -avX --partial "/mnt/disk11/media/tv" "/mnt/disk13/media") was interrupted: exit status 11 MOVE: These are the commands that were executed: MOVE: Operation Finished" Quote Link to comment
jbrodriguez Posted March 18, 2016 Author Share Posted March 18, 2016 Hey all. This plugin doesn't work for me, can anyone tell me where this logs to? EDIT: Ok, so i found where it logs. I cleared the logs to start fresh and tried a move operation but it didn't work. No logs were created, this was showed on the web page. "MOVE: Operation started MOVE: rsync -avX --partial "/mnt/disk11/media/tv" "/mnt/disk13/media" MOVE: sending incremental file list MOVE: Started: Mar 18, 2016 00:33:50 MOVE: Ended: Mar 18, 2016 00:33:50 MOVE: Elapsed: 201ms MOVE: Move command (rsync -avX --partial "/mnt/disk11/media/tv" "/mnt/disk13/media") was interrupted: exit status 11 MOVE: These are the commands that were executed: MOVE: Operation Finished" Hi Dimtar, Looking at rsync's error codes, it says: 11 Error in file I/O So ... a question ... what's the smart status on disk11 in the main unRAID dashboard ? I would have to believe that the disk has some kind of fault, for rsync to be throwing that error. Please let me know if that's not the case. Quote Link to comment
Dimtar Posted March 19, 2016 Share Posted March 19, 2016 [quote author=jbrodriguez link=topic=45352.msg456318#msg456318 date=1458338377 Hi Dimtar, Looking at rsync's error codes, it says: 11 Error in file I/O So ... a question ... what's the smart status on disk11 in the main unRAID dashboard ? I would have to believe that the disk has some kind of fault, for rsync to be throwing that error. Please let me know if that's not the case. Hey there. MOVE: Operation started MOVE: rsync -avX --partial "/mnt/disk10/backup/Bvckup" "/mnt/disk12/backup" MOVE: sending incremental file list MOVE: Started: Mar 19, 2016 11:53:18 MOVE: Ended: Mar 19, 2016 11:53:18 MOVE: Elapsed: 24ms MOVE: Move command (rsync -avX --partial "/mnt/disk10/backup/Bvckup" "/mnt/disk12/backup") was interrupted: exit status 11 MOVE: These are the commands that were executed: MOVE: Operation Finished This is with another drive, to my knowledge there is nothing wrong with these drives. The smart status on this drive and drive 11 are green on the basic checks. Quote Link to comment
jbrodriguez Posted March 19, 2016 Author Share Posted March 19, 2016 [quote author=jbrodriguez link=topic=45352.msg456318#msg456318 date=1458338377 Hi Dimtar, Looking at rsync's error codes, it says: 11 Error in file I/O So ... a question ... what's the smart status on disk11 in the main unRAID dashboard ? I would have to believe that the disk has some kind of fault, for rsync to be throwing that error. Please let me know if that's not the case. Hey there. MOVE: Operation started MOVE: rsync -avX --partial "/mnt/disk10/backup/Bvckup" "/mnt/disk12/backup" MOVE: sending incremental file list MOVE: Started: Mar 19, 2016 11:53:18 MOVE: Ended: Mar 19, 2016 11:53:18 MOVE: Elapsed: 24ms MOVE: Move command (rsync -avX --partial "/mnt/disk10/backup/Bvckup" "/mnt/disk12/backup") was interrupted: exit status 11 MOVE: These are the commands that were executed: MOVE: Operation Finished This is with another drive, to my knowledge there is nothing wrong with these drives. The smart status on this drive and drive 11 are green on the basic checks. Ok. If you run it from the command line rsync -avPX "/mnt/disk10/backup/Bvckup" "/mnt/disk12/backup" what happens ? Quote Link to comment
asdiasx Posted March 21, 2016 Share Posted March 21, 2016 Similar problem here... the dry run goes OK, but when moving got same "exit status 11"... Quote Link to comment
jbrodriguez Posted March 21, 2016 Author Share Posted March 21, 2016 Similar problem here... the dry run goes OK, but when moving got same "exit status 11"... Are you able to run the command manually (the rsync command that appears in the log), to check if you get the same error ? Quote Link to comment
asdiasx Posted March 21, 2016 Share Posted March 21, 2016 I'll try this and post here the results... Tks! Quote Link to comment
asdiasx Posted March 22, 2016 Share Posted March 22, 2016 Hi, I didn't try yet as a move process is working on a huge share... Yes, I could get this to work with another share ( a newer one). At this moment I suspect that there are problems to move shares with .Appledouble folders (hidden). By a quick research these folders are created by OSX when dealing with no fully compatible fs... I cant remember if the files inside the shares went from a ntfs disk handled by OSX (its possible!). There are other .Applexxx folders that I'm not sure causes the same problem... I'll make a test deleting manually some folders and retrying the move command. As there are a lot of these .apple folders, is there a way to make the command handle them? Tks! Quote Link to comment
Morpheus Posted March 23, 2016 Share Posted March 23, 2016 I seem to be unable to get to the Home page. It doesn't matter if I click on "Home" or "Settings" it stays on the page shown below. If I hover on the "Home" button, the url is shown as MYIP:6237:/#/ Is that correct? Suggestions? https://www.dropbox.com/s/rf5xza7vh5xwpkq/unbalance.jpg?dl=0 It would seem that I can't figure out the tags either. Quote Link to comment
asdiasx Posted March 24, 2016 Share Posted March 24, 2016 An update, I did some commands to erase the .AppleDouble folders, but I think that what really solves the problem was run the permissions tool... The first try I was using a new disk formated xfs as destination. Now I am empting one drive at time and formating xfs (they're reiserfs) and migrating data from next disk to be formated. I thougt that unraid format command should do a permission update but seems that this is not the case... Anyway, it's working fine after run permission tool. Regards Quote Link to comment
jbrodriguez Posted March 26, 2016 Author Share Posted March 26, 2016 An update, I did some commands to erase the .AppleDouble folders, but I think that what really solves the problem was run the permissions tool... The first try I was using a new disk formated xfs as destination. Now I am empting one drive at time and formating xfs (they're reiserfs) and migrating data from next disk to be formated. I thougt that unraid format command should do a permission update but seems that this is not the case... Anyway, it's working fine after run permission tool. Regards Hi asdiasx, glad to know it's working for you now ! That permission stuff seems to be a bit of a pain Quote Link to comment
jbrodriguez Posted March 26, 2016 Author Share Posted March 26, 2016 I seem to be unable to get to the Home page. It doesn't matter if I click on "Home" or "Settings" it stays on the page shown below. If I hover on the "Home" button, the url is shown as MYIP:6237:/#/ Is that correct? Suggestions? https://www.dropbox.com/s/rf5xza7vh5xwpkq/unbalance.jpg?dl=0 It would seem that I can't figure out the tags either. Hi Morpheus, are you related to the Morpheus from the Matrix movies Just kidding Try and select one of the shares .. I don't have the code at hand right now, but I check for first run of the app: if it's the first run, the app goes to the settings page. I believe I'm checking if some share has been selected, to decide whether it's the first run or not Quote Link to comment
Morpheus Posted March 27, 2016 Share Posted March 27, 2016 Hi Morpheus, are you related to the Morpheus from the Matrix movies Just kidding Try and select one of the shares .. I don't have the code at hand right now, but I check for first run of the app: if it's the first run, the app goes to the settings page. I believe I'm checking if some share has been selected, to decide whether it's the first run or not That worked, Thanks for the reply. Quote Link to comment
wgstarks Posted April 2, 2016 Share Posted April 2, 2016 Every time I try to use unBalance I get the same result- The dry run seems fine, but when I try the actual run I get this: Move command (rsync -avX --partial "/mnt/disk8/Media/XBMC" "/mnt/disk1/Media") was interrupted: exit status 13 No commands are being executed. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.