jbrodriguez Posted July 6, 2017 Author Share Posted July 6, 2017 v3.1.2 is available ! 2017-07-06 - 3.1.2 Before removing a file/folder on the source disk, make sure it exists on the destination disk. This is for the MOVE operation, as an additional check. 1 Quote Link to comment
jbrodriguez Posted July 7, 2017 Author Share Posted July 7, 2017 v3.2.0 is out ! ---------------------- 2017-07-07 - 3.2.0 VALIDATE operation is now implemented !! Available after a COPY operation, it performs a checksum copy, thus validating the integrity of the data. It works by replacing the rsync flags from -avPRX (default flags) to -rcPRX If you set custom rsync flags, "-a" MUST still be the first one. Also note that this operation takes a long time to complete. -------------- I'd like to stress how long this operation will take. I will take a LOT of time I'll now tackle the GATHER operation. I would like to thank all the people that has contributed to the beer/coffee/soft drinks fund I have enjoyed reading the notes on how unBALANCE helped you. Thanks a lot ! 2 Quote Link to comment
jbuszkie Posted July 11, 2017 Share Posted July 11, 2017 Great tool! I tried to do a move yesterday from one data disk to another and it seemed to work (a part from seeming very slow...) However when I looked today, it never deleted the source files. They all got copied, but never deleted... I basically used all the stock settings.. Jim Quote Link to comment
jbrodriguez Posted July 11, 2017 Author Share Posted July 11, 2017 Thanks for the comments ! It should run at approx the same speed as copying data to a parity protected drive. Can you check what the log says (/boot/logs/unbalance.log) ? Send a pm if you prefer. Quote Link to comment
jbuszkie Posted July 11, 2017 Share Posted July 11, 2017 Here is the relevant stuff from the log file.. I: 2017/07/10 20:54:50 core.go:596: _calc:End:srcDisk(/mnt/disk4) I: 2017/07/10 20:55:06 core.go:782: Running transfer operation ... I: 2017/07/10 20:55:06 core.go:814: Command Started: rsync -avPRX "HTPC_Stuff/edrive" "/mnt/disk10/" I: 2017/07/10 23:22:17 core.go:143: Sending config W: 2017/07/11 04:58:25 core.go:886: Command Interrupted: rsync -avPRX "HTPC_Stuff/edrive" "/mnt/disk10/" (exit status 23 : Partial transfer due to error) I: 2017/07/11 04:58:25 core.go:991: unBALANCE - TRANSFER operation INTERRUPTED Started: Jul 10, 2017 20:55:06 Ended: Jul 11, 2017 04:58:25 Elapsed: 8h3m18.906s Command Interrupted: rsync -avPRX "HTPC_Stuff/edrive" "/mnt/disk10/" (exit status 23 : Partial transfer due to error) Transferred 962.2G at ~ 33.98 MB/s And I don't see anything in the syslog around 4:58... What error? hmm... Quote Link to comment
jbrodriguez Posted July 11, 2017 Author Share Posted July 11, 2017 Ok, you're not running the latest unBALANCE version (which is 3.2.0, it would have logged which transfer operation you ran: move/copy). Yeah, rsync's error 23 (Partial transfer due to error) is almost like a catch-all But it's generally associated with permission errors or date-time issues with some of your files/folders. Did you get any permission issues during the calculate phase ? Quote Link to comment
jbrodriguez Posted July 11, 2017 Author Share Posted July 11, 2017 P.S.: Turbo Write really helps with write speeds, at the expense of having all of your disks spun up. Quote Link to comment
jbuszkie Posted July 11, 2017 Share Posted July 11, 2017 7 minutes ago, jbrodriguez said: Ok, you're not running the latest unBALANCE version (which is 3.2.0, it would have logged which transfer operation you ran: move/copy). Yeah, rsync's error 23 (Partial transfer due to error) is almost like a catch-all But it's generally associated with permission errors or date-time issues with some of your files/folders. Did you get any permission issues during the calculate phase ? I don't recall if I had any permission issues. I think I might have. And I just installed unbalance several days ago! I didn't think to check for an update so soon! :-) Quote Link to comment
Fireball3 Posted July 12, 2017 Share Posted July 12, 2017 (edited) 23 hours ago, jbuszkie said: Great tool! I tried to do a move yesterday from one data disk to another and it seemed to work (a part from seeming very slow...) However when I looked today, it never deleted the source files. They all got copied, but never deleted... I basically used all the stock settings.. Jim You need to fix the permission issues (run "newperms path_to_folder") - on the source AND on the target directory! Then run exactly the same move command again that you used yesterday. The sources should get removed then. Edited July 12, 2017 by Fireball3 Quote Link to comment
jbuszkie Posted July 12, 2017 Share Posted July 12, 2017 I have a new disk coming online soon. I'll probably have to move more file around. I'll check the permissions before I run it next time. Thanks for the help guys, Jim Quote Link to comment
kingy444 Posted July 18, 2017 Share Posted July 18, 2017 Just after some direction here, and what im looking for might be the gather function but im not sure what exactly that is planned to provide. Basically i have just moved all my data using unbalance in order to move from reinfs to xfs, now i have one completely empty disk. I would like to distribute the data on the other disks to this one, so that i end up with all drives having relatively the same amount of free space. Whats the besr way to accomplish this? Quote Link to comment
jbrodriguez Posted July 18, 2017 Author Share Posted July 18, 2017 I can tell you what GATHER will do with an example. Let's say you have Breaking Bad's season 1/2 on disk2, season 3/4 on disk3, season 5 on disk4. GATHER will help you consolidate all 5 seasons in a single disk. It will let you choose to consolidate on disk2, disk3 or disk4 (provided there's enough available space) or any other disk that has enough space. What it won't do is fill an empty disk to "rebalance" the space among all available disks. Hope it makes it a bit clearer. Quote Link to comment
kingy444 Posted July 18, 2017 Share Posted July 18, 2017 16 minutes ago, jbrodriguez said: I can tell you what GATHER will do with an example. Let's say you have Breaking Bad's season 1/2 on disk2, season 3/4 on disk3, season 5 on disk4. GATHER will help you consolidate all 5 seasons in a single disk. It will let you choose to consolidate on disk2, disk3 or disk4 (provided there's enough available space) or any other disk that has enough space. What it won't do is fill an empty disk to "rebalance" the space among all available disks. Hope it makes it a bit clearer. Definitely clears that up, obviously a useful feature. Any recommendations on 'rebalancing'... sounds like that might be another enhancement at another time? Any suggestions to speed up manually doing this? Quote Link to comment
jbuszkie Posted July 18, 2017 Share Posted July 18, 2017 54 minutes ago, kingy444 said: Definitely clears that up, obviously a useful feature. Any recommendations on 'rebalancing'... sounds like that might be another enhancement at another time? Any suggestions to speed up manually doing this? Not the best way... but could you move it all to the cache disk and then let the mover "rebalance" it with the share set to the correct setting (I forget which one it is..)? Quote Link to comment
JonathanM Posted July 18, 2017 Share Posted July 18, 2017 56 minutes ago, kingy444 said: Any recommendations on 'rebalancing'... sounds like that might be another enhancement at another time? Depending on your use, rebalancing may be counter productive. As you fill a disk, the speeds get lower and lower. Many people would rather fill a disk to capacity and leave it alone, so new writes and fresh data are written to the newest and fastest disks. Older less used data is left on the slower full disks. The more you erase and move data around on a disk, the more fragmented it gets, and modern filesystems don't really have a good method of defragmenting. They are designed in such a way that fragmentation doesn't have as much of an effect as in the past, but it's still noticeable on a badly thrashed disk. The default fill method is high water for a reason, it's the best compromise between spreading data all over the disks evenly and filling each disk fully before moving to the next one. tl;dr Rebalancing may hurt performance. Quote Link to comment
jbrodriguez Posted July 27, 2017 Author Share Posted July 27, 2017 Since it's coming along pretty well, I figured I might as well post a teaser screenshot for the GATHER op There's still some work to be done, but it's not that far away now ! 1 Quote Link to comment
ptirmal Posted August 7, 2017 Share Posted August 7, 2017 (edited) I've been using unbalance to move data off two disks to convert them to btrfs from rfs. My process was to move the data after calculating (no issues in calculation), and then stop the array and format the old drive. I noticed that it had copied the data not moved it after I did the second drive. The problem is, some of my shares are not showing the data that was moved... They are showing missing folders and files. I checked the share with my important folders and it's not showing all data, in crashplan it shows files are on the disk when you navigate through /mnt/user/share (pretty sure that's what it's saying, not showing as deleted from a backup) but the share isn't showing them. I'm running a parity sync now. Is there something that needs to be done for unraid to recognize the files moved disks? It seems like that should have been happening realtime with unbalance but it looks like the share is looking for data on disk 1 when it was moved to disk 2. All my share information has "all disks" and no excluded disks. Edited August 7, 2017 by ptirmal Quote Link to comment
trurl Posted August 7, 2017 Share Posted August 7, 2017 3 minutes ago, ptirmal said: All my share information has "all disks" and no excluded disks. How about your Global Share Settings? Quote Link to comment
ptirmal Posted August 7, 2017 Share Posted August 7, 2017 (edited) 41 minutes ago, trurl said: How about your Global Share Settings? Same, set to default, nothing is excluded. Just want to reiterate that I'm pretty sure the data has been copied correctly, because the drive usage percentage increased respective to the data I moved (which was copied) prior to formatting (which then went back to where it was before any file moving). Edit: So, looking at this thread: I think this is a samba issue. I'll have to check when I get home to be certain. I've never had to fix permissions before, is there a way I can go about doing that? Edited August 7, 2017 by ptirmal Quote Link to comment
graywolf Posted August 7, 2017 Share Posted August 7, 2017 Going from memory since not near machine. Tools tab. Docker safe permission (think part of fix common problem plugin) Quote Link to comment
graywolf Posted August 7, 2017 Share Posted August 7, 2017 BTW. Looking forward to the Gather function. Been meaning to consolidate some shares to specific drives but have not done it yet due to time constraints. How close are you for it? Quote Link to comment
JonathanM Posted August 7, 2017 Share Posted August 7, 2017 1 hour ago, ptirmal said: I'm running a parity sync now. Why? Quote Link to comment
jbrodriguez Posted August 7, 2017 Author Share Posted August 7, 2017 2 hours ago, ptirmal said: it shows files are on the disk when you navigate through /mnt/user/share What about taking a look at the disk itself ? $ ls -al /mnt/disk2/share I understand disk2 is where data is supposed to be now. You could also check the log (/boot/logs/unbalance.log) or send it over (pm if you want). Quote Link to comment
jbrodriguez Posted August 7, 2017 Author Share Posted August 7, 2017 1 hour ago, graywolf said: BTW. Looking forward to the Gather function. Been meaning to consolidate some shares to specific drives but have not done it yet due to time constraints. How close are you for it? It should be ready sometime next week, if there are no issues in the mean time Quote Link to comment
ptirmal Posted August 7, 2017 Share Posted August 7, 2017 33 minutes ago, jbrodriguez said: What about taking a look at the disk itself ? $ ls -al /mnt/disk2/share I understand disk2 is where data is supposed to be now. You could also check the log (/boot/logs/unbalance.log) or send it over (pm if you want). Thanks, I'll check when I get back. 1 hour ago, jonathanm said: Why? At the time I thought maybe the parity disk wasn't updating during the transfer. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.