OFark Posted January 28, 2019 Share Posted January 28, 2019 On 1/26/2019 at 12:46 PM, trurl said: And how do you define best? If you are just trying to get the current split level for files that were written before you applied the split level, you could just move them to cache and let mover take care of it. Since you probably only have enough cache for small batches you would have to move them manually to cache then let mover move them back to the array. Also might be worth noting that high-water allocation will keep things together most of the time without bothering with split level. Well in unBALANCE I would define the "best" drive to move them to as the one requiring the smallest transfer. Since I only have enough cache for small batches is exactly why I'm asking for this feature, so it will move and allocate folders one at a time, rather than the entire selection. That way I don't have to keep coming back to it, and it can run over-night and be done by the morning, not as I have had to do and be small batches each night. 1 Quote Link to comment
Cessquill Posted January 28, 2019 Share Posted January 28, 2019 1 hour ago, OFark said: Since I only have enough cache for small batches is exactly why I'm asking for this feature, so it will move and allocate folders one at a time, rather than the entire selection. That way I don't have to keep coming back to it, and it can run over-night and be done by the morning, not as I have had to do and be small batches each night. You could always add a User Script to check cache free space and invoke the mover if it is over x% - I have one that runs every minute (and 99.9% of the time does nothing). Then mover is invoked whenever my cache starts to fill up. Not always fail-safe, as data can often be written to the cache faster than it's copied off, but it might help to you run larger batches. Alternatively, do you have an old larger disk that you could temporarily use as a cache drive? Doesn't need to be SSD. Mass moving of data in batches over a week or so is something that I end up doing quite a lot, and assumed that a lot of Unraid users did it too. Quote Link to comment
NewDisplayName Posted January 28, 2019 Share Posted January 28, 2019 ehhhhhh... dont need to use user script for mover, just install CA mover plugin Quote Link to comment
OFark Posted January 29, 2019 Share Posted January 29, 2019 18 hours ago, Cessquill said: You could always add a User Script to check cache free space and invoke the mover if it is over x% - I have one that runs every minute (and 99.9% of the time does nothing). Then mover is invoked whenever my cache starts to fill up. Not always fail-safe, as data can often be written to the cache faster than it's copied off, but it might help to you run larger batches. Alternatively, do you have an old larger disk that you could temporarily use as a cache drive? Doesn't need to be SSD. Mass moving of data in batches over a week or so is something that I end up doing quite a lot, and assumed that a lot of Unraid users did it too. If I'm attempting to unbalance more files than I have space on another drive for that's not going to work, it's not going to let me start, as it won't be able to specify a destination disk. Quote Link to comment
jbrodriguez Posted January 30, 2019 Author Share Posted January 30, 2019 Not sure I understand ... what you're saying is for instance, gather only season 1 of the series, something like that ? But you can do that today, select just the season folder to be gathered. Wait, you mean to say, that you have to do this MANY times, so it becomes a pain. If that's the case, well there are some alternatives as mentioned, it hasn't come up before, so I don't know if it's something that could be 'frequently' used/'highly' requested. I'll make a mental model of how it would work in any case, for a possible future implementation. Quote Link to comment
FrozenGamer Posted January 30, 2019 Share Posted January 30, 2019 (edited) Edit.. I am going to go back and manually delete all the duplicate files with binhex krusader on disks2, 1 and 4. I was able to discern what was incomplete and delete destination files in that case. Source otherwise. Really lucky and it was all large files that were transferred and that i didn't chose to scatter to all disks - if someone wants to post an easy way could have undone this versus manually please post. Otherwise disregard my question. If i press stop on a long scatter move operation, will unbalance delete the files that have been successfully moved so far? 415GB out of 3.17TB set to transfer. or do i have to go back and manually delete all the files on the source? (edit in answer to my own question, i have pressed stop now i have 500gb of duplicate files as far as i can tell it didn't delete any source files). Is there an easy way to go back and delete the source files that i was copying? or how do i go about doing this? I am reading that first it copies, then deletes. Does this normally happen at the end of each completed rsync command versus the end all of the rsync commands queued? Edited January 30, 2019 by FrozenGamer more info Quote Link to comment
jbrodriguez Posted January 30, 2019 Author Share Posted January 30, 2019 Hi, Not sure why you're stopping, but ... It deletes the source folder after each (copy) rsync command has finished. Note: Deletes the source only if there were no errors during the rsync transfer. Seeing your edit, did it complete any rsync command? You can get more information in the History page. Quote Link to comment
FrozenGamer Posted January 31, 2019 Share Posted January 31, 2019 No, it was going to be quite a while, i had left it all night while i slept and it was only 1/3 through the first rsync command. I have pretty slow writes to parity - 10-15MB/s. Quote Link to comment
GHunter Posted January 31, 2019 Share Posted January 31, 2019 Love this plugin, Thanks for creating and maintaining it!!! I have a feature request. Can you show "Elapsed Time" like you do with "Time Remaining"? This would give me a more realistic idea on future move and copy functions. Thanks. Quote Link to comment
jbrodriguez Posted January 31, 2019 Author Share Posted January 31, 2019 (edited) 8 hours ago, FrozenGamer said: No, it was going to be quite a while, i had left it all night while i slept and it was only 1/3 through the first rsync command. I have pretty slow writes to parity - 10-15MB/s. Oh ok. Yes, at that point in time, some of the source files were duplicated on the target. Edited January 31, 2019 by jbrodriguez Quote Link to comment
jbrodriguez Posted January 31, 2019 Author Share Posted January 31, 2019 5 hours ago, GHunter said: Love this plugin, Thanks for creating and maintaining it!!! Thanks for the kind words ! 5 hours ago, GHunter said: Can you show "Elapsed Time" like you do with "Time Remaining"? Yes, it's easy in terms of complexity I have been unable to spare dev cycles for the plugin, but I'd like to push a new release before the end of 1Q. Quote Link to comment
FrozenGamer Posted February 2, 2019 Share Posted February 2, 2019 On 1/31/2019 at 12:55 AM, jbrodriguez said: Oh ok. Yes, at that point in time, some of the source files were duplicated on the target. Thanks for taking the time to comment and try and help! - At this point i am running a server with 22 8tb mostly seagate archive drives. Stuff is really slow when it involves writes. I need to look into getting better hardware also because i have a bottleneck in my backplate multiplier (if that is the right term). Need to find something on ebay that will keep my drives cool and has a bit more modern bandwidth for the drives to box. Parity check takes about 29 hours, a rebuild takes about 7 days. I am going to post a question once i find the appropriate forum for suggestions on what to buy on ebay. Quote Link to comment
bdydrp Posted February 5, 2019 Share Posted February 5, 2019 Not sure if it is something im missing, but im trying to move the contents from disk 2 to disk 6 Disk 2 has got 582GB, and disk 5 has 2.8TB Free When i run the Plan option, it tells me that a folder cant be moved because the target disk doesn't have enough space Am i missing something here? Quote Link to comment
jbrodriguez Posted February 5, 2019 Author Share Posted February 5, 2019 Hi, looking at your screenshot, it's reporting PC Backups/WindowsImageBackup as being 1.16 EB !!! 😮 😮 😮 Do you have any idea why that might be ? The plugin obtains folder size with du, a system level command, so I'm not sure what could be the issue. Let me know. Quote Link to comment
bdydrp Posted February 5, 2019 Share Posted February 5, 2019 (edited) @jbrodriguez Not sure why its reading like that. When i look at the share size in the GUI, it reports as 551 GB Ill try a restart, and see if it helps! Cant think of anything else Edited February 5, 2019 by bdydrp Quote Link to comment
tucansam Posted February 6, 2019 Share Posted February 6, 2019 (edited) On my primary server, this works great!!! On my secondary, it seems to hang at "PLANNING: Checking issues..." and never seems to finish, at least, the "Move" and "Copy" buttons never become available. Edited February 6, 2019 by tucansam Quote Link to comment
BBLV Posted February 6, 2019 Share Posted February 6, 2019 If I'm moving ~12TB of data (in a share folder) from 3 "old disks" to 4 new disks, what's the best practice to efficiently utilize unBalance for this move? I need to get this data moved in order to reallocate two of the "old disks" for dual parity, as they're larger. I plan to tackle this tonight, just want to try and avoid any roadblocks (I'm new to unRaid, learning as I go). Quote Link to comment
jbrodriguez Posted February 7, 2019 Author Share Posted February 7, 2019 On 2/5/2019 at 11:44 PM, tucansam said: it seems to hang at "PLANNING: Checking issues..." it's performing a find command at that point find "/mnt/disk1/folder/." -exec stat --format "%%A|%%U:%%G|%%F|%%n" {} \; Try to run this from your command line, let me know if it works. Replace 'disk1/folder' with the disk/folder you're trying to move/copy. Quote Link to comment
jbrodriguez Posted February 7, 2019 Author Share Posted February 7, 2019 On 2/5/2019 at 5:04 PM, bdydrp said: When i look at the share size in the GUI, it reports as 551 GB Run this from the command line find "/mnt/disk2/PC Backups/." ! -name . -prune -exec du -bs {} + Post back the result Quote Link to comment
bdydrp Posted February 7, 2019 Share Posted February 7, 2019 @jbrodriguez Result from cmd line 1155541323535496270 /mnt/disk2/PC Backups/./WindowsImageBackup 528 /mnt/disk2/PC Backups/./MediaID.bin 866 /mnt/disk2/PC Backups/./OFFICE-PC Quote Link to comment
jbrodriguez Posted February 7, 2019 Author Share Posted February 7, 2019 10 hours ago, BBLV said: what's the best practice to efficiently utilize unBalance for this move As for best practices, you should use unbalance exclusively (no other writes to at least the target disk(s)). Turbo Write (unraid settings) should get you a speed bump. Quote Link to comment
jbrodriguez Posted February 7, 2019 Author Share Posted February 7, 2019 1 hour ago, bdydrp said: 1155541323535496270 /mnt/disk2/PC Backups/./WindowsImageBackup Right, that's ~1.2 EB ! Can't tell why that's the case, though 🤷🏼♂️ Quote Link to comment
NewDisplayName Posted February 7, 2019 Share Posted February 7, 2019 What, etabyte? WTF Quote Link to comment
bdydrp Posted February 7, 2019 Share Posted February 7, 2019 How can it be reading EB? I dont have THAT much storage! I think i will delete that folder, as i can manually run a backup later. Quote Link to comment
bdydrp Posted February 7, 2019 Share Posted February 7, 2019 So i just tried deleting that folder contents on my windows machine, and windows popup reporting as 1.0EB Im baffled! Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.