January 28, 20197 yr On 1/26/2019 at 12:46 PM, trurl said: And how do you define best? If you are just trying to get the current split level for files that were written before you applied the split level, you could just move them to cache and let mover take care of it. Since you probably only have enough cache for small batches you would have to move them manually to cache then let mover move them back to the array. Also might be worth noting that high-water allocation will keep things together most of the time without bothering with split level. Well in unBALANCE I would define the "best" drive to move them to as the one requiring the smallest transfer. Since I only have enough cache for small batches is exactly why I'm asking for this feature, so it will move and allocate folders one at a time, rather than the entire selection. That way I don't have to keep coming back to it, and it can run over-night and be done by the morning, not as I have had to do and be small batches each night.
January 28, 20197 yr 1 hour ago, OFark said: Since I only have enough cache for small batches is exactly why I'm asking for this feature, so it will move and allocate folders one at a time, rather than the entire selection. That way I don't have to keep coming back to it, and it can run over-night and be done by the morning, not as I have had to do and be small batches each night. You could always add a User Script to check cache free space and invoke the mover if it is over x% - I have one that runs every minute (and 99.9% of the time does nothing). Then mover is invoked whenever my cache starts to fill up. Not always fail-safe, as data can often be written to the cache faster than it's copied off, but it might help to you run larger batches. Alternatively, do you have an old larger disk that you could temporarily use as a cache drive? Doesn't need to be SSD. Mass moving of data in batches over a week or so is something that I end up doing quite a lot, and assumed that a lot of Unraid users did it too.
January 29, 20197 yr 18 hours ago, Cessquill said: You could always add a User Script to check cache free space and invoke the mover if it is over x% - I have one that runs every minute (and 99.9% of the time does nothing). Then mover is invoked whenever my cache starts to fill up. Not always fail-safe, as data can often be written to the cache faster than it's copied off, but it might help to you run larger batches. Alternatively, do you have an old larger disk that you could temporarily use as a cache drive? Doesn't need to be SSD. Mass moving of data in batches over a week or so is something that I end up doing quite a lot, and assumed that a lot of Unraid users did it too. If I'm attempting to unbalance more files than I have space on another drive for that's not going to work, it's not going to let me start, as it won't be able to specify a destination disk.
January 30, 20197 yr Author Not sure I understand ... what you're saying is for instance, gather only season 1 of the series, something like that ? But you can do that today, select just the season folder to be gathered. Wait, you mean to say, that you have to do this MANY times, so it becomes a pain. If that's the case, well there are some alternatives as mentioned, it hasn't come up before, so I don't know if it's something that could be 'frequently' used/'highly' requested. I'll make a mental model of how it would work in any case, for a possible future implementation.
January 30, 20197 yr Edit.. I am going to go back and manually delete all the duplicate files with binhex krusader on disks2, 1 and 4. I was able to discern what was incomplete and delete destination files in that case. Source otherwise. Really lucky and it was all large files that were transferred and that i didn't chose to scatter to all disks - if someone wants to post an easy way could have undone this versus manually please post. Otherwise disregard my question. If i press stop on a long scatter move operation, will unbalance delete the files that have been successfully moved so far? 415GB out of 3.17TB set to transfer. or do i have to go back and manually delete all the files on the source? (edit in answer to my own question, i have pressed stop now i have 500gb of duplicate files as far as i can tell it didn't delete any source files). Is there an easy way to go back and delete the source files that i was copying? or how do i go about doing this? I am reading that first it copies, then deletes. Does this normally happen at the end of each completed rsync command versus the end all of the rsync commands queued? Edited January 30, 20197 yr by FrozenGamer more info
January 30, 20197 yr Author Hi, Not sure why you're stopping, but ... It deletes the source folder after each (copy) rsync command has finished. Note: Deletes the source only if there were no errors during the rsync transfer. Seeing your edit, did it complete any rsync command? You can get more information in the History page.
January 31, 20197 yr No, it was going to be quite a while, i had left it all night while i slept and it was only 1/3 through the first rsync command. I have pretty slow writes to parity - 10-15MB/s.
January 31, 20197 yr Love this plugin, Thanks for creating and maintaining it!!! I have a feature request. Can you show "Elapsed Time" like you do with "Time Remaining"? This would give me a more realistic idea on future move and copy functions. Thanks.
January 31, 20197 yr Author 8 hours ago, FrozenGamer said: No, it was going to be quite a while, i had left it all night while i slept and it was only 1/3 through the first rsync command. I have pretty slow writes to parity - 10-15MB/s. Oh ok. Yes, at that point in time, some of the source files were duplicated on the target. Edited January 31, 20197 yr by jbrodriguez
January 31, 20197 yr Author 5 hours ago, GHunter said: Love this plugin, Thanks for creating and maintaining it!!! Thanks for the kind words ! 5 hours ago, GHunter said: Can you show "Elapsed Time" like you do with "Time Remaining"? Yes, it's easy in terms of complexity I have been unable to spare dev cycles for the plugin, but I'd like to push a new release before the end of 1Q.
February 2, 20197 yr On 1/31/2019 at 12:55 AM, jbrodriguez said: Oh ok. Yes, at that point in time, some of the source files were duplicated on the target. Thanks for taking the time to comment and try and help! - At this point i am running a server with 22 8tb mostly seagate archive drives. Stuff is really slow when it involves writes. I need to look into getting better hardware also because i have a bottleneck in my backplate multiplier (if that is the right term). Need to find something on ebay that will keep my drives cool and has a bit more modern bandwidth for the drives to box. Parity check takes about 29 hours, a rebuild takes about 7 days. I am going to post a question once i find the appropriate forum for suggestions on what to buy on ebay.
February 5, 20197 yr Not sure if it is something im missing, but im trying to move the contents from disk 2 to disk 6 Disk 2 has got 582GB, and disk 5 has 2.8TB Free When i run the Plan option, it tells me that a folder cant be moved because the target disk doesn't have enough space Am i missing something here?
February 5, 20197 yr Author Hi, looking at your screenshot, it's reporting PC Backups/WindowsImageBackup as being 1.16 EB !!! 😮 😮 😮 Do you have any idea why that might be ? The plugin obtains folder size with du, a system level command, so I'm not sure what could be the issue. Let me know.
February 5, 20197 yr @jbrodriguez Not sure why its reading like that. When i look at the share size in the GUI, it reports as 551 GB Ill try a restart, and see if it helps! Cant think of anything else Edited February 5, 20197 yr by bdydrp
February 6, 20197 yr On my primary server, this works great!!! On my secondary, it seems to hang at "PLANNING: Checking issues..." and never seems to finish, at least, the "Move" and "Copy" buttons never become available. Edited February 6, 20197 yr by tucansam
February 6, 20197 yr If I'm moving ~12TB of data (in a share folder) from 3 "old disks" to 4 new disks, what's the best practice to efficiently utilize unBalance for this move? I need to get this data moved in order to reallocate two of the "old disks" for dual parity, as they're larger. I plan to tackle this tonight, just want to try and avoid any roadblocks (I'm new to unRaid, learning as I go).
February 7, 20197 yr Author On 2/5/2019 at 11:44 PM, tucansam said: it seems to hang at "PLANNING: Checking issues..." it's performing a find command at that point find "/mnt/disk1/folder/." -exec stat --format "%%A|%%U:%%G|%%F|%%n" {} \; Try to run this from your command line, let me know if it works. Replace 'disk1/folder' with the disk/folder you're trying to move/copy.
February 7, 20197 yr Author On 2/5/2019 at 5:04 PM, bdydrp said: When i look at the share size in the GUI, it reports as 551 GB Run this from the command line find "/mnt/disk2/PC Backups/." ! -name . -prune -exec du -bs {} + Post back the result
February 7, 20197 yr @jbrodriguez Result from cmd line 1155541323535496270 /mnt/disk2/PC Backups/./WindowsImageBackup 528 /mnt/disk2/PC Backups/./MediaID.bin 866 /mnt/disk2/PC Backups/./OFFICE-PC
February 7, 20197 yr Author 10 hours ago, BBLV said: what's the best practice to efficiently utilize unBalance for this move As for best practices, you should use unbalance exclusively (no other writes to at least the target disk(s)). Turbo Write (unraid settings) should get you a speed bump.
February 7, 20197 yr Author 1 hour ago, bdydrp said: 1155541323535496270 /mnt/disk2/PC Backups/./WindowsImageBackup Right, that's ~1.2 EB ! Can't tell why that's the case, though 🤷🏼♂️
February 7, 20197 yr How can it be reading EB? I dont have THAT much storage! I think i will delete that folder, as i can manually run a backup later.
February 7, 20197 yr So i just tried deleting that folder contents on my windows machine, and windows popup reporting as 1.0EB Im baffled!
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.