Jump to content
jbrodriguez

[Plug-In] unBALANCE

1149 posts in this topic Last Reply

Recommended Posts

20 minutes ago, OFark said:

the best drive possible for that directory

And how do you define best?

 

If you are just trying to get the current split level for files that were written before you applied the split level, you could just move them to cache and let mover take care of it.

 

Since you probably only have enough cache for small batches you would have to move them manually to cache then let mover move them back to the array.

 

Also might be worth noting that high-water allocation will keep things together most of the time without bothering with split level.

Share this post


Link to post
On 1/26/2019 at 12:46 PM, trurl said:

And how do you define best?

 

If you are just trying to get the current split level for files that were written before you applied the split level, you could just move them to cache and let mover take care of it.

 

Since you probably only have enough cache for small batches you would have to move them manually to cache then let mover move them back to the array.

 

Also might be worth noting that high-water allocation will keep things together most of the time without bothering with split level.

Well in unBALANCE I would define the "best" drive to move them to as the one requiring the smallest transfer.

 

Since I only have enough cache for small batches is exactly why I'm asking for this feature, so it will move and allocate folders one at a time, rather than the entire selection. That way I don't have to keep coming back to it, and it can run over-night and be done by the morning, not as I have had to do and be small batches each night.

Share this post


Link to post
1 hour ago, OFark said:

Since I only have enough cache for small batches is exactly why I'm asking for this feature, so it will move and allocate folders one at a time, rather than the entire selection. That way I don't have to keep coming back to it, and it can run over-night and be done by the morning, not as I have had to do and be small batches each night.

You could always add a User Script to check cache free space and invoke the mover if it is over x% - I have one that runs every minute (and 99.9% of the time does nothing).  Then mover is invoked whenever my cache starts to fill up.  Not always fail-safe, as data can often be written to the cache faster than it's copied off, but it might help to you run larger batches.

 

Alternatively, do you have an old larger disk that you could temporarily use as a cache drive?  Doesn't need to be SSD.

 

Mass moving of data in batches over a week or so is something that I end up doing quite a lot, and assumed that a lot of Unraid users did it too.

Share this post


Link to post

ehhhhhh... dont need to use user script for mover, just install CA mover plugin :)

Share this post


Link to post
18 hours ago, Cessquill said:

You could always add a User Script to check cache free space and invoke the mover if it is over x% - I have one that runs every minute (and 99.9% of the time does nothing).  Then mover is invoked whenever my cache starts to fill up.  Not always fail-safe, as data can often be written to the cache faster than it's copied off, but it might help to you run larger batches.

 

Alternatively, do you have an old larger disk that you could temporarily use as a cache drive?  Doesn't need to be SSD.

 

Mass moving of data in batches over a week or so is something that I end up doing quite a lot, and assumed that a lot of Unraid users did it too.

If I'm attempting to unbalance more files than I have space on another drive for that's not going to work, it's not going to let me start, as it won't be able to specify a destination disk.

Share this post


Link to post

Not sure I understand ... what you're saying is for instance, gather only season 1 of the series, something like that ?

 

But you can do that today, select just the season folder to be gathered.

 

Wait, you mean to say, that you have to do this MANY times, so it becomes a pain.

 

If that's the case, well there are some alternatives as mentioned, it hasn't come up before, so I don't know if it's something that could be 'frequently' used/'highly' requested.

 

I'll make a mental model of how it would work in any case, for a possible future implementation.

 

Share this post


Link to post

Edit..  I am going to go back and manually delete all the duplicate files with binhex krusader on disks2, 1 and 4.  I was able to discern what was incomplete and delete destination files in that case.  Source otherwise. Really lucky and it was all large files that were transferred and that i didn't chose to scatter to all disks - if someone wants to post an easy way could have undone this versus manually please post.  Otherwise disregard my question.

 

 

If i press stop on a long scatter move operation, will unbalance delete the files that have been successfully moved so far? 415GB out of 3.17TB set to transfer.  or do i have to go back and manually delete all the files on the source? (edit in answer to my own question, i have pressed stop now i have 500gb of duplicate files as far as i can tell it didn't delete any source files).  Is there an easy way to go back and delete the source files that i was copying? or how do i go about doing this? 

 

 I am reading that first it copies, then deletes.  Does this normally happen at the end of each completed rsync command versus the end all of the rsync commands queued?

Edited by FrozenGamer
more info

Share this post


Link to post

Hi,

 

Not sure why you're stopping, but ... It deletes the source folder after each (copy) rsync command has finished. Note: Deletes the source only if there were no errors during the rsync transfer.

 

Seeing your edit, did it complete any rsync command?

 

You can get more information in the History page.

 

Share this post


Link to post

No, it was going to be quite a while, i had left it all night while i slept and it was only 1/3 through the first rsync command.  I have pretty slow writes to parity - 10-15MB/s.

Share this post


Link to post

Love this plugin, Thanks for creating and maintaining it!!!

 

I have a feature request. Can you show "Elapsed Time" like you do with "Time Remaining"? This would give me a more realistic idea on future move and copy functions. Thanks.

Share this post


Link to post
8 hours ago, FrozenGamer said:

No, it was going to be quite a while, i had left it all night while i slept and it was only 1/3 through the first rsync command.  I have pretty slow writes to parity - 10-15MB/s.

Oh ok.

 

Yes, at that point in time, some of the source files were duplicated on the target.

Edited by jbrodriguez

Share this post


Link to post
5 hours ago, GHunter said:

Love this plugin, Thanks for creating and maintaining it!!!

Thanks for the kind words !

 

5 hours ago, GHunter said:

Can you show "Elapsed Time" like you do with "Time Remaining"?

Yes, it's easy in terms of complexity :)

 

I have been unable to spare dev cycles for the plugin, but I'd like to push a new release before the end of 1Q.

Share this post


Link to post
On 1/31/2019 at 12:55 AM, jbrodriguez said:

Oh ok.

 

Yes, at that point in time, some of the source files were duplicated on the target.

Thanks for taking the time to comment and try and help! - At this point i am running a server with 22 8tb mostly seagate archive drives.  Stuff is really slow when it involves writes.  I need to look into getting better hardware also because i have a bottleneck in my backplate multiplier (if that is the right term).  Need to find something on ebay that will keep my drives cool and has a bit more modern bandwidth for the drives to box.  Parity check takes about 29 hours, a rebuild takes about 7 days.  I am going to post a question once i find the appropriate forum for suggestions on what to buy on ebay.

Share this post


Link to post

Not sure if it is something im missing, but im trying to move the contents from disk 2 to disk 6

Disk 2 has got 582GB, and disk 5 has 2.8TB Free

When i run the Plan option, it tells me that a folder cant be moved because the target disk doesn't have enough space

Am i missing something here?

image.thumb.png.219f85b6cc9053084ee359d09edbbb59.png

 

Share this post


Link to post

Hi, looking at your screenshot, it's reporting

 

PC Backups/WindowsImageBackup as being 1.16 EB !!! 😮 😮 😮

 

Do you have any idea why that might be ?

 

The plugin obtains folder size with du, a system level command, so I'm not sure what could be the issue.

 

Let me know.

Share this post


Link to post

@jbrodriguez

Not sure why its reading like that.

When i look at the share size in the GUI, it reports as 551 GB

Ill try a restart, and see if it helps! Cant think of anything else

Edited by bdydrp

Share this post


Link to post

On my primary server, this works great!!!  On my secondary, it seems to hang at "PLANNING: Checking issues..." and never seems to finish, at least, the "Move" and "Copy" buttons never become available.  

Edited by tucansam

Share this post


Link to post

If I'm moving ~12TB of data (in a share folder) from 3 "old disks" to 4 new disks, what's the best practice to efficiently utilize unBalance for this move? I need to get this data moved in order to reallocate two of the "old disks" for dual parity, as they're larger. I plan to tackle this tonight, just want to try and avoid any roadblocks (I'm new to unRaid, learning as I go).

Share this post


Link to post
On 2/5/2019 at 11:44 PM, tucansam said:

it seems to hang at "PLANNING: Checking issues..."

it's performing a find command at that point

find "/mnt/disk1/folder/." -exec stat --format "%%A|%%U:%%G|%%F|%%n" {} \;

Try to run this from your command line, let me know if it works.

 

Replace 'disk1/folder' with the disk/folder you're trying to move/copy.

Share this post


Link to post
On 2/5/2019 at 5:04 PM, bdydrp said:

When i look at the share size in the GUI, it reports as 551 GB

Run this from the command line

find "/mnt/disk2/PC Backups/." ! -name . -prune -exec du -bs {} +

Post back the result

Share this post


Link to post

@jbrodriguez

Result from cmd line

1155541323535496270     /mnt/disk2/PC Backups/./WindowsImageBackup
528     /mnt/disk2/PC Backups/./MediaID.bin
866     /mnt/disk2/PC Backups/./OFFICE-PC

Share this post


Link to post
10 hours ago, BBLV said:

what's the best practice to efficiently utilize unBalance for this move

As for best practices, you should use unbalance exclusively (no other writes to at least the target disk(s)).

 

Turbo Write (unraid settings) should get you a speed bump.

Share this post


Link to post
1 hour ago, bdydrp said:

1155541323535496270     /mnt/disk2/PC Backups/./WindowsImageBackup

Right, that's ~1.2 EB !

 

Can't tell why that's the case, though 🤷🏼‍♂️

 

Share this post


Link to post

How can it be reading EB?

I dont have THAT much storage!

 

I think i will delete that folder, as i can manually run a backup later.

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now