[Plugin] unbalanced


Recommended Posts

On 1/26/2019 at 12:46 PM, trurl said:

And how do you define best?

 

If you are just trying to get the current split level for files that were written before you applied the split level, you could just move them to cache and let mover take care of it.

 

Since you probably only have enough cache for small batches you would have to move them manually to cache then let mover move them back to the array.

 

Also might be worth noting that high-water allocation will keep things together most of the time without bothering with split level.

Well in unBALANCE I would define the "best" drive to move them to as the one requiring the smallest transfer.

 

Since I only have enough cache for small batches is exactly why I'm asking for this feature, so it will move and allocate folders one at a time, rather than the entire selection. That way I don't have to keep coming back to it, and it can run over-night and be done by the morning, not as I have had to do and be small batches each night.

  • Like 1
Link to comment
1 hour ago, OFark said:

Since I only have enough cache for small batches is exactly why I'm asking for this feature, so it will move and allocate folders one at a time, rather than the entire selection. That way I don't have to keep coming back to it, and it can run over-night and be done by the morning, not as I have had to do and be small batches each night.

You could always add a User Script to check cache free space and invoke the mover if it is over x% - I have one that runs every minute (and 99.9% of the time does nothing).  Then mover is invoked whenever my cache starts to fill up.  Not always fail-safe, as data can often be written to the cache faster than it's copied off, but it might help to you run larger batches.

 

Alternatively, do you have an old larger disk that you could temporarily use as a cache drive?  Doesn't need to be SSD.

 

Mass moving of data in batches over a week or so is something that I end up doing quite a lot, and assumed that a lot of Unraid users did it too.

Link to comment
18 hours ago, Cessquill said:

You could always add a User Script to check cache free space and invoke the mover if it is over x% - I have one that runs every minute (and 99.9% of the time does nothing).  Then mover is invoked whenever my cache starts to fill up.  Not always fail-safe, as data can often be written to the cache faster than it's copied off, but it might help to you run larger batches.

 

Alternatively, do you have an old larger disk that you could temporarily use as a cache drive?  Doesn't need to be SSD.

 

Mass moving of data in batches over a week or so is something that I end up doing quite a lot, and assumed that a lot of Unraid users did it too.

If I'm attempting to unbalance more files than I have space on another drive for that's not going to work, it's not going to let me start, as it won't be able to specify a destination disk.

Link to comment

Not sure I understand ... what you're saying is for instance, gather only season 1 of the series, something like that ?

 

But you can do that today, select just the season folder to be gathered.

 

Wait, you mean to say, that you have to do this MANY times, so it becomes a pain.

 

If that's the case, well there are some alternatives as mentioned, it hasn't come up before, so I don't know if it's something that could be 'frequently' used/'highly' requested.

 

I'll make a mental model of how it would work in any case, for a possible future implementation.

 

Link to comment

Edit..  I am going to go back and manually delete all the duplicate files with binhex krusader on disks2, 1 and 4.  I was able to discern what was incomplete and delete destination files in that case.  Source otherwise. Really lucky and it was all large files that were transferred and that i didn't chose to scatter to all disks - if someone wants to post an easy way could have undone this versus manually please post.  Otherwise disregard my question.

 

 

If i press stop on a long scatter move operation, will unbalance delete the files that have been successfully moved so far? 415GB out of 3.17TB set to transfer.  or do i have to go back and manually delete all the files on the source? (edit in answer to my own question, i have pressed stop now i have 500gb of duplicate files as far as i can tell it didn't delete any source files).  Is there an easy way to go back and delete the source files that i was copying? or how do i go about doing this? 

 

 I am reading that first it copies, then deletes.  Does this normally happen at the end of each completed rsync command versus the end all of the rsync commands queued?

Edited by FrozenGamer
more info
Link to comment

Hi,

 

Not sure why you're stopping, but ... It deletes the source folder after each (copy) rsync command has finished. Note: Deletes the source only if there were no errors during the rsync transfer.

 

Seeing your edit, did it complete any rsync command?

 

You can get more information in the History page.

 

Link to comment
8 hours ago, FrozenGamer said:

No, it was going to be quite a while, i had left it all night while i slept and it was only 1/3 through the first rsync command.  I have pretty slow writes to parity - 10-15MB/s.

Oh ok.

 

Yes, at that point in time, some of the source files were duplicated on the target.

Edited by jbrodriguez
Link to comment
5 hours ago, GHunter said:

Love this plugin, Thanks for creating and maintaining it!!!

Thanks for the kind words !

 

5 hours ago, GHunter said:

Can you show "Elapsed Time" like you do with "Time Remaining"?

Yes, it's easy in terms of complexity :)

 

I have been unable to spare dev cycles for the plugin, but I'd like to push a new release before the end of 1Q.

Link to comment
On 1/31/2019 at 12:55 AM, jbrodriguez said:

Oh ok.

 

Yes, at that point in time, some of the source files were duplicated on the target.

Thanks for taking the time to comment and try and help! - At this point i am running a server with 22 8tb mostly seagate archive drives.  Stuff is really slow when it involves writes.  I need to look into getting better hardware also because i have a bottleneck in my backplate multiplier (if that is the right term).  Need to find something on ebay that will keep my drives cool and has a bit more modern bandwidth for the drives to box.  Parity check takes about 29 hours, a rebuild takes about 7 days.  I am going to post a question once i find the appropriate forum for suggestions on what to buy on ebay.

Link to comment

Not sure if it is something im missing, but im trying to move the contents from disk 2 to disk 6

Disk 2 has got 582GB, and disk 5 has 2.8TB Free

When i run the Plan option, it tells me that a folder cant be moved because the target disk doesn't have enough space

Am i missing something here?

image.thumb.png.219f85b6cc9053084ee359d09edbbb59.png

 

Link to comment

If I'm moving ~12TB of data (in a share folder) from 3 "old disks" to 4 new disks, what's the best practice to efficiently utilize unBalance for this move? I need to get this data moved in order to reallocate two of the "old disks" for dual parity, as they're larger. I plan to tackle this tonight, just want to try and avoid any roadblocks (I'm new to unRaid, learning as I go).

Link to comment
On 2/5/2019 at 11:44 PM, tucansam said:

it seems to hang at "PLANNING: Checking issues..."

it's performing a find command at that point

find "/mnt/disk1/folder/." -exec stat --format "%%A|%%U:%%G|%%F|%%n" {} \;

Try to run this from your command line, let me know if it works.

 

Replace 'disk1/folder' with the disk/folder you're trying to move/copy.

Link to comment
  • jbrodriguez changed the title to [Plugin] unbalanced

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.