[Plugin] unbalanced


Recommended Posts

Hi all,

 

I am an unRAID n00b, but am very much enjoying tinkering, so forgive me if this is a very stupid question, as I fear it might be!


I'm trying to remove an SSD and shrink my array in order to use it as a dedicated drive for a gaming VM, using Unassigned Devices. 

My plan was to first move all the data from Disk-1, using unBALANCED, then I believe the only way to really reduce the array is via New Config, or by just whipping out the drive then doing a parity rebuild on whats left. (This might all be wrong, and I'm sure you will tell me! 😀)

 

The main issue with my plan so far, is that I cannot get unBALANCED to let me use it. I don't believe one can disable Mover from what I've read, so I changed it to run monthly so it doesn't interfere with unBALANCED. I then disabled Dockers from within the unRAID settings, I shut down all the VMs I rebooted the server, but I cannot get past this error, which seems to block unBALANCED from being able to function...

 

Quote

unBALANCE needs exclusive access to disks, so disable mover and/or any dockers that write to disks, before running it.

 

Any pointers would be greatly appreciated. 

 

P.S.: I suppose I could just remove the disk and rebuild from parity without using unBALANCED at all, but I would like to get it working if possible for future use. 

 


Thanks in advance!

  • Like 1
Link to comment
Quote

unBALANCE needs exclusive access to disks, so disable mover and/or any dockers that write to disks, before running it.

I am having the exact same issue. It's incredibly frustrating. I've used unBalance on this array in the past (several months ago) without issue.

 

I am attempting to move data from one 8TB drive to two new 6TB drives so I can remove the 8TB from the array.

 

I've disabled both VMs and Dockers from the settings menu.

I've also disabled the cache drive for the folder I am attempting to work with and set Mover to a monthly schedule.

I am able to select the source and target for the scatter and select the 'plan' option, but cannot select the copy or move options.

 

 

(EDIT) It finally worked with these changes - I just had to run through the planning stage first. The banner at the top never went away, but it's finally scattering the files.

Edited by bshattuck
  • Like 1
Link to comment
On 5/3/2020 at 9:34 AM, jbrodriguez said:

Hi, not sure what you mean by that.

 

Can you elaborate ?

I'm attempting to change to a port that I don't already have forwarded, so the world can't mess with my server.

N8NDlNuPqQ.thumb.png.7a4661320a25ba6d54a49fa93e3fc1c1.png

 

Although after hitting apply, it defaults to the port below

SZ5sO64On7.thumb.png.d3741dab896ae8884720633f1d3642af.png

Link to comment

Is there a way to send data from my current drives in my array (22 disks) to my 1 new drive?  I would like to pull some data off the current ones I have are almost full and I keep getting warnings saying they're filling up.

 

Adding a new 14tb now so it would be nice to syphon the data from my array into it.  I don't see a way to select multiple sources though in unbalance.

Link to comment
  • 2 weeks later...

Reporting 2.27TB be moved from a 2TB drive with 162GB free???

 

I have one ReiserFS disk left in my system, that I would like to empty and convert to XFS. 

However, after planning, the Plan says it will move 2.27TB from a 2TB drive with only 1.84TB data.

Also - in the GUI it suggests 2.27TB, in the log 2.11TB

 

Does this mean, that some files on that disk are corrupt in some way (I would assume yes)?

 

Any suggestions for how to proceed?

 

znsP29T.png

NNTJ3YV.png

 

Thanks in advance for input.

Link to comment

Hi,

 

I remember a similar issue and it turned out that the filesystem was reporting a file with a size in the exabyte range, so it's certainly possible that something like this could be the issue in this case.

 

Can you check the logs for any folder/file that has an unusual size ?

 

The full log is located at /boot/logs/unbalance.log

Edited by jbrodriguez
Link to comment

Hey, I'd like to publicly thank all of the supporters who sent some hard-earned cash MY way.

 

Any amount you sent, you can be sure that I have spent on things I enjoy, a fine bottle of wine, a fancy cup of coffee, some new artisanal brew :)

 

Open source projects are notoriously difficult to make a buck, so I really appreciate the gesture.

 

The fact is I really like Unraid and for as long as I can, I'll keep supporting unbalance and my commercial projects in the Unraid ecosystem.

 

So, many thanks for your donations 🙌

Link to comment

Arffff...  I  cannot use unbalance anymore.

Quote

"unBALANCE needs exclusive access to disks, so disable mover and/or any dockers that write to disks, before running it. Also note that transfer speed may be affected by disk health"

 

I stopped docker, VM, clear the cache drive...  but the application remains locked.

 

Did I missed something ?

Link to comment
  • 4 weeks later...

I just started messing with this to move data in order to reformat drives.. so far so good.. Cool utility...   but I wish it were faster.

 

Not sure if this has been mentioned in any of the previous 61 pages ... forgive me.

 

Would it be possible/desired/wanted to change/modify/fork/clone/beta-test this to something that could do file transfers in a multi-threaded way?

 

I tried to do some searching around and apparently rsync itself doesn't offer any multi-thread capability, but in conjunction with 'parallel' and/or a couple other methods I saw it's at least possible to do, if it can be figured out.   <-- I can't figure it out, not a linux guy, too old to learn anything new ;)  One site I found with some code on it I had working a little bit, seemed ok, but not sure if I was doing it correctly.

 

So without getting into the nitty gritty details about, what's the possibility or interest level of creating a multi-threaded tool?  When you're moving TB's and TB's of data, single thread operations are very painful.

 

Thanks!

 

 

 

 

Link to comment
54 minutes ago, Energen said:

I wish it were faster.

Unless you are not running a parity drive at all, I doubt that the bottleneck is rsync.

 

When you are moving data from one parity protected drive to another, it's going to be slow, since you are restricted by the seek times and rotational latency of 3 drives. If you try to do more transfers simultaneously, it's going to be even slower than the cumulative speed of doing 2 transfers back to back, because you are going to be waiting on the parity drive to seek to yet another area, causing more latency.

 

If you are willing to forego parity protection for the duration, you could definitely speed things up.

Link to comment

Ah, yes, hmm.. I did not consider calculating parity at the same time.

 

I'm down to the last disk that I'll be able to move and format before having to move data off the first full disks, so perhaps then I'll disable parity and see what the speeds look like.

 

In my current operation I'm averaging around ~50MB/s so it will be easy to tell how much overhead parity accounts for, and then perhaps I can try some other stuff with the multi-thread moving without parity and see.. if I can figure out a easy multithread function.

 

Thanks for the feedback.

 

 

-- Edit,

After removing the parity drive, I had burst speeds of about 120MB/s, then for moving small files (photos and the like) it averaged around 85MB/s, fluctuating between 70-90MB/s.

 

-- Edit2,

So that's not advisable to do... just the parity rebuild time is worse.

 

Edited by Energen
Link to comment

So I haven't been able to find anyone else really mentioning anything about ghost files/directories... not sure what happened here but I'm finding that I have a lot of weird stuff happening after moving / stopping / restarting / etc the scatter process..   I've got what seems to be ghost file/directory entries..

 

For example, \\UNRAID\Photos\Photo1.jpg is actually a folder, with multiple sub-levels of the same path (i.e. UNRAID\Photos\Photo1.jpg\Photos\Photo1.jpg\Photos\Photo1.jpg\Photos\Photo1.jpg) where finally the last entry is the actual file.  

 

First I started moving all the actual files to a new path before finding out that I could just keep deleting the root folder and that would end up with the actual file where it was supposed to be.

 

Similarly related, I'm now deleting some stuff and I have to repeatedly delete it (from Windows) before it actually deletes, even though there's no "ghost" paths visible.

 

Any thoughts on what happened here?  If I stopped the process during a move operation and then started it again later, possibly on a different disk, could that leave these ghosts? My Windows share is seeing the file entry across all the unraid disks?

 

 

Edited by Energen
Link to comment
2 hours ago, Energen said:

If I stopped the process during a move operation and then started it again later, possibly on a different disk, could that leave these ghosts?

They probably are real files, left behind. A move operation is a copy followed by a delete. If you interrupt it before it finishes, it will have copied but not deleted. I suspect you have created quite a mess.

 

Before you start cleaning up that mess, I suggest learning how the unraid share system works, where each share is a root directory on one or more disks. If you have the same file in the same path on multiple disks, the user share system will only show the first one it finds. If you then rename or delete it, then the copy that is on the second disk will be visible.

Link to comment
  • 5 weeks later...

How can I select multiple drives at once to process in a 'scatter' operation?  I have 5 disks that I want to move all the data off and decommission; doing them one at a time (and needing to circle back and remember to move on to the next one at the appropriate time) is going to be a hassle.

Link to comment

Hello!

So...I made a mistake this past weekend.  I was replacing a drive with smart errors.  So, I installed some new drives, added them to my array and started a scatter from bad drive to disk 4 and 5...the new ones.  Only to see that I did not include the drives in my 'Global User Shares'

nLZukDO.png

😞

 

So...I am missing lots of data in my user shares.  I can see that my data did move because the drives are full but I'm not sure what the next step would be.  I  stopped the array and wondering if I should start in Maintenance mode - to see if the data is intact?

 

Thoughts? Ideas?

 

Thanks!

Link to comment
  • jbrodriguez changed the title to [Plugin] unbalanced

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.