Jump to content
jbrodriguez

[Plug-In] unBALANCE

1166 posts in this topic Last Reply

Recommended Posts

Can someone please help. I am having so many issues learning v6. I came from unrid v5 for 5 years without any issues...

 

I am encrypting drives so I am first using the unbalance plugin to move files from one drive to another, then exncrypt the empty one, etc...

 

For some reason everything has come to a crawl lately. I have disabled any parity and don't even have any parity drive assigned. I am getting max speeds of about 20MB/s! I just goan i5 8400 processor, LSI IBM m1015 controller. Never had issues with such slow speed before! Can someone help me out and let me know what's happening? I feel like this is slowing down my other dockers that I installed. I have 16 gigs of ram too. What's happening and why is everything so slow, even without parity!

Screen Shot 2018-12-25 at 12.18.35 AM.png

Edited by maxse

Share this post


Link to post

also, port 6238 seems to load but not port 6237 which is the actual port specified by unbalance server plugin. Nt even sure where 6238 comes from. Also it isn't a secure connection (no 's' in http) not sure if that  matters help! 20MB/s and over 38 hours to transfer 3.22TB with no parity assigned!

Share this post


Link to post

btw, I've already encrypted 2 drives, new drives (cache and array drive) but I never provisioned the SSLl Not even sure what that means really, or if that's what causing my issue. Anyway I tried to do it and got his error. Attaching screen shot.

 

I am using a pi-hole on a raspberry pi, with the router dns set to the local ip for for the raspberry pi. not sure if this is why I get this error or if this is something should fix, but hopefully you guys could help? Did i screw up  by encrypting drives without this ssl thing? I just saw that space invader one talked about it

 

This is slowly starting to get VERY overwhelming. Can you guys help me? I do I need to do this SSL thing? Are my drives not really encrypted without this? I don't know what to do, is this the issue with why my speed is so slow? Again I was able to encrypt 2 drives already without this SSL DNS, resolver thing which I don't even know what it is, or how to fix it on my ISP router with Pi-Hole running

Screen Shot 2018-12-25 at 1.25.58 AM.png

Edited by maxse

Share this post


Link to post

As stated in other threads, Stop your plex docker and any other dockers that may be scanning the system until after your unbalance operations finish.

Share this post


Link to post

For some reason I cant UN Check Dry Run and cannot do anything on this, tried reinstalling and tried with a different user and all result in the same thing. 

Share this post


Link to post
25 minutes ago, ceyo14 said:

For some reason I cant UN Check Dry Run and cannot do anything on this, tried reinstalling and tried with a different user and all result in the same thing. 

Have you tried a different browser?

Share this post


Link to post
3 minutes ago, trurl said:

Have you tried a different browser?

Yep. Forgot to mention this. Tried Chrome, Firefox, Edge, IE, IE in compatibility mode and even through my phone as well as on the server. Also tried from Win 10 and Server 2019 and Ubuntu and Mac OS. Lol...

 

Have never been able to use the plugin and same issue since I installed it initially some versions ago.

Share this post


Link to post
4 hours ago, ceyo14 said:

For some reason I cant UN Check Dry Run and cannot do anything on this

By can't uncheck, you mean it's grayed out right ?

 

Just to be sure, you did provide enough data to the plugin to perform any given operation (chose files/folders and target drive in gather, the full steps in scatter).

 

Other than that, I wouldn't really know.

 

In Chrome, you could take a look at the Developer Console and check for any error.

  • Like 1

Share this post


Link to post

I'd like to say a quick thanks to all of you people who donated this past year.

 

I didn't reply individually but I certainly appreciated it (and the notes you wrote).

 

Thanks !

Share this post


Link to post
On 1/3/2019 at 5:24 AM, jbrodriguez said:

By can't uncheck, you mean it's grayed out right ?

 

Just to be sure, you did provide enough data to the plugin to perform any given operation (chose files/folders and target drive in gather, the full steps in scatter).

 

Other than that, I wouldn't really know.

 

In Chrome, you could take a look at the Developer Console and check for any error.

I didn't know I had to do the Plan step first. It worked... Just my impatience I guess... It is now running. Thanks for this plugin!

Share this post


Link to post

I try to cleanr my disks up so i can transform them into encrypted... i choose gather disk1 -> disk 8

 

its only 30-40mb/s? is that normal? If i transfer from/to array i get around 100mbs normal. (smb)

 

Also i wonder, i did now 3 times gasther disk 1 -> disk 8 (had some errors about not readable files, deleted them) the last 2 times it finished without error, but when i look in unraid ui, it says there are still files left...?

 

(i even removed parity to speed it up)

Edited by nuhll

Share this post


Link to post

Send me the logs, I'll take a look (/boot/logs/unbalance.log)

30/40 mbps seems about par, it's reading from a drive, then writing to a different drive and parity.

I guess you reach 100mbps from another pc and using cache?



Sent from my iPhone using Tapatalk

Share this post


Link to post

Like i said, i removed parity, thats why i expected more.

 

I have nearly 1gbits from array.

 

< log removed after he saw it >

Edited by nuhll

Share this post


Link to post
8 hours ago, nuhll said:

Like i said, i removed parity, thats why i expected more.

Oh, ok. Not sure about the speed in that case.

 

The portion of the log you sent looks fine 🤷🏼‍♂️

 

Be aware that move is implemented as copy then delete.

 

If there's an operation-ending error during transfer, the original files are left intact in the source disk.

 

You can review which command failed in the history screen.

Share this post


Link to post

I restartet now and retried disk1 -> disk8

 

....

 

atleast transfer speed is np anymore :) (it look a second)

 

1123.png

 

edit:

okay nvm.

 

I found the last corrupt file.

 

It now finished and the ghost files (?) dissapeared.

 

Sad thing is that some corrupt files (it says then no permissions) break the whole process, thats pretty ... dumb?!

 

Ive run the permission thing ofc, but even that cant set some permissions and the files are non playable, so i guess, they are corrupt.

 

I need to do: start, wait until it fails, look at log, delete file, start.. and so on... 

 


W: 2019/01/23 15:22:09 core.go:821: getReadBytes:err(open /proc/29288/io: permission denied):xfer(0)
W: 2019/01/23 15:22:09 core.go:766: command:end:error(exit status 13)
I: 2019/01/23 15:22:09 core.go:770: command:retcode(0):exitcode(13)

he know ssomething is wrong

 


6 files(s) with a permission other than '-rw-rw-rw-' or '-r--r--r--'

would be cool if it could just display the items so i dont need to start top start stop...

Edited by nuhll

Share this post


Link to post
9 hours ago, nuhll said:

W: 2019/01/23 15:22:09 core.go:766: command:end:error(exit status 13)

This is the main issue

 

One of the rsync commands returned error 13, which per the manual is "13 - Errors with program diagnostics".

 

Currently, the plugin handles only error 23, meaning it flags the command as having issues, then keeps going (doesn't break the process).

 

I could probably do the same for error 13 (maybe for any error), but you would still need to handle the issue manually.

 

You're the only one with enough context to know what to do (which is what you actually did).

 

9 hours ago, nuhll said:

W: 2019/01/23 15:22:09 core.go:821: getReadBytes:err(open /proc/29288/io: permission denied):xfer(0)

This is a by-product of how rsync progress is being monitored (the process descriptor doesn't exist since the command finished), it's not an issue.

 

9 hours ago, nuhll said:

would be cool if it could just display the items so i dont need to start top start stop...

Could probably do something like that, but I'd need to take a deep look into it.

Share this post


Link to post

I did nothing.

 

I just want to move all files from one disk to antoher (so i can encrypt it).

 

I use scatter disk 2 -> disk 8.

 

It seems like some of my files are corrupt or something (most of them i cant playback, so i guess its corrupt or was just bad download)

 

for the corrupt files, the "use docker safe permissions" plugin does not work (it does not change the file permissions, probably because it also cant access it??? (idk) But since all good files are correctly permissioned after i run that plugin, i can find out the corrupt files left unpermissioned. because of that files the whole process stops and i need to manually start stop start stop start stop until i find the e.g. 10 failure files in 10000... :)

 

it would be great if the plugin just displays the failed items (or just log it to log file?) so i could remove all failed files and dont need to startt stop always.

 

 

You know what i mean? :D

 

Also i notcied i deleted the unbalanced.log, it doesnt get recreated...?

Edited by nuhll

Share this post


Link to post

Yes, I feel your pain.

That logging could be improved, although I'm not sure how to do it.

In general terms, the plugin says: rsync! copy this folder from disk a to disk b.

rsync goes about it and internally it copies all relevant files.

I'm not sure if I can correlate exactly which file causes an error by just looking at the rsync's progress output.

Maybe it's easy, maybe it isn't, in any case it'd need some investment in terms of the developer's busy time


Sent from my iPhone using Tapatalk

Share this post


Link to post

I mean, if you click plan, it shows X files are wrong permissions, so your script does know which files, right?

 

Just write that to the log file? :)

 

Also i would suggest recreating the unbalanced.log if u start a new job, if the file is not there, it only seems to get recreated when you disable and enable it again.

Edited by nuhll

Share this post


Link to post
6 hours ago, nuhll said:

Just write that to the log file? :)

That's one source of issues (permissions/ownership), but there could be potentially hundreds of files to be logged, so that wouldn't be very useful either.

 

And the docker-safe permissions script, pretty much takes care of that.

 

The other source of issues is rsync errors while transferring (not related to permission/ownership).

 

That's what I was referring to earlier, that needs more 'figuring out'.

Share this post


Link to post

No.

its the same issue.

 

The files which cant be permission changed, are the corrupt files. If i delete these files, then the count of permission issues goes down, so that are the correct files. The transfer error are on these files. If i one by one delete these, all works expected. But its very hard to do because if i had a full list, i just check that list, delete files and start, finish.

 

Then just make a checkbox to log the files if checked to log?

Edited by nuhll

Share this post


Link to post
2 hours ago, nuhll said:

The files which cant be permission changed, are the corrupt files. If i delete these files, then the count of permission issues goes down,

You're only considering your specific case, which is natural.

 

But there are instances where permissions/ownership are fine, yet you still find rsync issues, so the app can't generalize based on a single scenario 🤷🏼‍♂️

 

In any case, I'll take a look but I have some other things in my plate right now.

Share this post


Link to post

Yes, i know, thats my situation, thats why i said, maybe just add a checkbox, or config edit, so ppl who want to know which files exactly, can change it and find the files easier.. :)

Edited by nuhll

Share this post


Link to post

What i dont understand is:

I did now already 2 or 3 times gather move disk 2 -> disk 9, its 2 times successfull (with a checkmark in yellow), but if i go to history and redo it (because i still see files) it does the same again... and again... and again? Like if its not really moving it?

 

WHat does that mean?

skipping:deletion:(rsync command was flagged):(/mnt/disk9/Archiv/Spiele)

 

Nvm. i found out you can expand if its yellow flagged and can manually click rsmc, i checked that the dirs are empty and clicked it.

Speed is also relativly normal on other hdds, i guess its the one hdd which sucks. 80-180mb/s - while doin other stuff tho

Edited by nuhll

Share this post


Link to post

I'd just like to say everything is working fine for me, but I'd like to request a feature:

 

Backstory:

I have a load (100+) of directories scattered across the array, I've recently become aware of the split level feature so that playback is seamless. However, I have no way of bringing the files already scattered back together. That's when I started using unBalance. It works well, but I can't move everything at once as I don't have the space to shift it all. So I have to move in batches, work out what I can afford to move, it's not difficult but it could be easier if:

 

The Request:

There was an option to go through each of the selected gather directories and move each one individually to the best drive possible for that directory. Basically gathering directories to the best place individually.

 

Or a batching system so I could queue them all up, painstaking but it would be something.

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now