[Plugin] unbalanced


Recommended Posts

9 hours ago, huffsper said:

Going to https://UNRAIDIP:6237/ gives me ERR_SSL_PROTOCOL_ERROR. I'm using a self signed certificate for unRAID instead of Let's encrypt if that matters

 

I have it working on my server, but with the unRAID assigned LE certificate.

 

So, a couple of questions:

 

- Are you using the unRAID create self-signed cert ?

- the ERR_SSL_PROTOCOL_ERROR is the red warning from the browser or you have already trusted this self-signed cert in your browser ?

- In the first few lines of the log, it shows whether it's listening on http or https. What does your log show ?

 

itimpi, https support means, roughly, that if a cert is present and the ssl setting is yes (and auto IIRC), it will listen on https using that cert, otherwise it falls back to http.

Link to comment
14 hours ago, jbrodriguez said:

 

I have it working on my server, but with the unRAID assigned LE certificate.

 

So, a couple of questions:

 

- Are you using the unRAID create self-signed cert ?

- the ERR_SSL_PROTOCOL_ERROR is the red warning from the browser or you have already trusted this self-signed cert in your browser ?

- In the first few lines of the log, it shows whether it's listening on http or https. What does your log show ?

 

itimpi, https support means, roughly, that if a cert is present and the ssl setting is yes (and auto IIRC), it will listen on https using that cert, otherwise it falls back to http.

 

- I'm using a completely self signed cert, made in the CLI.

- The "ERR_SSL_PROTOCOL_ERROR" is not the typical warning for certs. It's the same screen as a broken page gives you in Chrome. No buttons other than retry.

- My log says it's only listening on HTTP. Here you go!

 

I: 2018/03/13 11:44:37 app.go:51: unbalance v5.0.0-1042-b4a3ac6-v2018.02.27 starting ...
I: 2018/03/13 11:44:37 app.go:59: No config file specified. Using app defaults ...
I: 2018/03/13 11:44:37 server.go:54: Starting service Server ...
I: 2018/03/13 11:44:37 server.go:71: Serving files from /usr/local/emhttp/plugins/unbalance
I: 2018/03/13 11:44:37 server.go:129: Server started listening http on :6237
I: 2018/03/13 11:44:37 array.go:45: starting service Array ...
I: 2018/03/13 11:44:37 planner.go:50: starting service Planner ...
I: 2018/03/13 11:44:37 core.go:98: starting service Core ...
W: 2018/03/13 11:44:37 core.go:113: Unable to read history: open /boot/config/plugins/unbalance/unbalance.hist: no such file or directory
I: 2018/03/13 11:44:37 app.go:73: Press Ctrl+C to stop ...
I: 2018/03/13 11:44:48 core.go:170: Sending config
I: 2018/03/13 11:44:48 core.go:175: Sending state
I: 2018/03/13 11:44:48 core.go:185: Sending storage

 

Edited by huffsper
Formatting
Link to comment
1 hour ago, huffsper said:

My log says it's only listening on HTTP. Here you go!

 

Ok, that's definitely the issue.

 

The plugin is listening on http and you're trying to access it via https.

 

The logic to determine whether to run on http or https is described here, as per standard 6.4.x+ operation.

 

This comes from the help text available in unRAID's SSL config page.

 

Is it possible that your certificate is named different? located elsewhere ?

 

A 'workaround' would be to put your cert in /boot/config/ssl/certs/<server-name>_unraid_bundle.pem.

Link to comment
14 minutes ago, jbrodriguez said:

 

Ok, that's definitely the issue.

 

The plugin is listening on http and you're trying to access it via https.

 

The logic to determine whether to run on http or https is described here, as per standard 6.4.x+ operation.

 

This comes from the help text available in unRAID's SSL config page.

 

Is it possible that your certificate is named different? located elsewhere ?

 

A 'workaround' would be to put your cert in /boot/config/ssl/certs/<server-name>_unraid_bundle.pem.

 

https://imgur.com/a/Vzg2K

 

Should be correct AFAIK?

Link to comment

Hello,

    I just happened to notice that after using this that it seems to de-sparsify (un-sparsify?) files. Rsync has a --sparse option, but I'm not sure what effect that would have on non-sparse files. Specifically my vdisk backups were affected. If I had moved docker.img, or libvirt.img, they would presumably have been affected as well. I also have some dockers that create sparse files.

 

Disclaimer: Unfortunately, I forgot to double-check the files it de-sparsed (de-sparsified?) before I removed them, so maybe I'm wrong about this. It would be worth double-checking.

 

You could run a check like this to return a list of sparse files. The command searches sub-directories.

find /directory/to/be/copied -type f -printf "%S\t%p\n" | gawk '$1 < 1.0 {print}'


Hopefully this information is useful to someone.

Thanks,

    JTok

Edit: I forgot to include a link to the code source: https://www.thegeekdiary.com/how-to-find-all-the-sparse-file-in-linux/

Edited by JTok
forgot to include code source
Link to comment

One thing mentioned I'd echo is that a cancel button would be helpful. I've found that if I have SSH access I can kill an RSYNC process and bring it down but it's not graceful and not something I'd expect a novice to be comfortable with.

 

That said, I was able to use this tool to successfully move over 35TB of data around to encrypt 16 devices. I still haven't done my cache but it's on the list - I think that can be done? Anyway, unBALANCE, the Krusader container, and MidnightCommander were all very helpful and I appreciate the effort that's gone into this tool - thank you!!

  • Like 1
Link to comment
18 hours ago, JTok said:

Hopefully this information is useful to someone.

 

Thanks for the heads up !

 

8 hours ago, BLKMGK said:

it's not graceful and not something I'd expect a novice to be comfortable with

 

Yes, it's not precisely user friendly.

 

I'll implement it with a confirmation (this will create duplicate copies of files, leave an incomplete operation, do it at your own risk, etc).

 

In any case, I'm glad the plugin was helpful, that's a lot of data to move around.

Link to comment

I was moving from ReiserFS so yeah it was time! I've been using unRAID a very very long time and there was finally a good reason to move to a new filesystem, crypto! I was on travel part of the time which is why using a portable device was so important. It's nice to be sitting in a bar somewhere on wifi and remote in to move things around :)

Link to comment

I converted one of my servers, using unBALANCE, from RFS to XFS, except for one 3Tb disk that is dying/is dead. I have to replace it.

 

If encryption was available then, I'd have moved and encrypted data at the same time :)

 

I'll be monitoring for in-place encryption. If it becomes available without data transfer between disks, I'll convert my second array, which is currently fully RFS (about 20Tb).

 

Remote access is really cool if you need to do some or a lot of traveling. OpenVPN FTW !!

 

Cheers !

Edited by jbrodriguez
Link to comment

I had a server lockup last night because of (?) unBALANCE :(

 

When I woke up my server wasn't responding to anything. All the services were down, and I was unable to access the web UI or connect via SSH. I hooked up a monitor and keyboard and tried to login but after entering the password nothing happened. The only thing that was responding was the unBALANCE web UI. It had finished transfering my folder for TV-Shows which is probably around 2.5TB, but was stuck at the "removing folder" part (there was some other folders waiting to be moved next). I had to hard reboot the server. When that was done I manually deleted the TV-shows folder because I could verify it had been copied but not deleted yet.

 

I was unable to run 'diagnostics' because of the lockup but I can see unBALANCE managed to save a logfile. I can share it in a private message for anyone who wanna help troubleshoot.

Edited by huffsper
Link to comment
On 2018-03-18 at 6:36 PM, jbrodriguez said:

I'll take a look

 

Did you find anything weird looking at the log? I noticed in my logs that unBALANCE is using 'rm -rf' to delete files, wouldn't using rsync's '--remove-source-files' be better?

Edited by huffsper
Link to comment
On 3/24/2018 at 12:02 PM, huffsper said:

 

Did you find anything weird looking at the log? I noticed in my logs that unBALANCE is using 'rm -rf' to delete files, wouldn't using rsync's '--remove-source-files' be better?

 

I took a look and didn't find anything unusual.

 

There were a couple of instances where the rsync proc was still available, but there was no file descriptor for the file being copied. But it eventually would go on.

 

I now see that I removed the disk type from the logs, I have to put that back.

 

Are any of your disks still on RFS ?

 

As for --remove-source, I can't remember if I ever considered it. It'll probably go down as to when rsync removes the source: is it after each file has been copied or after the entire command has finished ?

 

I'll give it some thought.

Link to comment

Hello, dumb question with unbalance (unRaid 6.5):

 

I get the warning:

Quote

155 folder(s) with a permission other than 'drwxrwxrwx'

 

But when I open the terminal and look into, they are 'drwxrwsrwx' (so a 's' instead of an 'x').

 

How can I change that? Should I change that?

Link to comment
5 hours ago, jbrodriguez said:

Are any of your disks still on RFS ?

 

All my disks in the array were XFS at the time (now XFS encrypted) and cache drive was BTRFS.

I just tried moving a folder containing 5 files at around 400mb each with --remove-source-files, and it didn't delete anything until all of them were transfered. But if the problem I had was just a freak accident of unknown causes it might not be worth changing from 'rm -rf' to this.

Link to comment
6 hours ago, Polopollo said:

Hello, dumb question with unbalance (unRaid 6.5):

 

I get the warning:

 

But when I open the terminal and look into, they are 'drwxrwsrwx' (so a 's' instead of an 'x').

 

How can I change that? Should I change that?

 

try the following: You need to remove the sticky bit from the group field. IIRC, the group-sticky just forces all subdirectories to keep the same group as the parent.

 


chmod -R g-s /mnt/disk#/share/directory-to-correct

Link to comment
1 hour ago, BRiT said:

 

try the following: You need to remove the sticky bit from the group field. IIRC, the group-sticky just forces all subdirectories to keep the same group as the parent.

 

 


chmod -R g-s /mnt/disk#/share/directory-to-correct

 

 

It worked!! And I now learned this command! Thank you again!!

Link to comment

Is it possible to scatter to the same source disk + other disks?

 

ex. i started off with 1 disk in the array (which is now full), but i recently just added a second disk with the same size. Now i want to split the data across the 2 disks.

 

Thanks!

Link to comment
On 2018-03-30 at 7:23 AM, endiz said:

Is it possible to scatter to the same source disk + other disks?

 

ex. i started off with 1 disk in the array (which is now full), but i recently just added a second disk with the same size. Now i want to split the data across the 2 disks.

 

Thanks!

 

When you press "Plan" it will show you how the data will be distributed before moving, so my recommendation would be to keep checking folders until you have an even split and then move it. Not the answer you had hoped for but hopefully helpful anyway.

 

To continue having an even split after the move you would go into the Share settings in unRAID and change Allocation method to "Most free".

Edited by huffsper
Link to comment
13 minutes ago, huffsper said:

To continue having an even split after the move you would go into the Share settings in unRAID and change Allocation method to "Most free".

 

endiz:

 

You should consider why you even want this though. If you get to a place where they are even, then most free is going to write one, then maybe write the other, then maybe write the first, etc., simply because each disk briefly has the most free. High-water is the default because it is a compromise between distributing the files, and constantly switching disks.

 

Unless your first disk is really nearly full you might just let it alone. Rather than worrying about even distribution, you should make sure you set the Minimum Free for each user share. Minimum Free should be set larger than the largest file you expect to write. It has no way to know how large a write will become, so if it has more than Minimum Free, it can choose the disk and if the file is too large, it will fail. If a disk has less than Minimum Free, it will choose a different disk.

  • Like 1
  • Upvote 1
Link to comment

What is the command line process to kill unbalance and rsync?

 

I am trying to scatter 2TB from one disk to a few others. I have unassigned the parity drive to speed the process but the transfer speed is still only 17MB/s it should be 4x that. I want to kill the process and troubleshoot the speed issue. My question is how does one do that. I know the GUI can't kill it but I have seen mention that it can be done via a terminal session.

 

Can someone post step by step instructions, please?

 

Thanks

Link to comment

Hi,

 

Yes, this is a pending issue I'll be addressing in the next release (not sure yet when that will happen).

 

For the time being, you can do the following 

 

- killall unbalance (to prevent any other rsyncs to be launched)

- killall rsync (to terminate the currently running rsync)

 

With the current architecture, it's quite simple to 'do' this from the plugin, I'm looking into packaging this with the per-sector calculation change, but maybe I'll bring this forward, as soon as I can allocate some time to unBALANCE.

Link to comment
  • jbrodriguez changed the title to [Plugin] unbalanced

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.