[Plugin] unbalanced

[jbrodriguez]

14 hours ago, Seven7527 said:

stat: cannot statx '/mnt/disk1/Movies/Music': No such file or directory

 

14 hours ago, Seven7527 said:

:~# stat /mnt/disk1/Media/Music

right, those are 2 different folders, not sure why on the unbalanced ui, the selection comes as Movies/Music instead of Media/Music 🤷‍♂️

[Harlo]

Hey there,

i used unbalanced for the first time here. i have a dead drive that is in my unraid server. there was about 2.5 tb of data on the failed drive and it was running in emulated mode. when i ran the unbalanced to scatter all the data, i selected the other 8 drives in my array as the destination and let it rip. it ran for about an hour than gave errors. when i looked at the data on the failed (emulated) drive its showing a pile of random folders nothing in regards to my data. than like a moron i rebooted my unraid box... i lost the log file for the unbalanced scatter, also the drive now just says unmountable:unsupported or no file system. there was quite a bit of important data on this drive. any chance i can fix this? i am no longer able to view contents of this device. when i look at the share its self it appears 4tb less space (would account for the missing drive now)

[JonathanM]

1 hour ago, Harlo said:

i have a dead drive that is in my unraid server.

How do you know it's dead? The majority of the time Unraid fails a drive is because of a cabling or power issue, dead drives are relatively rare. When a write fails, for whatever reason, Unraid disables access and emulates the data using all the rest of the drives.

 

See if you can get a SMART report on the "dead" drive, and see if it will mount in the Unassigned Devices plugin.

[Harlo]

33 minutes ago, JonathanM said:

How do you know it's dead? The majority of the time Unraid fails a drive is because of a cabling or power issue, dead drives are relatively rare. When a write fails, for whatever reason, Unraid disables access and emulates the data using all the rest of the drives.

 

See if you can get a SMART report on the "dead" drive, and see if it will mount in the Unassigned Devices plugin.

no shes dead. i pulled it and its just ticking. not sure if its the head or the board. im hoping the board as once i move everything ill have spares and can swap boards hopefully get it back and recover the data. there is some very critical stuff that was on this that i really cant loose. 

[Harlo]

im currently running a parity check - dont think that will do much but worth a shot. im not an expert in unraid at all. i was really excited to find this addon to move data quickly so i could shrink the array and pull this device from the array. but unfortunately now im just seeing what i posted above. 

[JonathanM]

2 hours ago, Harlo said:

the drive now just says unmountable:unsupported or no file system.

Do a filesystem check using the GUI, on the main tab click on the Disk #.

[Harlo]

2 hours ago, JonathanM said:

Do a filesystem check using the GUI, on the main tab click on the Disk #.

ok ill give that a try. im going to wait till the parity check has finished so some time tomorrow ill stop the array and try the check. with that work though being there is no real drive there?

[JonathanM]

8 hours ago, Harlo said:

there is no real drive there?

You are checking the emulated drive.

 

Instead of polluting the plugin support thread, please start a new thread in the general area, include your diagnostics and a link back to your post here so anyone wishing to help can see what's happened so far.

[nightknight]

Does the new "unbalanced" plugin still require that all VMs and Dockers be stopped? This used to be a warning of the old unbalance/depreceted plugin, but I do not see it with the new plugin. 

 

Just checking if perhaps this is an overisght in how the new plugin works, or whether its a nice little convenience feture that I no longer need to stop dockers

[jbrodriguez]

17 hours ago, nightknight said:

Does the new "unbalanced" plugin still require that all VMs and Dockers be stopped?

check out this reply

 

[nearcatch]

Restrict unbalanced LAN Access

 

I haven't liked that unbalanced is available to anyone in my LAN once it's started, but I have figured out a solution for myself. Sharing the steps here for anyone else who is curious and uses a reverse proxy for other things, like I was.

 

How-to

 

1. Set up your reverse proxy to have an authenticated subdomain for unbalanced. I use Traefik and Authelia. You will have to do something specific to your setup, but this is what I added to my Traefik config file:

http:
  routers:
    unbalanced-rtr:
      rule: "Host(`unbalanced.unraid.lan`)" 
      entryPoints:
        - websecure
      middlewares:
        - chain-authelia-lan
        - error-pages@docker
      service: unbalanced-svc
  services:
    unbalanced-svc:
      loadBalancer:
        servers:
          - url: "http://192.168.1.10:7090" # substitute your unraid server's IP address and unbalanced port

 

2. Run these iptables rules in a console session so that any request to unbalanced's port gets rejected, unless it is from Unraid's IP or the IP range of your reverse proxy's network. Substitute the correct IP addresses and ports for your network. You can also add these to your go file to have them activated every time Unraid is rebooted.

iptables -A INPUT -p tcp --dport 7090 -s 10.10.1.0/24 -j ACCEPT # substitute the subnet your reverse proxy uses. you can also limit this to the exact IP of your reverse proxy docker container if you want
iptables -A INPUT -p tcp --dport 7090 -s 192.168.1.10 -j ACCEPT
iptables -A INPUT -p tcp --dport 7090 -j REJECT --reject-with tcp-reset

 

Result

 

After these two steps, unbalanced cannot be accessed by the ip:port of my Unraid server. It can only be accessed using https://unbalanced.unraid.lan, and because I add Authelia using a Traefik middleware, it requires authentication instead of being freely accessible.

 

No reverse proxy?

 

If you don't use a reverse proxy, you can still do step 2 and edit the 1st iptables rule to reject any request to unbalanced's port except from one specific computer on your LAN, which should still help limit access.

 

 

 

EDIT: Also for those who don't know, you can remove the iptables rules by running them again with "- D" instead of "- A". Restarting your Unraid server will also reset your iptables if you haven't modified your go file with these rules.

Edited April 23 by nearcatch

[Andiroo2]

I’m getting 100% CPU usage on CPU Core0 when I use Unbalanced. Everything is working correctly, but just wondering if this is normal. 

 

 

[jbrodriguez]

14 hours ago, Andiroo2 said:

I’m getting 100% CPU usage on CPU Core0 when I use Unbalanced. Everything is working correctly, but just wondering if this is normal. 

i'd say it's a bit odd, can you run  `top` or `htop` to confirm that unbalanced is the one using that core at a 100% ?

Edited May 4 by jbrodriguez

[Andiroo2]

5 minutes ago, jbrodriguez said:

i'd say it's a bit odd, can you run  `top` or `htop` to confirm that unbalance is the one using that core at a 100% ?

This is where it gets weird. htop doesn’t show any cores at 100%.  I see it only in the Unraid Dashboard and in my Grafana panel (fed by telegram with out of the box CPU reporting).

[sdowen]

I ran unbalanced to "gather" the data from various shares into individual disks for each share, and that worked great!  And I've then gone and updated the include and exclude disks.  However, what's left behind on each drive is a set of empty directories where the share data used to be.

 

Example: Index of /mnt/disk1 now has directory names of "Music", "Video", etc. but they're empty.

 

If I go to the Shares screen and run a size "compute" on a share, it says for each disk "Disk 1    Share is outside the list of designated disks" and the size is 6B. 

 

Can I safely delete those empty directories?  I'm not 100% sure, but I think they might be causing the individual drives to stay spun up because I have some media server dockers which should only be trying to access shares, but if those shares are spread across many disks, they seem to stay spun up and I'm trying to avoid that.

 

Thanks!

[jbrodriguez]

hi, the plugin doesn't delete top-level folders, since those are Unraid shares

 

i would ask in the general thread to see if it's safe to delete empty shares folders

[jesseasi]

STOP option when a transfer is running.

 

Can someone tell me what exactly happens when you stop a file transfer?  What happened if you stop in the middle of a large file - does.   Does the incomplete file get deleted and the original file stay where it was?   I am not sure exactly how the process works.  

 

I have had some issues where users are streaming via plex and a file transfer is making playback stutter.  Stopping the process fixes plex but what happenes to the files that were being transferred?

[jbrodriguez]

hi, when you press the stop button, it will 

 

- "kill" the current rsync command (there might be a bit of delay)

- stops the current operation, so any upcoming rsync commands are not run

- doesn't perform any move/deletion on the source file/folder

 

when an rsync command is stopped (killed), the file willl be partially copied on the destination and unchanged on the source

 

if you run the same operation (rsync command) again, it would pick up from where it left

Edited May 16 by jbrodriguez

[Mossshine]

How long does SCATTER scan take? I wanted to move whole user share from a 1TB NVME (around 90% full) to 14TB HDD that had 3.5TB free space to remove the NVME from my array. After 1 hour of waiting I killed it. Does it really take so much time?

Edited May 16 by Mossshine

[jbrodriguez]

1TB should take approx 32 - 36 hours ?

Edited May 16 by jbrodriguez

[Mossshine]

Just to scan? Jezus christ

[jbrodriguez]

oh you meant the plan phase !

 

that shouldn't take that long

 

check the last lines of /var/log/unbalanced.log when you run the scatter plan, to check for any issues there

[Mossshine]

Yes, sorry, I mean the plan phase. The log ends with this

```
2024/05/16 21:01:54 ---------------------------------------------------------
2024/05/16 21:01:54 ---------------------------------------------------------
2024/05/16 21:01:54  
2024/05/16 21:01:54 =========================================================
2024/05/16 21:01:54 Bytes To Transfer: 673.21 GB
2024/05/16 21:01:54 ---------------------------------------------------------
2024/05/16 21:02:34 received terminated signal. shutting down the app ...
```
It seems it calculated the size just fine, but the GUI was stuck?

[arcane]

How does one run unbalanced as user "nobody"?  I ran new permissions on a share I was moving, but it appears some of the directories had changed ownership to root.  Thank you.

[jbrodriguez]

it used to run as nobody, but i changed that, i seem to remember it was due to permissions changes in Unraid 🤔

 

you could run it as nobody via command line (i think)