[Plugin] unbalanced


Recommended Posts

What do the yellow check marks mean?

image.thumb.png.5270b447762bb7494ede701c532a6d16.png

 

I don't think I've ever seen that before yesterday, and I saw nothing about it on the GitHub page. (Maybe I just missed seeing it)

 

It's also telling me

Quote

skipping:deletion:(rsync command was flagged):(/mnt/disk5/TV/Emergency!/Season 07)

Which looks like it's trying to delete from disk5, but disk 5 is the destination, not one of the sources.

Edited by FreeMan
Link to comment
2 hours ago, FreeMan said:

It's also telling me

Quote

skipping:deletion:(rsync command was flagged):(/mnt/disk5/TV/Emergency!/Season 07)

Which looks like it's trying to delete from disk5, but disk 5 is the destination, not one of the sources.

yes, that's the destination, it's just showing the rsync "to" command

the yellow check mark indicates that there was an issue with that specific command, so the app didn't delete the source folder, the reason can be anything that rsync reports

  • Thanks 1
Link to comment
  • 2 weeks later...

I just moved the contents of an old 8tb drive to a new 20tb drive, unbalance reported 7.76tb were moved but the 20tb disk itself shows 13tb used.

 

I had a lot of hardlinks on the 8tb drive, all of which broke when using unbalance to move all the contents to a different disk.

 

I have a couple more disks that i want to do this operation on, but if i could i would like to avoid having to recreate several terrabytes of data, to then just delete when i recreate the hardlinks.

 

Is it possible to keep hardlinks intact when scattering a drive to one other specific drive? All the relevant data is in /mnt/user/media/

Link to comment
On 12/28/2022 at 5:09 PM, jbrodriguez said:

yes, that's the destination, it's just showing the rsync "to" command

the yellow check mark indicates that there was an issue with that specific command, so the app didn't delete the source folder, the reason can be anything that rsync reports

 

I am moving 6TB at the moment and noticed the yellow indicators as well.
As I can not see the log until it is finished, will the log for those be obvious so I can see what was wrong? I am assuming it is mostly it could not delete the source folder as it was in use or not empty or something.

 

image.thumb.png.f6c9978d2f5accd9af05ed02d4110cda.png

Link to comment
  • 2 weeks later...
On 1/18/2023 at 6:27 PM, LFFPicard said:

 

I am moving 6TB at the moment and noticed the yellow indicators as well.
As I can not see the log until it is finished, will the log for those be obvious so I can see what was wrong? I am assuming it is mostly it could not delete the source folder as it was in use or not empty or something.

 

image.thumb.png.f6c9978d2f5accd9af05ed02d4110cda.png

rsync generally throws errors on permissions, dates, ownership, it can be anything and is very generic most of the times, if there is a serious issue, it will print something more "pressing" :)

Link to comment

so, i spent the morning getting unbalance to compile :)

 

some changes in my dev environment i guess, had to upgrade a bunch of stuff and patch some code.

 

it's working now though !

 

i accepted a PR from MHLoppy on github, since i still had references to lime-technology.com 😮, so, thanks MHLoppy.

 

so, i'm set up to look at adding some changes, i'll start looking into the effort needed

Link to comment
3 hours ago, orlando500 said:

does unbalance support multiple cache pools? seems only cache is visible in unbalance by default if supported?

no but there was talk about multiple pools previously (1 or 2 years ago i think), but boiled down to it being hard to identify pools that aren't "cache"

Edited by Cull2ArcaHeresy
  • Like 1
Link to comment

Just thought I'd mention some unintuitive behavior for scatter: if I select multiple individual files, it attempts to place each one on a separate drive, even if there is enough space on a single drive. If there are more individual files than target drives selected, it claims there is not enough space in the target disks. 

 

E.g. I'm moving a several small log files (10 MB each), and my disks have several TB free. It will place the first log file on the first target disk, second on the next target disk, and then if there are more individual files than disks available, it will say "PLANNING: The following items will not be transferred, because there's not enough space in the target disks"

 

I selected scatter because I want to move them from a specific full drive to other drives.

Link to comment

i've had scatters going from multiple drives to multiple drives (and multiple to one) do the same thing saying that will not be transferred (idr if had space comment), so i reselected what was left and it just took multiple passes to finish. But mine was more than 1 working transfers per drive. I'll dig into history to be more specific if needed to help hunt down the bug or cause (assuming not intentional)

Link to comment

hi, the logic for scatter is 

- Get the contents of the selected files/folders from the source disk

Order the target disks by free space available

For each target disk, calculate how much it can be filled up with files/folder from the source disk

a file or folder tree is a single unit, in the case of a folder it will calculate the size of the entire tree (du folder/*)

not sure about the several small files scenario though, if they're at root level should place as many of them as they fit on a single drive

Link to comment
On 2/12/2023 at 7:56 AM, jbrodriguez said:

hi, the logic for scatter is 

- Get the contents of the selected files/folders from the source disk

Order the target disks by free space available

For each target disk, calculate how much it can be filled up with files/folder from the source disk

a file or folder tree is a single unit, in the case of a folder it will calculate the size of the entire tree (du folder/*)

not sure about the several small files scenario though, if they're at root level should place as many of them as they fit on a single drive

Well certainly sounds like a bug as this is not what occurs for multiple small files selected (as opposed to a folder)

I've attached a log file of this scenario, let me know if you need any more information for this scenario or if I am doing something incorrectly

 

Thanks!

 

unbalance.log

Edited by robobub
Link to comment
  • 2 weeks later...
On 2/21/2023 at 1:32 PM, dopeytree said:

Could we add a password to this? Not a fan of having open access ports.

Would it be easy to add the arrr's login system?

no you can not take a random mono/.net app and snip out part of their code and just shove it in another app.   you should look at setting up a reverse proxy. but also you shouldnt just blindly have your nas exposed to the internet either. look at requiring a vpn..

Link to comment

It's just a feature request for the future.

 

It's not on the internet (well only plex port is open). It's more for if some other device on the local network got infected it could mess up the unraid array as there's no login to the unbalance gui on port 6237. We have passwords for every other app that has its own port that is outside of the unraid web GUI.

Edited by dopeytree
Link to comment
  • 2 weeks later...
On 1/28/2023 at 7:20 PM, jbrodriguez said:

that's odd, it should accept the custom flags.

in the history tab, is the "-H" showing in the rsync commands ?

on the hardlinks issue @jbrodriguez

 

On 1/11/2023 at 10:25 AM, MammothJerk said:

I just moved the contents of an old 8tb drive to a new 20tb drive, unbalance reported 7.76tb were moved but the 20tb disk itself shows 13tb used.

 

I had a lot of hardlinks on the 8tb drive, all of which broke when using unbalance to move all the contents to a different disk.

 

I have a couple more disks that i want to do this operation on, but if i could i would like to avoid having to recreate several terrabytes of data, to then just delete when i recreate the hardlinks.

 

Is it possible to keep hardlinks intact when scattering a drive to one other specific drive? All the relevant data is in /mnt/user/media/

On 1/12/2023 at 7:33 PM, MammothJerk said:

from my reading of the webui, unbalance seems to run different commands per each subdirectory one level deep

ZarbePq.png

this would break all of my hardlinks from seeding to movies even though they are correctly in the same unraid share. but unbalance is moving their directories separately for some reason.

 

Seems to be some logic that moves the subdirectories separately which is what is breaking the hardlinks between said subdirectories.

 

i.e if i have two hardlinked files
 

/disk9/media/movies/movie.mkv
/disk9/media/seeding/movie/movie.mkv

the /movies/ and /seeding/ subdirectories get moved separately and the hardlinks break.

Edited by MammothJerk
  • Thanks 1
Link to comment

Is there a way to scatter where all the files are spread across all disks in the array similar to RAID (how it sripes all files across all disks) but with files instead. File-level striping at the file level rather than at the block level.

 

I want to do this so it increase the I/O performance when accessing multiple files it will spin up and use multiple HDD instead of just a single one taking all the load. I want this as I perfer having higher I/O performance and don't mind the drives spinning up.

 

I don't want to use actual RAID because of how inflexable it is with different drive sizes.

 

Looks like this can be done in python, I used user script files plugin and just replace the "Disk = "with your dir destination location and "src_dir = " with your source directory.

import os
import shutil

# Define the paths of the disks
disks = ['D:/test', 'E:/test', 'F:/test']

# Define the path of the source directory
src_dir = 'D:/test'

# Define a variable to keep track of the current disk index
current_disk_index = 0

# Recursively walk through the source directory and move files
for dirpath, dirnames, filenames in os.walk(src_dir):
    for filename in filenames:
        src_path = os.path.join(dirpath, filename)
        # Get the next disk path in a round-robin fashion
        current_disk = disks[current_disk_index]
        current_disk_index = (current_disk_index + 1) % len(disks)
        dst_path = os.path.join(current_disk, os.path.relpath(src_path, start=src_dir))
        # Create the destination directory if it doesn't exist
        if not os.path.exists(os.path.dirname(dst_path)):
            os.makedirs(os.path.dirname(dst_path))
        # Move the file to the destination directory
        shutil.move(src_path, dst_path)
        print("Moved {} to {}".format(src_path, dst_path))

 

Edited by Hellomynameisleo
Link to comment
  • jbrodriguez changed the title to [Plugin] unbalanced

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.