[Plugin] unbalanced

[JonathanM]

On 3/18/2023 at 8:40 AM, Hellomynameisleo said:

I want to do this so it increase the I/O performance when accessing multiple files it will spin up and use multiple HDD instead of just a single one taking all the load.

You do realize this will still limit your writes, since every write is updating the parity disk(s). Have you looked into turbo write?

[itimpi]

On 3/18/2023 at 12:40 PM, Hellomynameisleo said:

Is there a way to scatter where all the files are spread across all disks in the array similar to RAID (how it sripes all files across all disks) but with files instead. File-level striping at the file level rather than at the block level.

 

I want to do this so it increase the I/O performance when accessing multiple files it will spin up and use multiple HDD instead of just a single one taking all the load. I want this as I perfer having higher I/O performance and don't mind the drives spinning up.

 

I don't want to use actual RAID because of how inflexable it is with different drive sizes.

 

Looks like this can be done in python, I used user script files plugin and just replace the "Disk = "with your dir destination location and "src_dir = " with your source directory.

import os
import shutil

# Define the paths of the disks
disks = ['D:/test', 'E:/test', 'F:/test']

# Define the path of the source directory
src_dir = 'D:/test'

# Define a variable to keep track of the current disk index
current_disk_index = 0

# Recursively walk through the source directory and move files
for dirpath, dirnames, filenames in os.walk(src_dir):
    for filename in filenames:
        src_path = os.path.join(dirpath, filename)
        # Get the next disk path in a round-robin fashion
        current_disk = disks[current_disk_index]
        current_disk_index = (current_disk_index + 1) % len(disks)
        dst_path = os.path.join(current_disk, os.path.relpath(src_path, start=src_dir))
        # Create the destination directory if it doesn't exist
        if not os.path.exists(os.path.dirname(dst_path)):
            os.makedirs(os.path.dirname(dst_path))
        # Move the file to the destination directory
        shutil.move(src_path, dst_path)
        print("Moved {} to {}".format(src_path, dst_path))

 

Have you checked this actually works as expected?  
 

I see a potential problem?    You have the same drive in source and destination list.   Copying a file to itself can.end up truncating the file to zero length.

[Hellomynameisleo]

7 hours ago, JonathanM said:

You do realize this will still limit your writes, since every write is updating the parity disk(s). Have you looked into turbo write?

I'll take a look at turbo write but it looks complicated in how it works. I'll try the plugin out.

7 hours ago, itimpi said:

Have you checked this actually works as expected?  
 

I see a potential problem?    You have the same drive in source and destination list.   Copying a file to itself can.end up truncating the file to zero length.

Yes it works as expected, just don't use /mnt/user as the source and then /mnt/disk1 /mnt/disk2 etc. as the destination otherwise it will end up deleting the file.

 

Also it seems if you want to use /mnt/user/ where you just move the files to the folder name and add a "1" or something extra to the folder name so there's no conflict, then at the end of the script move it will rename the folder back to the original folder name. below example where it will move the folder music to music1 to the 4 disks in a round robin when its done rename folder "music1" to "music".

 

import os
import shutil

# Define the paths of the disks
disks = ['/mnt/disk1/Backup/Music1', '/mnt/disk2/Backup/Music1', '/mnt/disk3/Backup/Music1', '/mnt/disk4/Backup/Music1']

# Define the path of the source directory
src_dir = '/mnt/user/Backup/Music'

# Define a variable to keep track of the current disk index
current_disk_index = 0

# Recursively walk through the source directory and move files
for dirpath, dirnames, filenames in os.walk(src_dir):
    for filename in filenames:
        src_path = os.path.join(dirpath, filename)
        # Get the next disk path in a round-robin fashion
        current_disk = disks[current_disk_index]
        current_disk_index = (current_disk_index + 1) % len(disks)
        dst_path = os.path.join(current_disk, os.path.relpath(src_path, start=src_dir))
        # Create the destination directory if it doesn't exist
        if not os.path.exists(os.path.dirname(dst_path)):
            os.makedirs(os.path.dirname(dst_path))
        # Move the file to the destination directory
        shutil.move(src_path, dst_path)
        print("Moved {} to {}".format(src_path, dst_path))

# Rename the Music1 folder to Music
old_dir = '/mnt/user/Backup/Music1'
new_dir = '/mnt/user/Backup/Music'
os.rename(old_dir, new_dir)

print("Finished")

 

Edited March 19, 2023 by Hellomynameisleo

[daTroll]

Tiny cosmetic request: the next time you update this plugin, can you please remove the "v" prefix in the version designation? Most of my other plugins use the same YYYY.MM.DD format but without any "v" prefix. Occasionally I like to sort the list of plugins in the WebUI based on version, to quickly identify the oldest/newest.

 

Cheers,

T

[quimnut]

I see unbalance writing logs to my usb stick, it is the only software that writes logs there and I feel it's not appropriate. Additionally if I was to enable cloud backups of my boot stick it would leak what secret files I have moved.

 

Is it possible to stop logging. Perhaps even prefix it into console logs/dmesg?

 

Thanks.

[cobalt027]

I just upgraded from 6.9.2 (unbalance working) to 6.11.5 and now unbalance wont show the UI. Stop, start, change port, uninstall, reinstall. It just doesn't show. Any ideas where i can look to diagnose the problem and fix it?

Thank you for this plugin. I use it all the time.

[cobalt027]

9 minutes ago, cobalt027 said:

I just upgraded from 6.9.2 (unbalance working) to 6.11.5 and now unbalance wont show the UI. Stop, start, change port, uninstall, reinstall. It just doesn't show. Any ideas where i can look to diagnose the problem and fix it?

Thank you for this plugin. I use it all the time.

Nevermind. It was chrome forcing it to https. once i changed it manually to http it worked. Also works completely normal in firefox.

 

On another note, I have 2 cache pools setup. Unblanace only see one of them. The one it sees is a normal 1 SSD pool. The one it doesn't see is 2 SSD drives in a RAID1. Does that make a difference?

[bobC]

On 2/9/2023 at 12:48 PM, Cull2ArcaHeresy said:

no but there was talk about multiple pools previously (1 or 2 years ago i think), but boiled down to it being hard to identify pools that aren't "cache"

Yes, it does not seem to recognize that there is a zfs pool to move/copy to/from , it would be wonderful to see an update on this fine app,

[redwolf]

hello, i am getting this message when i press the plan button on scatter, does it matter?

[kiki]

13 hours ago, redwolf said:

hello, i am getting this message when i press the plan button on scatter, does it matter?

I install today the plugin just to test it for a future migration, same thing,

i install the Fix common Problems plugin and launch the Docker Safe new permissions (in Tools tab of the Unraid WebUi) but the message stay.

The worst part is if a launch a migration, the mover doesn't.. move anything. but it tell me i succeed to move 50GB in less than 1 second.. 😄

Edited May 5, 2023 by kiki
orthograph edit

[veri745]

On 4/22/2023 at 9:57 AM, cobalt027 said:

On another note, I have 2 cache pools setup. Unblanace only see one of them. The one it sees is a normal 1 SSD pool. The one it doesn't see is 2 SSD drives in a RAID1. Does that make a difference?

I just came here to report this same issue.  I installed a second ssd and set it up as a second cache pool (btrfs), but it is not visible from Unbalance

Edited May 10, 2023 by veri745

[TheGleaner]

I am having the issue of when trying to transfer from a disk to the cache, some files will not transfer as it says there is not room in the cache, but the cache does have room and the files are small, small enough they could all fit on the cache.

 

version v2021.04.21 (5.6.4)

 

Unraid 6.11.5

[itimpi]

4 hours ago, TheGleaner said:

I am having the issue of when trying to transfer from a disk to the cache, some files will not transfer as it says there is not room in the cache, but the cache does have room and the files are small, small enough they could all fit on the cache.

 

version v2021.04.21 (5.6.4)

 

Unraid 6.11.5

Do you have a Minimum Free Space setting on the cache pool or on the User Share involved that my be kicking in and stopping the transfer?   You are likely to get better informed feedback if you attach your system’s diagnostics zip file to your next post in this thread.

[Vetteman]

When I run UNBALANCE and attempt to move a folder I get a warning about user nobody not having enough rights. When I go into the settings for UNBLANCE I on see Enable server and port. I do not see the user setting that is mentioned earlier in the thread. I am assuming I need to change the user to root or an equivalent admin. 

 

How do I run UNBALANCE as root?

 

Thanks

[TheGleaner]

On 5/23/2023 at 5:36 AM, itimpi said:

Do you have a Minimum Free Space setting on the cache pool or on the User Share involved that my be kicking in and stopping the transfer?   You are likely to get better informed feedback if you attach your system’s diagnostics zip file to your next post in this thread.

No. I was meaning I was trying to transfer MBs worth of files to a 2 TB SSD cache drive, but I have not had the issue since I posted. Go figure.

[dopeytree]

Is this all good to use with 6.12? the last update was v2021-04-21

[isvein]

On 6/15/2023 at 4:57 PM, dopeytree said:

Is this all good to use with 6.12? the last update was v2021-04-21

I have tested it now, it works just as usual, but if you try to use it to move data to an ZFS formatted disk, each folder does not become its own dataset as it does if you create a new share on an zfs drive.
All folders and files just gets moved/copied to the pool root

[jbrodriguez]

1 hour ago, isvein said:

each folder does not become its own dataset as it does if you create a new share on an zfs drive

creating a dataset on a zfs pool is a design decision from the zfs admin (user :)), unbalance will not see datasets only shares, so it's up to the user to create shares that unbalance can use (not sure if i'm missing something here)

[isvein]

2 hours ago, jbrodriguez said:

creating a dataset on a zfs pool is a design decision from the zfs admin (user :)), unbalance will not see datasets only shares, so it's up to the user to create shares that unbalance can use (not sure if i'm missing something here)

That makes sense, but can unbalanse put data into shares on another disk?
I thought it just copied or moved folders from root of one disk to root of another, since this works just fine on say xfs disks.

[madejackson]

So as with isvein discussed on reddit (here and here), unBALANCE does not create the nececcary datasets in advance of moving the files over (unlike core unRAID functions like mover that do work as they should). But I also experienced another issue with zfs and unBALANCE:

 

unBALANCE is unable to read the size of a zfs-formated disk correctly. As soon as there is some data filled onto the zfs formated disk, it just shows the remaining free space as "size":

[jbrodriguez]

3 hours ago, madejackson said:

...

unBALANCE is unable to read the size of a zfs-formated disk correctly. As soon as there is some data filled onto the zfs formated disk, it just shows the remaining free space as "size":

what is actual free space on these disks (i understand you're referring to disk 1 and 2 ) ?

[madejackson]

19 hours ago, jbrodriguez said:

what is actual free space on these disks (i understand you're referring to disk 1 and 2 ) ?

Free Space calculation is fine as far as i can see. It's just the total size that's wrong.

 

Updated screenshots, unRAID and ZFS-Master as comparison:

Edited June 27, 2023 by madejackson

[isvein]

Im getting this message now when I try to move data from my last xfs drive (the last one does not have much free space) to zfs drives. Tried different things but same happens
Disk9 is xfs and disk8 is zfs

Edit: Hmmm looks like this may be because of something else, but not sure why

Edited June 29, 2023 by isvein

[Can0n]

Hello @jbrodriguez

love the unbalance plugin however I am seeing one issue

one of my cache pools is now showing up

I have 2 party, 22 array  disks, 1 ZFS pool with two 1TB SSD's and the pool not showing is a second ZFS pool with two 4TB spinners in ZFS it is also reading the size of the ZFS SSD's incorrectly as 690GB in size

the pool is called download-cache so not sure if there is a character limit for it to display or what the issues is and wondering if you could take a look

Edited July 10, 2023 by Can0n

[madejackson]

Edit: solved.

I was able to find the culprit: It's the hard-drives from WD (WD50NPZZ). I can write to my Toshibas and Seagate ST5000LM000 with about 35Mbit/s, but the WD (WD50NPZZ) is stuck at <5MB/s.

 

I'm on 6.12.2 now and suddenly have some very slow speeds (<4MB/s), when trying to copy from btrfs pool to my newly zfs formatted array disk. I am 100% sure that nothing else is accessing my disks (fresh reboot, docker+VM stopped). Any Idea what this could be?

 

In the last couple of transfers on 6.12.1 from xfs disk to zfs disk I usually got 20-30MB/s.

 

Speed Fluctuate a bit. Sometimes it goes up to 25MB/s for a short time only for then to fall back down to 3MB/s, grinding to a complete halt and then speed up again. The Files are regular sized MP4's and maybe some .srt subtitels. There are no errors in the logs. Diskspeed lists the speed of this particular WD disk at about 90-150MB/s

 

 

 

unbalance.log

 

 

Edited July 11, 2023 by madejackson