[Plugin] unbalanced

[racingsquirrel]

I am trying to move files from one pool to another but neither of the pools/disks are showing up in the plugin, although other pools are.

Only difference I can see is that the 2 missing pools contain HDDs and the pools that are showing up are SSDs.

[NickI]

Hi i am getting the same strange behaviour from the plugin. I am a long time user of Unbalanced so i know this is this not normal. To explain in a bit more detail

 

I am trying for example to gather some files distributed to multiple disk to cache. So i select the GATHER action, select the share and click Next.  Usually the plan phase took around 30sec max. After looking at the log and waiting for the plan to complete, the planning phase took around 6 minutes to complete. Have a look at the log

 

2024/06/04 08:37:16 scanning:disk(/mnt/disk1):folder(nextcloud)
2024/06/04 08:40:13 issues:owner(0):group(0):folder(39465):file(91400)
2024/06/04 08:40:14 items:count(6):size(13.40 GB)
2024/06/04 08:40:14 scanning:disk(/mnt/disk2):folder(nextcloud)
2024/06/04 08:42:23 issues:owner(0):group(0):folder(69196):file(19822)
2024/06/04 08:42:24 items:count(5):size(34.94 GB)
2024/06/04 08:42:24 scanning:disk(/mnt/disk3):folder(nextcloud)
2024/06/04 08:42:24 issues:owner(0):group(0):folder(76):file(26)
2024/06/04 08:42:24 items:count(3):size(5.14 MB)
2024/06/04 08:42:24 scanning:disk(/mnt/disk4):folder(nextcloud)
2024/06/04 08:42:24 issues:not-available:(stat /mnt/disk4/nextcloud: no such file or directory)
2024/06/04 08:42:24 items:not-available:(stat /mnt/disk4/nextcloud: no such file or directory)
2024/06/04 08:42:24 scanning:disk(/mnt/disk5):folder(nextcloud)
2024/06/04 08:43:09 issues:owner(0):group(0):folder(25461):file(8091)
2024/06/04 08:43:09 items:count(8):size(5.34 GB)
2024/06/04 08:43:09 scanning:disk(/mnt/disk6):folder(nextcloud)
2024/06/04 08:43:11 issues:owner(0):group(0):folder(539):file(608)
2024/06/04 08:43:11 items:count(5):size(3.36 GB)
2024/06/04 08:43:11 scanning:disk(/mnt/cache):folder(nextcloud)
2024/06/04 08:43:11 issues:owner(0):group(0):folder(4):file(1)
2024/06/04 08:43:11 items:count(1):size(205.48 KB)

 

After that, the move phase worked as expected. 

[jbrodriguez]

20 hours ago, racingsquirrel said:

 

I am trying to move files from one pool to another but neither of the pools/disks are showing up in the plugin

 

i think i made a change to better identify pools, but maybe there's still something missing there

 

if you can share your `/var/local/emhttp/disks.ini` (remove any sensitive info you want), i'd like to take a look

[jbrodriguez]

5 hours ago, NickI said:

Usually the plan phase took around 30sec max. After looking at the log and waiting for the plan to complete, the planning phase took around 6 minutes to complete

yes, that seems odd

the biggest "culprits" seem to be disk1 and disk2, do the smart stats look good for them ?

[racingsquirrel]

Thanks for taking a look. 

The 2 pools not showing up are cache_downloads_hdd_3tb and cache_downloads_hdd_4tb

disks.ini

[MAM59]

I have a strange effect with unbalanced: source disks are missing (not offered):

If you compare the lists you see, Parity is missing (which is perfectly ok, makes no sense to copy from or to), but the "Samsung980Heat" and the "Force510Cruical" pool are missing too (And thats important, it contains all the private user files like music, homeshares and so on)!

It did not happen with the old "unbalance" (or I did not notice it?, maybe, getting old 🙂 )

 

How do I get the missing disk available for unbalanced???

 

I attach diagnostics in case you want to search deeper.

 

f-diagnostics-20240624-0655.zip

Edited June 24 by MAM59

[jbrodriguez]

hi, there's an issue with cache pools, the logic to detect them doesn't work appropriately at the moment, haven't really had time to look into it

 

in any case, if you can share your `/var/local/emhttp/disks.ini` (remove any sensitive info you want), i'll have more data to find a better solution as soon as i have some time to work on the plugin 🙏

[MAM59]

15 minutes ago, jbrodriguez said:

if you can share your `/var/local/emhttp/disks.ini` (remove any sensitive info you want)

No Problem (weren't it already included in the diagnostics ? )

 

Good that you already are aware of the problem, take your time, no need to hurry as long as it is not forgotten to be fixed someday.

 

(My non-educated guess would be that the names of the pools maybe too long for the plugin? ? ? )

disks.ini

Edited June 24 by MAM59

[jbrodriguez]

6 hours ago, MAM59 said:

(My non-educated guess would be that the names of the pools maybe too long for the plugin? ? ? )

yes, it's something about that 🤷‍♂️

[spicylizzard20]

Hello - Can someone advice why unbalanced won't detect enough free space to scatter, even when there's plenty? 

I'm trying to empty out an old drive, but as you can see in the attachments, unbalanced somehow does not detect enough free space on the other 3 drives. Any ideas? 

 

 

 

[jbrodriguez]

1 hour ago, spicylizzard20 said:

Can someone advice why unbalanced won't detect enough free space to scatter, even when there's plenty?

hi, instead of selecting /media, select the folders below: 4kmovies, for example

 

the plugin tries to move the immediate children of the selection, since 4kmovies is 7Tb, it doesn't find a disk with that free size

 

when you select 4kmovies itself, it will transfer each children folder individually, which will be certainly smaller

[spicylizzard20]

10 minutes ago, jbrodriguez said:

hi, instead of selecting /media, select the folders below: 4kmovies, for example

 

the plugin tries to move the immediate children of the selection, since 4kmovies is 7Tb, it doesn't find a disk with that free size

 

when you select 4kmovies itself, it will transfer each children folder individually, which will be certainly smaller

Ahh that did it!!  Thanks!

[jbrodriguez]

quick note, my heartfelt thanks to all of you sponsoring the app, i really appreciate it !!! 

[Remy4409]

Hi!

 

So basically, one of my drives is dying. Ordered a new one but in the meanwhile, I started moving stuff on other drives using Unbalanced.

 

I basically Moved a total of 2.2TB, but in the Main tab on Unraid, the used space on my drive only went down by about 1.5TB. I didn't think AT ALL about hardlinks created by radarr or sonarr, so now I have my Qbittorrent folder on the failing drive, and my Plex folder containing the same shows and movies on another drive. Did i fucked up everything? From what I understand it copied the whole file since it's another drive, hardlinks cannot work between 2 drives, I should have moved both at the same time. 

 

How can I came back from this? Can I put data back in any way to restore those links?

 

EDIT: Worse case scenario, I can just delete my stuff from the qBittorrent folder and it would solve it, but I'd like to avoid it if possible.

 

Thank you!

Edited July 23 by Remy4409

[_Shorty]

8 hours ago, Remy4409 said:

Hi!

 

So basically, one of my drives is dying. Ordered a new one but in the meanwhile, I started moving stuff on other drives using Unbalanced.

 

I basically Moved a total of 2.2TB, but in the Main tab on Unraid, the used space on my drive only went down by about 1.5TB. I didn't think AT ALL about hardlinks created by radarr or sonarr, so now I have my Qbittorrent folder on the failing drive, and my Plex folder containing the same shows and movies on another drive. Did i fucked up everything? From what I understand it copied the whole file since it's another drive, hardlinks cannot work between 2 drives, I should have moved both at the same time. 

 

How can I came back from this? Can I put data back in any way to restore those links?

 

EDIT: Worse case scenario, I can just delete my stuff from the qBittorrent folder and it would solve it, but I'd like to avoid it if possible.

 

Thank you!

No parity drives?  Don't need to do anything if you have parity drives until you put the new drive in.  Then you just do a rebuild.

[Xili]

hello (sorry for the google translation)

 

i used unbalanced a few years ago to reduce the size of the array, i followed this video without any problems:

 

today i try to do the same, but during the move, the files do not seem to be moved, but copied, that is to say that the original disk remains full. But above all the only target disk is full while it has room to receive everything, as if the hardlinks of radarr and sonarr were poorly recognized, and therefore were copied twice

 

do you have any idea of the origin of the problem?

Thanks

[AlanC]

On 6/25/2024 at 12:06 AM, jbrodriguez said:

yes, it's something about that 🤷‍♂️

I had the same issue, I had to remove numbers and underscores for the disk to show up.

[MAM59]

2 hours ago, AlanC said:

I had to remove numbers and underscores for the disk to show up.

Boy, that really hit the spot! Great Guessing!!!

Getting rid of the numbers suddenly made the missing disks pop up!

 

So @jbrodriguez should take a closer look to the part where disknames are read and scanned for non-alphabetical chars. I would guess he tries to find "disk[0-9]+" but forget to scan for "disk" at the beginning...

 

[jbrodriguez]

On 8/1/2024 at 2:00 AM, Xili said:

do you have any idea of the origin of the problem?

try and select some children folder(s) of the current folder you're selecting, see if that helps/solves it

[jbrodriguez]

2 hours ago, MAM59 said:

e tries to find "disk[0-9]+" but forget to scan for "disk" at the beginning

i strip spaces and numbers from a disk name and create a map to identify pool disks, but that is failing

 

not sure what you mean with disk at the beginning

 

i think what i was trying to merge disks with name Cache, Cache2, Cache3 and so on, for multi disk pools

 

not really sure what's the best way to approach this to be honest

 

[MAM59]

13 hours ago, jbrodriguez said:

not sure what you mean with disk at the beginning

My guess was that you were looking for array devices named disk1, disk2 and so on.

 

The filtered out disks here all have numbers in the middle of the name ("force501cruical", "samsung980heat"). Leaving out the numbers makes them appear again in unbalanced (but of course causes a lot of not-working-anymore links, so renaming is not really an option for me).

 

Your aggragation approach is not bad, but I guess, you only scan for digits and if found, consider them the end of the string and throw away letters that follow the digits. So I would check out the source for a regular expression with [0-9] and open end.

 

[MAM59]

To be more specific:

• "[0-9]*" looks for numbers anywhere (read: "an unspecified count of digits")
• "[0-9]*$" matches only numbers at the end

 

(whereas "at the end" also may be not very bright, because everybody could name a pool "TestPool34" where the number is just a number without any meaning.)

 

I'm not even really sure if it makes any sense to offer the subdisks of a pool seperately. I get the idea that it might be wanted to move off the data from a single disk of a pool, but this will work for very few pool types only and result into chaos for all others (try to unload a disk of a RAID1 mirrored pool). I guess the users wont understand what conditions must be met to allow such an action. And the rest will complain because it wont work.

 

[jbrodriguez]

On 8/4/2024 at 7:52 AM, MAM59 said:

I'm not even really sure if it makes any sense to offer the subdisks of a pool seperately

exactly, that's why i was aggregating the disks in a cache pool, in any case, i'll give it some more thought  

[MAM59]

5 hours ago, jbrodriguez said:

exactly, that's why i was aggregating the disks in a cache pool, in any case, i'll give it some more thought

If we agree that it is quite useless, why then aggragate at all ? the poolname itself is sufficient as source or target drive

 

[jbrodriguez]

13 hours ago, MAM59 said:

If we agree that it is quite useless, why then aggragate at all ? the poolname itself is sufficient as source or target drive

iirc, Unraid's web ui aggregates ?

 

also not sure what happens if someone chooses one of the "sub disks" as target or source 🤷‍♂️