Thanks for building and maintaining this great tool.

 

Does unbalanced handle/maintain hardlink?

 

I have trash folder setup and use *rr for hardlinked file management.

 

Recently did a Scatter job to clear a drive, but afterwards my non-hardlinked file number jumped by a lot. 

 

Since I moved many files, it's unclear to me if no hard links are preserved or just partially.

 

Currently using jdupes to fix, but wonder if I did anything wrong, or is it expected that hardlinks will be broken.

 

Thanks!

18 minutes ago, StylishQuoter said:

Does unbalanced handle/maintain hardlink?

not by default, it uses rsync internally which does support hardlinks, but you need to add the flag in the settings (custom rsync flags).

 

having said that, users have reported some issues with hardlinks, unraid shares (can't remember the details)

 

my suggestion, use another tool if you have hardlinks: i don't use them so i can't validate/test them, there have been report of issues (similar to yours) and finally, it's not a good idea to change the rsync default flags that unbalanced uses (i may remove the custom flag setting altogether)

HI

 

I have gone to update Unraid today 24 Oct 2024 and the Update Assistant Error on the Plugin

ANything i need to do or be aware of ? 

Issue Found: unbalanced.plg is not known to Community Applications. Compatibility for this plugin CANNOT be determined and it may cause you issues.

2 hours ago, OsoPolar said:

ANything i need to do or be aware of ? 

Issue Found: unbalanced.plg is not known to Community Applications. Compatibility for this plugin CANNOT be determined and it may cause you issues.

no issue that im aware of.

 

it's reported for unbalanced or for unbalance (the previous name of the plugin) ?

 

 

 

23 hours ago, OsoPolar said:

HI

 

I have gone to update Unraid today 24 Oct 2024 and the Update Assistant Error on the Plugin

ANything i need to do or be aware of ? 

Issue Found: unbalanced.plg is not known to Community Applications. Compatibility for this plugin CANNOT be determined and it may cause you issues.

 

21 hours ago, jbrodriguez said:

no issue that im aware of.

 

it's reported for unbalanced or for unbalance (the previous name of the plugin) ?

 

 

 

I was preparing for an upgrade this morning and saw this message as well, which led me to find this post.

I ran the Update Assistant tests several times.  Sometimes it returned this message, sometimes it didn't.

I did also notice that the Community Applications had some issues when attempting to check for plugin updates.

 

Based on this, I believe this is a false reporting.

 

With that being said, some additional eyes on it would be appreciated.

3 minutes ago, Seregur said:

Based on this, I believe this is a false reporting.

 

With that being said, some additional eyes on it would be appreciated.

thanks, if Community App is having some issues (most likely upstream though), it sounds like it might be temporary 

 

if it still continues happening, i'll ping squid

Hi, I'm trying to use Unbalanced to empty a drive before dropping it from the array to change the file system on it then add it back in. I've installed unbalanced and been able to this for a handful of drives, but it stopped working. Now when I go to access the unbalanced UI, I can pick my source disk, folders to move, destination drive, and target disks. When I then select move (dry run deselected), all the content in the columns disappeared and I have a spinning wheel up top. It's been doing that so long I can't tell if it's broken or stalled or what's up. The logs page is inaccessible and I don't see the read/write counters on my drives changing, so I don't think it's doing anything. I tried uninstalling/reinstalling it, but it made no difference. Any suggestions?

 

 

 

8 hours ago, poshmick907ak said:

Hi, I'm trying to use Unbalanced to empty a drive before dropping it from the array to change the file system on it then add it back in. I've installed unbalanced and been able to this for a handful of drives, but it stopped working. Now when I go to access the unbalanced UI, I can pick my source disk, folders to move, destination drive, and target disks. When I then select move (dry run deselected), all the content in the columns disappeared and I have a spinning wheel up top. It's been doing that so long I can't tell if it's broken or stalled or what's up. The logs page is inaccessible and I don't see the read/write counters on my drives changing, so I don't think it's doing anything. I tried uninstalling/reinstalling it, but it made no difference. Any suggestions?

 

 

 

Do you have any parity drives?

Yes, I have one parity drive active. I could try removing it before shuffling data then re-add it after the data shuffle is done and let it rebuild. Would that help?

23 minutes ago, poshmick907ak said:

Yes, I have one parity drive active. I could try removing it before shuffling data then re-add it after the data shuffle is done and let it rebuild. Would that help?

Actually, forget I asked.  I was going to say can't you just reformat it and then let parity rebuild the files, but I guess that would also rebuild the current filesystem type. Heh.  Sorry about that.

Any ideas as to why Unbalanced is doing what it's doing? If I can't get it to move the data off the drives so I can update the file system one drive at a time, I don't know what to do other than to trash the array then rebuild it with the file system I want before restoring all the data from my backup server.

9 hours ago, poshmick907ak said:

empty a drive before dropping it from the array to change the file system on it then add it back in

You don't need (or want) to drop it from the array. After you get the data moved off, just reformat it in the array to the new filesystem.

 

If you actually drop it from the array, you will have to rebuild parity, and then adding it back will require clearing.

10 hours ago, poshmick907ak said:

Hi, I'm trying to use Unbalanced to empty a drive before dropping it from the array to change the file system on it then add it back in. I've installed unbalanced and been able to this for a handful of drives, but it stopped working. Now when I go to access the unbalanced UI, I can pick my source disk, folders to move, destination drive, and target disks. When I then select move (dry run deselected), all the content in the columns disappeared and I have a spinning wheel up top. It's been doing that so long I can't tell if it's broken or stalled or what's up. The logs page is inaccessible and I don't see the read/write counters on my drives changing, so I don't think it's doing anything. I tried uninstalling/reinstalling it, but it made no difference. Any suggestions?

 

 

 

 

I know that's why I'm really trying to avoid that and would prefer to just move the data off, but Unbalanced isn't working.

Edited November 8, 20241 yr by poshmick907ak
typo

11 hours ago, poshmick907ak said:

It's been doing that so long I can't tell if it's broken or stalled or what's up. The logs page is inaccessible

it seems the frontend silently crashed or it's really just stuck, try these 2 things

- if you're using chrome, open the developer console (View -> Developer -> Javascript Console), check if there are any errors there

- run `tail -f /var/log/unbalanced.log` on the unraid terminal and check what it prints out

hopefully one or both of them give some clue 

I was able to find errors in both. Here's what I found:

 

 

 

have to remove some of those frontend logs, not sure why it's logging so much 😀

 

the error is about the operations history, not sure what the root cause is, but try renaming the history file and see how it goes

`mv /boot/config/plugins/unbalanced/unbalanced.hist /boot/config/plugins/unbalanced/unbalanced.bak` then restart unbalanced

 

can you also check that file, look for anything unusual ? it's a bit dense, id like to take a look if you can send it over (dm?)

I can confirm on reinstalling Unbalanced, I see the below entry in its log before I even make a fresh attempt to clear a drive, so the history file seems to be persistent between uninstall/reinstalls:

 

I tried running the command you suggested from the Unraid terminal to clear the old file out of use by renaming it, but the command didn't work. Forgive my ignorance, do plugins in Unraid have their own terminal that is separate from Unraid so they operate somewhat in a sandbox environment or do I need to cd to another directory to run the command? I tried running both "dir" and ls" from the terminal, but don't see any directories listed. Linux isn't my strong suit, so thanks for the patience and help with this one

Edited November 8, 20241 yr by poshmick907ak
Forgot to include screen capture

Actually, I was able to figure out how to get to that directory, but there was no unblanced.hist file there to move or rename.

17 hours ago, poshmick907ak said:

so the history file seems to be persistent between uninstall/reinstalls:

it's ok, the message is saying that there is no unbalanced.hist file, that's also why the other command i mentioned didn't work for you.

 

does the app work now or it still "crashes" ?

It's working now.... very weird, I don't know what changed, but thank you for the help!

Hi Juan! I appreciate your work on this plugin and have added you to my donate list. I tend to do my donations every year between Christmas and New Years Day so you'll be getting one this year. Thanks!

 

Now on to my request: it's a repeat of the requested addition of being able to use unBalanced with drives mounted under Unassigned Devices (UD). My case is probably a little different than most that have requested this, so I understand entirely if it's too much effort to support only a few users. I've read/skimmed through the entire thread over the last 2 days, making some notes along the way that I'll discuss below.

 

My power costs escalated rapidly in summer 2023, so that was the impetus for me to try and combine my two larger unRAID instances into one. I use the mergerfs plugin to create a pool of 33 disks that are all mounted under UD. 28 of them were from the array on my 2nd unRAID system. All 33 are single partition disks using the XFS filesystem. The other 5 were mounted under UD or as single disk pools on the other system.

 

I'm planning to reduce the size of the mergerfs pool to 28 disks. This is advance planning for when unRAID ships with the 'multiple arrays as pools' feature in a future 7.x release. My long term goal would be ending up with 4 pools of 14 data devices and 1 parity disk (60 disks total), using the unRAID parity system. We have no timeline on when the feature will be implemented but it never hurts to prep early.

 

I can manually do the moves of all the data so that the 5 disks I plan to remove will contain data that really should be on my backup unRAID instance. Those 5 disks will be added to the backup unRAID once I complete this. Moving my data around would be so much easier if I could use unBalanced.

 

To my understanding, you use /var/local/emhttp/disks.ini for the standard unRAID array and pools. Alas disks.ini doesn't contain any info about drives mounted under UD. DLandon provides two files as part of UD that can help with this. They are /var/state/unassigned.devices/unassigned.devices.ini and /var/state/unassigned.devices/unassigned.devices.json. I can provide these files to you after I obfuscate/remove any private data they contain.

 

Let me (and others that have requested this) know if this is something you have the time to look at and if there's any other data you need. I'd be happy to test things for you. Regardless, thanks again for your work on unBalance(d) over the last 5+ years.

3 hours ago, AgentXXL said:

I appreciate your work on this plugin and have added you to my donate list. I tend to do my donations every year between Christmas and New Years Day so you'll be getting one this year. Thanks!

thank you !!! i really appreciate all funding sponsorship !

 

this also goes to all the sponsors that keep sending me some funds, i won't buy a lambo off of them 🤣, however they're very meaningful to me, thanks !!!

 

wrt to unassigned devices, after itimpi pointed me to look at `/mnt` to support disk pools, i think it should be easier ... IF unassigned devices appear under `/mnt`

 

@AgentXXLif you run

df --block-size=1 /mnt/*

on your terminal, the unassigned devices show up there ?

On 11/18/2024 at 1:06 PM, jbrodriguez said:

wrt to unassigned devices, after itimpi pointed me to look at `/mnt` to support disk pools, i think it should be easier ... IF unassigned devices appear under `/mnt`

 

@AgentXXLif you run

df --block-size=1 /mnt/*

on your terminal, the unassigned devices show up there ?

Actually, UD mounted drives appear under /mnt/disks. For my mergerfs pool off 33 disks, I gave them all the same root name + disk#, i.e. DLPool01 through DLPool33. Here's the output of the df command (I did it with 1M as you need a modifier for --block-size):

 

root@AnimNAS:~# df --block-size=1M /mnt/disks/*
Filesystem     1M-blocks     Used Available Use% Mounted on
/dev/sdbn1      15257610 14741485    516126  97% /mnt/disks/DLPool01
/dev/sdaq1      15257610 14745048    512563  97% /mnt/disks/DLPool02
/dev/sdar1      15257610 14741624    515987  97% /mnt/disks/DLPool03
/dev/sdas1      15257610 14743789    513822  97% /mnt/disks/DLPool04
/dev/sdat1      15257610 14730938    526673  97% /mnt/disks/DLPool05
/dev/sdau1      15257610 15004264    253347  99% /mnt/disks/DLPool06
/dev/sdav1      15257610 14961329    296282  99% /mnt/disks/DLPool07
/dev/sdaw1      15257610 14741135    516476  97% /mnt/disks/DLPool08
/dev/sdax1      15257610 14758009    499602  97% /mnt/disks/DLPool09
/dev/sday1      15257610 14733555    524056  97% /mnt/disks/DLPool10
/dev/sdaz1       9535498  9014963    520536  95% /mnt/disks/DLPool11
/dev/sdba1       9535673  9020499    515175  95% /mnt/disks/DLPool12
/dev/sdbb1       9535673  9015630    520044  95% /mnt/disks/DLPool13
/dev/sdbc1       9535498  9040554    494945  95% /mnt/disks/DLPool14
/dev/sdbd1       9535498  9254896    280602  98% /mnt/disks/DLPool15
/dev/sdbe1       9535498  9018229    517270  95% /mnt/disks/DLPool16
/dev/sdbf1       9535498  9018114    517385  95% /mnt/disks/DLPool17
/dev/sdbg1       9537071  9013280    523791  95% /mnt/disks/DLPool18
/dev/sdbh1       9535673  9013193    522481  95% /mnt/disks/DLPool19
/dev/sdbi1       9535673  9100551    435123  96% /mnt/disks/DLPool20
/dev/sdbj1       9535498  9018443    517056  95% /mnt/disks/DLPool21
/dev/sdbk1       9535498  9018154    517345  95% /mnt/disks/DLPool22
/dev/sdbl1       9536139  9015517    520623  95% /mnt/disks/DLPool23
/dev/sdbm1       9535673  9018434    517240  95% /mnt/disks/DLPool24
/dev/sdb1        9535498  9020378    515121  95% /mnt/disks/DLPool25
/dev/sdc1        9535498  9016059    519440  95% /mnt/disks/DLPool26
/dev/sdv1       19072010 18555335    516676  98% /mnt/disks/DLPool27
/dev/sdj1       15257610 14789615    467996  97% /mnt/disks/DLPool28
/dev/sdf1        9535498  9020017    515482  95% /mnt/disks/DLPool29
/dev/sdg1        9535498  9020318    515181  95% /mnt/disks/DLPool30
/dev/sdh1        9535498  9019047    516452  95% /mnt/disks/DLPool31
/dev/sdi1        9535498  9017011    518488  95% /mnt/disks/DLPool32
/dev/sdan1       9535498  9015695    519804  95% /mnt/disks/DLPool33
root@AnimNAS:~# 

 

I haven't redacted those .ini or .json files yet but can do if they will help. Thanks! This would help greatly in allowing me to sort things in the limited space left on each drive.

Edited November 20, 20241 yr by AgentXXL

22 hours ago, AgentXXL said:

Actually, UD mounted drives appear under /mnt/disks.

man, i hoped they were straight under /mnt, not sure how much complexity this can add, i'd certainly need to have 2 passes

 

22 hours ago, AgentXXL said:

I haven't redacted those .ini or .json files yet but can do if they will help

that would be very helpful, i'd need to recreate the same disk structure as regular array disks, which requires

		disk.ID, _ = strconv.ParseUint(section.Key("idx").String(), 10, 64) // 1
		disk.Name = diskName                                                // disk1, cache
		disk.Path = "/mnt/" + disk.Name                                     // /mnt/disk1, /mnt/cache
		disk.Device = section.Key("device").String()                        // sdp
		disk.Type = diskType                                                // Flash, Parity, Data, Cache
		disk.FsType = section.Key("fsType").String()                        // xfs, reiserfs, btrfs
		disk.Free = free[disk.Path]
		disk.Size = size[disk.Path]
		disk.Serial = section.Key("id").String() // WDC_WDXXXXX-XXXXXXX_WD-WMXXXXXXXX
		disk.Status = diskStatus       

and some other fields that i calculate on the spot

31 minutes ago, jbrodriguez said:

that would be very helpful, i'd need to recreate the same disk structure as regular array disks, which requires

		disk.ID, _ = strconv.ParseUint(section.Key("idx").String(), 10, 64) // 1
		disk.Name = diskName                                                // disk1, cache
		disk.Path = "/mnt/" + disk.Name                                     // /mnt/disk1, /mnt/cache
		disk.Device = section.Key("device").String()                        // sdp
		disk.Type = diskType                                                // Flash, Parity, Data, Cache
		disk.FsType = section.Key("fsType").String()                        // xfs, reiserfs, btrfs
		disk.Free = free[disk.Path]
		disk.Size = size[disk.Path]
		disk.Serial = section.Key("id").String() // WDC_WDXXXXX-XXXXXXX_WD-WMXXXXXXXX
		disk.Status = diskStatus       

and some other fields that i calculate on the spot

 

OK, redacted 3 entries from both files - all I removed was the 8 character serial number for each and replaced with XXXXXXXX. All drives show the same info so hopefully 3 should be enough. I don't see exact matches to the fields you mention, but it looks like it's all there, just not using the same variable names.

 

As it's a pretty rare request, I'll understand if it's too much work to get UD mounted drives included in unBalanced. Thanks for taking a look at it regardless.

unassigned.devices-redacted-small.ini unassigned.devices-redacted-small.json