[Plugin] unbalanced

[NLS]

 

 

 

So I am thinking of REALLY merging my top level folders as much as I can (I have a whole plan) after years of spill-overs.

 

You know what would take "unbalanced" to the next level?

Really automatically make and handle a plan of shifts to arrive at planned "end shape".

An unbalanced "2.0" would do it for a single folder.

In other words "I want this folder" (which could be a full share, if it fits) "in this disk".
But that destination disk has things that make it NOT fit.
Then the user will be able to select folders in the destination that doesn't want touched (moved).

Then the system will make a quick check if folder to move there + folders that user doesn't want to be touched, fit that disk.
If they don't allow the user to modify locked folders or select different disk or whatever.

If they fit, then supposed unbalanced 2.0 will decide the moves that need to be done OUT of the disk (keeping top levels "together" as much as possible) and in the end (OR IN SMALLER STEPS if there is not enough contiguous space - this is the whole point of needing such an app) reach the point where locked folders are still there, folder to be moved has moved there and everything that doesn't need to move out is still there and what needed to move out is on different disk (even on the source disk that the folder the user selected moved out of, that now has more space) as much kept together as possible.

 

This would be magic.

 

Let me give an example with three disks in a google sheet...

https://docs.google.com/spreadsheets/d/1GApW5mRgftP0FmUmFbbsO-nsn0ZjohbIjvO6FJz-y_g/edit?usp=sharing

 

Version 3.0 would make this for multiple folders (this in disk 1, that in disk 2, the other in disk 3 and so on) and make a whole plan of moves to reach that destination. Or even handle organized "spills" (like this in disk 1 and rest in disk 2... which would be "Videos" in the example).

 

I can possibly help analyzing the logic (maybe pseudo-code, my original background includes development - but have it "semi-forgotten" over the years, working in different fields of IT).

 

[jbrodriguez]

14 hours ago, NLS said:

Really automatically make and handle a plan of shifts to arrive at planned "end shape".

...

I can possibly help analyzing the logic (maybe pseudo-code, my original background includes development - but have it "semi-forgotten" over the years, working in different fields of IT).

yes, that sounds like a next gen unbalanced

 

it seems to require various considerations and probably a more intelligent transfer phase, currently all the logic happens on the plan phase.

 

if you can think of a pseudo code, it would help clarify this use case (i didn't have access to the google doc)

 

[NLS]

45 minutes ago, jbrodriguez said:

yes, that sounds like a next gen unbalanced

 

it seems to require various considerations and probably a more intelligent transfer phase, currently all the logic happens on the plan phase.

 

if you can think of a pseudo code, it would help clarify this use case (i didn't have access to the google doc)

 

 

Try this link?

https://docs.google.com/spreadsheets/d/e/2PACX-1vSyL6hHkrzSQwik_tpBfTc_6cEMogKQAKzpB5YUkuimqnq8d92hbUUcXQeiWIU1Kfszrq7vKuDPvTld/pubhtml

 

(else I need you PM me your gmail)

Edited January 31 by NLS

[JPilla415]

Forgive me if I am missing something painfully obvious, but the new Unbalanced UI is indicating the following:

 

Quote

You can find more details about which files have issues in the log file (/var/log/unbalanced.log).

 

However, when looking through /var/log via Krusader, I am not seeing any log file. The last modified date for /var/log is back in 2023 for me.

 

 

 

Appreciate the help!

Edited January 31 by JPilla415

[itimpi]

36 minutes ago, JPilla415 said:

However, when looking through /var/log via Krusader, I am not seeing any log file. The last modified date for /var/log is back in 2023 for me.

Do you have the volume mapping for the Krusader docker container set to allow that level of access on the Unraid host?   If not then you are looking at the location inside the docker container.

[JPilla415]

6 minutes ago, itimpi said:

Do you have the volume mapping for the Krusader docker container set to allow that level of access on the Unraid host?   If not then you are looking at the location inside the docker container.

Shoot! Didn't think about that. Thanks very much!

[jbrodriguez]

4 hours ago, NLS said:

 

Try this link?

https://docs.google.com/spreadsheets/d/e/2PACX-1vSyL6hHkrzSQwik_tpBfTc_6cEMogKQAKzpB5YUkuimqnq8d92hbUUcXQeiWIU1Kfszrq7vKuDPvTld/pubhtml

 

(else I need you PM me your gmail)

i can see the doc now

[wulperdinger]

Hello Mister jbrodriguez,

 

is it possible to change the blue color of the font in the scatter section in dark mode?

Since my eyes were made wrong, I have problems seeing blue and red letters, all they do is jiggle around a little bit, and it is very tedious for me to read them, unless I move my head very close to the monitor.

I couldn't find anything in the settings, so I was wondering if it is even possible for me to change the color of the font?

 

[Revan335]

Can you add the deprecation to fix common problems or Communication Applications to notify all Users that's are using the old Version about the new Version? @Squid

[jbrodriguez]

18 hours ago, wulperdinger said:

Hello Mister jbrodriguez,

 

is it possible to change the blue color of the font in the scatter section in dark mode?

Since my eyes were made wrong, I have problems seeing blue and red letters, all they do is jiggle around a little bit, and it is very tedious for me to read them, unless I move my head very close to the monitor.

I couldn't find anything in the settings, so I was wondering if it is even possible for me to change the color of the font?

 

hi, the short term solution is to use light mode (click on the moon on the top right)

contrast is tricky, i'll look into it 

[jbrodriguez]

On 1/30/2024 at 4:43 PM, NLS said:

...

Really automatically make and handle a plan of shifts to arrive at planned "end shape".

 

Let me give an example with three disks in a google sheet...

https://docs.google.com/spreadsheets/d/1GApW5mRgftP0FmUmFbbsO-nsn0ZjohbIjvO6FJz-y_g/edit?usp=sharing

...

 

i took a look at the doc, it reminded me of this function in excel that had to solve a constraint, it isn't the one for loan payment, but just can't remember what it was, maybe the one for loan to value ? (🤷‍♂️)

 

i also remembered that the function could potentially not find a solution

 

my take is that this functionality requires multiple passes, each pass consists of multiple data transfers (rsync commands) and a reevaluation after each pass

 

you can't calculate in advance, since it involves data transfers, and even though unbalanced calculates space based on sectors and there's a safety space, there's still overhead depending on how many files are involved (because of inodes and such)

 

this brings me back to the excel function, it's possible that after multiple passes, a solution still can't be found, that is, there's not enough space to move a whole share to a single drive, although a partial fill would probably be a good compromise (maybe?)

 

i can't tackle this, but there might be something there, a simpler version with a different logic that can be further reviewed

 

p.s.: i was wondering if concentrating a share on a single drive is a good practice, parity will save any failure, although it still feels a bit risky to me (considering no backups, etc)  

Edited February 2 by jbrodriguez

[NLS]

On 2/2/2024 at 2:07 PM, jbrodriguez said:

i took a look at the doc, it reminded me of this function in excel that had to solve a constraint, it isn't the one for loan payment, but just can't remember what it was, maybe the one for loan to value ? (🤷‍♂️)

 

i also remembered that the function could potentially not find a solution

 

my take is that this functionality requires multiple passes, each pass consists of multiple data transfers (rsync commands) and a reevaluation after each pass

 

you can't calculate in advance, since it involves data transfers, and even though unbalance calculates space based on sectors and there's a safety space, there's still overhead depending on how many files are involved (because of inodes and such)

 

this brings me back to the excel function, it's possible that after multiple passes, a solution still can't be found, that is, there's not enough space to move a whole share to a single drive, although a partial fill would probably be a good compromise (maybe?)

 

i can't tackle this, but there might be something there, a simpler version with a different logic that can be further reviewed

 

p.s.: i was wondering if concentrating a share on a single drive is a good practice, parity will save any failure, although it still feels a bit risky to me (considering no backups, etc)  

 

You can definitely mathematically know if you can have a solution.
You can calculate in advance, taking care of OS intricacies. All these things are documented.
The process might get just a bit more complex, but that is why we use computers, right?

Anyway, I am showing a "probable" way to go. Just that.

Now about your PS. I have at times thought of "why bother, just let everything mix" etc.
My take is that UNRAID's parity method (which is a type of RAID4), can have advantages to harness.
Knowing (roughly) what is in every disk is an advantage in case disks die OVER the number your parities can save.
If that (knock wood) ever happens I definitely prefer to know "I lost my MP3" or "I lost that piece of my emulation data" etc. instead of "I lost probably a piece of everything".
It is of course a philosophical matter in the end.
This also helps with local backups (I use that). I make sure that for things that are too important I daily sync some specific folders to two different disks (while using those disks also for other things).
If I didn't care about all that, I might just use a "normal" RAID5 solution.

Anyway, I hope all this discussion evolves to something interesting (a future plugin?) 

 

Edited February 3 by NLS

[FreakyBigFoot]

So this has been deprecated but there's a replacement? How do we get the new version?

[jbrodriguez]

15 hours ago, FreakyBigFoot said:

So this has been deprecated but there's a replacement? How do we get the new version?

the new plugin can be found in community apps, search for unbalanced, more details here 

 

[The_Doctor]

I might have missed it, but any way of adding in a un/select all on the targets? When using scatter on the old version, it would auto pick all targets but the one that was being used as the source. With large arrays it can be tedious to pick all targets. I understand that the auto select all could also be the opposite needs for some, so a select all /un select all, might be a good option?

[jbrodriguez]

30 minutes ago, The_Doctor said:

... so a select all /un select all, might be a good option?

yes, i had planned to add it, but it fell through the cracks , will implement it in an upcoming release

[touching-concept2664]

When I move with unbalance (from cache to disk3), unbalance does not delete the file it moved. There are no issue during plan and dry run. Because of that, I manually stop unbalance. Now I have duplicate files. 

 

Any advice? Why does move not deleting files it moved? Or it only delete at the end of operation? How to remove duplicate files?

 

UPDATE:
Tried on smaller number of files. Once finished, I can verify that the files has been moved(deleted in original location). I guess all good. Just have to do it in smaller number of files/folders

Edited February 15 by touching-concept2664
Fix it

[dubbleup99]

12 hours ago, touching-concept2664 said:

When I move with unbalance (from cache to disk3), unbalance does not delete the file it moved. There are no issue during plan and dry run. Because of that, I manually stop unbalance. Now I have duplicate files. 

 

Any advice? Why does move not deleting files it moved? Or it only delete at the end of operation? How to remove duplicate files?

 

UPDATE:
Tried on smaller number of files. Once finished, I can verify that the files has been moved(deleted in original location). I guess all good. Just have to do it in smaller number of files/folders

 

I've noticed that it also occurs for me when dealing with a large number of files. While it won't be a major inconvenience for me since I'm planning to wipe and encrypting my disk, I'm hopeful that this can be resolved in a future update.

[JonathanM]

1 hour ago, dubbleup99 said:

 

I've noticed that it also occurs for me when dealing with a large number of files. While it won't be a major inconvenience for me since I'm planning to wipe and encrypting my disk, I'm hopeful that this can be resolved in a future update.

When moving a large number of files, the deletion can take quite a long time, especially when dealing with a "mature" file system that has been in use with many additions and deletions over time. The entire copy occurs first, then after that is successfully completed, the deletion is started. If there is an error during the copy, or the process is interrupted, the deletion will not happen.

 

This tool is definitely not the fastest or safest option if you are planning to wipe a disk after it is empty. Manually copying (NOT moving) is quicker and safer, as you can verify each copy and do it over if the destination and source aren't identical. Once you are satisfied that all data has been copied successfully, reformatting the disk takes a tiny fraction of the time required to delete the files.

 

tldr: This behaviour is by design and not easily "fixable".

 

I'm not the author of this app, and possibly @jbrodriguez could improve the status reporting, so it's more clear when the copy part isn't completed successfully, or the deletion part is in progress so you don't kill it while it's still working.

[jbrodriguez]

23 hours ago, JonathanM said:

tldr: This behaviour is by design and not easily "fixable".

 

I'm not the author of this app, and possibly @jbrodriguez could improve the status reporting, so it's more clear when the copy part isn't completed successfully, or the deletion part is in progress so you don't kill it while it's still working.

correct, that's how the app works

 

if there are errors during the copy part, the app will avoid deleting those files, to allow the user to double check what was the issue.

 

the app uses rsync underneath, and rsync errors are not that user friendly, but in general they are related to incorrect date/time, permission issues, etc., unless there's something wrong with the source disk which surface i/o errors

 

the app runs an `rm -rf <folder>` command when deleting files and that is a discrete part of the operation, so yes, there's room for improvement in terms of reporting back to the user there

 

thanks for the detailed explanation @JonathanM 👍

 

[TwistedCarnivore]

Getting ready to upgrade unraid from 6.12.6 to 6.12.8. I ran the update assistant and got this error concerning unbalanced

Issue Found: unbalanced.plg is not known to Community Applications. Compatibility for this plugin CANNOT be determined and it may cause you issues.

Is this going to cause problems on the upgrade?

 

[jbrodriguez]

i haven't heard about any new requirements related to plugins in 6.12.8, @Squidis there any change needed on the plugin template ?

 

thanks for the heads up @TwistedCarnivore👍

[Squid]

12 minutes ago, jbrodriguez said:

i haven't heard about any new requirements related to plugins in 6.12.8, @Squidis there any change needed on the plugin template ?

 

thanks for the heads up @TwistedCarnivore👍

No.  There was a period of about half hour yesterday when every plugin got tagged that way

[TwistedCarnivore]

I ran the update assistant twice a little while ago and got the same results.  Removed the plugin and update assistant ran clean. Updating unraid now and will try and add unbalanced when it finishes.

[TwistedCarnivore]

unbalanced installs and runs fine after the upgrade. fix common problems does not show any issues after installing unbalanced. upgrade assistant still shows and error with unbalanced after the upgrade. Must be something in the upgrade assistant causing it to report that problem.
 

Disclaimer: This script is NOT definitive. There may be other issues with your server that will affect compatibility.

Current unRaid Version: 6.12.8 Upgrade unRaid Version: 6.12.8

NOTE: You are currently running the latest version of unRaid.


Checking for plugin updates
OK: All plugins up to date

Checking for plugin compatibility
Issue Found: unbalanced.plg is not known to Community Applications. Compatibility for this plugin CANNOT be determined and it may cause you issues.
OK: All plugins are compatible

Checking for extra parameters on emhttp
OK: emhttp command in /boot/config/go contains no extra parameters

Checking for disabled disks
OK: No disks are disabled

Checking installed RAM
OK: You have 4+ Gig of memory

Checking flash drive
OK: Flash drive is read/write

Checking for valid NETBIOS name
OK: NETBIOS server name is compliant.

Checking for ancient version of dynamix.plg
OK: Dynamix plugin not found

Checking for VM MediaDir / DomainDir set to be /mnt
OK: VM domain directory and ISO directory not set to be /mnt

Checking for mover logging enabled
OK: Mover logging not enabled

Checking for reserved name being used as a user share
OK: No user shares using reserved names were found

Checking for extra.cfg
OK: /boot/config/extra.cfg does not exist


Issues have been found with your server that may hinder the OS upgrade. You should rectify those problems before upgrading