[Plugin] unbalanced


Recommended Posts

On 5/8/2021 at 8:19 PM, jbrodriguez said:

completely slipped my mind, you need to also select at least a folder from the "source" disk

That was it!! Cheers for that!

 

I can now click "plan", and the planning starts, but I can still not click move or copy, or uncheck "dry run".

 

But this is probably because there's not enough free space left on the destination drive.  Strange, I thought there was.  Certainly when I count the cache drive too, but maybe Unbalance doesn't move to the cache drive?

 

Edit : indeed, it was because of too little space on the destination drive, because if I go further up the tree, effectively lowering the amount of data that needs to be transferred, I can move it.

Still, I do wonder why it doesn't just copy to the cache drive too?

 

Edit 2 : turns out there was a problem with some of the files on the drive, and Unbalance broke its teeth on those.  See here.  So no problem with Unbalance at all.  Thanks a lot for all the help!

Edited by BartDG
added extra info
Link to comment

Just wanted to ask this to be sure.

 

I have a a share: Music, that have files on all my drives.

I've added a new drive and I want to concentrate all the content from inside Music to it's drive.

 

I'm moving everything with unBALANCE. 

 

After everything is move to the new drive, can I go into each drive and rm -rf all the music folders that are empty?

 

I want to do this to avoid disks spinning up whenever I access the Music share from another device.

Link to comment

So, something weird is going on with UnBalance. Used it many times before and never had an issue. Running Unraid 6.8.3 and latest UnBalance (uninstalled and reinstalled to be sure).

 

I want to move 3 ISO files (about 6GB) from my ISO folder that for has everything on a single drive. Use UnBalance to select the 3 files then select the 2 drives the files can be moved to. 

 

Run Plan just fine, but Plan says only 1 file (sometimes 2) won't be moved because there isn't enough space. UnBalance sees the available space on the target drives at 3.5TB for each drive, yet says it can't move 2GB worth of files to either drive. 

 

Been driving me crazy the last couple of days. Any thoughts as to why UnBlance doesn't think there is enough space?

 

Link to comment
2 hours ago, jbrodriguez said:

yes, sounds odd.

 

check the logs for any clues, /boot/logs/unbalance.log and see what the planning stage outputs 

 

Not seeing anything in the log to indicate an issue. The log even shows 3.5TB available on each target disk, yet says it can't copy a 900MB file.

 

And for this run I actually stopped every single docker, VM, script, etc.

unbalance.log

Link to comment
30 minutes ago, jmbailey2000 said:

 

Not seeing anything in the log to indicate an issue. The log even shows 3.5TB available on each target disk, yet says it can't copy a 900MB file.

 

And for this run I actually stopped every single docker, VM, script, etc.

unbalance.log 6.22 kB · 1 download

yes, it's odd

 

can you run

 

stat -f /mnt/disk1
stat -f /mnt/disk2
stat -f /mnt/disk3
stat -f /mnt/disk4

 

and check if the blocksize is the same for all of them ?

 

it's the only thing I can think of right now, some discrepancy in block size, although it would still fit based on bytes, since you have so much free space

 

Link to comment
6 hours ago, jbrodriguez said:

yes, it's odd

 

can you run

 


stat -f /mnt/disk1
stat -f /mnt/disk2
stat -f /mnt/disk3
stat -f /mnt/disk4

 

and check if the blocksize is the same for all of them ?

 

it's the only thing I can think of right now, some discrepancy in block size, although it would still fit based on bytes, since you have so much free space

 

 

Looks correct. (I have no drive 3 at the moment)

 


root@SOL:~# stat -f /mnt/disk1
  File: "/mnt/disk1"
    ID: 90100000000 Namelen: 255     Type: xfs
Block size: 4096       Fundamental block size: 4096
Blocks: Total: 1952984905 Free: 588132960  Available: 588132960
Inodes: Total: 781402624  Free: 781059886
root@SOL:~#
root@SOL:~# stat -f /mnt/disk2
  File: "/mnt/disk2"
    ID: 90200000000 Namelen: 255     Type: xfs
Block size: 4096       Fundamental block size: 4096
Blocks: Total: 1952984905 Free: 854124105  Available: 854124105
Inodes: Total: 781402624  Free: 781174211
root@SOL:~#
root@SOL:~# stat -f /mnt/disk3
stat: cannot read file system information for '/mnt/disk3': No such file or directory
root@SOL:~#
root@SOL:~# stat -f /mnt/disk4
  File: "/mnt/disk4"
    ID: 90400000000 Namelen: 255     Type: xfs
Block size: 4096       Fundamental block size: 4096
Blocks: Total: 1952984905 Free: 854140697  Available: 854140697
Inodes: Total: 781402624  Free: 781115618
root@SOL:~#
 

 

Link to comment

so, this was too intriguing for me, i had a test for this scatter, so i tried your data

 

this is the result, it's quite low level info but the idea is there :)

 

this is the body of the test

 

func TestKnap(t *testing.T) {
	blockSize := int64(4096)
	fBlockSize := float64(blockSize)

	disk := &domain.Disk{
		ID:          2,
		Name:        "md2",
		Path:        "/mnt/disk2",
		Device:      "sdc",
		Free:        3498492334080,
		Size:        7999426170880,
		BlocksTotal: 1952984905,
		BlocksFree:  854124105,
		Serial:      "SAMSUNG_HD15",
		Status:      "DISK_OK",
	}

	items := make([]*domain.Item, 0)
	items = append(items,
		&domain.Item{Name: "proxmox-backup-server_1.1-1.iso", Size: 710651904, Path: "/mnt/disk1/isos/proxmox-backup-server_1.1-1.iso", BlocksUsed: int64(math.Ceil(float64(710651904) / fBlockSize))},
		&domain.Item{Name: "proxmox-mailgateway_6.4-1.iso", Size: 1024268288, Path: "/mnt/disk1/isos/proxmox-mailgateway_6.4-1.iso", BlocksUsed: int64(math.Ceil(float64(1024268288) / fBlockSize))},
		&domain.Item{Name: "proxmox-ve_6.4-1.iso", Size: 928993280, Path: "/mnt/disk1/isos/proxmox-ve_6.4-1.iso", BlocksUsed: int64(math.Ceil(float64(928993280) / fBlockSize))},
	)

	reserved := int64(1073741824)

	packer := algorithm.NewKnapsack(disk, items, reserved, blockSize)
	bin := packer.BestFit()

	assert.Nil(t, bin)
}

 

and below you can find the result of the test, it's failing because it expected no bin (group of files/folders) to be created , but it actually created one (just a quick way to test this :)

 

it created a bin with 3 items, sized 2663913472, which is the sum of the size of the 3 files

 

❯ go test -v
=== RUN   TestKnap
    unbalance_test.go:764: 
                Error Trace:    unbalance_test.go:764
                Error:          Expected nil, but got: &domain.Bin{Size:2663913472, Items:[]*domain.Item{(*domain.Item)(0xc000012c00), (*domain.Item)(0xc000012c40), (*domain.Item)(0xc000012bc0)}, BlocksUsed:650370}
                Test:           TestKnap
--- FAIL: TestKnap (0.00s)
FAIL
exit status 1
FAIL    unbalance       0.003s

 

i tried also using bytes instead of blocks and it worked too.

 

so, with regards to the issue you're having, there's something odd about how many blocksUsed are being reported for each of those files (unfortunately i'm not logging that piece of data)

 

i set blocksUsed to 0 for those items as another test and i got the exact same behavior you're currently experiencing, i was able to fully replicate it.

 

so, this is very strange because in your logs you have

 

I: 2021/05/15 09:10:45 planner.go:520: planner:array(4 disks):blockSize(4096)

 

blockSize set to 4096, which means, items should have a blockSize (planner.go:325)

 

		if blockSize > 0 {
			blocks = int64(math.Ceil(float64(size) / fBlockSize))
		} else {
			blocks = 0
		}

 

and this is all to say that i'm out of fresh ideas :)

 

somewhere, somehow, something is getting misreported, just can't figure out what/where yet 🤷‍♂️

 

 

 

 

 

 

 

 

Link to comment
6 hours ago, jbrodriguez said:

so, this was too intriguing for me, i had a test for this scatter, so i tried your data

 

somewhere, somehow, something is getting misreported, just can't figure out what/where yet 🤷‍♂️

 

 

 

Glad to see I'm not the only one that has little things get stuck in my brain and drive me nuts!!

 

I did try something else (no difference in the end result) where I actually shutdown all dockers and the docker service itself. Now, maybe it was doing this before, but this time when I tried it, I did notice that disk 2's free space of 3.5TB was not highlighted blue like the others. But, it sometimes is highlighted blue like the others so assuming it is a cosmetic thing. 

 

Link to comment

I might be a little confused as to how this plugin works.  I'd essentially like to move some stuff from a drive that's nearing capacity to 3 other drives.  However, when I attempt to run this, it just puts it all on one drive, instead of "Scattering" it across the drives.  Any advice?

Link to comment

Thank you for this wonderful software it's like magic.

 

I've a minor problem with port number. Using default port number (6237), when I click on "Open Web UI" link, it tries to open port 6238. If I choose 6238 as port for UI, it tries to open 6239 and so on. I have to manually type port number in browser which is not difficult but just annoying. Unbalance 5.6.4 version.


Regards

Link to comment
On 5/25/2021 at 10:34 AM, Coolsaber57 said:

I might be a little confused as to how this plugin works.  I'd essentially like to move some stuff from a drive that's nearing capacity to 3 other drives.  However, when I attempt to run this, it just puts it all on one drive, instead of "Scattering" it across the drives.  Any advice?

It doesn't "balance" data across drives, it uses the first drive with enough space, if not enough, uses the second drive with enough space and so on

Link to comment
On 5/26/2021 at 2:15 AM, denisvic said:

Thank you for this wonderful software it's like magic.

 

I've a minor problem with port number. Using default port number (6237), when I click on "Open Web UI" link, it tries to open port 6238. If I choose 6238 as port for UI, it tries to open 6239 and so on. I have to manually type port number in browser which is not difficult but just annoying. Unbalance 5.6.4 version.


Regards

I think it has something to do with SSL vs no SSL, I'll take a look

Link to comment
On 5/27/2021 at 7:43 AM, jbrodriguez said:

It doesn't "balance" data across drives, it uses the first drive with enough space, if not enough, uses the second drive with enough space and so on

 

I just installed this hoping to accomplish what sounds similar to him. I am a bit confused now, though. Why it is called scatter, if it does not scatter the data across the target drives? If it just moves it from one full drive to one lesser full drive, that sounds more like moving than scattering?

 

Is there not a way to take a drive that is more full than others and scatter the data between the drives until they each are about equal? I had one single drive, now I have 5, I was hoping to take the data from the first drive and spread it out across all of them without having to do it by hand, moving files around manually.

Link to comment

Replaced my old post as I solved the puzzles.  Instead a little bit of feedback from a noob to this plugin.

 

First - I love the clear simplicity of this plugin.  Thanks.  Works great for a noob to UNRAID, and clearly lets me see which disks data is moving from.

 

1\ The GUI does not seem to like Vivaldi browser.  A Chromium based browser.  Had to swap to Firefox to get it to run correctly.  Maybe my security settings a bit high?

 

2\ The pink text that pops up for a few seconds complaining about file rights is a little confusing.  Not sure why Unbalance is worrying that folders were 770 instead of 777.  Still worked fine for me.

 

3\ Why is there no security on this plugin?  Anyone on my network could run this without permission checks.  Means I'll need to keep it turned off when not in use.

 

Edited by Batter Pudding
Link to comment
  • jbrodriguez changed the title to [Plugin] unbalanced

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.