[Plugin] unbalanced

[jbrodriguez]

19 hours ago, Gaertner said:

The files with the CKSUM error then have a current date (changed) on the reading hard drive

definitely weird, i can't think of a way that rsync would modify a filestamp while reading a file, unbalanced eventually deletes files on the source disk (if doing a move), but if there's an error like the one you posted ("operation flagged" in the logs), then it won't touch the file either

 

it would seem it's a hardware issue, i/o errors on a disk are somewhat difficult to surface, however im not an expert as to suggest ways to test/confirm 🤷‍♂️

[Gaertner]

If there is a hardware problem then it must be the ASM1166 card. So far the hard drives connected to the card have been affected. It's just strange that I've only had the errors when I moved files with unbalanced.
What errors/causes are there why it suddenly assigns the file with the current date during a read process? Problems with special characters? Access rights?

[jbrodriguez]

1 hour ago, Gaertner said:

What errors/causes are there why it suddenly assigns the file with the current date during a read process? Problems with special characters? Access rights?

i've never seen rsync change the timestamp on a file it's reading, can't really help there 🤷‍♂️

[Ezekial66]

Just curious, is it possible to run multiple instance of unbalanced in a certain scenario?

 

ie, I have 3x8TB disks in my array I would like to empty onto 3 other 16TB disks that are near empty. There is currently no parity. In this instance would in be possible to run 3 separate instances of unbalanced, moving the entirety of contents from say:

disk 14 (8TB) --> disk 3 (16TB)

disk 15 (8TB) --> disk 4 (16TB)

disk 16 (8TB) --> disk 5 (16TB)

 

so with no parity and no cross over between disks, it would essentially be a direct transfer from disk to disk.

[JonathanM]

1 hour ago, Ezekial66 said:

direct transfer from disk to disk

Why not just use the dynamix file manager plugin?

[Ezekial66]

36 minutes ago, JonathanM said:

Why not just use the dynamix file manager plugin?

In all honesty...because when I asked about disk to disk transfers I was recommended this plugin and forgot Dynamix FM existed lol XD

[Ezekial66]

Wow ya @JonathanM this is actually stupid fast compared to unbalanced. I'm not sure why but with both unbalanced and file manager going alongside each other, unbalanced has been going for 7 hours and only 50% done. File Manager has been going for barely an hour and is 75% done....

[JonathanM]

4 minutes ago, Ezekial66 said:

unbalanced has been going for 7 hours

Whatever you do, don't cancel the unbalanced run unless you are fine with it leaving the files behind on the source disk.

 

Personally I prefer to copy and compare, then delete when I'm sure the copy is accurate when I can, moving is too final for me.

[Ezekial66]

2 minutes ago, JonathanM said:

Whatever you do, don't cancel the unbalanced run unless you are fine with it leaving the files behind on the source disk.

 

Personally I prefer to copy and compare, then delete when I'm sure the copy is accurate when I can, moving is too final for me.

That's totally fair I can understand that. everything in these disks media and is categorized through Sonarr and Radarr so if anything goes missing I'll see it there. 

 

And no the unbalanced transfer is still going I'm going to let it finish, I was just surprised by the speed difference between the two when I can see the disk speeds in the Main Tab are pretty similar. Not sure what makes FM so much faster.

[methanoid]

On 8/15/2024 at 9:01 PM, jbrodriguez said:

a little too advanced for what unbalanced can support atm

 

i really want to implement pluggable algorithms, that is, some lua code that defines selection of source, selection of targets, planning and transferring, while the ui is taken care of by unbalanced, gather/scatter would be the first 2

 

it's just an idea, hopefully some time down the line i can flesh it out a bit more

 

I was thinking again about the desire to have all a single TV show on same HDD and how to "logically" do it. ... was gonna ask my son (coder) to do a bash script for me but thought I'd share here in case some clever person is good at bash scripts :-)

 

Define number of Disks (N)
Define minimum HDD space in GB (SPACE)
For X = 1 to N (x is HDD number)
For each directory in /mnt/diskX/data/tv/DIR)
Check size of directory DIR
Check size of (/mnt/user/data/media/TV/DIR) [whole across all HDDs)
If different then...(cos all data isnt on same HDD)....
For each HDD except current...
Check size of /mnt/diskY/data/tv/DIR 
If enough HDD space on diskX 
mv /mnt/diskY/data/tv/DIR to /mnt/diskX/data/tv/DIR

Until done

 

[Cessquill]

1 hour ago, methanoid said:

 

I was thinking again about the desire to have all a single TV show on same HDD

Apologies if I've missed the start of this conversation, but the "Split Level" setting when creating a share is exactly for this purpose.  If you haven't done already, set it up now on your shares to solve it going forward.

 

I don't know how much data you've got, or how patient you are, but I'd be tempted to set up your shares with the correct split level, manually move each show in turn onto your cache (via Midnight Commander, Krusader, command line, other file manager, etc.), and then have Mover put it back on your drives properly.  Or tidy them manually by looking at the TV share and identifying where a show spans disks and manually moving it.

 

I haven't fully thought this through, and it could be beyond how much time you want to spend (I tend to chip away at these sorts of jobs sending a copy/move operation off every now and then), but set the split level up correctly so that everything going forward is right.

 

Hope this helps, apologies if not.

[methanoid]

4 hours ago, Cessquill said:

Apologies if I've missed the start of this conversation, but the "Split Level" setting when creating a share is exactly for this purpose.  If you haven't done already, set it up now on your shares to solve it going forward.

 

I don't know how much data you've got, or how patient you are, but I'd be tempted to set up your shares with the correct split level, manually move each show in turn onto your cache (via Midnight Commander, Krusader, command line, other file manager, etc.), and then have Mover put it back on your drives properly.  Or tidy them manually by looking at the TV share and identifying where a show spans disks and manually moving it.

 

I haven't fully thought this through, and it could be beyond how much time you want to spend (I tend to chip away at these sorts of jobs sending a copy/move operation off every now and then), but set the split level up correctly so that everything going forward is right.

 

Hope this helps, apologies if not.

 

Thanks, I have unmanic putting my whole library in order and H265 so there will appx 60-70TB to "sort" so not gonna give my cache drive silly amounts of writes to sort it for me. I do have Split Level set but sometimes when a disk is full and a show gets "upgraded" the files end up elsewhere... and I prefer "Fill" HDDs

[[email protected]]

The initial speed was about 120M, but now the speed is almost non-existent. It seems that the previous version did not have this problem. Please forgive my English

[MAM59]

2 minutes ago, [email protected] said:

but now the speed is almost non-existent.

Make sure the "to be moved" files are not in use by any other process (known evildoes are "Plex" or something for instance), else rsync will stop and wait for the file to become free again.

This wait can last forever...

 

[[email protected]]

33 minutes ago, MAM59 said:

确保“要移动”的文件没有被任何其他进程使用（例如已知的恶意进程“Plex”或类似进程），否则 rsync 将停止并等待文件再次变为空闲。

这种等待可能会持续永远……

 

Got it, I'm observing it. Thank you!!!!!

[[email protected]]

I'm not moving now, I'm copying, right?

[[email protected]]

38 minutes ago, MAM59 said:

确保“要移动”的文件没有被任何其他进程使用（例如已知的恶意进程“Plex”或类似进程），否则 rsync 将停止并等待文件再次变为空闲。

这种等待可能会持续永远……

 

I didn't understand earlier that there is a related anime, but the anime hasn't been reading the screenshot of Leverage. Does this also have an impact?
I have now closed all Docker files associated with the TV directory, but the speed has not responded yet!!

[[email protected]]

Closing the associated Docker does not result in any changes, but the speed is still incorrect

[MAM59]

your dockers look still be running? stop them and stop the docker service (and the same for VMs)

Speed will not "jump" upward, it may take several minutes to see improvements

 

[[email protected]]

All have been closed,

[[email protected]]

 

我认为这是 WA 太高的问题

Edited August 30 by [email protected]

[MAM59]

if you run a copy job, its normal, that WA will raise and raise. First the (RAM) Buffers get filled, then Data is written directly to the drives. And because writing will be slow, WA (wait) cycles need to be added more and more.

You can pause unbalanced and wait a period. WA should drop to 0 again then.

Unpausing will copy fast again first (buffers!) then slow down again more and more.

The slower your drives, the faster (and longer) it will happen.

 

[[email protected]]

The problem is that the issue has not been resolved yet, and it is still slow. The files are being copied, but the time speed is only about 10M. The speed is not good now and it will take a long time

[jbrodriguez]

it also depends on your source disks, if they have some hardware issue, copying will also be slow

but mostly what mam59 mentioned, if you have other services reading from the source and/or writing to the source/target disks, it will result in a slower operation and unpredictable results, if writing to the same disks involved in the operation

[Ezekial66]

Apologies if some of this has been mentioned before, I didn’t find much that aligned with my personal use case so thought I would re-ask.
 

I’ve been focusing on getting my array back up to snuff after swapping out 5 disks to larger and newer ones. I wound up using the File Manager plugin as it seemed to work better for my use case here. Now that I’ve almost got everything transferred to the new disks and the new array built (still a little bit of transferring between disks left to go) I’m considering using something like ‘unbalanced’ to organize the disks now that everything is kinda all over the place. 

I know that the ‘gather’ function of unbalanced will do what I need, I just wanted to confirm some details for those that have used it, or if there’s a better way to do this. 
 

-Will unbalanced also more external subs as well as mkv files and folders when reorganizing? 

-Will old folders be left behind in disks even though the contents have been reorganized?

-When I am looking to reorganize my TV shows (1,400 series, 200TB) gather seems to only function on a disk to disk basis, so how would I reorganize these files across multiple disks?

Thanks all for your input.