[Plugin] unbalanced

[Wynter]

On 3/3/2024 at 12:11 PM, Starfox said:

Thanks - no yellow marked command for the operation no (And hence I can't see any button to delete the folders).

 

Tried to do the second run (Is that really the workflow here?). Unbalanced got stuck in plan mode. Refreshed and tried again. Would only allow me to do one folder at a time (Kept saying destination was full which given it was just an empty folder obvious was not the case). Got there in the end. Thanks

I have the same issue when using scatter to move folders with files.  The folders and files are moved leaving empty folders on the source disk.  I would then have to run the scatter again and again where it will delete the folders one at a time even though I've selected many.  History shows the transfers as complete without errors...

[MAM59]

1 hour ago, Wynter said:

The folders and files are moved leaving empty folders on the source disk. 

Yeah, this "feature" already existed in the previous plugin "unbalance". But it was sufficient to re-run the whole job once more to get rid of them.

[HellScytheX]

UPDATE: 

Was checking in periodically without adding any new files, and noticed that the total array capacity number went up from 6.57 TB -> 6.58 TB. 

I then restarted the 1 Transmission docker which would have been associated with that 1 folder, and after a few minutes it does look like the space was freed back up to 6.60TB, with the source disk in question freeing up the space to the expected amount. 

So looks like the deletion of the files from the source disks was held up by the docker in this case, but I have no way of seeing the held up files in the first place.

Unfortunately it still doesn't explain my first case of 130GB, since those files were not associated with any applications at the time, and also wasn't restored even after restarting the array or rebooting the server. But I'll assume it's all ok for now and will restart/disable any applications that may be associated with the files that I'm moving in question going forward to hopefully resolve the space problems I get in the future.

 

I am still curious where the 27.0GB number comes from in the unbalanced drive selection preview, but it's not something critical anymore since I have recovered the space.

 

For now, I'll be content with getting my 30GB 😝

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

ORIGINAL POST:

 

Hi there,

I'm looking for some help figuring out what is happening to my HDD space which sometimes diminishes when I use the Gather function of this plugin. 
It looks like the space taken up by files from the source disk are not being freed up, while the copy to the target disk is still writing successfully, resulting in what looks like a duplication of data consuming my usable drive space.
However I can't seem to find the data on the source disk when I browse through the disk under '/mnt/diskX'.
It also doesn't do this for every file/folder in the transfer. Could be only 1 or a few folders which doesn't release their space. 

The first time I saw it happen was when I was moving a couple hundred GB of data between some drives, and I noticed that 130GB of data wasn't freed up on the source drive that I was moving the data from. 
I had tried restarted my array and rebooting the server in hopes of some cleanup activity needing to be done, but sadly to no success. The 130GB wasn't being released.
I had given up on that 130GB as I had spent too much time trying to troubleshoot and didn't have any trace of my logs or screenshots for any proof which didn't help, so had given up on recovering that 130GB, but ever since I've become paranoid every time I am doing some transfers, and started taking screenshots to try and capture the behaviour again and haven't had any problems in the past few weeks till now...

For some background, I was on unRaid 6.11.5 and was using the old unbalanced plugin for the past 2 years and moved several TB's over the years with the Gather function without any problems, but I had to upgrade my unRaid to 6.12.12 in August this year, and so uninstalled the old plugin and installed the new unbalanced plugin to continue doing what I needed to do, which is where I am at now. 

Hopefully what I have captured below illustrates what I'm talking about.

 

Pic 1: This is what unbalanced was showing me. For some reason it shows 27.0GB, even though there are no source files existing on the target 'disk3'. All the files are on 'disk5 and disk9', and I expect a full transfer of the 66.6GB (similar to what is showing for disk1 and disk2, or the partial transfer to disk9).

There is also no star symbol, indicating existing files on the disk so I am not sure how the 27.0GB number is calculated...

The circled box also shows the current->planned state, and shows the full 66.6GB to be written to the drive (849 GB - 782 GB = 67 GB). Same thing with disk7, so I suspect if I selected to transfer to happen on disk7, then I will also lose disk space.

 

Pic 2: The before capacity of the drives shown on unRaid. I have highlighted the drives which were involved in the transfer operation.

 

Pic 3: The after state after the unbalance Gather was successfully completed.

• The target disk (disk3) looks correct, and corresponds with the current->planned 66.6GB full write to the disk which unbalanced had exactly shown, from 849 GB- > 782 GB.
• One of the source disks (disk9) also looks correct, as the space was freed up from the 3GB file that was on that disk that transferred and released the space correctly, going from 772 GB -> 775 GB.
• The problematic one is (disk5)... I am expecting ~63GB to be freed up from here, to go from ~1.17TB -> ~1.23TB, and the total array to remain the same at 6.60TB. But as you can see, only ~30GB looks to be freed up. It looks like the 32.7GB folder that was to be transferred wasn't freed up on disk 5. When I was monitoring the Gather/Move operation I could see the writes start consuming space once it started trying to transfer that folder over. 

I have downloaded the unbalanced logs and also looked through them, and don't see anything out of the norm. The copying looks to complete correctly without any errors, and the 'removing (rm-rf)' executes correctly on the source disks, as well as the pruning of the leftover empty folders. I have attached the logs for this operation here as well: unbalanced_log_20240908.txt

 

If anyone could maybe figure out what is happening, or if there is a way to recover the space which wasn't freed up after the transfer is already done. If it's unbalanced doing something weird, or if it's something at an OS/unRaid level?

Maybe what is unbalanced showing with the 27.0GB in Pic 1, if it's a bug or some other calculation that it's doing...?

Or even at least if I can just recover the unreleased HDD capacity, then I don't mind performing any extra actions after any transfers to release it as a workaround, since I would like to retain as much usable storage in my array if possible 😅.

 

Thanks for reading through this long post!

Any help would be greatly appreciated!

Edited September 8 by HellScytheX
Situation improved after continued monitoring and providing an update on the resolution.

[ap90033]

If I stop the move, will unbalance finish the current file being moved and then stop? 

[jbrodriguez]

12 hours ago, ap90033 said:

If I stop the move, will unbalance finish the current file being moved and then stop? 

stopping a command/operation, means sending a linux kill command to the rsync process ... sometimes it takes a bit of time to end, but i'm not sure if it waits for the current file to complete the transfer, i don't think it does though