On 3/3/2024 at 12:11 PM, Starfox said:

Thanks - no yellow marked command for the operation no (And hence I can't see any button to delete the folders).

 

Tried to do the second run (Is that really the workflow here?). Unbalanced got stuck in plan mode. Refreshed and tried again. Would only allow me to do one folder at a time (Kept saying destination was full which given it was just an empty folder obvious was not the case). Got there in the end. Thanks

I have the same issue when using scatter to move folders with files.  The folders and files are moved leaving empty folders on the source disk.  I would then have to run the scatter again and again where it will delete the folders one at a time even though I've selected many.  History shows the transfers as complete without errors...

1 hour ago, Wynter said:

The folders and files are moved leaving empty folders on the source disk. 

Yeah, this "feature" already existed in the previous plugin "unbalance". But it was sufficient to re-run the whole job once more to get rid of them.

UPDATE: 

Was checking in periodically without adding any new files, and noticed that the total array capacity number went up from 6.57 TB -> 6.58 TB. 

I then restarted the 1 Transmission docker which would have been associated with that 1 folder, and after a few minutes it does look like the space was freed back up to 6.60TB, with the source disk in question freeing up the space to the expected amount. 

So looks like the deletion of the files from the source disks was held up by the docker in this case, but I have no way of seeing the held up files in the first place.

Unfortunately it still doesn't explain my first case of 130GB, since those files were not associated with any applications at the time, and also wasn't restored even after restarting the array or rebooting the server. But I'll assume it's all ok for now and will restart/disable any applications that may be associated with the files that I'm moving in question going forward to hopefully resolve the space problems I get in the future.

 

I am still curious where the 27.0GB number comes from in the unbalanced drive selection preview, but it's not something critical anymore since I have recovered the space.

 

For now, I'll be content with getting my 30GB 😝

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

ORIGINAL POST:

 

Hi there,

I'm looking for some help figuring out what is happening to my HDD space which sometimes diminishes when I use the Gather function of this plugin. 
It looks like the space taken up by files from the source disk are not being freed up, while the copy to the target disk is still writing successfully, resulting in what looks like a duplication of data consuming my usable drive space.
However I can't seem to find the data on the source disk when I browse through the disk under '/mnt/diskX'.
It also doesn't do this for every file/folder in the transfer. Could be only 1 or a few folders which doesn't release their space. 

The first time I saw it happen was when I was moving a couple hundred GB of data between some drives, and I noticed that 130GB of data wasn't freed up on the source drive that I was moving the data from. 
I had tried restarted my array and rebooting the server in hopes of some cleanup activity needing to be done, but sadly to no success. The 130GB wasn't being released.
I had given up on that 130GB as I had spent too much time trying to troubleshoot and didn't have any trace of my logs or screenshots for any proof which didn't help, so had given up on recovering that 130GB, but ever since I've become paranoid every time I am doing some transfers, and started taking screenshots to try and capture the behaviour again and haven't had any problems in the past few weeks till now...

For some background, I was on unRaid 6.11.5 and was using the old unbalanced plugin for the past 2 years and moved several TB's over the years with the Gather function without any problems, but I had to upgrade my unRaid to 6.12.12 in August this year, and so uninstalled the old plugin and installed the new unbalanced plugin to continue doing what I needed to do, which is where I am at now. 

Hopefully what I have captured below illustrates what I'm talking about.

 

Pic 1: This is what unbalanced was showing me. For some reason it shows 27.0GB, even though there are no source files existing on the target 'disk3'. All the files are on 'disk5 and disk9', and I expect a full transfer of the 66.6GB (similar to what is showing for disk1 and disk2, or the partial transfer to disk9).

There is also no star symbol, indicating existing files on the disk so I am not sure how the 27.0GB number is calculated...

The circled box also shows the current->planned state, and shows the full 66.6GB to be written to the drive (849 GB - 782 GB = 67 GB). Same thing with disk7, so I suspect if I selected to transfer to happen on disk7, then I will also lose disk space.

 

Pic 2: The before capacity of the drives shown on unRaid. I have highlighted the drives which were involved in the transfer operation.

 

Pic 3: The after state after the unbalance Gather was successfully completed.

• The target disk (disk3) looks correct, and corresponds with the current->planned 66.6GB full write to the disk which unbalanced had exactly shown, from 849 GB- > 782 GB.
• One of the source disks (disk9) also looks correct, as the space was freed up from the 3GB file that was on that disk that transferred and released the space correctly, going from 772 GB -> 775 GB.
• The problematic one is (disk5)... I am expecting ~63GB to be freed up from here, to go from ~1.17TB -> ~1.23TB, and the total array to remain the same at 6.60TB. But as you can see, only ~30GB looks to be freed up. It looks like the 32.7GB folder that was to be transferred wasn't freed up on disk 5. When I was monitoring the Gather/Move operation I could see the writes start consuming space once it started trying to transfer that folder over. 

I have downloaded the unbalanced logs and also looked through them, and don't see anything out of the norm. The copying looks to complete correctly without any errors, and the 'removing (rm-rf)' executes correctly on the source disks, as well as the pruning of the leftover empty folders. I have attached the logs for this operation here as well: unbalanced_log_20240908.txt

 

If anyone could maybe figure out what is happening, or if there is a way to recover the space which wasn't freed up after the transfer is already done. If it's unbalanced doing something weird, or if it's something at an OS/unRaid level?

Maybe what is unbalanced showing with the 27.0GB in Pic 1, if it's a bug or some other calculation that it's doing...?

Or even at least if I can just recover the unreleased HDD capacity, then I don't mind performing any extra actions after any transfers to release it as a workaround, since I would like to retain as much usable storage in my array if possible 😅.

 

Thanks for reading through this long post!

Any help would be greatly appreciated!

Edited September 8, 20241 yr by HellScytheX
Situation improved after continued monitoring and providing an update on the resolution.

If I stop the move, will unbalance finish the current file being moved and then stop? 

12 hours ago, ap90033 said:

If I stop the move, will unbalance finish the current file being moved and then stop? 

stopping a command/operation, means sending a linux kill command to the rsync process ... sometimes it takes a bit of time to end, but i'm not sure if it waits for the current file to complete the transfer, i don't think it does though

Just checking if this is the right thing for the task at hand: I have three disks @16TB each (one of them parity) and would like to have my files sorted by folder into disk1 & disk2 (e.g. folder A is only on disk1, folder B only on disk2) to prepare for a data migration.

Would that be possible, even if folder A is bigger than the total free space on disk1 (but not bigger than total space), e.g. could the plugin move data around to free space so that folder A fits onto drive1?

Well, the question is, why do you need to move data around at all?  You say you have three 16 TB drives, and one of them is parity.  So you have 32 TB of usable space.  What are your plans?  You might not have to move data around at all.  All depends what you want to do.  Are you adding more disks to the existing three?  Do you want to replace one of the three with a different larger drive?  What you're planning on doing has a big effect one what you can actually do and what you actually may or may not need to do.  Give us some more info.

Quote

Give us some more info.

Ok, that I could have done before. 🙂

I'd like to migrate away from unraid and I planned on having a backup of the important files (personal photos, documents, ebooks, etc.) on disk1 and the recoverable files (music, videos, etc.) on disk2, so that I can temporarily remove one disk after the other from unraid to the new system. I have the important files backed up as during this period they would not be parity protected.

Ah, ok.  Well, unbalanced can help, but it isn't going to let you move more data than there is room for on the destination drive.  You're going to have to do some juggling.  Move some to disk 1, move some to disk 2, repeat as necessary.  It'll be a little easier if you have things in some nested sub-folders that can be selected independently.  But it isn't going to let you select one folder that contains 5 TB of stuff on disk 2 to move to disk 1 if you only have 4 TB free on disk 1.  If the data is split between the two drives it will show you how much would have to be moved and whether or not there's enough space for that job, though.

 

Mind if I ask why you're wanting to move away from unraid?  Just curious.  Been using it for years here and still like it a lot.  I'm up to two parity drives and 10 data drives now.

Thanks for the info, juggling it will be then...

 

As for the reasons, I tried unraid after ditching a Synology NAS (very good software, weak hardware) and a SFF PC for an all-in-one solution on a Ugreen 4800+ NAS. At first I thought, unraid would be good but there were several issues (mostly docker & compose related, clunkyness with compose, a docker apparently writing to rootfs despite not being mounted there, leading to complete unresponsiveness) that led me to believe TrueNAS could be better. Probably, in a few months/years/days I will think, "man, this unraid looks slick, if only TrueNAS would not be such a stickler with its ZFS volumes...".

Ah, I see.  Mine's nothing but a fileserver for me.  I know it is capable of doing a lot more, but that's all I want out of it, and it does that well.  Thanks.

File deleted when using "--max-size" or "--min-size"

 

When using the "Gather" function, I want to prevent small files in the cache from being moved to the HDD pool. I noticed that Unbalanced utilizes rsync for file transfers. To address this, I added the option --min-size=20M, intending to move only files larger than 20MB. However, I discovered that all files smaller than 20MB were deleted instead.

 

I stupidly used "Gather" again with the --max-size=20M option, which resulted in the loss of 2TB of movies.

 

I'm wondering if this is a bug in Unraid related to rsync, or if I simply executed the command incorrectly. 

23 minutes ago, SunSwallow said:

I want to prevent small files in the cache from being moved to the HDD pool.

(can't help you with your "deleted" problem, but...)

Pls reconsider the "small files stay on cache" wish. In general it is very unwise because small files allocate much larger space on disk than needed (this is true for all files, but with large files the wasted percentage is almost 0 whereas with small files it can be many 100%). So a cache with only small files will fill up much faster than with mixed/large files.

This is true for all file systems because they also manage some kind of blocks and all have a minimum amount of blocksize that will always be used (it once was NOT true for NTFS where content of very small files was already stored in the directory entry of this file, but I dont know if this is still used today anymore).

 

4 minutes ago, MAM59 said:

 

Pls reconsider the "small files stay on cache" wish. In general it is very unwise because small files allocate much larger space on disk than needed

 

Thanks for you kindly advice. I would check the usage of Cache regularly  following your advice! My cache usage is only about 150GB/500GB so I think it would not be a problem for me now.

5 hours ago, SunSwallow said:

I'm wondering if this is a bug in Unraid related to rsync, or if I simply executed the command incorrectly

 

i've never used those command line flags in an rsync command outside of unbalanced, much less as part of unbalanced 🤷‍♂️, but i think it did what you told it to do,  it copied some files (only those below 20M), then deleted the source files (as gather is a move which is copy/delete)

6 minutes ago, jbrodriguez said:

but i think it did what you told it to do,  it copied some files (only those below 20M), then deleted the source files (as gather is a move which is copy/delete)

Thanks a lot. It seems that I didn't understand and use gather correctly😞.

5 minutes ago, SunSwallow said:

Thanks a lot. It seems that I didn't understand and use gather correctly😞.

not really, i think the flags were the issue, i've stuck to the about 5 or 6, maybe 7 rsync flags for as long as i can remember, going beyond that is uncharted territory (at least for me)

 

i do have a warning on the unbalanced rsync settings for that reason (something about changing the flags is for expert users or similar iirc)

10 minutes ago, jbrodriguez said:

not really, i think the flags were the issue,

In fact, I think it's because I didn't realize gather would remove the source files using "rm". When using rsync in other situations, if I want to delete the source files, I would use "--remove-source-files" flag. It would only remove these files transferred. So when I check the rsync flags, I thought it was OK. 
Unbalanced always works well, so I never read the log before. When I found some files were deleted, for the first time, I read the log and found that Unbalanced will remove the old files using "rm". Maybe it's a good idea to use "--remove-source-files" instead of another "rm"? I dont know if this would work for unraid file system.

7 minutes ago, SunSwallow said:

In fact, I think it's because I didn't realize gather would remove the source files using "rm"

when you choose gather and before you start transferring, the button to start the operation reads Move, which signals files will be moved (transferred to destination then deleted on source, either via rm or remove-source-files)

 

8 minutes ago, SunSwallow said:

Maybe it's a good idea to use "--remove-source-files" instead of another "rm"

i think i considered this early on, i didn't use it at the time (not sure why) but i can revisit this sometime

I've successfully used unbalanced (gather) to move my Nextcloud folder to one disk so that only this disk spins up. However, if I try to do the same with /Media/Plex media/Shows, there are no target disks shown at all. Am I doing something wrong? I already used newperm on those media files, because I thought it was a rights problem.

3 hours ago, Hiko0 said:

I've successfully used unbalanced (gather) to move my Nextcloud folder to one disk so that only this disk spins up. However, if I try to do the same with /Media/Plex media/Shows, there are no target disks shown at all. Am I doing something wrong? I already used newperm on those media files, because I thought it was a rights problem.

Probably just isn't enough room.  Try selecting a subset of the child folders instead of just the main parent folder.

1 hour ago, _Shorty said:

Probably just isn't enough room.  Try selecting a subset of the child folders instead of just the main parent folder.

yup, try this

 

5 hours ago, Hiko0 said:

there are no target disks shown at all.

 

Thanks for your quick and informative answers. That was my guess already, so I have to try to figure out what to do with those huge media folders. Maybe you could add a message in those cases saying "not enough space for transfer"!?

They're not already organized into a bunch of subfolders?  Instead of trying to gather:

\TV Shows\

instead gather:

\TV Shows\The Simpsons\
\TV Shows\Futurama\
\TV Shows\Top Gear\
\TV Shows\Misfits\

etc., selecting a bunch of the child folders instead of the main one.  That's what I was trying to say earlier.  If you actually have tons of huge files in a single folder, well, I suppose you could temporarily put them into some temp subfolders like:

\Movies\A to G\
\Movies\H to N\

\Movies\O to U\

\Movies\V to Z\

 

and try gathering just one or two of those at a time.  Or something similar.  The more fine-grained you get, the easier it should be to fill up a drive.  Then after gathering you can just move them back out of the temp folders into their original folders.  If you don't have enough space to put everything onto a single drive you can at least put big chunks of things together.  That'll help stop it from having to fire up a handful of drives most of the time.

8 hours ago, Hiko0 said:

Maybe you could add a message in those cases saying "not enough space for transfer"!?

yea, that's currently missing