[Plugin] unbalanced

[FACENC]

Okay now I'm running into an even weirder issue.  I'm using unbalance to move data from one disk to another. And while I can confirm the data is moving (checking through SFTP or SSH session), the Unraid GUI still shows the same amount of disk usage. 

 

Here is unBalance showing it has already moved 597GB of data off of disk6.

 

And here is the unraid GUI still showing only 166GB free.

 

I'm at a loss for why my drive space is not moving.  Unless unbalance is moving hardlinks and still just taking forever to do such?

 

...and now it is at 100% complete, but it is still "moving" data.  Disk5 is nearing completely full (even though the plan showed it would have 1TB free after the move).

 

 

I don't understand why unbalance is acting this way.  I have used it before without issue, but now between the GUI crashing and this behavior, I can't really trust it with my data anymore.

Edited October 7, 2022 by FACENC

[Cessquill]

9 hours ago, FACENC said:

I don't understand why unbalance is acting this way.

Unbalance copies then deletes the source files (similar to how Midnight Commander does).  If you're moving a lot of data, there may be a cleanup period at the end where the original files are removed.

[Danyal]

Hi, I'm trying to use Unbalance to move my media share from 1 drive to several smaller drives. My directory structure is currently setup as Media/<sub-dirs> (e.g. Tv, Movies, etc). Media is a top level share. Split level is set to any level.

 

When I select the top level share dir (media), unbalanced says that it can't move media/movies because there isn't enough space on target drives. It seems like it has something to do with the fact that it's trying to keep all of movies on 1 drive rather than splitting it. I'm trying to move 20TB -> 3x 10TB that are all empty. If I select the sub directories instead (e.g. Media/Movies) then it does what I'd expect it to do which is 'plan' shows it moving the 20TB filling the first 2 drives and a bit left over on the 3rd (due to my reserved space setting).

 

Is this expected? Any issues with selecting the sub-dirs?

[miicar]

Hey guys.  I RAN INTO A HUGE PROBLEM! 

I know this plugin hasn't really been updated in a while; and yes, i realize that since its been left to die, it doesn't handle multiple cache pools that unRAID now (thankfully) has, BUT i loved this plugin for what it did before.
For the most part, i use MC or other methods to move files between disks, but the other day, i decided to move some files around my production server with unBALLANCE again (carefully avoiding the cache drive, as i know its not really supported anymore)...boy did that seem to turn into a nightmare! 

My goal was to move the active production files to a faster HDD installed on the array (and archive completed files on the older/slower drives).  The next day, after moving ("Gather") the files to the new array drive using unBALANCE overnight, my design team reported missing a bunch of their empty production and folder-structure template folders!!!  It seems like every folder that was touched by the unBALANCE move was stripped of its empty folders!  (at first we thought we lost actual files, but after HOURS of reviewing my regular backups, i've concluded its only the empty folders that were deleted). 

 

There wasn't anything suspect in the logs of unBALANCE, other then the permissions warning at the start.  But that's never been an issue in the past, as it had only left some files/folders behind that i could manually clean up later.

These deleted folders did not go into the Recycle Bin (unraid plug-in) folder, as any other deleted files do...just gonzo!  And again, ONLY the empty folders are missing...if there was even a hidden file in the folder, it survived!

I guess i should never use this plugin again, unless its brought back from the dead with proper support for current unRAID versions (currently running 6.10.3)...but i would still like to figure out what happened, why it happened, and how to make sure it never happens again...and THANK GOD FOR REGULAR BACKUPS!

Has anyone else had this experience with this plugin?  is there a setting I flipped by accident?  Can my team trust me again?  Should i hang up my SATA cables and work at Wall-Mart?

 

(i will also pose this question in general questions forum, as it seems this one is quite stale)

Edited October 27, 2022 by miicar

[te5s3rakt]

is there a way to use this plug to automatically reapply a split level for a share.

 

so i can find a way to say select a tv show, and then move that to a single disk.

 

but what if it wanted to shuffle all tv shows so that each shows seasons are huddled together single disks? but all shows are distributed evenly (as close as they're able that is) across the array.

Edited December 2, 2022 by te5s3rakt

[Steviewunda]

I ran Docker Safe New Permissions after a  dry run of unbalance reported numerous permissions errors. A second dry run of unbalance Gather reported no errors, so I did an actual Move run which should have consolidated 550GB of Music share on disks 1,2,4, and 5 with about 300GB on drive 3. After the run (which reported no errors) there is only about 380GB of music files in the Music share, and a lot of music is missing. When I look at the Music share in Windows File Explorer or Krusader, there are numerous empty or partially empty folders, sometimes containing only the .jpg's, playlists etc but few or no actual music files. Is there any way I can recover them? I have attached the diagnostics file, and the unbalance.log Even if I can't recover them, it would be good to know what went wrong

unbalance.log tower-diagnostics-20221214-1530.zip

[FreeMan]

I'm clearing an old disk, moving files off of it.

 

After running a number of scatter passes, I've noticed that unBalance doesn't always delete the empty directories.

 

Am I missing something or is this expected?

[jbrodriguez]

On 10/27/2022 at 12:10 PM, miicar said:

It seems like every folder that was touched by the unBALANCE move was stripped of its empty folders!

i think that's the default rsync behavior, to not copy empty folders ? not sure, but in any case, you can always add custom flags to rsync to behave the way you need it to

[jbrodriguez]

On 12/14/2022 at 2:09 PM, Steviewunda said:

I ran Docker Safe New Permissions after a  dry run of unbalance reported numerous permissions errors. A second dry run of unbalance Gather reported no errors, so I did an actual Move run which should have consolidated 550GB of Music share on disks 1,2,4, and 5 with about 300GB on drive 3. After the run (which reported no errors) there is only about 380GB of music files in the Music share, and a lot of music is missing. When I look at the Music share in Windows File Explorer or Krusader, there are numerous empty or partially empty folders, sometimes containing only the .jpg's, playlists etc but few or no actual music files. Is there any way I can recover them? I have attached the diagnostics file, and the unbalance.log Even if I can't recover them, it would be good to know what went wrong

unbalance.log 4.54 MB · 0 downloads tower-diagnostics-20221214-1530.zip 139.58 kB · 0 downloads

there was an issue with your disk 2

I: 2022/12/14 14:50:01 core.go:695: Command Started: (src: /mnt/disk2) rsync -avPR -X "Music/A Tribe Called Quest" "/mnt/disk3/" 
W: 2022/12/14 14:50:01 shell.go:50: flag:(rsync: [sender] link_stat "/mnt/disk2/Music/A Tribe Called Quest" failed: No such file or directory (2)
)
W: 2022/12/14 14:50:02 core.go:748: command:end:error(exit status 23)
I: 2022/12/14 14:50:02 core.go:752: command:retcode(0):exitcode(23)
I: 2022/12/14 14:50:02 core.go:1009: Command Finished
I: 2022/12/14 14:50:02 core.go:1022: Current progress: 17.94% done ~ 1h50m6s left (65.37 MB/s)
W: 2022/12/14 14:50:02 core.go:941: skipping:deletion:(rsync command was flagged):(/mnt/disk3/Music/A Tribe Called Quest)
I: 2022/12/14 14:50:02 core.go:695: Command Started: (src: /mnt/disk2) rsync -avPR -X "Music/Al Green" "/mnt/disk3/" 

it wasn't finding folders in disk2/Music so it started skipping each one that wasn't found, it's strange, looks like something changed on the disk between planning and actually moving files 

[jbrodriguez]

On 12/15/2022 at 9:49 PM, FreeMan said:

I'm clearing an old disk, moving files off of it.

 

After running a number of scatter passes, I've noticed that unBalance doesn't always delete the empty directories.

 

Am I missing something or is this expected?

it doesn't delete top level directories (the Unraid shares), other than it should delete folders under the share that are empty 

[Steviewunda]

1 hour ago, jbrodriguez said:

there was an issue with your disk 2

I: 2022/12/14 14:50:01 core.go:695: Command Started: (src: /mnt/disk2) rsync -avPR -X "Music/A Tribe Called Quest" "/mnt/disk3/" 
W: 2022/12/14 14:50:01 shell.go:50: flag:(rsync: [sender] link_stat "/mnt/disk2/Music/A Tribe Called Quest" failed: No such file or directory (2)
)
W: 2022/12/14 14:50:02 core.go:748: command:end:error(exit status 23)
I: 2022/12/14 14:50:02 core.go:752: command:retcode(0):exitcode(23)
I: 2022/12/14 14:50:02 core.go:1009: Command Finished
I: 2022/12/14 14:50:02 core.go:1022: Current progress: 17.94% done ~ 1h50m6s left (65.37 MB/s)
W: 2022/12/14 14:50:02 core.go:941: skipping:deletion:(rsync command was flagged):(/mnt/disk3/Music/A Tribe Called Quest)
I: 2022/12/14 14:50:02 core.go:695: Command Started: (src: /mnt/disk2) rsync -avPR -X "Music/Al Green" "/mnt/disk3/" 

it wasn't finding folders in disk2/Music so it started skipping each one that wasn't found, it's strange, looks like something changed on the disk between planning and actually moving files 

So 500GB of music files have gone? Fool that I was, I was consolidating my shares to make backup to another Unraid server easier - the successful dry runs convinced me that it was safe to run Unbalance. As always, it pays to backup before making changes. Unfortunately, I could never trust that Unbalance was going to perform correctly again.

[FreeMan]

3 hours ago, jbrodriguez said:

it doesn't delete top level directories (the Unraid shares), other than it should delete folders under the share that are empty 

That's the behavior I expected, however, that's not the behavior I'm seeing.

 

\mnt\disk3\Movies\Movie1

 

Is left as an empty directory after having moved Movie1 from disk3 to disk10.

 

As a matter of fact, I've got 179 empty directories under ..\disk3\Movies, 14 empty directories under ..\disk3\Sport and 194 under ..\disk3\TV

 

Fortunately, Krusader makes it easy to delete them all, but still...

[FreeMan]

1 hour ago, Steviewunda said:

So 500GB of music files have gone? Fool that I was, I was consolidating my shares to make backup to another Unraid server easier - the successful dry runs convinced me that it was safe to run Unbalance. As always, it pays to backup before making changes. Unfortunately, I could never trust that Unbalance was going to perform correctly again.

Did you have anything else reading/writing to the array at the time of the move? The instructions explicitly say not to...

[Steviewunda]

3 minutes ago, FreeMan said:

Did you have anything else reading/writing to the array at the time of the move? The instructions explicitly say not to...

No I followed SpaceinvaderOne's instructions very carefully

[FreeMan]

This is an awesome tool that works about 99% for me - I greatly appreciate it!

 

Two suggestions for you, @jbrodriguez, when you get a few minutes to do some new development.

 

1) When doing a Gather, make the top of the window a non-scrolling frame. That way, as I'm scrolling through more than a screen full of movies or TV shows, I can click on one way down in the "R" section and still see what disk(s) it's on so I can decide if I need to gather it up.

2) When doing a Scatter, give me some sort of idea of how many bytes I've selected to move. I may be clearing off an entire 4TB drive, but may not want to do it all in one pass. It'd be handy to know if I've selected 400MB or 400GB of data. (If things have been scattered, 20 directories may only contain a few 100s of bytes each.)

 

 

[jbrodriguez]

12 hours ago, Steviewunda said:

So 500GB of music files have gone? Fool that I was, I was consolidating my shares to make backup to another Unraid server easier - the successful dry runs convinced me that it was safe to run Unbalance. As always, it pays to backup before making changes. Unfortunately, I could never trust that Unbalance was going to perform correctly again.

right, i can't say for sure what happened, when rsync throws an error, unbalance shows that command with a yellow icon on the history page and doesn't delete the source folder

[jbrodriguez]

10 hours ago, FreeMan said:

As a matter of fact, I've got 179 empty directories under ..\disk3\Movies, 14 empty directories under ..\disk3\Sport and 194 under ..\disk3\TV

maybe a permission issue ?

[jbrodriguez]

10 hours ago, FreeMan said:

This is an awesome tool that works about 99% for me - I greatly appreciate it!

thanks !

 

10 hours ago, FreeMan said:

When doing a Gather, make the top of the window a non-scrolling frame. That way, as I'm scrolling through more than a screen full of movies or TV shows, I can click on one way down in the "R" section and still see what disk(s) it's on so I can decide if I need to gather it up.

can you provide an example ?

10 hours ago, FreeMan said:

When doing a Scatter, give me some sort of idea of how many bytes I've selected to move

sounds reasonable, unbalance knows that info before starting the operation

[FreeMan]

8 hours ago, jbrodriguez said:

maybe a permission issue ?

I've had no permissions warnings as I've been doing the moves. Could be, I suppose, though. The most recent run did delete the empty directories.

 

8 hours ago, jbrodriguez said:

can you provide an example ?

I have a couple thousand movies - I selected this movie:

Then I have to scroll back to the top of the window in order to see which/how many disks it's scattered across:

 

If the portion in the red box could be in an iframe (or similar construct - I know just enough web dev to be extremely dangerous) with a scroll bar as needed where I added a heavy green line. Obviously, the "Shares Explorer" part would be below the frame so they would scroll independently.

 

Just had a thought - maybe the "Shares Explorer" could scroll vertically on the left side, while the "Selected" could scroll vertically on the right side.

Edited December 18, 2022 by FreeMan

[FreeMan]

Here's a better drawing of my suggestion. Each colored box would be a separate frame.

The headers would always be available in the red frame. That way, when I've scrolled all the way down to "Z", I can still click "Next" without a lot of scrolling back up.

I can scroll through the Explorer in the blue frame, and as I check items, they appear in the yellow frame. The blue & yellow frames would scroll independently, so I could select movies starting with "A", "L", and "Z" in the Explorer, and they'd show at the top of the "Selected".

 

This way, I might choose to pick 10 (more or less) random movies that exist on disk7, and move them to some other location...

 

Additional thought: The blue & yellow frames might need a horizontal scroll. Having that dynamically appear only when needed would be the best bet, but if they need to be there all the time, that'd work, too.

Edited December 19, 2022 by FreeMan

[jbrodriguez]

19 hours ago, FreeMan said:

Here's a better drawing of my suggestion. Each colored box would be a separate frame.

The headers would always be available in the red frame. That way, when I've scrolled all the way down to "Z", I can still click "Next" without a lot of scrolling back up.

I can scroll through the Explorer in the blue frame, and as I check items, they appear in the yellow frame. The blue & yellow frames would scroll independently, so I could select movies starting with "A", "L", and "Z" in the Explorer, and they'd show at the top of the "Selected".

 

This way, I might choose to pick 10 (more or less) random movies that exist on disk7, and move them to some other location...

 

Additional thought: The blue & yellow frames might need a horizontal scroll. Having that dynamically appear only when needed would be the best bet, but if they need to be there all the time, that'd work, too.

ok, i get it now, if i get some free time i'll take a look at it

[FreeMan]

18 hours ago, jbrodriguez said:

ok, i get it now, if i get some free time i'll take a look at it

That would be awesome, thanks!

 

Please don't forget the Scatter side - give a total amount to be moved and show how much will be moved to each of the selected destination disks.

[miicar]

On 12/17/2022 at 1:53 PM, jbrodriguez said:

i think that's the default rsync behavior, to not copy empty folders ? not sure, but in any case, you can always add custom flags to rsync to behave the way you need it to

fine...if i was using Rsync and the missing folders were in the backup that i copied to...but this issue was the folders went missing from the copy-from folder...not the destination...in fact, they were in the destination (if i recall...its been a while and i've just stopped using that app)

[CryPt00n]

On 2/19/2022 at 11:28 PM, jbrodriguez said:

i'd still like to extend unbalance for multiple cache pools, but since i don't use it, i need some data to make it work.

 

Hi, any update on this?

[Seonac]

One thing that isn't clear to me... maybe I missed it seeing it stated or listed somewhere.

 

Is the Parity still valid during a Scatter, Move operation?

 

I have a disk that may fail soon and don't have a spare but have plenty of space within the array to move the data to, so I was going to shrink the array. I realize that I will have to create a new config and rebuild the parity after removing the disk. I started a scatter, move operation following SpaceInvaderOne's instructions and realized after starting the operation I'm not sure if the parity is maintained during the move.