[DOCKER] unBALANCE


Recommended Posts

Is there a way to ensure that a drive is not fill up fully? Because so far I have seen that it has only moved data to 1 drive and that is is now down to 40gb of free space, will that drop to 0gb free space or will unBALANCE automatically stop and move to the next drive?

Link to comment
  • Replies 234
  • Created
  • Last Reply

Top Posters In This Topic

Hi MrLondon,

 

It will leave some room, currently set to at least 450Mb.

 

How it works is

- It sorts the target disks by free space available

- Tries to fill up the first disk as much as possible (respecting the 450mb)

- Moves to the next disk, tries to fill it up as much as possible ...

- Repeat the previous step as many times as needed

 

You can choose one disk (or various) as not elegible (uncheck the TO column).

 

Link to comment

Thanks for the answer, yes I have about 550mb of free space left on that disk, it moved to the next disk, but now it seems to have hung itself as for the past 5min at least it has not moved from a file and I cannot see any activity on the system with unmenu or system stats. is there a way to see from your system if it's hanging?

 

from your logs these are the last lines

 

I: 2015/06/19 14:46:31 core.go:345: ./movies/Sports Movies/Field of Dreams (1989)/movie.xml

I: 2015/06/19 14:46:31 core.go:345: ./movies/Sports Movies/Field of Dreams (1989)/.actors/Timothy_Busfield.jpg

I: 2015/06/19 14:46:31 core.go:345: ./movies/Sports Movies/Field of Dreams (1989)/.actors/Dwier_Brown.jpg

I: 2015/06/19 14:46:31 core.go:345: ./movies/Sports Movies/Field of Dreams (1989)/.actors/Amy_Madigan.jpg

I: 2015/06/19 14:46:31 core.go:345: ./movies/Sports Movies/Field of Dreams (1989)/.actors/Ray_Liotta.jpg

I: 2015/06/19 14:46:31 core.go:345: ./movies/Sports Movies/Field of Dreams (1989)/.actors/Art_LaFleur.jpg

I: 2015/06/19 14:46:31 core.go:345: ./movies/Sports Movies/Field of Dreams (1989)/.actors/Kevin_Costner.jpg

I: 2015/06/19 14:46:31 core.go:345: ./movies/Sports Movies/Field of Dreams (1989)/.actors/James_Earl_Jones.jpg

I: 2015/06/19 14:46:31 core.go:345: ./movies/Sports Movies/Field of Dreams (1989)/.actors/Mike_Nussbaum.jpg

I: 2015/06/19 14:46:31 core.go:345: ./movies/Sports Movies/Field of Dreams (1989)/.actors/Gaby_Hoffmann.jpg

I: 2015/06/19 14:46:31 core.go:345: ./movies/Sports Movies/Field of Dreams (1989)/.actors/Burt_Lancaster.jpg

I: 2015/06/19 14:46:31 core.go:345: ./movies/Sports Movies/Field of Dreams (1989)/.actors/Lee_Garlington.jpg

I: 2015/06/19 14:46:32 core.go:345: ./movies/Sports Movies/Field of Dreams (1989)/.actors/Larry_Brandenburg.jpg

I: 2015/06/19 14:46:32 core.go:345: ./movies/Sports Movies/Field of Dreams (1989)/.actors/Frank_Whaley.jpg

I: 2015/06/19 14:46:32 core.go:345: ./movies/Sports Movies/Field of Dreams (1989)/Field of Dreams-poster.jpg

I: 2015/06/19 14:46:32 core.go:345: ./movies/Sports Movies/Field of Dreams (1989)/disc.png

I: 2015/06/19 14:46:32 core.go:345: ./movies/Sports Movies/Field of Dreams (1989)/metabrowser.dvdid.xml

I: 2015/06/19 14:46:32 core.go:345: ./movies/Sports Movies/Field of Dreams (1989)/mymovies-front.jpg

I: 2015/06/19 14:46:32 core.go:345: ./movies/Sports Movies/Field of Dreams (1989)/backdrop.jpg

 

this is now 30min ago since last update.

Now there hasn't been an update for 1 hour actually. still at the last line from above.

Link to comment

...

this is now 30min ago since last update.

Now there hasn't been an update for 1 hour actually. still at the last line from above.

 

It definitely looks like it's hung.

 

Is the web interface still running (the activity indicator still moving) ?

 

If you're ok with dropping to the command line, could you post the output of these commands

 

ps aux | grep unbalance

ps aux | grep rsync

 

From the last lines you're showing, there doesn't seem to be any error.

 

Not sure why it stopped.

 

The log file itself has a bit more detail.

 

Let me know if you could see anything different in the last lines of the file.

Link to comment

hi Iboregard,

 

here are the output you asked for,but the log is still flashing from left to right, but no update in the list.

 

root@Tower:/boot# ps aux | grep unbalance

root      1381  0.0  0.0    176    4 ?        Ss  10:52  0:00 runsv unbalance

nobody    1606  0.0  0.1  10540  5976 ?        Sl  10:52  0:01 /usr/bin/unbalance -c /config

root      6475  0.0  0.0  5104  1636 pts/0    R+  16:45  0:00 grep unbalance

root@Tower:/boot# ps aux | grep rsync

nobody    5162  0.0  0.0  17100  1792 ?        S    14:46  0:00 rsync -dIWRpEAXogtl --numeric-ids --inplace ./movies/Sports Movies/Field of Dreams (1989)/backdrop.jpg /mnt/disk13/

nobody    5163  0.0  0.0  17100  1692 ?        S    14:46  0:00 rsync -dIWRpEAXogtl --numeric-ids --inplace ./movies/Sports Movies/Field of Dreams (1989)/backdrop.jpg /mnt/disk13/

nobody    5164  0.0  0.0  17100  220 ?        S    14:46  0:00 rsync -dIWRpEAXogtl --numeric-ids --inplace ./movies/Sports Movies/Field of Dreams (1989)/backdrop.jpg /mnt/disk13/

root      6923  0.0  0.0  5104  1676 pts/0    S+  16:45  0:00 grep rsync

nobody  15365  0.0  0.0  13356  2352 ?        S    14:30  0:00 find ./movies/Sports Movies -depth ( ( -type f ! -exec fuser -s {} ; ) -o ( -type d -empty ) ) -print ( -exec [ -d /mnt/disk13/{} ] ; -o -exec [ ! -e /mnt/disk13/{} ] ; -o -exec echo Duplicate skipped: {} ; -a -exec false ; ) -exec rsync -dIWRpEAXogtl --numeric-ids --inplace {} /mnt/disk13/ ; -delete

root@Tower:/boot#

 

Link to comment

After discussing this with MrLondon, a failing disk seems to have caused the underlying rsync to get stuck.

 

I will further review if it's possible to make rsync timeout, so that a meaningful error can be sent back to the UI (or via email).

 

So this is mostly a heads up, to people that may have a S.M.A.R.T. test failing on an array disk.

 

 

Link to comment

I hag about a terabyte of data I moved from my disk 2.  It calculated fine as it was supposed to move data to two other array disks and leave plenty of space.  Kick it off last night.  This morning got the email that it completed.  I checked tonight and one drive was at zero bytes available and it appears it duplicated a lot of data.  I still have 500GB on my drive 2.  How can I find out what happened?

Link to comment

I hag about a terabyte of data I moved from my disk 2.  It calculated fine as it was supposed to move data to two other array disks and leave plenty of space.  Kick it off last night.  This morning got the email that it completed.  I checked tonight and one drive was at zero bytes available and it appears it duplicated a lot of data.  I still have 500GB on my drive 2.  How can I find out what happened?

 

Hi B_Sinn3d, what does the email say ?

 

It should list which operations were completed and which weren't (due to space restrictions).

 

Also, the log file should have more details about what went on.

Link to comment

I hag about a terabyte of data I moved from my disk 2.  It calculated fine as it was supposed to move data to two other array disks and leave plenty of space.  Kick it off last night.  This morning got the email that it completed.  I checked tonight and one drive was at zero bytes available and it appears it duplicated a lot of data.  I still have 500GB on my drive 2.  How can I find out what happened?

 

Hi B_Sinn3d, what does the email say ?

 

It should list which operations were completed and which weren't (due to space restrictions).

 

Also, the log file should have more details about what went on.

 

There were no errors, everything appeared to work correctly.  I will show the beginning and end of the email.

 

"Move operation completed.

 

These are the commands that were executed:"

 

...here are all the diskmv commands in this section.

 

"Started: 2015-06-23 21:42:55.184367525 -0400 EDT

Ended: 2015-06-24 05:04:59.728098283 -0400 EDT

 

Elapsed: 7h22m4.543730855s"

 

That is how it ended, there were no errors and the logs did not show anything.

Link to comment

hi

 

thanks for this cool tool.

tried it for reiserfs xfs convo needs but failed on me due to my diskmv lack of info ("/" stuff). so i had to use midnight commander instead it took ages and blocked my laptop :) but now i see other uses for unbalancer and want to use it time to time.

 

if u accept i got a few suggestions

 

pls auto add "/" to folder names in settings.

 

without the slash marks its a nightmare many will have to find out soon. e.g. directory tree; disk6\- Movies ->\-- watched --\Hitchcock Collection\Frenzy\Frenzy (1972).avi  now if i just say to unbalancer to get - Movies - folder folder from disk6 and move it to disk 3 it will fail to move most of the folders and files due to nonstandard characters. but i can make it work using /- Movies -/ folder in settings.

 

another thing i would like to see is to be able to pick more from disk tick boxes. so i can organise my all around data to single disks as much as possible with ease . dunno if that is possible but would be cool to collect all the folders and data scattered around disks and combine them in a disk or two with just a few clicks inside unraid without telnet and such.

 

or maybe user shares to disks option? that would be cool as well. like from xx share to disk2 and disk3

 

one last thing; if i move x folder from disk1 to disk2 with unbalancer the x folder is still in disk1 empty inside but folder is there dunno why.

 

anyway i dont know if any of this is possible or worth your time to implement/fix but these would make the unbalancer much better and usable.

 

  p.s. yes i made a big mess when moving data around for xfs conversation :) now i want an easy way to organize my data inside the array

 

 

Link to comment

Hi zivizivi,

 

I think I follow what you're saying about the trailing slash thing.

 

It has to do with the semantics of the underlying rsync command that is being used.

 

I will check this further.

 

With regards to organizing the data, you might want to take a look at consld8 (Github). It seems to do what you're mentioning.

 

Not sure why the original folders are not deleted. The rsync command should be deleting them, after the copy has finished.

 

I'll try to replicate this.

Link to comment
  • 2 weeks later...
  • 2 weeks later...

I didn't read thru this whole thread so this may have been brought up.

When using mv to move files the creation date gets changed.. I don't know if there is a way around it.

Instead I've been using cp with  -a to keep dates intact then I delete the original after checking the copies.

 

Link to comment

yes lboregard i red the diskvm info when i first found the problem and trying to fix it. then i come here to help others and let u know. in the end i had to use midnight commander. it was no biggy just a telnet session from a 7/24 open server/download laptop. was a long time ago cant really remember why unbalancer wasnt enough : )

Link to comment
  • 2 weeks later...

Hi wgstarks, you could stop the internal rsync from the command line.

 

Do a `ps aux | grep rsync` and kill the relevant process.

 

It should pick up from there and move onto the next file, but to be honest I've never tried it :)

 

Is it possible that your disk is failing, seems weird that it has been stuck for hours trying to move the file.

Link to comment

Not sure what's going on. Haven't had any indications of a bad disk. Have noticed that this disk shows "Active" all the time. I've seen lots of similar reports though, so I wasn't really concerned about it. Currently, Dashboard is showing disk 1 (from) is active and disk 9 (to) is inactive. I think that's a good indication that nothing is moving.

Link to comment

I ran the command you posted. Not sure it did anything though-

Last login: Mon Aug 10 19:15:46 on console
maggie:~ wgstarks$ telnet 10.0.1.20
Trying 10.0.1.20...
Connected to 10.0.1.20.
Escape character is '^]'.
Password: 
Login incorrect

Brunnhilde login: root
Linux 4.0.4-unRAID.
Last login: Sat Aug  8 16:17:05 -0400 2015 on /dev/pts/0 from 10.0.1.190.
root@Brunnhilde:~# ps aux | grep rsync
nobody   19528  0.0  0.0  13488  2620 ?        S    Aug10   0:00 find ./Media/XBMC -depth ( ( -type f ! -exec fuser -s {} ; ) -o ( -type d -empty ) ) -print ( -exec [ -d /mnt/disk9/{} ] ; -o -exec [ ! -e /mnt/disk9/{} ] ; -o -exec echo Duplicate skipped: {} ; -a -exec false ; ) -exec rsync -dIWRpEAXogtl --numeric-ids --inplace {} /mnt/disk9/ ; -delete
root     25100  0.0  0.0   5100  1752 pts/0    S+   10:04   0:00 grep rsync
nobody   32765  0.0  0.0  17184  2492 ?        S    Aug10   0:00 rsync -dIWRpEAXogtl --numeric-ids --inplace ./Media/XBMC/Movies/Captain Ron (1992) HDTV/extrafanart/mNwDE6QY2VqWilIJSLk8Rj7tDgK.jpg /mnt/disk9/
root@Brunnhilde:~# 



 

Did i need to run another command also to kill the process?

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.