Script to redistribute/rebalance disk usage


Recommended Posts

Hello all,

 

EDIT:

Now that I have had a chance to observe some things on my array, I've had a pretty significant change of heart about this script. DON'T USE IT, unless you have a really good reason.  I wrote it after making a lot of changes to my server, adding activity LEDs for each disk, and upgrading 3 of the disks to larger ones.  And I was trying to "get settled into the new config" by balancing the data across the disks.

 

Here are a few things I've noticed/learned since then (in order of importance IMO):

 

1. Free space fragmentation is a HUGE write performance hit, as in, instead of 80-85MB/sec, it's down to 10-15MB/sec in some cases.  And moving files between disks causes a lot of free space fragmentation.  Based on observing the activity LEDs and watching write speed fall through the floor, it appears that free space fragmentation causes substantial additional IOs on other disks in order to calculate parity.  In some cases, I was seeing a lot of activity on 3, 4 or 5 disks while writing a single large file.  I'd love to hear from the unRAID devs on this point.

 

2. It does actually really suck to have files scattered across disks.  It introduces significant lag when, for example, you are listening to a playlist that has songs on many disks, or binge-watching an entire TV series/season.  Each time a new (idle) disk is accessed, you have to wait for it to spin up, which can take a few seconds.  You can set the spin-down delay to something long, but that is a non-optimal workaround.

 

3. Once you scatter folders across disks, split levels are no longer effective, and the scattering gets worse.  The split level feature appears to look at what is already on disk and works within those limits.  So if there are folders all over the place, it has no bounds in which to hold itself and files are written anywhere there is already a folder.

 

I may actually dump my array or cycle out for empty disks in order to deal with these issues I've created for myself.  It's that bad.  I hope I haven't caused any headaches for others who have used this script.  Ugh.  #stilllearning

/EDIT

 

I created a rudimentary script I call 'balance.sh' for redistributing files in order to relieve space constraints on nearly full disks and/or help fill up a new or upgraded disk.  It's a very basic script, and you need to be sure it's safe for your installation, and modify it appropriately before trying to run it.  I am not responsible for any loss of data or other issues on your server as a result of using this script.

 

Re-balancing the free space of an unRAID array is a little controversial, because it sort of goes against one of the primary features of unRAID: power conservation.  But I do it because I don't want 'all my eggs in one basket'.  Data integrity is more important to me than power savings, and having files scattered across all the disks prevents a huge loss of any one kind of file in the event of a non-recoverable failure (as unlikely as that really is).  Really, though, that's mostly a smoke-screen excuse, and I'm just a little OCD and want all the disks to have similar free space.  ::)

 

My unRAID server has three shares: Media, Storage and Backups.  All of my shares use the 'Most Free' allocation method. I made the script run in only one share at a time in order to help limit the risk of running it.  And I am running 6.0-beta14b, with a six-disk array, plus a cache disk (not in use on any share).

 

This is a one-shot script.  It will move a maximum of one file each time it's run. I make it run continuously with the following command:

 

while [ 1 ]; do ./balance.sh; done

 

Hit Ctrl-C to stop it, BUT ONLY WHEN THE SCRIPT PROMPTS YOU TO DO SO.

 

Verbal breakdown of the script:

* Specify the share to run against.

* Set how much difference there must be between the fullest and emptiest disk before doing anything.

* Grab the current disk space info and pick it apart to find the largest and smallest free space numbers.

* Find the difference between them.

* If the difference is more than MARGIN, then proceed, otherwise just report we're OK and exit.

* Grab the current disk space info and pick it apart to find the disk with the least free space.

* Find all video files on that disk, randomly (un)sort the list and choose the first one.

* Rename the chosen file to file.tmp (avoids duplication).

* Wait one second for the file system to sync.

* Move the file from the disk location to the share location, which will cause it to write to the disk with the most free space.

* Report what file was moved and wait 5 seconds.

 

RISKS:

I'm running the script on my own server as I type this post and it's not having any troubles.  But here are some things to watch for:

* The script has the capacity to DESTROY DATA, or at least put it places that are hard to clean up.  Make sure you are paying attention.

* If you kill the script it may leave the file it was moving named as /path/to/file.ext.tmp.  You will have to rename it back to /path/to/file.ext manually.  It's best to wait for the 5-second pause at the end before killing the script.

* I made some assumptions about the disk and share layout that may not be consistent across installations. Verify before running.

* Currently the script only looks for files with extensions mkv, avi, mp4 and mov (case insensitive).  You will need to edit for other file extensions.

* If you are using a cache disk, you may need to keep an eye on its space and run the mover periodically

 

Customization suggestions:

* Add args to the find command to look for files of a certain size or range.

* Refine the find path to limit activity to a smaller section of the share.  This could be useful for re-balancing without being totally random.

* Remove the random sort in order to move a specific list of files.  You would have to do some testing to figure out what files find finds.

 

#!/bin/bash

SHARE="Media" # To help reduce risk, operate against a specific share only.
MARGIN=50000 # Number of MB required between the fullest and emptiest disk for anything to happen

LEAST_AVAIL=`df -hm | grep disk | awk '{ print $4" "$6 }' | sort | head -n 1 | awk '{ print $1 }'`
MOST_AVAIL=`df -hm | grep disk | awk '{ print $4" "$6 }' | sort | tail -n 1 | awk '{ print $1 }'`
#echo $LEAST_AVAIL
#echo $MOST_AVAIL

let AVAIL_DIFF=MOST_AVAIL-LEAST_AVAIL
#echo $AVAIL_DIFF

if [ $AVAIL_DIFF -gt $MARGIN ]
then
  DISK=`df -hm | grep disk | awk '{ print $4" "$6 }' | sort | head -n 1 | awk '{ print $2 }'`
  echo ""
  echo "$DISK has $LEAST_AVAIL MB available."
  echo "Which is $AVAIL_DIFF MB less than the disk with the most space."
  echo ""

  cd $DISK/$SHARE

  FILE=`find . -type f -iregex '.*\.\(mkv\|avi\|mp4\|mov\)' | sort -R | head -n 1`
  echo "Moving file $DISK/$SHARE/$FILE"
  echo "         to /mnt/user/$SHARE/$FILE"
  echo ""

  ls -l "$DISK/$SHARE/$FILE"

  mv "$FILE" "$FILE.tmp"
  #ls -l "$FILE.tmp"
  sleep 1

  mv "$FILE.tmp" "/mnt/user/$SHARE/$FILE"

  ls -l "/mnt/user/$SHARE/$FILE"
else
  echo "Disk utilization is balanced."
fi

echo ""
echo "Waiting a few seconds to allow you to cancel the script."
sleep 5

exit 0

 

And here is what the output looks like when it runs:

/mnt/disk1 has 289894 MB available.
Which is 310855 MB less than the disk with the most space.

Moving file /mnt/disk1/Media/./Movies/General/Harry Potter and the Chamber of Secrets (2002).mkv
         to /mnt/user/Media/./Movies/General/Harry Potter and the Chamber of Secrets (2002).mkv

-rw-rw-rw- 1 nobody users 6.7G Jul 27  2014 /mnt/disk1/Media/./Movies/General/Harry Potter and the Chamber of Secrets (2002).mkv
-rw-rw-rw- 1 nobody users 6.7G Jul 27  2014 /mnt/user/Media/./Movies/General/Harry Potter and the Chamber of Secrets (2002).mkv

Waiting a few seconds to allow you to cancel the script.

 

Enjoy!

Link to comment
  • 3 weeks later...

I thought this was a big issue in that you do not want to move from a defined disk to the user share level due to the potential loss of data.  I was under the impression you needed to move for disk_x to disk_y.

 

  echo "Moving file $DISK/$SHARE/$FILE"

  echo "        to /mnt/user/$SHARE/$FILE"

 

This could have been a big issue!  But there's a rename in the code, to avoid directly overwriting original file (which I completely missed on a quick examination!).

 

I do understand that the chances are somewhat small, in that it should copy from a full disk to a less full disk, but there's possibly still a risk that it could happen on the same disk, (perhaps because of split level logic?).  The risk needs to be investigated fully, avoided if possible, because files could be lost completely.

 

My apologies to koyaanisqatsi!

Link to comment

Sorry to freak people out.  The script does not move files directly from disk to share.  It moves first to a temporary file in place, then moves the temp file from disk to a real file on the share, renaming it to avoid file name duplication.

 

I did notice I omitted a sleep step and have added it back in.  The sleep gives the file system a moment to sync before trying to move the temp file to the share.  I ran into an issue once (but was never able to reproduce it) where the move from disk to share started before the move to temp file had fully synced.

Link to comment

Sorry to freak people out.  The script does not move files directly from disk to share.  It moves first to a temporary file in place, then moves the temp file from disk to a real file on the share, renaming it to avoid file name duplication.

 

And I apologize, I was late for something, and didn't examine the code closely, missed the rename.

 

So are we all convinced it's completely safe?

Link to comment

No problem, RobJ!  :)

 

If a sync causes the parity to be written, and the share's understanding of the file system state to be fully updated, then it will definitely help a little.  But the move operation works like a copy and delete because it's jumping mount points.  So you're having to wait for the data to copy anyway, and that extra second of time is negligible.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.