data "rebalance"

[ajeffco]

Hello,

 

I've recently replaced drives in my unraid array with larger drives.  I now have an "imbalance" in the utilization of my drives.  Some are nearly empty, while others are at >95% utilization.

 

I'd like to "rebalance" data across the drives.  I'm thinking an rsync from a full /mnt/disk#/sharedir to the /mnt/user/sharedir with a remove source and then delete the empty share sub-directories (but not the share directory itself) would achieve this.  Assuming this makes sense, my share allocation level is high-water, should I change it to most-free for this process?

 

Any thoughts or advice on this?  Is it even worth the effort?

 

*EDIT: I saw a couple of posts with re-balance.  One mentioned a cache drive, and then the mover would re-balance, but I don't have a cache drive.  There was also mention of a script to do this but I couldn't find it.

 

Thanks,

 

Al

[Joe L.]

That might do it... but you must use the newest of "rc6-test" betas that fixed a long existing bug when renaming a temp file (as rsyncs uses) to its actual name will delete the old original.

[ajeffco]

Thanks Joe,

 

I'm currently running 5.0-rc6-r8168-test.  I've got two older/smaller drives to pull still and then parity scan/check.  I'll post again with how it goes after all is said and done and the dust settles. 

 

Might even script it.

[opentoe]

This would be a cool plugin to have since basically a lot of people are always upgrading to larger drives which keep getting larger so quickly.

[lionelhutz]

I'm not sure that would work since the same data exists in both locations. You could rename the share directory on the disk and then run the rsync but that would empty the disk.

 

Why not open the source disk and the destination disk on your PC, pick a bunch of files from the source (about 300-400gig would be an appropriate maximum) and then drag and drop them to the destination. Once you do this, go to bed and it'll be done by the morning.

 

At any rate, I don't understand the need to move more than a 100 gig or so off any one disk. The only need might be with keeping each complete TV series together since new episodes could keep filling the disk. In those cases, you just kick some continuing series to a new disk when needed. Otherwise, just let the older disks stay fairly full and fill the new ones as you add new media. WHS with it's disk balancing routine has everyone believing each disk must be equally full at all times or everything goes bad. The newer disks will see more writing and plenty of reading as you access the newer media on them. At the end of the day, the older disks were accessed a fair bit as their media was new added and watched and now the newer disks get their turn at being used more as their media is added and watched.

[Helmonder]

The reason would be that balanced data reduces the amount of data lost during catastrophic disk failure.

 

Failure of one drive will be fixed by the parity drive, failure of two drives will result in loss of the data on both drives (but you keep the rest).

 

Knowing that there is a chance of loosing data on a drive it would make sense to have as little as possible data on each drive, and that is the same as a balanced system..

 

The comment on that though is that it statistically is not correct... Statistically you will loose the same amount of data on average (in a totally unbalanced system with one completely full drive and an empty you loose either 100% of all date (if you loose the first drive) or 0% of all data (if you loose the second). Average risk per disk is 50%. In a balanced system you would loose 50% in either case..

 

I have posed some questions on this before, issue is that all people that have reacted and are possible "plugin writers" are anxious writing such a plugin since it will mess around with data.. Error in the plugin could cause mass data loss.. I understand that also..

 

Personally I just do a telnet, start MC and have it run over night to mass copy a couple of directories to another disk if I feel I need to.. There is something inside of me that likes to see the disks balanced (though I realize it is basically nonsense)..

[jowi]

Another theory is just to leave the full disks alone, that will result in less writes and reads on those disks, which will result in a lower change of failure of those disks

[Helmonder]

And theorizing on the assumption that you are most likely to use data that was saved recently also not messing with the data would cause old drives to get very less access, meaning less usage in electricity and, indeed, wear and tear..

[Influencer]

Ill chime in, I've thought about making such plugin but I haven't because of the mentioned remark. It would be messing with data, moving around stuff and deleting from the old location. One missed ; : , /  [ ] ¦ '  and it could mean data loss. Of course testing before release would take care of most of those issues, but no guarantee it would find all the bugs. Considering most users utilize 1 TB drives at the smallest, the possibility of an oopsy on my part causing the loss of a minimum of 500 GB is a little much for me. Especially if the data in question are personal videos and not easily recovered.

 

Both theories for balanced data have their merits.

[BobPhoenix]

Remember Murphy.

 

The most important data drive is always the first to go.

[WeeboTech]

You can move files with rsync pretty easily. I do it all the time.

 

The only issue is a possible collision with share to share. If both files exist on the share, the rsync will not go forward.

 

You can do disk to disk easily. I do it all the time.

 

Your other option is to use the cache drive, or install one temporarily.

Move it to your cache drive with --remove-sent-files, then let the mover put it on the share.

 

I would not do it on a normal basis other then trying to keep certain types of movies together.

 

I.E. I have a couple drives for TV shows, scifi, animation, documentary, etc, etc. In other words my drives are set up by genres.