Rebalancing (split level) conceptual / practicle?

[chuck23322]

Ok,  sue me -- I got all the way up to a 13 drive 29tb Unraid server and then realized what the fuss about "split level" was all about.

 

I've run into the problem when a DVD rip stutters bad or completely fails during playback (and on re-start it works) because the needed disk for the "next" part must get spun up.  I've also run into it with some of my photo directories where I've added more photos, and they are spread across multiple disks and it gets painful or just annoying to spin up the other drives at random and wait... 

 

Now that I realize that I need to rebalance (not sure the right word here) the server so that I have files within the subdirectories all on the same drive -- I'm wondering on the right mechanics to do this.

 

What I've pondered doing is something like this:

 

1) Set the split-level properly

 

2) Starting on Disk1 -- rename the /mnt/disk1/DVDs folder to say -- /mnt/disk1/DVDs-temp

 

3) Then execute a "mv /mnt/disk1/DVDs-temp/* /mnt/DVDs" -- having unraid then "move" all the files from the referenced physical disk (Disk1) to the "share" DVDs -- and let the split-level logic take hold

 

4) Rinse / repeat for all the physical disks (disk2, disk3...) and again for each of the 'shares' -- 'Photos', 'Videos', etc, etc

 

I want to set this up to execute on the unraid server itself -- to run 20+ terabytes across the network wire makes no sense....

 

And, of course, I want to do it 'safely' -- eg, if for some reason it burps, or whatever -- I don't want to loose anything....

 

 

Is this conceptually correct?  A better way to do it?

[Frank1940]

 

First thing, I would do is to start is to find out how bad the problem really is.  I would suspect that you have only a small percentage of the DVD folders with the problem.  If this proves to be the case, I would make an inventory list of those disks.  I would then copy those off the server.  Then delete the folders from the server and copy the folders back.

 

You may run into a bigger problem with the photos.  I, personally, consider to be a minor issue compared to the DVD problem...

 

By the way, When you set the split level, you also need to set the "Minimum Free Space" setting on the same page.  You can find information about both these settings here:

 

  http://lime-technology.com/wiki/index.php/Un-Official_UnRAID_Manual#Min._Free_Space

 

Hope this helps

[BobPhoenix]

What I did recently was the following:

[*]Install cache disk - didn't have one.

[*]Make sure cache was NOT enabled on any share and mover was turned off by setting time to be once a month. (yes I know there are other ways to turn it off now however still would have to find them again)

[*]In default GUI browse the share and find those directories that have files on multiple disks since the disks the directory is on is listed to the right.

[*]Get into MC on the console and copy those directories to the cache drive with same folder structure as the share that way the files remain playable and in the same location from the shares perspective and any PC accessing the share (HTPC for example)

[*]Set split level and minimum free space correctly if I hadn't already

[*]Copy files back to user0/share/folder with MC but could just turn on mover and let it do it over night too.  You could also copy back to a specific disk if like me you like to control which disk the files end up on regardless of the split level settings.

By doing it that way the files are never offline and always usable while copying

[chuck23322]

Ah Bob -- hmm... while that means copying things twice -- once onto the cache drive, and then letting the mover put them back somewhere -- I do like the concept.  To your point, keeps the files "available" ....

 

I'll have to break up the moves as I have a 2TB cache drive, and effectively have about 2TB data free at the moment as well.  Would need to do it in <2TB chunks.

 

Your concept gives me something to mull over as a possible better solution -- albeit adding more 'time' involved -- but has benefits to the process.

 

I think I'll experiment with a small subset of my DVD files.

 

---

 

Frank -- agreed, it might not be as bad as I thought -- but due to laziness vs number of files/directories to analyze vs. missing something by hand -- I'm not sure I want to go down the road of figuring out "what's good and what's bad" about the allocation.  I'm OK with treating "everything" as bad and just letting split-level / move (or cache/mover) fix it....

[chuck23322]

What I did recently was the following:

[*]Install cache disk - didn't have one.

[*]Make sure cache was NOT enabled on any share and mover was turned off by setting time to be once a month. (yes I know there are other ways to turn it off now however still would have to find them again)

[*]In default GUI browse the share and find those directories that have files on multiple disks since the disks the directory is on is listed to the right.

[*]Get into MC on the console and copy those directories to the cache drive with same folder structure as the share that way the files remain playable and in the same location from the shares perspective and any PC accessing the share (HTPC for example)

[*]Set split level and minimum free space correctly if I hadn't already

[*]Copy files back to user0/share/folder with MC but could just turn on mover and let it do it over night too.  You could also copy back to a specific disk if like me you like to control which disk the files end up on regardless of the split level settings.

By doing it that way the files are never offline and always usable while copying

 

Bob, one thing hit me -- you're talking about _copying_ to the cache drive and then, potentially, letting the mover script put things back onto the data drives.

 

Two thoughts/questions:

 

1) Aren't you ending up (for a period of time) with duplicate files?  One copy on the data drives and one copy on the cache disk?    Probably not that big of a deal since they are identical files...  but conceptually that's bad, I think.  But probably minor or irrelevent issue in this exercise...

 

2) I was under the impression that if the file (or directory at split-level) already existed on, say "disk1" then split-level wouldn't apply.  Eg, if split-level would have normally put the directory (as a new directory) on "disk2" -- but since there  is already a directory on "disk1" (because of it being a copy not a _move_ to cache) -- then I wouldn't be getting the split-level logic at work.  I was assuming that I would have to ensure that the directories/files were being created "new/fresh" for the "split-level" logic to take hold.

[BobPhoenix]

Ah Bob -- hmm... while that means copying things twice -- once onto the cache drive, and then letting the mover put them back somewhere -- I do like the concept.  To your point, keeps the files "available" ....

 

I'll have to break up the moves as I have a 2TB cache drive, and effectively have about 2TB data free at the moment as well.  Would need to do it in <2TB chunks.

 

Your concept gives me something to mull over as a possible better solution -- albeit adding more 'time' involved -- but has benefits to the process.

 

I think I'll experiment with a small subset of my DVD files.

Understand.  I put my cold spare in the box and used it to copy to the temporary cache drive so I had extra space.  I actually went to individual drives not user0.  So after I got as many of the directories off the array drives to the cache as I could - I would move the remaining directories off an individual array drive to additional drives (usually just one but some times multiple other drives) and make that drive empty.  I then copied from the cache drive back to that drive.  Then repeated the process with directories that were on multiple drives until all directories were on single drives and my temporary cache drive was empty.  Then I removed the cache drive after I precleared it again so that it could become my cold spare again.  I probably moved some of the files several (5 or 6 maybe) times before it was all done.

[BobPhoenix]

What I did recently was the following:

[*]Install cache disk - didn't have one.

[*]Make sure cache was NOT enabled on any share and mover was turned off by setting time to be once a month. (yes I know there are other ways to turn it off now however still would have to find them again)

[*]In default GUI browse the share and find those directories that have files on multiple disks since the disks the directory is on is listed to the right.

[*]Get into MC on the console and copy those directories to the cache drive with same folder structure as the share that way the files remain playable and in the same location from the shares perspective and any PC accessing the share (HTPC for example)

[*]Set split level and minimum free space correctly if I hadn't already

[*]Copy files back to user0/share/folder with MC but could just turn on mover and let it do it over night too.  You could also copy back to a specific disk if like me you like to control which disk the files end up on regardless of the split level settings.

By doing it that way the files are never offline and always usable while copying

 

Bob, one thing hit me -- you're talking about _copying_ to the cache drive and then, potentially, letting the mover script put things back onto the data drives.

 

Two thoughts/questions:

 

1) Aren't you ending up (for a period of time) with duplicate files?  One copy on the data drives and one copy on the cache disk?    Probably not that big of a deal since they are identical files...  but conceptually that's bad, I think.  But probably minor or irrelevent issue in this exercise...

Should have said MOVE some people prefer to copy and then delete the original files so I get in the habit of saying copy when MOVE is what I really meant.

2) I was under the impression that if the file (or directory at split-level) already existed on, say "disk1" then split-level wouldn't apply.  Eg, if split-level would have normally put the directory (as a new directory) on "disk2" -- but since there  is already a directory on "disk1" (because of it being a copy not a _move_ to cache) -- then I wouldn't be getting the split-level logic at work.  I was assuming that I would have to ensure that the directories/files were being created "new/fresh" for the "split-level" logic to take hold.

By moving (sorry about that) the directories they disappear from the array drives when MC moves them to the cache drive so now they ARE new directories to the array since it doesn't exist on any drive yet.

[chuck23322]

 

Should have said MOVE some people prefer to copy and then delete the original files so I get in the habit of saying copy when MOVE is what I really meant.

 

By moving (sorry about that) the directories they disappear from the array drive when MC moves them to the cache drive so now they ARE new directories to the array since it doesn't exist on any drive yet.

 

Gotcha -- MOVE -- am in-sync with what you did now  

 

And thank you for interjecting your methodology...  probably wouldn't have thought of that way, and I like the idea.  Better yet, that somebody did it and it worked