Ok, I'm almost ready to jump on this bandwagon called unraid but I have some questions that I'm hoping this great community can help out and give me a sanity check.  Sorry for the long post but I want to get my point across and explained.

 

I guess I should give you a little background first.  I've been doing things since DOS 3 was around, I know all Microsoft OSes, and hardware is my specialty.  I've been using RAID arrays forever it seems and I'm really interesting in this unraid product to save my data but also save my power bill.  I can spell Linux and that is about it but I understand pretty much everything and I can wrap my brain around lots of things quickly but this split level thing is kicking my butt and I think it's all because of the way I've always stored my data and being solely on these Windows based RAID arrays.

 

Currently I have 5 NAS boxes.  I have a Acer Windows home server which is where my sabnzb runs as well as backs up my workstations (four 400gb drives), a dlink 2 bay DNS-323 (two 1tb drives) which has nothing but my kids media rips, a dlink 4 bay DNS-434 (four 1tb drives) which has hardly ever used media rips and then two home built windows server boxes.  One is my master media server.  It has (4) 2tb drives in it in software raid 5.  It also has another (4) 2tb drives in raid 5 that replicates the first array.  My second windows server is the same way but instead has all eight 1.5tb drives in it, also mirrored.  I do this because I don't want to lose any of this data.  I'm not worried about file deletion, it's locked down pretty well and in 8 years that I've ran this way, though multiple hard drive upgrades in the process, I've never had an issue and if I did I could always go back to the mirrored data before it replicates and pull what I need back.  All of these servers run 24x7 minus the dlinks which shut drives down.

 

Now, with all that said, let's get to the hardware in the new unraid server and then my questions.

 

I plan to use a Gigabyte GA-MA785G-UD3H motherboard with 6 sata ports onboard.  I have three processor choices.  An AMD LE-1600 single core, Semprom 140 single core, and a 4850e dual core.  All are 45w TDP.  My thinking is that file serving and storing isn't that demanding so I plan to use the LE-1600 unless someone explains otherwise to use something else (my current media servers all run the AMD E-350 dual cores and yes I do like AMD but I have lots of Intel stuff as well, this is just what I have laying around).  I plan on getting the 24 bay Norco case eventually but right now I only have 11 new drives to put in this new server.  I also will purchase this LSI 9201-16i card (http://www.newegg.com/Product/Product.aspx?Item=N82E16816118142).  Hard drives will be all 750gb with the exception of the parity drive being 2tb.  I am currently using the free version of unraid to pre-clear all the drives.  Eventually I will want to create another server just like this new one so I can continue replicating the data to and will most likely be offsite this time.  Call me crazy if you like, I just believe in several ways of redundancy. 

 

Now with the hardware out of the way, and let me know if anyone sees an issue with the hardware, it's on to the way I store my data and how to go about setting up the new servers split level which I can't understand.

 

On my main media server I have:

 

HD_movies

--> movie filename

 

Movies

--> Movie name folder

------->Movie filename (sometimes this will also be a VIDEO_TS folder on ones I haven't converted)

 

TV_shows

-->Show name

------->Season x

------------>Episode filename

 

mp3s

-->Artist name

------->Could be song or could be album name

------------>song name if the above is an album name

 

Photos

-->Description folder name

------->image files

 

Unsorted

-->mix of whatever I haven't sorted yet that will eventually be moved into the above folders

 

Stuff

-->random stuff that I rarely use and don't care what drive they are on

 

 

Now on my second server, it's all a bunch of random folders and files and I don't care what happens to those files honestly as long as they are somewhere.

 

So, again sorry for the long email but hopefully someone will take the time to read through this and decide to help me out.

I went ahead and purchased the pro version.  I still would like some input on what processor to use and if my hardware selection is ok but more importantly how to set up split level for my configuration.  If anyone could help me out it would be greatly appreciated.

 

The MB looks ok, just hope it doen't have the HPA-bug. For only fileserving a single-core processor will do fine (I'm leaning towards the Sempron).

I wouldn't spend that much on the LSI-controller, I would get a couple of IBM M1015 of ebay for about half the price, if available. You should go with 5.0beta if you want a LSI controller. I have a M1015 and 5.0b12a.

 

That split-level caused me headches too. Right now I use a Split-level of 2 and a Allocation method: Most-Free.

 

Works for me. I have a few folders named Blu-ray, DVD, MKV....and more

Inside the folders every title is in a separate folder. Blu-ray and DVD either are in folder-structure or ISO.

I did this mostly by trial and error and asking here.

 

This helped me:

http://lime-technology.com/wiki/index.php/Un-Official_UnRAID_Manual#Split_level

I'm still new at this, so take my advice with a grain of salt, but here goes anyway.

 

I may have missed it, but make sure you use a single rail power supply. I haven't figured out how to post links yet, but there is a wiki page devoted to PSU selection. That motherboard is listed on the compatibility page, but you have to disable HPA in the BIOS. Not sure if that means you can change the default or not, but it would be worth looking into.

 

As far as your split levels go, you should have a fairly easy time with it since you have so many discrete folders. From what I understand of it, the split level must be set at the lowest folder in the tree that you can tolerate your files being split over multiple drives. So you wouldn't want the movie files split over drives because a movie might need to spin up another drive in the middle of a movie. You've probably figured that out by now, though. Are you storing your movies as .iso's? The structure you have looks like you don't have your movies stored in individual folders, correct?

 

On my main media server I have:

 

HD_movies

--> movie filename

 

Not sure about this. Are you storing your files as .iso's?

 

Movies

--> Movie name folder

------->Movie filename (sometimes this will also be a VIDEO_TS folder on ones I haven't converted)

 

SPLIT LEVEL = 1 - This would allow different movie folders to be stored on different drives, but not allow the files within the folder to be split. E.g. Movie_name_folder_1 could be on one disk, but Movie_name_folder_2 could be on a different drive.

 

TV_shows

-->Show name

------->Season x

------------>Episode filename

 

SPLIT LEVEL = 2 - This would allow a show to be split over multiple disks, but would put each season on a single disk. E.g. Show_name_1 could be on one disk and Show_name_2 could be on another disk

If I understand this correctly, a SPLIT LEVEL = 3 would allow episodes to be split over multiple disks. However, if each episode is a single file, that may not be a problem.  I'm not really sure about that.

 

mp3s

-->Artist name

------->Could be song or could be album name

------------>song name if the above is an album name

 

SPLIT LEVEL = 1 - If I understand your structure, I think you would be stuck with a level 1 split. This would put all the songs by a single artist on one disk.E.g. Artist_name_1 could be on one disk and Artist_name_2 could be on another disk. If everything had the album name folder, you could go with a level 2 split and that would put albums on a single disk

 

Photos

-->Description folder name

------->image files

 

SPLIT LEVEL = 1 - This would put everything in a  specific "Descriptions folder name" on a single disk

 

Unsorted

-->mix of whatever I haven't sorted yet that will eventually be moved into the above folders

 

Is everything in this folder in it's own folder? If so, then you could go with a split level = 1. Then the random folders would be stored on a single disk.

 

Stuff

-->random stuff that I rarely use and don't care what drive they are on

 

Same as above

 

 

Again, hopefully someone will chime in that's a little more familiar with this than I am, but this is how I understand the split level business.

 

Welcome to the madness

 

EDIT: Keep in mind that split levels work in conjunction with the share's "Min free space" setting to ensure you don't try to copy a file to a drive that exceeds the remaining space.

I appreciate the responses.  As far as power supplies, I only use PC Power & Cooling power supplies which have single 12v rails.  It's the first time I've heard about the HPA issue so I'm looking into that.

 

I've looked at the wiki and every post on split level and it just doesn't make sense to me.  I look at my arrays and the drive letter is there, I put stuff in the folder I want to and never have to worry about it again or about what drive it's on because they are in RAID 5, though all my drives are spinning 24x7 also.  What I don't get is say for instance a season of a tv show isn't over, and I'm not done putting files in that folder, how does it know to continue to put those files on that hard drive, what if it fills up before I put my last file there and doesn't have space for the new files, does it create another folder on another drive and start putting them there or move the whole season to another drive?  I rarely have any .iso files so the VIDEO_TS folders very seldom get used so I'm not too worried about some of those files not being on a certain drive, all my hd movies are not in their own folder but they are all .mkv, .avi, .ts, etc under that one folder name.

 

Most of the mp3s can all fit on one disk tbh.  Most of the regular movies could as well.  It's the HD movies, and TV shows that will have to span multiple disks.  All of the other random stuff I could care less where it is as long as it's on a disk.

 

So if I understand this correctly, I should create a share called HD_movies and set split level to 1.  Create a share called Movies and also set split level to 1.  Then the only one I really have to worry about is on TV_Shows, I create that share and set split level to 2?  All other shares I create as 1.

 

For someone who's been around this stuff for a long time I don't know why this is so hard for me to understand.

This is one of those things that was a bit of a surprise to me as it's not really advertised. Essentially, you have to specify how much free space you want a share to leave on a drive. That way unRaid will not try to write to that share, on a drive that is below that limit. So you would set your HD_Movies share to have a limit that's about twice the size of the largest movie folder you expect to put there. Then if a drive has exceeded your min free space setting for that share, it will use a different drive the next time you copy a file. Unfortunately, unRaid isn't "smart" enough to move files if you run out of space, and you will get an out of space error and your copy will fail if you add data to a folder that doesn't span multiple drives.

So I've been playing around with some spare disks before I go with live data and drives and I have one question.  Say I have 4 400gb drives with a 400gb parity drive and I have a share called TV_SHOWS with a split level of 2, highwater of 20000000 and it's about 1tb in size.  I have a share called PICTURES with a sl of 1, hw of 20000000 and it's about 60gb in size.  I have a share called MP3s with a sl of 1, hw of 20000000 and it's about 30gb in size.  When I start my migration of data would I copy the PICTURES and MP3s folders first because they would reside on the first hard drive since nothing is on the drives yet and then start the copy of the TV_SHOWS next or reverse it....or does it really matter what I copy at any time?

It doesn't matter.. if they're all set to a high water of 20000000, any of those will simply switch to the next disk once it hits that figure.

Here's what I don't like about this product.  I copied 1.2tb of information into that TV_SHOWS share I created.  Then I copied 15gb of data into the PICTURES share, then copied 70gb of data into the MP3s folder.  All copied fine.  Then I decided to add another 400gb drive to the array.  I then proceeded to copy another 460gb of data into the TV_SHOWS share and it never touched the new drive, instead it it filled the other drives up, one drive it completely filled up copying a season to it because of the sl being 2 it had to keep that season on that drive.  Now that one drive can't be used for much of anything and is way less than the hwm.

 

So the system is pretty dumb on how it places data because how does it know how much data is going to be in one season of a show...it doesn't, so it just keeps filling up those drives.  It has yet to start using the new 400gb drive, was I supposed to do something to it do make it start using it for those shares other than adding it to the array?  I created the shares with no include or exclude of drives.  IMO this is where there should be some mover app that rearranges the data on the drives at night to make more sense.

That's how it can happen. If you want to keep the data organized in some fashion on the drives then you have to accept that drives may end up full. That is one difference between a striped storage solution and having individual file systems on every drive.

 

Personally, I keep complete TV series confined to a single disk by using split level = 1 on my TV series. So, I occasionally have to shuffle a few TV series to another disk.

 

The unRAID storage decision process is basically like this. Compare the split level setting to the file directory structure of the new file vs the existing share directory structure. If the file must go to a certain disk due to the split level then it goes to that disk. If the file is not confined to a certain disk by the split level, then the allocation method is used to determine which disk to use. So, if a TV series is confined to a certain disk by the split level then unRAID will either place the file on that disk or give you an out of space error.

 

Here's a hint that can help but might be too late. If you use the fill-up allocation method then it will jump around between drives as the split level allows. In your case, the first new TV season placed goes to the most free disk. The next new TV season placed goes to another disk with the most free space. And so on and so on.  Then, the new TV seasons are more likely to fit because all the new TV seasons don't end up on a single disk to start, they are spread around.

 

Look at it a different way, say you started 5 new TV seasons in the fall all at about the same time. Most likely, all the new seasons would end up on a single disk due to the high water allocation method currently partly filling a certain drive. Then, as these seasons fill later in time, the disk could become full due to all those new episodes being written to it.

 

Finally, if you don't care about the structure per disk then set all your split levels to 99 or some other high number. Your server will then fill until it's full.

I've been playing with this for a few weeks now with spare drives so I can figure this out whether or not to use this product or not, I deleted the data and recreated the shares.  This time I copied the smaller directories to the array first and then the large directories after and it seems to work out better, so to me deciding on when to copy certain amounts of data is important.  I also found out that when you add a new drive to the array you must reboot for it to start using it, I added new drives and it never used them but as soon as I rebooted it started using them.  I don't think that's stated anywhere is it?

 

Now, I have already bought 2 copies of the pro software, but my goal was to figure out if it's right for me or to choose something else.  Just like everyone else I want to make sure if I'm going to change the way I've been storing data for all these years it's reliable, makes sense and is easy to use.

 

So, with that said, I'd like to see responses from people on why they went with unraid instead of WHS w/drivepool or another product.  I'm testing that product as well, mainly because I have a WHS v1 server and I have had zero issues out of it and it makes sense to me since I live in the Windows world.  Don't get me wrong, I'm not bashing this product in any way, I'm simply stating pros and cons that are issues of mine, but probably not someone else's.  We all test products differently and ultimately the decision will be mine but I'd still like some input from people on this board and I don't want to give up on this product.

 

Here's the pros I see for unraid.

 

1) it spins the drives down that are not in use

2) it boots off a usb stick and doesn't require a hard drive spinning constantly for the OS

3) runs on about any hardware

4) remote web administration

5) add any size drive

 

Cons:

1) takes a while to get it configured how you want it

2) learning curve can be steep if you don't know linux

3) shares are hard to understand

4) drives can fill up without much notice due to how the shares are set up and how it copies data

5) no software to move files efficiently to other drives for balancing

6) adding drives requires stopping array, rebooting server and restarting array

7) could require constant monitoring of the shares

 

 

 

 

So, with that said, I'd like to see responses from people on why they went with unraid instead of WHS w/drivepool or another product.  I'm testing that product as well, mainly because I have a WHS v1 server and I have had zero issues out of it and it makes sense to me since I live in the Windows world.  Don't get me wrong, I'm not bashing this product in any way, I'm simply stating pros and cons that are issues of mine, but probably not someone else's.  We all test products differently and ultimately the decision will be mine but I'd still like some input from people on this board and I don't want to give up on this product.

 

 

I will give you my two cents, I was a WHS V1 user for a couple of years, I went for a different solution because micrsoft decided to drop the one and only added value of the system in the new VAIL version (drive manager). I know there are 3rd party solutions to fit that gap but that did not feel comfortable with me, I have had two occassions with serious errors with WHS where I really needed the supportline and I feel that having that support split in different companies would be an issue.

 

Remaining with the previous version was no option also because the windows version was so old that it took more and more tinkering to keep my functionality up (I needed to change itunes installation packages to get it installed for example). 

 

It was also getting to be quite expensive, I wanted to keep redundancy and the only way WHS could do that is by duplicate storing files, it simply stores the files on two different disks. That means that the amount of storage you have available is basically half of the amount of drive space you have. And then price is only one thing. I had an HP system with 4 internal drives and added an external esata drive cabinet with 4 more. All drives running 24/7 making a lot of noise. I was not able to add more drives to the system (unless I would attach an external USB drive which would mean ANOTHER drive cabinet, ANOTHER power adapter, etc. etc.)

 

The disk doubling was also a big hassle with the failures I had, I have had to resetup my system two times because the DM system got mixed up, I had to do a completely new setup copying all data back to the WHS system, but because you have no control on what data is doubled to what drives it meant that I basicallt had to copy all data two times (with a lot of "replace" copy actions).

 

In short: I needed new hardware, AND new software. So I could look to a "greenfield" new solution

 

Unraid fits my needs in that it solves my specific issues (but it might not do that for a different user):

 

- The support system is great (but that is because I like community based support like it is delivered here);

- I like the amount of influence I can have on the system config wise but also "on the background" ( I like being able to copy data between disks to "balance" myself)

 

(the above beiing sad: I think a plugin that balances data over the drives in a user share would be really nice, also for data redundancy, I prefer to have two half full drives instead of one full and one empty)

 

- The powerusage is really low, I kind of make it a hobby to get my electricity usage as low as possible without going back in comfort, and unraid has helped that also. With WHS I had 8 drives full time running because I had the system downloading torrents, with unraid I have one drive allocated to a share that receives that info, I have disabled cache drive for it. On that drive I have also located my music collection and user files, so in 80% of all cases only this one drive will be spun up. Cache-dirs make sure that even when I browse ALL my files my disks remain spun down.

 

 

Here's the pros I see for unraid.

 

1) it spins the drives down that are not in use

2) it boots off a usb stick and doesn't require a hard drive spinning constantly for the OS

3) runs on about any hardware

4) remote web administration

5) add any size drive

 

Agree on all, see above for what I feel are extra pro's

 

 

 

 

Cons:

1) takes a while to get it configured how you want it

2) learning curve can be steep if you don't know linux

3) shares are hard to understand

4) drives can fill up without much notice due to how the shares are set up and how it copies data

5) no software to move files efficiently to other drives for balancing

6) adding drives requires stopping array, rebooting server and restarting array

7) could require constant monitoring of the shares

 

For

 

1) Partly true, but that is due to the tinkering thing... If you don't do all the things that unraid lets you do and you cannot do in others you do not need a lot of knowledge. But I do agree that I have used some of my nearly forgotten linux knowledge of 10 years ago.

 

2) If you use unraid pure vanilla then that is not true imho... Install it, add user shares and you can basically just let it running without messing with it.. The fact that you can do so much more makes you do those things and that leads to the need of more knowledge.

 

3) Why ?  That is rediculously easy... Only thing that might need a bit of thinking is the split level, but you can also just nog bother about that, there is absolutely no way to control this on whs, if you do not touch it your data will just be spread over the whole array. It works like whs than without the need for double drives.

 

4) That is because you look at the drives, If you want to see the system as a black box you should not worry about that, the system fills up drives and will switch to a new drive if the first one gets full. But here also, the ability to control this is nice, but if you want to do this you will need to spend a bit more time.

 

5) See 4), if you want to not be bothered with administration then there is no need to do this.. And if you want to bother it actually is very easy and 100% controllable, just telnet in and use midnight copy (but as said: a nice plugin that automates some stuff would be nice, I agree)

 

6) This is partly correct, if you have hardware capable of hot swapping and you are using unraid v5 then you only need to stop the array, no need for a reboot. I do not really mind, since unraid is mostly home-build it happens that cables get rocked loose and I am happy that the array does not automatically responds with actions when that happens, could give more problems then it solves.

 

7) No, not needed, unless you start using the system in  a way that makes this necessary (like having a share on only one drive)

 

That's my two cents !

 

This is exactly the type of response I was looking for, thank you.  Now let me play devils advocate.  I too am playing the "how much can I lower my electricity bill" game as well but still get what I want accomplished.  All that I stated above are running 24/7 which is one of the main reasons I am looking for something new.  The new WHS w/drivepool can shut the drives down so they aren't running and spins only the drives up that it needs and it moves files at night for balancing.  You're not locked in with how many drives you can use either plus it keeps a copy of the directories in memory like cache-dirs.  But you do have to have at least one 160gb drive for the OS which will spin all the time but I could use that for downloading I guess just use a larger drive than 160gb.

 

You are correct, with unraid I can look at what's on a particular disk and whs you can't.  If you read my original post I state that I keep my data on two different NAS servers so the data is mirrored.  I plan on doing this with the new systems as well.  I know what you're thinking, why not just have one whs system and have whs duplicate the data.  This way I'm not limited by a hardware issue or total array failure.  I would have two unraid systems that mirror each other or two whs servers probably using robocopy like I'm doing now.  If something happens I can easily switch to the backup and get the other one fixed.  Heck, I might have to play with one of both to see how they work together.

 

Point is, both unraid and whs w/dp have pros and cons and I'm not trying to say whs is better and unraid sucks or vice-versa.  I'm not leaning toward any one software right now, just stating that they are very similar now and coming from a windows world what makes more sense to me and I don't have to worry about in the future.  I think I might have to try that option of having one of each so I have the best of both worlds.  As long as I know how to replicate the data between the two, that may be the way to go for me.

 

For your cons

 

2 - There is nothing hard about running a web interface. I think you're more concerned with running other apps or certain troubleshooting.

 

3 - Not sure what you mean. The split level and idea behind the storage may be a little harder to grasp compared to a single striped array. Maybe you're really just over thinking it.

 

4 - Don't forget that this is basically self inflicted due to your choices for share settings. Set the split level to a high number and this isn't an issue.

 

6 - Adding a new drive - did you try just stopping and starting the array?

 

I have a friend who built a server like mine because he liked the setup. He's constantly having Windows PC problems which are often self inflicted. His server on the other hand has been rock steady for about 4 years now.

 

You are correct, with unraid I can look at what's on a particular disk and whs you can't.  If you read my original post I state that I keep my data on two different NAS servers so the data is mirrored.  I plan on doing this with the new systems as well.  I know what you're thinking, why not just have one whs system and have whs duplicate the data.  This way I'm not limited by a hardware issue or total array failure.  I would have two unraid systems that mirror each other or two whs servers probably using robocopy like I'm doing now.  If something happens I can easily switch to the backup and get the other one fixed.  Heck, I might have to play with one of both to see how they work together.

 

 

 

The nail on the head is that no product is the ideal product.. Unraid is not the -ideal- product for everyone simply because such a product does not exist. You have to check what stuff is important for you and then make your pick. I did and Unraid won from WHS, someone else might go for WHS or for a basic NAS solution.

 

The important thing is that you investigate (and I see you doing just that), jumping into something without doing investigations is what will cause unhappiness in the  end.

 

Lots of luck with your quest !

I hate to bring up an old thread, but, I'm almost done with my testing of all the potential software for my two new servers.  Without getting into details my final results are that I am planning on running two unraid servers in a mirrored fashion (if I can figure out rsync) as well as a windows home server running drivepool.  I think I know the answer to these last questions which will be a no, but my last two questions are....  Say I have six 2tb drives, one being parity.  If one drive fails, can I add two 1.5tb drives to the array as a replacement if I don't have another 2tb drive to replace it with?  Or, if I have those six 2tb drives and I have four 1.5tb drives already in the array but those are empty will unraid move the data or reconstruct parity to those empty drives when a 2tb drive fails?

I hate to bring up an old thread, but, I'm almost done with my testing of all the potential software for my two new servers.  Without getting into details my final results are that I am planning on running two unraid servers in a mirrored fashion (if I can figure out rsync) as well as a windows home server running drivepool.  I think I know the answer to these last questions which will be a no, but my last two questions are....  Say I have six 2tb drives, one being parity.  If one drive fails, can I add two 1.5tb drives to the array as a replacement if I don't have another 2tb drive to replace it with?

Sorry... no, not unless all the data drives are 1.5TB or smaller. quote]Or, if I have those six 2tb drives and I have four 1.5tb drives already in the array but those are empty will unraid move the data or reconstruct parity to those empty drives when a 2tb drive fails?

unRAID NEVER moves any data.  If you have a single drive failure, (a write to the drive fails, for ANY reason) it will be marked as not-valid and disabled.  All further activity to that drive will be simulated by reads from the remaining drives in combination with parity.

 

In many cases, new users are surprised to see they can still read and write to a drive that has failed. (or even physically disconnected)

 

You can, if yo wish copy the entire failed (simulated) drive to other working data drives in your array.  If however, you have a concurrent second failure of a second disk data on both the failed disks could be lost.  Data on the remaining disks is safe.

 

It is up to YOU to migrate any data.  You do not need to migrate any data, but you are at risk of loss of data should a second disk faile while a first is marked invalid.. 

 

unRAID (or and RAID) is not a backup of data.  it is a way to prevent a single hard disk failure from causing loss of data.    Read here:

http://lime-technology.com/forum/index.php?topic=2601.msg21033#msg21033

 

then, as time permits, check out the rest of the wiki, especially the user-contributed section.

http://lime-technology.com/wiki/index.php/Unofficial_Documentation