Need a dummy's guide to what unRaid server I need and how to get it

[steve317]

Hi, I posted here a couple of years ago and then had to backburner our unRaid project.  I'm back and looking to finally get an unRaid system set up asap.  I have a couple of questions, and I'm hoping you guys can help.  Here are my questions, and then the details:

 

1) Is unRaid even the right solution?  Part of what happened is that some people (including I think the Lime Tech folks themselves) were saying it's not optimal for video; but I know everything has evolved rapidly and I also don't know if I'm right about this.

 

2) What is the best server hardware for us and how do I get it?

 

 

 

 

OUR USE CASE:

 

We're an educational institution on an extremely low budget. We have a huge backlog of videos -- thousands of hours of our educational programs recorded over the past decade+ that for now will be used for internal training purposes but eventually we may edit into some public videos.  For now, our big challenge is to fit them all on a storage device, organize them, and access them easily (but not regularly -- so they aren't being constantly broadcast or reviewed 24/7).

 

We will have them all organized (and sometimes edited) on Final Cut Pro; with some old ones still in iMovie that will eventually be imported into Final Cut Pro.  The big issue is that they are currently strewn chaotically across 10-20 old 1-4tb external HDDs we've purchased over the years, some of which may be ready to fail, as HDDs do.

 

The videos (and Final Cut Pro library data and/or whatever else is related to storing media in FCP) will be stored on the unRaid.  I'm assuming the way we'll do this is then access the unRaid via the LAN from a Mac running FCP.

 

So, hopefully this clarifies what we need to build (below), and lets everyone be able to tell me if unRaid is unsuited to this. (Should we just buy some kind of NAS?  Should we just sell a couple of kidneys and buy some sort of SAN system?)  I keep looking at other things, and keep coming back to unRaid, but I still don't know if that's a correct judgement on my part.

 

 

 

 

OUR HARDWARE NEEDS:

 

We are on a budget, so, for example, if we can get something for $500 instead of $1,000, or $1,000 instead of $1,500, that's a big deal and we want to do so.  On the other hand I've seen some forum posts listing a home-made system where a person itemizes $300-400 for each of three different components plus other items.  At that point it would be cheaper to buy the one Lime Tech sells for $1,500.

 

And if the Lime Tech built option (currently listed at $1,500) is best for us, then we will do that.  Or if our needs require something more powerful, but it can be had for the same $1,500 but by building it ourselves, then we can do that.  We might also interested in some other person or company that sells unRaid hardware; but Googling for that has not returned obvious pathways to move on that plan. 

 

We're new to this kind of hardware build, so that's why if there's a fairly priced pre-built option, we're very interested.  On the other hand, we'd love for this system to cost $500 to well under $1,000, so we can buy more HDDs now.  But maybe our capacity needs mean we need to buy something higher end than $500 will afford.

 

Our capacity needs are that we'd like to be able to get up to that aprox 70TB+ capacity that a few of the systems have:  In other words, we want as many drive bays as possible -- not 5, prob at least 8, but prob much better to have at least 10 or more.  We don't want to and can't afford to build a new storage server in the near future, so we want this one to last a while; and our video collection grows fast.

 

Our competency is that I regularly open computer cases and do routine upgrades like HDDs/SSDs and RAM, with the occasional repair; I've been doing this and working with computers for 30 years, so while I've never built a PC from scratch, it's not a giant leap of experience or thinking for me.  That said, I haven't ever done it, so some of those list of parts needed just to get it operational (not to upgrade it!) are a little intimidating.  I'll do it if it makes sense and won't result in a terrible system, or I'll go the turnkey route and buy from some Lime or some other fine proprietor if that's best and affordable.

 

 

 

 

Thanks in advance for any help, expertise, insight, opinions, and/or proselytizing you can offer.

 

 

[c3]

If you have repaired PCs for 30 years, you have touched every part.

 

You know the parts;

case - you have touched many - this time you want a big one, for 13+ drives (70+TB/6TB drives + parity)

power supply - again I am sure you have replaced these - look for ~650+W, I would recommend 80 Plus Gold.

motherboard+CPU+RAM - maybe you toss the PC when these go bad, but it's mostly a few screws. You wont need a lot here as this is just a NAS.

Controller card - This is perhaps the only thing you haven't touched before, but it is a card, like video or network. This card gets cabled to the drives.

Drives - again, you have replaced these, it is the same in unRAID.

 

The only thing I wish to add is backup. Don't make this or anything else the single copy of anything.

[steve317]

Thank you for the encouragement and guidance.

 

Now, assuming unRaid is suitable for a large video storage system used for infrequent editing on Final Cut Pro, what I'm hoping for is some guidance on what system to get or build:

 

I've looked around the web and these forums, and it's not clear to me where a current list is.  I've seen some recommended builds from many years ago, but I assume that some of those are out of date.

 

Are there any current lists of good configurations of large capacity unRaid servers, preferably with prices listed? 

[hernandito]

Thank you for the encouragement and guidance.

 

Now, assuming unRaid is suitable for a large video storage system used for infrequent editing on Final Cut Pro, what I'm hoping for is some guidance on what system to get or build:

 

I've looked around the web and these forums, and it's not clear to me where a current list is.  I've seen some recommended builds from many years ago, but I assume that some of those are out of date.

 

Are there any current lists of good configurations of large capacity unRaid servers, preferably with prices listed?

 

Hi Steve,

 

I dont know how much experience you have building a whole computer from parts... It is not that hard. You buy all the individual components and you assemble yourself. On your first build, it may take you 2-3 hours to achieve. Once you put it all together, its just a matter on configuring some thing in the bios, create your unRAID thumb drive and off you go.

 

Other than what Limetech sells, (and I forgot the other maker) there is nothing that is built specifically for unRAID. UnRAID will run perfectly on a ton of different hardware, from a 5 year old Celeron to the latest greatest... from consumer cpu/mobo to server cpu/mobo... It all depends on what you want the server to do. For simple file serving, you do not need much. For transcoding 5 simultaneous Plex streams, you may want better hardware.

 

For my sister, I built server from retired desktop machine's motherboard and cpu (an Intel Core 2 Duo).

 

Most of us here I think are guys who like to get under the computer's hood...

 

The biggest expense is going to be the cost of the hard drives. 4TB drives cost around $130 (not sure this size is the lowest cost per GB). If you need 70TB  you will need 18 4TB drives at roughly $2,340 just in drive alone. A 24 drive case are in the $400 range (not sure there are 19-20 drive cases)... But remember, you CAN add drives as you need... so you can start with 5 and go from there.

 

 

 

 

 

 

[meep]

I would suggest that you try not to build a combined archive and edit storage array. You will end up chasing performance metrics for editing and end up way over specifying your storage.

 

If I was doing this, I'd focus on reliable,  robust storage for all the media files. Then, on the occasion I needed to work on a project, I'd copy it to a dedicated edit system with faster local storage and copy it back to the unraid array when done.

 

Plus as already mentioned, don't relay on a single system.as your sole arechive - back it up or have a mirrored system or similar.

[tr0910]

Agree with meep. For a photo studio we work on workstations with local fast SSD storage but use unRaid to host it all.  Trying to do it differently will push you into pioneering territory which is best left for the techies for now.

 

Gigabit Ethernet will be a limiting factor. 10 gig Ethernet is coming but is not yet ready for the "I just need it to work crowd".

[steve317]

Thanks for all this info.  Can you clarify -- are you saying I shouldn't use the unRAID to view the archived videos -- I shouldn't play them on my local computer while they're stored on the unRAID, but instead should copy them to a local HDD or SSD even to view them, or do I only need to to do that to edit them?

 

Also, the big next step is knowing what to build.  Is there a good up-to-date list of parts to get for the kind of system I want to build?

 

Or is it best to go with Lime Tech's -- will it wind up costing me the same or more just to build it from scratch?

 

Thanks!

[trurl]

Thanks for all this info.  Can you clarify -- are you saying I shouldn't use the unRAID to view the archived videos -- I shouldn't play them on my local computer while they're stored on the unRAID, but instead should copy them to a local HDD or SSD even to view them, or do I only need to to do that to edit them?

You don't want to do the editing over the network. Viewing videos on the other hand is probably one of the main things people use unRAID for.

 

We don't even edit photos over the network.

[Russ Uno]

Thanks for all this info.  Can you clarify -- are you saying I shouldn't use the unRAID to view the archived videos -- I shouldn't play them on my local computer while they're stored on the unRAID, but instead should copy them to a local HDD or SSD even to view them, or do I only need to to do that to edit them?

You don't want to do the editing over the network. Viewing videos on the other hand is probably one of the main things people use unRAID for.

 

We don't even edit photos over the network.

 

I agree it's better to edit on a local drive (Firewire is usually best for Final Cut) but with Final Cut you don't actually edit the video itself, you are working with a small file that keeps a list of the edits, settings & etc.

I haven't tried it from Final Cut but I have edited 30 tracks of HD audio from within ProTools, which also uses a Session file that keeps a list of edit's, setting & etc. off UnRaid and it did work just fine..

These high level Apps use non destructive editing, they don't actually cut or edit Video or Audio directly. I think if Final Cut doesn't complain about using a network drive and plays the Video without a problem it could work.

The problem with Final Cut is there is a whole host of files that come into play, the Actual video, the reference files (wave forms, render files and etc), the session/Final Cut file. All those can be stored in different places and with the older versions of Final Cut before Final Cut Pro X all files from all video edit sessions are stored together in various folders so finding, separating and moving the files for just one video editing session is not easy. Final Cut Pro X may be easier but not as simple as moving just the actual Video.

[tr0910]

What we are trying to tell you is that for intensely working on large files, doing it over the network can be slower than a local SSD drive. However for small files and not for intense working, unRaid will be fine.

 

You can read at gigabit wireline speeds of over 100meg/sec. Unless you install a cache drive writes to unRaid are going to be about 1/2 as fast due to the parity protected process. 

 

No matter what it will be better than what you have today.

 

[00b5]

If you want to store, and then watch any videos you described across a wired network, unRAID will be fine for that. If you are expecting to EDIT videos, especially LARGE single files, doing so FROM the network storage (ie, unRAID) isn't going to be optimal. Bringing the file/video/etc locally, then workign on it, and they storing the changes each day/final copy/few extra copies back on the unRAID should also be fine.

 

Probably the #1 use of unRAID these days is people watching movies/tvshows. They store very large files (1gb to 16gb in my case) tvshows/movies, then play them back from unRAID to some other device all around. Some people even transcode (which requires server hp, ie, a faster CPU) to various devices. When one movie (lets say bluray copy, 1080p quality, etc) ranges from 4GB for something run of the mill, up to a full iso copy for 25-50GB (PER MOVIE!!) people need LOTS of space pretty quickly.

 

I'll also add, that if all you wanted to do was have ONE central place to store all the media you referenced, then unRAID is a possible solution. It allows you to highly customize your setup, plan for possible upgrades like cpu power, expanding to additional drives, and to build it out to the physical size that works for you (small box in the corner, big rack mount server case, etc). Heck, you could even start with just recycling some existing hardware you have laying around. If you can get a recent computer that can hold/connect 4 sata hdds, (just a # i picked) you could just proof of concept this entire thing yourself, and only spend the cost of the license and maybe some new hdds.

 

Here are some new core build options that I suggested recently, just to give you some ideas:

Cheap: ~ $190 to $320+ for mb/cpu/ram

ASRock H97M Pro4 LGA 1150 Intel H97 HDMI SATA 6Gb/s USB 3.0 Micro ATX Intel Motherboard $79

http://www.newegg.com/Product/Product.aspx?Item=N82E16813157512

 

Intel Pentium G3258 Haswell Dual-Core 3.2 GHz LGA 1150 53W BX80646G3258 Desktop Processor $69

http://www.newegg.com/Product/Product.aspx?Item=N82E16819117374

 

2x4GB DDR3 $40

Lots to choose from, so i'm not linking anything specific. Just verify you get the minimum speed (probably ddr3 1600)

 

This mb would be VT-d ready, but the cpu isn't. this means you can't pass videocards/etc to VMs. You can still run VMs, dockers, etc all just fine. You just can't build something like a gaming machine inside your unRAID box without these features. If you wanted VT-d now, swap to a compatible cpu (most haswell i5 or i7), probably upping your cpu cost from $70 to ~$200+

 

Expensive: $560 (Server mb (12 sata)/xeon /w video/ecc ram)

SUPERMICRO MBD-X10SL7-F-O uATX Server Motherboard LGA 1150 Intel C222 DDR3 1600 $250

http://www.newegg.com/Product/Product.aspx?Item=N82E16813182821

 

Intel Xeon E3-1226 v3 Haswell 3.3GHz 8MB L3 Cache LGA 1150 84W Server Processor BX80646E31226V3  $225

http://www.newegg.com/Product/Product.aspx?Item=N82E16819117319

 

2x4GB DDR3 EEC $85

 

Going with server hardware is usually going to be better, but if you don't need it, you can often just use desktop components. If you search the forums you'll see plenty of discussions about why one way or the other.

 

Add your case (with enough bays for reasonable expansion for the future), an appropriately sized single rail PSU for the amount of drives you might want down the road, and then add drives.

 

I'd also suggest a good backup, since you aren't referring to regular movies/tvshows that many of us can just as easily acquire again a 2nd time if needed.

 

I'd suggest a cheaper (like the G3250 based system) setup with a few large drives just to keep a 2nd copy. Some people do this 1:1 (so two 70TB+ setups), but if you can get by with say, 20TB of CRITICAL backups, you can build that on the cheap.