28x8TB Reds and 2x12TB WD Golds + 2x1TB SSD Cache - MEGAUNRAID BUILD

[engin33rh3r3]

I was going to use a sa120 + a 12 bay Norco but the sa120 is so loud and still wasn't enough space. Now I am looking for a chassis to support this many drives as well as a what CPU+RAM+MOBO combo. My primary unraid server is a 14x8tb TS440 with extra drive caddies running 32gb ECC and a Xeon 1275L CPU. The new mega-unraid server will allow me to shift my original server to being a backup. My CPU+RAM+MOBO+CHASSIS budget is less than $2k 

 

Already have,

28x8TB Reds and 2x12TB WD Golds 

2xDell LSI SAS 9207-8I PCI-E 3.0

2x1TB Samsung 860 SSD's

 

And if there is a way to use what I have on hand idea would be appreciated,

SA120 w/drive caddies (Currently Empty)

TS440 w/12 empty drive caddies

64gb of ddr4 2400MHz Corsair Dominator Platinum Ram ( *not using - from previous main rig)

64gb of ddr3 ECC ram and xeon 1275L v3 on a TS440 mobo

Dell PowerEdge T30 8GB UDIMM, 2400MT/s Intel Xeon E3-1225 (*not using - from a failed project)

Lots of rack space

 

 

What is the most efficient build? Would like to have current generation equipment unless I have enough lanes on with the TS440 or need to go to the latest generation. Also have a spare 7700k or could steal the 8700k from my new rig but unsure about not using ECC ram.

 

 

[engin33rh3r3]

This is not a joke. I am an engineer for a fortune 500 company and wanted to use my bonus to go balls out. Especially since I am out of space and don't have a backup for my original unraid/plex server. 

[jbartlett]

I've been pondering the same thing for my annual bonus with a balls-to-the-wall multi-CPU powerhouse though not as crazy on the drives as you. Pondering the balance between core speed vs core count -the primary use would be to re-encode my huge Plex library to h265 from lossless MKV's and plex encodings is leading me towards core count.

[jbartlett]

I can tell you with that many drives, you'll want to run dual parity or break it up into two NAS units. Maybe use a cube case that supports dual motherboards and create two NAS servers in one? Tom has spoken up in the past about the future possibility of having a master/slave unRAID setup where multiple unRAID servers would look like one on the Network.

[whipdancer]

Sorry, I don't think I've seen any real discussion of something that big before.

The only people I've seen discuss anything close to it are photo/video pros and use it for their work - but even they don't keep that much data live. Heck, I don't even know what the use-case is for it. I don't know how much help you will actually get.

 

 

Quote

What is the most efficient build?

What is the goal for the equipment? Transcoding, rendering, engineering, playing 20 video streams at the same time, running a torrent site?

The use of the equipment would dictate most efficient setup.

Edited March 2, 2018 by whipdancer

[Lev]

I build this big. My advice... Don't trouble yourself with those items you listed you want to reuse. Build for where you going, not where you've been.

 

Write speed becomes an issue to fill it in a reasonable amount of time if all drives are empty. Look at some of my previous post and you might find some ideas on how to speed it up.

 

You already spent a lot of money on drives, now is not the time to be cheap. Finish the job.

 

Let's see some pics

 

[engin33rh3r3]

Thanks. After more research and talking to people I am now stuck on the fence between unraid and stable bit+snap raid. Each have there own pros cons and they are significant in each of there own ways. 

 

Almost everything has shown up now and I have made a few changes since I posted this.

 

Now it will be 

 

• SUPERMICRO 1U SERVER X9DRI-LN4F+ 2x E5-2680V2 32GB 4x TRAYS RAILS
• 164GB DDR3 ECC ram
• 6x12TB WD Golds 7200rpm
• 1x12TB Seagate 12TB Enterprise Capacity 7200rpm
• 24x8TB Reds&White Labels mix
• 2x500GB Samsung 860 SSD's (2xCache drives dual parity)
• 2x2TB 2TB Micron
• 2xDell LSI SAS 9207-8I PCI-E 3.0
• Supermicro 846E16-R1200B 4U Server BPN-SAS2-846EL1 24x TRAYS

 

Also got a Rosewill Rosewill 4U Server Chassis(RSV-L4500) that I considered moving the 1u supermicro into to give me more bays without using the SA120
just didn't know if I want to use something like an HP 24 BAY 3GB SAS expander card with the drives in it or add another 2xDell LSI SAS 9207-8I PCI-E 3.0 cards?

 

 

I am also assuming that I want the cache drives and parity drives directly connected to motherboard and not an the EXPANDER or the 846E16-R1200B JBOD? Also, will the 846E16 not choke if I try to access to many of the 24 drives???

 

I really like the idea that Stable Bit+Snap Raid is in Windows 2016 Server and that I am the one that controls when the redundancy information is updated. Also, unriad can't identify silent data corruption.Then on top of that Windows 2016 Server can now also run Windows dockers and support for Linux dockers is currently in preview/coming soon and through Plex the tool I am primarily building this for now supports GPU hardware acceleration which is significant.... However, I really really enjoy how reliable and straight forward unraid has been for me for the past couple of years and feel that MANY people in IT overlook the value of this especially with big data... I have never built a single FREENAS/NAS4FREE or any Windows Servers that has given me anything even close to the order of magnitude of reliability my unraid rig has given me... That is a big deal because of the size of the data set... Also, I don't feel like I have to keep re-learning things and it just always works with almost no headaches... Any words of wisdom here? I have more time effort and money invested into this and rather not mess it up lol... 

 

 

 

 

 

On 3/4/2018 at 2:32 AM, Lev said:

I build this big. My advice... Don't trouble yourself with those items you listed you want to reuse. Build for where you going, not where you've been.

 

Write speed becomes an issue to fill it in a reasonable amount of time if all drives are empty. Look at some of my previous post and you might find some ideas on how to speed it up.

 

You already spent a lot of money on drives, now is not the time to be cheap. Finish the job.

 

Let's see some pics

 

 

 

[pwm]

9 hours ago, engin33rh3r3 said:

Also, unriad can't identify silent data corruption.

You can, if you use a file integrity plugin.

[JorgeB]

You can, if you use a file integrity plugin.

Or btrfs as the default filesystem

[pwm]

1 minute ago, johnnie.black said:

47 minutes ago, pwm said:

You can, if you use a file integrity plugin.

Or btrfs as the default filesystem

The BTRFS checksums isn't as explicit as the checksumming done by file integrity plugins and SnapRAID. But compared to SnapRAID etc, it isn't limited to just look at checksums for individual files - it also checksums the file system structures.

 

The good thing is that it's possible to use both a file integrity plugin and BTRFS, where a file integrity plugin can also help with locating duplicated files i.e. files with the same hash. And a file integrity plugins can catch intentionally changed files while BTRFS can only catch bit rot.

[jbrodriguez]

I'd go with Limetech just for forum support, but sound technology is a killer.

 

I still remember when my motherboard died, basically because my Intel adapter caught on fire !!

 

I switched motherboard, started unRAID and it was as if nothing had happened. Those days I still believed in Windows, and then I recognized the switch wouldn't have been possible with a Windows or any other (non-Linux/unRAID) server.

 

ZFS is an awesome technology, but being able to add disks to your array 'for free' (within your appetite for risk, one or two parity disks), is comparable to none.

[HellDiverUK]

I sometimes spin up a Windows Server system and use it for a while, but unRAID is much easier to deal with.  I fight with Server all day at work, when I get home I don't want to be doing it there either. 

 

unRAID, in general, just works.  I've not had any show-stoppers with unRAID since I started using it, even hardware failures are trivial to recover from.  I think I first started running unRAID at version 4.5, now on 6.5 and run pretty much every beta/RC on the way through.