AMD Fileserver Build (100TB+)


Recommended Posts

Hey all,

Pretty proud of my first Unraid machine. Finally have it to a "finished" state. I wanted to share some good photos but I'm no photographer. I scored all 10 14TB HDD's from shucking Seagate external drives a few months ago, before the price boom, so $225 apiece. Hoping price comes back down because I have space for a few more. I'm still in the learning phase but I have set up OpenVPN, Nextcloud with reverse proxy, Plex, Windows 10 VM, and Ubuntu VM. Running headless for now until I have a need for a GPU assuming they come down to MSRP. I have a 10Gbps card in this and my regular PC, so they're directly connected without a switch. All-in cost right now is north of $5k, but I am also interested in upgrading to a Ryzen 5900/5950x when those are available at MSRP also. Before anyone gets concerned, the drive power lines are not all chained together. 

PXL_20210617_163200684.PORTRAIT.jpg

PXL_20210617_163222724.PORTRAIT.jpg

PXL_20210617_163313917.PORTRAIT.jpg

PXL_20210617_163359867.PORTRAIT.jpg

PXL_20210617_163421931.PORTRAIT.jpg

PXL_20210617_163601219.PORTRAIT.jpg

PXL_20210617_164030775.jpg

  • Like 6
  • Thanks 1
Link to comment
Just now, PSYCHOPATHiO said:

Amazing build, & a great choice of hardware overall, This is what I would call a perfect build. Just make sure you protect your investment with a decent UPS, it saved my server at least 2 times in the last 6 months from outages.

 

Welcome to the club.

Thanks dude, yep I have a UPS tucked back there too.

Link to comment
4 hours ago, Lolight said:

What are your HDD's temps when active?

Do you run the stock front fans?

Looks like the highest temp any of the HDDs has seen is 36C, with most of them showing highest of 32C. I actually have a temperature sensor probe coming tomorrow, that I am planning to put in the HDD area, to set the fan curve for the front fans, which are just the 3 stock fans. I haven't spent much time on fan settings yet, I just sort of set everything at balanced. I'm also going to be test fitting some 40mm fans for my 3 cards tomorrow.

Link to comment
11 hours ago, stev067 said:

Looks like the highest temp any of the HDDs has seen is 36C, with most of them showing highest of 32C.

Those are impressively low temps for such a dense HDD arrangement.

What are your ambient temps?

Edited by Lolight
Link to comment
On 6/19/2021 at 8:59 PM, Lolight said:

Those are impressively low temps for such a dense HDD arrangement.

What are your ambient temps?

I dug into the server last night and added the thermal probe, and checked out some temps in the bios. Apparently there is a motherboard sensor that isn't registering within Unraid, which was at 40C, and I'm measuring 25-30C on the HDDs. They look close together, but the airflow through them is really great, probably because the 3 fans are right there, and there's no cable clutter in that space. I left the temp probe between two of the HDDs and set the front fans to use that temp for its curve. I also added 40mm Noctua fans to my sas expander and raid card, and that lowered their temps an insane amount. Anyone have any idea why the dynamix temperature plugin is only registering my CPU and NVMe temperature sensors? Also, the dynamix fan plugin doesn't seem to register anything whatsoever.

Edited by stev067
Link to comment
1 hour ago, stev067 said:

Pic of the 40mm fan addition if anyone is interested. I just used the rubber things that Noctua includes, instead of going and buying screws. It works perfectly but takes a bit of force.

PXL_20210621_212313498.jpg

Nice work, might have to copy that idea 😛

Finally got mine yesterday.

 

You should be able to add some bandwidth (will help for parity checks) by dual linking your HBA, see below

 

  

On 9/21/2015 at 10:47 PM, JorgeB said:

Intel® RAID SAS2 Expander RES2SV240 - 10w

 

Single Link with Dell H310 (2400MB/s)

8 x 275MB/s

12 x 185MB/s

16 x 140MB/s (112MB/s*)

20 x 110MB/s (92MB/s*)

 

* Avoid using slower linking speed disks with expanders, as it will bring total speed down, in this example 4 of the SSDs were SATA2, instead of all SATA3.

 

Dual Link with Dell H310 (4000MB/s)

12 x 205MB/s

16 x 155MB/s (185MB/s*)

 

* Two different boards had consistently different results, will need to test a third one to see what's normal, 155MB/s is the max on a Supermicro X9SCM-F, 185MB/s on Asrock B150M-Pro4S.

 

Link to comment
2 minutes ago, tjb_altf4 said:

Nice work, might have to copy that idea 😛

Finally got mine yesterday.

 

You should be able to add some bandwidth (will help for parity checks) by dual linking your HBA, see below

 

  

 

Good call, I'll give it a try. Any idea how I should go about testing drive RW speeds?

Link to comment
On 6/22/2021 at 10:00 PM, tjb_altf4 said:

Should be able to do it though diskspeed

 

Hmm I'm not seeing any performance gain from dual-link, when testing with diskspeed. Looks like this only tests one drive at a time though. Wouldn't the benefit of the dual-link bandwidth be more for simultaneous disk RWs?

Link to comment

Since I can't stop making upgrades, I figured I would keep this thread updated. I decided the best way of beefing up my system was to replace my 3600xt with a 5950x. So I'm running 16 cores now. Part of the reasoning was I added a new windows VM that will be dedicated to running Blue Iris. Once the new PoE/VLAN switch comes, I will be setting up some perimeter cameras on my house which will record to a 4TB WD purple HDD which I plan to add as an unassigned drive, passed into the VM.

I also updated my mobo BIOS yesterday around 2:30 PM, and spent the next 7 hours trying to boot back in. The problem just ended up being one of the BIOS settings I was missing. Always save or write down your important BIOS settings, because they don't persist after an update.

Edited by stev067
TB not GB
  • Like 1
Link to comment
14 hours ago, stev067 said:

Hmm I'm not seeing any performance gain from dual-link, when testing with diskspeed. Looks like this only tests one drive at a time though. Wouldn't the benefit of the dual-link bandwidth be more for simultaneous disk RWs?

Yes, so the situation that where it would be noticeable would be during parity check when your reading from all drives, where with a single link you would see slower (but not terrible) speeds.

Unless you plan on using the other port for more drives, using dual link should just give you headroom when the array is under heavy use.

 

Link to comment
11 hours ago, tjb_altf4 said:

Yes, so the situation that where it would be noticeable would be during parity check when your reading from all drives, where with a single link you would see slower (but not terrible) speeds.

Unless you plan on using the other port for more drives, using dual link should just give you headroom when the array is under heavy use.

 

Thanks that's what I figured. I guess I could do a comparison when doing a turbo write too. I'll leave the dual link since I have it, but might be adding a 8087 to 8088 card to expand to an additional drive enclosure in the future. I think my personal max capacity on this case will be 16 drives.

  • Like 1
Link to comment
  • 2 weeks later...
On 6/27/2021 at 6:38 PM, stev067 said:

Thanks that's what I figured. I guess I could do a comparison when doing a turbo write too. I'll leave the dual link since I have it, but might be adding a 8087 to 8088 card to expand to an additional drive enclosure in the future. I think my personal max capacity on this case will be 16 drives.

Hi, where did you use the Icy Dock flexiDOCK MB795SP-B ? i have a new define 7 xl that im going to start building a new server in. But was thinking of going only with standard drive trays. Didnt see the ice in the picture i think :-)

 

Link to comment
1 hour ago, orlando500 said:

Hi, where did you use the Icy Dock flexiDOCK MB795SP-B ? i have a new define 7 xl that im going to start building a new server in. But was thinking of going only with standard drive trays. Didnt see the ice in the picture i think 🙂

 

 

Hey yeah I didn't get a pic of the dock, but it's in the front top slot. This blocks the top 2 HDD tray slots, which I don't need yet. I haven't been able to get the dock to work with hot plugging. A drive is only recognized if it was detected on boot, which kind of makes it pointless to me. Maybe it can be fixed but I haven't needed it yet. Probably in the long term, I will end up taking this out, so I can add 2 more drives to the top trays.

Link to comment

Great looking setup!

One question, I have a similar LSI card which i smacked a 40mm Nocuta on like you, wonder how you knew the temp of the card? You mentioned it decreased the temp a lot.

Love to see if I can measure how well my one is working.

 

Also wondering why you went with an Expander card rather than just use the ports on your mobo for the rest?

Link to comment
7 minutes ago, deanpelton said:

Great looking setup!

One question, I have a similar LSI card which i smacked a 40mm Nocuta on like you, wonder how you knew the temp of the card? You mentioned it decreased the temp a lot.

Love to see if I can measure how well my one is working.

 

Also wondering why you went with an Expander card rather than just use the ports on your mobo for the rest?

Hey!

So I have one of those IR thermal temp guns where you just point it. This one.  I'm not sure if there is an onboard sensor on the LSI cards, but I know that the Dynanix system temperature plugin didn't see one.

Regarding the SATA ports, I might not have really considered it haha. For me, it was sort of an exercise in learning about HBA's and expanders, wanting to have the IO to expand to another enclosure in the future, if necessary. Also I didn't really like the idea of having HDDs in the same array but on mixed interfaces. Cleanest to keep everything on the same single PCIe slot.

Link to comment
1 minute ago, stev067 said:

Regarding the SATA ports, I might not have really considered it haha. For me, it was sort of an exercise in learning about HBA's and expanders, wanting to have the IO to expand to another enclosure in the future, if necessary. Also I didn't really like the idea of having HDDs in the same array but on mixed interfaces. Cleanest to keep everything on the same single PCIe slot.

Fascinating, I hadn't thought about that!

 

I run an mtax case, so I only have the 1 PCIe slot and I am maxed out between the 8 from the LSI and the 4 on my motherboard, looks like you just gave me a way out.

Had no idea about these expander cards.

Link to comment
Just now, deanpelton said:

Fascinating, I hadn't thought about that!

 

I run an mtax case, so I only have the 1 PCIe slot and I am maxed out between the 8 from the LSI and the 4 on my motherboard, looks like you just gave me a way out.

Had no idea about these expander cards.

Yeah sounds like you're only a month or two behind me, as I had no idea either. It's dead simple to set up. One port needs to be plugged into your HBA card (doesn't matter which port), and then all the other ports are free ports for 4x HDDs. So with one HBA and two expanders, you could connect 40 drives to the same PCIe slot.

Here is the link to the expander card I bought from eBay. Another user gave me a tip that this guy will accept offers of $75, which is much lower than his listed price. It has a legit intel chip on it, though I'm not sure the card is genuine intel or some kind of OEM / grey market.

I just drilled holes in my case and used brass standoffs to mount it, since I didn't want it rattling around and potentially shorting out.

  • Thanks 1
Link to comment
  • 3 weeks later...
1 hour ago, JBa said:

This is a good build.. Does anyone has a recommendation for having a rack mounted case for this ?

I'm also interested, because my curiosity may take me to rack-mounting things at some point. You may get more/better answers by making a new thread, because I don't know if people will really see your question here.

Link to comment
  • 2 weeks later...

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.