AMD Fileserver Build (100TB+)


22 posts in this topic Last Reply

Recommended Posts

Hey all,

Pretty proud of my first Unraid machine. Finally have it to a "finished" state. I wanted to share some good photos but I'm no photographer. I scored all 10 14TB HDD's from shucking Seagate external drives a few months ago, before the price boom, so $225 apiece. Hoping price comes back down because I have space for a few more. I'm still in the learning phase but I have set up OpenVPN, Nextcloud with reverse proxy, Plex, Windows 10 VM, and Ubuntu VM. Running headless for now until I have a need for a GPU assuming they come down to MSRP. I have a 10Gbps card in this and my regular PC, so they're directly connected without a switch. All-in cost right now is north of $5k, but I am also interested in upgrading to a Ryzen 5900/5950x when those are available at MSRP also. Before anyone gets concerned, the drive power lines are not all chained together. 

PXL_20210617_163200684.PORTRAIT.jpg

PXL_20210617_163222724.PORTRAIT.jpg

PXL_20210617_163313917.PORTRAIT.jpg

PXL_20210617_163359867.PORTRAIT.jpg

PXL_20210617_163421931.PORTRAIT.jpg

PXL_20210617_163601219.PORTRAIT.jpg

PXL_20210617_164030775.jpg

  • Like 2
  • Thanks 1
Link to post

Amazing build, & a great choice of hardware overall, This is what I would call a perfect build. Just make sure you protect your investment with a decent UPS, it saved my server at least 2 times in the last 6 months from outages.

 

Welcome to the club.

Link to post
Just now, PSYCHOPATHiO said:

Amazing build, & a great choice of hardware overall, This is what I would call a perfect build. Just make sure you protect your investment with a decent UPS, it saved my server at least 2 times in the last 6 months from outages.

 

Welcome to the club.

Thanks dude, yep I have a UPS tucked back there too.

Link to post
On 6/17/2021 at 11:59 PM, stev067 said:

I scored all 10 14TB HDD's from shucking Seagate external drives

What are your HDD's temps when active?

Do you run the stock front fans?

Link to post
4 hours ago, Lolight said:

What are your HDD's temps when active?

Do you run the stock front fans?

Looks like the highest temp any of the HDDs has seen is 36C, with most of them showing highest of 32C. I actually have a temperature sensor probe coming tomorrow, that I am planning to put in the HDD area, to set the fan curve for the front fans, which are just the 3 stock fans. I haven't spent much time on fan settings yet, I just sort of set everything at balanced. I'm also going to be test fitting some 40mm fans for my 3 cards tomorrow.

Link to post
11 hours ago, stev067 said:

Looks like the highest temp any of the HDDs has seen is 36C, with most of them showing highest of 32C.

Those are impressively low temps for such a dense HDD arrangement.

What are your ambient temps?

Edited by Lolight
Link to post
5 minutes ago, Lolight said:

Those are impressively low temps for such a dense HDD arrangement.

What are your ambient temps?

I'll take some temps tomorrow when the temperature sensor comes and let you know.

Link to post
Posted (edited)
On 6/19/2021 at 8:59 PM, Lolight said:

Those are impressively low temps for such a dense HDD arrangement.

What are your ambient temps?

I dug into the server last night and added the thermal probe, and checked out some temps in the bios. Apparently there is a motherboard sensor that isn't registering within Unraid, which was at 40C, and I'm measuring 25-30C on the HDDs. They look close together, but the airflow through them is really great, probably because the 3 fans are right there, and there's no cable clutter in that space. I left the temp probe between two of the HDDs and set the front fans to use that temp for its curve. I also added 40mm Noctua fans to my sas expander and raid card, and that lowered their temps an insane amount. Anyone have any idea why the dynamix temperature plugin is only registering my CPU and NVMe temperature sensors? Also, the dynamix fan plugin doesn't seem to register anything whatsoever.

Edited by stev067
Link to post
Posted (edited)

Pic of the 40mm fan addition if anyone is interested. I just used the rubber things that Noctua includes, instead of going and buying screws. It works perfectly but takes a bit of force.

PXL_20210621_212313498.jpg

Edited by stev067
Link to post
1 hour ago, stev067 said:

Pic of the 40mm fan addition if anyone is interested. I just used the rubber things that Noctua includes, instead of going and buying screws. It works perfectly but takes a bit of force.

PXL_20210621_212313498.jpg

Nice work, might have to copy that idea 😛

Finally got mine yesterday.

 

You should be able to add some bandwidth (will help for parity checks) by dual linking your HBA, see below

 

  

On 9/21/2015 at 10:47 PM, JorgeB said:

Intel® RAID SAS2 Expander RES2SV240 - 10w

 

Single Link with Dell H310 (2400MB/s)

8 x 275MB/s

12 x 185MB/s

16 x 140MB/s (112MB/s*)

20 x 110MB/s (92MB/s*)

 

* Avoid using slower linking speed disks with expanders, as it will bring total speed down, in this example 4 of the SSDs were SATA2, instead of all SATA3.

 

Dual Link with Dell H310 (4000MB/s)

12 x 205MB/s

16 x 155MB/s (185MB/s*)

 

* Two different boards had consistently different results, will need to test a third one to see what's normal, 155MB/s is the max on a Supermicro X9SCM-F, 185MB/s on Asrock B150M-Pro4S.

 

Link to post
2 minutes ago, tjb_altf4 said:

Nice work, might have to copy that idea 😛

Finally got mine yesterday.

 

You should be able to add some bandwidth (will help for parity checks) by dual linking your HBA, see below

 

  

 

Good call, I'll give it a try. Any idea how I should go about testing drive RW speeds?

Link to post
On 6/22/2021 at 10:00 PM, tjb_altf4 said:

Should be able to do it though diskspeed

 

Hmm I'm not seeing any performance gain from dual-link, when testing with diskspeed. Looks like this only tests one drive at a time though. Wouldn't the benefit of the dual-link bandwidth be more for simultaneous disk RWs?

Link to post
Posted (edited)

Since I can't stop making upgrades, I figured I would keep this thread updated. I decided the best way of beefing up my system was to replace my 3600xt with a 5950x. So I'm running 16 cores now. Part of the reasoning was I added a new windows VM that will be dedicated to running Blue Iris. Once the new PoE/VLAN switch comes, I will be setting up some perimeter cameras on my house which will record to a 4TB WD purple HDD which I plan to add as an unassigned drive, passed into the VM.

I also updated my mobo BIOS yesterday around 2:30 PM, and spent the next 7 hours trying to boot back in. The problem just ended up being one of the BIOS settings I was missing. Always save or write down your important BIOS settings, because they don't persist after an update.

Edited by stev067
TB not GB
Link to post
14 hours ago, stev067 said:

Hmm I'm not seeing any performance gain from dual-link, when testing with diskspeed. Looks like this only tests one drive at a time though. Wouldn't the benefit of the dual-link bandwidth be more for simultaneous disk RWs?

Yes, so the situation that where it would be noticeable would be during parity check when your reading from all drives, where with a single link you would see slower (but not terrible) speeds.

Unless you plan on using the other port for more drives, using dual link should just give you headroom when the array is under heavy use.

 

Link to post
11 hours ago, tjb_altf4 said:

Yes, so the situation that where it would be noticeable would be during parity check when your reading from all drives, where with a single link you would see slower (but not terrible) speeds.

Unless you plan on using the other port for more drives, using dual link should just give you headroom when the array is under heavy use.

 

Thanks that's what I figured. I guess I could do a comparison when doing a turbo write too. I'll leave the dual link since I have it, but might be adding a 8087 to 8088 card to expand to an additional drive enclosure in the future. I think my personal max capacity on this case will be 16 drives.

Link to post
  • 2 weeks later...
On 6/27/2021 at 6:38 PM, stev067 said:

Thanks that's what I figured. I guess I could do a comparison when doing a turbo write too. I'll leave the dual link since I have it, but might be adding a 8087 to 8088 card to expand to an additional drive enclosure in the future. I think my personal max capacity on this case will be 16 drives.

Hi, where did you use the Icy Dock flexiDOCK MB795SP-B ? i have a new define 7 xl that im going to start building a new server in. But was thinking of going only with standard drive trays. Didnt see the ice in the picture i think :-)

 

Link to post
1 hour ago, orlando500 said:

Hi, where did you use the Icy Dock flexiDOCK MB795SP-B ? i have a new define 7 xl that im going to start building a new server in. But was thinking of going only with standard drive trays. Didnt see the ice in the picture i think 🙂

 

 

Hey yeah I didn't get a pic of the dock, but it's in the front top slot. This blocks the top 2 HDD tray slots, which I don't need yet. I haven't been able to get the dock to work with hot plugging. A drive is only recognized if it was detected on boot, which kind of makes it pointless to me. Maybe it can be fixed but I haven't needed it yet. Probably in the long term, I will end up taking this out, so I can add 2 more drives to the top trays.

Link to post

Great looking setup!

One question, I have a similar LSI card which i smacked a 40mm Nocuta on like you, wonder how you knew the temp of the card? You mentioned it decreased the temp a lot.

Love to see if I can measure how well my one is working.

 

Also wondering why you went with an Expander card rather than just use the ports on your mobo for the rest?

Link to post
7 minutes ago, deanpelton said:

Great looking setup!

One question, I have a similar LSI card which i smacked a 40mm Nocuta on like you, wonder how you knew the temp of the card? You mentioned it decreased the temp a lot.

Love to see if I can measure how well my one is working.

 

Also wondering why you went with an Expander card rather than just use the ports on your mobo for the rest?

Hey!

So I have one of those IR thermal temp guns where you just point it. This one.  I'm not sure if there is an onboard sensor on the LSI cards, but I know that the Dynanix system temperature plugin didn't see one.

Regarding the SATA ports, I might not have really considered it haha. For me, it was sort of an exercise in learning about HBA's and expanders, wanting to have the IO to expand to another enclosure in the future, if necessary. Also I didn't really like the idea of having HDDs in the same array but on mixed interfaces. Cleanest to keep everything on the same single PCIe slot.

Link to post
1 minute ago, stev067 said:

Regarding the SATA ports, I might not have really considered it haha. For me, it was sort of an exercise in learning about HBA's and expanders, wanting to have the IO to expand to another enclosure in the future, if necessary. Also I didn't really like the idea of having HDDs in the same array but on mixed interfaces. Cleanest to keep everything on the same single PCIe slot.

Fascinating, I hadn't thought about that!

 

I run an mtax case, so I only have the 1 PCIe slot and I am maxed out between the 8 from the LSI and the 4 on my motherboard, looks like you just gave me a way out.

Had no idea about these expander cards.

Link to post
Just now, deanpelton said:

Fascinating, I hadn't thought about that!

 

I run an mtax case, so I only have the 1 PCIe slot and I am maxed out between the 8 from the LSI and the 4 on my motherboard, looks like you just gave me a way out.

Had no idea about these expander cards.

Yeah sounds like you're only a month or two behind me, as I had no idea either. It's dead simple to set up. One port needs to be plugged into your HBA card (doesn't matter which port), and then all the other ports are free ports for 4x HDDs. So with one HBA and two expanders, you could connect 40 drives to the same PCIe slot.

Here is the link to the expander card I bought from eBay. Another user gave me a tip that this guy will accept offers of $75, which is much lower than his listed price. It has a legit intel chip on it, though I'm not sure the card is genuine intel or some kind of OEM / grey market.

I just drilled holes in my case and used brass standoffs to mount it, since I didn't want it rattling around and potentially shorting out.

Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.