Jump to content
MaxNL

MaxMe: My New Server

67 posts in this topic Last Reply

Recommended Posts

So at the end I decided to build my new server. (I should blame this forum the most for make me do it :D)

 

I decided to go for a "beast" kind of server (inspired by Raj's Prototype Designs).

 

The building and the setting has been fun and rewarding, but let's start with the lists:

 

Components - List:

Item - DescriptionLink#Note

Case - Norco RPC-4224>>124 HDD in the frontal bays. The plan is to add the cache drive not using them

Motherboard - ASUS P6T7 WS Supercomputer>>

17 PCIe 2.0 at 8X, plenty of room for expansion

CPU - Intel Xeon DP E5620

>>

1Should be more than enough to cover all the possible Add-On and media decoding

RAM - Kingstone ValueRAM - KVR1333D3N9K3/12G

>>

2I grew up in the '80 were we used to say: there is no such thing as too much RAM - 24GB Total

PSU - Enermax Platimax 1500

>>

1Plenty of power, no redundancy, soon I will have a UPS

SATA Controller - Supermicro AOC-SAS2LP-MV8

>>

3I won't need anything more for long time, 24 HDD ready

Fanplate - Norco 120mm fan wall bracket>>1Keeping the noise down

Fans - CPU Cooler - Noctua NH-C12P SE14>>1Keeping the noise down

Fans - Fanplate - Noctua NF-S12B ULN>>3Keeping the noise down

Fans - Case - Noctua NF-R8>>2Keeping the noise down

Video Card - ASUS EN210 >>1No Video on the motherboard just a cheap card without cooling fan

Hard Disks - Data - Hitachi Deskstar 7k4000 4TB>>522x4=88TB, and 2 parity disk, waiting for that upgrade to UnRaid

Hard Disks - Cache - Western Digital Velociraptor 2,5" SATA2>>2For cache, plugged in raid0 on motherboard SAS connector

Case Accessories - Air Slot RZ065>>7Just nicer than the standard one

Case Accessories - Scythe Bay Rafter 2,5 to 3,5 HDD Mounting>>1To mount the 2 WD Cache HDD without using the front bays

Case Accessories - NZXT Premium Cable 3pin fan Y cable>>14 fan plug on the motherboard vs 5 needed, the splitter solve that

Case Accessories - Lian Li Power Supply Rubber PT-PP03B>>1noise control

Case Accessories - Inline SATA To 2xMolex splitter>>1Keep everything properly connected reducing the cable cluttering

OS - unRAID Server Release 5.0-beta14>>1PRO Licence

 

 

Components - Pictures:

The Norco Case

ASUS Motherboard

Kingston RAM

width=300http://www.norcotek.com/product_images/rpc-4224b.jpg[/img]

width=300http://www.forumpcs.com.br/galeria/albums/userpics/20281/000000088286.jpg[/img]

width=300http://www.nix.ru/autocatalog/memory_modules_kingston/113461_3159_draft.jpg[/img]

Enermax PSU

SATA Controller

Fanplate

width=300http://www.enermax.com/files/ProductPicture_eng/PP_Pic/1285.jpg[/img]

width=300http://www.supermicro.com/a_images/products/Accessories/AOC-SASLP-MV8.gif[/img]

width=300http://cybershop.ri-vier.nl/images/Accessoires/120mm-fwp.jpg[/img]

Noctua CPU Cooler

Noctua 120mm Fan

Noctua 80mm Fan

width=300http://www.noctua.at/images/products/fullsize/noctua_nh_c12p_se14_2.jpg[/img]

width=300http://www.noctua.at/images/products/fullsize/nf-s12b_1.jpg[/img]

width=300http://www.noctua.at/images/products/fullsize/noctua_nf_r8_1.jpg[/img]

ASUS Video Card

Hard Disks - Data

Hard Disks - Cache

width=300http://www.mobilx.hu/images/detailed/en210_silent_di_512md3_v2(lp)_00.jpg[/img]

width=300http://www.hitachigst.com/components/com_virtuemart/shop_image/product/Deskstar_7K4000_4f036682bcca3.jpg[/img]

width=300http://www.boendiy.com/image/pro_img/Western-Digital-WD3000BLFS.jpg[/img]

Slot Covers

2,5" HDD Mounting

Fan Cable Splitter

width=300http://www.revoltec.net/revoltec.net/data/media/_pms/rz065/scaled/rz065_m.jpg[/img]

width=300http://www.scythe-usa.com/product/acc/058/images/bayrafter2.5-main2_400.jpg[/img]

width=300http://a248.e.akamai.net/origin-cdn.volusion.com/2thoc.9vr6n/v/vspfiles/photos/PC-CB%203P%20Y-2.jpg[/img]

Power Supply Rubber

SATA To Molex Adapter

unRAID

width=300http://lian-li.com/v2/tw/product/upload/image/pt-pp03b/pt-pp03b-01.jpg[/img]

width=300http://www.overclockers-store.com/images/large/INLINEsataTO2molex_6fRnFA6x071x_large.jpg[/img]

Share this post


Link to post

Here is the mounting tale:

 

First Round - START (20 april)

[*]Unmounted the original fanplate (4x80) from the norco case

[*]Unmounted the 2x80mm fan from the norco case

[*]Mounted the 3 120mm Noctua fan on the norco 120mm fan wall (with the rubber screws)

[*]Mounted the 2 80mm Noctua fan on the back side (close to the PSU - with the rubber screws)

[*]Mounted the CPU Cooler and CPU on the motherboard[br]The first time I did put the huge cooling fins in the wrong direction, blocking almost the first PCIe slot. Fortunately with the same setting of the CPU Cooler backplate, you can rotate the fins by 180°, et voilà, perfect

[*]Mounted the 140mm Noctua CPU Fan and used the ULN adapter to set the speed at the minimum

[*]Mounted the 6 memory stick (it is more easy if you plug them before the CPU cooler)

[*]Mounted the PSU - using the power supply rubber

[*]Mounted the ASUS Video Card[br]Mounting the video card I noticed that the cooling fins block the second PCIe slot, I will change the GPU with one that take only 1 slot (when and if I will need)

[*]Mounted the 3 AOC-SAS2LP-MV8[br]At the end PCIe slot 1,3,4,5 are taken, 2 is locked by the GPU, 6 and 7 are free

[*]Completed all the connections; Reset and Power switch, Power LED, IDE LED (this one has been connected to one of the two NIC LED of the case as there is no IDE LED on the norco), USB ports, fans (all with the ULN adapter to make as much silent as possible)

At this point >> first start of the HW

[*]I let the first start to go till the end (no system disk present)

[*]During the Boot process I noticed that I got only two Marvell Boot Bios

[*]Time to restart and check/setting the BIOS

[*]Long story short:

  • It is not possible to set the 6 SATA Port on AHCI or RAID if you want to use also the 2 SAS port in a RAID Configuration
  • At the boot if you have one or more AOC cards you will see just one Boot ROM (the second I was seeing was the one from the on-board Marvel SAS ports. You can completely ignore it as in unRAID all the three card are correctly installed[br]I tried to disable the BOOT ROM completely but I wasn't able to do it
  • Complete some settings and tweaking in the BIOS, disabling everything I was not going to use (eg. HD audio) and changing some settings related to fan, boot, etc.

 

First Round - END (20 april)

 

Second Round - START (21 april)

[*]Mounted the HDDs:

  • 1 4TB HDD on the first AOC-SAS2LP card
  • 2 4TB HDD on the second AOC-SAS2LP card
  • 2 4TB HDD on the third AOC-SAS2LP card
  • 2 300GB HDD on the internal SAS ports

 

[*]Unmounted the ULN adapter from all the noctua fan but the CPU, the three 120mm and the 2 80mm run at the highest speed

[*]Fixed the cables

At this point >> first start of unRAID.

[*]unRAID starts

[*]Installed preclear script

[*]Executing preclear on all 7 drive (5 4TB Hitachi, 2 200GB WD Velociraptor)

  • 1st 300GB WD Velociraptor SATA2 (WD-WXH1E61CJEY4) - Successfully Precleared
    • Total Time: 3:52:42
    • Cycle Ran: 1
    • Pre Read Time: 0:52:02 (95 MB/s)
    • Zeroing Time: 0:50:10 (99 MB/s)
    • Post Read Time: 1:42:28 (48 MB/s)
    • Start TEMP: 42C
    • End TEMP: 47C

     

    [*]2nd 300GB WD Velociraptor SATA2 (WD-WXH1E61CJXZ3) - Successfully Precleared

    • Total Time: 1:24:19
    • Cycle Ran: 1
    • Pre Read Time: 0:51:36 (96 MB/s)
    • Zeroing Time: -1:-9:-58 (-71 MB/s)
    • Post Read Time: 1:41:41 (49 MB/s)
    • Start TEMP: 51C
    • End TEMP: 57C
    • Something strange here - I'll run the preclear again later

     

    [*]3rd 4TB Hitachi 7K4000 SATA3 (HDS724040ALE640-PK2310PAG164PM) - Successfully Precleared

    • Total Time: 35:26:40
    • Cycle Ran: 1
    • Pre Read Time: 7:27:54 (148 MB/s)
    • Zeroing Time: 10:24:51 (106 MB/s)
    • Post Read Time: 17:32:47 (63 MB/s)
    • Start TEMP: 39C
    • End TEMP: 47C

     

    [*]4th 4TB Hitachi 7K4000 SATA3 (HDS724040ALE640-PK2310PAG16KJM) - Successfully Precleared

    • Total Time: 37:24:04
    • Cycle Ran: 1
    • Pre Read Time: 7:54:56 (140 MB/s)
    • Zeroing Time: 10:51:35 (102 MB/s)
    • Post Read Time: 18:36:25 (59 MB/s)
    • Start TEMP: 42C
    • End TEMP: 49C

     

    [*]5th 4TB Hitachi 7K4000 SATA3 (HDS724040ALE640-PK2310PAG16M4M) - Successfully Precleared

    • Total Time: 37:08:50
    • Cycle Ran: 1
    • Pre Read Time: 7:49:04 (142 MB/s)
    • Zeroing Time: 10:51:56 (102 MB/s)
    • Post Read Time: 18:26:41 (60 MB/s)
    • Start TEMP: 41C
    • End TEMP: 46C

     

    [*]6th 4TB Hitachi 7K4000 SATA3 (HDS724040ALE640-PK2310PAG16MTM) - Successfully Precleared

    • Total Time: 36:56:15
    • Cycle Ran: 1
    • Pre Read Time: 7:45:16 (143 MB/s)
    • Zeroing Time: 10:47:52 (102 MB/s)
    • Post Read Time: 18:21:59 (60 MB/s)
    • Start TEMP: 39C
    • End TEMP: 44C

     

    [*]7th 4TB Hitachi 7K4000 SATA3 (HDS724040ALE640-PK2310PAG16ULM) - Successfully Precleared

    • Total Time: 36:38:34
    • Cycle Ran: 1
    • Pre Read Time: 7:41:04 (144 MB/s)
    • Zeroing Time: 10:40:43 (104 MB/s)
    • Post Read Time: 18:15:39 (60 MB/s)
    • Start TEMP: 40C
    • End TEMP: 46C

     

 

Second Round - END (22 april)

 

Third Round - START (22 april)

[*]Unmounted one of the two WD Velociraptor (300GB), I will use just one as cache drive, but I'm starting to have doubt

[*]Following the Johnm's suggestion closed all the hole in the fanplate

[*]Completed the cable fixing (picture available soon)

[*]Executing preclear again on the cache drive (200GB WD)

 


  •    
  • 1st 300GB WD Velociraptor SATA2 (WDC_WD3000BLFS-01YBU4_WD-WXH1E61CJXZ3) - Preclear running
         

    •        
  • Total Time: 3:25:16
           
  • Cycle Ran: 1
           
  • Pre Read Time: 0:51:45 (96 MB/s)
           
  • Zeroing Time: 0:52:10 (95 MB/s)
           
  • Post Read Time: 1:40:21 (49 MB/s)
           
  • Start TEMP: 40C
           
  • End TEMP: 49C
         

   

 

 

Third Round - END (running)

Share this post


Link to post

Here are the images (click to see a bigger version):

DSC03723.JPG

DSC03724.JPG

DSC03736.JPG

Starting from top left.[br]The three AOC-SAS2LP-MV8 Cards[br]The ASUS Video Card[br]Processor with Noctua cooler[br]Noctua 80mm fan[br]Emermax 1500W PSU[br]I'm missing one SF8087 cable...[br]All cable are - I think - perfectly settled[br]24GB RAM[br]WD Cache drive[br]Power cables - also - perfectly settled

To be noticed.[br]The cables work[br]The three Noctua S12B 120mm (wrong choice)[br]The isolation of  the fanplate holes

To be noticed.[br]The video card that occupy 2 slots (wrong choice)[br]The three SAS Card (SATA3 PCIe 8x)

 

DSC03742.JPG

DSC03743.JPG

IMG_0332.JPG

A better view of the three Noctua 120mm fan and the isolation I used to close the holes in the Norco fan plate

I also closed the holes above the PCIe slots

All the HDDs bays not occupied are closed with the same method/material (recycled from the Norco shipping package)

 

DSC03740.JPG

DSC03725.JPG

DSC03729.JPG

Here is the details of the cache disk mounting (I'm starting to think not to use it... see a performance post later on)

Here is the cables work for the power connections to the HDDs backplane

This is the only part I can still improve related to cable settling, I will do it when I will get the sixth missing SF8087 cable

 

DSC03722.JPG

Here is the case closed and laying on my table. I put also some silicon feet just for the time being (until it stays on the table)

 

 

 

MySlots.jpg

And this is the Slot Schema

 

Share this post


Link to post

A couple of questions.

 

1 - Raid0 as Cache drive

1.1) I attached the 2 WD HDD to the SAS port of the motherboad

1.2) I have created a RAID 0 with the 2 HDD

1.3) In unraid I keep seeing the 2 HDD as single disk no RAID is available

Any suggestions?

 

2 - HDD Temperature

2.1) All the Noctua Fan are mounted without reducing the speed

2.2) The air flow is present and in the right direction (cold air enter from the front)

2.3) During the pre-clear the HDDs (Hitachi 7K4000 4TB) get very HOT going sometime over 47C

Any suggestions?

 

Thanks and let me know what do you think of my compulsive unraid.

Share this post


Link to post

A couple of questions.

 

1 - Raid0 as Cache drive

1.1) I attached the 2 WD HDD to the SAS port of the motherboad

1.2) I have created a RAID 0 with the 2 HDD

1.3) In unraid I keep seeing the 2 HDD as single disk no RAID is available

Any suggestions?

Can't be done with the motherboard RAID. Those are software raid with hardware assist, you need Windows to do the software bit. You would need a true hardware raid card.

2 - HDD Temperature

2.1) All the Noctua Fan are mounted without reducing the speed

2.2) The air flow is present and in the right direction (cold air enter from the front)

2.3) During the pre-clear the HDDs (Hitachi 7K4000 4TB) get very HOT going sometime over 47C

Any suggestions?

Put the lid on the server? if you have the top off, no air goes across the drives.

if you have an older norco, make sure the bays with the drives are open and close off most of the empty bays.

If you have a new Norco (yellow backplanes), you might have to block off some of the empty bays in a creative way.

plug up all of the extra openings in the fanwall. the holes in the fanwall are way to big and cause a circular vortex inside the chassis.

your ambient room temperature is to high? it needs cool air (70F(ish) or lower)

I am running the same setup with Hitachi 3TB drives (5400's) and they are all low 30's

Thanks and let me know what do you think of my compulsive unraid.

 

EDIT:

I just noticed you are using the NF-S12B Fans.

You actually should be using the NF-P12 fans...

 

The high static pressure on the NF-P12's is what is needed to get the air to move through the tight space of the drive bays.

 

when you use the weaker low RPM fans, they actually just slow down straining to get the air to move through the tight space.

If you were to monitor the speed of those fans, You will see they are probably spinning much slower then they should be.

 

Unfortunately, there is a fine line between silent and and effective cooling. if you cross the line, your temps will raise to a danger point.

The design of the Norco proves this by the confined space of the drive bays and the unevenly pressured motherboard compartment.

By design, you have to build a high pressure area in the motherboard area requiring fans with high torque..

 

You will never Silence a Norco and have it work correctly. BUT you can get it whisper quiet with a bit of work.

Share this post


Link to post

Johnm

 

thanks for yours suggestions.

I will try as soon as the preclear is finished.

I'll get back as soon as I have some update.

Share this post


Link to post

Edited 2nd round with the results of the preclear script on all the 4TB HDDs

Share this post


Link to post

Ugh those disk temps are seriously ordinary ???    I've gotta agree with John here - either those fans or something with the case setup isn't right.

Share this post


Link to post

I'm uploading some picture of the setup inside the case after everything has been fixed up.

 

I followed the tips from Johnm.

Now last things I can do is to change the 3 120mm fan (they run at about 600rpm...).

 

Soon I'll post the pictures.

 

In the mean time started the 3rd round...  (see 2nd POST)

Share this post


Link to post

Will be interesting to see - for reference, I have 3x 120mm and 2x 80mm Arctic F PWM fans in my build.  Preclears never went above 36 degrees (ambient temperature anywhere between 20 and 25 degrees).  For the most part, they floated around the 32 degree mark.

 

 

Share this post


Link to post

Interesting data.

 

I will keep those in mind.

 

The temperature of my HDDs are rising even with just one in preclearing....

Share this post


Link to post

Wrapping up after the preclear script... I noticed this performances:

 

  [*]Cache drive - Plugged on SATA motherboard port (AHCI - SATA2 - 10.000 RPM

   


  •      
  • Pre Read Time: 0:52:02 (95 MB/s)
         
  • Zeroing Time: 0:50:10 (99 MB/s)
         
  • Post Read Time: 1:42:28 (48 MB/s)
       

 

  [*]Data Drive - AOC-SAS2LP-MV8 PCIe 8x - SATA3 - 7.200 RPM

   


  •      
  • Pre Read Time: 7:27:54 (148 MB/s)
         
  • Zeroing Time: 10:24:51 (106 MB/s)
         
  • Post Read Time: 17:32:47 (63 MB/s)
       

 

 

It does look like the cache disk is not going to change a lot...

 

What do you think guys?

Share this post


Link to post

I think in the end, you will find that the cache drive will help. You will only get about 1/3 of the data dirves speed when writing across the network in my experiences.

 

One think I did notice, in your pictures, you blocked the vents above the PCI slots. I think you might want to leave that open. that would allow the hot air from the cards to escape with a combination of natural convection and air pressure created by the fanwall.

 

Thats a big honking video card.. i would see if you can find a cheap basic card.

 

I like the feet idea...

 

it's starting to look like a server...

Share this post


Link to post

I think in the end, you will find that the cache drive will help. You will only get about 1/3 of the data dirves speed when writing across the network in my experiences.

I think I will see soon... planning to start copying stuff on it by tuesday (monday I'm out of the city for work)

One think I did notice, in your pictures, you blocked the vents above the PCI slots. I think you might want to leave that open. that would allow the hot air from the cards to escape with a combination of natural convection and air pressure created by the fanwall.

Thanks, I was thinking it would help... going to reverse that

Thats a big honking video card.. i would see if you can find a cheap basic card.

I know... I already saw some cheap video card with only one slot size... sooner or later...

I like the feet idea...

it's starting to look like a server...

Well I need to know what's needed to have it ending to look like a server  ;D

 

Thanks for your comments and suggestion.

Share this post


Link to post

I like the feet idea...

it's starting to look like a server...

Well I need to know what's needed to have it ending to look like a server  ;D

 

Thanks for your comments and suggestion.

 

DATA...  :P

 

For my old Q9950 server, I pulled an old PCIe ATI card from an old Dell or HP from work that was getting tossed in the trash.. price was perfect for a low-end card that cant do any HD video..

 

I do like that drive map template. I should make one for my server.

There is no need for the /dev/xxx.

That can change every time you boot your server. I would use the "Disk#" instead.

Share this post


Link to post

I do like that drive map template. I should make one for my server.

There is no need for the /dev/xxx.

That can change every time you boot your server. I would use the "Disk#" instead.

I shall change that!

 

Attached here the excel file...

MySlots_4224_Worksheet.zip

Share this post


Link to post

There is no need for the /dev/xxx.

That can change every time you boot your server. I would use the "Disk#" instead.

 

not if you use udev rules.

Share this post


Link to post

Hi Max,

 

I like the build so far. Did you fix those Harddisk temps?

 

Did you get all the parts in the Netherlands? I'm also interested in buying such a Norco case, how is availability and pricing in the Netherlands?

Share this post


Link to post

not if you use udev rules.

 

Could you please explain it a little more?

 

4.7 uses /dev/sda

 

from a 5.0 beta (not 5.0b1), unraid looks at the HD serialnumber (= udev rule)

 

Share this post


Link to post

Did you get all the parts in the Netherlands? I'm also interested in buying such a Norco case, how is availability and pricing in the Netherlands?

 

Price is what it is, look at:

 

http://cybershop.ri-vier.nl/4u-server-case-w-24-hotswappable-satasas-drive-bay-rpc4224-p-67.html    € 440.3

 

Yep, that is where I get it.

 

I got the case, the fanplate and the SFF-8087 cables from The Netherlands; the 3 AOC-SAS2LP-MV8 cards from the US; the CPU, motherboard, memory, fan, cables, video card, from germany and a couple of accessories from UK.

 

Share this post


Link to post

After a whole day without any use the HDDs temperatures are:

Dev

Align

Size

Destination

Model

Serial #

Temp

sdb

63

300GB

Cache

WDC WD3000BLFS-

WD-WXH1E61CJXZ3

41

sdc

1

4T

Data

Hitachi HDS7240

PK2310PAG16KJM

40

sdd

1

4T

Data

Hitachi HDS7240

PK2310PAG16ULM

38

sde

1

4T

Data

Hitachi HDS7240

PK2310PAG164PM

39

sdf

1

4T

Data

Hitachi HDS7240

PK2310PAG16M4M

39

sdg

1

4T

Data

Hitachi HDS7240

PK2310PAG16MTM

37

 

Next step will be to change the 120mm fan as suggested earlier by Johnm

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.