HYEPYC | High End EPYC Milan Build

[tazire]

What kind of CPU frequencies do you see your CPU getting up to? I am having a weird situation where once the array is started my CPU is pegged at 1.5Ghz... I can see that before the array is started the CPU boosts to its rated 3.35... Just curious if this is just me?

[Pri]

Usually only my completely idle cores are at 1.5GHz, active cores are between 3GHz and 3.8GHz. When there's low CPU utilisation and only say 3-4 threads are active I've seen them hit 4GHz.

 

No issues with CPU utilisation here, frequencies quite high all the time, I'd say 3.2GHz is the most common frequency I see due to multithreaded loads. I'm using the OnDemand CPU scaler with Performance Boost enabled.

 

 

Edited April 3, 2023 by Pri

[tazire]

yea m8 you almost answered my issue. I always thought unraid was pre set to on demand performance governor scheduling and it just never dawned on me to check it. Went into the tips and tricks plugin and low and behold it was set to powersave. Changed to on demand solved my issue. 

 

Server is very nearly finished... a busted 9201-16e has slowed up my finalized setup. new one on the way and then its done! Really have felt the pain of trying to go the ebay route of obtaining hardware.... thankfully the really expensive stuff was all perfect

 

[IMone]

ask yourself if you can mount redundant power supplies on the goxxi case?

[tazire]

24 minutes ago, IMone said:

ask yourself if you can mount redundant power supplies on the goxxi case?

you can if you want to. xcase sell it with redundant power supplies installed if you want them. 

[Pri]

17 hours ago, IMone said:

ask yourself if you can mount redundant power supplies on the goxxi case?

 

My case came with two brackets from Gooxi, they allowed for single or dual-redundant power supplies to be fitted. I also had a 3rd bracket from Xcase, the retailer I bought mine from which was an ATX bracket so that I could use a normal power supply and that was the one I installed.

 

So to be clear, that ATX bracket is not stock, but the single and dual server power supply brackets are standard from Gooxi and included in the cases accessory kit which ships with the case.

[IMone]

grazie

[Stankyjawnz]

I had a question for you, there are very few people who post builds online using a 9500 HBA. It is fairly well known the older LSI chipsets do not allow systems to achieve low idle power consumption because they prevent the cpu from entering lower power c-states. It seems like with the 9500 hba broadcom made more of an attempt to reduce power. Have you ever looked into what cpu c-states your server reaches with powertop? 

 

I actually have a very low performance needs (spinning rust) however there could be break even in multiple years if the 9500 can allow the system to idle well. 

 

[Pri]

3 hours ago, Stankyjawnz said:

I had a question for you, there are very few people who post builds online using a 9500 HBA. It is fairly well known the older LSI chipsets do not allow systems to achieve low idle power consumption because they prevent the cpu from entering lower power c-states. It seems like with the 9500 hba broadcom made more of an attempt to reduce power. Have you ever looked into what cpu c-states your server reaches with powertop? 

 

I actually have a very low performance needs (spinning rust) however there could be break even in multiple years if the 9500 can allow the system to idle well. 

 

 

I did try the powertop script months ago, it didn't have any effect for my system and by that I mean the power use never changed before or after. I think the lowest C state I saw was C2 but that's just going by my memory.

 

I did notice though that the power use of the x86 cores are already very low, they can go sub 0.25 watts. But the I/O die which handles PCIe, Memory, Infinity Fabric to/from the x86 cores uses a lot of energy, I think the lowest I saw it was 19.5 Watts while the x86 cores were all sub 0.25 etc

 

I suspect an Intel system would have better luck with testing what the 9500-8i can do sleep state wise.

[Stankyjawnz]

28 minutes ago, Pri said:

 

I did try the powertop script months ago, it didn't have any effect for my system and by that I mean the power use never changed before or after. I think the lowest C state I saw was C2 but that's just going by my memory.

 

I did notice though that the power use of the x86 cores are already very low, they can go sub 0.25 watts. But the I/O die which handles PCIe, Memory, Infinity Fabric to/from the x86 cores uses a lot of energy, I think the lowest I saw it was 19.5 Watts while the x86 cores were all sub 0.25 etc

 

I suspect an Intel system would have better luck with testing what the 9500-8i can do sleep state wise.

 

Thanks for your reply. The fact that you report your cores could go so low is promising.  I had similar results with the powertop --auto-tune script, I didn't find it that useful on its own, and it actually kept causing the ethernet controller to hang. I did find the tool itself to be fairly useful just to see what it reports as "bad" to see if there were more things to turn off in bios, etc. 

[tazire]

@Pri Just a question on the 9500... Is IT mode a thing on the newer HBA's? I keep getting errors with my 9300-16i so I've invested in a 9500 16i. Havent seen it mentioned anywhere and I'm guessing the newer HBA's work out of the box?

[Pri]

2 minutes ago, tazire said:

@Pri Just a question on the 9500... Is IT mode a thing on the newer HBA's? I keep getting errors with my 9300-16i so I've invested in a 9500 16i. Havent seen it mentioned anywhere and I'm guessing the newer HBA's work out of the box?

The 9500-8i I purchased is a IT mode HBA yes. It lacks the RAM for raid modes. You would have to buy a 9560 to get RAID functionality (RAID5 / RAID6 etc).

[tazire]

46 minutes ago, Pri said:

The 9500-8i I purchased is a IT mode HBA yes. It lacks the RAM for raid modes. You would have to buy a 9560 to get RAID functionality (RAID5 / RAID6 etc).

I've gone with the 9500-16i again though without the RAM. I havent seen anything anywhere about people having to flash the firmware to IT mode and was just guessing it came in IT mode. Cheers for the quick response. 

[IMone]

HI

I have the same gooxi case and a mb h12ssl - c, can I ask you for 2 courtesies?

1) Is there no way to connect the front panel USB sockets on the H12?

2) can you take a photo of the connectors on the front panel power, hd led etc. I'm a noob but something in the diagram on the instructions doesn't add up to me?

Thank you

[Pri]

To connect the front panel USB you will need to use a USB3 to USB2 adapter. I did buy one but didn't end up using it and just left the front disconnected.

 

I'm unable to take the photo you requested as the server is all powered on and everything and I can't turn it off etc

[IMone]

46 minuti fa, Pri ha detto:

Per collegare l'USB del pannello frontale sarà necessario utilizzare un adattatore da USB3 a USB2. Ne ho comprato uno ma alla fine non l'ho usato e ho semplicemente lasciato la parte anteriore scollegata.

 

Non riesco a scattare la foto richiesta poiché il server è tutto acceso e tutto e non posso spegnerlo, ecc.

I thank you

I have already solved both, I have already found the USB2-USB3 adapter, and for the connections I have found the problem, I am Italian and I am a mechanical designer, drawing details in the ISO system, things like the one I am about to describe give me difficulty, in the manual supermicro there is the drawing of the MB with the connection pins on the front panel circled, and under the connection diagram, but you look at the MB with the lens and a light (since, moreover, I can barely see it) you realize that between the MB drawing and schema are rotated 180 degrees

[Pri]

22 minutes ago, IMone said:

I thank you

I have already solved both, I have already found the USB2-USB3 adapter, and for the connections I have found the problem, I am Italian and I am a mechanical designer, drawing details in the ISO system, things like the one I am about to describe give me difficulty, in the manual supermicro there is the drawing of the MB with the connection pins on the front panel circled, and under the connection diagram, but you look at the MB with the lens and a light (since, moreover, I can barely see it) you realize that between the MB drawing and schema are rotated 180 degrees

 

Ah yes, you're 100% correct, the Supermicro motherboard has it inverted compared to the manual. I remember now having to think that one out aswell.

[IMone]

HI
I finally managed to finish the assembly and everything works, or almost, the only thing is the cache SSDs that only see one because I haven't yet entered the bios to correct the PCI Express bifurcation settings, I was wondering if there 'there was a way to do it remotely, not having installed a video card it would be inconvenient I would have to borrow one from a PC and it bothered me a little, I tried on "supermicro bmc", but couldn't find a way to access the bios remotely , at most I can update it

[Pri]

7 minutes ago, IMone said:

HI
I finally managed to finish the assembly and everything works, or almost, the only thing is the cache SSDs that only see one because I haven't yet entered the bios to correct the PCI Express bifurcation settings, I was wondering if there 'there was a way to do it remotely, not having installed a video card it would be inconvenient I would have to borrow one from a PC and it bothered me a little, I tried on "supermicro bmc", but couldn't find a way to access the bios remotely , at most I can update it

 

Yes you can access the IPMI KVM from the WebUI. That will allow you to enter the BIOS remotely over network etc

 

No video card is necessary as there is one included in the BMC chip of the motherboard.

[IMone]

Did you also have problems with the fans? bmc continues to report errors to me because the pwm fans do not reach 3500 rpm but only 2000 therefore it is a fault for the system. if so, how did you solve it? because I don't even see the fans in the various fan controls, like the one in Dynamix

Edited September 17, 2023 by IMone

[Pri]

Just now, IMone said:

Did you also have problems with the fans? bmc continues to report errors to me because the pwm fans do not reach 3500 rpm but only 2000 therefore it is a fault for the system. if so, how did you solve it? because I don't even see the fans in the various fan controls, like the one in Dynamix

Set the fan speed in the motherboards IPMI menus to 100% then use the IPMI Tools plugin in unRAID to set the RPM.

 

If you set the motherboards IPMI fan system to anything other than full speed / 100%, it will cause conflicts with the unRAID IPMI plugin.

[IMone]

hi, what did you choose as allocation method and division method? I don't understand why even though I chose high water and automatically divide each folder and I also tried automatically divide up to the fifth level, he currently puts all the files on the first disk and that's it

[Pri]

I've done a whole bunch of updates to the server since the initial build which I think would be good to share. Most recently I added an RTX A4000 for PLEX transcoding and AI/LLM use. In the initial build I did use a GTX 1080 Ti but its Pascal and a bit power hungry and lacks newer codec support that I needed. I removed the 1080 Ti a long time ago and just recently replaced it with the RTX A4000.

 

Main reason I chose this card is it's single-slot and so I'd only use up one slot instead of two. I still have one left unused that I may use for something in the future. Below are two photos of the card, I think it looks really quite beautiful.

 

 

 

I've found the card idles around 10 Watts with the appropriate idle power mode script applied, the fan is not audible to me and stays around 40% in my chassis.

 

In addition to adding this card I've also made some other system changes. I removed the Supermicro CPU cooler because I found it quite loud when the CPU got warm and I've since swapped it for an Arctic 4U-M which is basically inaudible.

 

The Supermicro had a 92mm fan which I found idled around 2,500 RPM to keep my chip cool and would go as high as 3,500 under load. The newer Arctic 4U-M cooler features two 120mm fans and runs much quieter with them spinning around 950 RPM at idle and 1200 RPM under load. Quite remarkable difference in not only temperatures but noise.

 

I also switched out my Intel X540-T2 for an X550-T2 because I needed the 2.5GbE capability that the X550 features for a cable modem I'm using while I needed its other 10GbE port for a Fibre modem from my second service provider. The X540 only features 1Gb/10Gb capability so it needed to be swapped out.

 

I also swapped out the generic PCIe to M.2 card I was using for the 4 x 980 Pro SSD's because it wasn't stable. After about 6 months I had issues with one or more SSD's randomly disconnecting from the system which I believe was down to the weak VRM situation on the PCIe card. For this reason I switched to an Asus HyperM.2 PCIe 4.0 card which has a much stronger VRM setup, since I made that switch about half a year ago I've had no more issues with my 980 Pros.

 

Another addition to the server was an Intel X710-DA4 card. This is a four port SFP+ (10Gb) card. I'm using this for the LAN side of my pfSense Virtual Machine, unRAID itself and a Windows VM. My motherboard does have dual 10GbE built in but I just prefer SFP+ and so I've stopped using those ports for the time being (but I may use them for VM's in the future that I want to have dedicated connectivity).

 

Below is a photo of the current internal state of the server with all the changes I mentioned above.

 

 

That's the update for right now. The system is still humming along dutifully.

Edited February 2 by Pri

[tazire]

Very nice updates. I'm using that cooler and 2 of the Asus hyper cards as well. I've been half looking at an a4000 myself. Currently using a p4000. But I'm also tempted to hang on a while and get an ada generation a4000 for av1 support when the prices drop a bit. The p4000 is plenty for the time being anyway. I added a chenbro 24 bay 2u chassis for ssd's and moved everything to my spare room and put it in a lack rack! Eventually I'll move it out to the shed but I need to insulate and run power and networking before I can do that. 

[Pri]

1 hour ago, tazire said:

Very nice updates. I'm using that cooler and 2 of the Asus hyper cards as well. I've been half looking at an a4000 myself. Currently using a p4000. But I'm also tempted to hang on a while and get an ada generation a4000 for av1 support when the prices drop a bit. The p4000 is plenty for the time being anyway. I added a chenbro 24 bay 2u chassis for ssd's and moved everything to my spare room and put it in a lack rack! Eventually I'll move it out to the shed but I need to insulate and run power and networking before I can do that.

I considered the same thing, to wait and get an RTX 4000 Ada 20GB instead or even buy one brand new. But when I looked at the situation codec wise.

 

Everything I get 4K wise is H.265 whether it's 4K HDR or 4K SDR. Stuff I get that isn't 4K is H.264. There's no source of media I use that has AV1 files but if they did, the RTX A4000 can decode AV1, it just can't encode in it. But all the clients I need to transcode for either support H.264 or H.265.

 

So it felt like a good fit. I think if I were to buy a newer card I'd want one with VVC (H.266) support which likely wont come until the next generation or even the one after that etc - That appears to be the codec that streamers will be settling on and the one to go mainstream, it's looking like AV1 will not make it big and in-fact some TV's at CES this year lacked AV1 support but had VVC support.

 

Something to think about!

Edited February 2 by Pri