UnRAID on Asus Pro WS W680-ACE IPMI

[venicenerd]

Hey guys,

 

I’m also considering this board for my Plex/VM gaming machine but I have some questions about the IOMMU groupings. 
 

I’d like to pass (bind) the following directly to my VM:

 

- RTX 4070 located in PCI 5.0x16 slot 

- USB 3 card located in PCI 3.0x1 slot

- One of the three M2 slots (for bare metal SSD access on the VM)

 

Then I’d like to use the two remaining PCI 3.0x4 slots for a 10G network card and a SATA expansion card. Those two would be used by the Unraid OS, though, and NOT passed on the the VM. 

 

Would that be possible with this board? I can’t find any schematics that outline the IOMMU groupings. 
 

Any help would be greatly appreciated!

 

PS: and just to confirm it is possible to use the IPMI card while using the igpu for Plex transcoding and the RTX4070 for my VM? A monitor would always be connected to the RTX4070 but it would be excluded from Unraid used (via binding to the VM)

[Daniel15]

7 hours ago, venicenerd said:

I can’t find any schematics that outline the IOMMU groupings. 
 

 

I've got the Micro ATX version which only has two M.2 slots. They're two separate controllers (one is on the CPU and the other is on the chipset) so they're each in a separate IOMMU group. I'm not sure about the full ATX version that has another M.2 slot that goes to the chipset - it's possible the two that go to the chipset are in the same IOMMU group but I don't really have a way to check

 

7 hours ago, venicenerd said:

PS: and just to confirm it is possible to use the IPMI card while using the igpu for Plex transcoding and the RTX4070 for my VM?

The person in the comment directly above yours did exactly this and got it working.

[venicenerd]

3 hours ago, Daniel15 said:

 

I've got the Micro ATX version which only has two M.2 slots. They're two separate controllers (one is on the CPU and the other is on the chipset) so they're each in a separate IOMMU group. I'm not sure about the full ATX version that has another M.2 slot that goes to the chipset - it's possible the two that go to the chipset are in the same IOMMU group but I don't really have a way to check

 

The person in the comment directly above yours did exactly this and got it working.

that’s encouraging to hear. Hopefully someone with the full ATX Version can chime in as well. Hoping all the PCI slots will have their own IOMMU group as well.
 

I also saw the board has 4 DDR RAM slots. Is it okay to just populate two of them or should you always populate all 4? Most DDR kits come in a set of two.  

[venicenerd]

On 5/28/2023 at 11:07 AM, NAS-newbie said:

I can report that I am successfully running UnRAID with the Asus Pro WS W680-ACE, a "Raptorlake" CPU and this Crucial memory (2 x 32GB DDR5 ECC sticks at 4800) for weeks. 

 

So I’m curious to hear everyone’s thoughts about ECC RAM vs “regular” RAM on this board. 
 

I was looking at the G.Skill Trident Z5 DDR5-6400. How much faster would this be for gaming compared to the 4800 ECC?

 

Also, if you go with the ECC RAM can you still overclock the CPU via XMP profiles or is that only possible with faster RAM? 
 

I’m basically going to use my Unraid machine as my Plex server and virtual Gaming machine. So I’m unsure which way to go in regards to RAM. 

[Daniel15]

7 hours ago, venicenerd said:

Is it okay to just populate two of them

Yes. With DDR you should always have pairs, so either two or four is fine. There's two slots on the motherboard labeled "A" and two labeled "B" - just make sure you use the A slots before the B slots. This is documented in the manual too

 

7 hours ago, venicenerd said:

How much faster would this be for gaming compared to the 4800 ECC?

It depends on the game and how heavy its RAM usage is. LTT has this video:

I usually prefer linking to Gamers Nexus instead (their testing is way more thorough) but they don't seem to have a video about RAM speeds, just about DDR4 vs 5.

 

7 hours ago, venicenerd said:

Also, if you go with the ECC RAM can you still overclock the CPU via XMP profiles or is that only possible with faster RAM? 

Generally you'd only use ECC RAM at "standard" RAM speeds, meaning no XMP profiles for the RAM.

Overclocking the CPU is probably still possible but I haven't tried overclocking a CPU in maybe 15 years now. The turbo speeds are usually more than sufficient for me, and you can bump the maximum wattage for non-turbo modes in the BIOS.

[frodr]

I have this board, and I guess I'm the unlucky duck here. I had all kinds of trouble:

• Startup sequence randomly stuck on F6, specially after adding/removing pcie cards.
• Startup sequence randomly stays on OData Server info screen forever, or fully stuck.
• IPMI card looses contact with mainboard, sometime I have to physically remove/install it to get it up and running. 
• IPMI remote connection fall out "all the time".
• IPMI remote control due not follow the startup procedure until end. 
• Bios resets itself to default, mostly after tuning main power off. 

I'm in contact with Asus Support, that is sometimes like talking to a dement person asking the same question over and over again. 

 

A question to this board. Is the 2 pcie5 slots "connected? My understanding is the if I run 16 lanes card in slot 1, there is only 4 lanes to the CPU in pcie slot2? Does this mean a 8 lane card in pcie slot will not work? Or will it work with 4 lanes bandwidth?

 

// 

[eicar]

1 hour ago, frodr said:

A question to this board. Is the 2 pcie5 slots "connected? My understanding is the if I run 16 lanes card in slot 1, there is only 4 lanes to the CPU in pcie slot2? Does this mean a 8 lane card in pcie slot will not work? Or will it work with 4 lanes bandwidth?

It's x16/NA or x8/x8 for the two gen5 slots, i.e. if you run 16 lanes in slot 1, the second gen5 slot cannot be used. However, you still have PCIe 4.0 x4 left, which is used by the CPU-direct M.2 slot.

Edited September 18, 2023 by eicar

[eicar]

fyi… comparing the Asus Pro WS 680-ACE (IPMI) and the new ASRock Industrial IMB-X1714:

 

https://forums.unraid.net/topic/145312-asrock-industrial-imb-x1714-atx-intel-w680/

[frodr]

22 minutes ago, eicar said:

fyi… comparing the Asus Pro WS 680-ACE (IPMI) and the new ASRock Industrial IMB-X1714:

 

https://forums.unraid.net/topic/145312-asrock-industrial-imb-x1714-atx-intel-w680/

 

I have ordered this Supermicro. I want the IPMI functionality if possible. 

[JimmyGerms]

Hey all!

 

I wanted to share my experience as well. Just got it up and running. Took a bit but things seem to be smooth with the IPMI (for those who are pulling their hair out to get to the web interface put 'https://' in front of your BMC IP)

 

I did have issues when I updated to to the latest bios with my RAM. It kept down-clocking them to 3200 instead of 4800 and I could not get the speeds back up without going back to bios version 2602.

 

One question I had for the group, is there a way to tie the fan speed to HDD temps? I currently don't see a way without buying separate temp probes and using that in the fan settings. There doesn't seem to be a way to use anything other then CPU or temp probes. I learned the Fan tab in the IPMI tools plugin only appears for ASRock and SuperMicro boards. Wonder if there's a way to hack that?

[Daniel15]

8 minutes ago, JimmyGerms said:

is there a way to tie the fan speed to HDD temps?

I couldn't find an out-of-the-box way, so I just set the speed of the fans closest to the HDDs to a constant speed. They're Noctua fans so they're not too loud even if I run them at 60% constantly (plus my server is in a closet away from all the bedrooms, so noise is less of an issue)

 

The fan speeds are controllable via IPMI since it's visible in the web interface, so there should be a way to do it programmatically too. You could see if the fan speeds can be controlled via ipmitool or ipmicfg at the command line, and if so, it should be possible to script it or update the IPMI Tools plugin to handle it.

Edited September 18, 2023 by Daniel15

[frodr]

14 minutes ago, JimmyGerms said:

Hey all!

 

I wanted to share my experience as well. Just got it up and running. Took a bit but things seem to be smooth with the IPMI (for those who are pulling their hair out to get to the web interface put 'https://' in front of your BMC IP)

 

I did have issues when I updated to to the latest bios with my RAM. It kept down-clocking them to 3200 instead of 4800 and I could not get the speeds back up without going back to bios version 2602.

 

One question I had for the group, is there a way to tie the fan speed to HDD temps? I currently don't see a way without buying separate temp probes and using that in the fan settings. There doesn't seem to be a way to use anything other then CPU or temp probes. I learned the Fan tab in the IPMI tools plugin only appears for ASRock and SuperMicro boards. Wonder if there's a way to hack that?

This one in IPMI?

 

[JimmyGerms]

Yup, but changing the source to look at HDD or HDD pool. Doesn't look like there's a default way. Only other options in here are temp probes.

Edited September 18, 2023 by JimmyGerms

[frodr]

10 minutes ago, JimmyGerms said:

Yup, but changing the source to look at HDD or HDD pool. Doesn't look like there's a default way. Only other options in here are temp probes.

 

 

You have the fans I miss in my setup. Wonder why?

 

[JimmyGerms]

Under "Monitor" in the bios you can switch it over to IPMI. Hope that helps!

[JimmyGerms]

On 6/22/2023 at 4:04 PM, Omid said:

So, I've finally reached some challenges with this motherboard. Well, more specifically, its IPMI card which doesn't seem to be documented too well. For starters, the full manual (only available online) says that the box should contain an SPI cable and a TPM adapter; neither of which I appear to have. This is also inconsistent with the list of accessories the Quick Start guide mentions, which doesn't include those two items. But never mind, it doesn't seem like those are mandatory.

 

It would be great if anyone who purchased this motherboard (the IPMI variant) could confirm what cables they received in the box.

 

The BIOS detects and shows some details from the IPMI card, so it appears to be functioning fine. My issue now is that I can open the web interface, but it's not accepting the default credentials (admin:admin). I can't find a way to change/reset the password via the BIOS or via something like a jumper. Some results online for older versions suggest flashing the card or potentially running some commands to do it, but I'd need to boot into something and I'd rather do that before moving Unraid into this build.

 

I'm wondering if anyone can confirm what their default credentials were. I've already tried many variations with CAPS, the full administrator word, etc.

Thanks!

@Omid take a look at this PDF for a password reset within windows or linux. Section is called "How to Reset the IPMI expansion card login password"

https://dlcdnets.asus.com/pub/ASUS/mb/Add-on_card/IPMI_EXPANSION_CARD/IPMI_Expansion_Card_FAQ_EN.pdf?model=IPMI EXPANSION CARD

[Daniel15]

6 hours ago, JimmyGerms said:

@Omid take a look at this PDF for a password reset within windows or linux. Section is called "How to Reset the IPMI expansion card login password"

https://dlcdnets.asus.com/pub/ASUS/mb/Add-on_card/IPMI_EXPANSION_CARD/IPMI_Expansion_Card_FAQ_EN.pdf?model=IPMI EXPANSION CARD

 

I used Supermicro's ipmicfg to reset the password. It works well with other brands too. ipmicfg -user list to get a list of all the users and their IDs, then ipmicfg -user setpwd 1 new_password to set user ID 1's password to new_password.

 

The default admin user is either ID 1 or 2; I can't remember.

Edited September 19, 2023 by Daniel15

[eicar]

Has anyone had any experience using 48GB DDR5 ECC UDIMMs on this MB? The W680 boards by Asus and Supermicro officially support 192GB max—the ASRock Industrial board doesn't (yet?)—, and according to Intel, the 13th generation Core CPUs do too: https://ark.intel.com/content/www/us/en/ark/products/230580.html

 

At any rate, manufacturers seem to be slowly rolling them out, e.g. here: https://www.smartm.com/product/ddr5-udimm-ecc

[eicar]

19 hours ago, frodr said:

I have ordered this Supermicro. I want the IPMI functionality if possible. 

It's built-in IPMI, no extra card, right? This is nice, actually. The X13SAE-F is a really good alternate option.

[frodr]

10 hours ago, eicar said:

It's built-in IPMI, no extra card, right? This is nice, actually. The X13SAE-F is a really good alternate option.

 

That's right. IPMI onboard. Nice to get rid of 3 untidy cables. 

[JimmyGerms]

For anyone who is using the Kingston 32gig 4800 RAM (KSM48E40BD8KM-32HM), is there an XMP profile selection for you? I'm curious if I'm just overlooking where the option is or do these modules not have an XMP?

 

EDIT: Has anyone done any CPU undervolting? If so, do you use the "Actual VRM Core Voltage" setting? There's no documentation in the manual for that one

Edited September 20, 2023 by JimmyGerms

[firstTimer]

@JimmyGerms And anyone interested I created a special thread for optimisations of this MB with a i7 13700k

 

Edited September 20, 2023 by firstTimer

[Daniel15]

On 9/19/2023 at 1:52 PM, frodr said:

 

That's right. IPMI onboard. Nice to get rid of 3 untidy cables. 

 

The MicroATX version of this motherboard ahas onboard IPMI, so I'm surprised the ATX version doesn't...

 

17 hours ago, JimmyGerms said:

is there an XMP profile selection for you?

You generally don't get XMP profiles for ECC RAM, as the ECC functionality is only tested with standard (non-XMP) memory speeds.

[JimmyGerms]

I'm at my wits end. I am trying to under-volt the board to give me some thermal headroom on the 13900k but I cannot get good performance. Anything I touch with ACLL, DCLL, Actual Voltage or Global Voltage immediately drops performance by half. In Cinebench R23 I get 16k instead of the 35-36k when set at default auto settings (But I hit 100 degrees).

 

Intel XTU also is all greyed out so I cant even test adjust there...no idea what is going on. UnderVolt Protection is disabled too. I'm on latest BIOS and read a thread about switching microcode as well which I've done but same results.

 

What the hell is the correct way to do it on this board?

Edited September 21, 2023 by JimmyGerms

[eicar]

On 9/21/2023 at 1:22 AM, Daniel15 said:

The MicroATX version of this motherboard ahas onboard IPMI, so I'm surprised the ATX version doesn't...

Maybe for space reasons? You'd need an additional PCIe slot for optional IPMI.

 

fyi: there's an industrial variant of this board (available? in the making?), the R680EA-IM-A:

https://www.asus.com/networking-iot-servers/aiot-industrial-solutions/industrial-motherboards/r680ea-im-a/

datasheet incl. block diagram: https://www.asus.com/Websites/global/products/e8zjxlzraxczv2lr/pdf/wot1ujdkz9kusqdy.pdf

It has the R680E chipset, which is the embedded version of the W680, i.e. the same number of PCIe lanes on the PCH (12 x 4.0 + 16 x 3.0) in addition to the CPU lanes with 16 x 5.0 + 4 x 4.0. Same DMI (4.0 x8).

Main difference: 7 SATA ports, i.e. no MiniSAS. SATA port #8 is missing, because you can use the CPU-direct gen4 M.2 slot in SATA mode via the R680E and an ASM2480B.

There are no gen4 M.2 slots via the PCH, but that's why you get three x4 PCIe slots in addition to the two gen5 CPU-direct slots (x16/NA or x8/x8): two PCIe 4.0 x4 open slots and one PCIe 3.0 x4 open slot, plus two legacy 32-bit PCI slots coming from a single lane of PCIe 3.0.

Pretty nice, I think.