UnRAID on Asus Pro WS W680-ACE IPMI

[Daniel15]

19 minutes ago, Omid said:

as long as you have enough clearance from your cooler

 

I've got the MicroATX version (W680M-ACE SE) and can't even use my PCIe x1 slot as my Noctua NH-D15 cooler partially covers both the PCIe slot and one M.2 slot, so to remove the M.2 I have to remove the CPU cooler first 😂. The MicroATX one has a different layout where the M.2 slot sits where the RAM normally is, so RAM clearance isn't an issue at least. Not sure what I was thinking putting a giant CPU cooler on a MicroATX board, but at least it keeps the CPU nice and cool (~30°C with ~20% CPU usage even though the system is sitting in a closet).

 

What I don't understand is that since they could embed the BMC/IPMI directly on the board in the MicroATX version, why couldn't they also do that on the ATX version where there's even more space available on the board?? So weird.

Edited October 30, 2023 by Daniel15

[NAS-newbie]

Still no experiences of running this board with the full 128GB RAM? Curious about what ECC sticks that works for this and what speed that was possible?

[SShadow]

Quote

Curious about what ECC sticks that works for this and what speed that was possible?

 

@NAS-newbie  I am running the Pro WS W680M-ACE SE (mATX) version of this board and have two:

Kingston 32GB 4800MHz CL40 DDR5 ECC Memory Module KSM48E40BS8KM-32HM.
It runs at the advertised 4800MHz.  Everything appears to work fine so far.  I have only had it running for two weeks. 

Edited November 3, 2023 by SShadow

[NAS-newbie]

Yes I also run with two 32GB modules at 4800 (and do does several people who have reported this in the thread) I but would like to know if anybody managed to run with 4x32GB and in that case with what modules and what speed...

[Daniel15]

6 minutes ago, NAS-newbie said:

Yes I also run with two 32GB modules at 4800 (and do does several people who have reported this in the thread) I but would like to know if anybody managed to run with 4x32GB and in that case with what modules and what speed...

 

If 2 x 32GB works, then 4 x 32GB should also work with the same type of RAM. In theory the board supports 48GB DIMMs too (so you could do 2 x 48GB to get 96GB total RAM) but I haven't seen my suppliers actually stocking 48GB DIMMs. Seems like they're still rare.

[Zerax]

On 11/4/2023 at 6:30 AM, NAS-newbie said:

Yes I also run with two 32GB modules at 4800 (and do does several people who have reported this in the thread) I but would like to know if anybody managed to run with 4x32GB and in that case with what modules and what speed...

I run 4x32GB Kingston KSM48E40BD8KM-32HM on a WS W680-ACE IPMI, the board will sett the modules automatically to to 4200.  

[casperse]

4 hours ago, Roger Jemterud said:

I run 4x32GB Kingston KSM48E40BD8KM-32HM on a WS W680-ACE IPMI, the board will sett the modules automatically to to 4200.  

Can you manually set it to 4800? and if so is it stable?

[Daniel15]

2 hours ago, casperse said:

Can you manually set it to 4800? and if so is it stable?

ECC RAM generally doesn't let you change the timings. The ECC functionality is only guaranteed to work properly at the standard speed.

Having said that, this RAM is 4800MT/s so I'm not sure why it's only being detected at 4200.

[Zerax]

4 hours ago, casperse said:

Can you manually set it to 4800? and if so is it stable?

 

1 hour ago, Daniel15 said:

ECC RAM generally doesn't let you change the timings. The ECC functionality is only guaranteed to work properly at the standard speed.

Having said that, this RAM is 4800MT/s so I'm not sure why it's only being detected at 4200.

 

Looks like manually set it to 4800MT/s works, no issues yet. 

 

Edited November 14, 2023 by Roger Jemterud

[casperse]

On 6/22/2023 at 10:50 AM, Omid said:

Thanks for confirming and for the link! Now to try find the cable in the UK... 😅 

 

And I totally agree with you regarding the motherboard. The build quality and finish isn't too far off ASUS's ROG series of motherboards, so I feel like it's using the same premium components. Its features, ports and PCIe configurations seem very appropriate for an Unraid build (e.g. no bifurcation when maxing out M.2 slots or the SlimSAS connector). My only qualm is that the big cooler I got for the 13700k, the Noctua NH-D15, will be almost touching the IPMI card if I put it in the top PCIe x1 slot (1mm gap; maybe less!). I'm pretty sure they will eventually touch after some time, because...gravity.

 

I'll confirm that I received 2x 32GB Kingston (KSM48E40BD8KM-32HM) and am currently putting them through memtest where they were detected as supporting ECC.

Hi @Omid I am in the same situation, what CPU cooler did you get in order to have the IPMI in the first slot?
I really wanted the Noctua NH-D15 and cant find a cooler that is in the same league and can fit?

[Omid]

5 hours ago, casperse said:

Hi @Omid I am in the same situation, what CPU cooler did you get in order to have the IPMI in the first slot?
I really wanted the Noctua NH-D15 and cant find a cooler that is in the same league and can fit?

I decided to keep my NH-D15 and instead moved the IPMI card to one of the x16 slots at the bottom of the board. It's annoying, I know, but won't be a problem for me as I've only got 1 other card in there right now and it's for storage.

 

Honestly, I think that, with a bit of work, I could create a ~2mm gap between the cooler and the card if it's in the x1 slot. I just didn't have the time when I was doing it. If you're really creative and up for the challenge, I think you could make it work with NH-D15 and top slot. It was really close; I had to look very closely to see how small the gap was. Ultimately, I wasn't happy or comfortable with closing the case with that configuration. When I was thinking of options, I was exploring a combination of the following:

• Mounting the cooler just a couple of mm higher and screwing it down
• Place a thin insulation layer (plastic/rubber sheet) on top of the IPMI board
• Maybe shave/fold/clip off a mm of the cooler's fins
• File down/clip off some of the soldered tips of the IPMI board

As you would probably agree, none of these are great solutions and I really dislike the last two.

 

There's also the idea of using a riser cable/card of some sort, but I didn't really explore that much, since I would also have to figure out how the network cable would reach the card.

Edited November 22, 2023 by Omid

[Zerax]

18 hours ago, casperse said:

Hi @Omid I am in the same situation, what CPU cooler did you get in order to have the IPMI in the first slot?
I really wanted the Noctua NH-D15 and cant find a cooler that is in the same league and can fit?

 

I use the Noctua NH-U12A with Thermalright Contact Frame on i9-13900T.

And the IPMI card fits in the first slot. In many cases the NH-U12A perform beter then NH-D15.

 

 

 

Edited November 23, 2023 by Zerax

[Omid]

5 hours ago, Zerax said:

 

I use the Noctua NH-U12A with Thermalright Contact Frame on i9-13900T.

And the IPMI card fits in the first slot. In many cases the NH-U12A perform beter then NH-D15.

 

 

 

 

Damn, I wish I had seen this video before! @casperse This should be great info for you! 

[casperse]

On 11/23/2023 at 11:43 AM, Omid said:

 

Damn, I wish I had seen this video before! @casperse This should be great info for you! 

Yes I also found that video, looking into the data sheet for size, looks okay!
(Just got the ASUS Pro WS W680-ACE IPMI on a Black Friday deal) - now looking at getting the right CPU?
Since the platform is end of live (Intel will have new sockets in 2024) I thought better to go high!
i9 14900K and the "tame" the power usage? - That way I can dial it up later if I need extra speed/computing power?
Also my last build I always regretted that I only got 64G ram and the Intel’s Xeon E-2176G should have maxed it out.
(Now that's going to be the backup server)

[JimmyGerms]

For anyone that has a 13900k, does unraid report it as 2970 MHz for the base clock speed on this board? I'm not sure what's causing this but it's also causing me not to hit my full turbo either. Always just short.

 

[Daniel15]

23 minutes ago, JimmyGerms said:

For anyone that has a 13900k, does unraid report it as 2970 MHz for the base clock speed on this board? I'm not sure what's causing this but it's also causing me not to hit my full turbo either. Always just short.

 

 

The speed shown here is arbitrary and honestly I don't know why Unraid even shows it. The clock speed on modern CPUs varies per core so I don't know where it's getting a single speed from. It might be the average across all cores at some arbitrary point in time. There's no such thing as a base clock with Intel's scaling driver - it uses a minimum and maximum, and scales between them.

You can run this command to watch the current speed across all cores, updated once per 100ms:

watch -n0.1 'cat /proc/cpuinfo | grep MHz'

 

This will show you the configured minimum frequency in Hz, per core. For my i5-13500 it's 800Mhz by default:

cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_min_freq

 

This will show the max (4.8Ghz for mine):

cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_max_freq

 

These are both configurable - for example, if you want to save power, the max may be lower than the actual maximum supported by the processor. Check cpuinfo_min_freq and cpuinfo_max_freq (cpuinfo_ prefix instead of scaling_ prefix) to see the minimum and maximum supported by the processor.

 

If scaling_max_freq is lower than cpuinfo_max_freq, you can run this to set it to the maximum:

for x in /sys/devices/system/cpu/cpu*/cpufreq/; do
        cat $x/cpuinfo_max_freq > $x/scaling_max_freq
done

 

(stick it in /boot/config/go to make it run on every boot)

 

You can bump the minimum higher if you want to, too.

 

Also check if /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor is set to "powersave" - that'll tell the CPU to prefer lower clock speeds.

 

Finally, check if turbo is disabled:

cat /sys/devices/system/cpu/intel_pstate/no_turbo

(0 means enabled and 1 means disabled)

Edited December 3, 2023 by Daniel15

[Daniel15]

On 11/26/2023 at 4:50 AM, casperse said:

i9 14900K and the "tame" the power usage? - That way I can dial it up later if I need extra speed/computing power?

Yeah you can either reduce the power limits from their defaults in the BIOS, or configure the CPU governor to use powersave mode and reduce the maximum clock speed. The "Tips and Tweaks" plugin can do some of that - it sets some powersave settings (but not all of them), and lets you disable turbo. 

[golfsands7]

Would anyone be kind enough to share the IOMMU groups of this board? (Link to command)

 

In a different thread it looked like that all of the M.2 slots + all of the SATA ports are all in the same group and must therefore all be passed to a VM together. I'm new, but that seems odd to me. And indeed someone with the M-ATX version shared IOMMU groups that suggested they were not all together. Anyway I'd love if someone could confirm what they see or share their experiences. 

[Omid]

Help! I've just hooked up an LSI 9201-8i HBA to this motherboard and Unraid won't boot.

 

I get POST with the Asus splash screen showing, then the IPMI card initialising with the firmware/OData configuration step, as usual, but then it changes to a completely black screen. No cursor, blinker or anything. I left it 10+ mins and confirmed on the router that the server has not connected. The screen shows black through IPMI and with a VGA monitor attached.

 

Have I missed any special steps? It's my first time with an HBA card. I added it to the top PCIe 5.0 x16 slot.

 

Next steps:

1. Check BIOS to downgrade PCIe version, although I thought PCIe was good at backwards-compatibility
   • Done: I forced PCIe Gen2 - no success
2. Remove the HBA card just to see if I can get a boot again
   • Done: Confirmed - successful boot
3. Try a different PCIe slot
   • Done: I tried one of the PCIe 3.0 slots and also tried it with Gen2 forced - no success

 

In the 5.0 slots, Q-LED would mostly stay on 40 for ~10 seconds and briefly switch to 39 for just a couple of seconds, before returning to 40 [repeat]. In the 3.0 slots, Q-LED would instead mostly stay on 38 and briefly switch to 37 or 39 but never 40. I assume the slight variation in Q-LED codes is due to the PCIe 5.0 lanes running to the CPU whilst the PCI 3.0 lanes via PCH (?).

 

Regardless, these codes don't help much and only seem to confirm that there's some issue with the HBA card...unless someone can interpret this better than I can?

 

Anyone have any other ideas?

Could it be the card's firmware? Would an older firmware have such a significant negative impact?

How can I check the current firmware version?

 

EDIT: Based on this thread I have the following next steps.  Interestingly, what resolved the OP's issue in that thread was switching from an ASRock board to an Asus one 🤦‍♂️

 

1. Check boot device order (I stupidly haven't checked this yet)
   • Done: looks good, no different to when card is detached
2. Check and disable "legacy storage boot" in BIOS
   • I can't find this setting anywhere -- any help?
3. Insulate some PCIe pins (sounds scary)
   • Done: no success

 

@golfsands7 I'm unfamiliar but will try to get that information for you when I'm up and running again. See below

Edited December 9, 2023 by Omid

[Omid]

@golfsands7 Here are the IOMMU groups on my Asus Pro WS W680-ACE IPMI (ATX, not mATX), in two different formats:

 

IOMMU Group 0 00:02.0 VGA compatible controller [0300]: Intel Corporation Raptor Lake-S GT1 [UHD Graphics 770] [8086:a780] (rev 04)
IOMMU Group 1 00:00.0 Host bridge [0600]: Intel Corporation Device [8086:a703] (rev 01)
IOMMU Group 2 00:01.0 PCI bridge [0604]: Intel Corporation Device [8086:a70d] (rev 01)
IOMMU Group 3 00:06.0 PCI bridge [0604]: Intel Corporation Raptor Lake PCIe 4.0 Graphics Port [8086:a74d] (rev 01)
IOMMU Group 4 00:0a.0 Signal processing controller [1180]: Intel Corporation Raptor Lake Crashlog and Telemetry [8086:a77d] (rev 01)
IOMMU Group 5 00:0e.0 RAID bus controller [0104]: Intel Corporation Volume Management Device NVMe RAID Controller Intel Corporation [8086:a77f]
IOMMU Group 6 00:14.0 USB controller [0c03]: Intel Corporation Alder Lake-S PCH USB 3.2 Gen 2x2 XHCI Controller [8086:7ae0] (rev 11)
IOMMU Group 6 00:14.2 RAM memory [0500]: Intel Corporation Alder Lake-S PCH Shared SRAM [8086:7aa7] (rev 11)
IOMMU Group 7 00:15.0 Serial bus controller [0c80]: Intel Corporation Alder Lake-S PCH Serial IO I2C Controller #0 [8086:7acc] (rev 11)
IOMMU Group 7 00:15.1 Serial bus controller [0c80]: Intel Corporation Alder Lake-S PCH Serial IO I2C Controller #1 [8086:7acd] (rev 11)
IOMMU Group 7 00:15.2 Serial bus controller [0c80]: Intel Corporation Alder Lake-S PCH Serial IO I2C Controller #2 [8086:7ace] (rev 11)
IOMMU Group 8 00:16.0 Communication controller [0780]: Intel Corporation Alder Lake-S PCH HECI Controller #1 [8086:7ae8] (rev 11)
IOMMU Group 8 00:16.3 Serial controller [0700]: Intel Corporation Device [8086:7aeb] (rev 11)
IOMMU Group 9 00:17.0 SATA controller [0106]: Intel Corporation Alder Lake-S PCH SATA Controller [AHCI Mode] [8086:7ae2] (rev 11)
IOMMU Group 10 00:1a.0 PCI bridge [0604]: Intel Corporation Alder Lake-S PCH PCI Express Root Port #25 [8086:7ac8] (rev 11)
IOMMU Group 11 00:1b.0 PCI bridge [0604]: Intel Corporation Device [8086:7ac0] (rev 11)
IOMMU Group 12 00:1b.4 PCI bridge [0604]: Intel Corporation Device [8086:7ac4] (rev 11)
IOMMU Group 13 00:1c.0 PCI bridge [0604]: Intel Corporation Alder Lake-S PCH PCI Express Root Port #1 [8086:7ab8] (rev 11)
IOMMU Group 14 00:1c.1 PCI bridge [0604]: Intel Corporation Alder Lake-S PCH PCI Express Root Port #2 [8086:7ab9] (rev 11)
IOMMU Group 15 00:1d.0 PCI bridge [0604]: Intel Corporation Alder Lake-S PCH PCI Express Root Port #9 [8086:7ab0] (rev 11)
IOMMU Group 16 00:1f.0 ISA bridge [0601]: Intel Corporation Device [8086:7a88] (rev 11)
IOMMU Group 16 00:1f.3 Audio device [0403]: Intel Corporation Alder Lake-S HD Audio Controller [8086:7ad0] (rev 11)
IOMMU Group 16 00:1f.4 SMBus [0c05]: Intel Corporation Alder Lake-S PCH SMBus Controller [8086:7aa3] (rev 11)
IOMMU Group 16 00:1f.5 Serial bus controller [0c80]: Intel Corporation Alder Lake-S PCH SPI Controller [8086:7aa4] (rev 11)
IOMMU Group 17 01:00.0 SATA controller [0106]: Marvell Technology Group Ltd. 88SE9215 PCIe 2.0 x1 4-port SATA 6 Gb/s Controller [1b4b:9215] (rev 11)
IOMMU Group 18 02:00.0 Non-Volatile memory controller [0108]: Sandisk Corp Western Digital WD Black SN850X NVMe SSD [15b7:5030] (rev 01)
IOMMU Group 19 03:00.0 Non-Volatile memory controller [0108]: Sandisk Corp Western Digital WD Black SN850X NVMe SSD [15b7:5030] (rev 01)
IOMMU Group 20 05:00.0 Non-Volatile memory controller [0108]: Sandisk Corp Western Digital WD Black SN850X NVMe SSD [15b7:5030] (rev 01)
IOMMU Group 21 06:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I226-LM [8086:125b] (rev 06)
IOMMU Group 22 07:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I226-LM [8086:125b] (rev 06)
IOMMU Group 23 08:00.0 PCI bridge [0604]: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge [1a03:1150] (rev 06)
IOMMU Group 23 09:00.0 VGA compatible controller [0300]: ASPEED Technology, Inc. ASPEED Graphics Family [1a03:2000] (rev 52)
IOMMU Group 23 09:01.0 IPMI Interface [0c07]: ASPEED Technology, Inc. Device [1a03:2402] (rev 01)

 

Alternative format:

Group 0:        [8086:a780] [R] 00:02.0  VGA compatible controller                Raptor Lake-S GT1 [UHD Graphics 770]
Group 1:        [8086:a703]     00:00.0  Host bridge                              Device a703
Group 2:        [8086:a70d] [R] 00:01.0  PCI bridge                               Device a70d
Group 3:        [8086:a74d] [R] 00:06.0  PCI bridge                               Raptor Lake PCIe 4.0 Graphics Port
Group 4:        [8086:a77d]     00:0a.0  Signal processing controller             Raptor Lake Crashlog and Telemetry
Group 5:        [8086:a77f]     00:0e.0  RAID bus controller                      Volume Management Device NVMe RAID Controller Intel Corporation
Group 6:        [8086:7ae0]     00:14.0  USB controller                           Alder Lake-S PCH USB 3.2 Gen 2x2 XHCI Controller
USB:            [05ac:0250]              Bus 001 Device 004                       Apple, Inc. Aluminium Keyboard (ISO) 
USB:            [05ac:1006]              Bus 001 Device 002                       Apple, Inc. Hub in Aluminum Keyboard 
USB:            [046b:ff10]              Bus 001 Device 009                       American Megatrends, Inc. Virtual Keyboard and Mouse 
USB:            [046b:ffb0]              Bus 001 Device 008                       American Megatrends, Inc. Virtual Ethernet. 
USB:            [046b:ff01]              Bus 001 Device 007                       American Megatrends, Inc. Virtual Hub 
USB:            [1b1c:1c1e]              Bus 001 Device 006                       Corsair HX1000i Power Supply 
USB:            [05e3:0608]              Bus 001 Device 005                       Genesys Logic, Inc. Hub 
USB:            [0781:5567]              Bus 001 Device 003                       SanDisk Corp. Cruzer Blade 
USB:            [1d6b:0002]              Bus 001 Device 001                       Linux Foundation 2.0 root hub 
USB:            [1d6b:0003]              Bus 002 Device 001                       Linux Foundation 3.0 root hub 
                [8086:7aa7]     00:14.2  RAM memory                               Alder Lake-S PCH Shared SRAM
Group 7:        [8086:7acc]     00:15.0  Serial bus controller                    Alder Lake-S PCH Serial IO I2C Controller #0
                [8086:7acd]     00:15.1  Serial bus controller                    Alder Lake-S PCH Serial IO I2C Controller #1
                [8086:7ace]     00:15.2  Serial bus controller                    Alder Lake-S PCH Serial IO I2C Controller #2
Group 8:        [8086:7ae8]     00:16.0  Communication controller                 Alder Lake-S PCH HECI Controller #1
                [8086:7aeb]     00:16.3  Serial controller                        Device 7aeb
Group 9:        [8086:7ae2]     00:17.0  SATA controller                          Alder Lake-S PCH SATA Controller [AHCI Mode]
Group 10:       [8086:7ac8] [R] 00:1a.0  PCI bridge                               Alder Lake-S PCH PCI Express Root Port #25
Group 11:       [8086:7ac0]     00:1b.0  PCI bridge                               Device 7ac0
Group 12:       [8086:7ac4] [R] 00:1b.4  PCI bridge                               Device 7ac4
Group 13:       [8086:7ab8] [R] 00:1c.0  PCI bridge                               Alder Lake-S PCH PCI Express Root Port #1
Group 14:       [8086:7ab9] [R] 00:1c.1  PCI bridge                               Alder Lake-S PCH PCI Express Root Port #2
Group 15:       [8086:7ab0] [R] 00:1d.0  PCI bridge                               Alder Lake-S PCH PCI Express Root Port #9
Group 16:       [8086:7a88]     00:1f.0  ISA bridge                               Device 7a88
                [8086:7ad0]     00:1f.3  Audio device                             Alder Lake-S HD Audio Controller
                [8086:7aa3]     00:1f.4  SMBus                                    Alder Lake-S PCH SMBus Controller
                [8086:7aa4]     00:1f.5  Serial bus controller                    Alder Lake-S PCH SPI Controller
Group 17:       [1b4b:9215] [R] 01:00.0  SATA controller                          88SE9215 PCIe 2.0 x1 4-port SATA 6 Gb/s Controller
Group 18:       [15b7:5030] [R] 02:00.0  Non-Volatile memory controller           Western Digital WD Black SN850X NVMe SSD
Group 19:       [15b7:5030] [R] 03:00.0  Non-Volatile memory controller           Western Digital WD Black SN850X NVMe SSD
Group 20:       [15b7:5030] [R] 05:00.0  Non-Volatile memory controller           Western Digital WD Black SN850X NVMe SSD
Group 21:       [8086:125b] [R] 06:00.0  Ethernet controller                      Ethernet Controller I226-LM
Group 22:       [8086:125b] [R] 07:00.0  Ethernet controller                      Ethernet Controller I226-LM
Group 23:       [1a03:1150] [R] 08:00.0  PCI bridge                               AST1150 PCI-to-PCI Bridge
                [1a03:2000] [R] 09:00.0  VGA compatible controller                ASPEED Graphics Family
                [1a03:2402] [R] 09:01.0  IPMI Interface                           Device 2402

 

[golfsands7]

 @Omid Thank you so much!

[golfsands7]

2 hours ago, Omid said:

Help! I've just hooked up an LSI 9201-8i HBA to this motherboard and Unraid won't boot...

I dont know if it's related but I recall reading a post somewhere of someone saying they had trouble with an HBA in the x16 slot of Supermicro's W680. Again no idea if it's related at all though.

[Daniel15]

On 12/5/2023 at 1:10 AM, golfsands7 said:

and must therefore all be passed to a VM togethe

If you want to pass a whole drive to a VM, it's usually a better idea to have it as an unassigned drive in Unraid, and pass it through as a block device from `/dev/disk/by-id/`. Don't use the legacy devices like /dev/sda as the order can change between reboots or if you add a new drive. 

 

When creating/editing the VM, select VirtIO for vDisk bus and enter the path in vDisk location. Performance will essentially be the same as passing the SATA device through directly, but it means you'll still see SMART data for the drive in Unraid. 

Edited December 9, 2023 by Daniel15

[JimmyGerms]

19 hours ago, Omid said:

Help! I've just hooked up an LSI 9201-8i HBA to this motherboard and Unraid won't boot.

 

I'll hook up my LSI 9207-8i HBA a little later today and see if I get any abnormalities.

 

For now though, did you disable "fast boot" in bios?

Also, try out enabling CSM. I can't remember if this helped on my current server or not. Need to play around and see.

Edited December 9, 2023 by JimmyGerms

[maydaytek]

20 hours ago, Omid said:

Help! I've just hooked up an LSI 9201-8i HBA to this motherboard and Unraid won't boot.

 

I get POST with the Asus splash screen showing, then the IPMI card initialising with the firmware/OData configuration step, as usual, but then it changes to a completely black screen. No cursor, blinker or anything. I left it 10+ mins and confirmed on the router that the server has not connected. The screen shows black through IPMI and with a VGA monitor attached.

 

Have I missed any special steps? It's my first time with an HBA card. I added it to the top PCIe 5.0 x16 slot.

 

Next steps:

1. Check BIOS to downgrade PCIe version, although I thought PCIe was good at backwards-compatibility
   • Done: I forced PCIe Gen2 - no success
2. Remove the HBA card just to see if I can get a boot again
   • Done: Confirmed - successful boot
3. Try a different PCIe slot
   • Done: I tried one of the PCIe 3.0 slots and also tried it with Gen2 forced - no success

 

In the 5.0 slots, Q-LED would mostly stay on 40 for ~10 seconds and briefly switch to 39 for just a couple of seconds, before returning to 40 [repeat]. In the 3.0 slots, Q-LED would instead mostly stay on 38 and briefly switch to 37 or 39 but never 40. I assume the slight variation in Q-LED codes is due to the PCIe 5.0 lanes running to the CPU whilst the PCI 3.0 lanes via PCH (?).

 

Regardless, these codes don't help much and only seem to confirm that there's some issue with the HBA card...unless someone can interpret this better than I can?

 

Anyone have any other ideas?

Could it be the card's firmware? Would an older firmware have such a significant negative impact?

How can I check the current firmware version?

 

EDIT: Based on this thread I have the following next steps.  Interestingly, what resolved the OP's issue in that thread was switching from an ASRock board to an Asus one 🤦‍♂️

 

1. Check boot device order (I stupidly haven't checked this yet)
   • Done: looks good, no different to when card is detached
2. Check and disable "legacy storage boot" in BIOS
   • I can't find this setting anywhere -- any help?
3. Insulate some PCIe pins (sounds scary)
   • Done: no success

 

@golfsands7 I'm unfamiliar but will try to get that information for you when I'm up and running again. See below

Similar issue for me with lsi 9205-8i

 

If I disable "fast boot" it will boot fine from the USB drive but the HBA card won't be detected.

 

My next steps to try are:

- Install discrete GPU to get CSM enabled source: https://www.asus.com/us/support/FAQ/1045467

- Enable Interrupt 19 Capture in BIOS. Source: https://hardforum.com/threads/hba-cards-for-modern-motherboards-and-os.2030137/

 

for now, if I go into Bios and manually pick the USB drive it will boot with Fast Boot enable which also strangely enough picks up my HBA card.