9600-24i for Unraid

[LeetDonkey]

Hello,

I am considering a 9600-24i for Unraid.
I am going to use it purely for mechanical SATA drives.

Are there any quirks or issues I should be aware of?
Also, is anyone able to confirm that it supports ASPM and will allow the CPU PKG to go beyond C3 state?

[JorgeB]

Should be plug and play, don't know about ASPM.

[Wody]

On 3/31/2024 at 10:23 PM, LeetDonkey said:

Are there any quirks or issues I should be aware of?

I don't know about C-states since I don't really pay attention to that, but its important to note that the 9600 drops support for anything below 6Gb/s, meaning if you have any SATA 1 or 2, or SAS 1 drives, they won't work at all, and simply won't be detected.

Also, the 9500 and 9600 no longer have a legacy BIOS, only UEFI, so even though it isn't important for HBA's, there is no information on screen during booting like with older cards, and no key to press to enter their BIOS. Instead, they only show up in the mainboard's BIOS Setup screen (I'm not sure what it is called for UEFI). Sometimes they don't show up there at all though, the 9600 isn't visible for me. But they work fine with Unraid and drive-detection is fine too.

 

[LeetDonkey]

On 4/2/2024 at 3:05 PM, Wody said:

I don't know about C-states since I don't really pay attention to that, but its important to note that the 9600 drops support for anything below 6Gb/s, meaning if you have any SATA 1 or 2, or SAS 1 drives, they won't work at all, and simply won't be detected.

Also, the 9500 and 9600 no longer have a legacy BIOS, only UEFI, so even though it isn't important for HBA's, there is no information on screen during booting like with older cards, and no key to press to enter their BIOS. Instead, they only show up in the mainboard's BIOS Setup screen (I'm not sure what it is called for UEFI). Sometimes they don't show up there at all though, the 9600 isn't visible for me. But they work fine with Unraid and drive-detection is fine too.

 

Thank you for the clarification on SATA 1, 2 and SAS 1, I did not know that.
I don't think it will be an issue since all my drivers are SATA-3.

My main concern now is ASPM support, so I hope someone can confirm/deny that.
 

[Wody]

20 hours ago, LeetDonkey said:

My main concern now is ASPM support, so I hope someone can confirm/deny that.

I had a look at lspci, and my 9600-24i reports ASPM level 1 only with a latency of both L0s and L1 as unlimited, my 9500-16i reports both L0s and L1, with latency for L0s unlimited and for L1 < 64us. Since they are different, i figured I'd mentioned both. Also each card is in a different server, both running Unraid though, and the latest firmware at the moment (Phase 30 for the 9500, Phase 8.8 for the 9600)

Edited April 5 by Wody

[LeetDonkey]

3 hours ago, Wody said:

I had a look at lspci, and my 9600-24i reports ASPM level 1 only with a latency of both L0s and L1 as unlimited, my 9500-16i reports both L0s and L1, with latency for L0s unlimited and for L1 < 64us. Since they are different, i figured I'd mentioned both. Also each card is in a different server, both running Unraid though, and the latest firmware at the moment (Phase 30 for the 9500, Phase 8.8 for the 9600)

Thank you for investigating, this certainly helps me pick the correct hardware.

For ASPM support I was considering a carrier board with a PCIE packet switch and 4 x ASM1166 controllers, but I think the sheer amount of connectors and components make the setup a bit janky and error prone.
With your feedback I will definately put the 9600-24i into consideration.

I am currently awaiting the new hardware and will conduct some experiments on how big an impact ASPM support will make on the total power consumption, and then make a decision.

[firstTimer]

On 4/5/2024 at 9:43 AM, Wody said:

I had a look at lspci, and my 9600-24i reports ASPM level 1 only with a latency of both L0s and L1 as unlimited, my 9500-16i reports both L0s and L1, with latency for L0s unlimited and for L1 < 64us. Since they are different, i figured I'd mentioned both. Also each card is in a different server, both running Unraid though, and the latest firmware at the moment (Phase 30 for the 9500, Phase 8.8 for the 9600)

Could you please check with powertop if your cpu drops to lower c-states? I mean if the pckg state goes below C2 or C3

[Wody]

On 4/13/2024 at 12:45 AM, firstTimer said:

Could you please check with powertop if your cpu drops to lower c-states? I mean if the pckg state goes below C2 or C3

I'm not sure if I'm reading things correctly, but if I look at the Idle stats in Powertop, it lists the Pkg as spending a lot of time in C6 and none in C3, and the Core also a lot of time in C6 and none in C3 on the server with the 9600. For the 9500, that server is a lot more busy, so it only reaches C3 in Pkg and none in C6, and the cores run some time both in C3 and C6.

[firstTimer]

3 hours ago, Wody said:

I'm not sure if I'm reading things correctly, but if I look at the Idle stats in Powertop, it lists the Pkg as spending a lot of time in C6 and none in C3, and the Core also a lot of time in C6 and none in C3 on the server with the 9600. For the 9500, that server is a lot more busy, so it only reaches C3 in Pkg and none in C6, and the cores run some time both in C3 and C6.

Oh nice, so at least the server with the 9600 is reaching a good result I would say (best would be C8 to C10) but with some many disks and so many variables that can influence this... it means you have a good result. Regarding the 9500, as you said, your server is quite busy so it is difficult to say if the lower C-states are not reached because of the high demand or because any component of the server could influence this result. I am kinda undecided  whether to buy the 9500 or the 9600... both cost a good amount of money and calculating if those boards are more convenient in the long run because they could eventually lead the whole server to lower C-states is a hard problem to think about... My server doesn't go lower than C3 and it consumes 60W-70W at idle while similar builds consume 40ish watts with C8.

[fenglerf1]

Does this card need to be flashed into IT mode like the 9400 or prior cards? I have tried searching but have been unable to find anything about this.

 

I'm interested in the 9500/9600 due to the ability to use NVME drives and SATA at the same time and only taking up one PCIe slot.

[LeetDonkey]

I have the same experience as Wody,
I rarely if ever see any of the deeper C states on the CPU pkg, simply because alot of stuff is going on with dockers etc.
If I stop the array it will go to C7 for like 40% of the time.

I have a 9305-24i which does not support ASPM, but I have placed it in a PCIE slot connected to the PCH instead of the CPU.
This will allow the CPU PKG to reach C7, at least on my W480 platform.
I have a X710-DA2 in a CPU PCIE slot and this limits the PKG state to C7, otherwise I would see C8.

[Wody]

17 hours ago, fenglerf1 said:

Does this card need to be flashed into IT mode like the 9400 or prior cards?

The 9400 and newer cards can't be used for RAID so they don't have IT mode. The 9400s (for the models that support it) have two firmwares, one including PCIe support, and one without it which has bigger buffers/queues.The 9500 and 9600 also drop legacy bios support, and don't display anything on the screen anymore at startup.

 

If you only want one slot, you probably want the 9600, since you can get that in a 24i version with 3 connectors, the 9500 only has the 16i with 2 connectors. The 9600 is SAS 4, meaning it has no support for older drives below 6gbit though. That's of course not an issue with PCIe though

 

Cables may be an issue, since Broadcom doesn't have a lot of different options, but the 9500 and 9600 use the same ones so that's good at least.