(SOLVED) unRAID not seeing drives connected to HP SAS expander

[NightOps]

LSI 9207-8i using 8087 to breakout SATA cables works fine.  When running it through the expander unRAID reports that the disks are missing.  However, my BIOS can see the drives connected through the PCIe slot that the expander is connected to.  This same expander worked fine on a Windows Server 2022 box I just decommissioned when connected to an LSI 9750-4i card I was running.  I just picked up the 9207-8i card, and it is supposedly running in IT mode.

 

When I look in the BIOS it shows the card as "LSI SAS2 MPT Controller SAS2308" with Version 7.27.04.00

Physical Disk Count: 4

Firmware Type: IT

Firmware Version: 20.0.7.0

 

So it appears that the LSI card is seeing the disks on the expander... what am I missing?

legion-diagnostics-20211215-0604.zip

Edited December 21, 2021 by NightOps

[JorgeB]

Strange issue, the expander is being correctly detected:

 

Dec 15 06:02:42 Legion kernel: scsi 9:0:4:0: Enclosure         HP       HP SAS EXP Card  2.06 PQ: 0 ANSI: 5
Dec 15 06:02:42 Legion kernel: scsi 9:0:4:0: set ignore_delay_remove for handle(0x000e)
Dec 15 06:02:42 Legion kernel: scsi 9:0:4:0: SES: handle(0x000e), sas_addr(0x500143801001d6e5), phy(36), device_name(0x0000000000000000)
Dec 15 06:02:42 Legion kernel: scsi 9:0:4:0: enclosure logical id (0x500143801001d6e5), slot(0)
Dec 15 06:02:42 Legion kernel: scsi 9:0:4:0: qdepth(254), tagged(1), scsi_level(6), cmd_que(1)
Dec 15 06:02:42 Legion kernel: scsi 9:0:4:0: Attached scsi generic sg1 type 13

 

But the 4 disks are failing to initialize, maybe a timeout issue?

 

Dec 15 06:02:42 Legion kernel: mpt2sas_cm0: log_info(0x31110101): originator(PL), code(0x11), sub_code(0x0101)
### [PREVIOUS LINE REPEATED 3 TIMES] ###
Dec 15 06:02:42 Legion kernel: end_device-9:0:0: add: handle(0x000a), sas_addr(0x500143801001d6c4)
Dec 15 06:02:42 Legion kernel: mpt2sas_cm0: log_info(0x31110101): originator(PL), code(0x11), sub_code(0x0101)
### [PREVIOUS LINE REPEATED 3 TIMES] ###
Dec 15 06:02:42 Legion kernel: end_device-9:0:1: add: handle(0x000b), sas_addr(0x500143801001d6c5)
Dec 15 06:02:42 Legion kernel: mpt2sas_cm0: log_info(0x31110101): originator(PL), code(0x11), sub_code(0x0101)
### [PREVIOUS LINE REPEATED 3 TIMES] ###
Dec 15 06:02:42 Legion kernel: end_device-9:0:2: add: handle(0x000c), sas_addr(0x500143801001d6c6)
Dec 15 06:02:42 Legion kernel: mpt2sas_cm0: log_info(0x31110101): originator(PL), code(0x11), sub_code(0x0101)
### [PREVIOUS LINE REPEATED 3 TIMES] ###
Dec 15 06:02:42 Legion kernel: end_device-9:0:3: add: handle(0x000d), sas_addr(0x500143801001d6c7)

 

Do you have a different brand disk/device you could use to test?

[NightOps]

2 minutes ago, JorgeB said:

Strange issue, the expander is being correctly detected:

 

Dec 15 06:02:42 Legion kernel: scsi 9:0:4:0: Enclosure         HP       HP SAS EXP Card  2.06 PQ: 0 ANSI: 5
Dec 15 06:02:42 Legion kernel: scsi 9:0:4:0: set ignore_delay_remove for handle(0x000e)
Dec 15 06:02:42 Legion kernel: scsi 9:0:4:0: SES: handle(0x000e), sas_addr(0x500143801001d6e5), phy(36), device_name(0x0000000000000000)
Dec 15 06:02:42 Legion kernel: scsi 9:0:4:0: enclosure logical id (0x500143801001d6e5), slot(0)
Dec 15 06:02:42 Legion kernel: scsi 9:0:4:0: qdepth(254), tagged(1), scsi_level(6), cmd_que(1)
Dec 15 06:02:42 Legion kernel: scsi 9:0:4:0: Attached scsi generic sg1 type 13

 

But the 4 disks are failing to initialize, maybe a timeout issue?

 

Dec 15 06:02:42 Legion kernel: mpt2sas_cm0: log_info(0x31110101): originator(PL), code(0x11), sub_code(0x0101)
### [PREVIOUS LINE REPEATED 3 TIMES] ###
Dec 15 06:02:42 Legion kernel: end_device-9:0:0: add: handle(0x000a), sas_addr(0x500143801001d6c4)
Dec 15 06:02:42 Legion kernel: mpt2sas_cm0: log_info(0x31110101): originator(PL), code(0x11), sub_code(0x0101)
### [PREVIOUS LINE REPEATED 3 TIMES] ###
Dec 15 06:02:42 Legion kernel: end_device-9:0:1: add: handle(0x000b), sas_addr(0x500143801001d6c5)
Dec 15 06:02:42 Legion kernel: mpt2sas_cm0: log_info(0x31110101): originator(PL), code(0x11), sub_code(0x0101)
### [PREVIOUS LINE REPEATED 3 TIMES] ###
Dec 15 06:02:42 Legion kernel: end_device-9:0:2: add: handle(0x000c), sas_addr(0x500143801001d6c6)
Dec 15 06:02:42 Legion kernel: mpt2sas_cm0: log_info(0x31110101): originator(PL), code(0x11), sub_code(0x0101)
### [PREVIOUS LINE REPEATED 3 TIMES] ###
Dec 15 06:02:42 Legion kernel: end_device-9:0:3: add: handle(0x000d), sas_addr(0x500143801001d6c7)

 

Do you have a different brand disk/device you could use to test?

These are WD Red Plus 4TB drives that work fine when connected to the LSI card.  I have some other seagate 2TB drives that I’ll check.  I’ll also try swapping out the 8087 to 8087 cable that connects the HBA to the expander if you think that’s a possibility???

[JorgeB]

3 minutes ago, NightOps said:

I’ll also try swapping out the 8087 to 8087 cable that connects the HBA to the expander if you think that’s a possibility???

Worth trying, though it seems unlikely to me, suspect more some sort of compatibility issue with the WD disks and the expander/LSI combo.

 

[NightOps]

4 minutes ago, JorgeB said:

Worth trying, though it seems unlikely to me, suspect more some sort of compatibility issue with the WD disks and the expander/LSI combo.

 

Oh now I'm really confused.  I just booted after switching the PCIe slot to run at Gen3 mode, and 2 of the 4 drives are showing up...

[JorgeB]

1 minute ago, NightOps said:

Oh now I'm really confused

Me too.

[NightOps]

4 minutes ago, JorgeB said:

Me too.

Not sure if I still have any molex to SATA power adapters around, but do you think it might be related the WD drives and the issue with 3.3v? (

)

[NightOps]

Upon a restart, the disks now all show as missing

[NightOps]

1 hour ago, JorgeB said:

Me too.

My Norco RPC-4220 has a molex-driven backplane...so I decided to throw it in the case and see if that took care of things.  Unfortunately no drives are being detected...

 

So I switched the WD drives to being ran off of the LSI HBA, and then I have my remaining slots being ran from the expander.  I can hot plug in Seagate 2TB and 10TB drives, and they get recognized immediately within unRAID and show up in the Unassigned Devices immediate.

[JorgeB]

13 minutes ago, NightOps said:

I can hot plug in Seagate 2TB and 10TB drives, and they get recognized immediately within unRAID and show up in the Unassigned Devices immediate.

Yes, this was my suspicion.

2 hours ago, JorgeB said:

some sort of compatibility issue with the WD disks and the expander/LSI combo.

 

[NightOps]

41 minutes ago, JorgeB said:

Yes, this was my suspicion.

 

I'm leaning towards a problem with the expander.  I've tried 2 HP expanders (468405-001 and 468405-002).  Do you know of any expanders that do not have this incompatibility?

[JorgeB]

Intel SAS2 based expanders should work fine, like the RES2SV240, chip is LSI, so less likely to run into compatibility issues.

[NightOps]

On 12/15/2021 at 10:16 AM, JorgeB said:

Intel SAS2 based expanders should work fine, like the RES2SV240, chip is LSI, so less likely to run into compatibility issues.

New expander should be in on Monday.  I've been testing things out and it seems that Seagate drives are working fine, but HP branded Seagate drives seem to periodically require me to (after bootup) pull the drive and re-insert it before they initialize.  I may also try this with the WD Red drives I have once I get some to test with.

[JorgeB]

I have one of those expanders also with an LSI HBA and never had any issues, but there could be with some specific models, also could be firmware related, but IIRC to update it requires a supported HP controller.

[NightOps]

20 hours ago, JorgeB said:

I have one of those expanders also with an LSI HBA and never had any issues, but there could be with some specific models, also could be firmware related, but IIRC to update it requires a supported HP controller.

Yuck... not worth it to buy a supported HP controller, lol.  So here's what I'm finding:

LSI HBA

SAS port1: all 4 WD Red drives always detected correctly

SAS port2: pushed to HP SAS Expander

HP Expander connected to Norco Backplane

-- 1 Seagate and 1 HP branded Seagate drive always show up (always the same ones...)

-- 6 HP branded Seagate drives always show up as missing

Since all drives are in hot swap caddies, if I remove them and plug them back in, then the other 6 get detected!

 

Also, if I stick another WD Red 4TB drive in a caddy connected to the expander and unplug/plug in, it gets detected as well... This seems like something akin to the 3.3v WD issue...but the Norco backplane is molex-powered, so I would have though that would have fixed it.

[NightOps]

So I got the Intel expander in... and at first it was doing the same thing as the HP one... no drives were seen...  And then I went ahead and plugged into the molex power on the Intel expander board, and all the drives came right up on the next boot.  So it appears that possibly newer gen slots don't supply enough power on their own??  Either way, it's working now!  🙂  Thanks @JorgeB!

[JorgeB]

1 minute ago, NightOps said:

So it appears that possibly newer gen slots don't supply enough power on their own?? 

They should, but glad it's working anyway.