October 30, 20241 yr Hi, So i have been reading and ssearching forums for 3 days, i havent found the exact answer which is odd, i usually do. I think my problem may be my LSI SAS2008 isnt in IT mode but now sure. Here is my scenario. 1. had 20 drives with 2 iin parity running on USB enclosures for a long time, with the occasional problem of a drive disappearing. but a reboot of the drives and server would always fix. 2. Read a post about movin to a desk shelf - This post here So i bought 2 DS4243 figured cant hurt to have spare parts. 3. same ebay guy sold me two cables and this card LSI Server Expansion Array JBOD PCIe Dual Port SFF-8088 6Gbs SAS 4. after following all of the directions from the above post everything was great i was one of the lucky ones all of my drives showed up, but then it start throwing thousands of errors on each of the drives, from what i read this meant bad cable or bad card. 5. so i went to amazon and bought this card LSI SAS 2008 which was advertised by the company as the same as SAS 9211-8i also bough another QSFP. SFF-8436 to SFF 8088 for good measure 6 installed the new hardware and boom all the drives showed up the whole world was great. no errors. and then i made one dumb mistake i forgot to included. my pools when creating a new config file. All kinds of inconsistent problems began to occur trying to solve the issue of my pools all of a sudden not mounting and wanting to be formatted. 7. so now i am burnt out looking for help as none of hdd will show except for my pool disks that are all nvme or external SSD. i am attaching my diagnosics hoping someone can help guide me. unraid-diagnostics-20241029-2336.zip
October 30, 20241 yr HBA is failing to initialize: Oct 29 23:03:53 UNRAID kernel: mpt2sas_cm0: diag reset: FAILED Oct 29 23:03:53 UNRAID kernel: mpt2sas_cm0: failure at drivers/scsi/mpt3sas/mpt3sas_scsih.c:12348/_scsih_probe()! Try a different PCIe slot and/or test it in a different PC to make sure it's working.
October 30, 20241 yr Author good morning JoreB, Thank you for the reply, ive read so many of your posts over the year i must take the opportunity to thank you for the countless hours you have saved me learning and fixing issues with my server. So THANK YOU! right before i called it quits last night and after i posted in this thread i decided to swap pci slots, so i put my 10gbps intel nic card into the same slot as my HBA and vice vera HBA into the previous pci slot for my nic card to eliminate the slot as the problem and dial into the card. as expected since both of my previously tested HBA cards were first seeing the drives and now neither seeing any i got the same result with the card in a new slot. before i go and put this hardware in a new machine please take a look at the fresh diags attached i saw a reset occur with success but still no luck seeing the drives in unraid. Oct 30 07:17:37 UNRAID kernel: mpt3sas version 43.100.00.00 loaded Oct 30 07:17:37 UNRAID kernel: mpt3sas 0000:07:00.0: can't disable ASPM; OS doesn't have ASPM control Oct 30 07:17:37 UNRAID kernel: mpt2sas_cm0: 64 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (32651152 kB) Oct 30 07:17:37 UNRAID kernel: mpt2sas_cm0 fault info from func: _base_wait_for_iocstate Oct 30 07:17:37 UNRAID kernel: mpt2sas_cm0: fault_state(0x2622)! Oct 30 07:17:37 UNRAID kernel: mpt2sas_cm0: sending diag reset !! Then i saw this Oct 30 07:17:37 UNRAID kernel: mpt2sas_cm0: diag reset: SUCCESS Oct 30 07:17:37 UNRAID kernel: ixgbe 0000:09:00.1: 16.000 Gb/s available PCIe bandwidth, limited by 5.0 GT/s PCIe x4 link at 0000:00:1c.4 (capable of 32.000 Gb/s with 5.0 GT/s PCIe x8 link) Oct 30 07:17:37 UNRAID kernel: mpt2sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k Oct 30 07:17:37 UNRAID kernel: mpt2sas_cm0: MSI-X vectors supported: 1 Oct 30 07:17:37 UNRAID kernel: no of cores: 32, max_msix_vectors: -1 Oct 30 07:17:37 UNRAID kernel: mpt2sas_cm0: 0 1 1 Oct 30 07:17:37 UNRAID kernel: mpt2sas_cm0: High IOPs queues : disabled Oct 30 07:17:37 UNRAID kernel: mpt2sas0-msix0: PCI-MSI-X enabled: IRQ 251 Oct 30 07:17:37 UNRAID kernel: mpt2sas_cm0: iomem(0x0000000080dc0000), mapped(0x00000000d309768f), size(16384) Oct 30 07:17:37 UNRAID kernel: mpt2sas_cm0: ioport(0x0000000000003000), size(256) Oct 30 07:17:37 UNRAID kernel: mpt2sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k Oct 30 07:17:37 UNRAID kernel: ixgbe 0000:09:00.1: MAC: 3, PHY: 0, PBA No: G45270-003 Oct 30 07:17:37 UNRAID kernel: ixgbe 0000:09:00.1: a0:36:9f:23:a5:e2 Oct 30 07:17:37 UNRAID kernel: mpt2sas_cm0: scatter gather: sge_in_main_msg(1), sge_per_chain(9), sge_per_io(128), chains_per_io(15) Oct 30 07:17:37 UNRAID kernel: mpt2sas_cm0: request pool(0x00000000eb00c68e) - dma(0x171280000): depth(3492), frame_size(128), pool_size(436 kB) Oct 30 07:17:37 UNRAID kernel: mpt2sas_cm0: sense pool(0x00000000ef92b2c2) - dma(0x178400000): depth(3367), element_size(96), pool_size (315 kB) Oct 30 07:17:37 UNRAID kernel: mpt2sas_cm0: reply pool(0x000000005924be29) - dma(0x188100000): depth(3556), frame_size(128), pool_size(444 kB) Oct 30 07:17:37 UNRAID kernel: mpt2sas_cm0: config page(0x00000000249a0f35) - dma(0x14b990000): size(512) Oct 30 07:17:37 UNRAID kernel: mpt2sas_cm0: Allocated physical memory: size(7579 kB) Oct 30 07:17:37 UNRAID kernel: mpt2sas_cm0: Current Controller Queue Depth(3364),Max Controller Queue Depth(3432) Oct 30 07:17:37 UNRAID kernel: mpt2sas_cm0: Scatter Gather Elements per IO(128) Oct 30 07:17:37 UNRAID kernel: mpt2sas_cm0: overriding NVDATA EEDPTagMode setting Oct 30 07:17:37 UNRAID kernel: mpt2sas_cm0: LSISAS2008: FWVersion(20.00.06.00), ChipRevision(0x03), BiosVersion(07.27.01.01) Oct 30 07:17:37 UNRAID kernel: mpt2sas_cm0: Protocol=(Initiator,Target), Capabilities=(TLR,EEDP,Snapshot Buffer,Diag Trace Buffer,Task Set Full,NCQ) Oct 30 07:17:37 UNRAID kernel: scsi host11: Fusion MPT SAS Host Oct 30 07:17:37 UNRAID kernel: mpt2sas_cm0: sending port enable !! and then the last bit waas from what i can tell it recognizing the eclosue Oct 30 07:17:37 UNRAID kernel: mpt2sas_cm0: hba_port entry: 0000000044a548cd, port: 255 is added to hba_port list Oct 30 07:17:37 UNRAID kernel: mpt2sas_cm0: host_add: handle(0x0001), sas_addr(0x56c92bf0000390f0), phys(8) Oct 30 07:17:37 UNRAID kernel: mpt2sas_cm0: expander_add: handle(0x0009), parent(0x0001), sas_addr(0x500a0980056c977f), phys(37) Oct 30 07:17:37 UNRAID kernel: expander-11:0: add: handle(0x0009), sas_addr(0x500a0980056c977f) Oct 30 07:17:37 UNRAID kernel: mpt2sas_cm0: handle(0xa) sas_address(0x500a0980056c977e) port_type(0x1) Oct 30 07:17:37 UNRAID kernel: scsi 11:0:0:0: Enclosure NETAPP DS424IOM6 0191 PQ: 0 ANSI: 5 Oct 30 07:17:37 UNRAID kernel: scsi 11:0:0:0: set ignore_delay_remove for handle(0x000a) Oct 30 07:17:37 UNRAID kernel: scsi 11:0:0:0: SES: handle(0x000a), sas_addr(0x500a0980056c977e), phy(36), device_name(0x0000000000000000) Oct 30 07:17:37 UNRAID kernel: scsi 11:0:0:0: enclosure logical id (0x500a0980055230c6), slot(0) Oct 30 07:17:37 UNRAID kernel: scsi 11:0:0:0: qdepth(254), tagged(1), scsi_level(6), cmd_que(1) Oct 30 07:17:37 UNRAID kernel: scsi 11:0:0:0: Power-on or device reset occurred Oct 30 07:17:37 UNRAID kernel: mpt2sas_cm0: log_info(0x31120434): originator(PL), code(0x12), sub_code(0x0434) Oct 30 07:17:37 UNRAID kernel: mpt2sas_cm0: log_info(0x31120434): originator(PL), code(0x12), sub_code(0x0434) Oct 30 07:17:37 UNRAID kernel: mpt2sas_cm0: log_info(0x31120434): originator(PL), code(0x12), sub_code(0x0434) Oct 30 07:17:37 UNRAID kernel: mpt2sas_cm0: log_info(0x31120434): originator(PL), code(0x12), sub_code(0x0434) Oct 30 07:17:37 UNRAID kernel: scsi 11:0:0:0: Attached scsi generic sg5 type 13 Oct 30 07:17:37 UNRAID kernel: end_device-11:0:0: add: handle(0x000a), sas_addr(0x500a0980056c977e) Oct 30 07:17:37 UNRAID kernel: mpt2sas_cm0: port enable: SUCCESS Thanks again in advance for your help. unraid-diagnostics-20241030-0727.zip
October 30, 20241 yr It appears to be working now, and it detected the enclosure: [11:0:0:0] enclosu NETAPP DS424IOM6 0191 - /dev/sg5 state=running queue_depth=254 scsi_level=6 type=13 device_blocked=0 timeout=30 dir: /sys/bus/scsi/devices/11:0:0:0 [/sys/devices/pci0000:00/0000:00:1c.1/0000:07:00.0/host11/port-11:0/expander-11:0/port-11:0:0/end_device-11:0:0/target11:0:0/11:0:0:0]
October 30, 20241 yr Author correct but i am still not seeing any drives in unraid main tab or in system devices. this is all i see SCSI Devices [0:0:0:0] disk WD Game Drive 2003 /dev/sda 4.00TB [1:0:0:0] disk WD Game Drive 2003 /dev/sdb 4.00TB [2:0:0:0] disk VendorCo ProductCode 2.00 /dev/sdc 31.4GB [N:0:0:1] disk MSI M450 2TB__1 /dev/nvme0n1 2.00TB [N:1:1:1] disk CT4000P3PSSD8__1 /dev/nvme1n1 4.00TB [N:2:1:1] disk CT4000P3PSSD8__1 /dev/nvme2n1 4.00TB [N:3:1:1] disk CT4000P3PSSD8__1 /dev/nvme3n1 4.00TB Edited October 30, 20241 yr by RemmyCH3CK
October 30, 20241 yr That now suggests an enclosure issue, or config. Do you hear the disks spinning up? I'm not familiar with those enclosures, but if you are using interposers, try without them.
October 30, 20241 yr Author So i hear them spin up when i first power on the enclosure, but its not what i am used to hearing from my old usb enclosure where you could distinctly hear the drives spin up first and once that settled you would start the server. and then there is no activity through the unraid boot process like i am used to hearing and no activity lights i am not using any interposers outside of the enclosure, dont think there are any internally. just control card (IOM3 module) then QSFP. SFF-8436 to SFF 8088 cable to HBA card Edited October 30, 20241 yr by RemmyCH3CK
October 30, 20241 yr Author just a follow up on the interposers, check the trays again to confirm. the back of the tray is open, so id imagine that if there was going to be an interposer it would be attached there.
October 30, 20241 yr Author good news and bad news. So i figured out my vanishing drives problem. last night with tired eyes i plugged my cable into the bottom IOM6 instead of the top IOM6. spoke to the person i bought them from he helped me realize this. great ebay seller. very helpful person. he also is sending me interposers to maybe help with all of the udma crc errors i am getting for free. but this takes me back to the beginning of this journey but there is hope. is it ok if i ask for help with something related to this process on this thread? in one of my many reconfigs yesterday. I made the mistake of creating a new config without having the pools checked. now my first pool cache came back correctly and is fine, but my other two pools look like this: Those drive are supposed to be part of zfs pools. I need to retain the data and get them back into the correct pools. I am 90% sure this is how i had them configured but the pic i took cut them out by mistake. any guidance on what can be done here Edited October 30, 20241 yr by RemmyCH3CK
October 30, 20241 yr Post the diagnostics and the output from zpool import 6 minutes ago, RemmyCH3CK said: he also is sending me interposers I believe they work best without them, or you will lose SMART.
October 30, 20241 yr Author diags attached. Here is the output from zpool import: pool: utility-cache id: 182752850668142062 state: ONLINE action: The pool can be imported using its name or numeric identifier. config: utility-cache ONLINE sdb1 ONLINE pool: utility-cache id: 9217438907639438045 state: ONLINE action: The pool can be imported using its name or numeric identifier. config: utility-cache ONLINE raidz1-0 ONLINE nvme3n1p1 ONLINE sda1 ONLINE nvme0n1p1 ONLINE unraid-diagnostics-20241030-1135.zip
October 30, 20241 yr Both pools are showing the same name, type: zpool import 9217438907639438045 plex Then zpool export plex Then restart the array and post new diags if they still don't mount.
October 30, 20241 yr Author all good now. thank you for helping me navigate this! Edited October 30, 20241 yr by RemmyCH3CK
October 30, 20241 yr Interposers are only there to allow multi-pathing for sata drives, which you aren't using. In some scenarios they can also be useful to avoid compatibility issues with mixed sas/sata deployments. Unless you know you need them, you don't need them.
November 3, 20241 yr Author Hi all, so the saga continues a bit. Drives have been stable and not throwing real time errors at me, however i just ran my first parity check and i ran it with corrections. It took roughly two days. 66 billions errors on parity check and read/write on my array, but it reports that the parity was built successfully. From what i read on here this can happen, and the next step would be to run the parity check again without corrections without rebooting. Which i am doing right now, but it immediately showed tons of errors in the parity check as soon as it started. i have attached my diagnostics for you to look at and maybe provide some insight. thank you in advance. unraid-diagnostics-20241103-1007.zip Edited November 3, 20241 yr by RemmyCH3CK
November 3, 20241 yr There are disk read errors, that is different from parity sync errors, errors are affecting multiple devices, so most likely a power/connection issue, or the controller/enclosure
November 3, 20241 yr Author ok i just got a new controller in the mail. its a 4 port PM8003 REV: 5.0 i can try installing that card and see if it helps.
November 3, 20241 yr Author so i switched two different HAB controller cards, two different cables. i made sure the drives are all securely mounted in their slots. i even swapped the iom6 controllers on the disk shelf. still getting all off those errors. any other ideas before i return these disk shelfs and go back to usb shelfs lol? unraid-diagnostics-20241103-1209.zip
November 3, 20241 yr I'm not familiar with those enclosures, but other users have them with Unraid, so they are generally fine, could be an issue with that one, or the controller.
November 3, 20241 yr Author well i think i got it worked out. yesterday i received the interposers. just installed them, all the drives showed up as healthy. under SMART and i received notices saying that all of the drives returned to normal operation. i am also see temp read outs which leads me to believe SMART is working with the interposers installed. interestingly enough i went to chat gpt and asked in i neededd to use interposers with my HBA/enclosure config and this was the response: Yes, for SATA drives in a NetApp DS4243, interposers are typically required to allow the SATA drives to communicate over the SAS backplane. Here’s why: Protocol Conversion: The DS4243 uses a SAS backplane, which expects SAS communication protocols. Interposers convert the SATA protocol of the drives to SAS, allowing them to be recognized over SAS interfaces. Connection Reliability: Without interposers, there may be issues with connection stability and drive detection since SATA drives might not communicate reliably with the SAS backplane. HBA Compatibility: Some HBAs and systems may not detect SATA drives at all on SAS backplanes without interposers, as they rely on the SAS protocol for both connectivity and management. If you don't have interposers, you might be unable to see the SATA drives. Adding interposers should allow the drives to appear on your Unraid server, provided the other connections and configurations are correct.
November 4, 20241 yr 13 hours ago, RemmyCH3CK said: just installed them, all the drives showed up as healthy. under SMART IIRC they don't pass the correct SMART attributes, but if reliability is better it may still be worth it, post new diags just to check SMART.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.