March 14, 20251 yr I have currently cabled my back-plane to single link as i found i received link resets and read/write errors (UDMA CRC errors) if i use dual link. I unfortunately do not have another SAS controller to eliminate it as the cause of the problems, but I have recabled. Currently I am saturating the single channel link when I do parity actions and it tops out at 2.2G. SAS Expander: Super Mei TOP24 Expander 05G451625012 REV CO BLACK 12Gb (there is a sticker saying FW 0a00 2521 V1.00) SAS Controller: Broadcom / LSI SAS3224 PCI-Express Fusion-MPT SAS-3 (rev 01) (during boot it says LSISAS3224: FWVersion(16.00.01.00), ChipRevision(0x01), BiosVersion(18.00.00.00) ) [1000:00c4] 02:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS3224 PCI-Express Fusion-MPT SAS-3 (rev 01) [10:0:20:0] disk ATA WDC WD200EDGZ-11 0A85 /dev/sdaa 20.0TB [10:0:21:0] disk ATA WDC WD200EDGZ-11 0A85 /dev/sdab 20.0TB [10:0:22:0] disk ATA WDC WD200EDGZ-11 0A85 /dev/sdac 20.0TB [10:0:23:0] disk ATA WDC WD200EDGZ-11 0A85 /dev/sdad 20.0TB [10:0:0:0] disk ATA WDC WD140EMFZ-11 0A81 /dev/sdc 14.0TB [10:0:1:0] disk ATA WDC WD140EMFZ-11 0A81 /dev/sdd 14.0TB [10:0:2:0] disk ATA WDC WD140EMFZ-11 0A81 /dev/sde 14.0TB [10:0:3:0] disk ATA ST12000VN0007-2G SC60 /dev/sdf 12.0TB [10:0:4:0] disk ATA WDC WD180EDGZ-11 0A85 /dev/sdg 18.0TB [10:0:5:0] disk ATA WDC WD180EDGZ-11 0A85 /dev/sdi 18.0TB [10:0:6:0] disk ATA WDC WD180EDGZ-11 0A85 /dev/sdj 18.0TB [10:0:7:0] disk ATA WDC WD140EMFZ-11 0A81 /dev/sdn 14.0TB [10:0:8:0] disk ATA WDC WD180EDGZ-11 0A85 /dev/sdo 18.0TB [10:0:9:0] disk ATA WDC WD180EDGZ-11 0A85 /dev/sdp 18.0TB [10:0:10:0] disk ATA WDC WD180EDGZ-11 0A85 /dev/sdq 18.0TB [10:0:11:0] disk ATA WDC WD180EDGZ-11 0A85 /dev/sdr 18.0TB [10:0:12:0] disk ATA WDC WD200EDGZ-11 0A85 /dev/sds 20.0TB [10:0:13:0] disk ATA WDC WD200EDGZ-11 0A85 /dev/sdt 20.0TB [10:0:14:0] disk ATA WDC WD200EDGZ-11 0A85 /dev/sdu 20.0TB [10:0:15:0] disk ATA WDC WD200EDGZ-11 0A85 /dev/sdv 20.0TB [10:0:16:0] disk ATA WDC WD200EDGZ-11 0A85 /dev/sdw 20.0TB [10:0:17:0] disk ATA WDC WD200EDGZ-11 0A85 /dev/sdx 20.0TB [10:0:18:0] disk ATA WDC WD200EDGZ-11 0A85 /dev/sdy 20.0TB [10:0:19:0] disk ATA WDC WD200EDGZ-11 0A85 /dev/sdz 20.0TB All of the diagnostics are from the single link setup, but perhaps there are some hints as to what is the problem. This link looks to be established fine, and you can see the firmware and versions of the controller. [ 36.314490] mpt3sas_cm0: iomem(0x0000000082500000), mapped(0x00000000430be8e9), size(65536) [ 36.316459] mpt3sas_cm0: ioport(0x0000000000004000), size(256) [ 36.341341] mpt3sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k [ 36.342958] mpt3sas_cm0: sending message unit reset !! [ 36.345780] mpt3sas_cm0: message unit reset: SUCCESS [ 36.348106] mpt3sas_cm0: scatter gather: sge_in_main_msg(9), sge_per_chain(15), sge_per_io(128), chains_per_io(8) [ 36.355313] mpt3sas_cm0: request pool(0x00000000d091fde2) - dma(0x139200000): depth(5852), frame_size(256), pool_size(1463 kB) [ 36.371186] mpt3sas_cm0: sense pool(0x00000000ebd46e7d) - dma(0x13a700000): depth(5791), element_size(96), pool_size (542 kB) [ 36.373700] mpt3sas_cm0: reply pool(0x000000004f53c277) - dma(0x13a800000): depth(5916), frame_size(128), pool_size(739 kB) [ 36.376178] mpt3sas_cm0: config page(0x00000000035a8032) - dma(0x13234d000): size(512) [ 36.378734] mpt3sas_cm0: Allocated physical memory: size(17295 kB) [ 36.381342] mpt3sas_cm0: Current Controller Queue Depth(5788),Max Controller Queue Depth(5824) [ 36.384083] mpt3sas_cm0: Scatter Gather Elements per IO(128) [ 36.525341] mpt3sas_cm0: _base_display_fwpkg_version: complete [ 36.525557] mpt3sas_cm0: overriding NVDATA EEDPTagMode setting [ 36.526162] mpt3sas_cm0: LSISAS3224: FWVersion(16.00.01.00), ChipRevision(0x01), BiosVersion(18.00.00.00) [ 36.526167] mpt3sas_cm0: Protocol=(Initiator,Target), Capabilities=(TLR,EEDP,Snapshot Buffer,Diag Trace Buffer,Task Set Full,NCQ) [ 36.530365] mpt3sas_cm0: sending port enable !! [ 36.534865] mpt3sas_cm0: hba_port entry: 0000000081440fd9, port: 255 is added to hba_port list [ 36.546212] mpt3sas_cm0: host_add: handle(0x0001), sas_addr(0x500062b202991611), phys(24) [ 36.553485] mpt3sas_cm0: expander_add: handle(0x0019), parent(0x0011), sas_addr(0xcc1be0310015463f), phys(37) [ 36.559900] expander-10:0: add: handle(0x0019), sas_addr(0xcc1be0310015463f) [ 36.559922] mpt3sas_cm0: port enable: SUCCESS [ 36.577354] mpt3sas_cm0: handle(0x1a) sas_address(0xcc1be03100154600) port_type(0x1) [ 36.590904] scsi 10:0:0:0: SATA: handle(0x001a), sas_addr(0xcc1be03100154600), phy(0), device_name(0x5000cca264d70f29) [ 36.644352] end_device-10:0:0: add: handle(0x001a), sas_addr(0xcc1be03100154600) [ 36.644814] mpt3sas_cm0: handle(0x1b) sas_address(0xcc1be03100154601) port_type(0x1) [ 36.651453] scsi 10:0:1:0: SATA: handle(0x001b), sas_addr(0xcc1be03100154601), phy(1), device_name(0x5000cca264df1ce0) [ 36.655977] end_device-10:0:1: add: handle(0x001b), sas_addr(0xcc1be03100154601) [ 36.656265] mpt3sas_cm0: handle(0x1c) sas_address(0xcc1be03100154602) port_type(0x1) [ 36.657287] scsi 10:0:2:0: SATA: handle(0x001c), sas_addr(0xcc1be03100154602), phy(2), device_name(0x5000cca28dc54e4b) [ 36.669233] end_device-10:0:2: add: handle(0x001c), sas_addr(0xcc1be03100154602) [ 36.678566] mpt3sas_cm0: handle(0x24) sas_address(0xcc1be03100154603) port_type(0x1) [ 36.702145] scsi 10:0:3:0: SATA: handle(0x0024), sas_addr(0xcc1be03100154603), phy(3), device_name(0x5000c500a392550d) [ 36.801469] end_device-10:0:3: add: handle(0x0024), sas_addr(0xcc1be03100154603) [ 36.811823] mpt3sas_cm0: handle(0x1d) sas_address(0xcc1be03100154604) port_type(0x1) [ 36.845256] scsi 10:0:4:0: SATA: handle(0x001d), sas_addr(0xcc1be03100154604), phy(4), device_name(0x5000cca2afc6dedc) [ 36.892815] end_device-10:0:4: add: handle(0x001d), sas_addr(0xcc1be03100154604) [ 36.902157] mpt3sas_cm0: handle(0x1e) sas_address(0xcc1be03100154605) port_type(0x1) [ 36.919262] scsi 10:0:5:0: SATA: handle(0x001e), sas_addr(0xcc1be03100154605), phy(5), device_name(0x5000cca284d5623f) [ 36.978479] end_device-10:0:5: add: handle(0x001e), sas_addr(0xcc1be03100154605) [ 37.003230] mpt3sas_cm0: handle(0x1f) sas_address(0xcc1be03100154606) port_type(0x1) [ 37.098613] scsi 10:0:6:0: SATA: handle(0x001f), sas_addr(0xcc1be03100154606), phy(6), device_name(0x5000cca2afc3c57c) [ 37.287258] end_device-10:0:6: add: handle(0x001f), sas_addr(0xcc1be03100154606) [ 37.295286] mpt3sas_cm0: handle(0x20) sas_address(0xcc1be03100154607) port_type(0x1) [ 37.305559] scsi 10:0:7:0: SATA: handle(0x0020), sas_addr(0xcc1be03100154607), phy(7), device_name(0x5000cca264dfc5ee) [ 37.321216] end_device-10:0:7: add: handle(0x0020), sas_addr(0xcc1be03100154607) [ 37.333591] mpt3sas_cm0: handle(0x22) sas_address(0xcc1be03100154608) port_type(0x1) [ 37.350951] scsi 10:0:8:0: SATA: handle(0x0022), sas_addr(0xcc1be03100154608), phy(8), device_name(0x5000cca284ef2699) [ 37.515223] end_device-10:0:8: add: handle(0x0022), sas_addr(0xcc1be03100154608) [ 37.520455] mpt3sas_cm0: handle(0x23) sas_address(0xcc1be03100154609) port_type(0x1) [ 37.544456] scsi 10:0:9:0: SATA: handle(0x0023), sas_addr(0xcc1be03100154609), phy(9), device_name(0x5000cca2afc510fb) [ 37.582514] end_device-10:0:9: add: handle(0x0023), sas_addr(0xcc1be03100154609) [ 37.591047] mpt3sas_cm0: handle(0x25) sas_address(0xcc1be0310015460a) port_type(0x1) [ 37.607455] scsi 10:0:10:0: SATA: handle(0x0025), sas_addr(0xcc1be0310015460a), phy(10), device_name(0x5000cca2afc41fea) [ 37.641779] end_device-10:0:10: add: handle(0x0025), sas_addr(0xcc1be0310015460a) [ 37.650665] mpt3sas_cm0: handle(0x26) sas_address(0xcc1be0310015460b) port_type(0x1) [ 37.667995] scsi 10:0:11:0: SATA: handle(0x0026), sas_addr(0xcc1be0310015460b), phy(11), device_name(0x5000cca284d4e2cf) [ 37.716350] end_device-10:0:11: add: handle(0x0026), sas_addr(0xcc1be0310015460b) [ 37.725195] mpt3sas_cm0: handle(0x27) sas_address(0xcc1be0310015460c) port_type(0x1) [ 37.747290] scsi 10:0:12:0: SATA: handle(0x0027), sas_addr(0xcc1be0310015460c), phy(12), device_name(0x5000cca2bec4ff5b) [ 37.849489] end_device-10:0:12: add: handle(0x0027), sas_addr(0xcc1be0310015460c) [ 37.849629] mpt3sas_cm0: handle(0x28) sas_address(0xcc1be0310015460d) port_type(0x1) [ 37.857081] scsi 10:0:13:0: SATA: handle(0x0028), sas_addr(0xcc1be0310015460d), phy(13), device_name(0x5000cca407e2e65a) [ 37.935803] end_device-10:0:13: add: handle(0x0028), sas_addr(0xcc1be0310015460d) [ 37.936005] mpt3sas_cm0: handle(0x29) sas_address(0xcc1be0310015460e) port_type(0x1) [ 37.943492] scsi 10:0:14:0: SATA: handle(0x0029), sas_addr(0xcc1be0310015460e), phy(14), device_name(0x5000cca2c0c512a2) [ 38.022516] end_device-10:0:14: add: handle(0x0029), sas_addr(0xcc1be0310015460e) [ 38.022733] mpt3sas_cm0: handle(0x2a) sas_address(0xcc1be0310015460f) port_type(0x1) [ 38.031128] scsi 10:0:15:0: SATA: handle(0x002a), sas_addr(0xcc1be0310015460f), phy(15), device_name(0x5000cca2bfc040d3) [ 38.115380] end_device-10:0:15: add: handle(0x002a), sas_addr(0xcc1be0310015460f) [ 38.116866] mpt3sas_cm0: handle(0x2b) sas_address(0xcc1be03100154610) port_type(0x1) [ 38.126201] scsi 10:0:16:0: SATA: handle(0x002b), sas_addr(0xcc1be03100154610), phy(16), device_name(0x5000cca2b3c380e6) [ 38.205629] end_device-10:0:16: add: handle(0x002b), sas_addr(0xcc1be03100154610) [ 38.206807] mpt3sas_cm0: handle(0x2c) sas_address(0xcc1be03100154611) port_type(0x1) [ 38.215695] scsi 10:0:17:0: SATA: handle(0x002c), sas_addr(0xcc1be03100154611), phy(17), device_name(0x5000cca2b3c34289) [ 38.293204] end_device-10:0:17: add: handle(0x002c), sas_addr(0xcc1be03100154611) [ 38.294389] mpt3sas_cm0: handle(0x2d) sas_address(0xcc1be03100154612) port_type(0x1) [ 38.304437] scsi 10:0:18:0: SATA: handle(0x002d), sas_addr(0xcc1be03100154612), phy(18), device_name(0x5000cca2becd9d1c) [ 38.389970] end_device-10:0:18: add: handle(0x002d), sas_addr(0xcc1be03100154612) [ 38.391154] mpt3sas_cm0: handle(0x2e) sas_address(0xcc1be03100154613) port_type(0x1) [ 38.401483] scsi 10:0:19:0: SATA: handle(0x002e), sas_addr(0xcc1be03100154613), phy(19), device_name(0x5000cca2bec8c3a3) [ 38.491219] end_device-10:0:19: add: handle(0x002e), sas_addr(0xcc1be03100154613) [ 38.492407] mpt3sas_cm0: handle(0x2f) sas_address(0xcc1be03100154614) port_type(0x1) [ 38.501773] scsi 10:0:20:0: SATA: handle(0x002f), sas_addr(0xcc1be03100154614), phy(20), device_name(0x5000cca2b3c4afc3) [ 38.583468] end_device-10:0:20: add: handle(0x002f), sas_addr(0xcc1be03100154614) [ 38.585420] mpt3sas_cm0: handle(0x30) sas_address(0xcc1be03100154615) port_type(0x1) [ 38.596504] scsi 10:0:21:0: SATA: handle(0x0030), sas_addr(0xcc1be03100154615), phy(21), device_name(0x5000cca2b3c35cc0) [ 38.679991] end_device-10:0:21: add: handle(0x0030), sas_addr(0xcc1be03100154615) [ 38.682711] mpt3sas_cm0: handle(0x31) sas_address(0xcc1be03100154616) port_type(0x1) [ 38.692968] scsi 10:0:22:0: SATA: handle(0x0031), sas_addr(0xcc1be03100154616), phy(22), device_name(0x5000cca2b3c35804) [ 38.773938] end_device-10:0:22: add: handle(0x0031), sas_addr(0xcc1be03100154616) [ 38.779631] mpt3sas_cm0: handle(0x32) sas_address(0xcc1be03100154617) port_type(0x1) [ 38.789334] scsi 10:0:23:0: SATA: handle(0x0032), sas_addr(0xcc1be03100154617), phy(23), device_name(0x5000cca2bec2f265) [ 38.880027] end_device-10:0:23: add: handle(0x0032), sas_addr(0xcc1be03100154617) [ 38.884685] mpt3sas_cm0: handle(0x21) sas_address(0xcc1be0310015463d) port_type(0x1) [ 38.901648] scsi 10:0:24:0: SES: handle(0x0021), sas_addr(0xcc1be0310015463d), phy(36), device_name(0xcc1be0310015463d) [ 38.919628] end_device-10:0:24: add: handle(0x0021), sas_addr(0xcc1be0310015463d) I was unable to find either the firmware for the expander nor the firmware for the controller. I am attaching diagnostics from the SAS controller in single link to the expander and running a parity check, as i didn't want to risk any more write errors. When it was in dual link mode i would get a flood of the following messages and an increase in UDMA CRC errors: mpt3sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000) mpt3sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000) mpt3sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000) mpt3sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000) mpt3sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000) mpt3sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000) beyonder-nas-diagnostics-20250314-1045.zip
March 14, 20251 yr 1 hour ago, scs3jb said: Currently I am saturating the single channel link when I do parity actions and it tops out at 2.2G. As 12Gb (SAS3) single link have 48Gb ( ~4GB ), so faraway to saturate. Edited March 14, 20251 yr by Vr2Io
March 14, 20251 yr Community Expert Solution 9 minutes ago, Vr2Io said: As 12Gb (SAS3) single link have 48Gb ( ~4GB ), so faraway to saturate. That's true, but keep in mind that it only applies if the devices are SAS3, or the expander supports LSI's Databolt or similar technology, or it will be limited to 2.2GB per link with SATA3 devices. 1 hour ago, scs3jb said: FWVersion(16.00.01.00) I would start by upgrading the HBA firmware, I believe the latest is 16.00.10.00 or 16.00.12.00
March 14, 20251 yr Author 27 minutes ago, JorgeB said: I would start by upgrading the HBA firmware, I believe the latest is 16.00.10.00 or 16.00.12.00 Do you know where i could find this? I struggled to find the latest and broadcom site seems to no longer have download for anything but the latest. Edit: nevermind, i found what i need: https://www.broadcom.com/site-search?q=Installer_P16_for_Linux https://docs.broadcom.com/docs/9305_24i_Pkg_P16.12_IT_FW_BIOS_for_MSDOS_Windows.zip run sas3flash -list to confirm the version and for some reason its for MSDOS, you can change the model and 24i bit of the url to get the right package. Edited March 14, 20251 yr by scs3jb fishing rod
March 14, 20251 yr Author Let me write a miniguide for muppets like me: Download https://www.broadcom.com/site-search?q=Installer_P16_for_Linux and extract it to somewhere you can run things. Confirm the make, model and firmware with: root@beyonder-nas:/mnt/scratch_old/backups/storage# ./sas3flash -list Avago Technologies SAS3 Flash Utility Version 17.00.00.00 (2018.04.02) Copyright 2008-2018 Avago Technologies. All rights reserved. Adapter Selected is a Avago SAS: SAS3224(A1) Controller Number : 0 Controller : SAS3224(A1) PCI Address : 00:02:00:00 SAS Address : 500062b-2-0299-1611 NVDATA Version (Default) : 10.00.00.03 NVDATA Version (Persistent) : 10.00.00.03 Firmware Product ID : 0x2228 (IT) Firmware Version : 16.00.01.00 NVDATA Vendor : LSI NVDATA Product ID : SAS9305-24i BIOS Version : 08.37.00.00 UEFI BSD Version : 18.00.00.00 FCODE Version : N/A Board Name : SAS9305-24i Board Assembly : 03-25699-02004 Board Tracer Number : XW841903CD Finished Processing Commands Successfully. Exiting SAS3Flash. I have a 9305-24i, so need this https://docs.broadcom.com/docs/9305_24i_Pkg_P16.12_IT_FW_BIOS_for_MSDOS_Windows.zip . extract it and look for the firmware, it will be a .bin file. Now we can flash the firmware: root@beyonder-nas:/mnt/scratch_old/backups/storage# ./sas3flash -o -c 00 -f SAS9305_24i_IT_P.bin Avago Technologies SAS3 Flash Utility Version 17.00.00.00 (2018.04.02) Copyright 2008-2018 Avago Technologies. All rights reserved. Advanced Mode Set Adapter Selected is a Avago SAS: SAS3224(A1) Executing Operation: Flash Firmware Image Firmware Image has a Valid Checksum. Firmware Version 16.00.12.00 Firmware Image compatible with Controller. Valid NVDATA Image found. NVDATA Major Version 10.00 Checking for a compatible NVData image... NVDATA Device ID and Chip Revision match verified. NVDATA Versions Compatible. Valid Initialization Image verified. Valid BootLoader Image verified. Beginning Firmware Download... Firmware Download Successful. Verifying Download... Firmware Flash Successful. Resetting Adapter... Adapter Successfully Reset. NVDATA Version 10.00.00.03 Finished Processing Commands Successfully. Exiting SAS3Flash. Finally you can confirm it is flashed: root@beyonder-nas:/mnt/scratch_old/backups/storage# ./sas3flash -list Avago Technologies SAS3 Flash Utility Version 17.00.00.00 (2018.04.02) Copyright 2008-2018 Avago Technologies. All rights reserved. Adapter Selected is a Avago SAS: SAS3224(A1) Controller Number : 0 Controller : SAS3224(A1) PCI Address : 00:02:00:00 SAS Address : 500062b-2-0299-1611 NVDATA Version (Default) : 10.00.00.03 NVDATA Version (Persistent) : 10.00.00.03 Firmware Product ID : 0x2228 (IT) Firmware Version : 16.00.12.00 NVDATA Vendor : LSI NVDATA Product ID : SAS9305-24i BIOS Version : 08.37.00.00 UEFI BSD Version : 18.00.00.00 FCODE Version : N/A Board Name : SAS9305-24i Board Assembly : 03-25699-02004 Board Tracer Number : XW841903CD Finished Processing Commands Successfully. Exiting SAS3Flash. Now i recable and see if this solved my problem
March 15, 20251 yr Author Thank you @JorgeB, dual link is now working with the firmware updated. Dual link in use, no more errors in syslog/dmesg and running a parity check is getting ~150 mb/s across 21 disks. I'm happy with that for sata drives on an expander. Marking as solution.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.