Super slow MB/s on drive rebuild

[TheBawse]

I just upgraded a drive to something larger and upon starting my rebuild I noticed my speeds are really slow.  I usually am between 75-100mb/s and I haven't gone above 25mb/s.

I took a look through my logs but I didn't see anything that could explain the bottleneck.  I have attached my diagnostics if someone could take a look and give me a second opinion it would be much appreciated.

tower-diagnostics-20230121-2028.zip

[apandey]

Do you have any other read / writes happening on the array during the rebuild? That severely slows does the rebuild because the drive head needs to shuffle between near sequential rebuild access to random access needed for other activities

[JorgeB]

You are having problems with multiple disks, check/replace power and SATA cables, could also be a PSU problem.

[TheBawse]

4 hours ago, apandey said:

Do you have any other read / writes happening on the array during the rebuild? That severely slows does the rebuild because the drive head needs to shuffle between near sequential rebuild access to random access needed for other activities

No, I turned off docker and want doing any other work. 

[TheBawse]

3 hours ago, JorgeB said:

You are having problems with multiple disks, check/replace power and SATA cables, could also be a PSU problem.

PSU is new and all cables were replaced a few months ago with startech SAS to SATA cables.

[JorgeB]

20 hours ago, TheBawse said:

PSU is new and all cables were replaced a few months ago with startech SAS to SATA cables.

Still need to try and change something to fix those errors, you should also upgrade the LSI firmware since it's quite old.

[TheBawse]

7 hours ago, JorgeB said:

Still need to try and change something to fix those errors, you should also upgrade the LSI firmware since it's quite old.

Thanks. Did you see disk errors in the diag file? I'll look into the firmware on my HBAs. I assumed I had the lastest.

[JorgeB]

28 minutes ago, TheBawse said:

Did you see disk errors in the diag file?

Yes, in the syslog.

[TheBawse]

18 minutes ago, JorgeB said:

Yes, in the syslog.

I'll check again. Didn't see any first time.

[trurl]

32 minutes ago, TheBawse said:

I'll check again. Didn't see any first time.

Only the last 2000+ lines in syslog

[TheBawse]

2 hours ago, trurl said:

Only the last 2000+ lines in syslog

Jan 21 20:28:33 Tower kernel: sd 10:0:10:0: task abort: SUCCESS scmd(0x000000004380e880)
Jan 21 20:28:33 Tower kernel: sd 10:0:10:0: attempting task abort!scmd(0x0000000051f87a4f), outstanding for 30516 ms & timeout 30000 ms
Jan 21 20:28:33 Tower kernel: sd 10:0:10:0: [sdu] tag#2860 CDB: opcode=0x88 88 00 00 00 00 00 01 96 80 f0 00 00 04 00 00 00
Jan 21 20:28:33 Tower kernel: scsi target10:0:10: handle(0x0014), sas_address(0x5001e67464526ff6), phy(22)
Jan 21 20:28:33 Tower kernel: scsi target10:0:10: enclosure logical id(0x5001e67464526fff), slot(22) 
Jan 21 20:28:33 Tower kernel: sd 10:0:10:0: No reference found at driver, assuming scmd(0x0000000051f87a4f) might have completed
Jan 21 20:28:33 Tower kernel: sd 10:0:10:0: task abort: SUCCESS scmd(0x0000000051f87a4f)
Jan 21 20:28:33 Tower kernel: sd 10:0:10:0: attempting task abort!scmd(0x000000001715f78a), outstanding for 30431 ms & timeout 30000 ms
Jan 21 20:28:33 Tower kernel: sd 10:0:10:0: [sdu] tag#2848 CDB: opcode=0x88 88 00 00 00 00 00 01 96 9c f0 00 00 04 00 00 00
Jan 21 20:28:33 Tower kernel: scsi target10:0:10: handle(0x0014), sas_address(0x5001e67464526ff6), phy(22)
Jan 21 20:28:33 Tower kernel: scsi target10:0:10: enclosure logical id(0x5001e67464526fff), slot(22) 
Jan 21 20:28:33 Tower kernel: sd 10:0:10:0: No reference found at driver, assuming scmd(0x000000001715f78a) might have completed
Jan 21 20:28:33 Tower kernel: sd 10:0:10:0: task abort: SUCCESS scmd(0x000000001715f78a)

Yeah...No idea what it means though

[trurl]

They are problems communicating with disks. If you skim over those you will see references to multiple disks, here are some examples (from syslog):

Jan 21 20:07:15 Tower kernel: sd 10:0:0:0: [sdk] tag#2316 CDB: opcode=0x88 88 00 00 00 00 00 00 0b 7f 58 00 00 00 20 00 00
Jan 21 20:08:36 Tower kernel: sd 10:0:5:0: [sdp] tag#2880 CDB: opcode=0x88 88 00 00 00 00 00 00 16 af 40 00 00 04 00 00 00
Jan 21 20:08:36 Tower kernel: sd 10:0:10:0: [sdu] tag#2884 CDB: opcode=0x88 88 00 00 00 00 00 00 16 af 40 00 00 04 00 00 00
Jan 21 20:10:11 Tower kernel: sd 10:0:9:0: [sdt] tag#2780 CDB: opcode=0x88 88 00 00 00 00 00 00 22 15 b0 00 00 04 00 00 00
Jan 21 20:11:36 Tower kernel: sd 10:0:4:0: [sdo] tag#2709 CDB: opcode=0x88 88 00 00 00 00 00 00 2f cb 58 00 00 02 98 00 00
Jan 21 20:12:20 Tower kernel: sd 10:0:3:0: [sdn] tag#3004 CDB: opcode=0x88 88 00 00 00 00 00 00 3f 83 88 00 00 01 68 00 00
Jan 21 20:13:29 Tower kernel: sd 10:0:1:0: [sdl] tag#3126 CDB: opcode=0x88 88 00 00 00 00 00 00 47 93 68 00 00 04 00 00 00
Jan 21 20:14:28 Tower kernel: sd 10:0:2:0: [sdm] tag#3039 CDB: opcode=0x88 88 00 00 00 00 00 00 68 21 68 00 00 02 b8 00 00
Jan 21 20:19:47 Tower kernel: sd 10:0:11:0: [sdv] tag#2871 CDB: opcode=0x88 88 00 00 00 00 00 00 e1 2a 48 00 00 04 00 00 00

This controller (from system/lspci):

10:00.0 Serial Attached SCSI controller [0107]: Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] [1000:0072] (rev 03)
	Subsystem: Dell 6Gbps SAS HBA Adapter [1028:1f1c]
	Kernel driver in use: mpt3sas
	Kernel modules: mpt3sas