Read Errors on New Server Build


Recommended Posts

Last night I moved all my hard drives over to a new server I built.  I finally upgraded to a chassis that has 16 bays and connects to the backplane with 2 SAS cables.  I bought the chassis off ebay used but the cables were new.  I have been reading that most of these errors are fixed with new cables but this a brand new cable so I hope it is not that and it would seem like it would be more than just disk 5 with errors.  I will post my diagnostics but some advice would be much appreciated.  I have already rebuilt disk 5 using the same drive and same spot in the chassis.  I am hoping it is not a bad spot in the chassis.  I noticed the drive showed up in Unassigned Devices while being spun down in the array and then shortly after I got the read error.  

finalizer-diagnostics-20191203-1855.zip

Link to comment

There are communication issues with multiple disks:

Dec  3 11:52:37 Finalizer kernel: sd 9:0:1:0: attempting task abort! scmd(00000000375afb15)
Dec  3 11:52:37 Finalizer kernel: sd 9:0:1:0: [sdd] tag#6528 CDB: opcode=0x85 85 06 20 00 d8 00 00 00 00 00 4f 00 c2 00 b0 00
Dec  3 11:52:37 Finalizer kernel: scsi target9:0:1: handle(0x000b), sas_address(0x5003048001d979ad), phy(13)
Dec  3 11:52:37 Finalizer kernel: scsi target9:0:1: enclosure logical id(0x5003048001d979bf), slot(1)
Dec  3 11:52:37 Finalizer kernel: sd 9:0:1:0: device_block, handle(0x000b)
Dec  3 11:52:38 Finalizer kernel: sd 9:0:1:0: task abort: SUCCESS scmd(00000000375afb15)
Dec  3 11:52:39 Finalizer kernel: sd 9:0:1:0: device_unblock and setting to running, handle(0x000b)
Dec  3 11:52:46 Finalizer kernel: sd 9:0:7:0: attempting task abort! scmd(00000000ff7cd007)
Dec  3 11:52:46 Finalizer kernel: sd 9:0:7:0: [sdj] tag#6568 CDB: opcode=0x85 85 06 20 00 d8 00 00 00 00 00 4f 00 c2 00 b0 00
Dec  3 11:52:46 Finalizer kernel: scsi target9:0:7: handle(0x0011), sas_address(0x5003048001d979b3), phy(19)
Dec  3 11:52:46 Finalizer kernel: scsi target9:0:7: enclosure logical id(0x5003048001d979bf), slot(7)
Dec  3 11:52:46 Finalizer kernel: sd 9:0:7:0: device_block, handle(0x0011)
Dec  3 11:52:48 Finalizer kernel: sd 9:0:7:0: task abort: SUCCESS scmd(00000000ff7cd007)
Dec  3 11:52:48 Finalizer kernel: sd 9:0:7:0: device_unblock and setting to running, handle(0x0011)
Dec  3 11:59:29 Finalizer kernel: mdcmd (50): spindown 5
Dec  3 12:00:08 Finalizer kernel: sd 9:0:13:0: attempting task abort! scmd(00000000454c3926)
Dec  3 12:00:08 Finalizer kernel: sd 9:0:13:0: [sdo] tag#6783 CDB: opcode=0x85 85 06 20 00 d8 00 00 00 00 00 4f 00 c2 00 b0 00
Dec  3 12:00:08 Finalizer kernel: scsi target9:0:13: handle(0x000f), sas_address(0x5003048001d979b1), phy(17)
Dec  3 12:00:08 Finalizer kernel: scsi target9:0:13: enclosure logical id(0x5003048001d979bf), slot(5)
Dec  3 12:00:08 Finalizer kernel: sd 9:0:13:0: device_block, handle(0x000f)
Dec  3 12:00:10 Finalizer kernel: sd 9:0:13:0: device_unblock and setting to running, handle(0x000f)
Dec  3 12:00:11 Finalizer kernel: sd 9:0:13:0: [sdo] Synchronizing SCSI cache
Dec  3 12:00:11 Finalizer kernel: sd 9:0:13:0: [sdo] Synchronize Cache(10) failed: Result: hostbyte=0x01 driverbyte=0x00

 

So unlikely to be a bad slot, most likely it can be the cables, power issue, backplane expander or the HBA, unfortunately not easy to say which one is the culprit without swapping things around to rule them out.

Link to comment

 

2 minutes ago, johnnie.black said:

There are communication issues with multiple disks:


Dec  3 11:52:37 Finalizer kernel: sd 9:0:1:0: attempting task abort! scmd(00000000375afb15)
Dec  3 11:52:37 Finalizer kernel: sd 9:0:1:0: [sdd] tag#6528 CDB: opcode=0x85 85 06 20 00 d8 00 00 00 00 00 4f 00 c2 00 b0 00
Dec  3 11:52:37 Finalizer kernel: scsi target9:0:1: handle(0x000b), sas_address(0x5003048001d979ad), phy(13)
Dec  3 11:52:37 Finalizer kernel: scsi target9:0:1: enclosure logical id(0x5003048001d979bf), slot(1)
Dec  3 11:52:37 Finalizer kernel: sd 9:0:1:0: device_block, handle(0x000b)
Dec  3 11:52:38 Finalizer kernel: sd 9:0:1:0: task abort: SUCCESS scmd(00000000375afb15)
Dec  3 11:52:39 Finalizer kernel: sd 9:0:1:0: device_unblock and setting to running, handle(0x000b)
Dec  3 11:52:46 Finalizer kernel: sd 9:0:7:0: attempting task abort! scmd(00000000ff7cd007)
Dec  3 11:52:46 Finalizer kernel: sd 9:0:7:0: [sdj] tag#6568 CDB: opcode=0x85 85 06 20 00 d8 00 00 00 00 00 4f 00 c2 00 b0 00
Dec  3 11:52:46 Finalizer kernel: scsi target9:0:7: handle(0x0011), sas_address(0x5003048001d979b3), phy(19)
Dec  3 11:52:46 Finalizer kernel: scsi target9:0:7: enclosure logical id(0x5003048001d979bf), slot(7)
Dec  3 11:52:46 Finalizer kernel: sd 9:0:7:0: device_block, handle(0x0011)
Dec  3 11:52:48 Finalizer kernel: sd 9:0:7:0: task abort: SUCCESS scmd(00000000ff7cd007)
Dec  3 11:52:48 Finalizer kernel: sd 9:0:7:0: device_unblock and setting to running, handle(0x0011)
Dec  3 11:59:29 Finalizer kernel: mdcmd (50): spindown 5
Dec  3 12:00:08 Finalizer kernel: sd 9:0:13:0: attempting task abort! scmd(00000000454c3926)
Dec  3 12:00:08 Finalizer kernel: sd 9:0:13:0: [sdo] tag#6783 CDB: opcode=0x85 85 06 20 00 d8 00 00 00 00 00 4f 00 c2 00 b0 00
Dec  3 12:00:08 Finalizer kernel: scsi target9:0:13: handle(0x000f), sas_address(0x5003048001d979b1), phy(17)
Dec  3 12:00:08 Finalizer kernel: scsi target9:0:13: enclosure logical id(0x5003048001d979bf), slot(5)
Dec  3 12:00:08 Finalizer kernel: sd 9:0:13:0: device_block, handle(0x000f)
Dec  3 12:00:10 Finalizer kernel: sd 9:0:13:0: device_unblock and setting to running, handle(0x000f)
Dec  3 12:00:11 Finalizer kernel: sd 9:0:13:0: [sdo] Synchronizing SCSI cache
Dec  3 12:00:11 Finalizer kernel: sd 9:0:13:0: [sdo] Synchronize Cache(10) failed: Result: hostbyte=0x01 driverbyte=0x00

 

So unlikely to be a bad slot, most likely it can be the cables, power issue, backplane expander or the HBA, unfortunately not easy to say which one is the culprit without swapping things around to rule them out.

 

 

Any advice on the best way to troubleshoot this?  I don't really have any extra of anything to start troubleshooting with a new component.  Wouldn't other disks get read errors if it was a cable or HBA issue?

Link to comment

Well maybe this answers something.  I was doing a rebuild on Disk 5 (new disk but same slot) and I got more errors on the parity and Disk 4 and Disk 3.  This makes me think it has something more to do with a cable, backplane, or HBA.  I have 2 power supplies so I hope it is not the but I guess it is possible.  I am going to let it rebuild Disk 5 then probably shut it down until I can get new cables unless there is no risk of corrupting the data?  Before I tried rebuilding the drive I tried to check all the connections to make sure they were tightly connected.

 

image.png.5e3db4541f0077e25a5a9d1ab433956a.png

Link to comment
5 hours ago, mgranger said:

I got more errors on the parity and Disk 4 and Disk 3. 

UDMA CRC errors are usually a bad cable, much less likely but can also be controller or backplane.

 

5 hours ago, mgranger said:

unless there is no risk of corrupting the data?

Not with UDMA CRC errors, but if more disks have read errors it can, just post the diags after the rebuild is done if you're not sure.

 

Link to comment

So the drive i took out of Drive 5 was a 4TB HGST drive.  I have it in the unassigned devices now but it is refusing to mount.  Is there a reason that this is happening?  I figured I would be able to mount it using UD.  I think I can reformat it now that the other disk was rebuilt but just trying to troubleshoot why this would happen.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.