Jump to content
We're Hiring! Full Stack Developer ×

Getting some Disk issues, Logs included

Blaze9

By Blaze9
in General Support

Recommended Posts

Blaze9 Rookie

Blaze9

Posted
Posted

Hey All,

 

All of a sudden I'm getting some errors on my cache disk (SDJ1)

  

\Mar 17 17:04:06 Mk4Alpha kernel: ata12.00: exception Emask 0x0 SAct 0xc000000 SErr 0x0 action 0x6 frozen
Mar 17 17:04:06 Mk4Alpha kernel: ata12.00: failed command: READ FPDMA QUEUED
Mar 17 17:04:06 Mk4Alpha kernel: ata12.00: cmd 60/30:d0:50:3e:03/00:00:11:00:00/40 tag 26 ncq dma 24576 in
Mar 17 17:04:06 Mk4Alpha kernel: res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Mar 17 17:04:06 Mk4Alpha kernel: ata12.00: status: { DRDY }
Mar 17 17:04:06 Mk4Alpha kernel: ata12.00: failed command: READ FPDMA QUEUED
Mar 17 17:04:06 Mk4Alpha kernel: ata12.00: cmd 60/08:d8:78:54:03/00:00:11:00:00/40 tag 27 ncq dma 4096 in
Mar 17 17:04:06 Mk4Alpha kernel: res 40/00:ff:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Mar 17 17:04:06 Mk4Alpha kernel: ata12.00: status: { DRDY }
Mar 17 17:04:06 Mk4Alpha kernel: ata12: hard resetting link
Mar 17 17:04:07 Mk4Alpha kernel: ata12: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
Mar 17 17:04:07 Mk4Alpha kernel: ata12.00: supports DRM functions and may not be fully accessible
Mar 17 17:04:07 Mk4Alpha kernel: ata12.00: supports DRM functions and may not be fully accessible
Mar 17 17:04:07 Mk4Alpha kernel: ata12.00: configured for UDMA/133
Mar 17 17:04:07 Mk4Alpha kernel: sd 12:0:0:0: [sdl] tag#26 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08
Mar 17 17:04:07 Mk4Alpha kernel: sd 12:0:0:0: [sdl] tag#26 Sense Key : 0x5 [current]
Mar 17 17:04:07 Mk4Alpha kernel: sd 12:0:0:0: [sdl] tag#26 ASC=0x21 ASCQ=0x4
Mar 17 17:04:07 Mk4Alpha kernel: sd 12:0:0:0: [sdl] tag#26 CDB: opcode=0x28 28 00 11 03 3e 50 00 00 30 00
Mar 17 17:04:07 Mk4Alpha kernel: print_req_error: I/O error, dev sdl, sector 285425232
Mar 17 17:04:07 Mk4Alpha kernel: sd 12:0:0:0: [sdl] tag#27 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08
Mar 17 17:04:07 Mk4Alpha kernel: sd 12:0:0:0: [sdl] tag#27 Sense Key : 0x5 [current]
Mar 17 17:04:07 Mk4Alpha kernel: sd 12:0:0:0: [sdl] tag#27 ASC=0x21 ASCQ=0x4
Mar 17 17:04:07 Mk4Alpha kernel: sd 12:0:0:0: [sdl] tag#27 CDB: opcode=0x28 28 00 11 03 54 78 00 00 08 00
Mar 17 17:04:07 Mk4Alpha kernel: print_req_error: I/O error, dev sdl, sector 285430904
Mar 17 17:04:07 Mk4Alpha kernel: ata12: EH complete
Mar 17 17:04:07 Mk4Alpha kernel: BTRFS info (device sdj1): read error corrected: ino 835828 off 22508195840 (dev /dev/sdl1 sector 285430840)
Mar 17 17:05:01 Mk4Alpha sSMTP[20947]: Creating SSL connection to host
Mar 17 17:05:01 Mk4Alpha sSMTP[20947]: SSL connection using TLS_AES_256_GCM_SHA384
Mar 17 17:05:16 Mk4Alpha unassigned.devices: Error: shell_exec(/usr/bin/lsof '/mnt/disks/Profile' 2>/dev/null | /bin/sort -k8 | /bin/uniq -f7 | /bin/grep -c -e REG) took longer than 5s!
Mar 17 17:06:23 Mk4Alpha kernel: ata12.00: exception Emask 0x0 SAct 0x40000000 SErr 0x0 action 0x6 frozen
Mar 17 17:06:23 Mk4Alpha kernel: ata12.00: failed command: READ FPDMA QUEUED
Mar 17 17:06:23 Mk4Alpha kernel: ata12.00: cmd 60/18:f0:d0:65:56/00:00:40:00:00/40 tag 30 ncq dma 12288 in
Mar 17 17:06:23 Mk4Alpha kernel: res 40/00:00:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
Mar 17 17:06:23 Mk4Alpha kernel: ata12.00: status: { DRDY }
Mar 17 17:06:23 Mk4Alpha kernel: ata12: hard resetting link
Mar 17 17:06:23 Mk4Alpha kernel: ata12: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
Mar 17 17:06:23 Mk4Alpha kernel: ata12.00: supports DRM functions and may not be fully accessible
Mar 17 17:06:23 Mk4Alpha kernel: ata12.00: supports DRM functions and may not be fully accessible
Mar 17 17:06:23 Mk4Alpha kernel: ata12.00: configured for UDMA/133
Mar 17 17:06:23 Mk4Alpha kernel: sd 12:0:0:0: [sdl] tag#30 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08
Mar 17 17:06:23 Mk4Alpha kernel: sd 12:0:0:0: [sdl] tag#30 Sense Key : 0x5 [current]
Mar 17 17:06:23 Mk4Alpha kernel: sd 12:0:0:0: [sdl] tag#30 ASC=0x21 ASCQ=0x4
Mar 17 17:06:23 Mk4Alpha kernel: sd 12:0:0:0: [sdl] tag#30 CDB: opcode=0x28 28 00 40 56 65 d0 00 00 18 00
Mar 17 17:06:23 Mk4Alpha kernel: print_req_error: I/O error, dev sdl, sector 1079403984
Mar 17 17:06:23 Mk4Alpha kernel: ata12: EH complete
Mar 17 17:06:39 Mk4Alpha unassigned.devices: Error: shell_exec(/usr/bin/lsof '/mnt/disks/Profile' 2>/dev/null | /bin/sort -k8 | /bin/uniq -f7 | /bin/grep -c -e REG) took longer than 5s!
Mar 17 17:07:33 Mk4Alpha kernel: ata12.00: exception Emask 0x0 SAct 0x42 SErr 0x0 action 0x6 frozen
Mar 17 17:07:33 Mk4Alpha kernel: ata12.00: failed command: READ FPDMA QUEUED
Mar 17 17:07:33 Mk4Alpha kernel: ata12.00: cmd 60/08:08:60:0d:30/00:00:34:00:00/40 tag 1 ncq dma 4096 in
Mar 17 17:07:33 Mk4Alpha kernel: res 40/00:ff:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Mar 17 17:07:33 Mk4Alpha kernel: ata12.00: status: { DRDY }
Mar 17 17:07:33 Mk4Alpha kernel: ata12.00: failed command: READ FPDMA QUEUED
Mar 17 17:07:33 Mk4Alpha kernel: ata12.00: cmd 60/08:30:90:0a:30/00:00:34:00:00/40 tag 6 ncq dma 4096 in
Mar 17 17:07:33 Mk4Alpha kernel: res 40/00:00:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
Mar 17 17:07:33 Mk4Alpha kernel: ata12.00: status: { DRDY }
Mar 17 17:07:33 Mk4Alpha kernel: ata12: hard resetting link
Mar 17 17:07:33 Mk4Alpha kernel: ata12: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
Mar 17 17:07:33 Mk4Alpha kernel: ata12.00: supports DRM functions and may not be fully accessible
Mar 17 17:07:33 Mk4Alpha kernel: ata12.00: supports DRM functions and may not be fully accessible
Mar 17 17:07:33 Mk4Alpha kernel: ata12.00: configured for UDMA/133
Mar 17 17:07:33 Mk4Alpha kernel: sd 12:0:0:0: [sdl] tag#1 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08
Mar 17 17:07:33 Mk4Alpha kernel: sd 12:0:0:0: [sdl] tag#1 Sense Key : 0x5 [current]
Mar 17 17:07:33 Mk4Alpha kernel: sd 12:0:0:0: [sdl] tag#1 ASC=0x21 ASCQ=0x4
Mar 17 17:07:33 Mk4Alpha kernel: sd 12:0:0:0: [sdl] tag#1 CDB: opcode=0x28 28 00 34 30 0d 60 00 00 08 00
Mar 17 17:07:33 Mk4Alpha kernel: print_req_error: I/O error, dev sdl, sector 875564384
Mar 17 17:07:33 Mk4Alpha kernel: sd 12:0:0:0: [sdl] tag#6 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08
Mar 17 17:07:33 Mk4Alpha kernel: sd 12:0:0:0: [sdl] tag#6 Sense Key : 0x5 [current]
Mar 17 17:07:33 Mk4Alpha kernel: sd 12:0:0:0: [sdl] tag#6 ASC=0x21 ASCQ=0x4
Mar 17 17:07:33 Mk4Alpha kernel: sd 12:0:0:0: [sdl] tag#6 CDB: opcode=0x28 28 00 34 30 0a 90 00 00 08 00
Mar 17 17:07:33 Mk4Alpha kernel: print_req_error: I/O error, dev sdl, sector 875563664
Mar 17 17:07:33 Mk4Alpha kernel: ata12: EH complete
Mar 17 17:07:33 Mk4Alpha kernel: BTRFS info (device sdj1): read error corrected: ino 24266 off 70832128 (dev /dev/sdl1 sector 875563600)
Mar 17 17:07:33 Mk4Alpha kernel: BTRFS info (device sdj1): read error corrected: ino 24266 off 70807552 (dev /dev/sdl1 sector 875564320)
Mar 17 17:07:38 Mk4Alpha shutdown[24595]: shutting down for system reboot
Mar 17 17:08:53 Mk4Alpha kernel: ata9.00: exception Emask 0x0 SAct 0x4004 SErr 0x0 action 0x6 frozen
Mar 17 17:08:53 Mk4Alpha kernel: ata9.00: failed command: READ FPDMA QUEUED
Mar 17 17:08:53 Mk4Alpha kernel: ata9.00: cmd 60/08:10:78:4d:54/00:00:12:00:00/40 tag 2 ncq dma 4096 in
Mar 17 17:08:53 Mk4Alpha kernel: res 40/00:00:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
Mar 17 17:08:53 Mk4Alpha kernel: ata9.00: status: { DRDY }
Mar 17 17:08:53 Mk4Alpha kernel: ata9.00: failed command: READ FPDMA QUEUED
Mar 17 17:08:53 Mk4Alpha kernel: ata9.00: cmd 60/08:70:f8:a3:50/00:00:12:00:00/40 tag 14 ncq dma 4096 in
Mar 17 17:08:53 Mk4Alpha kernel: res 40/00:ff:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Mar 17 17:08:53 Mk4Alpha kernel: ata9.00: status: { DRDY }
Mar 17 17:08:53 Mk4Alpha kernel: ata9: hard resetting link
Mar 17 17:08:54 Mk4Alpha kernel: ata9: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
Mar 17 17:08:54 Mk4Alpha kernel: ata9.00: supports DRM functions and may not be fully accessible
Mar 17 17:08:54 Mk4Alpha kernel: ata9.00: NCQ Send/Recv Log not supported
Mar 17 17:08:54 Mk4Alpha kernel: ata9.00: supports DRM functions and may not be fully accessible
Mar 17 17:08:54 Mk4Alpha kernel: ata9.00: NCQ Send/Recv Log not supported
Mar 17 17:08:54 Mk4Alpha kernel: ata9.00: configured for UDMA/133
Mar 17 17:08:54 Mk4Alpha kernel: sd 9:0:0:0: [sdj] tag#2 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08
Mar 17 17:08:54 Mk4Alpha kernel: sd 9:0:0:0: [sdj] tag#2 Sense Key : 0x5 [current]
Mar 17 17:08:54 Mk4Alpha kernel: sd 9:0:0:0: [sdj] tag#2 ASC=0x21 ASCQ=0x4
Mar 17 17:08:54 Mk4Alpha kernel: sd 9:0:0:0: [sdj] tag#2 CDB: opcode=0x28 28 00 12 54 4d 78 00 00 08 00
Mar 17 17:08:54 Mk4Alpha kernel: print_req_error: I/O error, dev sdj, sector 307514744
Mar 17 17:08:54 Mk4Alpha kernel: sd 9:0:0:0: [sdj] tag#14 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08
Mar 17 17:08:54 Mk4Alpha kernel: sd 9:0:0:0: [sdj] tag#14 Sense Key : 0x5 [current]
Mar 17 17:08:54 Mk4Alpha kernel: sd 9:0:0:0: [sdj] tag#14 ASC=0x21 ASCQ=0x4
Mar 17 17:08:54 Mk4Alpha kernel: sd 9:0:0:0: [sdj] tag#14 CDB: opcode=0x28 28 00 12 50 a3 f8 00 00 08 00
Mar 17 17:08:54 Mk4Alpha kernel: print_req_error: I/O error, dev sdj, sector 307274744
Mar 17 17:08:54 Mk4Alpha kernel: ata9: EH complete
Mar 17 17:08:54 Mk4Alpha kernel: BTRFS info (device sdj1): read error corrected: ino 24266 off 561152 (dev /dev/sdj1 sector 307274680)
Mar 17 17:08:54 Mk4Alpha kernel: BTRFS info (device sdj1): read error corrected: ino 24266 off 67854336 (dev /dev/sdj1 sector 307514680)

 

Looks like sdj is reporting errors, but I restarted the unresponsive server and got no errors on SMART.... sdj and sdl are my cache pool drives (RAID1) so I can replace the defunct drive w/ a new one soon, but I need to purchase one! Is this the only issue seen? 

 

Aside from replacing these old drives proactively, do you guys suggest anything? I'm attaching the full log here!

 

Thanks

 

mk4alpha-diagnostics-20200317-1732.zip

Link to comment
Blaze9 Rookie

Blaze9

Posted
  • Author
Posted

Taking a look at the controllers:

  

root@Mk4Alpha:~# lspci | grep -i 5:00.0
05:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9230 PCIe SATA 6Gb/s Controller (rev 11)
85:00.0 VGA compatible controller: NVIDIA Corporation GM206 [GeForce GTX 960] (rev a1)
root@Mk4Alpha:~# ls -al /sys/block/sd*
lrwxrwxrwx 1 root root 0 Mar 19 10:03 /sys/block/sda -> ../devices/pci0000:00/0000:00:1a.7/usb1/1-1/1-1:1.0/host0/target0:0:0/0:0:0:0/block/sda/
lrwxrwxrwx 1 root root 0 Mar 19 10:03 /sys/block/sdb -> ../devices/pci0000:00/0000:00:1f.2/ata3/host3/target3:0:0/3:0:0:0/block/sdb/
lrwxrwxrwx 1 root root 0 Mar 19 10:03 /sys/block/sdc -> ../devices/pci0000:00/0000:00:07.0/0000:04:00.0/host17/port-17:0/end_device-17:0/target17:0:0/17:0:0:0/block/sdc/
lrwxrwxrwx 1 root root 0 Mar 19 10:03 /sys/block/sdd -> ../devices/pci0000:00/0000:00:07.0/0000:04:00.0/host17/port-17:1/end_device-17:1/target17:0:1/17:0:1:0/block/sdd/
lrwxrwxrwx 1 root root 0 Mar 19 10:03 /sys/block/sde -> ../devices/pci0000:00/0000:00:1f.2/ata4/host4/target4:0:0/4:0:0:0/block/sde/
lrwxrwxrwx 1 root root 0 Mar 19 10:03 /sys/block/sdf -> ../devices/pci0000:00/0000:00:1f.2/ata5/host5/target5:0:0/5:0:0:0/block/sdf/
lrwxrwxrwx 1 root root 0 Mar 19 10:03 /sys/block/sdg -> ../devices/pci0000:00/0000:00:1f.2/ata6/host6/target6:0:0/6:0:0:0/block/sdg/
lrwxrwxrwx 1 root root 0 Mar 19 10:03 /sys/block/sdh -> ../devices/pci0000:00/0000:00:1f.2/ata7/host7/target7:0:0/7:0:0:0/block/sdh/
lrwxrwxrwx 1 root root 0 Mar 19 10:03 /sys/block/sdi -> ../devices/pci0000:00/0000:00:1f.2/ata8/host8/target8:0:0/8:0:0:0/block/sdi/
lrwxrwxrwx 1 root root 0 Mar 19 10:03 /sys/block/sdj -> ../devices/pci0000:00/0000:00:09.0/0000:05:00.0/ata9/host9/target9:0:0/9:0:0:0/block/sdj/
lrwxrwxrwx 1 root root 0 Mar 19 10:03 /sys/block/sdk -> ../devices/pci0000:00/0000:00:09.0/0000:05:00.0/ata11/host11/target11:0:0/11:0:0:0/block/sdk/
lrwxrwxrwx 1 root root 0 Mar 19 10:03 /sys/block/sdl -> ../devices/pci0000:00/0000:00:09.0/0000:05:00.0/ata12/host12/target12:0:0/12:0:0:0/block/sdl/

Looks like sdk, sdj, sdl are all on my marvel controller which I should move away from. 

 

The LSI card I have right now ( 04:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)) doesn't seem to support TRIM and the machine won't boot when I attach my SSDs to that.

 

I have a question however. If I remove the SATA card and rely on the LSI card, can I reconfigure my drive ports? All my mobo ports are used by HDDs, can I swap two out and give them to my SSDs, and then put those two on the LSI card, or will the physical address change cause issues for the array? Thanks @johnnie.black!

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
Go to topic listing
×
×
  • Create New...