I am having an issue with my server. My zpool which is dedicated to running containers (has appdata & system shares) is throwing UDMA CRC error counts. I had this issue a little over a month ago and it corrupted two SSDs. I chalked it up to bad drives, so I replaced them. Now, the server has been up for a month and the issues is happening again.

 

In the syslog, I constantly see:

Dec 14 17:46:10 homegrown kernel: mpt2sas_cm0: log_info(0x31120303): originator(PL), code(0x12), sub_code(0x0303)
Dec 14 17:46:10 homegrown kernel: mpt2sas_cm0: log_info(0x31120303): originator(PL), code(0x12), sub_code(0x0303)
Dec 14 17:46:10 homegrown kernel: mpt2sas_cm0: log_info(0x31120303): originator(PL), code(0x12), sub_code(0x0303)
Dec 14 17:46:10 homegrown kernel: sd 2:0:1:0: [sdd] tag#716 UNKNOWN(0x2003) Result: hostbyte=0x0b driverbyte=DRIVER_OK cmd_age=2s
Dec 14 17:46:10 homegrown kernel: sd 2:0:1:0: [sdd] tag#716 CDB: opcode=0x2a 2a 00 02 77 04 f0 00 01 00 00
Dec 14 17:46:10 homegrown kernel: I/O error, dev sdd, sector 41354480 op 0x1:(WRITE) flags 0x700 phys_seg 32 prio class 2
Dec 14 17:46:10 homegrown kernel: mpt2sas_cm0: log_info(0x31120303): originator(PL), code(0x12), sub_code(0x0303)
Dec 14 17:46:10 homegrown kernel: zio pool=cache-docker vdev=/dev/sdd1 error=5 type=2 offset=21172445184 size=131072 flags=180880
Dec 14 17:46:10 homegrown kernel: mpt2sas_cm0: log_info(0x31120303): originator(PL), code(0x12), sub_code(0x0303)
Dec 14 17:46:10 homegrown kernel: sd 2:0:1:0: [sdd] tag#715 UNKNOWN(0x2003) Result: hostbyte=0x0b driverbyte=DRIVER_OK cmd_age=2s
Dec 14 17:46:10 homegrown kernel: sd 2:0:1:0: [sdd] tag#715 CDB: opcode=0x2a 2a 00 02 77 03 f0 00 01 00 00
Dec 14 17:46:10 homegrown kernel: I/O error, dev sdd, sector 41354224 op 0x1:(WRITE) flags 0x700 phys_seg 32 prio class 2
Dec 14 17:46:10 homegrown kernel: zio pool=cache-docker vdev=/dev/sdd1 error=5 type=2 offset=21172314112 size=131072 flags=180880
Dec 14 17:46:10 homegrown kernel: sd 2:0:1:0: [sdd] tag#714 UNKNOWN(0x2003) Result: hostbyte=0x0b driverbyte=DRIVER_OK cmd_age=2s
Dec 14 17:46:10 homegrown kernel: sd 2:0:1:0: [sdd] tag#714 CDB: opcode=0x2a 2a 00 02 77 02 f0 00 01 00 00
Dec 14 17:46:10 homegrown kernel: I/O error, dev sdd, sector 41353968 op 0x1:(WRITE) flags 0x700 phys_seg 32 prio class 2
Dec 14 17:46:10 homegrown kernel: zio pool=cache-docker vdev=/dev/sdd1 error=5 type=2 offset=21172183040 size=131072 flags=180880
Dec 14 17:46:10 homegrown kernel: sd 2:0:1:0: [sdd] tag#713 UNKNOWN(0x2003) Result: hostbyte=0x0b driverbyte=DRIVER_OK cmd_age=2s
Dec 14 17:46:10 homegrown kernel: sd 2:0:1:0: [sdd] tag#713 CDB: opcode=0x2a 2a 00 02 77 01 f0 00 01 00 00
Dec 14 17:46:10 homegrown kernel: I/O error, dev sdd, sector 41353712 op 0x1:(WRITE) flags 0x700 phys_seg 32 prio class 2
Dec 14 17:46:10 homegrown kernel: zio pool=cache-docker vdev=/dev/sdd1 error=5 type=2 offset=21172051968 size=131072 flags=180880
Dec 14 17:46:10 homegrown kernel: sd 2:0:1:0: [sdd] tag#712 UNKNOWN(0x2003) Result: hostbyte=0x0b driverbyte=DRIVER_OK cmd_age=2s
Dec 14 17:46:10 homegrown kernel: sd 2:0:1:0: [sdd] tag#712 CDB: opcode=0x2a 2a 00 02 77 00 f0 00 01 00 00
Dec 14 17:46:10 homegrown kernel: I/O error, dev sdd, sector 41353456 op 0x1:(WRITE) flags 0x700 phys_seg 32 prio class 2
Dec 14 17:46:10 homegrown kernel: zio pool=cache-docker vdev=/dev/sdd1 error=5 type=2 offset=21171920896 size=131072 flags=180880
Dec 14 17:46:10 homegrown kernel: sd 2:0:1:0: [sdd] tag#711 UNKNOWN(0x2003) Result: hostbyte=0x0b driverbyte=DRIVER_OK cmd_age=3s
Dec 14 17:46:10 homegrown kernel: sd 2:0:1:0: [sdd] tag#711 CDB: opcode=0x2a 2a 00 02 76 f6 f0 00 01 00 00
Dec 14 17:46:10 homegrown kernel: I/O error, dev sdd, sector 41350896 op 0x1:(WRITE) flags 0x700 phys_seg 32 prio class 2
Dec 14 17:46:10 homegrown kernel: zio pool=cache-docker vdev=/dev/sdd1 error=5 type=2 offset=21170610176 size=131072 flags=180880
Dec 14 17:46:10 homegrown kernel: sd 2:0:1:0: [sdd] tag#709 UNKNOWN(0x2003) Result: hostbyte=0x0b driverbyte=DRIVER_OK cmd_age=3s
Dec 14 17:46:10 homegrown kernel: sd 2:0:1:0: [sdd] tag#709 CDB: opcode=0x2a 2a 00 02 76 ff f0 00 01 00 00
Dec 14 17:46:10 homegrown kernel: I/O error, dev sdd, sector 41353200 op 0x1:(WRITE) flags 0x700 phys_seg 32 prio class 2
Dec 14 17:46:10 homegrown kernel: zio pool=cache-docker vdev=/dev/sdd1 error=5 type=2 offset=21171789824 size=131072 flags=180880
Dec 14 17:46:10 homegrown kernel: sd 2:0:1:0: [sdd] tag#708 UNKNOWN(0x2003) Result: hostbyte=0x0b driverbyte=DRIVER_OK cmd_age=3s
Dec 14 17:46:10 homegrown kernel: sd 2:0:1:0: [sdd] tag#708 CDB: opcode=0x2a 2a 00 02 76 fe f0 00 01 00 00
Dec 14 17:46:10 homegrown kernel: I/O error, dev sdd, sector 41352944 op 0x1:(WRITE) flags 0x700 phys_seg 32 prio class 2
Dec 14 17:46:10 homegrown kernel: zio pool=cache-docker vdev=/dev/sdd1 error=5 type=2 offset=21171658752 size=131072 flags=180880
Dec 14 17:46:10 homegrown kernel: sd 2:0:1:0: [sdd] tag#707 UNKNOWN(0x2003) Result: hostbyte=0x0b driverbyte=DRIVER_OK cmd_age=3s
Dec 14 17:46:10 homegrown kernel: sd 2:0:1:0: [sdd] tag#707 CDB: opcode=0x2a 2a 00 02 76 fd f0 00 01 00 00
Dec 14 17:46:10 homegrown kernel: I/O error, dev sdd, sector 41352688 op 0x1:(WRITE) flags 0x700 phys_seg 32 prio class 2
Dec 14 17:46:10 homegrown kernel: zio pool=cache-docker vdev=/dev/sdd1 error=5 type=2 offset=21171527680 size=131072 flags=180880
Dec 14 17:46:10 homegrown kernel: sd 2:0:1:0: Power-on or device reset occurred

 

I did some research on the forums and think the issue is either HBA / SAS expander firmware, cables or perhaps too many drives on a power supply leg. Can someone help me diagnose?

 

Thank you!

homegrown-diagnostics-20241214-1741.zip

Edited December 14, 20241 yr by ramair02

Replacing the cables and/or making sure they're properly seated is the first thing to check.

 

 

UDMA CRC errors are typically a bad SATA cable.

11 hours ago, ramair02 said:

perhaps too many drives on a power supply leg

This is unlikely to be an issue if using the plugs built into the PSU cables.   It can be an issue if using splitters to get extra connections (especially if splitting a SATA connector more than 2 way).

Cables are seated properly, but I just ordered another set of cabledeconn mini sas breakout cables to see if that's the issue.

 

I read in another thread that there are some problem firmwares for the LSi HBA cards. How can I check the firmware and does anyone know which firmware versions have issues?

 

I also use an HP SAS expander along with the LSi card.

 

It's odd since the issue happened initially after months of being stable. Then I replaced the SSDs and things have been stable for more than a month -- until now.

13 hours ago, ramair02 said:

I read in another thread that there are some problem firmwares for the LSi HBA cards.

There are some older firmware versions that can cause them, but the syslog already rotated in the diags posted, reboot and post new ones.