June 28, 20242 yr I originally had several disk "fail" the same day and I didn't buy it. I moved disks around and then in the old slots the moved disks were working fine in the new location but disks moved to the old locations now "failed" Replaced the backplan and things seemed good for a while, and then something similar happened. Replaced backplanE AGAIN and .. again seemed ok for a few weeks and now .. single disk is broken .. parity is broken ( by me) and even I even attempt to build parity it starts out at 1.6GB/sec and then effectively crashes the array. Completely unresponsive even via keyboard/monitor. Reboot makes it start again. more or less fine but no way to build party and at least one supposedly failed disk. diag attached. vortex-diagnostics-20240627-2024.zip
June 28, 20242 yr Community Expert Jun 27 20:23:32 vortex kernel: md: recovery thread: multiple disk errors, sector=0 Jun 27 20:23:32 vortex kernel: md: recovery thread: multiple disk errors, sector=8 Jun 27 20:23:32 vortex kernel: md: recovery thread: multiple disk errors, sector=16 Jun 27 20:23:32 vortex kernel: md: recovery thread: multiple disk errors, sector=24 Jun 27 20:23:32 vortex kernel: md: recovery thread: multiple disk errors, sector=32 Hmm, this is quite strange, md errors without associated disk/controller errors, don't remember ever seeing that before, not sure what to make of it.
July 1, 20242 yr Author Damn.. I dont know what to do.. feels.. half unusable..Cant make it reach parity.. beyond one disabled disk.. i dont know what if any other disks are bad..
July 1, 20242 yr Community Expert Can you connect some disks bypassing the backplane and retest? Even if you cannot connect them all.
July 2, 20242 yr Author After a bunch of restarts and research of everything i discovered, i found this thread. The most telling part was I too had 3-4 disks with read errors on the same exact sectors. Which is very similar..many disks at the same time showing read errors. One of the disks that was originally disabled was mountable and everything was there no issues. I upgraded the the controllar card firmware and right now its stable at 0 errors I still cannot explain how this all of a sudden happened. Generally I'm not a fan of upgrading things that were not previously broken. Either wayl.. its ok so far. Adding the last disk back into the array and building parity. That'll be the last check Edited July 2, 20242 yr by leon
July 3, 20242 yr Author Next day Jul 3 15:07:43 vortex kernel: mpt3sas_cm0: log_info(0x31110d00): originator(PL), code(0x11), sub_code(0x0d00) Jul 3 15:07:43 vortex kernel: sd 9:0:16:0: device_block, handle(0x0024) Jul 3 15:07:46 vortex kernel: sd 9:0:16:0: device_unblock and setting to running, handle(0x0024) Jul 3 15:07:46 vortex kernel: md: disk26 read error, sector=28182497896 Jul 3 15:07:46 vortex kernel: md: disk26 read error, sector=28182497904 Jul 3 15:07:46 vortex kernel: md: disk26 read error, sector=28182497912 Jul 3 15:07:46 vortex kernel: md: disk26 read error, sector=28182497920 Jul 3 15:07:46 vortex kernel: md: disk26 read error, sector=28182497928 Jul 3 15:07:46 vortex kernel: md: disk26 read error, sector=28182497936 Jul 3 15:07:46 vortex kernel: md: disk26 read error, sector=28182497944 Jul 3 15:07:46 vortex kernel: md: disk26 read error, sector=28182497952 Jul 3 15:07:46 vortex kernel: md: disk26 read error, sector=28182497960 Jul 3 15:07:46 vortex kernel: md: disk26 read error, sector=28182497968 Jul 3 15:07:46 vortex kernel: md: disk26 read error, sector=28182497976 Jul 3 15:07:46 vortex kernel: md: disk26 read error, sector=28182497984 Jul 3 15:07:46 vortex kernel: md: disk26 read error, sector=28182497992 Jul 3 15:07:46 vortex kernel: md: disk26 read error, sector=28182498000 Jul 3 15:07:46 vortex kernel: md: disk26 read error, sector=28182498008 Jul 3 15:07:46 vortex kernel: md: disk26 read error, sector=28182498016 Jul 3 15:07:46 vortex kernel: md: disk26 read error, sector=28182498024 Jul 3 15:07:46 vortex kernel: md: disk26 read error, sector=28182498032 Jul 3 15:07:46 vortex kernel: md: disk26 read error, sector=28182498040 Jul 3 15:07:46 vortex kernel: md: disk26 read error, sector=28182498048 Jul 3 15:07:46 vortex kernel: md: disk26 read error, sector=28182498056 Jul 3 15:07:46 vortex kernel: md: disk26 read error, sector=28182498064 Jul 3 15:07:46 vortex kernel: md: disk26 read error, sector=28182498072 Jul 3 15:07:46 vortex kernel: md: disk26 read error, sector=28182498080 Jul 3 15:07:46 vortex kernel: md: disk26 read error, sector=28182498088 Jul 3 15:07:46 vortex kernel: md: disk26 read error, sector=28182498096 Jul 3 15:07:46 vortex kernel: md: disk26 read error, sector=28182498104 Jul 3 15:07:46 vortex kernel: md: disk26 read error, sector=28182498112 Jul 3 15:07:46 vortex kernel: md: disk26 read error, sector=28182498120 Jul 3 15:07:46 vortex kernel: md: disk26 read error, sector=28182498128 Jul 3 15:07:46 vortex kernel: md: disk26 read error, sector=28182498136 Jul 3 15:07:46 vortex kernel: md: disk26 read error, sector=28182498144 Jul 3 15:07:46 vortex kernel: sd 9:0:16:0: [sdo] Synchronizing SCSI cache Jul 3 15:07:46 vortex kernel: sd 9:0:16:0: [sdo] Synchronize Cache(10) failed: Result: hostbyte=0x01 driverbyte=DRIVER_OK Jul 3 15:07:46 vortex kernel: mpt3sas_cm0: mpt3sas_transport_port_remove: removed: sas_addr(0x300062b206514053) Jul 3 15:07:46 vortex kernel: mpt3sas_cm0: removing handle(0x0024), sas_addr(0x300062b206514053) Jul 3 15:07:46 vortex kernel: mpt3sas_cm0: enclosure logical id(0x500062b206514040), slot(9) Jul 3 15:07:46 vortex kernel: mpt3sas_cm0: enclosure level(0x0000), connector name( ) Jul 3 15:08:05 vortex kernel: mpt3sas_cm0: handle(0x24) sas_address(0x300062b206514053) port_type(0x1) Jul 3 15:08:05 vortex kernel: scsi 9:0:17:0: Direct-Access ATA ST18000NM000J-2T SN04 PQ: 0 ANSI: 6 Jul 3 15:08:05 vortex kernel: scsi 9:0:17:0: SATA: handle(0x0024), sas_addr(0x300062b206514053), phy(19), device_name(0x0000000000000000) Jul 3 15:08:05 vortex kernel: scsi 9:0:17:0: enclosure logical id (0x500062b206514040), slot(9) Jul 3 15:08:05 vortex kernel: scsi 9:0:17:0: enclosure level(0x0000), connector name( ) Jul 3 15:08:05 vortex kernel: scsi 9:0:17:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y) Jul 3 15:08:05 vortex kernel: scsi 9:0:17:0: qdepth(32), tagged(1), scsi_level(7), cmd_que(1) Jul 3 15:08:05 vortex kernel: sd 9:0:17:0: Attached scsi generic sg14 type 0 Jul 3 15:08:05 vortex kernel: end_device-9:17: add: handle(0x0024), sas_addr(0x300062b206514053) Jul 3 15:08:05 vortex kernel: sd 9:0:17:0: Power-on or device reset occurred Jul 3 15:08:05 vortex kernel: sd 9:0:17:0: [sdah] 35156656128 512-byte logical blocks: (18.0 TB/16.4 TiB) Jul 3 15:08:05 vortex kernel: sd 9:0:17:0: [sdah] 4096-byte physical blocks Jul 3 15:08:05 vortex kernel: sd 9:0:17:0: [sdah] Write Protect is off Jul 3 15:08:05 vortex kernel: sd 9:0:17:0: [sdah] Mode Sense: 9b 00 10 08 Jul 3 15:08:05 vortex kernel: sd 9:0:17:0: [sdah] Write cache: enabled, read cache: enabled, supports DPO and FUA Jul 3 15:08:05 vortex kernel: sdah: sdah1 Jul 3 15:08:05 vortex kernel: sd 9:0:17:0: [sdah] Attached SCSI disk Jul 3 15:08:06 vortex unassigned.devices: Disk with ID 'ST18000NM000J-2TV103_ZR5FA1YG ()' is not set to auto mount. Jul 3 15:08:07 vortex emhttpd: error: hotplug_devices, 1706: No such file or directory (2): tagged device ST18000NM000J-2TV103_ZR5FA1YG was (sdo) is now (sdah) Jul 3 15:08:07 vortex emhttpd: read SMART /dev/sdah Jul 3 15:08:07 vortex kernel: emhttpd[10455]: segfault at 67c ip 000056534c9cc75f sp 00007ffec82f8a90 error 4 in emhttpd[56534c9b7000+24000] likely on CPU 4 (core 4, socket 0) Jul 3 15:08:07 vortex kernel: Code: c4 36 01 00 48 89 45 f8 48 8d 05 f9 23 01 00 48 89 45 f0 e9 79 01 00 00 8b 45 ec 89 c7 e8 1a 88 ff ff 48 89 45 d8 48 8b 45 d8 <8b> 80 7c 06 00 00 85 c0 0f 94 c0 0f b6 c0 89 45 d4 48 8b 45 e0 48
July 4, 20242 yr Author And worse.. I dont know what this is. Seem HW related.. but I dont know what Jul 3 19:03:51 vortex kernel: sd 9:0:17:0: device_block, handle(0x0024) Jul 3 19:03:52 vortex kernel: sd 9:0:7:0: device_block, handle(0x0020) Jul 3 19:03:52 vortex kernel: sd 9:0:15:0: device_block, handle(0x0022) Jul 3 19:03:54 vortex kernel: sd 9:0:17:0: device_unblock and setting to running, handle(0x0024) Jul 3 19:03:54 vortex kernel: sd 9:0:17:0: [sdah] Synchronizing SCSI cache Jul 3 19:03:54 vortex kernel: sd 9:0:17:0: [sdah] Synchronize Cache(10) failed: Result: hostbyte=0x01 driverbyte=DRIVER_OK Jul 3 19:03:54 vortex kernel: mpt3sas_cm0: mpt3sas_transport_port_remove: removed: sas_addr(0x300062b206514053) Jul 3 19:03:54 vortex kernel: mpt3sas_cm0: removing handle(0x0024), sas_addr(0x300062b206514053) Jul 3 19:03:54 vortex kernel: mpt3sas_cm0: enclosure logical id(0x500062b206514040), slot(9) Jul 3 19:03:54 vortex kernel: mpt3sas_cm0: enclosure level(0x0000), connector name( ) Jul 3 19:03:55 vortex kernel: sd 9:0:7:0: device_unblock and setting to running, handle(0x0020) Jul 3 19:03:55 vortex kernel: sd 9:0:15:0: device_unblock and setting to running, handle(0x0022) Jul 3 19:03:55 vortex kernel: sd 9:0:7:0: [sdk] Synchronizing SCSI cache Jul 3 19:03:55 vortex kernel: sd 9:0:7:0: [sdk] Synchronize Cache(10) failed: Result: hostbyte=0x01 driverbyte=DRIVER_OK Jul 3 19:03:55 vortex kernel: mpt3sas_cm0: mpt3sas_transport_port_remove: removed: sas_addr(0x300062b206514050) Jul 3 19:03:55 vortex kernel: mpt3sas_cm0: removing handle(0x0020), sas_addr(0x300062b206514050) Jul 3 19:03:55 vortex kernel: mpt3sas_cm0: enclosure logical id(0x500062b206514040), slot(11) Jul 3 19:03:55 vortex kernel: mpt3sas_cm0: enclosure level(0x0000), connector name( ) Jul 3 19:03:55 vortex kernel: sd 9:0:15:0: [sdm] Synchronizing SCSI cache Jul 3 19:03:55 vortex kernel: sd 9:0:15:0: [sdm] Synchronize Cache(10) failed: Result: hostbyte=0x01 driverbyte=DRIVER_OK Jul 3 19:03:55 vortex kernel: mpt3sas_cm0: mpt3sas_transport_port_remove: removed: sas_addr(0x300062b206514051) Jul 3 19:03:55 vortex kernel: mpt3sas_cm0: removing handle(0x0022), sas_addr(0x300062b206514051) Jul 3 19:03:55 vortex kernel: mpt3sas_cm0: enclosure logical id(0x500062b206514040), slot(10) Jul 3 19:03:55 vortex kernel: mpt3sas_cm0: enclosure level(0x0000), connector name( ) Jul 3 19:04:12 vortex kernel: mpt3sas_cm0: handle(0x24) sas_address(0x300062b206514053) port_type(0x1) Jul 3 19:04:12 vortex kernel: scsi 9:0:18:0: Direct-Access ATA ST18000NM000J-2T SN04 PQ: 0 ANSI: 6 Jul 3 19:04:12 vortex kernel: scsi 9:0:18:0: SATA: handle(0x0024), sas_addr(0x300062b206514053), phy(19), device_name(0x0000000000000000) Jul 3 19:04:12 vortex kernel: scsi 9:0:18:0: enclosure logical id (0x500062b206514040), slot(9) Jul 3 19:04:12 vortex kernel: scsi 9:0:18:0: enclosure level(0x0000), connector name( ) Jul 3 19:04:12 vortex kernel: scsi 9:0:18:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y) Jul 3 19:04:12 vortex kernel: scsi 9:0:18:0: qdepth(32), tagged(1), scsi_level(7), cmd_que(1) Jul 3 19:04:12 vortex kernel: sd 9:0:18:0: Attached scsi generic sg10 type 0 Jul 3 19:04:12 vortex kernel: end_device-9:18: add: handle(0x0024), sas_addr(0x300062b206514053) Jul 3 19:04:12 vortex kernel: sd 9:0:18:0: Power-on or device reset occurred Jul 3 19:04:12 vortex kernel: sd 9:0:18:0: [sdah] 35156656128 512-byte logical blocks: (18.0 TB/16.4 TiB) Jul 3 19:04:12 vortex kernel: sd 9:0:18:0: [sdah] 4096-byte physical blocks Jul 3 19:04:12 vortex kernel: sd 9:0:18:0: [sdah] Write Protect is off Jul 3 19:04:12 vortex kernel: sd 9:0:18:0: [sdah] Mode Sense: 9b 00 10 08 Jul 3 19:04:12 vortex kernel: sd 9:0:18:0: [sdah] Write cache: enabled, read cache: enabled, supports DPO and FUA Jul 3 19:04:12 vortex kernel: sdah: sdah1 Jul 3 19:04:12 vortex kernel: sd 9:0:18:0: [sdah] Attached SCSI disk Jul 3 19:04:12 vortex kernel: mpt3sas_cm0: handle(0x20) sas_address(0x300062b206514050) port_type(0x1) Jul 3 19:04:13 vortex kernel: scsi 9:0:19:0: Direct-Access ATA ST18000NM000J-2T SN02 PQ: 0 ANSI: 6 Jul 3 19:04:13 vortex kernel: scsi 9:0:19:0: SATA: handle(0x0020), sas_addr(0x300062b206514050), phy(16), device_name(0x0000000000000000) Jul 3 19:04:13 vortex kernel: scsi 9:0:19:0: enclosure logical id (0x500062b206514040), slot(11) Jul 3 19:04:13 vortex kernel: scsi 9:0:19:0: enclosure level(0x0000), connector name( ) Jul 3 19:04:13 vortex kernel: scsi 9:0:19:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y) Jul 3 19:04:13 vortex kernel: scsi 9:0:19:0: qdepth(32), tagged(1), scsi_level(7), cmd_que(1) Jul 3 19:04:13 vortex kernel: sd 9:0:19:0: Attached scsi generic sg12 type 0 Jul 3 19:04:13 vortex kernel: end_device-9:19: add: handle(0x0020), sas_addr(0x300062b206514050) Jul 3 19:04:13 vortex kernel: sd 9:0:19:0: Power-on or device reset occurred Jul 3 19:04:13 vortex kernel: sd 9:0:19:0: [sdai] 35156656128 512-byte logical blocks: (18.0 TB/16.4 TiB) Jul 3 19:04:13 vortex kernel: sd 9:0:19:0: [sdai] 4096-byte physical blocks Jul 3 19:04:13 vortex kernel: sd 9:0:19:0: [sdai] Write Protect is off Jul 3 19:04:13 vortex kernel: sd 9:0:19:0: [sdai] Mode Sense: 9b 00 10 08 Jul 3 19:04:13 vortex kernel: sd 9:0:19:0: [sdai] Write cache: enabled, read cache: enabled, supports DPO and FUA Jul 3 19:04:13 vortex kernel: sdai: sdai1 Jul 3 19:04:13 vortex kernel: sd 9:0:19:0: [sdai] Attached SCSI disk Jul 3 19:04:13 vortex unassigned.devices: Disk with ID 'ST18000NM000J-2TV103_ZR5FA1YG (sdah)' is not set to auto mount. Jul 3 19:04:13 vortex kernel: mpt3sas_cm0: handle(0x22) sas_address(0x300062b206514051) port_type(0x1) Jul 3 19:04:14 vortex kernel: scsi 9:0:20:0: Direct-Access ATA ST18000NM000J-2T SN02 PQ: 0 ANSI: 6 Jul 3 19:04:14 vortex kernel: scsi 9:0:20:0: SATA: handle(0x0022), sas_addr(0x300062b206514051), phy(17), device_name(0x0000000000000000) Jul 3 19:04:14 vortex kernel: scsi 9:0:20:0: enclosure logical id (0x500062b206514040), slot(10) Jul 3 19:04:14 vortex kernel: scsi 9:0:20:0: enclosure level(0x0000), connector name( ) Jul 3 19:04:14 vortex kernel: scsi 9:0:20:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y) Jul 3 19:04:14 vortex kernel: scsi 9:0:20:0: qdepth(32), tagged(1), scsi_level(7), cmd_que(1) Jul 3 19:04:14 vortex kernel: sd 9:0:20:0: Attached scsi generic sg14 type 0 Jul 3 19:04:14 vortex kernel: end_device-9:20: add: handle(0x0022), sas_addr(0x300062b206514051) Jul 3 19:04:14 vortex kernel: sd 9:0:20:0: Power-on or device reset occurred Jul 3 19:04:14 vortex kernel: sd 9:0:20:0: [sdaj] 35156656128 512-byte logical blocks: (18.0 TB/16.4 TiB) Jul 3 19:04:14 vortex kernel: sd 9:0:20:0: [sdaj] 4096-byte physical blocks Jul 3 19:04:14 vortex kernel: sd 9:0:20:0: [sdaj] Write Protect is off Jul 3 19:04:14 vortex kernel: sd 9:0:20:0: [sdaj] Mode Sense: 9b 00 10 08 Jul 3 19:04:14 vortex kernel: sd 9:0:20:0: [sdaj] Write cache: enabled, read cache: enabled, supports DPO and FUA Jul 3 19:04:14 vortex kernel: sdaj: sdaj1 Jul 3 19:04:14 vortex kernel: sd 9:0:20:0: [sdaj] Attached SCSI disk Jul 3 19:04:14 vortex unassigned.devices: Disk with ID 'ST18000NM000J-2TV103_ZR58490G ()' is not set to auto mount. Jul 3 19:04:15 vortex unassigned.devices: Disk with ID 'ST18000NM000J-2TV103_ZR5CTC3K ()' is not set to auto mount.
July 4, 20242 yr Community Expert 5 hours ago, leon said: Seem HW related Yep, seems like a power/connection issue to me.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.