- Minor
I upgraded from 6.11.5 to 6.12.10.
After the upgrade my array would not start up and got stuck at mounting disks. I had a quick look at syslog and it was getting some errors with some of my SSD's in my BTRFS Pools. I panicked as i didn't want to lose any data in one of the pools so rolled back to 6.11.5. After the roll back everything was working fine again with the array starting and I could access the data on both BTRFS pools. I quickly moved all the data off the pools and onto my array without any issues.
I had 2 btrfs pools working on 6.11.5. One with 8 NVME drives in raid 0 and another with 8 enterprise SAS SSDs in raid 10.
Once I had the data off the pools I decided to delete them and upgrade to 6.12.10 and then try to set them up again. Upgrade went fine and I could see the SSD's showing up in unassigned devices without issues. I setup the 8 x NVME SSD in a ZFS raid 0 pool and this started up and formatted with out any issues.
I then setup the pool of 8 SAS SSD's with ZFS in Raidz with one pool of 8 drives. Array started up ok but when I ticked the box to start the formatting of the drives in the new pool that's when I started getting the same errors again in SYSLOG. Some of the errors below, there is a lot more you can view in the diagnostics that ill attach.
kernel: I/O error, dev sdk, sector 6232642688 op 0x3:(DISCARD) flags 0x4000 phys_seg 1 prio class 2
kernel: mpt2sas_cm0: _scsih_sas_broadcast_primitive_event: enter: phy number(4), width(1)
kernel: sd 9:0:0:0: device_block, handle(0x0009)
kernel: sd 9:0:1:0: device_block, handle(0x000a)
kernel: sd 9:0:2:0: device_block, handle(0x000b)
kernel: sd 9:0:3:0: device_block, handle(0x000c)
kernel: sd 9:0:4:0: device_block, handle(0x000d)
kernel: sd 9:0:5:0: device_block, handle(0x000e)
kernel: sd 9:0:6:0: device_block, handle(0x000f)
kernel: sd 9:0:7:0: device_block, handle(0x0010)
kernel: sd 9:0:0:0: device_unblock and setting to running, handle(0x0009)
kernel: sd 9:0:1:0: device_unblock and setting to running, handle(0x000a)
kernel: sd 9:0:2:0: device_unblock and setting to running, handle(0x000b)
kernel: sd 9:0:3:0: device_unblock and setting to running, handle(0x000c)
kernel: sd 9:0:4:0: device_unblock and setting to running, handle(0x000d)
kernel: sd 9:0:5:0: device_unblock and setting to running, handle(0x000e)
kernel: sd 9:0:6:0: device_unblock and setting to running, handle(0x000f)
kernel: sd 9:0:7:0: device_unblock and setting to running, handle(0x0010)
kernel: sd 9:0:4:0: Power-on or device reset occurred
kernel: sd 9:0:4:0: attempting task abort!scmd(0x00000000c8003593), outstanding for 7078 ms & timeout 7000 ms
kernel: sd 9:0:4:0: [sdk] tag#7077 CDB: opcode=0x12 12 00 00 00 fe 00
kernel: scsi target9:0:4: handle(0x000d), sas_address(0x5001173100a25b6d), phy(4)
kernel: scsi target9:0:4: enclosure logical id(0x50014380328b2188), slot(7)
kernel: sd 9:0:4:0: task abort: SUCCESS scmd(0x00000000c8003593)
kernel: sd 9:0:4:0: attempting device reset! scmd(0x00000000ac175889)
kernel: sd 9:0:4:0: [sdk] tag#7040 CDB: opcode=0x42 42 00 00 00 00 00 00 00 18 00
kernel: scsi target9:0:4: handle(0x000d), sas_address(0x5001173100a25b6d), phy(4)
kernel: scsi target9:0:4: enclosure logical id(0x50014380328b2188), slot(7)
kernel: sd 9:0:4:0: device reset: FAILED scmd(0x00000000ac175889)
kernel: scsi target9:0:4: enclosure logical id(0x50014380328b2188), slot(7)
kernel: sd 9:0:4:0: task abort: SUCCESS scmd(0x00000000c1d71bc4)
kernel: sd 9:0:4:0: attempting task abort!scmd(0x0000000023403f0c), outstanding for 30185 ms & timeout 30000 ms
kernel: sd 9:0:4:0: [sdk] tag#7094 CDB: opcode=0x42 42 00 00 00 00 00 00 00 18 00
kernel: scsi target9:0:4: handle(0x000d), sas_address(0x5001173100a25b6d), phy(4)
I couldn't cancel the format process and trying to force a reboot wouldn't work so i had to eventually pull the power. I thought perhaps it was just a ZFS thing so after a reboot I attempted the same again but this time using BTRFS and again ran into the same errors. Powered of again as its seemed to get stuck in a loop or not be making a lot of progress.
I can format the drives ok using unassigned devices but anytime I add them into a pool and start the format process that way I get the same errors showing up a lot.
I'm not sure what's changed between 6.11.5 and 6.12.10 for these errors to occur? The disks and pool were working fine previously with no smart errors. I'm getting the same errors on all disks so far so i don't think its a disk failure. Is there some sort of conflict or compatibility issues in 6.12.10 with my onboard LSI SAS2308 controller setup in IT mode? Or some compatibility with the SAS SSD's them selves?
The SSD's are SanDisk DOPM3840S5xnNMRI. They come out of a 3par storage array and i had to re-format and change the sector size from 520bytes to 4096bytes to get them visible to unraid. They have been rock solid with no issues or errors at all in my 6.11.5 btrfs pool.
I'm now trying again with a ZFS pool and just letting it run over night as it does seem to slowly be making its way through the 8 disks one by one even with all the errors. Not sure how it will turn out. If it fails I may try rolling back to 6.11.5 again and attempt to setup another BTRFS pool and see if the same errors come up again in syslog as i don't recall the same issues occurring when I first set this up a few years back.
Update: I didn't post this until this morning as the diagnostics took forever to run while the pool was formatting. Pool is now up after it took most of the night.. I'm unsure if its safe to use the pool given all the errors. I've downloaded diagnostics again so will upload both here.
Any suggestions?
thanks.
prometheus-diagnostics-20240612-0946.zip prometheus-diagnostics-20240611-2341.zip