September 6, 20241 yr Hello mindhive. I may have messed up, but I know to stop and wait when this happens and research and ask. I had two disks go UNAVAIL in my ZFS pool (Ocean is its name) and purchased two to replace them. since they were unavail, I swapped the drives and started the array. It now is erroring out and wanting me to format the entire pool, which I would love not to do, as restoring that much data from backups will be time consuming. I still have both old disks and the two new disks, but the pool recognizes the two new disks and wont accept the old disks. In my research I found i likely messed up by replacing both, even though they are unavail. I should have done one at a time even though the redundancy is capable of dual loss. Please help me restore my pool and fix this. blinky-diagnostics-20240906-1252.zip Edited September 6, 20241 yr by DieFalse
September 6, 20241 yr Community Expert 7.0.0-beta should be able to replace both devices at the same time, post the output from: zpool import
September 6, 20241 yr Author 1 minute ago, JorgeB said: 7.0.0-beta should be able to replace both devices at the same time, post the output from: zpool import root@blinky:~# zpool import pool: puddle id: 11867148198770777971 state: ONLINE status: Some supported features are not enabled on the pool. (Note that they may be intentionally disabled if the 'compatibility' property is set.) action: The pool can be imported using its name or numeric identifier, though some features will not be available without an explicit 'zpool upgrade'. config: puddle ONLINE sdi1 ONLINE sdj1 ONLINE pool: pond id: 8288541694324865914 state: ONLINE status: Some supported features are not enabled on the pool. (Note that they may be intentionally disabled if the 'compatibility' property is set.) action: The pool can be imported using its name or numeric identifier, though some features will not be available without an explicit 'zpool upgrade'. config: pond ONLINE sdc1 ONLINE sdd1 ONLINE sde1 ONLINE sdf1 ONLINE sdg1 ONLINE sdh1 ONLINE pool: lake id: 14754233077931072669 state: ONLINE status: Some supported features are not enabled on the pool. (Note that they may be intentionally disabled if the 'compatibility' property is set.) action: The pool can be imported using its name or numeric identifier, though some features will not be available without an explicit 'zpool upgrade'. config: lake ONLINE raidz1-0 ONLINE sdai1 ONLINE sdaj1 ONLINE sdak1 ONLINE sdal1 ONLINE sdam1 ONLINE sdan1 ONLINE sdao1 ONLINE sdap1 ONLINE sdaq1 ONLINE sdar1 ONLINE pool: ocean id: 7071264282089903179 state: ONLINE status: One or more devices were being resilvered. action: The pool can be imported using its name or numeric identifier. config: ocean ONLINE raidz2-0 ONLINE sdl1 ONLINE sdm1 ONLINE sdn1 ONLINE sdo1 ONLINE sdp1 ONLINE sdq1 ONLINE sdr1 ONLINE sds1 ONLINE sdt1 ONLINE sdu1 ONLINE sdv1 ONLINE sdw1 ONLINE raidz2-1 ONLINE sdk1 ONLINE sdx1 ONLINE sdy1 ONLINE sdz1 ONLINE sdaa1 ONLINE sdab1 ONLINE sdac1 ONLINE sdad1 ONLINE sdas1 ONLINE sdaf1 ONLINE sdag1 ONLINE sdah1 ONLINE
September 6, 20241 yr Community Expert With the array stopped import the pool manually with zpool import ocean then post the output from zpool status
September 6, 20241 yr Author 30 minutes ago, JorgeB said: With the array stopped import the pool manually with zpool import ocean then post the output from zpool status root@blinky:~# zpool import ocean root@blinky:~# zpool status pool: ocean state: ONLINE status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scan: resilver in progress since Tue Aug 27 10:01:17 2024 25.2T / 161T scanned at 2.21G/s, 2.70T / 138T issued 234G resilvered, 1.95% done, no estimated completion time config: NAME STATE READ WRITE CKSUM ocean ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 sdl1 ONLINE 0 0 0 sdm1 ONLINE 0 0 0 sdn1 ONLINE 0 0 0 sdo1 ONLINE 0 0 0 sdp1 ONLINE 0 0 0 sdq1 ONLINE 0 0 0 sdr1 ONLINE 0 0 0 sds1 ONLINE 0 0 0 sdt1 ONLINE 0 0 0 sdu1 ONLINE 0 0 0 sdv1 ONLINE 0 0 0 sdw1 ONLINE 0 0 0 raidz2-1 ONLINE 0 0 0 sdk1 ONLINE 0 0 0 sdx1 ONLINE 0 0 0 sdy1 ONLINE 0 0 0 sdz1 ONLINE 0 0 0 sdaa1 ONLINE 0 0 0 sdab1 ONLINE 42 0 2 (resilvering) sdac1 ONLINE 0 0 0 sdad1 ONLINE 0 0 0 sdas1 ONLINE 0 0 105 (resilvering) sdaf1 ONLINE 0 0 0 sdag1 ONLINE 0 0 0 sdah1 ONLINE 0 0 5 errors: 5251538 data errors, use '-v' for a list
September 6, 20241 yr Community Expert Are these the original disks or the replacements? Pool is showing data errors, meaning data corruption has occurred.
September 6, 20241 yr Author 8 minutes ago, JorgeB said: Are these the original disks or the replacements? Pool is showing data errors, meaning data corruption has occurred. These are the replacements, bad disks were SDAB1 and SDAH
September 6, 20241 yr Author 33 minutes ago, JorgeB said: Post the current diagnostics. Attached. blinky-diagnostics-20240906-1511.zip
September 7, 20241 yr Community Expert Sep 6 15:04:38 blinky kernel: zio pool=ocean vdev=/dev/sdah1 error=61 type=1 offset=11951779422208 size=4096 flags=1605793 Sep 6 15:08:41 blinky kernel: sd 7:0:34:0: [sdah] tag#5600 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=3s Sep 6 15:08:41 blinky kernel: sd 7:0:34:0: [sdah] tag#5600 Sense Key : 0x3 [current] [descriptor] Sep 6 15:08:41 blinky kernel: sd 7:0:34:0: [sdah] tag#5600 ASC=0x11 ASCQ=0x0 Sep 6 15:08:41 blinky kernel: sd 7:0:34:0: [sdah] tag#5600 CDB: opcode=0x88 88 00 00 00 00 05 6f 5e c8 10 00 00 03 40 00 00 Sep 6 15:08:41 blinky kernel: critical medium error, dev sdah, sector 23343319248 op 0x0:(READ) flags 0x0 phys_seg 51 prio class 0 There are issues with sdah, the last disk from that pool, it's logged as a disk problem, but there's no SMART for that disk, check/replace cables and post new diags.
September 8, 20241 yr Author 23 hours ago, JorgeB said: There are issues with sdah, the last disk from that pool, it's logged as a disk problem, but there's no SMART for that disk, check/replace cables and post new diags. I have moved SDAH1 to a completely different bay. blinky-diagnostics-20240907-2316.zip
September 8, 20241 yr Community Expert Looks like that drive is failing: === START OF READ SMART DATA SECTION === SMART Health Status: Failure prediction threshold exceeded [asc=5d, ascq=0] Import the pool again manually and see if there are more errors.
September 8, 20241 yr Author root@blinky:~# diagnostics Starting diagnostics collection... SG_IO: bad/missing sense data, sb[]: 72 05 20 00 00 00 00 34 00 0a 00 00 00 00 00 00 00 00 00 00 01 0a 00 00 00 00 00 00 00 00 00 00 SG_IO: bad/missing sense data, sb[]: 72 05 20 00 00 00 00 34 00 0a 00 00 00 00 00 00 00 00 00 00 01 0a 00 00 00 00 00 00 00 00 00 00 SG_IO: bad/missing sense data, sb[]: 72 05 20 00 00 00 00 34 00 0a 00 00 00 00 00 00 00 00 00 00 01 0a 00 00 00 00 00 00 00 00 00 00 SG_IO: bad/missing sense data, sb[]: 72 05 20 00 00 00 00 34 00 0a 00 00 00 00 00 00 00 00 00 00 01 0a 00 00 00 00 00 00 00 00 00 00 SG_IO: bad/missing sense data, sb[]: 72 05 20 00 00 00 00 34 00 0a 00 00 00 00 00 00 00 00 00 00 01 0a 00 00 00 00 00 00 00 00 00 00 SG_IO: bad/missing sense data, sb[]: 72 05 20 00 00 00 00 34 00 0a 00 00 00 00 00 00 00 00 00 00 01 0a 00 00 00 00 00 00 00 00 00 00 SG_IO: bad/missing sense data, sb[]: 72 05 20 00 00 00 00 34 00 0a 00 00 00 00 00 00 00 00 00 00 01 0a 00 00 00 00 00 00 00 00 00 00 SG_IO: bad/missing sense data, sb[]: 72 05 20 00 00 00 00 34 00 0a 00 00 00 00 00 00 00 00 00 00 01 0a 00 00 00 00 00 00 00 00 00 00 SG_IO: bad/missing sense data, sb[]: 72 05 20 00 00 00 00 34 00 0a 00 00 00 00 00 00 00 00 00 00 01 0a 00 00 00 00 00 00 00 00 00 00 SG_IO: bad/missing sense data, sb[]: 72 05 20 00 00 00 00 34 00 0a 00 00 00 00 00 00 00 00 00 00 01 0a 00 00 00 00 00 00 00 00 00 00 SG_IO: bad/missing sense data, sb[]: 72 05 20 00 00 00 00 1c 02 06 00 00 cf 00 00 00 03 02 00 01 80 0e 00 00 00 00 00 00 00 00 00 00 SG_IO: bad/missing sense data, sb[]: 72 05 20 00 00 00 00 1c 02 06 00 00 cf 00 00 00 03 02 00 01 80 0e 00 00 00 00 00 00 00 00 00 00 SG_IO: bad/missing sense data, sb[]: 72 05 20 00 00 00 00 1c 02 06 00 00 cf 00 00 00 03 02 00 01 80 0e 00 00 00 00 00 00 00 00 00 00 SG_IO: bad/missing sense data, sb[]: 72 05 20 00 00 00 00 1c 02 06 00 00 cf 00 00 00 03 02 00 01 80 0e 00 00 00 00 00 00 00 00 00 00 SG_IO: bad/missing sense data, sb[]: 72 05 20 00 00 00 00 1c 02 06 00 00 cf 00 00 00 03 02 00 01 80 0e 00 00 00 00 00 00 00 00 00 00 SG_IO: bad/missing sense data, sb[]: 72 05 20 00 00 00 00 1c 02 06 00 00 cf 00 00 00 03 02 00 01 80 0e 00 00 00 00 00 00 00 00 00 00 SG_IO: bad/missing sense data, sb[]: 72 05 20 00 00 00 00 1c 02 06 00 00 cf 00 00 00 03 02 00 01 80 0e 00 00 00 00 00 00 00 00 00 00 SG_IO: bad/missing sense data, sb[]: 72 05 20 00 00 00 00 1c 02 06 00 00 cf 00 00 00 03 02 00 01 80 0e 00 00 00 00 00 00 00 00 00 00 SG_IO: bad/missing sense data, sb[]: 72 05 20 00 00 00 00 1c 02 06 00 00 cf 00 00 00 03 02 00 01 80 0e 00 00 00 00 00 00 00 00 00 00 SG_IO: bad/missing sense data, sb[]: 72 05 20 00 00 00 00 1c 02 06 00 00 cf 00 00 00 03 02 00 01 80 0e 00 00 00 00 00 00 00 00 00 00 SG_IO: bad/missing sense data, sb[]: 72 05 20 00 00 00 00 1c 02 06 00 00 cf 00 00 00 03 02 00 01 80 0e 00 00 00 00 00 00 00 00 00 00 SG_IO: bad/missing sense data, sb[]: 70 00 05 00 00 00 00 18 00 00 00 00 20 00 21 c0 00 00 00 00 f8 21 00 00 00 00 00 00 00 00 00 00 SG_IO: bad/missing sense data, sb[]: 70 00 05 00 00 00 00 18 00 00 00 00 20 00 21 c0 00 00 00 00 f8 21 00 00 00 00 00 00 00 00 00 00 SG_IO: bad/missing sense data, sb[]: 70 00 05 00 00 00 00 28 00 00 00 00 20 00 00 00 00 00 00 00 11 05 01 00 00 00 00 00 00 00 00 00 SG_IO: bad/missing sense data, sb[]: 70 00 05 00 00 00 00 28 00 00 00 00 20 00 00 00 00 00 00 00 11 05 01 00 00 00 00 00 00 00 00 00 blinky-diagnostics-20240908-1426.zip
September 8, 20241 yr Community Expert Did you import the pool? Not seeing any disk errors on the diags.
September 12, 20241 yr Author On 9/8/2024 at 6:31 PM, JorgeB said: Did you import the pool? Not seeing any disk errors on the diags. Ok, running zpool import ocean stalls - I gave it 24 hours and the command never finishes. zpool status does the same, I gave it 24 hours also. Trying to start the array stalls on mounting disks. blinky-diagnostics-20240912-0951.zip
September 12, 20241 yr Community Expert Sep 12 09:47:53 blinky kernel: WARNING: Pool 'ocean' has encountered an uncorrectable I/O failure and has been suspended. That pool is possibly toast, since there are 2 two new dives and 1 bad disk on the same raidz2 vdev, best bet it to destroy the pool and recreate with good disks only, then restore from a backup
September 12, 20241 yr Author 29 minutes ago, JorgeB said: Sep 12 09:47:53 blinky kernel: WARNING: Pool 'ocean' has encountered an uncorrectable I/O failure and has been suspended. That pool is possibly toast, since there are 2 two new dives and 1 bad disk on the same raidz2 vdev, best bet it to destroy the pool and recreate with good disks only, then restore from a backup Is it SDAH still the bad one?
September 12, 20241 yr Author 31 minutes ago, JorgeB said: Sep 12 09:47:53 blinky kernel: WARNING: Pool 'ocean' has encountered an uncorrectable I/O failure and has been suspended. That pool is possibly toast, since there are 2 two new dives and 1 bad disk on the same raidz2 vdev, best bet it to destroy the pool and recreate with good disks only, then restore from a backup Also, I still have the old disks. It's over 90TB of data on that pool, so restorning will take quite a while, if there is an alternative.
September 13, 20241 yr Community Expert Solution There are no disk read errors logged at this time, but that "pool suspended" error means that there are errors in more devices than the current pool redundancy can recover from. The old disks have likely been wiped, but if you still have them you can try, not sure if that's possible since the pool will be looking for the new ones, you can disconnect both new disks and connect the old ones, then output of zpool import
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.