February 28, 20251 yr Hello I have a failing disk and I woud like to replace it with a spare. I stopped the array and selected it in the place of the old disk I'd like to replace and when I started the array it said there was no file system. I used RAIDZ2 for this array. Any help would greatly be appreciated! fileserver-diagnostics-20250227-2047.zip Edited February 28, 20251 yr by impenetrable-aviator2287 Add diags
February 28, 20251 yr Looks like all the pools are mounted, which one is it? Also post the output from zpool import
February 28, 20251 yr Author root@Fileserver:~# zpool import no pools available to import root@Fileserver:~# zpool status pool: docker_vms state: ONLINE config: NAME STATE READ WRITE CKSUM docker_vms ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 nvme1n1p1 ONLINE 0 0 0 nvme2n1p1 ONLINE 0 0 0 errors: No known data errors pool: main_array state: ONLINE status: One or more devices are faulted in response to persistent errors. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Replace the faulted device, or use 'zpool clear' to mark the device repaired. config: NAME STATE READ WRITE CKSUM main_array ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 sdb1 FAULTED 658 578 0 too many errors sdc1 ONLINE 0 0 0 sdd1 FAULTED 101 141 0 too many errors sde1 ONLINE 0 0 0 sdk1 ONLINE 0 0 0 sdl1 ONLINE 0 0 0 sdm1 ONLINE 0 0 0 sdn1 ONLINE 0 0 0 sdo1 ONLINE 0 0 0 sdp1 ONLINE 0 0 0 sdq1 ONLINE 0 0 0 sdr1 ONLINE 0 0 0 cache nvme0n1 ONLINE 0 0 0 errors: No known data errors Looks like 2 I need to swap now.
February 28, 20251 yr Author New diags, should show faulted disks now fileserver-diagnostics-20250228-0655.zip
February 28, 20251 yr So there appears to be two failing disks, are you going to replace both at the same time? You should just need to stop the array, unassign both failing devices and assign the new ones, but I would recommend first offlining sdd at least, it keeps spamming the logs, you can physically disconnect it or just use: echo 1 > /sys/block/sdX/device/delete
February 28, 20251 yr Author So the last time I stopped the array and unassigned the failing disks and assigned new ones the GUI said that there was no file system. And question about offlining the disk. If I do that and I can't get the new disks to rebuild, can I bring it back in to start the array?
February 28, 20251 yr If that happens again post new diags. 12 minutes ago, impenetrable-aviator2287 said: And question about offlining the disk. If I do that and I can't get the new disks to rebuild, can I bring it back in to start the array? Reboot.
February 28, 20251 yr Author Thanks @JorgeB. It's doing it's thing now however I'm getting a ton of "Warning [FILESERVER] - offline uncorrectable" from the other disks now that had no signs of issues.
February 28, 20251 yr NAME STATE READ WRITE CKSUM main_array DEGRADED 0 0 0 raidz2-0 DEGRADED 0 0 0 sdb1 FAULTED 1.09K 1.44K 0 too many errors sdc1 ONLINE 0 0 4.36K replacing-2 DEGRADED 0 0 4.36K 10611756595589239980 UNAVAIL 0 0 0 was /dev/sdd1 sds1 ONLINE 0 0 0 (resilvering) sde1 ONLINE 0 0 4.36K sdk1 ONLINE 0 0 4.36K sdl1 ONLINE 0 0 4.36K sdm1 ONLINE 74 0 4.36K (resilvering) sdn1 ONLINE 0 0 4.36K sdo1 ONLINE 0 0 4.36K sdp1 ONLINE 0 0 4.36K sdq1 ONLINE 0 0 4.36K sdr1 ONLINE 4.42K 0 0 (resilvering) smd and sdr are also failing according to that and SMART, so really bad luck with the disks, or there is something like a power issue causing problems, do you hear the disks clicking, or spinning up and down?
February 28, 20251 yr Author No signs of mechanical issue. Bad luck for sure. The errors stopped about 40 min ago and haven't gone up since. The speed has drasitcally increased on the resilvering as well. I'm going to replace those drives as well just incase. I'm thinking about rebuilding that array to have 2 vdevs of 6 and mirror them.
February 28, 20251 yr When the resilver is done, I would recommend running an extended SMART test on all drives, all the ones showing errors have pending sectors, so they appear to be really failing.
February 28, 20251 yr Author It looks like another drive faulted while rebuilding. Might be hosed. Is there a way to bring one of the faulted drives back online?
February 28, 20251 yr You can try restarting the array and trying to resilver again, but most likely the same thing will happen.
February 28, 20251 yr Author It's not restarting the array. It's saying there is no fs. So was hoping to clear the fault and hopefully rebuild from the two that faulted enough to get by before replacing them as well to rebuild.
February 28, 20251 yr Author cannot import 'main_array': I/O error Destroy and re-create the pool from a backup source. Looking pretty bad lol
February 28, 20251 yr Author Since I really don't have any other options, going to try "pool import -fFX main_array" Nothing left to lose lol
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.