January 2, 20251 yr Hello! Thank you for taking the time to read! A couple of weeks ago I noticed my docker containers went offline. After looking everything over I noticed my btrfs NVMe pool named "Nvme_cache" was in a degraded state (2 drives in raid 1). It appeared that one of the drives had failed. I have extra NVMe drives in my server so I stopped the array and removed the one NVMe and replaced it with another. Everything appeared fine and it looked like the btrfs was working on rewriting the mirror to the new drive. A day later it appeared to have finished successfully and everything was working. But less than a week later I'm alerted to my dockers going offline and see the Nvme_cache pool saying "Unmountable: Unsupported or no file system". I have tried some basic trouble shooting but have not been able to get the pool back online. Any ideas would be greatly appreciated! My diagnostics file is attached. anton-diagnostics-20250102-1622.zip
January 3, 20251 yr Author 9 hours ago, JorgeB said: Post the output from btrfs fi show Attached. Thanks!
January 3, 20251 yr Community Expert There's an issue with this pool, it should only be 2 devices but it's showing 3: Probably one of them was used before with the same pool, if you know which device that is (possibly nvme4n1?), disconnect it and post that output again.
January 3, 20251 yr Author 16 minutes ago, JorgeB said: There's an issue with this pool, it should only be 2 devices but it's showing 3: Probably one of them was used before with the same pool, if you know which device that is (possibly nvme4n1?), disconnect it and post that output again. I'll give that a try. I'm physically away from my server until Sunday but will do this and post the results then. Thank you!
January 4, 20251 yr Community Expert You can also drop the device offline with the array stopped by typing: echo 1 > /sys/block/nvmeXn1/device/device/remove But rebooting will bring it back.
January 6, 20251 yr Author On 1/3/2025 at 2:26 PM, JorgeB said: There's an issue with this pool, it should only be 2 devices but it's showing 3: Probably one of them was used before with the same pool, if you know which device that is (possibly nvme4n1?), disconnect it and post that output again. Physically removing the prior NVMe device appears to have fixed the issue. I have also attached the output following the removal of the NVMe device. Thank you for the idea! However, I am still trying to understand this. The device I removed wasn't added to any pool. Would there be a way to remove this device from the btrfs pool history to prevent this from happening again? My hope is to somehow wipe the old NVMe and try to get some use out of its remaining life.
January 6, 20251 yr Community Expert Solution See if the pool mounts normally, now, if yes, just wipe the other device, you can do it with blkdiscard -f /dev/nvmeXn1
January 8, 20251 yr Author On 1/6/2025 at 5:39 AM, JorgeB said: See if the pool mounts normally, now, if yes, just wipe the other device, you can do it with blkdiscard -f /dev/nvmeXn1 This worked perfectly. Thank you so much for the help!
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.