February 4, 20251 yr Hey folks. Looking for some help here. Overnight, all my dockers crashed. I poked around a bit and figured I'd just reboot the server to start. After reboot, my cache wouldn't mount. Getting an error that says "Unmountable: Unsupported or no file system" I had one cache NVMe SSD that seemingly wouldn't generate a SMART report, so I removed it from the cache pool and tried to start the array, but got the same error. Now, when I try to add it back, I get the "All existing data on this device will be OVERWRITTEN when array is Started" warning. I do not remember how to resolve this, but figure I should before I tackle whatever caused my overnight issue. My presumption is that no data has been modified on these drives since they can't mount, so perhaps there is come config thing I can do to re-establish the 4 drive cache. No hardware or other changes in months. I've just been plugging along for a while now. As you can see, the drives here are a gross mishmash of all sorts of stuff. Posting both diag with the array stopped and after trying to start it with 3 of the 4 cache drives listed. __819 is without the array started. __823 is with it started but that 4th cache drive not selected in the pool so I don't accidentally overwrite something if I can avoid it. I would appreciate any help! castle-diagnostics-20250204-0819.zip Edited February 4, 20251 yr by KingArthur
February 4, 20251 yr Author Looks like 2nd diag file didn't post on my original thing. Here is that file, which is diag after starting the array.castle-diagnostics-20250204-0823.zip
February 4, 20251 yr Author root@Castle:~# btrfs fi show Label: none uuid: bbe1c830-2781-400e-8a8e-6dda9da14c36 Total devices 4 FS bytes used 841.49GiB devid 1 size 238.47GiB used 154.00GiB path /dev/sdb1 devid 3 size 223.57GiB used 134.00GiB path /dev/sde1 devid 4 size 953.87GiB used 499.03GiB path /dev/sdc1 devid 5 size 238.47GiB used 150.00GiB path /dev/nvme0n1p1
February 4, 20251 yr Community Expert You should disable Docker and VM Manager in Settings until you get this fixed. They have already created (appdata, domains, system) shares on the array.
February 4, 20251 yr Author Thanks for the advice @trurl. I am hoping that this won't be too awfully messy to resolve, but I'm not the most optimistic of that right now. Edited February 4, 20251 yr by KingArthur
February 4, 20251 yr Author EDIT: Whoops. Definitely not fixed. Getting Fix Common Problems Errors "unable to write to cache" and "unable to write to Docker Image" I was re-reading some other posts I had pulled up earlier where @JorgeB recommended to another person that the way to address the "All existing data on this device will be OVERWRITTEN when array is Started" warning if you sort of mess up a drive config is to un-assign everything, start the array, stop the array, re-assign. I did that. Then, I figured I'd try to start the array, and the cache pool started totally fine? I then went back and re-enabled Docker and VM Manager and the docker service started, as did the cache. But now I have to figure out what broke here. castle-diagnostics-20250204-1005.zip Edited February 4, 20251 yr by KingArthur
February 4, 20251 yr Community Expert Solution Pool is going read-only: Feb 4 10:05:19 Castle kernel: BTRFS error (device sdb1): parent transid verify failed on logical 6665296314368 mirror 1 wanted 2853913 found 2853884 Feb 4 10:05:19 Castle kernel: BTRFS: error (device sdb1: state A) in __btrfs_free_extent:3077: errno=-5 IO failure Feb 4 10:05:19 Castle kernel: BTRFS info (device sdb1: state EA): forced readonly This transid verify failed error is fatal, you can try copying anything important from the pool to the array, then reformat the pool and restore the data. P.S. pool is using the single profile, even for metadata, if that is what you want, make sure at least the metadata is raid1, that is the default for single.
February 4, 20251 yr Author Thanks, @JorgeB. So to be clear, that BTRFS error forces the pool into read-only? Is there a suggestion that this error is the result of a hardware failure with device sdb1, or is this a one-off pool error of some kind? Should I be excluding that device from my new config? As for the best way to approach this, is the process to copy what I want to attempt to save, shut down the array, configure my cache drives the way I want them, reformat, start the array, copy content back to the cache, then go from there? Edited February 4, 20251 yr by KingArthur
February 4, 20251 yr Community Expert 25 minutes ago, KingArthur said: that BTRFS error forces the pool into read-only? Correct. 25 minutes ago, KingArthur said: Is there a suggestion that this error is the result of a hardware failure with device sdb1, or is this a one-off pool error of some kind? Should I be excluding that device from my new config? This error means that some writes were lost, but a device lied about it and maked them as complete, could be a firmware issue, device or controller. 25 minutes ago, KingArthur said: As for the best way to approach this, is the process to copy what I want to attempt to save, shut down the array, configure my cache drives the way I want them, reformat, start the array, copy content back to the cache, then go from there? Basically yes.
February 4, 20251 yr Author I appreciate it, Jorge. I'm starting a new thread to ask for some more advice regarding backups and the least-disruptive way to do this.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.