January 28Jan 28 So here's what happened so far:I was running a mirrored BTRFS pool for a while just fine, until I eventually woke up to a ton of errors being thrown by my 2nd SSD (also 2nd in the screenshot).So I went for a reboot, at which point the 2nd SSD completely disappeared off the list of devices. At this point I assumed it might just be dead as I didn't have much time to spare when it happened - and I was fine running the server with a single cache drive, I do weekly appdata backups anyways.I changed my pool to a single device, the SSD did its rebalancing, and everything continued to run fine ever since.Now about a couple months later, I finally found some time to check the 2nd SSD. I took it out of the system and plugged it into my regular PC for some tests - turns out the SSD is completely fine.I was still weirded out that my unRAID server wouldn't recognize it, even throughout further reboots throughout the last couple months, but I figured I'd just plug it back into the server again, to do some testing there, maybe it's the slot that's broken after all.To my surprise, the SSD was recognized again. This didn't happen after previous reboots, and all I did was run some SMART tests on my PC, nothing was formatted or written to it.Foolishly, despite having trouble before, I figured I'd add the device back to the pool.So I changed the pool devices to 2 slots again, added the drive back into its Cache 2 slot - I was of course greeted with the regular "All existing data on this device will be OVERWRITTEN when array is Started", shown next to the 2nd SSD, which was fine, as it's the main cache drive which was holding the up to date data anyways - and started the array.That's when I was greeted with an error. Unfortunately I can't remember which one it was, but starting the array/pool failed.I then tried removing the second drive again, to start the pool with its singular drive.This gave me the Wrong Pool State - cache - too many wrong or missing devices error.I went for another reboot, thinking it would allow me to change the pool back to a single slot, but it remained set to 2 slots.At this point I realized that my original SSD, the first one, was being recognized as a new device, while my second one wasn't.Now the "All existing data on this device will be OVERWRITTEN when array is Started" message is showing next to my first drive - but this is of course the one with the data I want to keep.Trying to start with just my 1st SSD keeps giving me Wrong Pool State - cache - too many wrong or missing devices - obviously as it's recognized as a new device all of a sudden.Now I'm left in a position where I can only start with 2 devices if I erase my first drive, which I don't want to - and I can't change the pool back to a single device either.I'm sure I messed up along the way, probably should've erased the second SSD before trying to add it back to the pool, but I thought that's what would happen either way, given the GUI message.I'd appreciate any help at this point! davenas-diagnostics-20260128-1408.zip Edited January 28Jan 28 by xDaveN added diagnostics
January 28Jan 28 Community Expert Please post your system diagnostics (<---Click this link to see how) in a reply to my comment. Edited January 28Jan 28 by MowMdown
January 28Jan 28 Author Oh sure, here you go. I've also added the diagnostics to the main post.Also made sure my drive wasn't somehow erased in the process. Seems recoverable so far?root@DaveNAS:~# btrfs filesystem show /dev/nvme1n1p1Label: none uuid: 26ed106b-e8b3-429c-982a-93ad830330c3Total devices 1 FS bytes used 251.41GiBdevid 1 size 931.51GiB used 293.06GiB path /dev/nvme1n1p1devid 2 size 931.51GiB used 310.03GiB path /dev/nvme0n1p1Could removing the pool to have it forget its config, and adding it back with the same name be a solution here?Docker/VM are already disabled.I suppose what's happening is the following. Changing my pool back to 2 devices made my 1st SSD not work anymore, since it was set up for a single device pool before.My 2nd SSD however was still configured for a 2 device pool, hence the cache pool now detecting my 2nd SSD as the one with the correct data, and the 1st one to be erased.Does this sound about right?With that in mind, the solution should be adding a new pool with a singular device, my 1st SSD, which should at least give me a running system?I'd have to make sure the new pool has the old "Cache" name, I could remove the existing one and simply recreate it with 1 slot. I'd then check if my shares still point where I want them to.Does this sound about right? davenas-diagnostics-20260128-1408.zip Edited January 28Jan 28 by xDaveN additional info
January 28Jan 28 Community Expert Solution If the 2nd device is out of sync and the other one was converted to single, it would be better to reimport the pool with just that device first to see if it mounts correctly, but for that you have to disconnect the other device; you can drop it offline by typing:echo 1 > /sys/block/nvmeXn1/device/device/removeReimport the pool with the remaining device, start the array, and post the diags.To reimport the pool:on main click on the first device for that pool and then "remove pool"back on main, create a new pool with the same name and 1 of slotassign the pool device, leave the filesystem set to autostart the array to import the pool
January 28Jan 28 Author Followed your steps - seems like we're good.Keeping Docker/VM turned off for now but what would be the correct approach if I wanted to mirror the pool to the 2nd SSD again?I've dropped it offline but it's otherwise untouched.Appreciate the help so far!davenas-diagnostics-20260128-1705.zip Edited January 28Jan 28 by xDaveN typo
February 5Feb 5 Author @JorgeB sorry for bothering you again, would you mind providing an answer for my question above?
February 5Feb 5 Community Expert Missed your earlier reply.If it works fine with just that device, you can disable array auto-start, reboot to bring the other device back, wipe it withblkdiscard -f /dev/nvmeXn1Then you can re-add it to the pool to create a mirror.
April 2Apr 2 Author @JorgeB I might have to bug you again here I haven't gotten to adding the second device back to the pool yet, but I've noticed that something similar to my initial issue re-appears on every restart.Whenever I restart the system (which I didn't have to do until now, since I fixed things thanks to your help) my cache drive is unmountable and tells me that no file system can be found.What seems to work is your initial suggestion: deleting the pool and recreating it.Now I obviously don't want to do that upon every restart, and it also made me hesitate adding the second drive back to the pool to mirror it.SMART returns no errors, but btrfs check returns this:Opening filesystem to check...No valid Btrfs found on /dev/nvme0n1ERROR: cannot open file systemI can however start the array, and docker/VMs work fine, but it eventually crashes again.Time until crash seems to vary a lot, from hours to weeks.Unfortunately I was writing my log file to the very same drive, effectively making the logs inconclusive as they simply stop at one point.Given I can't check btrfs, I suspect there might be corruption.I'm kindly asking you for some advice on how to proceed. Edited April 2Apr 2 by xDaveN
April 2Apr 2 Author Thank you once again for showing up!I've attached the new diagnostics.Funny enough, upon this restart my pool started just fine - btrfs check still says there's no valid filesystem though, and the system crashed eventually - as expected.davenas-diagnostics-20260402-2015.zip Edited April 2Apr 2 by xDaveN
April 3Apr 3 Community Expert For a check you need to specify the partition /dev/nvme0n1p1, not /dev/nvme0n1Since the pool mounted can't see what the issue is, but btrfs is detecting data corruption, unless these are old errors, I recommend running Memtest.
April 3Apr 3 Author My bad. I should've copied commands instead of typing them, lol.Here's the actual output of the command with the array started in maintenance mode:Opening filesystem to check... Checking filesystem on /dev/nvme0n1p1 UUID: 26ed106b-e8b3-429c-982a-93ad830330c3 [1/8] checking log skipped (none written) [2/8] checking root items [3/8] checking extents [4/8] checking free space tree We have a space info key for a block group that doesn't exist [5/8] checking fs roots [6/8] checking only csums items (without verifying data) [7/8] checking root refs [8/8] checking quota groups skipped (not enabled on this FS) found 260820193280 bytes used, error(s) found total csum bytes: 207499560 total tree bytes: 897204224 total fs tree bytes: 404783104 total extent tree bytes: 209240064 btree space waste bytes: 211968322 file data blocks allocated: 824150417408 referenced 240684015616I'll run memtest next.
April 3Apr 3 Author Memtest passed. 👍🏻I'd like to mention again, that the mounting issue somehow magically resolved itself. At least I haven't run into it again for the last couple of restarts.No clue how, as it was happening with 100% consistency before. The only thing I did differently was having the machine shut off for an extended period instead of immediately restarting - ever since the issue disappeared.Either way, my system will still crash eventually.What stands out is a segfault from qbit, which I'll have to investigate, but ideally I'd still like to make sure btrfs is fine. Edited April 3Apr 3 by xDaveN
April 3Apr 3 Community Expert Next time it fails to mount, please post new diags.Also recommend scrubbing the pool.
April 3Apr 3 Author Will do. Guess I'll have to figure out what causes the crashes currently.Scrub seems fine:Starting scrub on devid 1 scrub done for 26ed106b-e8b3-429c-982a-93ad830330c3 Scrub started: Fri Apr 3 13:53:13 2026 Status: finished Duration: 0:05:11 Total to scrub: 243.75GiB Rate: 802.56MiB/s Error summary: no errors foundI can see the following mentions of corruption in my syslog:Apr 3 12:40:54 DaveNAS emhttpd: mounting /mnt/cache Apr 3 12:40:54 DaveNAS emhttpd: shcmd (55): mkdir -m 0666 -p /mnt/cache Apr 3 12:40:54 DaveNAS emhttpd: cache: btrfs verify devices Apr 3 12:40:54 DaveNAS emhttpd: /sbin/btrfs filesystem show /dev/nvme0n1p1 2>&1 Apr 3 12:40:54 DaveNAS emhttpd: Label: none uuid: 26ed106b-e8b3-429c-982a-93ad830330c3 Apr 3 12:40:54 DaveNAS emhttpd: #011Total devices 1 FS bytes used 242.91GiB Apr 3 12:40:54 DaveNAS emhttpd: #011devid 1 size 931.51GiB used 303.06GiB path /dev/nvme0n1p1 Apr 3 12:40:54 DaveNAS emhttpd: mounting: cache devices: 1 slots: 1 missing: 0 Apr 3 12:40:54 DaveNAS emhttpd: shcmd (56): mount -t btrfs -o noatime,nodiscard,space_cache=v2 /dev/nvme0n1p1 /mnt/cache Apr 3 12:40:54 DaveNAS kernel: BTRFS: device fsid 26ed106b-e8b3-429c-982a-93ad830330c3 devid 1 transid 1380862 /dev/nvme0n1p1 (259:2) scanned by mount (8947) Apr 3 12:40:54 DaveNAS kernel: BTRFS info (device nvme0n1p1): first mount of filesystem 26ed106b-e8b3-429c-982a-93ad830330c3 Apr 3 12:40:54 DaveNAS kernel: BTRFS info (device nvme0n1p1): using crc32c (crc32c-intel) checksum algorithm Apr 3 12:40:54 DaveNAS kernel: BTRFS info (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 15, gen 0 Apr 3 12:40:54 DaveNAS kernel: BTRFS info (device nvme0n1p1): enabling ssd optimizations Apr 3 12:40:54 DaveNAS kernel: BTRFS info (device nvme0n1p1): enabling free space tree Apr 3 12:40:54 DaveNAS emhttpd: shcmd (57): mount -o remount,discard=async /mnt/cache Apr 3 12:40:54 DaveNAS kernel: BTRFS info (device nvme0n1p1 state M): turning on async discard Apr 3 12:40:54 DaveNAS emhttpd: update_pool_cfg: 30 cache 0as well as:Apr 3 12:40:58 DaveNAS emhttpd: Starting services... Apr 3 12:40:58 DaveNAS emhttpd: shcmd (69): /etc/rc.d/rc.samba reload Apr 3 12:40:59 DaveNAS emhttpd: shcmd (73): /etc/rc.d/rc.avahidaemon reload Apr 3 12:40:59 DaveNAS emhttpd: shcmd (76): /usr/local/sbin/mount_image '/mnt/user/system/docker/docker.img' /var/lib/docker 50 Apr 3 12:40:59 DaveNAS rsyslogd: [origin software="rsyslogd" swVersion="8.2102.0" x-pid="10138" x-info="https://www.rsyslog.com"] start Apr 3 12:41:00 DaveNAS kernel: loop2: detected capacity change from 0 to 104857600 Apr 3 12:41:00 DaveNAS kernel: BTRFS: device fsid 116a4be1-79f8-4af3-a2f2-595fd9cbd64a devid 1 transid 1173838 /dev/loop2 (7:2) scanned by mount (10372) Apr 3 12:41:00 DaveNAS kernel: BTRFS info (device loop2): first mount of filesystem 116a4be1-79f8-4af3-a2f2-595fd9cbd64a Apr 3 12:41:00 DaveNAS kernel: BTRFS info (device loop2): using crc32c (crc32c-intel) checksum algorithm Apr 3 12:41:00 DaveNAS kernel: BTRFS info (device loop2): bdev /dev/loop2 errs: wr 0, rd 0, flush 0, corrupt 1, gen 0 Apr 3 12:41:00 DaveNAS kernel: BTRFS info (device loop2): enabling ssd optimizations Apr 3 12:41:00 DaveNAS kernel: BTRFS info (device loop2): turning on async discard Apr 3 12:41:00 DaveNAS kernel: BTRFS info (device loop2): enabling free space tree Apr 3 12:41:00 DaveNAS root: Resize device id 1 (/dev/loop2) from 50.00GiB to max Apr 3 12:41:00 DaveNAS emhttpd: shcmd (78): /etc/rc.d/rc.docker start Apr 3 12:41:00 DaveNAS rc.docker: Starting Docker daemon...I'm not exactly sure how to interpret those or if they're relevant at all, given my crashes seem to follow a qbittorrent segfault.If I understand correctly, this displays the count of previous corruption events? Edited April 3Apr 3 by xDaveN
April 3Apr 3 Author Turns out the latest Jellyfin update leaks RAM, which ends up crashing the server: https://github.com/jellyfin/jellyfin/issues/16549Thanks for all the help @JorgeB - unless my old issue reappears we should be good here. Edited April 3Apr 3 by xDaveN
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.