Starko

A little progress update. Thanks @JorgeB and @trurl and your suggestions. Can confirm that NVME0 was dead, and NVME1 is barely hanging on. My immediate worry at the start was that as no shares were listed in /Shares/, that I could end up losing everything. Even though I could see the data on the main array was still there, I didn't want to risk losing it by re-creating anything. Reading back, if I had said that in the first post, I would have saved myself a good deal of anxiety. Reading this guide helped me understand that removing 1 disk from the cache pool was fine and supported. I connected an old (working) SSD via USB, and tried swapping both nvme0 and nvme1 with it. With nvme1 swapped out, Unassigned Devices helped me see that I could still access the data, even if in Read Only mode, and grabbed the Docker image and /appdata/ and copied via the webUI. With nvme0 swapped out, I could start the array, and while the cache pool would not mount, all the Shares re-appeared! I changed all the shares to Primary location being Array and no secondary (i.e. leaving cache unused) - I'm unsure if this was needed (as the cache pool was not mounted) but it did make me feel a bit more comfortable. As the pool was already unmounted, I figured it would be fine to remove it from the config. Did that, and everything is now running. As far as I can tell, no critical data has been lost. Most crucially, my HomeAssistant set up was saved - I was not looking forward to trying to rebuild that. Fix Common Problems seems happy, and I've added both the Backup/Restore Appdata, and Cleanup Appdata plugins to add a bit more redundancy to this in the future. Phew!

I haven't replaced the failing drive yet, mostly cos I don't have any spare nvmes lying around. Given unraids jbod-like approach, would a different drive of the same or larger size be OK? Edit: Yes it can be. And no worries, appreciate the clarification.

Ok, this is where I'm getting really confused (or am just in denial). If I start the arrays, the nvmes mount in to the cache pool, but when I go to /mnt/cache via the webUI is shows 0 objects, directories, or files. Syslog shows the following Nov 21 16:51:57 macflix emhttpd: shcmd (6951): mount -o remount,discard=async /mnt/cache Nov 21 16:51:57 macflix kernel: BTRFS info (device dm-7 state M): turning on async discard Nov 21 16:51:57 macflix kernel: BTRFS error (device dm-7): parent transid verify failed on logical 1102118912 mirror 1 wanted 3675066 found 3674422 Nov 21 16:51:57 macflix kernel: nvme1n1: I/O Cmd(0x1) @ LBA 2146432, 8 blocks, I/O Error (sct 0x2 / sc 0x80) Nov 21 16:51:57 macflix kernel: critical medium error, dev nvme1n1, sector 2146432 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 2 Nov 21 16:51:57 macflix kernel: BTRFS error (device dm-7): bdev /dev/dm-8 errs: wr 620034, rd 6496, flush 23526, corrupt 553263, gen 6345 Nov 21 16:51:57 macflix kernel: BTRFS error (device dm-7): parent transid verify failed on logical 1101135872 mirror 1 wanted 3675062 found 3674422 Nov 21 16:51:57 macflix kernel: nvme1n1: I/O Cmd(0x1) @ LBA 2144512, 8 blocks, I/O Error (sct 0x2 / sc 0x80) Nov 21 16:51:57 macflix kernel: critical medium error, dev nvme1n1, sector 2144512 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 2 Nov 21 16:51:57 macflix kernel: BTRFS error (device dm-7): bdev /dev/dm-8 errs: wr 620035, rd 6496, flush 23526, corrupt 553263, gen 6345 Nov 21 16:51:57 macflix kernel: BTRFS error (device dm-7): parent transid verify failed on logical 1604272128 mirror 1 wanted 3675053 found 3674475 Nov 21 16:51:57 macflix kernel: nvme1n1: I/O Cmd(0x1) @ LBA 3127200, 8 blocks, I/O Error (sct 0x2 / sc 0x80) Nov 21 16:51:57 macflix kernel: critical medium error, dev nvme1n1, sector 3127200 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 2 Nov 21 16:51:57 macflix kernel: BTRFS error (device dm-7): bdev /dev/dm-8 errs: wr 620036, rd 6496, flush 23526, corrupt 553263, gen 6345 Nov 21 16:51:57 macflix emhttpd: update_pool_cfg: 30 cache 0 Nov 21 16:51:57 macflix kernel: BTRFS error (device dm-7): parent transid verify failed on logical 1604567040 mirror 1 wanted 3675053 found 3674478 Nov 21 16:51:57 macflix kernel: nvme1n1: I/O Cmd(0x1) @ LBA 3127776, 8 blocks, I/O Error (sct 0x2 / sc 0x80) Nov 21 16:51:57 macflix kernel: critical medium error, dev nvme1n1, sector 3127776 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 2 Nov 21 16:51:57 macflix kernel: BTRFS error (device dm-7): bdev /dev/dm-8 errs: wr 620037, rd 6496, flush 23526, corrupt 553263, gen 6345 Nov 21 16:51:57 macflix kernel: BTRFS error (device dm-7): parent transid verify failed on logical 1605877760 mirror 1 wanted 3675053 found 3674475 Nov 21 16:51:57 macflix kernel: nvme1n1: I/O Cmd(0x1) @ LBA 3130336, 8 blocks, I/O Error (sct 0x2 / sc 0x80) Nov 21 16:51:57 macflix kernel: critical medium error, dev nvme1n1, sector 3130336 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 2 Nov 21 16:51:57 macflix kernel: BTRFS error (device dm-7): bdev /dev/dm-8 errs: wr 620038, rd 6496, flush 23526, corrupt 553263, gen 6345 Nov 21 16:51:57 macflix kernel: BTRFS error (device dm-7): parent transid verify failed on logical 1101332480 mirror 1 wanted 3675062 found 3674422 Nov 21 16:51:57 macflix kernel: nvme1n1: I/O Cmd(0x1) @ LBA 2144896, 8 blocks, I/O Error (sct 0x2 / sc 0x80) Nov 21 16:51:57 macflix kernel: critical medium error, dev nvme1n1, sector 2144896 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 2 Nov 21 16:51:57 macflix kernel: BTRFS error (device dm-7): bdev /dev/dm-8 errs: wr 620039, rd 6496, flush 23526, corrupt 553263, gen 6345 Nov 21 16:51:57 macflix emhttpd: shcmd (6952): /usr/sbin/zfs mount -a Nov 21 16:51:57 macflix emhttpd: shcmd (6953): sync Nov 21 16:51:57 macflix kernel: nvme1n1: I/O Cmd(0x1) @ LBA 2134016, 32 blocks, I/O Error (sct 0x2 / sc 0x80) Nov 21 16:51:57 macflix kernel: critical medium error, dev nvme1n1, sector 2134016 op 0x1:(WRITE) flags 0x1800 phys_seg 1 prio class 2 Nov 21 16:51:57 macflix kernel: nvme0n1: I/O Cmd(0x1) @ LBA 2174976, 32 blocks, I/O Error (sct 0x2 / sc 0x80) Nov 21 16:51:57 macflix kernel: critical medium error, dev nvme0n1, sector 2174976 op 0x1:(WRITE) flags 0x1800 phys_seg 1 prio class 2 Nov 21 16:51:57 macflix kernel: nvme0n1: I/O Cmd(0x1) @ LBA 2175040, 32 blocks, I/O Error (sct 0x2 / sc 0x80) Nov 21 16:51:57 macflix kernel: nvme1n1: I/O Cmd(0x1) @ LBA 2134080, 32 blocks, I/O Error (sct 0x2 / sc 0x80) Nov 21 16:51:57 macflix kernel: critical medium error, dev nvme0n1, sector 2175040 op 0x1:(WRITE) flags 0x1800 phys_seg 1 prio class 2 Nov 21 16:51:57 macflix kernel: critical medium error, dev nvme1n1, sector 2134080 op 0x1:(WRITE) flags 0x1800 phys_seg 1 prio class 2 Nov 21 16:51:57 macflix kernel: BTRFS error (device dm-7 state A): Transaction aborted (error -5) Nov 21 16:51:57 macflix kernel: BTRFS: error (device dm-7 state A) in __btrfs_update_delayed_inode:1096: errno=-5 IO failure Nov 21 16:51:57 macflix kernel: BTRFS info (device dm-7 state EA): forced readonly Nov 21 16:51:57 macflix kernel: BTRFS: error (device dm-7 state EA) in __btrfs_run_delayed_items:1174: errno=-5 IO failure Nov 21 16:51:57 macflix kernel: BTRFS warning (device dm-7 state EA): Skipping commit of aborted transaction. Nov 21 16:51:57 macflix kernel: BTRFS: error (device dm-7 state EA) in cleanup_transaction:2027: errno=-5 IO failure Nov 21 16:51:57 macflix emhttpd: shcmd (6954): mkdir /mnt/user0 Nov 21 16:51:57 macflix emhttpd: shcmd (6955): /usr/libexec/unraid/shfs /mnt/user0 -disks 254 -o default_permissions,allow_other,noatime Nov 21 16:51:57 macflix emhttpd: shcmd (6956): mkdir /mnt/userIn terminal, if I cd in to /mnt/cache, and ls - i get "ls: cannot open directory '.': Input/output error". The cache pool is/was RAID1, so I'm guess/hope-ing that if I remove the failed drive (nvme0n1) from the pool (following this guide) then I might be able to get something off to nvme1n1. Does that make any sense?

I don't understand why - but if I try and view /Main/Device?name=cache on the web GUI, the "Cache settings" tab is blank, and I get this error in Firefox and Chrome console Device?name=cache:1649 Error evaluating content in webGui/DeviceInfo.page: round(): Argument #1 ($num) must be of type int|float, string given (anonymous) @ Device?name=cache:1649But if I view /Main/Device?name=cache2 then there is no error, and the tab loads as expected (i.e. Name, Identification, Partition size, Partition format, and then the SMART section)

I've started it running and will report back. Although admittedly, I'm not clear on what a RAM test will say about the NVMe cache drives - would appreciate the lesson. Update: Memtest passed with no errors, two loops, all tests.

Background: Docker locked up late yesterday evening. After restarting Docker, the service failed to start. I tried the temp workaround of increasing the allowable image size to see if this helped, but the image itself almost immediately expanded to fill it. Today, I tried the same approach as there is (was?) at lot of spare capacity in the pool, so upped to the rather silly limit of 150GB. This again was almost immediately filled. Everything else was running fine - but Docker was still being weird. Checking System Logs, I got lots of these: I checked a bunch of threads here, found out about btrfs check status, started in to Maintenance mode, and a very big output of this appeared: Which runs for several hundred lines filled almost entirely with parent transid... Viewing the pool via "Main" - it says there is still 42.8GB free, but if I viewed nvme1n1's Attributes, I got this. Whilst nvme1n0 had lots of "Available spare" space. I found a thread here recommending to Scrub to even it out - my thinking being that one drive had ran out of space, and couldn't balance itself - which I did....and instead now both drives are in this state. If I attempt to run SMART self test, it immediately aborts on both drives. Fix Common Problems now throws an error pointing me here. and I felt that as I've probably bricked it all, I might as well finally start asking for some help(!). Diagnostics below - if any help is possible that would be awesome. macflix-diagnostics-20251120-1943.zip