Background: Docker locked up late yesterday evening. After restarting Docker, the service failed to start. I tried the temp workaround of increasing the allowable image size to see if this helped, but the image itself almost immediately expanded to fill it. Today, I tried the same approach as there is (was?) at lot of spare capacity in the pool, so upped to the rather silly limit of 150GB. This again was almost immediately filled.

Everything else was running fine - but Docker was still being weird. Checking System Logs, I got lots of these:

nvme1n1: I/O Cmd(0x1) @ LBA 8534880, 8 blocks, I/O Error (sct 0x2 / sc 0x80)

critical medium error, dev nvme1n1, sector 8534880 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 2

and

BTRFS error (device dm-7): bdev /dev/dm-8 errs: wr 620034, rd 6496, flush 23526, corrupt 553263, gen 6345

I checked a bunch of threads here, found out about btrfs check status, started in to Maintenance mode, and a very big output of this appeared:

checksum verify failed on 8145951784960 wanted 0x8f07b4c6 found 0xe39dd5c6
checksum verify failed on 8145951801344 wanted 0x1e6e87b6 found 0x9076c198
parent transid verify failed on 4373004288 wanted 3675045 found 3667748
checksum verify failed on 8145951817728 wanted 0x47806902 found 0x94b7afd3
parent transid verify failed on 1101987840 wanted 3675066 found 3647318
parent transid verify failed on 1102053376 wanted 3675066 found 3674422

Which runs for several hundred lines filled almost entirely with parent transid...

Viewing the pool via "Main" - it says there is still 42.8GB free, but if I viewed nvme1n1's Attributes, I got this.

- Critical warning 0x09

- Temperature 35 Celsius

- Available spare 0%

- Available spare threshold 10%

- Percentage used 39%

- Data units read 233,782,560 [119 TB]

- Data units written 642,810,296 [329 TB]

- Host read commands 442,066,239

- Host write commands 2,725,490,982

- Controller busy time 16,235

- Power cycles 64

- Power on hours 14,558 (1y, 7m, 27d, 14h)

- Unsafe shutdowns 37

- Media and data integrity errors 410

- Error information log entries 549

- Warning comp. temperature time 0

- Critical comp. temperature time 0

- Temperature sensor 1 35 Celsius

- Temperature sensor 2 47 Celsius

- SSD endurance remaining 61 %

Whilst nvme1n0 had lots of "Available spare" space. I found a thread here recommending to Scrub to even it out - my thinking being that one drive had ran out of space, and couldn't balance itself - which I did....and instead now both drives are in this state.

If I attempt to run SMART self test, it immediately aborts on both drives.

Fix Common Problems now throws an error pointing me here. and I felt that as I've probably bricked it all, I might as well finally start asking for some help(!).

Diagnostics below - if any help is possible that would be awesome.

macflix-diagnostics-20251120-1943.zip

Edited November 22, 2025Nov 22 by Starko
Solved.

Memtest is on the boot menu.

50 minutes ago, trurl said:

Memtest is on the boot menu.

I've started it running and will report back.

Although admittedly, I'm not clear on what a RAM test will say about the NVMe cache drives - would appreciate the lesson.

Update: Memtest passed with no errors, two loops, all tests.

Edited November 20, 2025Nov 20 by Starko

I don't understand why - but if I try and view /Main/Device?name=cache on the web GUI, the "Cache settings" tab is blank, and I get this error in Firefox and Chrome console

Device?name=cache:1649 Error evaluating content in webGui/DeviceInfo.page: round(): Argument #1 ($num) must be of type int|float, string given
(anonymous) @ Device?name=cache:1649

But if I view /Main/Device?name=cache2 then there is no error, and the tab loads as expected (i.e. Name, Identification, Partition size, Partition format, and then the SMART section)

Edited November 20, 2025Nov 20 by Starko

Nov 20 18:17:45 macflix kernel: BTRFS info (device dm-7): bdev /dev/dm-7 errs: wr 0, rd 23536, flush 0, corrupt 0, gen 0

Nov 20 18:17:45 macflix kernel: BTRFS info (device dm-7): bdev /dev/dm-8 errs: wr 620033, rd 6496, flush 23526, corrupt 553263, gen 6345

There are errors on both pool devices; the second one dropped offline at some point in the past, and the 1st pool device has failed

SMART overall-health self-assessment test result: FAILED!

- available spare has fallen below threshold

- media has been placed in read only mode

I would recommend backing up what you can and reformatting the pool, then checking here for better pool monitoring:

https://forums.unraid.net/topic/46802-faq-for-unraid-v6/page/2/#findComment-700582

8 hours ago, JorgeB said:

I would recommend backing up what you can and reformatting the pool, then checking here for better pool monitoring:

https://forums.unraid.net/topic/46802-faq-for-unraid-v6/page/2/#findComment-700582

Ok, this is where I'm getting really confused (or am just in denial).

If I start the arrays, the nvmes mount in to the cache pool, but when I go to /mnt/cache via the webUI is shows 0 objects, directories, or files.

Syslog shows the following

Nov 21 16:51:57 macflix emhttpd: shcmd (6951): mount -o remount,discard=async /mnt/cache
Nov 21 16:51:57 macflix kernel: BTRFS info (device dm-7 state M): turning on async discard
Nov 21 16:51:57 macflix kernel: BTRFS error (device dm-7): parent transid verify failed on logical 1102118912 mirror 1 wanted 3675066 found 3674422
Nov 21 16:51:57 macflix kernel: nvme1n1: I/O Cmd(0x1) @ LBA 2146432, 8 blocks, I/O Error (sct 0x2 / sc 0x80) 
Nov 21 16:51:57 macflix kernel: critical medium error, dev nvme1n1, sector 2146432 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 2
Nov 21 16:51:57 macflix kernel: BTRFS error (device dm-7): bdev /dev/dm-8 errs: wr 620034, rd 6496, flush 23526, corrupt 553263, gen 6345
Nov 21 16:51:57 macflix kernel: BTRFS error (device dm-7): parent transid verify failed on logical 1101135872 mirror 1 wanted 3675062 found 3674422
Nov 21 16:51:57 macflix kernel: nvme1n1: I/O Cmd(0x1) @ LBA 2144512, 8 blocks, I/O Error (sct 0x2 / sc 0x80) 
Nov 21 16:51:57 macflix kernel: critical medium error, dev nvme1n1, sector 2144512 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 2
Nov 21 16:51:57 macflix kernel: BTRFS error (device dm-7): bdev /dev/dm-8 errs: wr 620035, rd 6496, flush 23526, corrupt 553263, gen 6345
Nov 21 16:51:57 macflix kernel: BTRFS error (device dm-7): parent transid verify failed on logical 1604272128 mirror 1 wanted 3675053 found 3674475
Nov 21 16:51:57 macflix kernel: nvme1n1: I/O Cmd(0x1) @ LBA 3127200, 8 blocks, I/O Error (sct 0x2 / sc 0x80) 
Nov 21 16:51:57 macflix kernel: critical medium error, dev nvme1n1, sector 3127200 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 2
Nov 21 16:51:57 macflix kernel: BTRFS error (device dm-7): bdev /dev/dm-8 errs: wr 620036, rd 6496, flush 23526, corrupt 553263, gen 6345
Nov 21 16:51:57 macflix emhttpd: update_pool_cfg: 30 cache 0
Nov 21 16:51:57 macflix kernel: BTRFS error (device dm-7): parent transid verify failed on logical 1604567040 mirror 1 wanted 3675053 found 3674478
Nov 21 16:51:57 macflix kernel: nvme1n1: I/O Cmd(0x1) @ LBA 3127776, 8 blocks, I/O Error (sct 0x2 / sc 0x80) 
Nov 21 16:51:57 macflix kernel: critical medium error, dev nvme1n1, sector 3127776 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 2
Nov 21 16:51:57 macflix kernel: BTRFS error (device dm-7): bdev /dev/dm-8 errs: wr 620037, rd 6496, flush 23526, corrupt 553263, gen 6345
Nov 21 16:51:57 macflix kernel: BTRFS error (device dm-7): parent transid verify failed on logical 1605877760 mirror 1 wanted 3675053 found 3674475
Nov 21 16:51:57 macflix kernel: nvme1n1: I/O Cmd(0x1) @ LBA 3130336, 8 blocks, I/O Error (sct 0x2 / sc 0x80) 
Nov 21 16:51:57 macflix kernel: critical medium error, dev nvme1n1, sector 3130336 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 2
Nov 21 16:51:57 macflix kernel: BTRFS error (device dm-7): bdev /dev/dm-8 errs: wr 620038, rd 6496, flush 23526, corrupt 553263, gen 6345
Nov 21 16:51:57 macflix kernel: BTRFS error (device dm-7): parent transid verify failed on logical 1101332480 mirror 1 wanted 3675062 found 3674422
Nov 21 16:51:57 macflix kernel: nvme1n1: I/O Cmd(0x1) @ LBA 2144896, 8 blocks, I/O Error (sct 0x2 / sc 0x80) 
Nov 21 16:51:57 macflix kernel: critical medium error, dev nvme1n1, sector 2144896 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 2
Nov 21 16:51:57 macflix kernel: BTRFS error (device dm-7): bdev /dev/dm-8 errs: wr 620039, rd 6496, flush 23526, corrupt 553263, gen 6345
Nov 21 16:51:57 macflix emhttpd: shcmd (6952): /usr/sbin/zfs mount -a
Nov 21 16:51:57 macflix emhttpd: shcmd (6953): sync
Nov 21 16:51:57 macflix kernel: nvme1n1: I/O Cmd(0x1) @ LBA 2134016, 32 blocks, I/O Error (sct 0x2 / sc 0x80) 
Nov 21 16:51:57 macflix kernel: critical medium error, dev nvme1n1, sector 2134016 op 0x1:(WRITE) flags 0x1800 phys_seg 1 prio class 2
Nov 21 16:51:57 macflix kernel: nvme0n1: I/O Cmd(0x1) @ LBA 2174976, 32 blocks, I/O Error (sct 0x2 / sc 0x80) 
Nov 21 16:51:57 macflix kernel: critical medium error, dev nvme0n1, sector 2174976 op 0x1:(WRITE) flags 0x1800 phys_seg 1 prio class 2
Nov 21 16:51:57 macflix kernel: nvme0n1: I/O Cmd(0x1) @ LBA 2175040, 32 blocks, I/O Error (sct 0x2 / sc 0x80) 
Nov 21 16:51:57 macflix kernel: nvme1n1: I/O Cmd(0x1) @ LBA 2134080, 32 blocks, I/O Error (sct 0x2 / sc 0x80) 
Nov 21 16:51:57 macflix kernel: critical medium error, dev nvme0n1, sector 2175040 op 0x1:(WRITE) flags 0x1800 phys_seg 1 prio class 2
Nov 21 16:51:57 macflix kernel: critical medium error, dev nvme1n1, sector 2134080 op 0x1:(WRITE) flags 0x1800 phys_seg 1 prio class 2
Nov 21 16:51:57 macflix kernel: BTRFS error (device dm-7 state A): Transaction aborted (error -5)
Nov 21 16:51:57 macflix kernel: BTRFS: error (device dm-7 state A) in __btrfs_update_delayed_inode:1096: errno=-5 IO failure
Nov 21 16:51:57 macflix kernel: BTRFS info (device dm-7 state EA): forced readonly
Nov 21 16:51:57 macflix kernel: BTRFS: error (device dm-7 state EA) in __btrfs_run_delayed_items:1174: errno=-5 IO failure
Nov 21 16:51:57 macflix kernel: BTRFS warning (device dm-7 state EA): Skipping commit of aborted transaction.
Nov 21 16:51:57 macflix kernel: BTRFS: error (device dm-7 state EA) in cleanup_transaction:2027: errno=-5 IO failure
Nov 21 16:51:57 macflix emhttpd: shcmd (6954): mkdir /mnt/user0
Nov 21 16:51:57 macflix emhttpd: shcmd (6955): /usr/libexec/unraid/shfs /mnt/user0 -disks 254 -o default_permissions,allow_other,noatime 
Nov 21 16:51:57 macflix emhttpd: shcmd (6956): mkdir /mnt/user

In terminal, if I cd in to /mnt/cache, and ls - i get "ls: cannot open directory '.': Input/output error".

The cache pool is/was RAID1, so I'm guess/hope-ing that if I remove the failed drive (nvme0n1) from the pool (following this guide) then I might be able to get something off to nvme1n1.

Does that make any sense?

21 minutes ago, Starko said:

Syslog shows the following

Did you replace the failing device?

9 hours ago, JorgeB said:

and the 1st pool device has failed

When I said to recreate the pool, I meant using a new device in place of that one, but rereading now, I may not have been very clear.

I haven't replaced the failing drive yet, mostly cos I don't have any spare nvmes lying around. Given unraids jbod-like approach, would a different drive of the same or larger size be OK? Edit: Yes it can be.

And no worries, appreciate the clarification.

Edited November 21, 2025Nov 21 by Starko

A larger device will work with both btrfs and zfs, in a mirror, only the capacity of the smaller one can be used, since the failed device has gone read-only, no way to create a pool with it.

A little progress update.

Thanks @JorgeB and @trurl and your suggestions.

Can confirm that NVME0 was dead, and NVME1 is barely hanging on.

My immediate worry at the start was that as no shares were listed in /Shares/, that I could end up losing everything. Even though I could see the data on the main array was still there, I didn't want to risk losing it by re-creating anything. Reading back, if I had said that in the first post, I would have saved myself a good deal of anxiety.

Reading this guide helped me understand that removing 1 disk from the cache pool was fine and supported. I connected an old (working) SSD via USB, and tried swapping both nvme0 and nvme1 with it. With nvme1 swapped out, Unassigned Devices helped me see that I could still access the data, even if in Read Only mode, and grabbed the Docker image and /appdata/ and copied via the webUI. With nvme0 swapped out, I could start the array, and while the cache pool would not mount, all the Shares re-appeared!

I changed all the shares to Primary location being Array and no secondary (i.e. leaving cache unused) - I'm unsure if this was needed (as the cache pool was not mounted) but it did make me feel a bit more comfortable.

As the pool was already unmounted, I figured it would be fine to remove it from the config. Did that, and everything is now running. As far as I can tell, no critical data has been lost. Most crucially, my HomeAssistant set up was saved - I was not looking forward to trying to rebuild that.

Fix Common Problems seems happy, and I've added both the Backup/Restore Appdata, and Cleanup Appdata plugins to add a bit more redundancy to this in the future.

Phew!

Edited November 22, 2025Nov 22 by Starko

I didn't notice initially, but, yes, bit devices have failed and need to be replaced.

On 11/20/2025 at 4:22 PM, Starko said:

I'm not clear on what a RAM test will say about the NVMe cache drives - would appreciate the lesson.

You must never attempt to run any computer unless RAM is working perfectly. Everything goes through RAM. The OS and other executable code, your DATA. EVERYTHING!. The CPU can't do anything with anything until it is loaded into RAM.

On 11/23/2025 at 4:58 AM, JorgeB said:

both devices have failed and need to be replaced.

FTFY