pr0x1ma

This is what I had in the configuration when it failed: append initrd=/bzroot nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off acpi_enforce_resources=lax

Alright so it's been a while. I ended up buying a different SSD. However, now the new SSD is also giving me BTRFS errors. Like last time, it encountered it while performing a backup. Is this due to my RAM @JorgeB ? Jan 8 04:04:31 Unraid Docker Auto Update: Community Applications Docker Autoupdate finished Jan 8 04:14:18 Unraid emhttpd: read SMART /dev/sdi Jan 8 04:25:54 Unraid emhttpd: read SMART /dev/sdb Jan 8 04:27:22 Unraid kernel: nvme nvme0: I/O tag 112 (c070) opcode 0x2 (I/O Cmd) QID 1 timeout, aborting req_op:READ(0) size:16384 Jan 8 04:27:22 Unraid kernel: nvme nvme0: I/O tag 113 (a071) opcode 0x2 (I/O Cmd) QID 1 timeout, aborting req_op:READ(0) size:532480 Jan 8 04:27:22 Unraid kernel: nvme nvme0: I/O tag 114 (5072) opcode 0x2 (I/O Cmd) QID 1 timeout, aborting req_op:READ(0) size:524288 Jan 8 04:27:22 Unraid kernel: nvme nvme0: I/O tag 115 (e073) opcode 0x2 (I/O Cmd) QID 1 timeout, aborting req_op:READ(0) size:524288 Jan 8 04:27:22 Unraid kernel: nvme nvme0: I/O tag 116 (a074) opcode 0x2 (I/O Cmd) QID 1 timeout, aborting req_op:READ(0) size:524288 Jan 8 04:27:22 Unraid kernel: nvme nvme0: I/O tag 117 (6075) opcode 0x2 (I/O Cmd) QID 1 timeout, aborting req_op:READ(0) size:540672 Jan 8 04:27:22 Unraid kernel: nvme nvme0: I/O tag 75 (904b) opcode 0x0 (I/O Cmd) QID 4 timeout, aborting req_op:FLUSH(2) size:0 Jan 8 04:27:22 Unraid kernel: nvme nvme0: Abort status: 0x0 Jan 8 04:27:22 Unraid kernel: nvme nvme0: Abort status: 0x0 Jan 8 04:27:22 Unraid kernel: nvme nvme0: Abort status: 0x0 Jan 8 04:27:22 Unraid kernel: nvme nvme0: Abort status: 0x0 Jan 8 04:27:22 Unraid kernel: nvme0n1: I/O Cmd(0x2) @ LBA 3385720960, 1024 blocks, I/O Error (sct 0x0 / sc 0x7) Jan 8 04:27:22 Unraid kernel: nvme nvme0: Abort status: 0x0 Jan 8 04:27:22 Unraid kernel: I/O error, dev nvme0n1, sector 3385720960 op 0x0:(READ) flags 0x84700 phys_seg 128 prio class 2 Jan 8 04:27:22 Unraid kernel: nvme0n1: I/O Cmd(0x2) @ LBA 3385721984, 1056 blocks, I/O Error (sct 0x0 / sc 0x7) Jan 8 04:27:22 Unraid kernel: I/O error, dev nvme0n1, sector 3385721984 op 0x0:(READ) flags 0x80700 phys_seg 128 prio class 2 Jan 8 04:27:22 Unraid kernel: nvme nvme0: Abort status: 0x0 Jan 8 04:27:22 Unraid kernel: nvme nvme0: Abort status: 0x0 Jan 8 04:27:25 Unraid kernel: nvme nvme0: I/O tag 847 (734f) opcode 0x2 (I/O Cmd) QID 11 timeout, aborting req_op:READ(0) size:57344 Jan 8 04:27:26 Unraid kernel: nvme nvme0: I/O tag 998 (f3e6) opcode 0x2 (I/O Cmd) QID 5 timeout, aborting req_op:READ(0) size:16384 Jan 8 04:27:32 Unraid kernel: nvme nvme0: I/O tag 13 (a00d) opcode 0x2 (I/O Cmd) QID 9 timeout, aborting req_op:READ(0) size:16384 Jan 8 04:27:32 Unraid kernel: nvme nvme0: I/O tag 666 (629a) opcode 0x2 (I/O Cmd) QID 2 timeout, aborting req_op:READ(0) size:16384 Jan 8 04:27:38 Unraid kernel: nvme nvme0: I/O tag 364 (316c) opcode 0x9 (I/O Cmd) QID 3 timeout, aborting req_op:DISCARD(3) size:7573504 Jan 8 04:27:40 Unraid kernel: nvme nvme0: I/O tag 5 (7005) opcode 0x2 (I/O Cmd) QID 10 timeout, aborting req_op:READ(0) size:16384 Jan 8 04:27:52 Unraid kernel: nvme nvme0: I/O tag 112 (d070) opcode 0x2 (I/O Cmd) QID 1 timeout, aborting req_op:READ(0) size:16384 Jan 8 04:27:52 Unraid kernel: nvme nvme0: I/O tag 113 (a071) opcode 0x2 (I/O Cmd) QID 1 timeout, reset controller Jan 8 04:29:16 Unraid kernel: nvme nvme0: Device not ready; aborting reset, CSTS=0x1 Jan 8 04:29:16 Unraid kernel: nvme0n1: I/O Cmd(0x2) @ LBA 3385717872, 1040 blocks, I/O Error (sct 0x3 / sc 0x71) Jan 8 04:29:16 Unraid kernel: I/O error, dev nvme0n1, sector 3385717872 op 0x0:(READ) flags 0x80700 phys_seg 128 prio class 2 Jan 8 04:29:16 Unraid kernel: nvme0n1: I/O Cmd(0x2) @ LBA 3385718912, 1024 blocks, I/O Error (sct 0x3 / sc 0x71) Jan 8 04:29:16 Unraid kernel: I/O error, dev nvme0n1, sector 3385718912 op 0x0:(READ) flags 0x84700 phys_seg 128 prio class 2 Jan 8 04:29:16 Unraid kernel: nvme0n1: I/O Cmd(0x2) @ LBA 3385719936, 1024 blocks, I/O Error (sct 0x3 / sc 0x71) Jan 8 04:29:16 Unraid kernel: I/O error, dev nvme0n1, sector 3385719936 op 0x0:(READ) flags 0x80700 phys_seg 128 prio class 2 Jan 8 04:29:16 Unraid kernel: nvme0n1: I/O Cmd(0x2) @ LBA 802484856, 32 blocks, I/O Error (sct 0x3 / sc 0x71) Jan 8 04:29:16 Unraid kernel: I/O error, dev nvme0n1, sector 802484856 op 0x0:(READ) flags 0x80700 phys_seg 4 prio class 2 Jan 8 04:29:16 Unraid kernel: nvme0n1: I/O Cmd(0x2) @ LBA 786791032, 112 blocks, I/O Error (sct 0x3 / sc 0x71) Jan 8 04:29:16 Unraid kernel: I/O error, dev nvme0n1, sector 786791032 op 0x0:(READ) flags 0x80700 phys_seg 14 prio class 2 Jan 8 04:29:16 Unraid kernel: nvme0n1: I/O Cmd(0x2) @ LBA 432564896, 64 blocks, I/O Error (sct 0x3 / sc 0x71) Jan 8 04:29:16 Unraid kernel: I/O error, dev nvme0n1, sector 432564896 op 0x0:(READ) flags 0x80700 phys_seg 8 prio class 2 Jan 8 04:29:16 Unraid kernel: nvme nvme0: Abort status: 0x371 Jan 8 04:29:16 Unraid kernel: nvme nvme0: Abort status: 0x371 Jan 8 04:29:16 Unraid kernel: nvme nvme0: Abort status: 0x371 Jan 8 04:29:16 Unraid kernel: nvme nvme0: Abort status: 0x371 Jan 8 04:29:16 Unraid kernel: nvme nvme0: Abort status: 0x371 Jan 8 04:29:16 Unraid kernel: nvme nvme0: Abort status: 0x371 Jan 8 04:29:16 Unraid kernel: nvme nvme0: Abort status: 0x371 Jan 8 04:29:36 Unraid kernel: nvme nvme0: Device not ready; aborting reset, CSTS=0x1 Jan 8 04:29:36 Unraid kernel: nvme nvme0: Disabling device after reset failure: -19

Ok so I set those settings and recreated the docker.img. Everything seems to be working just fine for the last day or so but I don’t know how to recreate the issue so I suppose I’ll just have to wait and see if it does it again. Thanks for your help!

I have the original syslog as well which shows the I/O errors starting right after midnight (see attached).syslog.1.b.txt This is what the extended smart test came back with (spoiler: no errors) Samsung_SSD_990_PRO_2TB_S7KHNU0X633796F-20251012-2120.txt Is it possible that a corrupted docker.img could cause the I/O errors? I'm pretty sure that the "system-previous" log is the log after I rebooted the first time (I rebooted twice) and then ran the appdata backup plugin which produced the original error.

What do I do about this? Is this fixable?

Here’s the diagnostics. One other thing is that when I was on the command line when the docker wasn’t working I was getting an input/output error when running something as simple as “la -la”. unraid-diagnostics-20251012-2042.zip

Something happened last night that basically caused all my containers to fail. Initially, while investigating the problem, I tried to shutdown the array but it was stuck on "stopping". The logs indicated that it was one of my cache disks that couldn't be unmounted. During this process I read somewhere that it could be because there wasn't enough space for docker allocated. So I increased it--although the system was already in a weird state and I probably shouldn't have done that. Well it didn't help. So I ended up shutting the system down and starting it again. When booted I started the array and then ran the CA Appdata Backup plugin because I think that's what originally running when it failed last night. It worked for a few containers then it started to fail and failed consistently from then on. The docker service was then rendered kaput and was in basically the same state as it was earlier. Stopping the array didn't work. Oct 12 12:46:05 Unraid kernel: BTRFS error (device loop2 state A): Transaction aborted (error -5) Oct 12 12:46:05 Unraid kernel: BTRFS: error (device loop2 state A) in do_free_extent_accounting:2983: errno=-5 IO failure Oct 12 12:46:05 Unraid kernel: BTRFS info (device loop2 state EA): forced readonly Oct 12 12:46:05 Unraid kernel: BTRFS error (device loop2 state EA): failed to run delayed ref for logical 1778130944 num_bytes 4096 type 178 action 2 ref_mod 1: -5 Oct 12 12:46:05 Unraid kernel: BTRFS: error (device loop2 state EA) in btrfs_run_delayed_refs:2215: errno=-5 IO failure Oct 12 12:46:05 Unraid kernel: traps: postgres[406881] general protection fault ip:153cbbc32507 sp:7ffe29db9b50 error:0 in libc.so.6[28507,153cbbc32000+165000] Oct 12 12:46:05 Unraid kernel: traps: postgres[439417] general protection fault ip:1543b77b6a1a sp:7ffee4672fa0 error:0 in ld-musl-x86_64.so.1[41a1a,1543b7789000+57000] I think this is a clue but I don't know what it is that I should do from here. Any help is appreciated! [edit] This shows some unallocated space on my cache drive but I don't know if it's related?

Something happened and my unraid server became completely unresponsive. I looked in the syslog and it showed many entries with the following: It's not clear to me though what has caused this error? Something about NMI? syslog-previous

@JorgeB @itimpi Hey folks this worked! Thanks so so much!

@JorgeB I don't see a lost+found folder on either disk... should one be there? For now it seems I should let the rebuild happen for both disk3 and disk5?

@itimpi I restarted the array in normal mode and it's now reconstructing both disk3 and disk5--attached are the diagnostics.

@JorgeB here's the output with -L: Disk 3: Disk 5:

@JorgeB Here are the results after un-assigning disk 3 and 5: Output for Disk 3: Output for Disk 5:

@JorgeB attached are the updated diagnostics.

@JorgeB Quick question -- will it be an issue if I start the array given the parity errors and my data disk that was disabled? I suspect I will be forced to start the rebuild process when starting the array?