czonin

First, just wanted to say thanks for the help! I ended up finding unbalanced after my last post and got that process started. I've now removed the drive from my array. The drive is covered under warranty and is going to be replaced. Is it worth creating a new config with that drive removed and rebuilding parity now or should I wait until the replacement comes in?

The RAM I switched to was new so hoping it's good. I'll still test soon though. Poor time to have RAM and a drive failing with current prices. Haven't dealt with a drive failure yet on Unraid so need to figure that out. Edit: I only have 5.5 TB on the drive that's failing. Is there an easy way to move data off that drive so I don't need to replace right away?

I was started to suspect RAM. I was able to swap to a different set to avoid downtime. I scrubbed the NVME drive afterwards and removed the corrupted files. Everything seems good so far, but I did see these errors earlier today that I hadn't seen before. Could they be related or is this a separate issue and that drive is starting to fail? ST16000NM001G-2KK103_ZL28FRCX-20260131-0959.txt

I've also been seeing these errors, but they seem to be related to my NVME pool which is separate from the cache pool where appdata is. Jan 29 08:00:02 CZ-Server kernel: BTRFS warning (device nvme0n1p1): csum failed root 5 ino 5538472 off 1076047872 csum 0x64475f37 expected csum 0x122384bd mirror 1 Jan 29 08:00:02 CZ-Server kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 215, gen 0 Jan 29 08:00:02 CZ-Server shfs: copy_file: /mnt/fast-cache/data/media/movies/Curious George 2 Follow That Monkey! (2009)/Curious George 2 Follow That Monkey! (2009) {imdb-tt1350484} [WEBDL-1080p][EAC3 5.1][h264]-ABM.mkv /mnt/disk3/data/media/movies/Curious George 2 Follow That Monkey! (2009)/Curious George 2 Follow That Monkey! (2009) {imdb-tt1350484} [WEBDL-1080p][EAC3 5.1][h264]-ABM.mkv.partial (5) Input/output error Jan 29 08:00:02 CZ-Server kernel: BTRFS warning (device nvme0n1p1): csum failed root 5 ino 5538472 off 1076047872 csum 0x64475f37 expected csum 0x122384bd mirror 1 Jan 29 08:00:02 CZ-Server kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 216, gen 0 Jan 29 08:00:02 CZ-Server kernel: BTRFS warning (device nvme0n1p1): csum failed root 5 ino 5538472 off 1076047872 csum 0x64475f37 expected csum 0x122384bd mirror 1 Jan 29 08:00:02 CZ-Server kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 217, gen 0 Jan 29 08:00:02 CZ-Server kernel: BTRFS warning (device nvme0n1p1): csum failed root 5 ino 5538472 off 1076047872 csum 0x64475f37 expected csum 0x122384bd mirror 1 Jan 29 08:00:02 CZ-Server kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 218, gen 0 Jan 29 08:00:04 CZ-Server kernel: BTRFS warning (device nvme0n1p1): csum failed root 5 ino 5538512 off 4347924480 csum 0x870a3ef2 expected csum 0xfe2a9576 mirror 1 Jan 29 08:00:04 CZ-Server kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 219, gen 0 Jan 29 08:00:04 CZ-Server shfs: copy_file: /mnt/fast-cache/data/media/movies/Curious George Cape Ahoy (2021)/Curious George Cape Ahoy (2021) {imdb-tt15437042} [WEBDL-1080p][EAC3 5.1][x264]-NTb.mkv /mnt/disk3/data/media/movies/Curious George Cape Ahoy (2021)/Curious George Cape Ahoy (2021) {imdb-tt15437042} [WEBDL-1080p][EAC3 5.1][x264]-NTb.mkv.partial (5) Input/output error Jan 29 08:00:04 CZ-Server kernel: BTRFS warning (device nvme0n1p1): csum failed root 5 ino 5538512 off 4347924480 csum 0x870a3ef2 expected csum 0xfe2a9576 mirror 1 Jan 29 08:00:04 CZ-Server kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 220, gen 0 Jan 29 08:00:04 CZ-Server kernel: BTRFS warning (device nvme0n1p1): csum failed root 5 ino 5538512 off 4347924480 csum 0x870a3ef2 expected csum 0xfe2a9576 mirror 1 Jan 29 08:00:04 CZ-Server kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 221, gen 0 Jan 29 08:00:04 CZ-Server kernel: BTRFS warning (device nvme0n1p1): csum failed root 5 ino 5538512 off 4347924480 csum 0x870a3ef2 expected csum 0xfe2a9576 mirror 1 Jan 29 08:00:04 CZ-Server kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 222, gen 0 Jan 29 08:00:04 CZ-Server kernel: BTRFS warning (device nvme0n1p1): csum failed root 5 ino 5538512 off 4347924480 csum 0x870a3ef2 expected csum 0xfe2a9576 mirror 1 Jan 29 08:00:04 CZ-Server kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 223, gen 0 Jan 29 08:00:08 CZ-Server kernel: BTRFS warning (device nvme0n1p1): csum failed root 5 ino 5529102 off 5407424512 csum 0x8cefd35a expected csum 0xc602a72d mirror 1 Jan 29 08:00:08 CZ-Server kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 224, gen 0 Jan 29 08:00:08 CZ-Server shfs: copy_file: /mnt/fast-cache/data/.Recycle.Bin/media/movies/Greenland 2 Migration (2026)/Greenland 2 Migration (2026) {imdb-tt14850054} [WEBDL-2160p][HDR10Plus][EAC3 Atmos 5.1][HEVC]-QHstudIo.mp4 /mnt/disk3/data/.Recycle.Bin/media/movies/Greenland 2 Migration (2026)/Greenland 2 Migration (2026) {imdb-tt14850054} [WEBDL-2160p][HDR10Plus][EAC3 Atmos 5.1][HEVC]-QHstudIo.mp4.partial (5) Input/output error Jan 29 08:00:08 CZ-Server kernel: BTRFS warning (device nvme0n1p1): csum failed root 5 ino 5529102 off 5407424512 csum 0x8cefd35a expected csum 0xc602a72d mirror 1 Jan 29 08:00:08 CZ-Server kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 225, gen 0 Jan 29 08:00:08 CZ-Server kernel: BTRFS warning (device nvme0n1p1): csum failed root 5 ino 5529102 off 5407424512 csum 0x8cefd35a expected csum 0xc602a72d mirror 1 Jan 29 08:00:08 CZ-Server kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 226, gen 0 Jan 29 08:00:08 CZ-Server kernel: BTRFS warning (device nvme0n1p1): csum failed root 5 ino 5529102 off 5407424512 csum 0x8cefd35a expected csum 0xc602a72d mirror 1 Jan 29 08:00:08 CZ-Server kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 227, gen 0 Jan 29 08:01:00 CZ-Server move: mover: finished Jan 29 10:02:42 CZ-Server emhttpd: spinning down /dev/sdh Jan 29 16:00:01 CZ-Server move: mover: started Jan 29 16:00:02 CZ-Server kernel: BTRFS warning (device nvme0n1p1): csum failed root 5 ino 5538472 off 1076047872 csum 0x64475f37 expected csum 0x122384bd mirror 1 Jan 29 16:00:02 CZ-Server kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 228, gen 0 Jan 29 16:00:02 CZ-Server shfs: copy_file: /mnt/fast-cache/data/media/movies/Curious George 2 Follow That Monkey! (2009)/Curious George 2 Follow That Monkey! (2009) {imdb-tt1350484} [WEBDL-1080p][EAC3 5.1][h264]-ABM.mkv /mnt/disk3/data/media/movies/Curious George 2 Follow That Monkey! (2009)/Curious George 2 Follow That Monkey! (2009) {imdb-tt1350484} [WEBDL-1080p][EAC3 5.1][h264]-ABM.mkv.partial (5) Input/output error Jan 29 16:00:02 CZ-Server kernel: BTRFS warning (device nvme0n1p1): csum failed root 5 ino 5538472 off 1076047872 csum 0x64475f37 expected csum 0x122384bd mirror 1 Jan 29 16:00:02 CZ-Server kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 229, gen 0 Jan 29 16:00:02 CZ-Server kernel: BTRFS warning (device nvme0n1p1): csum failed root 5 ino 5538472 off 1076047872 csum 0x64475f37 expected csum 0x122384bd mirror 1 Jan 29 16:00:02 CZ-Server kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 230, gen 0 Jan 29 16:00:02 CZ-Server kernel: BTRFS warning (device nvme0n1p1): csum failed root 5 ino 5538472 off 1076047872 csum 0x64475f37 expected csum 0x122384bd mirror 1 Jan 29 16:00:02 CZ-Server kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 231, gen 0 Jan 29 16:00:03 CZ-Server kernel: BTRFS warning (device nvme0n1p1): csum failed root 5 ino 5538512 off 4347924480 csum 0x870a3ef2 expected csum 0xfe2a9576 mirror 1 Jan 29 16:00:03 CZ-Server kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 232, gen 0 Jan 29 16:00:03 CZ-Server shfs: copy_file: /mnt/fast-cache/data/media/movies/Curious George Cape Ahoy (2021)/Curious George Cape Ahoy (2021) {imdb-tt15437042} [WEBDL-1080p][EAC3 5.1][x264]-NTb.mkv /mnt/disk3/data/media/movies/Curious George Cape Ahoy (2021)/Curious George Cape Ahoy (2021) {imdb-tt15437042} [WEBDL-1080p][EAC3 5.1][x264]-NTb.mkv.partial (5) Input/output error Jan 29 16:00:03 CZ-Server kernel: BTRFS warning (device nvme0n1p1): csum failed root 5 ino 5538512 off 4347924480 csum 0x870a3ef2 expected csum 0xfe2a9576 mirror 1 Jan 29 16:00:03 CZ-Server kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 233, gen 0 Jan 29 16:00:03 CZ-Server kernel: BTRFS warning (device nvme0n1p1): csum failed root 5 ino 5538512 off 4347924480 csum 0x870a3ef2 expected csum 0xfe2a9576 mirror 1 Jan 29 16:00:03 CZ-Server kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 234, gen 0 Jan 29 16:00:03 CZ-Server kernel: BTRFS warning (device nvme0n1p1): csum failed root 5 ino 5538512 off 4347924480 csum 0x870a3ef2 expected csum 0xfe2a9576 mirror 1 Jan 29 16:00:03 CZ-Server kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 235, gen 0 Jan 29 16:00:03 CZ-Server kernel: BTRFS warning (device nvme0n1p1): csum failed root 5 ino 5538512 off 4347924480 csum 0x870a3ef2 expected csum 0xfe2a9576 mirror 1 Jan 29 16:00:03 CZ-Server kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 236, gen 0 Jan 29 16:00:05 CZ-Server kernel: BTRFS warning (device nvme0n1p1): csum failed root 5 ino 5529102 off 5407424512 csum 0x8cefd35a expected csum 0xc602a72d mirror 1 Jan 29 16:00:05 CZ-Server kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 237, gen 0 Jan 29 16:00:05 CZ-Server shfs: copy_file: /mnt/fast-cache/data/.Recycle.Bin/media/movies/Greenland 2 Migration (2026)/Greenland 2 Migration (2026) {imdb-tt14850054} [WEBDL-2160p][HDR10Plus][EAC3 Atmos 5.1][HEVC]-QHstudIo.mp4 /mnt/disk3/data/.Recycle.Bin/media/movies/Greenland 2 Migration (2026)/Greenland 2 Migration (2026) {imdb-tt14850054} [WEBDL-2160p][HDR10Plus][EAC3 Atmos 5.1][HEVC]-QHstudIo.mp4.partial (5) Input/output error

cz-server-diagnostics-20260129-1732.zip

Hey everyone, Over the past couple weeks I've been having docker issues. It started where all of my containers weren't showing in my dashboard. Going into the docker tab showed the message that docker failed to start. I stopped docker, deleted vdisk, and tried to start docker but still saw the same message. I rebooted my server and strangely all of my containers were back and running. I still went in and deleted the vdisk and added back all of my containers as I suspected that was the issue. Everything seemed fine up until last night when the same thing happened. I copied some logs that I'll post below. Is one of my cache drives failing where my appdata lives? I did a quick SMART test on both but it didn't show any issues. Unraid 7.2.3 System specs Jan 24 01:11:46 CZ-Server kernel: I/O error, dev loop2, sector 55725536 op 0x1:(WRITE) flags 0x10100 phys_seg 1 prio class 2 Jan 24 01:11:46 CZ-Server kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 7, rd 0, flush 0, corrupt 0, gen 0 Jan 24 01:11:46 CZ-Server kernel: I/O error, dev loop2, sector 57822688 op 0x1:(WRITE) flags 0x10100 phys_seg 1 prio class 2 Jan 24 01:11:46 CZ-Server kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 8, rd 0, flush 0, corrupt 0, gen 0 Jan 24 01:11:46 CZ-Server kernel: I/O error, dev loop2, sector 55725568 op 0x1:(WRITE) flags 0x10100 phys_seg 1 prio class 2 Jan 24 01:11:46 CZ-Server kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 9, rd 0, flush 0, corrupt 0, gen 0 Jan 24 01:11:46 CZ-Server kernel: I/O error, dev loop2, sector 57822720 op 0x1:(WRITE) flags 0x10100 phys_seg 1 prio class 2 Jan 24 01:11:46 CZ-Server kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 10, rd 0, flush 0, corrupt 0, gen 0 Jan 24 01:11:52 CZ-Server kernel: blk_print_req_error: 19 callbacks suppressed Jan 24 01:11:52 CZ-Server kernel: I/O error, dev loop2, sector 53473064 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 2 Jan 24 01:11:52 CZ-Server kernel: btrfs_dev_stat_inc_and_print: 19 callbacks suppressed Jan 24 01:11:52 CZ-Server kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 30, rd 0, flush 0, corrupt 0, gen 0 Jan 24 01:11:52 CZ-Server kernel: I/O error, dev loop2, sector 42819984 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 2 Jan 24 01:11:52 CZ-Server kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 31, rd 0, flush 0, corrupt 0, gen 0 Jan 24 01:11:52 CZ-Server kernel: I/O error, dev loop2, sector 43462768 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 2 Jan 24 01:11:52 CZ-Server kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 32, rd 0, flush 0, corrupt 0, gen 0 Jan 24 01:11:52 CZ-Server kernel: I/O error, dev loop2, sector 42820000 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 2 Jan 24 01:11:52 CZ-Server kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 33, rd 0, flush 0, corrupt 0, gen 0 Jan 24 01:12:02 CZ-Server kernel: I/O error, dev loop2, sector 55725856 op 0x1:(WRITE) flags 0x10100 phys_seg 1 prio class 2 Jan 24 01:12:02 CZ-Server kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 34, rd 0, flush 0, corrupt 0, gen 0 Jan 24 01:12:02 CZ-Server kernel: I/O error, dev loop2, sector 57823008 op 0x1:(WRITE) flags 0x10100 phys_seg 1 prio class 2 Jan 24 01:12:02 CZ-Server kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 35, rd 0, flush 0, corrupt 0, gen 0 Jan 24 01:12:02 CZ-Server kernel: I/O error, dev loop2, sector 55725888 op 0x1:(WRITE) flags 0x10100 phys_seg 1 prio class 2 Jan 24 01:12:02 CZ-Server kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 36, rd 0, flush 0, corrupt 0, gen 0 Jan 24 01:12:02 CZ-Server kernel: I/O error, dev loop2, sector 57823040 op 0x1:(WRITE) flags 0x10100 phys_seg 1 prio class 2 Jan 24 01:12:02 CZ-Server kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 37, rd 0, flush 0, corrupt 0, gen 0 Jan 24 01:12:02 CZ-Server kernel: I/O error, dev loop2, sector 55725920 op 0x1:(WRITE) flags 0x10100 phys_seg 1 prio class 2 Jan 24 01:12:02 CZ-Server kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 38, rd 0, flush 0, corrupt 0, gen 0 Jan 24 01:12:02 CZ-Server kernel: I/O error, dev loop2, sector 57823072 op 0x1:(WRITE) flags 0x10100 phys_seg 1 prio class 2 Jan 24 01:12:02 CZ-Server kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 39, rd 0, flush 0, corrupt 0, gen 0 Jan 24 01:12:02 CZ-Server kernel: I/O error, dev loop2, sector 55725952 op 0x1:(WRITE) flags 0x10100 phys_seg 1 prio class 2 Jan 24 01:12:02 CZ-Server kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 40, rd 0, flush 0, corrupt 0, gen 0 Jan 24 01:12:02 CZ-Server kernel: I/O error, dev loop2, sector 57823104 op 0x1:(WRITE) flags 0x10100 phys_seg 1 prio class 2 Jan 24 01:12:02 CZ-Server kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 41, rd 0, flush 0, corrupt 0, gen 0 Jan 24 01:12:02 CZ-Server kernel: I/O error, dev loop2, sector 55725984 op 0x1:(WRITE) flags 0x10100 phys_seg 1 prio class 2 Jan 24 01:12:02 CZ-Server kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 42, rd 0, flush 0, corrupt 0, gen 0 Jan 24 01:12:02 CZ-Server kernel: I/O error, dev loop2, sector 57823136 op 0x1:(WRITE) flags 0x10100 phys_seg 1 prio class 2 Jan 24 01:12:02 CZ-Server kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 43, rd 0, flush 0, corrupt 0, gen 0 Jan 24 01:12:05 CZ-Server kernel: BTRFS error (device loop2 state A): Transaction aborted (error -5) Jan 24 01:12:05 CZ-Server kernel: BTRFS: error (device loop2 state A) in do_free_extent_accounting: IO failure Jan 24 01:12:05 CZ-Server kernel: BTRFS info (device loop2 state EA): forced readonly Jan 24 01:12:05 CZ-Server kernel: BTRFS error (device loop2 state EA): failed to run delayed ref for disk Jan 24 01:12:05 CZ-Server kernel: BTRFS: error (device loop2 state EA) in btrfs_run_delayed_refs: IO failure

I'm currently using 4. I tried the other day with just 2, but still eventually ran into a random restart. I could try some different memory configs though.

Right. Got a another PSU ordered to test that should be here in a couple days. Really hoping the cause is something easy like that to switch out. Also took out the system yesterday after that last restart to check all connections, but there didn't seem to be anything loose.

Yes to both

Looking at the latest syslog that I attached, it basically just looks like the session ends and then we see the server start up right? There's no indication from Unraid that something is going wrong before the restart.

Right. I thought you meant that Unraid was triggering a reboot. It's definitely being restarted one way or another. Hardware is what I was starting to think. First thing that came to mind was the PSU if the system is just shutting down.

Could be. Not sure why that would happen though.

Just had another unclean shutdown. The syslog is attached. For reference, the parity check after boot started at 1:13 PM if that helps to track down the issue. syslog-192.168.86.32.log

Didn't realize I had to set that up. I guess ill need to let it run until the problem happens again?