My syslog is full of this:

Jan 3 08:46:43 Watchtower kernel: BTRFS warning (device nvme1n1p1): csum failed root 5 ino 144866576 off 16384 csum 0x8941f998 expected csum 0xf230cf28 mirror 1 Jan 3 08:46:43 Watchtower kernel: BTRFS error (device nvme1n1p1): bdev /dev/nvme0n1p1 errs: wr 74575594, rd 127015, flush 2688585, corrupt 795283, gen 0 Jan 3 08:46:43 Watchtower kernel: BTRFS warning (device nvme1n1p1): csum failed root 5 ino 144866576 off 49152 csum 0x8941f998 expected csum 0xb9fd18fc mirror 1 Jan 3 08:46:43 Watchtower kernel: BTRFS error (device nvme1n1p1): bdev /dev/nvme0n1p1 errs: wr 74575594, rd 127015, flush 2688585, corrupt 795284, gen 0 Jan 3 08:46:43 Watchtower kernel: repair_io_failure: 4 callbacks suppressed Jan 3 08:46:43 Watchtower kernel: BTRFS info (device nvme1n1p1): read error corrected: ino 144866576 off 16384 (dev /dev/nvme0n1p1 sector 44269288) Jan 3 08:46:43 Watchtower kernel: BTRFS info (device nvme1n1p1): read error corrected: ino 144866576 off 49152 (dev /dev/nvme0n1p1 sector 44269352) Jan 3 08:46:43 Watchtower kernel: BTRFS warning (device nvme1n1p1): csum failed root 5 ino 144866576 off 327680 csum 0x8941f998 expected csum 0x0ccf29d1 mirror 1 Jan 3 08:46:43 Watchtower kernel: BTRFS error (device nvme1n1p1): bdev /dev/nvme0n1p1 errs: wr 74575594, rd 127015, flush 2688585, corrupt 795285, gen 0 Jan 3 08:46:43 Watchtower kernel: BTRFS warning (device nvme1n1p1): csum failed root 5 ino 144866576 off 331776 csum 0x8941f998 expected csum 0xf904b254 mirror 1 Jan 3 08:46:43 Watchtower kernel: BTRFS error (device nvme1n1p1): bdev /dev/nvme0n1p1 errs: wr 74575594, rd 127015, flush 2688585, corrupt 795286, gen 0 Jan 3 08:46:43 Watchtower kernel: BTRFS warning (device nvme1n1p1): csum failed root 5 ino 144866576 off 335872 csum 0x8941f998 expected csum 0x5437483e mirror 1 Jan 3 08:46:43 Watchtower kernel: BTRFS error (device nvme1n1p1): bdev /dev/nvme0n1p1 errs: wr 74575594, rd 127015, flush 2688585, corrupt 795287, gen 0 Jan 3 08:46:43 Watchtower kernel: BTRFS warning (device nvme1n1p1): csum failed root 5 ino 144866576 off 339968 csum 0x8941f998 expected csum 0x4f0074a8 mirror 1 Jan 3 08:46:43 Watchtower kernel: BTRFS error (device nvme1n1p1): bdev /dev/nvme0n1p1 errs: wr 74575594, rd 127015, flush 2688585, corrupt 795288, gen 0 Jan 3 08:46:43 Watchtower kernel: BTRFS warning (device nvme1n1p1): csum failed root 5 ino 144866576 off 344064 csum 0x8941f998 expected csum 0x77433822 mirror 1 Jan 3 08:46:43 Watchtower kernel: BTRFS error (device nvme1n1p1): bdev /dev/nvme0n1p1 errs: wr 74575594, rd 127015, flush 2688585, corrupt 795289, gen 0 Jan 3 08:46:43 Watchtower kernel: BTRFS warning (device nvme1n1p1): csum failed root 5 ino 144866576 off 348160 csum 0x8941f998 expected csum 0x3f262c36 mirror 1 Jan 3 08:46:43 Watchtower kernel: BTRFS error (device nvme1n1p1): bdev /dev/nvme0n1p1 errs: wr 74575594, rd 127015, flush 2688585, corrupt 795290, gen 0 Jan 3 08:46:43 Watchtower kernel: BTRFS warning (device nvme1n1p1): csum failed root 5 ino 144866576 off 352256 csum 0x8941f998 expected csum 0x0f648a88 mirror 1 Jan 3 08:46:43 Watchtower kernel: BTRFS error (device nvme1n1p1): bdev /dev/nvme0n1p1 errs: wr 74575594, rd 127015, flush 2688585, corrupt 795291, gen 0 Jan 3 08:46:43 Watchtower kernel: BTRFS warning (device nvme1n1p1): csum failed root 5 ino 144866576 off 356352 csum 0x8941f998 expected csum 0xd92f4cd2 mirror 1 Jan 3 08:46:43 Watchtower kernel: BTRFS error (device nvme1n1p1): bdev /dev/nvme0n1p1 errs: wr 74575594, rd 127015, flush 2688585, corrupt 795292, gen 0 Jan 3 08:46:43 Watchtower kernel: BTRFS info (device nvme1n1p1): read error corrected: ino 144866576 off 327680 (dev /dev/nvme0n1p1 sector 44269896) Jan 3 08:46:43 Watchtower kernel: BTRFS info (device nvme1n1p1): read error corrected: ino 144866576 off 335872 (dev /dev/nvme0n1p1 sector 44269912) Jan 3 08:46:43 Watchtower kernel: BTRFS info (device nvme1n1p1): read error corrected: ino 144866576 off 331776 (dev /dev/nvme0n1p1 sector 44269904) Jan 3 08:46:43 Watchtower kernel: BTRFS info (device nvme1n1p1): read error corrected: ino 144866576 off 339968 (dev /dev/nvme0n1p1 sector 44269920) Jan 3 08:46:43 Watchtower kernel: BTRFS info (device nvme1n1p1): read error corrected: ino 144866576 off 368640 (dev /dev/nvme0n1p1 sector 44269976) Jan 3 08:46:43 Watchtower kernel: BTRFS info (device nvme1n1p1): read error corrected: ino 144866576 off 401408 (dev /dev/nvme0n1p1 sector 44270040) Jan 3 08:46:43 Watchtower kernel: BTRFS info (device nvme1n1p1): read error corrected: ino 144866576 off 344064 (dev /dev/nvme0n1p1 sector 44269928) Jan 3 08:46:43 Watchtower kernel: BTRFS info (device nvme1n1p1): read error corrected: ino 144866576 off 372736 (dev /dev/nvme0n1p1 sector 44269984)

I haven't had any system issues yet, save Plex stopping last night but a system restart fixed that. I've got no smart errors on any drive, cache or array. I ran memtest for 8.5 hours last night, and it passed. All my google searches say either memtest or failing drive, neither of which seem to be the case. I will mention that, according to the device attributes in UnRaid, the drive has a bunch of writes:

Data units written 1,365,844,436 [699 TB]

which I think is over WD specs for lifetime data writes. Could that be the issue? It's also pretty full, but there is 4GB of free space. I'm kinda at a loss on what to do at this point.

Thanks!

watchtower-diagnostics-20230103-1106.zip

Jan  3 07:54:07 Watchtower kernel: BTRFS info (device nvme1n1p1): bdev /dev/nvme1n1p1 errs: wr 386140, rd 7076, flush 56562, corrupt 4568, gen 0
Jan  3 07:54:07 Watchtower kernel: BTRFS info (device nvme1n1p1): bdev /dev/nvme0n1p1 errs: wr 74575594, rd 127015, flush 2688585, corrupt 794981, gen 0

 

Both NVMe devices dropped offline in the post, likely at different times, run a correcting scrub on the pool and post the results.

7 minutes ago, JorgeB said:

run a correcting scrub on the pool and post the results

Uhhh... This?

 

UUID: 8c2cf7ae-792d-4e0b-88bb-544a2c6d33e9 
Scrub started: Tue Jan 3 11:29:40 2023 
Status: finished 
Duration: 0:03:12 
Total to scrub: 918.92GiB 
Rate: 4.79GiB/s 
Error summary: verify=310 csum=90910 
Corrected: 91220 
Uncorrectable: 0 
Unverified: 0

Is that all I needed to do? Is that something that I should run on a schedule? I saw the run on schedule just past the manual option, and it's disabled.

You mostly need to monitor the pool to see if the devices keep dropping, see below for better pool monitoring:

https://forums.unraid.net/topic/46802-faq-for-unraid-v6/?do=findComment&comment=700582