Hello,

 

After I updated I was unable to get docker to start.  It also looks like I have a corrupt cache drive or its full?  No idea what is going on.

 

Any help would be appreciated, thank you.

One of my cache disks (The one where system and dockers are run from) is two nvme drives running in Raid 1

 

Dec 20 23:13:59 brain kernel: CPU: 0 PID: 1109 Comm: udevd Not tainted 6.1.64-Unraid #1
Dec 20 23:13:59 brain kernel: Call Trace:
Dec 20 23:14:26 brain kernel: BTRFS info (device nvme0n1p1): read error corrected: ino 0 off 211861504 (dev /dev/nvme0n1p1 sector 413792)
Dec 20 23:14:26 brain kernel: BTRFS info (device nvme0n1p1): read error corrected: ino 0 off 211865600 (dev /dev/nvme0n1p1 sector 413800)
Dec 20 23:14:26 brain kernel: BTRFS info (device nvme0n1p1): read error corrected: ino 0 off 211869696 (dev /dev/nvme0n1p1 sector 413808)
Dec 20 23:14:26 brain kernel: BTRFS info (device nvme0n1p1): read error corrected: ino 0 off 211873792 (dev /dev/nvme0n1p1 sector 413816)
Dec 20 23:14:26 brain kernel: BTRFS info (device nvme0n1p1): read error corrected: ino 0 off 876822528 (dev /dev/nvme0n1p1 sector 1712544)
Dec 20 23:14:26 brain kernel: BTRFS info (device nvme0n1p1): read error corrected: ino 0 off 876826624 (dev /dev/nvme0n1p1 sector 1712552)
Dec 20 23:14:26 brain kernel: BTRFS info (device nvme0n1p1): read error corrected: ino 0 off 876830720 (dev /dev/nvme0n1p1 sector 1712560)
Dec 20 23:14:26 brain kernel: BTRFS info (device nvme0n1p1): read error corrected: ino 0 off 876834816 (dev /dev/nvme0n1p1 sector 1712568)
Dec 20 23:14:26 brain kernel: BTRFS info (device nvme0n1p1): read error corrected: ino 0 off 148733952 (dev /dev/nvme0n1p1 sector 290496)
Dec 20 23:14:26 brain kernel: BTRFS info (device nvme0n1p1): read error corrected: ino 0 off 148738048 (dev /dev/nvme0n1p1 sector 290504)
Dec 20 23:14:32 brain kernel: BTRFS info (device nvme0n1p1): read error corrected: ino 0 off 725033123840 (dev /dev/nvme0n1p1 sector 1298639808)
Dec 20 23:14:32 brain kernel: BTRFS info (device nvme0n1p1): read error corrected: ino 0 off 725033127936 (dev /dev/nvme0n1p1 sector 1298639816)
Dec 20 23:14:32 brain kernel: BTRFS info (device nvme0n1p1): read error corrected: ino 0 off 725033132032 (dev /dev/nvme0n1p1 sector 1298639824)
Dec 20 23:14:32 brain kernel: BTRFS info (device nvme0n1p1): read error corrected: ino 0 off 86294528 (dev /dev/nvme0n1p1 sector 168544)
Dec 20 23:14:32 brain kernel: BTRFS info (device nvme0n1p1): read error corrected: ino 0 off 725033136128 (dev /dev/nvme0n1p1 sector 1298639832)
Dec 20 23:14:32 brain kernel: BTRFS info (device nvme0n1p1): read error corrected: ino 0 off 86298624 (dev /dev/nvme0n1p1 sector 168552)
Dec 20 23:14:32 brain kernel: BTRFS info (device nvme0n1p1): read error corrected: ino 0 off 86302720 (dev /dev/nvme0n1p1 sector 168560)
Dec 20 23:14:32 brain kernel: BTRFS info (device nvme0n1p1): read error corrected: ino 0 off 86306816 (dev /dev/nvme0n1p1 sector 168568)
Dec 20 23:14:32 brain kernel: BTRFS info (device nvme0n1p1): read error corrected: ino 0 off 724843757568 (dev /dev/nvme0n1p1 sector 1298269952)
Dec 20 23:14:32 brain kernel: BTRFS info (device nvme0n1p1): read error corrected: ino 0 off 724843761664 (dev /dev/nvme0n1p1 sector 1298269960)
Dec 20 23:14:32 brain kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 54562938, rd 208875, flush 4704339, corrupt 1, gen 0
Dec 20 23:14:32 brain kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 54562938, rd 208875, flush 4704339, corrupt 2, gen 0
Dec 20 23:14:36 brain kernel: ata5.00: exception Emask 0x10 SAct 0x2 SErr 0x280100 action 0x6 frozen
Dec 20 23:14:36 brain kernel: ata5.00: irq_stat 0x08000000, interface fatal error

 

 

 

brain-diagnostics-20231220-2344.zip

Edited December 21, 20232 yr by dirkinthedark

Dec 20 23:14:26 brain kernel: BTRFS info (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 54562938, rd 208875, flush 4704339, corrupt 0, gen 0
Dec 20 23:14:26 brain kernel: BTRFS info (device nvme0n1p1): bdev /dev/nvme1n1p1 errs: wr 0, rd 0, flush 0, corrupt 36, gen 0

This shows that one of the devices dropped offline in the past, there are also data corruption errors on the other one:

 

Dec 20 23:14:27 brain kernel: BTRFS info (device sdf1): bdev /dev/sdf1 errs: wr 0, rd 0, flush 0, corrupt 889, gen 0
Dec 20 23:14:27 brain kernel: BTRFS info (device sdf1): bdev /dev/sdi1 errs: wr 0, rd 0, flush 0, corrupt 1174, gen 0

More data corruption on the other pool, start by running memtest.

Drats, this only happend after I updated, but who knows could have been there lurking.  Of course I'm out of town

Thank you Jorge

I never got around to Memtesting as I am out of town and the family wanted their movies and minecraft servers back for the Christmas holidays.

 

I ended up copying the data off the failed nvmecache, formatting, and repopulating it.

 

I seem to be having other issues now since the upgrade but I guess I'll burn those bridges when I get there.

 

You can take a look here to reset the current pool stats and then keep monitoring, if more corruptions are found there's still a problem.

Yes its been clewr for a while, don't know what happened

17 hours ago, dirkinthedark said:

never got around to Memtesting

You shouldn't even attempt to run any computer unless memory is working perfectly. Everything goes through RAM, the OS and other executable code, your data, everything. The CPU can't do anything with anything until it is loaded into RAM.

I did quick test after building.  I work out of town and I wont be back home for weeks.  Just doing what I can.  Maybe need to get a rasberry pi kvm