My unraid server has been crashing after running normally for a while. I cannot see all the logs, but what I did see said kernel panic. Previous to this I was having docker issues where the docker service would crash. I saw someone say on the forums to switch to ipvlan instead of macvlan and that resolved the docker service issues, but now I'm getting random kernel panics. Has anyone else seen this?

 

Unraid was working great for me for years, but seems like there are some issues with 6.12.

 

Here's my diagnostics, but they don't seem to have the kernel panic log, but maybe someone with more knowledge will know what to do with it.

 

Thanks!

tower-diagnostics-20231014-1628.zip

Enable the syslog server and post that after a crash.

 

thanks just enabled. In the meantime my docker service crashed again (in the UI) and i'm seeing these logs 

 

Oct 15 08:35:45 Tower kernel: CPU: 4 PID: 5955 Comm: smartctl_type Tainted: P      D    O       6.1.49-Unraid #1
Oct 15 08:35:45 Tower kernel: Hardware name: Gigabyte Technology Co., Ltd. B450 AORUS PRO WIFI/B450 AORUS PRO WIFI-CF, BIOS F63 01/31/2023
Oct 15 08:35:45 Tower kernel: Call Trace:
Oct 15 08:35:45 Tower kernel: <TASK>
Oct 15 08:35:45 Tower kernel: dump_stack_lvl+0x44/0x5c
Oct 15 08:35:45 Tower kernel: __schedule_bug+0x9a/0xac
Oct 15 08:35:45 Tower kernel: __schedule+0x59/0x612
Oct 15 08:35:45 Tower kernel: ? _printk+0x58/0x73
Oct 15 08:35:45 Tower kernel: do_task_dead+0x3f/0x41
Oct 15 08:35:45 Tower kernel: make_task_dead+0x114/0x11c
Oct 15 08:35:45 Tower kernel: rewind_stack_and_make_dead+0x17/0x17
Oct 15 08:35:45 Tower kernel: RIP: 0033:0x14e5daff1425
Oct 15 08:35:45 Tower kernel: Code: Unable to access opcode bytes at 0x14e5daff13fb.
Oct 15 08:35:45 Tower kernel: RSP: 002b:00007ffccf9867b8 EFLAGS: 00000206 ORIG_RAX: 00000000000000e7
Oct 15 08:35:45 Tower kernel: RAX: ffffffffffffffda RBX: 000014e5db1002e8 RCX: 000014e5daff1425
Oct 15 08:35:45 Tower kernel: RDX: 00000000000000e7 RSI: ffffffffffffff78 RDI: 0000000000000000
Oct 15 08:35:45 Tower kernel: RBP: 0000000000000000 R08: 00007ffccf986760 R09: 0000000001643370
Oct 15 08:35:45 Tower kernel: R10: 000014e5daf27b10 R11: 0000000000000206 R12: 0000000000000000
Oct 15 08:35:45 Tower kernel: R13: 000014e5db0fe840 R14: 0000000000000000 R15: 000014e5db100300
Oct 15 08:35:45 Tower kernel: </TASK>
Oct 15 09:14:02 Tower kernel: nvidia_uvm: module uses symbols nvUvmInterfaceDisableAccessCntr from proprietary module nvidia, inheriting taint.
Oct 15 09:14:02 Tower kernel: nvidia-uvm: Loaded the UVM driver, major device number 241.
Oct 15 09:36:07 Tower kernel: traps: PMS HttpClient[23279] general protection fault ip:1469b13b8658 sp:1469ac1f7880 error:0 in ld-musl-x86_64.so.1[1469b13a8000+53000]
Oct 15 10:52:28 Tower kernel: traps: unraid-api[14073] trap int3 ip:ebf052 sp:154b6912b6c0 error:0 in unraid-api[91c000+167b000]
Oct 15 11:57:58 Tower kernel: Scheduler frequency invariance went wobbly, disabling!
Oct 15 13:39:12 Tower kernel: BTRFS warning (device sdc1): csum failed root 5 ino 21223 off 118784 csum 0xc834b506 expected csum 0xc2b739ab mirror 1
Oct 15 13:39:12 Tower kernel: BTRFS warning (device sdc1): csum failed root 5 ino 21223 off 524288 csum 0x2c64439a expected csum 0xd7bd1fd5 mirror 1
Oct 15 13:39:12 Tower kernel: BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 0, rd 0, flush 0, corrupt 1, gen 0
Oct 15 13:39:12 Tower kernel: BTRFS warning (device sdc1): csum failed root 5 ino 21223 off 118784 csum 0xc834b506 expected csum 0xc2b739ab mirror 1
Oct 15 13:39:12 Tower kernel: BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0
Oct 15 13:39:12 Tower kernel: BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 0, rd 0, flush 0, corrupt 3, gen 0
Oct 15 13:39:12 Tower kernel: BTRFS warning (device sdc1): csum failed root 5 ino 21223 off 798720 csum 0xe5785487 expected csum 0x0196ed0f mirror 1
Oct 15 13:39:12 Tower kernel: BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 0, rd 0, flush 0, corrupt 4, gen 0
Oct 15 13:39:12 Tower kernel: BTRFS warning (device sdc1): csum failed root 5 ino 21223 off 524288 csum 0x2c64439a expected csum 0xd7bd1fd5 mirror 1
Oct 15 13:39:12 Tower kernel: BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 0, rd 0, flush 0, corrupt 5, gen 0
Oct 15 13:52:42 Tower kernel: traps: unraid-api[24897] trap int3 ip:ebf052 sp:14d53245e6c0 error:0 in unraid-api[91c000+167b000]
Oct 15 14:15:37 Tower webGUI: Successful login user root from 192.168.86.25
Oct 15 14:35:00 Tower kernel: traps: php[4252] general protection fault ip:14ae625e9a7c sp:7ffe723ce640 error:0 in ld-musl-x86_64.so.1[14ae625d8000+4c000]
Oct 15 14:54:59 Tower monitor: Stop running nchan processes
Oct 15 22:13:04 Tower monitor: Stop running nchan processes
Oct 16 04:23:49 Tower kernel: traps: unraid-api[5904] trap int3 ip:ebf052 sp:148d2ecf76c0 error:0 in unraid-api[91c000+167b000]

 

Btrfs is detecting data corruption, suggest running memtest.

memtest isn't working choosing it as an option just puts me in a reboot cycle.

 

I ran diagnostics. not sure if this helps.

tower-diagnostics-20231016-1105.zip

8 minutes ago, droyster said:

memtest isn't working choosing it as an option just puts me in a reboot cycle.

That will happen if you are booting UEFI, boot in legacy mode or use the free passmark memtest instead.

i have logs here of when docker crashed again.

anything stand out here? 

I'm going to try replacing the ram, but I'm wondering if these logs have anything useful

syslog_crash.zip

Only thing I see is btrfs continuing to detect new data corruption, also some possible flash drives issues (could also be RAM related)

server has been running consistently for days now. Not positive what fixed it, but I changed my drive storage to `auto` and I turned off every docker container, but my plex one. I did see some errors in the docker logs. Since doing those 2 steps things seem fine

Any fix? I'm getting a very similar call trace

try seeing if your server will run without panicking with all docker containers shut down, then turn them on one by one and wait a day between each to give it time to panic