I recently updated to 6.12.13 and had some weird issues with Unraid being unresponsive after some time, I've had it die almost instantly after starting my array and also after a few days of it working without any issues.

At first, I thought it was the jumbo package issue since I have an RTL8111 NIC but I have the driver installed. But I also noticed that when I physically plugged in a keyboard/monitor I didn't get any response until I restarted it.

I've attached the diagnostics after it's crashed and I've restarted without starting arrays or anything else, any advice would be greatly appreciated

rocinante-diagnostics-20240930-2336.zip

Edited September 30, 20241 yr by ALameLlama

The syslog in the diagnostics is the RAM copy and only shows what happened since the reboot.   It could be worth enabling the syslog server to get a log that survives a reboot so we can see what happened prior to the reboot. The mirror to flash option is the easiest to set up, but if you are worried about excessive wear on the flash drive you can put your server’s address into the Remote Server field.

I've enabled that now, I'll update you once it's done it again.

Thanks

It seems to be a case of Schrödinger's crash, Ever since I've enabled mirroring the syslog to the USB, it's been stable.

I'll leave it over the weekend if nothing happens. I'll disable it again and see if that's any different

@itimpi Finally, This is the end of the syslog-previous file that was on the USB after rebooting it after the crash. seems like samba is dying and crashing?

 

Oct  5 03:07:22 Rocinante kernel: eth0: renamed from vethc1e5873
Oct  5 03:07:22 Rocinante kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth135303c: link becomes ready
Oct  5 03:07:22 Rocinante kernel: br-fa8dec63aaac: port 7(veth135303c) entered blocking state
Oct  5 03:07:22 Rocinante kernel: br-fa8dec63aaac: port 7(veth135303c) entered forwarding state
Oct  5 03:08:14 Rocinante kernel: BTRFS warning (device nvme0n1p1): csum failed root 5 ino 5954788 off 1348083712 csum 0xcdeade6f expected csum 0x36e7aeaa mirror 1
Oct  5 03:08:14 Rocinante kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme1n1p1 errs: wr 0, rd 0, flush 0, corrupt 20, gen 0
Oct  5 03:08:14 Rocinante kernel: BTRFS warning (device nvme0n1p1): csum failed root 5 ino 5954788 off 1348083712 csum 0xcdeade6f expected csum 0x36e7aeaa mirror 1
Oct  5 03:08:14 Rocinante kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme1n1p1 errs: wr 0, rd 0, flush 0, corrupt 21, gen 0
Oct  5 03:08:15 Rocinante kernel: BTRFS warning (device nvme0n1p1): csum failed root 5 ino 5954429 off 564867072 csum 0xa2e600ff expected csum 0xa78c04e1 mirror 1
Oct  5 03:08:15 Rocinante kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 19, gen 0
Oct  5 03:08:15 Rocinante kernel: BTRFS warning (device nvme0n1p1): csum failed root 5 ino 5954429 off 564867072 csum 0xa2e600ff expected csum 0xa78c04e1 mirror 1
Oct  5 03:08:15 Rocinante kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 20, gen 0
Oct  5 05:00:01 Rocinante Plugin Auto Update: Checking for available plugin updates
Oct  5 05:00:01 Rocinante Docker Auto Update: Community Applications Docker Autoupdate running
Oct  5 05:00:01 Rocinante Docker Auto Update: Checking for available updates
Oct  5 05:00:07 Rocinante Plugin Auto Update: fix.common.problems.plg version 2024.10.02 does not meet age requirements to update - 2 days old
Oct  5 05:00:07 Rocinante Plugin Auto Update: folder.view.plg version 2024.10.02 does not meet age requirements to update - 2 days old
Oct  5 05:00:07 Rocinante Plugin Auto Update: unassigned.devices.plg version 2024.10.02 does not meet age requirements to update - 2 days old
Oct  5 05:00:07 Rocinante Plugin Auto Update: Checking for language updates
Oct  5 05:00:07 Rocinante Plugin Auto Update: Community Applications Plugin Auto Update finished
Oct  5 05:00:50 Rocinante Docker Auto Update: Found update for redis-auth.  Not set to autoupdate
Oct  5 05:00:50 Rocinante Docker Auto Update: Found update for redis-pterodactyl.  Not set to autoupdate
Oct  5 05:00:50 Rocinante Docker Auto Update: Found update for traefik.  Not set to autoupdate
Oct  5 05:00:50 Rocinante Docker Auto Update: Stopping FileFlows
Oct  5 05:01:01 Rocinante kernel: br-fa8dec63aaac: port 23(vethb4465c7) entered disabled state
Oct  5 05:01:01 Rocinante kernel: vethd7e54eb: renamed from eth0
Oct  5 05:01:01 Rocinante kernel: br-fa8dec63aaac: port 23(vethb4465c7) entered disabled state
Oct  5 05:01:01 Rocinante kernel: device vethb4465c7 left promiscuous mode
Oct  5 05:01:01 Rocinante kernel: br-fa8dec63aaac: port 23(vethb4465c7) entered disabled state
Oct  5 05:01:02 Rocinante Docker Auto Update: Installing Updates for FileFlows
Oct  5 05:02:26 Rocinante Docker Auto Update: Restarting FileFlows
Oct  5 05:02:26 Rocinante kernel: br-fa8dec63aaac: port 23(vethc20ee0e) entered blocking state
Oct  5 05:02:26 Rocinante kernel: br-fa8dec63aaac: port 23(vethc20ee0e) entered disabled state
Oct  5 05:02:26 Rocinante kernel: device vethc20ee0e entered promiscuous mode
Oct  5 05:02:28 Rocinante kernel: eth0: renamed from veth05326fa
Oct  5 05:02:28 Rocinante kernel: IPv6: ADDRCONF(NETDEV_CHANGE): vethc20ee0e: link becomes ready
Oct  5 05:02:28 Rocinante kernel: br-fa8dec63aaac: port 23(vethc20ee0e) entered blocking state
Oct  5 05:02:28 Rocinante kernel: br-fa8dec63aaac: port 23(vethc20ee0e) entered forwarding state
Oct  5 05:02:29 Rocinante Docker Auto Update: Community Applications Docker Autoupdate finished
Oct  5 17:42:12 Rocinante webGUI: Successful login user root from 192.168.1.165
Oct  5 18:03:31 Rocinante smbd[13056]: [2024/10/05 18:03:31.674881,  0] ../../source3/smbd/close.c:1485(close_directory)
Oct  5 18:03:31 Rocinante smbd[13056]:   Could not close dir! fname=torrents/Ethically-sourced-media fd=40, err=116=Stale file handle
Oct  5 18:03:31 Rocinante smbd[13056]: [2024/10/05 18:03:31.674949,  0] ../../source3/smbd/fd_handle.c:39(fd_handle_destructor)
Oct  5 18:03:31 Rocinante smbd[13056]:   PANIC: assert failed at ../../source3/smbd/fd_handle.c(39): (fh->fd == -1) || (fh->fd == AT_FDCWD)
Oct  5 18:03:31 Rocinante smbd[13056]: [2024/10/05 18:03:31.674960,  0] ../../lib/util/fault.c:173(smb_panic_log)
Oct  5 18:03:31 Rocinante smbd[13056]:   ===============================================================
Oct  5 18:03:31 Rocinante smbd[13056]: [2024/10/05 18:03:31.674969,  0] ../../lib/util/fault.c:174(smb_panic_log)
Oct  5 18:03:31 Rocinante smbd[13056]:   INTERNAL ERROR: assert failed: (fh->fd == -1) || (fh->fd == AT_FDCWD) in pid 13056 (4.17.12)
Oct  5 18:03:31 Rocinante smbd[13056]: [2024/10/05 18:03:31.674974,  0] ../../lib/util/fault.c:178(smb_panic_log)
Oct  5 18:03:31 Rocinante smbd[13056]:   If you are running a recent Samba version, and if you think this problem is not yet fixed in the latest versions, please consider reporting this bug, see https://wiki.samba.org/index.php/Bug_Reporting
Oct  5 18:03:31 Rocinante smbd[13056]: [2024/10/05 18:03:31.674980,  0] ../../lib/util/fault.c:183(smb_panic_log)
Oct  5 18:03:31 Rocinante smbd[13056]:   ===============================================================
Oct  5 18:03:31 Rocinante smbd[13056]: [2024/10/05 18:03:31.674984,  0] ../../lib/util/fault.c:184(smb_panic_log)
Oct  5 18:03:31 Rocinante smbd[13056]:   PANIC (pid 13056): assert failed: (fh->fd == -1) || (fh->fd == AT_FDCWD) in 4.17.12
Oct  5 18:03:31 Rocinante smbd[13056]: [2024/10/05 18:03:31.675188,  0] ../../lib/util/fault.c:292(log_stack_trace)
Oct  5 18:03:31 Rocinante smbd[13056]:   BACKTRACE: 27 stack frames:
Oct  5 18:03:31 Rocinante smbd[13056]:    #0 /usr/lib64/libgenrand-samba4.so(log_stack_trace+0x2e) [0x148db832064e]
Oct  5 18:03:31 Rocinante smbd[13056]:    #1 /usr/lib64/libgenrand-samba4.so(smb_panic+0x9) [0x148db83208a9]
Oct  5 18:03:31 Rocinante smbd[13056]:    #2 /usr/lib64/libsmbd-base-samba4.so(+0x4d0fb) [0x148db87020fb]
Oct  5 18:03:31 Rocinante smbd[13056]:    #3 /usr/lib64/libtalloc.so.2(+0x44df) [0x148db82cf4df]
Oct  5 18:03:31 Rocinante smbd[13056]:    #4 /usr/lib64/libsmbd-base-samba4.so(file_free+0xd6) [0x148db870f266]
Oct  5 18:03:31 Rocinante smbd[13056]:    #5 /usr/lib64/libsmbd-base-samba4.so(+0xc0781) [0x148db8775781]
Oct  5 18:03:31 Rocinante smbd[13056]:    #6 /usr/lib64/libsmbd-base-samba4.so(smbd_smb2_request_process_close+0x211) [0x148db8775f01]
Oct  5 18:03:31 Rocinante smbd[13056]:    #7 /usr/lib64/libsmbd-base-samba4.so(smbd_smb2_request_dispatch+0xdfa) [0x148db8769bfa]
Oct  5 18:03:31 Rocinante smbd[13056]:    #8 /usr/lib64/libsmbd-base-samba4.so(+0xb5bc1) [0x148db876abc1]
Oct  5 18:03:31 Rocinante smbd[13056]:    #9 /usr/lib64/libtevent.so.0(tevent_common_invoke_fd_handler+0x91) [0x148db82e28c1]
Oct  5 18:03:31 Rocinante smbd[13056]:    #10 /usr/lib64/libtevent.so.0(+0xee07) [0x148db82e8e07]
Oct  5 18:03:31 Rocinante smbd[13056]:    #11 /usr/lib64/libtevent.so.0(+0xcef7) [0x148db82e6ef7]
Oct  5 18:03:31 Rocinante smbd[13056]:    #12 /usr/lib64/libtevent.so.0(_tevent_loop_once+0x91) [0x148db82e1ba1]
Oct  5 18:03:31 Rocinante smbd[13056]:    #13 /usr/lib64/libtevent.so.0(tevent_common_loop_wait+0x1b) [0x148db82e1e7b]
Oct  5 18:03:31 Rocinante smbd[13056]:    #14 /usr/lib64/libtevent.so.0(+0xce97) [0x148db82e6e97]
Oct  5 18:03:31 Rocinante smbd[13056]:    #15 /usr/lib64/libsmbd-base-samba4.so(smbd_process+0x817) [0x148db8758be7]
Oct  5 18:03:31 Rocinante smbd[13056]:    #16 /usr/sbin/smbd(+0xb090) [0x55efeb70b090]
Oct  5 18:03:31 Rocinante smbd[13056]:    #17 /usr/lib64/libtevent.so.0(tevent_common_invoke_fd_handler+0x91) [0x148db82e28c1]
Oct  5 18:03:31 Rocinante smbd[13056]:    #18 /usr/lib64/libtevent.so.0(+0xee07) [0x148db82e8e07]
Oct  5 18:03:31 Rocinante smbd[13056]:    #19 /usr/lib64/libtevent.so.0(+0xcef7) [0x148db82e6ef7]
Oct  5 18:03:31 Rocinante smbd[13056]:    #20 /usr/lib64/libtevent.so.0(_tevent_loop_once+0x91) [0x148db82e1ba1]
Oct  5 18:03:31 Rocinante smbd[13056]:    #21 /usr/lib64/libtevent.so.0(tevent_common_loop_wait+0x1b) [0x148db82e1e7b]
Oct  5 18:03:31 Rocinante smbd[13056]:    #22 /usr/lib64/libtevent.so.0(+0xce97) [0x148db82e6e97]
Oct  5 18:03:31 Rocinante smbd[13056]:    #23 /usr/sbin/smbd(main+0x1489) [0x55efeb708259]
Oct  5 18:03:31 Rocinante smbd[13056]:    #24 /lib64/libc.so.6(+0x236b7) [0x148db80e86b7]
Oct  5 18:03:31 Rocinante smbd[13056]:    #25 /lib64/libc.so.6(__libc_start_main+0x85) [0x148db80e8775]
Oct  5 18:03:31 Rocinante smbd[13056]:    #26 /usr/sbin/smbd(_start+0x21) [0x55efeb708b31]
Oct  5 18:03:31 Rocinante smbd[13056]: [2024/10/05 18:03:31.675254,  0] ../../source3/lib/dumpcore.c:315(dump_core)
Oct  5 18:03:31 Rocinante smbd[13056]:   dumping core in /var/log/samba/cores/smbd
Oct  5 18:03:31 Rocinante smbd[13056]: 

 

 

Edited October 5, 20241 yr by ALameLlama

Btrfs is detecting data corruption on the pool, run a correcting scrub and post the results.

 

Also, Samba is crashing, so it may be a good idea to run memtest.

 

This was the btrfs scrub, I'll shart the mem test shortly but that will take a while to finish.

UUID:             2a375431-ec6e-43a0-95ec-bb06d85a107c
Scrub started:    Sat Oct  5 20:08:02 2024
Status:           finished
Duration:         0:05:58
Total to scrub:   870.00GiB
Rate:             2.43GiB/s
Error summary:    csum=10
  Corrected:      0
  Uncorrectable:  10
  Unverified:     0

Check the syslog for the list of corrupt files, then delete/restore from a backup and re-run another scrub to confirm no more errors.

It looks like something is wrong with the memory. I've moved the sticks around and am running another mem test to see if it's the stick or the slot.

Yep, just one error is too many.

Surprisingly, after moving the RAM around, the memtest passes without any issues. I've removed the corrupted files and I'll see if that resolves the issue

After that recommend to reset the stats and keep monitoring the pool, if new corruptions are found there's still a problem.

Had it kernel panic, The syslog didn't show anything from what I can tell. the syslog-previous just stopped in the middle of what it was doing. I've included the syslog on the reboot in case there is anything useful there.

 

I've started doing another memtest and am gonna leave it over night to see if that'll give more accurate results

syslog

Note that memtest is only definitive if it finds errors, if you have multiple sticks try using the server with just one, if the same try with a different one, that will basically rule out bad RAM.

After lots of testing ended up being a dying mobo