unRAID has been running great for a long time. About 3 months ago, it started completely freezing every 2-3 days, I have no idea why, and I cannot figure out what's causing it.

 

Notable information:

• Running v6.12.4
• CPU - i9-14900k
• Motherboard - MSI Z790
• Memory - 2 x 48GB
• Latest crash/freeze was last night
• Seems to happen at random
• During freezes, the system becomes completely unresponsive. VMs/Dockers/GUI/shares not functional. When monitor is plugged into the system, it shows either CLI or Web GUI (depending on which mode I booted into), but no keyboard/mouse input is recognized.
• A hard shutdown via holding power button down is the only way to reboot into a functional state again
• Nothing found via Fix Common Problems
• Diagnostics and Syslog attached

 

Things I've tried so far:

• Tried changing 'Docker custom network type:' to ipvlan
• Removed 'Mover Tuning' plugin
• Upgraded CPU, motherboard, and Memory to new hardware

 

Any help would be greatly appreciated.

 

EDIT: Tried running memtest from the unRAID boot menu, but it gets stuck on 'loading /memtest... ok' and does nothing after that.  Should my next step be rebuilding the USB drive?

syslog-192.168.1.25.log unraid-diagnostics-20231114-0832.zip

Edited November 14, 20232 yr by barnowan

Do see some call traces but no indication of what's causing them, could be hardware related.

 

1 hour ago, barnowan said:

EDIT: Tried running memtest from the unRAID boot menu, but it gets stuck on 'loading /memtest... ok' and does nothing after that.

Current memtest only works for legacy boot, if possible boot legacy or use the free Passmark memtest that works with UEFI, alternatively try running with just one stick of RAM, if the same try the other one, that will basically rule out the RAM.

Just ran 4 passes of memtest over 5+ hours, and it passed with zero errors with both sticks of memory in.  I just swapped out the dual power supplies (Supermicro CSE-846 enclosure) with a couple I had in another chassis that weren't actively being used.  We'll see if that makes any difference.  Meanwhile, is there anything else you can think of that I can try if that fails?

Unfortunately locked up again after about 2 hours with different PSUs installed.

The server is acting up a different way this time.  It's overall functional and responsive, but none of the dockers are working at the moment.  New diagnostics attached.

 

According to syslog, it's looking like nvme1n1p1 may be failing:

 

"Nov 16 21:38:19 unRAID kernel: BTRFS critical (device nvme1n1p1): corrupt leaf: root=18446744073709551610 block=184367316992 slot=154 ino=50661511, invalid location key type, have 5, expect 132 or 1"

 

unRAID doesn't do anything when I try to run a SMART test on it, but it says it's in a healthy state.

 

 

  The log for that disk is also repeating these messages over and over again:

 

Nov 17 00:02:39 unRAID kernel: BTRFS error (device nvme1n1p1: state EAL): parent transid verify failed on logical 184360189952 mirror 1 wanted 2543847 found 2543846
Nov 17 00:02:39 unRAID kernel: BTRFS error (device nvme1n1p1: state EAL): parent transid verify failed on logical 184360189952 mirror 2 wanted 2543847 found 2543846

 

If I run a scrub, no errors are found.

 

 

Is this NVMe drive definitely bad?  If I replace it with another one, is there anything special I need to do afterwards?  It's in a cache pool with another identical drive running in raid 1 mode.

 

Thanks in advance for any help.

unraid-diagnostics-20231117-0047.zip

Nov 16 13:04:40 unRAID kernel: BTRFS info (device nvme1n1p1): bdev /dev/nvme2n1p1 errs: wr 3077252, rd 5647, flush 2008, corrupt 270, gen 0

 

This shows one your cache devices dropped offline in the past, see here for more info.

 

This can sometimes help, on the main GUI page click on the flash drive, scroll down to "Syslinux Configuration", make sure it's set to "menu view" (top right) and add this to your default boot option, after "append initrd=/bzroot"

nvme_core.default_ps_max_latency_us=0 pcie_aspm=off

e.g.:

append initrd=/bzroot nvme_core.default_ps_max_latency_us=0 pcie_aspm=off

Reboot and see if it makes a difference.

Thanks for the suggestion. It didn't seem to help much though.

 

I'm still getting constant freezes, I have some disks that are now emulated, and I can't keep unRAID operational long enough to complete the parity check (up to 22TB disks) which is making me a bit nervous.  I tried nuking my systemcache (docker, appdata, VMs, etc.) completely, formatting the NVMe drives, recreating the pool, and that didn't seem to help either. 

 

I also seem to be getting lots of call traces and possibly some CPU errors.

 

Any help would be greatly appreciated.  Updated diagnostics and syslog attached.

syslog-192.168.1.25.log unraid-diagnostics-20231122-2017.zip

Nov 17 03:40:28 unRAID kernel: macvlan_broadcast+0x10a/0x150 [macvlan]
Nov 17 03:40:28 unRAID kernel: ? _raw_spin_unlock+0x14/0x29
Nov 17 03:40:28 unRAID kernel: macvlan_process_broadcast+0xbc/0x12f [macvlan]

Macvlan call traces will usually end up crashing the server, switching to ipvlan should fix it (Settings -> Docker Settings -> Docker custom network type -> ipvlan (advanced view must be enabled, top right)), then reboot.
 

That's odd because I had already changed that to ipvlan a while back and have had successful reboots since then.

 

Is it possible there's an error where the UI is displaying ipvlan when macvlan is still actually in use? Do you know of a way to verify that?

That call trace is from the 17th, if you changed and rebooted after that it won't be the problem, unfortunately there's nothing else relevant logged after that.

That may be during the short period where I tried going back to macvlan for a bit to see if there was any change in stability.

 

At this point, what would you suggest? Are there any other logs I can gather? I'm willing to do whatever it takes to get this stable again. 

Could be hardware, one thing you can try is to boot the server in safe mode with all docker/VMs disabled, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one. 

This latest session was able to stay up for a couple days to finish the parity sync with no VMs started and with very limited docker containers running.  After the sync finished, I started up some more dockers (heimdall, overseer, sonarr, radarr, sabnzbd, prowlarr, possibly tautulli), and within 30 minutes, it totally froze again.  I tried looking around to see if I can grab the logs from the individual docker containers, but it seems there's no way to view those from a previous session that I've found. 

 

I had to do a hard shutdown, and I have emulated disks again.  However, the monitor hooked up to the machine had the below info this time.  Does this help indicate what may be causing this?

 

Quote

IPv6 address: not set
UnRAID login: kernel tried to execute NX-protected page - exploit attempt? (uid: 99)
BUG: unable to handle page fault for address: ffff888107ad207f SPF: supervisor instruction fetch in kernel mode
APF: error_code (0x0011) - permissions violation PGD 4c01067 P4D 4c01067 PUD 101739063 PMD 1051a2063 PTE 8000000107ad2163
4
Oops: 0011 [#1] PREEMPT SMP NOPTI
CPU: 10 PID: 30086 Comn: Plex Transcoder Tainted: P
0
6.1.49-Unraid #1 Hardware name: Micro-Star International Co., Ltd. MS-7E07/PRO 2790-A WIFI (MS-7E07), BIOS A.70 09/08/2023 RIP: 0010:0xffff888107ad207f
RSP: 0018:ffffc9002ae0fc68 EFLAGS: 00010246
Code:
69864560636f 64 65 72 2f 43 6f 6e 76 65 72 74 20 74 6f 20 57 41 56 20 28 74 64 20 30 63 68 20 6f 72 20 6c 65 73 73 29 28 <34> 34 36 34 61 65 62 33 24 61 34 64 31 24 34 30 64 64 24 39 36 38
RAX: 0000000000000000 RBX: ffffffff8124bda0 RCX: ffff888108107000 RDX: ffff888268727740 RSI: ffffffff8223e9c0 RDI: ffffffff8223e9ce
RBP: ffff8887068070f8 R08: 2100000000000000 R09: 2f297373656c2072 R10: 930153db8b126e9d R11: 0000000000000fc0 R12: 0000000000000081
R13: 0000000000000000 R14: ffffffff8ZZ3e9c0 R15: 0000000000000018
FS: 0000147707fa2808(0000) GS:ffff88981f280000(0000) knl6S:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CRO: 0000000080050033
CR2: ffff888107ad207f CR3: 0000000543d7c000 CR4: 0000000000750cc0
PKRU: 55555554
Call Trace:
<TASK>
?_die_body+0x1a/0x5c
?page_fault_oops+0x329/0x376
? exc_page_fault+0xf4/0x11d
? asn_exc_page_fault+0x22/0x30
? generic permission+0x21/0x1d2
? inode_pernission+0x90/0x131
? link path_ualk+0x9d/0x308
? path_init+0x107/0x2fc
? path openat+0x129/Oxald
? slab_free_freelist_hook.constprop.0+0x3b/0xaf
? knen_cache_free 0xc9/0x154
? do_filp_open+0x55/0xb8 ? nt_destroy_walk.isra.0+0x101/0x14b ? getnane_flags+0x29/0x152
? knen_cache_alloc+0x122/0x14d
?_raw_spin_unlock+0x14/0x29
? do_sys_openat2+0x6c/0xd9
? do_sys_open+0x3a/0x5a
?do_syscall_64+0x68/0x81
? entry SYSCALL_64_after_Jufrane+0x64/exce
<TASK>
Modules linked in: xt_nat xt_tcpudp veth xt commtrack xt MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrn_algo iptable_nat nf_nat of_conntrack of_defrag_ipus of defrag iput xt addrtype ir_netfilter xfs nfsd auth_rpcgss old_registry
lockd grace sunrpc nd_nod zfs(PO) zunicode(PO) zzstd(0) zlua(0) zavl(PO) icp(PO) zconnon(PO) znupair(PO) spl(0) tcp_diag inet_diag net6683 ipútable_filter ip6_tables iptable filter ip tables x_tables efivarfs
stp lic bonding tis 1915 intel_rapl_nar intel_rapl_connon x86_pkg_temp_thermal intel_powerclamp coretenp kun_intel last_abi den_buddy 12c_algo_bit tin dra display helper kun den kns helper den crct1@dif pelul crc32 peleul creze intel gha
af packet 8821q garp arp bridge
sh_cinulni_intel sha512 ssse3 input_leds aesni_intel led class joydev crypto_sind intel gtt cryptd rapl nei jhdcp net_pxp intel_estate mon_uni uni_baof nune 12c 1601 agpgart alci nei ne ize snbus syscopyarea sysfilirect intel_uncore sysingbl
t igc aacraid iZc_core
nume_core nei libahci thermal fb_sys_fops fan tpn_crb video ton_tis tom_tis_core ton uni backlight intel_pac_core acpi_pad acpi_tad button unix
CRZ: ffff888107ad207f
I end trace 0000000000000000 1-
RIP: 0010:0xffff888107ad207f
Code: 69 6f 45 6e 63 6f 64 65 72 2f 43 6f 6e 76 65 72 74 20 74 6f 20 57 41 56 20 28 74 6f 20 38 63 68 20 6f 72 20 6c 65 73 73 29 2f <34> 34 36 34 61 65 62 33 21 61 34 64 31 2d 34 30 64 64 24 39 36 38
RSP: 0018:ffffc9002ae0fc68 EFLAGS: 00010246
RAX: 0000000000000000 REX: ffffffff8124bda0 RCK: ffff888108107000
RDX: ffff888268727740 RSI: ffffffff8223e9c0 RDI: ffffffff8ZZ3e9c0
RBP: ffff888706807018 R08: 2100000000000000 R09: 21297373656c2072
R10: 934153db8b126e9d R11: 0000000000000fc0 R1Z: 0000000000000081
R13: 0000000000000000 R14: ffffffff8223e9c0 R15: 0000000000000018
FS: 0000147707fa2808(0000) GS:ffff88981f280000(0000) kn1GS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CRO: 0000000000050033
CKZ: ffff888107ad207f CR3: 0000000543d7e000 CR4: 0000000000750cc0
PKRU: 55555554
note: Plex Transcoder [30086] exited with irqs disabled

 

 

For now, I've switched to a new USB boot drive created from the unRAID USB tool using a recent backup (let me know if there's something I should do differently here), and I currently have the server up running another parity sync.

 

I also noticed that one of the two 'systemcache' NVMe drives has the below warning while the other does not. They are the exact same model. Any idea why?

 

 

I really appreciate your assistance.

unraid-diagnostics-20231124-2215.zip

Edited November 25, 20232 yr by barnowan

On 11/25/2023 at 4:17 AM, barnowan said:

I also noticed that one of the two 'systemcache' NVMe drives has the below warning while the other does not. They are the exact same model. Any idea why?

This is normal, panic photo doesn't really help me unfortunately.

Just providing updates in case someone else comes across this in the future.

• Changing out the NVMe cache drives (RAID 1) for a separate, single NVMe drive didn't help.
• The system was stable enough to complete the parity rebuild with 2 of my normally active ~15 docker containers running.
• I really started noticing a trend that when certain dockers and their associated dockers were enabled, the server would freeze.
• I later noticed the omada-controller docker was having odd issues with it dropping APs, unable to adopt, etc. and other containers also seemed to have some odd errors in the logs.
• I deleted the appdata folders for the troublesome dockers and reconfigured them from scratch.  This got those specific dockers working great again.
• The server seemed much more stable (been running all normal dockers for ~24 hours now which is unheard of lately)
• Some other docker containers were having issues with starting, even with the appdata deleted starting fresh along with re-installing them.
• I deleted the docker folder entirely (along with appdata for dockers I didn't notice earlier that were having issues), reinstalled all dockers, and everything is at least up and running now.

Time will tell if this was the solution or not, but it's looking much better now than it has been in a long time.

Edited December 1, 20232 yr by barnowan

One last follow-up.  Everything still seems to be rock solid with everything running.  Seems maybe the docker folder and appdata for certain dockers was bad/corrupt somehow.

 

Thank you, @JorgeB, for helping guide me through this!

@barnowanhaving very similar issues.  When you said you deleted the appdata, did you completely nuke and start from scratch?

@user2579 Not for all apps, just the troublesome ones.

 

The issues I was experiencing ended up coming back though. Something with the RAID 1 NVMe cache pool was causing issues even though SMART status on both drives was fine, no errors were seen in the details, etc. Happened on 3 different motherboards, so I don't think it was a board issue either. I went to a single NVMe drive setup using a different drive for my appdata, and it's been rock solid ever since. I still don't know the true root cause though.

9 hours ago, barnowan said:

@user2579 Not for all apps, just the troublesome ones.

 

The issues I was experiencing ended up coming back though. Something with the RAID 1 NVMe cache pool was causing issues even though SMART status on both drives was fine, no errors were seen in the details, etc. Happened on 3 different motherboards, so I don't think it was a board issue either. I went to a single NVMe drive setup using a different drive for my appdata, and it's been rock solid ever since. I still don't know the true root cause though.

This has been something concerning for me as well.  Everything has been fine for about 6 months and then recently it has progressively gotten worse to the point I have to wipe that raidz pool and start fresh or else I won't make it past mounting disks.  I have new RAM from a different vendor inbound to test that (this past week I got through an 80 hour memtest on the current RAM with no issues).  I re-seated/inspected the CPU, clean there as well.  If the RAM swap doesn't do anything, then yea I'm investigating the NVME.

 

What make/model NVME was giving you trouble?  When you say RAID 1 NVME cache pool, are you talking about raidz1 or raid 1?  On your current stable config, is the single NVME one from your original pool?

Edited January 29, 20242 yr by user2579
specificity

The two in the pool earlier were both 1 TB WD SN850. The single drive I'm using now is a 1 TB WD SN750.

 

I think RAID 1. I just created a btrfs cache pool with the 2 drives, went into pool settings, and made sure they were set to mirror each other. I don't remember the exact settings.