bigbangus

Changed Status to Solved

Ran memtest, got random errors. Noticed I has the two 16GB memory sticks in the A1/B1 slots instead of the correct A2/B2 slots so the XMP profile wasn't stable. Switched them to the correct A2/B2 slots @ 3200MHz and it has been running for days without a single error. Thank you for all the help.

OK I managed to get out OK with just deleting what was affected. Re-ran the scrub and now I see no errors or warning in my log and an exit code 0 Nov 21 13:31:02 UnraidNAS kernel: BTRFS info (device dm-3): scrub: finished on devid 1 with status: 0 However, during lunch I did have another error in my syslog making me wonder if the remaining RAM is still bad or if I need to troubleshoot components like the motherboard or peripherals. Or slow my RAM clock down. Right now I'm 3200MHz which agrees with the B550M Pro4 manual for 2 slots of RAM. Also back in OS 6.9.x I was running this way for a long time with no issues on those same two sticks. Nov 21 12:42:38 UnraidNAS kernel: BUG: Bad page state in process shfs pfn:1befce Nov 21 12:42:38 UnraidNAS kernel: page:000000009ad4d718 refcount:0 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x1befce unraidnas-diagnostics-20221121-1337.zip

not sure what to make of this. SCRUB report has a 0/0/0 error summary, but syslog is littered with warnings and errors SCRUB report: UUID: 8fc31d8d-1584-4ae3-a473-1894b03996f3 Scrub started: Mon Nov 21 11:38:48 2022 Status: finished Duration: 0:11:14 Total to scrub: 492.07GiB Rate: 747.58MiB/s Error summary: read=10 csum=6 Corrected: 0 Uncorrectable: 0 Unverified: 0 Nov 21 11:41:20 UnraidNAS kernel: BTRFS warning (device dm-3): checksum error at logical 3079392104448 on dev /dev/mapper/nvme0n1p1, physical 67479179264, root 5, inode 31329563, offset 8192, length 4096, links 1 (path: appdata/binhex-plex/Plex Media Server/Cache/PhotoTranscoder/97/976adb790e407d0bd4ba7d0dd9809d2dc58d3244.jpg) Nov 21 11:41:20 UnraidNAS kernel: BTRFS error (device dm-3): bdev /dev/mapper/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 3, gen 0 Nov 21 11:41:33 UnraidNAS autofan: Highest disk temp is 38C, adjusting fan speed from: 190 (74% @ 1115rpm) to: 210 (82% @ 1113rpm) Nov 21 11:42:13 UnraidNAS autofan: Highest disk temp is 37C, adjusting fan speed from: 210 (82% @ 1118rpm) to: 190 (74% @ 1034rpm) Nov 21 11:43:18 UnraidNAS autofan: Highest disk temp is 38C, adjusting fan speed from: 190 (74% @ 1028rpm) to: 210 (82% @ 1114rpm) Nov 21 11:44:09 UnraidNAS autofan: Highest disk temp is 38C, adjusting fan speed from: 190 (74% @ 2014rpm) to: 210 (82% @ 1965rpm) Nov 21 11:45:35 UnraidNAS kernel: BTRFS warning (device dm-3): checksum error at logical 3256795521024 on dev /dev/mapper/nvme0n1p1, physical 253472530432, root 5, inode 31329650, offset 499712, length 4096, links 1 (path: appdata/binhex-plex/Plex Media Server/Cache/PhotoTranscoder/e4/e42eaed652dadd123dbe4809b217b70a37fa677d.jpg) Nov 21 11:45:35 UnraidNAS kernel: BTRFS error (device dm-3): bdev /dev/mapper/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 4, gen 0 Nov 21 11:45:54 UnraidNAS kernel: BTRFS warning (device dm-3): checksum error at logical 3280635400192 on dev /dev/mapper/nvme0n1p1, physical 277312409600, root 5, inode 11017814, offset 253952, length 4096, links 1 (path: appdata/binhex-plex/Plex Media Server/Metadata/TV Shows/7/48e11ccd198ef58ee9cd436a4c8e896a5032f03.bundle/Contents/_combined/posters/tv.plex.agents.series_f85b35997126ad291e407c43271f62cee4ffcef1) Nov 21 11:45:54 UnraidNAS kernel: BTRFS error (device dm-3): bdev /dev/mapper/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 5, gen 0 Nov 21 11:46:39 UnraidNAS autofan: Highest disk temp is 37C, adjusting fan speed from: 210 (82% @ 1104rpm) to: 190 (74% @ 1116rpm) Nov 21 11:47:35 UnraidNAS kernel: BTRFS warning (device dm-3): checksum error at logical 3360888418304 on dev /dev/mapper/nvme0n1p1, physical 365081620480, root 5, inode 31329467, offset 827392, length 4096, links 1 (path: appdata/binhex-plex/Plex Media Server/Cache/PhotoTranscoder/36/3675b4617d307130c996a8c288e4ab33ad400e72.jpg) Nov 21 11:47:35 UnraidNAS kernel: BTRFS warning (device dm-3): checksum error at logical 3360887865344 on dev /dev/mapper/nvme0n1p1, physical 365081067520, root 5, inode 31329467, offset 274432, length 4096, links 1 (path: appdata/binhex-plex/Plex Media Server/Cache/PhotoTranscoder/36/3675b4617d307130c996a8c288e4ab33ad400e72.jpg) Nov 21 11:47:35 UnraidNAS kernel: BTRFS error (device dm-3): bdev /dev/mapper/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 6, gen 0 Nov 21 11:47:35 UnraidNAS kernel: BTRFS error (device dm-3): bdev /dev/mapper/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 7, gen 0 Nov 21 11:47:44 UnraidNAS autofan: Highest disk temp is 38C, adjusting fan speed from: 190 (74% @ 1116rpm) to: 210 (82% @ 1112rpm) Nov 21 11:48:45 UnraidNAS kernel: BTRFS warning (device dm-3): checksum error at logical 3406798528512 on dev /dev/mapper/nvme0n1p1, physical 422802890752, root 5, inode 31329505, offset 3088384, length 4096, links 1 (path: appdata/binhex-plex/Plex Media Server/Cache/PhotoTranscoder/bb/bbc8674fca63d0f667f64db95b53fd4f84bb4b68.jpg) Nov 21 11:48:45 UnraidNAS kernel: BTRFS error (device dm-3): bdev /dev/mapper/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 8, gen 0 Nov 21 11:48:56 UnraidNAS kernel: nvme0n1: I/O Cmd(0x2) @ LBA 861063264, 1024 blocks, I/O Error (sct 0x2 / sc 0x81) MORE DNR Nov 21 11:48:56 UnraidNAS kernel: critical medium error, dev nvme0n1, sector 861063264 op 0x0:(READ) flags 0x4000 phys_seg 94 prio class 0 Nov 21 11:48:56 UnraidNAS kernel: nvme0n1: I/O Cmd(0x2) @ LBA 861069408, 1024 blocks, I/O Error (sct 0x2 / sc 0x81) MORE DNR Nov 21 11:48:56 UnraidNAS kernel: critical medium error, dev nvme0n1, sector 861069408 op 0x0:(READ) flags 0x4000 phys_seg 20 prio class 0 Nov 21 11:48:56 UnraidNAS kernel: nvme0n1: I/O Cmd(0x2) @ LBA 861133560, 768 blocks, I/O Error (sct 0x2 / sc 0x81) MORE DNR Nov 21 11:48:56 UnraidNAS kernel: critical medium error, dev nvme0n1, sector 861133560 op 0x0:(READ) flags 0x4000 phys_seg 96 prio class 0 Nov 21 11:48:56 UnraidNAS kernel: nvme0n1: I/O Cmd(0x2) @ LBA 861063336, 8 blocks, I/O Error (sct 0x2 / sc 0x81) MORE DNR Nov 21 11:48:56 UnraidNAS kernel: critical medium error, dev nvme0n1, sector 861063336 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0 Nov 21 11:48:56 UnraidNAS kernel: BTRFS warning (device dm-3): i/o error at logical 3418399789056 on dev /dev/mapper/nvme0n1p1, physical 440846602240, root 5, inode 1304841, offset 31049420800, length 4096, links 1 (path: domains/Ubuntu/vdisk1.img) Nov 21 11:48:56 UnraidNAS kernel: BTRFS error (device dm-3): bdev /dev/mapper/nvme0n1p1 errs: wr 0, rd 1, flush 0, corrupt 8, gen 0 Nov 21 11:48:56 UnraidNAS kernel: nvme0n1: I/O Cmd(0x2) @ LBA 861069512, 8 blocks, I/O Error (sct 0x2 / sc 0x81) MORE DNR Nov 21 11:48:56 UnraidNAS kernel: critical medium error, dev nvme0n1, sector 861069512 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0 Nov 21 11:48:56 UnraidNAS kernel: BTRFS warning (device dm-3): i/o error at logical 3418402951168 on dev /dev/mapper/nvme0n1p1, physical 440849764352, root 5, inode 1304841, offset 31052582912, length 4096, links 1 (path: domains/Ubuntu/vdisk1.img) Nov 21 11:48:56 UnraidNAS kernel: BTRFS error (device dm-3): bdev /dev/mapper/nvme0n1p1 errs: wr 0, rd 2, flush 0, corrupt 8, gen 0 Nov 21 11:48:56 UnraidNAS kernel: nvme0n1: I/O Cmd(0x2) @ LBA 861063368, 8 blocks, I/O Error (sct 0x2 / sc 0x81) MORE DNR Nov 21 11:48:56 UnraidNAS kernel: critical medium error, dev nvme0n1, sector 861063368 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0 Nov 21 11:48:56 UnraidNAS kernel: BTRFS warning (device dm-3): i/o error at logical 3418399805440 on dev /dev/mapper/nvme0n1p1, physical 440846618624, root 5, inode 1304841, offset 31049437184, length 4096, links 1 (path: domains/Ubuntu/vdisk1.img) Nov 21 11:48:56 UnraidNAS kernel: BTRFS error (device dm-3): bdev /dev/mapper/nvme0n1p1 errs: wr 0, rd 3, flush 0, corrupt 8, gen 0 Nov 21 11:48:56 UnraidNAS kernel: nvme0n1: I/O Cmd(0x2) @ LBA 861069520, 8 blocks, I/O Error (sct 0x2 / sc 0x81) MORE DNR Nov 21 11:48:56 UnraidNAS kernel: critical medium error, dev nvme0n1, sector 861069520 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0 Nov 21 11:48:56 UnraidNAS kernel: BTRFS warning (device dm-3): i/o error at logical 3418402955264 on dev /dev/mapper/nvme0n1p1, physical 440849768448, root 5, inode 1304841, offset 31052587008, length 4096, links 1 (path: domains/Ubuntu/vdisk1.img) Nov 21 11:48:56 UnraidNAS kernel: BTRFS error (device dm-3): bdev /dev/mapper/nvme0n1p1 errs: wr 0, rd 4, flush 0, corrupt 8, gen 0 Nov 21 11:48:56 UnraidNAS kernel: nvme0n1: I/O Cmd(0x2) @ LBA 861134120, 8 blocks, I/O Error (sct 0x2 / sc 0x81) MORE DNR Nov 21 11:48:56 UnraidNAS kernel: critical medium error, dev nvme0n1, sector 861134120 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0 Nov 21 11:48:56 UnraidNAS kernel: BTRFS warning (device dm-3): i/o error at logical 3418436030464 on dev /dev/mapper/nvme0n1p1, physical 440882843648, root 5, inode 1304841, offset 23452639232, length 4096, links 1 (path: domains/Ubuntu/vdisk1.img) Nov 21 11:48:56 UnraidNAS kernel: BTRFS error (device dm-3): bdev /dev/mapper/nvme0n1p1 errs: wr 0, rd 5, flush 0, corrupt 8, gen 0 Nov 21 11:48:56 UnraidNAS kernel: nvme0n1: I/O Cmd(0x2) @ LBA 861063376, 8 blocks, I/O Error (sct 0x2 / sc 0x81) MORE DNR Nov 21 11:48:56 UnraidNAS kernel: critical medium error, dev nvme0n1, sector 861063376 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0 Nov 21 11:48:56 UnraidNAS kernel: BTRFS warning (device dm-3): i/o error at logical 3418399809536 on dev /dev/mapper/nvme0n1p1, physical 440846622720, root 5, inode 1304841, offset 31049441280, length 4096, links 1 (path: domains/Ubuntu/vdisk1.img) Nov 21 11:48:56 UnraidNAS kernel: BTRFS error (device dm-3): bdev /dev/mapper/nvme0n1p1 errs: wr 0, rd 6, flush 0, corrupt 8, gen 0 Nov 21 11:48:56 UnraidNAS kernel: nvme0n1: I/O Cmd(0x2) @ LBA 861134136, 8 blocks, I/O Error (sct 0x2 / sc 0x81) MORE DNR Nov 21 11:48:56 UnraidNAS kernel: critical medium error, dev nvme0n1, sector 861134136 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0 Nov 21 11:48:56 UnraidNAS kernel: BTRFS warning (device dm-3): i/o error at logical 3418436038656 on dev /dev/mapper/nvme0n1p1, physical 440882851840, root 5, inode 1304841, offset 23452647424, length 4096, links 1 (path: domains/Ubuntu/vdisk1.img) Nov 21 11:48:56 UnraidNAS kernel: BTRFS error (device dm-3): bdev /dev/mapper/nvme0n1p1 errs: wr 0, rd 7, flush 0, corrupt 8, gen 0 Nov 21 11:48:56 UnraidNAS kernel: BTRFS warning (device dm-3): i/o error at logical 3418399813632 on dev /dev/mapper/nvme0n1p1, physical 440846626816, root 5, inode 1304841, offset 31049445376, length 4096, links 1 (path: domains/Ubuntu/vdisk1.img) Nov 21 11:48:56 UnraidNAS kernel: BTRFS error (device dm-3): bdev /dev/mapper/nvme0n1p1 errs: wr 0, rd 8, flush 0, corrupt 8, gen 0 Nov 21 11:48:56 UnraidNAS kernel: BTRFS warning (device dm-3): i/o error at logical 3418436042752 on dev /dev/mapper/nvme0n1p1, physical 440882855936, root 5, inode 1304841, offset 23452651520, length 4096, links 1 (path: domains/Ubuntu/vdisk1.img) Nov 21 11:48:56 UnraidNAS kernel: BTRFS error (device dm-3): bdev /dev/mapper/nvme0n1p1 errs: wr 0, rd 9, flush 0, corrupt 8, gen 0 Nov 21 11:48:56 UnraidNAS kernel: BTRFS warning (device dm-3): i/o error at logical 3418399817728 on dev /dev/mapper/nvme0n1p1, physical 440846630912, root 5, inode 1304841, offset 31049449472, length 4096, links 1 (path: domains/Ubuntu/vdisk1.img) Nov 21 11:48:56 UnraidNAS kernel: BTRFS error (device dm-3): bdev /dev/mapper/nvme0n1p1 errs: wr 0, rd 10, flush 0, corrupt 8, gen 0 Nov 21 11:50:02 UnraidNAS kernel: BTRFS info (device dm-3): scrub: finished on devid 1 with status: 0

ok scrub in process thanks. I did a inode find and deleted the affected file prior. root@UnraidNAS:~# find /mnt/cache -inum 17390531 /mnt/cache/appdata/binhex-radarr/logs/radarr.39.txt root@UnraidNAS:~# cd /mnt/cache/appdata/binhex-radarr/logs/ root@UnraidNAS:/mnt/cache/appdata/binhex-radarr/logs# rm radarr.39.txt root@UnraidNAS:/mnt/cache/appdata/binhex-radarr/logs#

Server still going strong, but noticed there was an error this morning that I overlooked. @JorgeB Is there something I need to dig into to? Nov 21 04:17:20 UnraidNAS CA Backup/Restore: Verifying backup Nov 21 04:17:20 UnraidNAS CA Backup/Restore: Using command: cd '/mnt/user/appdata/' && /usr/bin/tar --diff -C '/mnt/user/appdata/' -af '/mnt/user/backups/appdata/2022-11-21@04.00/CA_backup.tar.gz' > /var/lib/docker/unraid/ca.backup2.datastore/appdata_backup.log & echo $! > /tmp/ca.backup2/tempFiles/verifyInProgress Nov 21 04:18:00 UnraidNAS kernel: BTRFS warning (device dm-3): csum failed root 5 ino 17390531 off 327680 csum 0x32c11707 expected csum 0x43348dfc mirror 1 Nov 21 04:18:00 UnraidNAS kernel: BTRFS error (device dm-3): bdev /dev/mapper/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0 Should I do a BTRFS scrub or something? unraidnas-diagnostics-20221121-1127.zip

After several more crashes not reported here yesterday, I pulled the 2nd half of my RAM out and now the server runs 100% for 12 hours. Thank you! Do you think the bad RAM could be responsible for the previous nvme / usb device crashes I've had this year:

Had another crash with the VM off, then decided to leave off compreface-gpu and double-take dockers knowing these guys are heavy lifters (really no evidence says to turn them off, just a hunch). Then left the house, came home to my zigbee2mqtt docker dead. The log said something about write-only only. Looking at the pfsense remote syslog (see attached): Nov 20 12:30:34 unraid kernel: BTRFS critical (device dm-3): corrupt leaf: root=7 block=3573797388288 slot=263, unaligned key offset for csum item, have 3069387368448 should be aligned to 4096 Nov 20 12:30:34 unraid kernel: BTRFS info (device dm-3): leaf 3573797388288 gen 2132420 total ptrs 329 free space 4214 owner 7 Nov 20 12:30:34 unraid kernel: item 0 key (18446744073709551606 128 3069384982528) itemoff 16271 itemsize 12 Nov 20 12:30:34 unraid kernel: item 1 key (18446744073709551606 128 3069384994816) itemoff 16267 itemsize 4 Nov 20 12:30:34 unraid kernel: item 2 key (18446744073709551606 128 3069384998912) itemoff 16259 itemsize 8 Nov 20 12:30:34 unraid kernel: item 3 key (18446744073709551606 128 3069385007104) itemoff 16243 itemsize 16 Then: Nov 20 12:30:34 unraid kernel: BTRFS error (device dm-3): block=3573797388288 write time tree block corruption detected Nov 20 12:30:34 unraid kernel: BTRFS: error (device dm-3) in btrfs_commit_transaction:2418: errno=-5 IO failure (Error while writing out transaction) Nov 20 12:30:34 unraid kernel: BTRFS info (device dm-3: state E): forced readonly Nov 20 12:30:34 unraid kernel: BTRFS warning (device dm-3: state E): Skipping commit of aborted transaction. Nov 20 12:30:34 unraid kernel: BTRFS: error (device dm-3: state EA) in cleanup_transaction:1982: errno=-5 IO failure Nov 20 12:30:34 unraid kernel: I/O error, dev loop2, sector 1953024 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0 so I noticed 6.11.5 what the heck why not I upgrade the OS to that. also in pure desperation I removed the following append from my flash: nvme_core.default_ps_max_latency_us=0 pcie_aspm=off this was suggested in the log from a previous crash this year. Do I just have a shitty nvme here? pfsense-unraid2022-11-20_morecrash.log

yeah def provide diags and logs on a dedicated thread if possible to the ops to solve it. otherwise it's not possible to know if the issues are related.

OK now crashing on 6.11.3 when I launched my VM. Now I am a bit confused, especially since I've been on 6.11.3 for over a week with my win10 running no issue. See attached for crash log and diags. Nov 20 07:35:24 unraid kernel: BUG: Bad page state in process uwsgi pfn:4b47cc Nov 20 07:35:24 unraid kernel: page:000000000856407b refcount:0 mapcount:0 mapping:0000000000000000 index:0x1 pfn:0x4b47cc Nov 20 07:35:24 unraid kernel: flags: 0x2ffff0000000008(dirty|node=0|zone=2|lastcpupid=0xffff) Nov 20 07:35:24 unraid kernel: raw: 02ffff0000000008 dead000000000100 dead000000000122 0000000000000000 Nov 20 07:35:24 unraid kernel: raw: 0000000000000001 0000000000000000 00000000ffffffff 0000000000000000 Nov 20 07:35:24 unraid kernel: page dumped because: PAGE_FLAGS_CHECK_AT_PREP flag(s) set Nov 20 07:35:24 unraid kernel: Modules linked in: af_packet nvidia_uvm(PO) xt_nat veth xt_CHECKSUM ipt_REJECT nf_reject_ipv4 xt_tcpudp ip6table_mangle ip6table_nat iptable_mangle vhost_net tun vhost vhost_iotlb tap xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 br_netfilter xfs dm_crypt dm_mod dax md_mod nct6775 nct6775_core hwmon_vid efivarfs ip6table_filter ip6_tables iptable_filter ip_tables x_tables bridge stp llc bonding tls wmi_bmof nvidia_drm(PO) nvidia_modeset(PO) edac_mce_amd edac_core nvidia(PO) kvm_amd drm_kms_helper kvm drm backlight crct10dif_pclmul i2c_piix4 crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl r8169 nvme i2c_core k10temp ccp syscopyarea ahci sysfillrect nvme_core realtek joydev sysimgblt libahci fb_sys_fops wmi tpm_crb tpm_tis tpm_tis_core tpm acpi_cpufreq button unix Nov 20 07:35:24 unraid kernel: CPU: 1 PID: 25802 Comm: uwsgi Tainted: P B O 5.19.17-Unraid #2 Nov 20 07:35:24 unraid kernel: Hardware name: To Be Filled By O.E.M. B550M Pro4/B550M Pro4, BIOS P2.30 02/24/2022 pfsense-unraid2022-11-20_6.11.3.log unraidnas-diagnostics-20221120-0833.zip

Crashed again. Not sure what to do anymore but remove my google coral for a month and see if it stabilizes? Or maybe try to relocate my Google Coral to another usb hub? See crash log (captured on remote pfSense syslog-ng server) and latest diagnostics below. Nov 4 07:07:59 unraid kernel: xhci_hcd 0000:02:00.0: WARN Set TR Deq Ptr cmd failed due to incorrect slot or ep state. Nov 4 07:07:59 unraid kernel: xhci_hcd 0000:02:00.0: WARN Successful completion on short TX Nov 4 07:07:59 unraid kernel: xhci_hcd 0000:02:00.0: ERROR Transfer event TRB DMA ptr not part of current TD ep_index 2 comp_code 1 unraidnas-diagnostics-20221104-0736.zip unraidnas_pfSense_syslog-ng_crash_2022-11-04_070800.log

Server crash, but seems unrelated. Seems like the more common [02:00.0] usb hub crash I've been having. I will follow up on other thread but thought it was worth mentioning it here... Nov 4 07:07:59 unraid kernel: xhci_hcd 0000:02:00.0: WARN Set TR Deq Ptr cmd failed due to incorrect slot or ep state. Nov 4 07:07:59 unraid kernel: xhci_hcd 0000:02:00.0: WARN Successful completion on short TX Nov 4 07:07:59 unraid kernel: xhci_hcd 0000:02:00.0: ERROR Transfer event TRB DMA ptr not part of current TD ep_index 2 comp_code 1 Nov 4 07:07:59 unraid kernel: xhci_hcd 0000:02:00.0: Looking for event-dma 000000026ad7d410 trb-start 000000026ad7d420 trb-end 000000026ad7d420 seg-start 000000026ad7d000 seg-end 000000026ad7dff0 Nov 4 07:08:00 unraid kernel: general protection fault, probably for non-canonical address 0xcec7beccd6afa8c1: 0000 [#1] PREEMPT SMP NOPTI Nov 4 07:08:00 unraid kernel: CPU: 6 PID: 16336 Comm: nginx Tainted: P O 5.19.14-Unraid #1 Nov 4 07:08:00 unraid kernel: Hardware name: To Be Filled By O.E.M. B550M Pro4/B550M Pro4, BIOS P2.30 02/24/2022 Nov 4 07:08:00 unraid kernel: RIP: 0010:__kmalloc_node_track_caller+0x126/0x1d9 Nov 4 07:08:00 unraid kernel: Code: 19 8b 54 24 04 4c 89 f9 4c 89 e7 8b 34 24 e8 78 fe ff ff 48 89 44 24 10 eb 2c 41 8b 44 24 28 48 8d 8a 00 01 00 00 49 8b 3c 24 <49> 8b 1c 06 4c 89 f0 65 48 0f c7 0f 0f 94 c0 84 c0 74 87 41 8b 44 Nov 4 07:08:00 unraid kernel: RSP: 0018:ffffc9000448fbf0 EFLAGS: 00010202 Nov 4 07:08:00 unraid kernel: RAX: 0000000000000200 RBX: ffff8881a4a91c00 RCX: 000000153c2fed06 Nov 4 07:08:00 unraid kernel: RDX: 000000153c2fec06 RSI: 00000000ffffffff RDI: 000000000002f650 Nov 4 07:08:00 unraid kernel: RBP: ffff888100042b00 R08: 0000000000082a20 R09: 0000000000000000 Nov 4 07:08:00 unraid kernel: R10: ffffc9000448fe40 R11: 0000000000000000 R12: ffff888100042b00 Nov 4 07:08:00 unraid kernel: R13: 0000000000000280 R14: cec7beccd6afa6c1 R15: ffffffff816b81f4 Nov 4 07:08:00 unraid kernel: FS: 00001513ba288740(0000) GS:ffff88900e980000(0000) knlGS:0000000000000000 Nov 4 07:08:00 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 4 07:08:00 unraid kernel: CR2: 000000c0000ea000 CR3: 0000000fd72a8000 CR4: 0000000000350ee0 Nov 4 07:08:00 unraid kernel: Call Trace: Nov 4 07:08:00 unraid kernel: <TASK> Nov 4 07:08:00 unraid kernel: kmalloc_reserve+0x2d/0x73 Nov 4 07:08:00 unraid kernel: __alloc_skb+0xb2/0x15e Nov 4 07:08:00 unraid kernel: ? preempt_latency_start+0x2b/0x46 Nov 4 07:08:00 unraid kernel: __tcp_send_ack+0x3b/0xdc Nov 4 07:08:00 unraid kernel: tcp_recvmsg_locked+0x6a1/0x6cf Nov 4 07:08:00 unraid kernel: tcp_recvmsg+0x101/0x1a2 Nov 4 07:08:00 unraid kernel: inet_recvmsg+0x69/0xa9 Nov 4 07:08:00 unraid kernel: __sys_recvfrom+0x97/0xf8 Nov 4 07:08:00 unraid kernel: __x64_sys_recvfrom+0x20/0x27 Nov 4 07:08:00 unraid kernel: do_syscall_64+0x6b/0x81 Nov 4 07:08:00 unraid kernel: entry_SYSCALL_64_after_hwframe+0x63/0xcd Nov 4 07:08:00 unraid kernel: RIP: 0033:0x1513bca9f5b0 Nov 4 07:08:00 unraid kernel: Code: 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 41 89 ca 64 8b 04 25 18 00 00 00 85 c0 75 1d 45 31 c9 45 31 c0 b8 2d 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 68 c3 0f 1f 80 00 00 00 00 55 48 83 ec 20 48 Nov 4 07:08:00 unraid kernel: RSP: 002b:00007ffc6a372808 EFLAGS: 00000246 ORIG_RAX: 000000000000002d Nov 4 07:08:00 unraid kernel: RAX: ffffffffffffffda RBX: 0000000000001012 RCX: 00001513bca9f5b0 Nov 4 07:08:00 unraid kernel: RDX: 0000000000001012 RSI: 0000560813db8e80 RDI: 0000000000000010 Nov 4 07:08:00 unraid kernel: RBP: 0000151399cb8400 R08: 0000000000000000 R09: 0000000000000000 Nov 4 07:08:00 unraid kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 Nov 4 07:08:00 unraid kernel: R13: 0000560813db8e80 R14: 0000560813aee180 R15: 0000560813d4d4c0 Nov 4 07:08:00 unraid kernel: </TASK> Nov 4 07:08:00 unraid kernel: Modules linked in: xt_mark af_packet nvidia_uvm(PO) xt_nat veth xt_CHECKSUM ipt_REJECT nf_reject_ipv4 xt_tcpudp ip6table_mangle ip6table_nat iptable_mangle vhost_net tun vhost vhost_iotlb tap xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 br_netfilter xfs dm_crypt dm_mod dax md_mod nct6775 nct6775_core hwmon_vid efivarfs ip6table_filter ip6_tables iptable_filter ip_tables x_tables bridge stp llc bonding tls ipv6 wmi_bmof edac_mce_amd edac_core nvidia_drm(PO) nvidia_modeset(PO) kvm_amd nvidia(PO) kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel drm_kms_helper aesni_intel crypto_simd cryptd rapl drm r8169 i2c_piix4 ccp k10temp nvme backlight realtek i2c_core joydev ahci nvme_core syscopyarea sysfillrect sysimgblt libahci fb_sys_fops wmi tpm_crb tpm_tis tpm_tis_core tpm acpi_cpufreq button unix Nov 4 07:08:00 unraid kernel: ---[ end trace 0000000000000000 ]--- Nov 4 07:08:00 unraid kernel: RIP: 0010:__kmalloc_node_track_caller+0x126/0x1d9 Nov 4 07:08:00 unraid kernel: Code: 19 8b 54 24 04 4c 89 f9 4c 89 e7 8b 34 24 e8 78 fe ff ff 48 89 44 24 10 eb 2c 41 8b 44 24 28 48 8d 8a 00 01 00 00 49 8b 3c 24 <49> 8b 1c 06 4c 89 f0 65 48 0f c7 0f 0f 94 c0 84 c0 74 87 41 8b 44 Nov 4 07:08:00 unraid kernel: RSP: 0018:ffffc9000448fbf0 EFLAGS: 00010202 Nov 4 07:08:00 unraid kernel: RAX: 0000000000000200 RBX: ffff8881a4a91c00 RCX: 000000153c2fed06 Nov 4 07:08:00 unraid kernel: RDX: 000000153c2fec06 RSI: 00000000ffffffff RDI: 000000000002f650 Nov 4 07:08:00 unraid kernel: RBP: ffff888100042b00 R08: 0000000000082a20 R09: 0000000000000000 Nov 4 07:08:00 unraid kernel: R10: ffffc9000448fe40 R11: 0000000000000000 R12: ffff888100042b00 Nov 4 07:08:00 unraid kernel: R13: 0000000000000280 R14: cec7beccd6afa6c1 R15: ffffffff816b81f4 Nov 4 07:08:00 unraid kernel: FS: 00001513ba288740(0000) GS:ffff88900e980000(0000) knlGS:0000000000000000 Nov 4 07:08:00 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 4 07:08:00 unraid kernel: CR2: 000000c0000ea000 CR3: 0000000fd72a8000 CR4: 0000000000350ee0 Nov 4 07:08:00 unraid kernel: general protection fault, probably for non-canonical address 0xcec7beccd6afa8c1: 0000 [#2] PREEMPT SMP NOPTI Nov 4 07:08:00 unraid kernel: CPU: 6 PID: 3055 Comm: kworker/6:0 Tainted: P D O 5.19.14-Unraid #1 Nov 4 07:08:00 unraid kernel: Hardware name: To Be Filled By O.E.M. B550M Pro4/B550M Pro4, BIOS P2.30 02/24/2022 Nov 4 07:08:00 unraid kernel: Workqueue: events efi_pstore_update_entries Nov 4 07:08:00 unraid kernel: RIP: 0010:__kmalloc+0xf2/0x19e Nov 4 07:08:00 unraid kernel: Code: 00 48 89 04 24 74 05 48 85 c0 75 17 4c 89 f9 83 ca ff 44 89 e6 48 89 ef e8 50 f9 ff ff 48 89 04 24 eb 25 8b 4d 28 48 8b 7d 00 <48> 8b 1c 08 48 8d 8a 00 01 00 00 65 48 0f c7 0f 0f 94 c0 84 c0 74 Nov 4 07:08:00 unraid kernel: RSP: 0018:ffffc90003effd90 EFLAGS: 00010282 Nov 4 07:08:00 unraid kernel: RAX: cec7beccd6afa6c1 RBX: ffff888102eaf000 RCX: 0000000000000200 Nov 4 07:08:00 unraid kernel: RDX: 000000153c2fec06 RSI: 0000000000000dc0 RDI: 000000000002f650 Nov 4 07:08:00 unraid kernel: RBP: ffff888100042b00 R08: 0000000000000dc0 R09: 0000000000000001 Nov 4 07:08:00 unraid kernel: R10: 8080808080808080 R11: fefefefefefefeff R12: 0000000000000dc0 Nov 4 07:08:00 unraid kernel: R13: ffff888100042b00 R14: 0000000000000400 R15: ffffffff8168a2f0 Nov 4 07:08:00 unraid kernel: FS: 0000000000000000(0000) GS:ffff88900e980000(0000) knlGS:0000000000000000 Nov 4 07:08:00 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 4 07:08:00 unraid kernel: CR2: 000000c0000ea000 CR3: 0000000fd72a8000 CR4: 0000000000350ee0 Nov 4 07:08:00 unraid kernel: Call Trace: Nov 4 07:08:00 unraid kernel: <TASK> Nov 4 07:08:00 unraid kernel: efivar_init+0x78/0x330 Nov 4 07:08:00 unraid kernel: ? efi_pstore_read_func+0x275/0x275 Nov 4 07:08:00 unraid kernel: ? efi_pstore_update_entries+0x1c/0x67 Nov 4 07:08:00 unraid kernel: ? kmem_cache_alloc_trace+0x11e/0x149 Nov 4 07:08:00 unraid kernel: efi_pstore_update_entries+0x3c/0x67 Nov 4 07:08:00 unraid kernel: process_one_work+0x1ab/0x295 Nov 4 07:08:00 unraid kernel: worker_thread+0x18b/0x244 Nov 4 07:08:00 unraid kernel: ? rescuer_thread+0x281/0x281 Nov 4 07:08:00 unraid kernel: kthread+0xe7/0xef Nov 4 07:08:00 unraid kernel: ? kthread_complete_and_exit+0x1b/0x1b Nov 4 07:08:00 unraid kernel: ret_from_fork+0x22/0x30 Nov 4 07:08:00 unraid kernel: </TASK> Nov 4 07:08:00 unraid kernel: Modules linked in: xt_mark af_packet nvidia_uvm(PO) xt_nat veth xt_CHECKSUM ipt_REJECT nf_reject_ipv4 xt_tcpudp ip6table_mangle ip6table_nat iptable_mangle vhost_net tun vhost vhost_iotlb tap xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 br_netfilter xfs dm_crypt dm_mod dax md_mod nct6775 nct6775_core hwmon_vid efivarfs ip6table_filter ip6_tables iptable_filter ip_tables x_tables bridge stp llc bonding tls ipv6 wmi_bmof edac_mce_amd edac_core nvidia_drm(PO) nvidia_modeset(PO) kvm_amd nvidia(PO) kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel drm_kms_helper aesni_intel crypto_simd cryptd rapl drm r8169 i2c_piix4 ccp k10temp nvme backlight realtek i2c_core joydev ahci nvme_core syscopyarea sysfillrect sysimgblt libahci fb_sys_fops wmi tpm_crb tpm_tis tpm_tis_core tpm acpi_cpufreq button unix Nov 4 07:08:00 unraid kernel: ---[ end trace 0000000000000000 ]---

Going strong for 5 days... will update here again if/when it ever crashes.

What's your motherboard and BIOS version?