bigbangus

Members
  • Posts

    201
  • Joined

  • Last visited

1 Follower

Recent Profile Visitors

The recent visitors block is disabled and is not being shown to other users.

bigbangus's Achievements

Explorer

Explorer (4/14)

32

Reputation

1

Community Answers

  1. Ran memtest, got random errors. Noticed I has the two 16GB memory sticks in the A1/B1 slots instead of the correct A2/B2 slots so the XMP profile wasn't stable. Switched them to the correct A2/B2 slots @ 3200MHz and it has been running for days without a single error. Thank you for all the help.
  2. OK I managed to get out OK with just deleting what was affected. Re-ran the scrub and now I see no errors or warning in my log and an exit code 0 Nov 21 13:31:02 UnraidNAS kernel: BTRFS info (device dm-3): scrub: finished on devid 1 with status: 0 However, during lunch I did have another error in my syslog making me wonder if the remaining RAM is still bad or if I need to troubleshoot components like the motherboard or peripherals. Or slow my RAM clock down. Right now I'm 3200MHz which agrees with the B550M Pro4 manual for 2 slots of RAM. Also back in OS 6.9.x I was running this way for a long time with no issues on those same two sticks. Nov 21 12:42:38 UnraidNAS kernel: BUG: Bad page state in process shfs pfn:1befce Nov 21 12:42:38 UnraidNAS kernel: page:000000009ad4d718 refcount:0 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x1befce unraidnas-diagnostics-20221121-1337.zip
  3. not sure what to make of this. SCRUB report has a 0/0/0 error summary, but syslog is littered with warnings and errors SCRUB report: UUID: 8fc31d8d-1584-4ae3-a473-1894b03996f3 Scrub started: Mon Nov 21 11:38:48 2022 Status: finished Duration: 0:11:14 Total to scrub: 492.07GiB Rate: 747.58MiB/s Error summary: read=10 csum=6 Corrected: 0 Uncorrectable: 0 Unverified: 0 Nov 21 11:41:20 UnraidNAS kernel: BTRFS warning (device dm-3): checksum error at logical 3079392104448 on dev /dev/mapper/nvme0n1p1, physical 67479179264, root 5, inode 31329563, offset 8192, length 4096, links 1 (path: appdata/binhex-plex/Plex Media Server/Cache/PhotoTranscoder/97/976adb790e407d0bd4ba7d0dd9809d2dc58d3244.jpg) Nov 21 11:41:20 UnraidNAS kernel: BTRFS error (device dm-3): bdev /dev/mapper/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 3, gen 0 Nov 21 11:41:33 UnraidNAS autofan: Highest disk temp is 38C, adjusting fan speed from: 190 (74% @ 1115rpm) to: 210 (82% @ 1113rpm) Nov 21 11:42:13 UnraidNAS autofan: Highest disk temp is 37C, adjusting fan speed from: 210 (82% @ 1118rpm) to: 190 (74% @ 1034rpm) Nov 21 11:43:18 UnraidNAS autofan: Highest disk temp is 38C, adjusting fan speed from: 190 (74% @ 1028rpm) to: 210 (82% @ 1114rpm) Nov 21 11:44:09 UnraidNAS autofan: Highest disk temp is 38C, adjusting fan speed from: 190 (74% @ 2014rpm) to: 210 (82% @ 1965rpm) Nov 21 11:45:35 UnraidNAS kernel: BTRFS warning (device dm-3): checksum error at logical 3256795521024 on dev /dev/mapper/nvme0n1p1, physical 253472530432, root 5, inode 31329650, offset 499712, length 4096, links 1 (path: appdata/binhex-plex/Plex Media Server/Cache/PhotoTranscoder/e4/e42eaed652dadd123dbe4809b217b70a37fa677d.jpg) Nov 21 11:45:35 UnraidNAS kernel: BTRFS error (device dm-3): bdev /dev/mapper/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 4, gen 0 Nov 21 11:45:54 UnraidNAS kernel: BTRFS warning (device dm-3): checksum error at logical 3280635400192 on dev /dev/mapper/nvme0n1p1, physical 277312409600, root 5, inode 11017814, offset 253952, length 4096, links 1 (path: appdata/binhex-plex/Plex Media Server/Metadata/TV Shows/7/48e11ccd198ef58ee9cd436a4c8e896a5032f03.bundle/Contents/_combined/posters/tv.plex.agents.series_f85b35997126ad291e407c43271f62cee4ffcef1) Nov 21 11:45:54 UnraidNAS kernel: BTRFS error (device dm-3): bdev /dev/mapper/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 5, gen 0 Nov 21 11:46:39 UnraidNAS autofan: Highest disk temp is 37C, adjusting fan speed from: 210 (82% @ 1104rpm) to: 190 (74% @ 1116rpm) Nov 21 11:47:35 UnraidNAS kernel: BTRFS warning (device dm-3): checksum error at logical 3360888418304 on dev /dev/mapper/nvme0n1p1, physical 365081620480, root 5, inode 31329467, offset 827392, length 4096, links 1 (path: appdata/binhex-plex/Plex Media Server/Cache/PhotoTranscoder/36/3675b4617d307130c996a8c288e4ab33ad400e72.jpg) Nov 21 11:47:35 UnraidNAS kernel: BTRFS warning (device dm-3): checksum error at logical 3360887865344 on dev /dev/mapper/nvme0n1p1, physical 365081067520, root 5, inode 31329467, offset 274432, length 4096, links 1 (path: appdata/binhex-plex/Plex Media Server/Cache/PhotoTranscoder/36/3675b4617d307130c996a8c288e4ab33ad400e72.jpg) Nov 21 11:47:35 UnraidNAS kernel: BTRFS error (device dm-3): bdev /dev/mapper/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 6, gen 0 Nov 21 11:47:35 UnraidNAS kernel: BTRFS error (device dm-3): bdev /dev/mapper/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 7, gen 0 Nov 21 11:47:44 UnraidNAS autofan: Highest disk temp is 38C, adjusting fan speed from: 190 (74% @ 1116rpm) to: 210 (82% @ 1112rpm) Nov 21 11:48:45 UnraidNAS kernel: BTRFS warning (device dm-3): checksum error at logical 3406798528512 on dev /dev/mapper/nvme0n1p1, physical 422802890752, root 5, inode 31329505, offset 3088384, length 4096, links 1 (path: appdata/binhex-plex/Plex Media Server/Cache/PhotoTranscoder/bb/bbc8674fca63d0f667f64db95b53fd4f84bb4b68.jpg) Nov 21 11:48:45 UnraidNAS kernel: BTRFS error (device dm-3): bdev /dev/mapper/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 8, gen 0 Nov 21 11:48:56 UnraidNAS kernel: nvme0n1: I/O Cmd(0x2) @ LBA 861063264, 1024 blocks, I/O Error (sct 0x2 / sc 0x81) MORE DNR Nov 21 11:48:56 UnraidNAS kernel: critical medium error, dev nvme0n1, sector 861063264 op 0x0:(READ) flags 0x4000 phys_seg 94 prio class 0 Nov 21 11:48:56 UnraidNAS kernel: nvme0n1: I/O Cmd(0x2) @ LBA 861069408, 1024 blocks, I/O Error (sct 0x2 / sc 0x81) MORE DNR Nov 21 11:48:56 UnraidNAS kernel: critical medium error, dev nvme0n1, sector 861069408 op 0x0:(READ) flags 0x4000 phys_seg 20 prio class 0 Nov 21 11:48:56 UnraidNAS kernel: nvme0n1: I/O Cmd(0x2) @ LBA 861133560, 768 blocks, I/O Error (sct 0x2 / sc 0x81) MORE DNR Nov 21 11:48:56 UnraidNAS kernel: critical medium error, dev nvme0n1, sector 861133560 op 0x0:(READ) flags 0x4000 phys_seg 96 prio class 0 Nov 21 11:48:56 UnraidNAS kernel: nvme0n1: I/O Cmd(0x2) @ LBA 861063336, 8 blocks, I/O Error (sct 0x2 / sc 0x81) MORE DNR Nov 21 11:48:56 UnraidNAS kernel: critical medium error, dev nvme0n1, sector 861063336 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0 Nov 21 11:48:56 UnraidNAS kernel: BTRFS warning (device dm-3): i/o error at logical 3418399789056 on dev /dev/mapper/nvme0n1p1, physical 440846602240, root 5, inode 1304841, offset 31049420800, length 4096, links 1 (path: domains/Ubuntu/vdisk1.img) Nov 21 11:48:56 UnraidNAS kernel: BTRFS error (device dm-3): bdev /dev/mapper/nvme0n1p1 errs: wr 0, rd 1, flush 0, corrupt 8, gen 0 Nov 21 11:48:56 UnraidNAS kernel: nvme0n1: I/O Cmd(0x2) @ LBA 861069512, 8 blocks, I/O Error (sct 0x2 / sc 0x81) MORE DNR Nov 21 11:48:56 UnraidNAS kernel: critical medium error, dev nvme0n1, sector 861069512 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0 Nov 21 11:48:56 UnraidNAS kernel: BTRFS warning (device dm-3): i/o error at logical 3418402951168 on dev /dev/mapper/nvme0n1p1, physical 440849764352, root 5, inode 1304841, offset 31052582912, length 4096, links 1 (path: domains/Ubuntu/vdisk1.img) Nov 21 11:48:56 UnraidNAS kernel: BTRFS error (device dm-3): bdev /dev/mapper/nvme0n1p1 errs: wr 0, rd 2, flush 0, corrupt 8, gen 0 Nov 21 11:48:56 UnraidNAS kernel: nvme0n1: I/O Cmd(0x2) @ LBA 861063368, 8 blocks, I/O Error (sct 0x2 / sc 0x81) MORE DNR Nov 21 11:48:56 UnraidNAS kernel: critical medium error, dev nvme0n1, sector 861063368 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0 Nov 21 11:48:56 UnraidNAS kernel: BTRFS warning (device dm-3): i/o error at logical 3418399805440 on dev /dev/mapper/nvme0n1p1, physical 440846618624, root 5, inode 1304841, offset 31049437184, length 4096, links 1 (path: domains/Ubuntu/vdisk1.img) Nov 21 11:48:56 UnraidNAS kernel: BTRFS error (device dm-3): bdev /dev/mapper/nvme0n1p1 errs: wr 0, rd 3, flush 0, corrupt 8, gen 0 Nov 21 11:48:56 UnraidNAS kernel: nvme0n1: I/O Cmd(0x2) @ LBA 861069520, 8 blocks, I/O Error (sct 0x2 / sc 0x81) MORE DNR Nov 21 11:48:56 UnraidNAS kernel: critical medium error, dev nvme0n1, sector 861069520 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0 Nov 21 11:48:56 UnraidNAS kernel: BTRFS warning (device dm-3): i/o error at logical 3418402955264 on dev /dev/mapper/nvme0n1p1, physical 440849768448, root 5, inode 1304841, offset 31052587008, length 4096, links 1 (path: domains/Ubuntu/vdisk1.img) Nov 21 11:48:56 UnraidNAS kernel: BTRFS error (device dm-3): bdev /dev/mapper/nvme0n1p1 errs: wr 0, rd 4, flush 0, corrupt 8, gen 0 Nov 21 11:48:56 UnraidNAS kernel: nvme0n1: I/O Cmd(0x2) @ LBA 861134120, 8 blocks, I/O Error (sct 0x2 / sc 0x81) MORE DNR Nov 21 11:48:56 UnraidNAS kernel: critical medium error, dev nvme0n1, sector 861134120 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0 Nov 21 11:48:56 UnraidNAS kernel: BTRFS warning (device dm-3): i/o error at logical 3418436030464 on dev /dev/mapper/nvme0n1p1, physical 440882843648, root 5, inode 1304841, offset 23452639232, length 4096, links 1 (path: domains/Ubuntu/vdisk1.img) Nov 21 11:48:56 UnraidNAS kernel: BTRFS error (device dm-3): bdev /dev/mapper/nvme0n1p1 errs: wr 0, rd 5, flush 0, corrupt 8, gen 0 Nov 21 11:48:56 UnraidNAS kernel: nvme0n1: I/O Cmd(0x2) @ LBA 861063376, 8 blocks, I/O Error (sct 0x2 / sc 0x81) MORE DNR Nov 21 11:48:56 UnraidNAS kernel: critical medium error, dev nvme0n1, sector 861063376 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0 Nov 21 11:48:56 UnraidNAS kernel: BTRFS warning (device dm-3): i/o error at logical 3418399809536 on dev /dev/mapper/nvme0n1p1, physical 440846622720, root 5, inode 1304841, offset 31049441280, length 4096, links 1 (path: domains/Ubuntu/vdisk1.img) Nov 21 11:48:56 UnraidNAS kernel: BTRFS error (device dm-3): bdev /dev/mapper/nvme0n1p1 errs: wr 0, rd 6, flush 0, corrupt 8, gen 0 Nov 21 11:48:56 UnraidNAS kernel: nvme0n1: I/O Cmd(0x2) @ LBA 861134136, 8 blocks, I/O Error (sct 0x2 / sc 0x81) MORE DNR Nov 21 11:48:56 UnraidNAS kernel: critical medium error, dev nvme0n1, sector 861134136 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0 Nov 21 11:48:56 UnraidNAS kernel: BTRFS warning (device dm-3): i/o error at logical 3418436038656 on dev /dev/mapper/nvme0n1p1, physical 440882851840, root 5, inode 1304841, offset 23452647424, length 4096, links 1 (path: domains/Ubuntu/vdisk1.img) Nov 21 11:48:56 UnraidNAS kernel: BTRFS error (device dm-3): bdev /dev/mapper/nvme0n1p1 errs: wr 0, rd 7, flush 0, corrupt 8, gen 0 Nov 21 11:48:56 UnraidNAS kernel: BTRFS warning (device dm-3): i/o error at logical 3418399813632 on dev /dev/mapper/nvme0n1p1, physical 440846626816, root 5, inode 1304841, offset 31049445376, length 4096, links 1 (path: domains/Ubuntu/vdisk1.img) Nov 21 11:48:56 UnraidNAS kernel: BTRFS error (device dm-3): bdev /dev/mapper/nvme0n1p1 errs: wr 0, rd 8, flush 0, corrupt 8, gen 0 Nov 21 11:48:56 UnraidNAS kernel: BTRFS warning (device dm-3): i/o error at logical 3418436042752 on dev /dev/mapper/nvme0n1p1, physical 440882855936, root 5, inode 1304841, offset 23452651520, length 4096, links 1 (path: domains/Ubuntu/vdisk1.img) Nov 21 11:48:56 UnraidNAS kernel: BTRFS error (device dm-3): bdev /dev/mapper/nvme0n1p1 errs: wr 0, rd 9, flush 0, corrupt 8, gen 0 Nov 21 11:48:56 UnraidNAS kernel: BTRFS warning (device dm-3): i/o error at logical 3418399817728 on dev /dev/mapper/nvme0n1p1, physical 440846630912, root 5, inode 1304841, offset 31049449472, length 4096, links 1 (path: domains/Ubuntu/vdisk1.img) Nov 21 11:48:56 UnraidNAS kernel: BTRFS error (device dm-3): bdev /dev/mapper/nvme0n1p1 errs: wr 0, rd 10, flush 0, corrupt 8, gen 0 Nov 21 11:50:02 UnraidNAS kernel: BTRFS info (device dm-3): scrub: finished on devid 1 with status: 0
  4. ok scrub in process thanks. I did a inode find and deleted the affected file prior. root@UnraidNAS:~# find /mnt/cache -inum 17390531 /mnt/cache/appdata/binhex-radarr/logs/radarr.39.txt root@UnraidNAS:~# cd /mnt/cache/appdata/binhex-radarr/logs/ root@UnraidNAS:/mnt/cache/appdata/binhex-radarr/logs# rm radarr.39.txt root@UnraidNAS:/mnt/cache/appdata/binhex-radarr/logs#
  5. Server still going strong, but noticed there was an error this morning that I overlooked. @JorgeB Is there something I need to dig into to? Nov 21 04:17:20 UnraidNAS CA Backup/Restore: Verifying backup Nov 21 04:17:20 UnraidNAS CA Backup/Restore: Using command: cd '/mnt/user/appdata/' && /usr/bin/tar --diff -C '/mnt/user/appdata/' -af '/mnt/user/backups/appdata/2022-11-21@04.00/CA_backup.tar.gz' > /var/lib/docker/unraid/ca.backup2.datastore/appdata_backup.log & echo $! > /tmp/ca.backup2/tempFiles/verifyInProgress Nov 21 04:18:00 UnraidNAS kernel: BTRFS warning (device dm-3): csum failed root 5 ino 17390531 off 327680 csum 0x32c11707 expected csum 0x43348dfc mirror 1 Nov 21 04:18:00 UnraidNAS kernel: BTRFS error (device dm-3): bdev /dev/mapper/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0 Should I do a BTRFS scrub or something? unraidnas-diagnostics-20221121-1127.zip
  6. After several more crashes not reported here yesterday, I pulled the 2nd half of my RAM out and now the server runs 100% for 12 hours. Thank you! Do you think the bad RAM could be responsible for the previous nvme / usb device crashes I've had this year:
  7. Had another crash with the VM off, then decided to leave off compreface-gpu and double-take dockers knowing these guys are heavy lifters (really no evidence says to turn them off, just a hunch). Then left the house, came home to my zigbee2mqtt docker dead. The log said something about write-only only. Looking at the pfsense remote syslog (see attached): Nov 20 12:30:34 unraid kernel: BTRFS critical (device dm-3): corrupt leaf: root=7 block=3573797388288 slot=263, unaligned key offset for csum item, have 3069387368448 should be aligned to 4096 Nov 20 12:30:34 unraid kernel: BTRFS info (device dm-3): leaf 3573797388288 gen 2132420 total ptrs 329 free space 4214 owner 7 Nov 20 12:30:34 unraid kernel: item 0 key (18446744073709551606 128 3069384982528) itemoff 16271 itemsize 12 Nov 20 12:30:34 unraid kernel: item 1 key (18446744073709551606 128 3069384994816) itemoff 16267 itemsize 4 Nov 20 12:30:34 unraid kernel: item 2 key (18446744073709551606 128 3069384998912) itemoff 16259 itemsize 8 Nov 20 12:30:34 unraid kernel: item 3 key (18446744073709551606 128 3069385007104) itemoff 16243 itemsize 16 Then: Nov 20 12:30:34 unraid kernel: BTRFS error (device dm-3): block=3573797388288 write time tree block corruption detected Nov 20 12:30:34 unraid kernel: BTRFS: error (device dm-3) in btrfs_commit_transaction:2418: errno=-5 IO failure (Error while writing out transaction) Nov 20 12:30:34 unraid kernel: BTRFS info (device dm-3: state E): forced readonly Nov 20 12:30:34 unraid kernel: BTRFS warning (device dm-3: state E): Skipping commit of aborted transaction. Nov 20 12:30:34 unraid kernel: BTRFS: error (device dm-3: state EA) in cleanup_transaction:1982: errno=-5 IO failure Nov 20 12:30:34 unraid kernel: I/O error, dev loop2, sector 1953024 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0 so I noticed 6.11.5 what the heck why not I upgrade the OS to that. also in pure desperation I removed the following append from my flash: nvme_core.default_ps_max_latency_us=0 pcie_aspm=off this was suggested in the log from a previous crash this year. Do I just have a shitty nvme here? pfsense-unraid2022-11-20_morecrash.log
  8. yeah def provide diags and logs on a dedicated thread if possible to the ops to solve it. otherwise it's not possible to know if the issues are related.
  9. OK now crashing on 6.11.3 when I launched my VM. Now I am a bit confused, especially since I've been on 6.11.3 for over a week with my win10 running no issue. See attached for crash log and diags. Nov 20 07:35:24 unraid kernel: BUG: Bad page state in process uwsgi pfn:4b47cc Nov 20 07:35:24 unraid kernel: page:000000000856407b refcount:0 mapcount:0 mapping:0000000000000000 index:0x1 pfn:0x4b47cc Nov 20 07:35:24 unraid kernel: flags: 0x2ffff0000000008(dirty|node=0|zone=2|lastcpupid=0xffff) Nov 20 07:35:24 unraid kernel: raw: 02ffff0000000008 dead000000000100 dead000000000122 0000000000000000 Nov 20 07:35:24 unraid kernel: raw: 0000000000000001 0000000000000000 00000000ffffffff 0000000000000000 Nov 20 07:35:24 unraid kernel: page dumped because: PAGE_FLAGS_CHECK_AT_PREP flag(s) set Nov 20 07:35:24 unraid kernel: Modules linked in: af_packet nvidia_uvm(PO) xt_nat veth xt_CHECKSUM ipt_REJECT nf_reject_ipv4 xt_tcpudp ip6table_mangle ip6table_nat iptable_mangle vhost_net tun vhost vhost_iotlb tap xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 br_netfilter xfs dm_crypt dm_mod dax md_mod nct6775 nct6775_core hwmon_vid efivarfs ip6table_filter ip6_tables iptable_filter ip_tables x_tables bridge stp llc bonding tls wmi_bmof nvidia_drm(PO) nvidia_modeset(PO) edac_mce_amd edac_core nvidia(PO) kvm_amd drm_kms_helper kvm drm backlight crct10dif_pclmul i2c_piix4 crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl r8169 nvme i2c_core k10temp ccp syscopyarea ahci sysfillrect nvme_core realtek joydev sysimgblt libahci fb_sys_fops wmi tpm_crb tpm_tis tpm_tis_core tpm acpi_cpufreq button unix Nov 20 07:35:24 unraid kernel: CPU: 1 PID: 25802 Comm: uwsgi Tainted: P B O 5.19.17-Unraid #2 Nov 20 07:35:24 unraid kernel: Hardware name: To Be Filled By O.E.M. B550M Pro4/B550M Pro4, BIOS P2.30 02/24/2022 pfsense-unraid2022-11-20_6.11.3.log unraidnas-diagnostics-20221120-0833.zip
  10. Upgraded to 6.11.4 from 6.11.3. Two crashes were observed soon after: 1) Right after upgrading, I tried to launch my WIN10VM while the dockers were loading. The system froze and my win10vm was unresponsive at login screen. All my dockers were down and my unraid GUI was unreachable. Nov 19 17:01:17 unraid kernel: BUG: Bad page state in process qemu-system-x86 pfn:7559e9 Nov 19 17:01:17 unraid kernel: page:00000000c01ab2ff refcount:0 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x7559e9 Nov 19 17:01:17 unraid kernel: flags: 0x2ffff0000000008(dirty|node=0|zone=2|lastcpupid=0xffff) Nov 19 17:01:17 unraid kernel: raw: 02ffff0000000008 ffffea001d567a48 ffffea001d567a48 0000000000000000 Nov 19 17:01:17 unraid kernel: raw: 0000000000000000 0000000000000000 00000000ffffffff 0000000000000000 Nov 19 17:01:17 unraid kernel: page dumped because: PAGE_FLAGS_CHECK_AT_PREP flag(s) set 2) During morning CA backup while my Win10VM was running. Same symptoms as before. Nov 20 04:09:55 unraid kernel: BUG: kernel NULL pointer dereference, address: 0000000000000088 Nov 20 04:09:55 unraid kernel: #PF: supervisor read access in kernel mode Nov 20 04:09:55 unraid kernel: #PF: error_code(0x0000) - not-present page Nov 20 04:09:55 unraid kernel: PGD 0 P4D 0 Nov 20 04:09:55 unraid kernel: Oops: 0000 [#1] PREEMPT SMP NOPTI Nov 20 04:09:55 unraid kernel: CPU: 5 PID: 263 Comm: kswapd0 Tainted: P B O 5.19.17-Unraid #2 Nov 20 04:09:55 unraid kernel: Hardware name: To Be Filled By O.E.M. B550M Pro4/B550M Pro4, BIOS P2.30 02/24/2022 Nov 20 04:09:55 unraid kernel: RIP: 0010:mem_cgroup_lruvec+0x35/0x4c The consistent trend is that it crashed during docker loading I guess once on startup and once during CA backup. Both times with the VM on. pfsense-unraid2022-11-19.log pfsense-unraid2022-11-20.log unraidnas-diagnostics-20221120-0657.zip
  11. Crashed again. Not sure what to do anymore but remove my google coral for a month and see if it stabilizes? Or maybe try to relocate my Google Coral to another usb hub? See crash log (captured on remote pfSense syslog-ng server) and latest diagnostics below. Nov 4 07:07:59 unraid kernel: xhci_hcd 0000:02:00.0: WARN Set TR Deq Ptr cmd failed due to incorrect slot or ep state. Nov 4 07:07:59 unraid kernel: xhci_hcd 0000:02:00.0: WARN Successful completion on short TX Nov 4 07:07:59 unraid kernel: xhci_hcd 0000:02:00.0: ERROR Transfer event TRB DMA ptr not part of current TD ep_index 2 comp_code 1 unraidnas-diagnostics-20221104-0736.zip unraidnas_pfSense_syslog-ng_crash_2022-11-04_070800.log
  12. Server crash, but seems unrelated. Seems like the more common [02:00.0] usb hub crash I've been having. I will follow up on other thread but thought it was worth mentioning it here... Nov 4 07:07:59 unraid kernel: xhci_hcd 0000:02:00.0: WARN Set TR Deq Ptr cmd failed due to incorrect slot or ep state. Nov 4 07:07:59 unraid kernel: xhci_hcd 0000:02:00.0: WARN Successful completion on short TX Nov 4 07:07:59 unraid kernel: xhci_hcd 0000:02:00.0: ERROR Transfer event TRB DMA ptr not part of current TD ep_index 2 comp_code 1 Nov 4 07:07:59 unraid kernel: xhci_hcd 0000:02:00.0: Looking for event-dma 000000026ad7d410 trb-start 000000026ad7d420 trb-end 000000026ad7d420 seg-start 000000026ad7d000 seg-end 000000026ad7dff0 Nov 4 07:08:00 unraid kernel: general protection fault, probably for non-canonical address 0xcec7beccd6afa8c1: 0000 [#1] PREEMPT SMP NOPTI Nov 4 07:08:00 unraid kernel: CPU: 6 PID: 16336 Comm: nginx Tainted: P O 5.19.14-Unraid #1 Nov 4 07:08:00 unraid kernel: Hardware name: To Be Filled By O.E.M. B550M Pro4/B550M Pro4, BIOS P2.30 02/24/2022 Nov 4 07:08:00 unraid kernel: RIP: 0010:__kmalloc_node_track_caller+0x126/0x1d9 Nov 4 07:08:00 unraid kernel: Code: 19 8b 54 24 04 4c 89 f9 4c 89 e7 8b 34 24 e8 78 fe ff ff 48 89 44 24 10 eb 2c 41 8b 44 24 28 48 8d 8a 00 01 00 00 49 8b 3c 24 <49> 8b 1c 06 4c 89 f0 65 48 0f c7 0f 0f 94 c0 84 c0 74 87 41 8b 44 Nov 4 07:08:00 unraid kernel: RSP: 0018:ffffc9000448fbf0 EFLAGS: 00010202 Nov 4 07:08:00 unraid kernel: RAX: 0000000000000200 RBX: ffff8881a4a91c00 RCX: 000000153c2fed06 Nov 4 07:08:00 unraid kernel: RDX: 000000153c2fec06 RSI: 00000000ffffffff RDI: 000000000002f650 Nov 4 07:08:00 unraid kernel: RBP: ffff888100042b00 R08: 0000000000082a20 R09: 0000000000000000 Nov 4 07:08:00 unraid kernel: R10: ffffc9000448fe40 R11: 0000000000000000 R12: ffff888100042b00 Nov 4 07:08:00 unraid kernel: R13: 0000000000000280 R14: cec7beccd6afa6c1 R15: ffffffff816b81f4 Nov 4 07:08:00 unraid kernel: FS: 00001513ba288740(0000) GS:ffff88900e980000(0000) knlGS:0000000000000000 Nov 4 07:08:00 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 4 07:08:00 unraid kernel: CR2: 000000c0000ea000 CR3: 0000000fd72a8000 CR4: 0000000000350ee0 Nov 4 07:08:00 unraid kernel: Call Trace: Nov 4 07:08:00 unraid kernel: <TASK> Nov 4 07:08:00 unraid kernel: kmalloc_reserve+0x2d/0x73 Nov 4 07:08:00 unraid kernel: __alloc_skb+0xb2/0x15e Nov 4 07:08:00 unraid kernel: ? preempt_latency_start+0x2b/0x46 Nov 4 07:08:00 unraid kernel: __tcp_send_ack+0x3b/0xdc Nov 4 07:08:00 unraid kernel: tcp_recvmsg_locked+0x6a1/0x6cf Nov 4 07:08:00 unraid kernel: tcp_recvmsg+0x101/0x1a2 Nov 4 07:08:00 unraid kernel: inet_recvmsg+0x69/0xa9 Nov 4 07:08:00 unraid kernel: __sys_recvfrom+0x97/0xf8 Nov 4 07:08:00 unraid kernel: __x64_sys_recvfrom+0x20/0x27 Nov 4 07:08:00 unraid kernel: do_syscall_64+0x6b/0x81 Nov 4 07:08:00 unraid kernel: entry_SYSCALL_64_after_hwframe+0x63/0xcd Nov 4 07:08:00 unraid kernel: RIP: 0033:0x1513bca9f5b0 Nov 4 07:08:00 unraid kernel: Code: 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 41 89 ca 64 8b 04 25 18 00 00 00 85 c0 75 1d 45 31 c9 45 31 c0 b8 2d 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 68 c3 0f 1f 80 00 00 00 00 55 48 83 ec 20 48 Nov 4 07:08:00 unraid kernel: RSP: 002b:00007ffc6a372808 EFLAGS: 00000246 ORIG_RAX: 000000000000002d Nov 4 07:08:00 unraid kernel: RAX: ffffffffffffffda RBX: 0000000000001012 RCX: 00001513bca9f5b0 Nov 4 07:08:00 unraid kernel: RDX: 0000000000001012 RSI: 0000560813db8e80 RDI: 0000000000000010 Nov 4 07:08:00 unraid kernel: RBP: 0000151399cb8400 R08: 0000000000000000 R09: 0000000000000000 Nov 4 07:08:00 unraid kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 Nov 4 07:08:00 unraid kernel: R13: 0000560813db8e80 R14: 0000560813aee180 R15: 0000560813d4d4c0 Nov 4 07:08:00 unraid kernel: </TASK> Nov 4 07:08:00 unraid kernel: Modules linked in: xt_mark af_packet nvidia_uvm(PO) xt_nat veth xt_CHECKSUM ipt_REJECT nf_reject_ipv4 xt_tcpudp ip6table_mangle ip6table_nat iptable_mangle vhost_net tun vhost vhost_iotlb tap xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 br_netfilter xfs dm_crypt dm_mod dax md_mod nct6775 nct6775_core hwmon_vid efivarfs ip6table_filter ip6_tables iptable_filter ip_tables x_tables bridge stp llc bonding tls ipv6 wmi_bmof edac_mce_amd edac_core nvidia_drm(PO) nvidia_modeset(PO) kvm_amd nvidia(PO) kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel drm_kms_helper aesni_intel crypto_simd cryptd rapl drm r8169 i2c_piix4 ccp k10temp nvme backlight realtek i2c_core joydev ahci nvme_core syscopyarea sysfillrect sysimgblt libahci fb_sys_fops wmi tpm_crb tpm_tis tpm_tis_core tpm acpi_cpufreq button unix Nov 4 07:08:00 unraid kernel: ---[ end trace 0000000000000000 ]--- Nov 4 07:08:00 unraid kernel: RIP: 0010:__kmalloc_node_track_caller+0x126/0x1d9 Nov 4 07:08:00 unraid kernel: Code: 19 8b 54 24 04 4c 89 f9 4c 89 e7 8b 34 24 e8 78 fe ff ff 48 89 44 24 10 eb 2c 41 8b 44 24 28 48 8d 8a 00 01 00 00 49 8b 3c 24 <49> 8b 1c 06 4c 89 f0 65 48 0f c7 0f 0f 94 c0 84 c0 74 87 41 8b 44 Nov 4 07:08:00 unraid kernel: RSP: 0018:ffffc9000448fbf0 EFLAGS: 00010202 Nov 4 07:08:00 unraid kernel: RAX: 0000000000000200 RBX: ffff8881a4a91c00 RCX: 000000153c2fed06 Nov 4 07:08:00 unraid kernel: RDX: 000000153c2fec06 RSI: 00000000ffffffff RDI: 000000000002f650 Nov 4 07:08:00 unraid kernel: RBP: ffff888100042b00 R08: 0000000000082a20 R09: 0000000000000000 Nov 4 07:08:00 unraid kernel: R10: ffffc9000448fe40 R11: 0000000000000000 R12: ffff888100042b00 Nov 4 07:08:00 unraid kernel: R13: 0000000000000280 R14: cec7beccd6afa6c1 R15: ffffffff816b81f4 Nov 4 07:08:00 unraid kernel: FS: 00001513ba288740(0000) GS:ffff88900e980000(0000) knlGS:0000000000000000 Nov 4 07:08:00 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 4 07:08:00 unraid kernel: CR2: 000000c0000ea000 CR3: 0000000fd72a8000 CR4: 0000000000350ee0 Nov 4 07:08:00 unraid kernel: general protection fault, probably for non-canonical address 0xcec7beccd6afa8c1: 0000 [#2] PREEMPT SMP NOPTI Nov 4 07:08:00 unraid kernel: CPU: 6 PID: 3055 Comm: kworker/6:0 Tainted: P D O 5.19.14-Unraid #1 Nov 4 07:08:00 unraid kernel: Hardware name: To Be Filled By O.E.M. B550M Pro4/B550M Pro4, BIOS P2.30 02/24/2022 Nov 4 07:08:00 unraid kernel: Workqueue: events efi_pstore_update_entries Nov 4 07:08:00 unraid kernel: RIP: 0010:__kmalloc+0xf2/0x19e Nov 4 07:08:00 unraid kernel: Code: 00 48 89 04 24 74 05 48 85 c0 75 17 4c 89 f9 83 ca ff 44 89 e6 48 89 ef e8 50 f9 ff ff 48 89 04 24 eb 25 8b 4d 28 48 8b 7d 00 <48> 8b 1c 08 48 8d 8a 00 01 00 00 65 48 0f c7 0f 0f 94 c0 84 c0 74 Nov 4 07:08:00 unraid kernel: RSP: 0018:ffffc90003effd90 EFLAGS: 00010282 Nov 4 07:08:00 unraid kernel: RAX: cec7beccd6afa6c1 RBX: ffff888102eaf000 RCX: 0000000000000200 Nov 4 07:08:00 unraid kernel: RDX: 000000153c2fec06 RSI: 0000000000000dc0 RDI: 000000000002f650 Nov 4 07:08:00 unraid kernel: RBP: ffff888100042b00 R08: 0000000000000dc0 R09: 0000000000000001 Nov 4 07:08:00 unraid kernel: R10: 8080808080808080 R11: fefefefefefefeff R12: 0000000000000dc0 Nov 4 07:08:00 unraid kernel: R13: ffff888100042b00 R14: 0000000000000400 R15: ffffffff8168a2f0 Nov 4 07:08:00 unraid kernel: FS: 0000000000000000(0000) GS:ffff88900e980000(0000) knlGS:0000000000000000 Nov 4 07:08:00 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 4 07:08:00 unraid kernel: CR2: 000000c0000ea000 CR3: 0000000fd72a8000 CR4: 0000000000350ee0 Nov 4 07:08:00 unraid kernel: Call Trace: Nov 4 07:08:00 unraid kernel: <TASK> Nov 4 07:08:00 unraid kernel: efivar_init+0x78/0x330 Nov 4 07:08:00 unraid kernel: ? efi_pstore_read_func+0x275/0x275 Nov 4 07:08:00 unraid kernel: ? efi_pstore_update_entries+0x1c/0x67 Nov 4 07:08:00 unraid kernel: ? kmem_cache_alloc_trace+0x11e/0x149 Nov 4 07:08:00 unraid kernel: efi_pstore_update_entries+0x3c/0x67 Nov 4 07:08:00 unraid kernel: process_one_work+0x1ab/0x295 Nov 4 07:08:00 unraid kernel: worker_thread+0x18b/0x244 Nov 4 07:08:00 unraid kernel: ? rescuer_thread+0x281/0x281 Nov 4 07:08:00 unraid kernel: kthread+0xe7/0xef Nov 4 07:08:00 unraid kernel: ? kthread_complete_and_exit+0x1b/0x1b Nov 4 07:08:00 unraid kernel: ret_from_fork+0x22/0x30 Nov 4 07:08:00 unraid kernel: </TASK> Nov 4 07:08:00 unraid kernel: Modules linked in: xt_mark af_packet nvidia_uvm(PO) xt_nat veth xt_CHECKSUM ipt_REJECT nf_reject_ipv4 xt_tcpudp ip6table_mangle ip6table_nat iptable_mangle vhost_net tun vhost vhost_iotlb tap xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 br_netfilter xfs dm_crypt dm_mod dax md_mod nct6775 nct6775_core hwmon_vid efivarfs ip6table_filter ip6_tables iptable_filter ip_tables x_tables bridge stp llc bonding tls ipv6 wmi_bmof edac_mce_amd edac_core nvidia_drm(PO) nvidia_modeset(PO) kvm_amd nvidia(PO) kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel drm_kms_helper aesni_intel crypto_simd cryptd rapl drm r8169 i2c_piix4 ccp k10temp nvme backlight realtek i2c_core joydev ahci nvme_core syscopyarea sysfillrect sysimgblt libahci fb_sys_fops wmi tpm_crb tpm_tis tpm_tis_core tpm acpi_cpufreq button unix Nov 4 07:08:00 unraid kernel: ---[ end trace 0000000000000000 ]---
  13. Going strong for 5 days... will update here again if/when it ever crashes.
  14. What's your motherboard and BIOS version?