jxjelly Posted October 21, 2022 Share Posted October 21, 2022 Tried turning the computer off with the power button which took down docker but didn't complete shutting down. I was able to ssh in and /etc/rc.d/rc.php-fpm restart allowed me to reach the gui again. I saw the server was prevented from shutting down because it ran into trouble shutting down the array. Shutting down cache drive around 18:41 Couldn't get the diagnostics because it was freezing at nvidia-smi Gui went unresponsive a few times while navigating the server. the php restart got it back every time. Ultimately the restart button in the gui got it to restart. Looking at the syslogs Oct 20 07:47:36 Monolith webGUI: Successful login user root from 209.107.189.2 [...] Oct 20 07:48:47 Monolith php-fpm[5983]: [WARNING] [pool www] server reached max_children setting (50), consider raising it At 07:47 was the last time I was able to log in. The other line I included above I wonder if it is the problem. Not sure why I'm hitting so many max children. Oct 20 20:18:48 Monolith kernel: _swap_info_get: Bad swap file entry 3ffffffffffbf Oct 20 20:18:48 Monolith kernel: BUG: Bad page map in process php-fpm pte:00008000 pmd:1a6f38067 Oct 20 20:18:48 Monolith kernel: addr:0000154493f33000 vm_flags:00000075 anon_vma:0000000000000000 mapping:ffff888107c6f3e8 index:ee Oct 20 20:18:48 Monolith kernel: file:libc-2.36.so fault:shmem_fault mmap:shmem_mmap read_folio:0x0 Oct 20 20:18:48 Monolith kernel: CPU: 0 PID: 29542 Comm: php-fpm Tainted: P D W O 5.19.14-Unraid #1 Oct 20 20:18:48 Monolith kernel: Hardware name: Gigabyte Technology Co., Ltd. Z390 AORUS PRO WIFI/Z390 AORUS PRO WIFI-CF, BIOS F12 11/05/2021 This bit also looked suspicious And then apologies for the end of the syslogs I had to reboot a few times to get access and download them and diagnostics (different issue for another time) monolith-diagnostics-20221020-2114.zip syslog Quote Link to comment
jxjelly Posted October 21, 2022 Author Share Posted October 21, 2022 You know what, I think I also sent my syslogs to a share on the cache drive which might be why it wouldn't go offline from the physical power button Quote Link to comment
JorgeB Posted October 21, 2022 Share Posted October 21, 2022 Problem appears to start with a Nvidia driver crash: Oct 20 05:00:18 Monolith kernel: general protection fault, probably for non-canonical address 0x841f0f2e66c3f3: 0000 [#1] PREEMPT SMP NOPTI Oct 20 05:00:18 Monolith kernel: CPU: 2 PID: 23970 Comm: PMS Timer Tainted: P O 5.19.14-Unraid #1 Oct 20 05:00:18 Monolith kernel: Hardware name: Gigabyte Technology Co., Ltd. Z390 AORUS PRO WIFI/Z390 AORUS PRO WIFI-CF, BIOS F12 11/05/2021 Oct 20 05:00:18 Monolith kernel: RIP: 0010:_nv027480rm+0x12/0x20 [nvidia] Oct 20 05:00:18 Monolith kernel: Code: f6 b8 01 00 00 00 89 d1 d3 e0 09 04 b7 c3 66 2e 0f 1f 84 00 00 00 00 00 48 8b bf 80 01 00 00 89 f6 b8 fe ff ff ff 89 d1 d3 c0 <21> 04 b7 c3 66 2e 0f 1f 84 00 00 00 00 00 48 8b bf 80 01 00 00 89 Oct 20 05:00:18 Monolith kernel: RSP: 0018:ffffc900037df8d0 EFLAGS: 00010246 Oct 20 05:00:18 Monolith kernel: RAX: 00000000fffffffe RBX: ffff8881ab1b0008 RCX: 0000000000000000 Oct 20 05:00:18 Monolith kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 00841f0f2e66c3f3 Oct 20 05:00:18 Monolith kernel: RBP: ffff88814fd45b90 R08: 0000000000000020 R09: ffff88814fd45b78 Oct 20 05:00:18 Monolith kernel: R10: ffff8881ab1b0008 R11: 00000001010f0000 R12: ffff8881ab1b0008 Oct 20 05:00:18 Monolith kernel: R13: ffff8881ab1b0008 R14: 0000000000000000 R15: 0000000000000001 Oct 20 05:00:18 Monolith kernel: FS: 0000000000000000(0000) GS:ffff88889dc80000(0000) knlGS:0000000000000000 Oct 20 05:00:18 Monolith kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Oct 20 05:00:18 Monolith kernel: CR2: 000014e33580ff1c CR3: 000000000200a006 CR4: 00000000003706e0 Oct 20 05:00:18 Monolith kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Oct 20 05:00:18 Monolith kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Oct 20 05:00:18 Monolith kernel: Call Trace: Oct 20 05:00:18 Monolith kernel: <TASK> Oct 20 05:00:18 Monolith kernel: ? _nv019327rm+0x8b/0x100 [nvidia] Oct 20 05:00:18 Monolith kernel: ? _nv019776rm+0x340/0x7f0 [nvidia] Oct 20 05:00:18 Monolith kernel: ? _nv019777rm+0xda/0x1e0 [nvidia] Oct 20 05:00:18 Monolith kernel: ? _nv019777rm+0xb8/0x1e0 [nvidia] Oct 20 05:00:18 Monolith kernel: ? _nv013890rm+0x438/0x920 [nvidia] Oct 20 05:00:18 Monolith kernel: ? _nv013890rm+0x418/0x920 [nvidia] Oct 20 05:00:18 Monolith kernel: ? _nv013878rm+0x78/0xf0 [nvidia] Oct 20 05:00:18 Monolith kernel: ? _nv014058rm+0x196/0x7f0 [nvidia] Oct 20 05:00:18 Monolith kernel: ? _nv035922rm+0xac/0xe0 [nvidia] Oct 20 05:00:18 Monolith kernel: ? _nv037394rm+0xac/0x140 [nvidia] Oct 20 05:00:18 Monolith kernel: ? _nv037393rm+0x2f7/0x4d0 [nvidia] Oct 20 05:00:18 Monolith kernel: ? _nv035835rm+0xbe/0x140 [nvidia] Oct 20 05:00:18 Monolith kernel: ? _nv035836rm+0x42/0x70 [nvidia] Oct 20 05:00:18 Monolith kernel: ? _nv031339rm+0x113/0x230 [nvidia] Oct 20 05:00:18 Monolith kernel: ? rm_gpu_ops_channel_destroy+0x1c/0x60 [nvidia] Oct 20 05:00:18 Monolith kernel: ? nvUvmGetSafeStack+0x42/0x92 [nvidia] Oct 20 05:00:18 Monolith kernel: ? nvUvmInterfaceChannelDestroy+0x1e/0x29 [nvidia] Oct 20 05:00:18 Monolith kernel: ? channel_destroy+0x89/0xc0 [nvidia_uvm] Oct 20 05:00:18 Monolith kernel: ? uvm_channel_manager_destroy.part.0+0x4c/0xa0 [nvidia_uvm] Oct 20 05:00:18 Monolith kernel: ? remove_gpu+0x183/0x3a0 [nvidia_uvm] Oct 20 05:00:18 Monolith kernel: ? uvm_gpu_release_locked+0x1f/0x40 [nvidia_uvm] Oct 20 05:00:18 Monolith kernel: ? uvm_va_space_destroy+0x45f/0x4c0 [nvidia_uvm] Oct 20 05:00:18 Monolith kernel: ? uvm_release.constprop.0+0x3d/0xa0 [nvidia_uvm] Oct 20 05:00:18 Monolith kernel: ? uvm_release_entry.part.0.isra.0+0x7a/0xb0 [nvidia_uvm] Oct 20 05:00:18 Monolith kernel: ? __fput+0x101/0x1c8 Oct 20 05:00:18 Monolith kernel: ? task_work_run+0x66/0x7e Oct 20 05:00:18 Monolith kernel: ? do_exit+0x39f/0x8e5 Oct 20 05:00:18 Monolith kernel: ? _raw_spin_unlock_irqrestore+0x24/0x3a Oct 20 05:00:18 Monolith kernel: ? do_group_exit+0x8f/0x8f Oct 20 05:00:18 Monolith kernel: ? get_signal+0x606/0x63e Oct 20 05:00:18 Monolith kernel: ? hrtimer_init_sleeper+0x41/0x41 Oct 20 05:00:18 Monolith kernel: ? arch_do_signal_or_restart+0x36/0x607 Oct 20 05:00:18 Monolith kernel: ? do_futex+0xcd/0x143 Oct 20 05:00:18 Monolith kernel: ? exit_to_user_mode_prepare+0x58/0x10d Oct 20 05:00:18 Monolith kernel: ? syscall_exit_to_user_mode+0x18/0x27 Oct 20 05:00:18 Monolith kernel: ? do_syscall_64+0x77/0x81 Oct 20 05:00:18 Monolith kernel: ? entry_SYSCALL_64_after_hwframe+0x63/0xcd Oct 20 05:00:18 Monolith kernel: </TASK> Oct 20 05:00:18 Monolith kernel: Modules linked in: xt_mark xt_nat veth tcp_diag udp_diag inet_diag nvidia_uvm(PO) xt_CHECKSUM ipt_REJECT nf_reject_ipv4 xt_tcpudp ip6table_mangle ip6table_nat iptable_mangle vhost_net tun vhost vhost_iotlb tap xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 br_netfilter xfs input_leds led_class joydev r8152 mii md_mod ip6table_filter ip6_tables iptable_filter ip_tables x_tables bridge stp llc ipv6 nvidia_drm(PO) nvidia_modeset(PO) x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm nvidia(PO) crct10dif_pclmul crc32_pclmul gigabyte_wmi wmi_bmof intel_wmi_thunderbolt mxm_wmi crc32c_intel ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl drm_kms_helper intel_cstate intel_uncore nvme i2c_i801 e1000e drm thunderbolt nvme_core i2c_smbus ahci i2c_core libahci syscopyarea sysfillrect sysimgblt fb_sys_fops intel_pch_thermal thermal fan tpm_crb tpm_tis video Oct 20 05:00:18 Monolith kernel: tpm_tis_core wmi backlight tpm acpi_pad button unix Oct 20 05:00:18 Monolith kernel: ---[ end trace 0000000000000000 ]--- Quote Link to comment
jxjelly Posted October 28, 2022 Author Share Posted October 28, 2022 I'm still having issues this time no nvidia crash in the logs. Nginx keeps giving me issues. syslog Quote Link to comment
JorgeB Posted October 28, 2022 Share Posted October 28, 2022 Are you using ipv6? If not disable it. Quote Link to comment
jxjelly Posted October 31, 2022 Author Share Posted October 31, 2022 It is disabled on my router and in my network settings. This weekend I had my longest period of uptime in a while from Friday till 4AM Monday. I'll have to pull the logs again when I get home. Quote Link to comment
Solution jxjelly Posted October 31, 2022 Author Solution Share Posted October 31, 2022 Booted up windows to get the log and ran into a memory related blue screen. Running memtest64 getting errors on 4, 6 & 7. So this is probably the cause of my issues. 1 Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.