Kulisch Posted July 14, 2022 Share Posted July 14, 2022 (edited) Hello everyone, my Unraid server is not accessible. (SSH, WebUI, Docker, VM,...) But there is no timeout, no drop, no block, nothing. SSH (no connection reset) and HTTP requests (Just blank, no error code) taking forever without any response. Only pinging the server is possible. This happens randomly. (uptime: 2 days, 4 days, 10 days) I can see that the server has a peak at power consumption in that time it happened. It "frozen" at 13:25 - 13:30 It is an AMD Ryzen 7 3800X build, so I did some UEFI settings like deactivating C-States, VFIO-Passthrough, B550 Taichi, BIOS Update Ver 2.0, no Memory Overclock (default 2133 MHZ, 2 x 16 GB). Maybe I'm missing something... Im having the feeling that this happens randomly during using a VM (Windows with GPU Passthrough). I do not remember any scenario where this happened and the VM was not started. I dont have a second GPU or a APU. Syslog to USB is activated and I can provide the logs after I restart the server. But before I do this, is there anything I should try before shutting down the server by pushing the button or cutting off the power? Thank you in advance Edited July 14, 2022 by Kulisch Quote Link to comment
JorgeB Posted July 14, 2022 Share Posted July 14, 2022 Enable the syslog server and post that together with the diagnostics after the next crash. Quote Link to comment
Kulisch Posted July 14, 2022 Author Share Posted July 14, 2022 Thank you. There you go: unraid-diagnostics-20220702-1242.zipusb-syslogremote-syslog Quote Link to comment
JorgeB Posted July 14, 2022 Share Posted July 14, 2022 Nothing obvious, some OOM errors and kvm related call traces so you might see if the issue doesn't occur if the VM(s) is(are) not running. Quote Link to comment
Kulisch Posted July 15, 2022 Author Share Posted July 15, 2022 (edited) After 19 hours it happened again, while using Remote Desktop. This time another VM (Win 11). Used it for 1 hour (Not passtroughing anything) and it froze at 11:18. This time, system is completely frozen. Trying to access over SSH and HTTP give me timeout responses. unraid-diagnostics-20220715-1144.zipserver-syslogusb-syslog Is it possible to enable and save debug logs of a VM? Or other advices? Because I dont remember having a frozen system without running a VM. Edited July 15, 2022 by Kulisch Quote Link to comment
JorgeB Posted July 15, 2022 Share Posted July 15, 2022 Again nothing relevant logged, make sure you're following the Ryzen specif recommendations, but if it only happens when using VMs it's probably not that. Quote Link to comment
Kulisch Posted July 22, 2022 Author Share Posted July 22, 2022 (edited) I checked the settings and changed from "Auto" to "typical current idle". After using a VM for several hours, unraid "freezes" again. Reverted to BIOS Update from 2.0 to 1.8 because I know that this Version worked proberly. Of course I checked the BIOS Settings after Downgrade and made the ryzen specific changes. Now I reverted Unraid 6.10.3 to 6.10.2. This time I see the VM (newly created Windows 11, nothing changed but windows updates) is not responsive after est. 2 hours idling. Settings VM: Now I'm pretty sure, that this problem does only occurs using a VM. Unraid was running for 3 days without using a VM and nothing bad happened. Every time I start a machine, after 1 or 2 hours the server is not responsive. Just pingable. unraid-diagnostics-20220722-1002.zipusb-syslogserver-syslog Edited July 22, 2022 by Kulisch Quote Link to comment
trurl Posted July 22, 2022 Share Posted July 22, 2022 Do you have an attached monitor and keyboard? Quote Link to comment
Kulisch Posted July 22, 2022 Author Share Posted July 22, 2022 Update: After the revert to 6.10.2 the Windows 10 and 11 machines are freezing after 1 hour. Not Unraid anymore. 👍 The Kali machine (without GUI) is online over 4 hours and nothing special is going wrong. So I consider the Guest OS makes the trouble. Do I have to check the "Energy saving" options from Windows? (Both options are on "never") 37 minutes ago, trurl said: Do you have an attached monitor and keyboard? Yes both of them. But there are not used for the VM in that case. Mouse and keyboard are passthrough(ed?) with the USB Controller. Monitor is connected with the only GPU which is also passthrough(ed?). Quote Link to comment
trurl Posted July 22, 2022 Share Posted July 22, 2022 2 hours ago, trurl said: Do you have an attached monitor and keyboard? I wanted to know if you could access the Unraid command line or the Unraid webUI on the server itself instead of over the network. Quote Link to comment
Kulisch Posted July 22, 2022 Author Share Posted July 22, 2022 21 minutes ago, trurl said: I wanted to know if you could access the Unraid command line or the Unraid webUI on the server itself instead of over the network. I could. I would have to remove the GPU and the USB controller from the passthrough, but that should not affect the error pattern. After all, it affected the VMs with and without passthrough. Should I check something specific or should I just see if the system reacts at all when "freezing". Quote Link to comment
trurl Posted July 22, 2022 Share Posted July 22, 2022 See what you get from command line with this dh -h Also might see if you can get diagnostics from command line Quote Link to comment
Kulisch Posted July 22, 2022 Author Share Posted July 22, 2022 See what you get from command line with thisdh -h Also might see if you can get diagnostics from command lineOk, I will try that in the next days. Trank youGesendet von iPhone mit Tapatalk Quote Link to comment
Kulisch Posted July 25, 2022 Author Share Posted July 25, 2022 (edited) So I updated to 6.10.3 again, took the IOMMU Groups back to Unraid, started a VM without any connected hardware device and it froze again. After checking the monitor I saw a Kernel Panic happened. So I couldn't check the diagnostics or "disk free" (df -h). Edit: Forgot to give you the diagnostics and logs after reboot. server-syslogusb-syslogunraid-diagnostics-20220726-0759.zip After starting Unraid, this messages showed up: Edited July 26, 2022 by Kulisch Quote Link to comment
Kulisch Posted August 24, 2022 Author Share Posted August 24, 2022 1. I installed the RC2 to test if there are still the same problems. After 18 Days and 5 hours uptime I started a Windows 11 and a Kali VM which I started installing (Stopped at user creation). Just to check if the problem appears again. I lost my RDP Session, tried to ping, nothing happened. This freeze happened after 20 - 30 minutes. 2. i checked the Syslog and it increased a little bit. 42.5 mb... Jul 27 08:06:18 unRAID kernel: vfio-pci 0000:0a:00.0: BAR 3: can't reserve [mem 0xe0000000-0xe1ffffff 64bit pref] Jul 27 08:06:18 unRAID kernel: vfio-pci 0000:0a:00.0: BAR 3: can't reserve [mem 0xe0000000-0xe1ffffff 64bit pref] Jul 27 08:06:18 unRAID kernel: vfio-pci 0000:0a:00.0: BAR 3: can't reserve [mem 0xe0000000-0xe1ffffff 64bit pref] Jul 27 08:06:18 unRAID kernel: vfio-pci 0000:0a:00.0: BAR 3: can't reserve [mem 0xe0000000-0xe1ffffff 64bit pref] Jul 27 08:06:18 unRAID kernel: vfio-pci 0000:0a:00.0: BAR 3: can't reserve [mem 0xe0000000-0xe1ffffff 64bit pref] Jul 27 08:06:18 unRAID kernel: vfio-pci 0000:0a:00.0: BAR 3: can't reserve [mem 0xe0000000-0xe1ffffff 64bit pref] This line shows 364433 times up. I removed these lines and replaced it with --- syslog-edit.log unraid-diagnostics-20220824-0756.zip Quote Link to comment
Kulisch Posted August 25, 2022 Author Share Posted August 25, 2022 Yesterday I made a memtest which passed. I'm running out of ideas. Does anyone have any suggestions what I could do? Because unfortunately this can not stay like this. I depend on the VMs and it is among other things the reason why I use Unraid at all. Quote Link to comment
trurl Posted August 25, 2022 Share Posted August 25, 2022 The point of df -h was to see if you had filled rootfs, which is in RAM and is where the OS is stored. If you fill it the OS won't have any room to work in. Quote Link to comment
Kulisch Posted August 26, 2022 Author Share Posted August 26, 2022 5 hours ago, trurl said: The point of df -h was to see if you had filled rootfs, which is in RAM and is where the OS is stored. If you fill it the OS won't have any room to work in. Thanks for the reply. Then I take back the GPU from passtrough again, hoping that no crash happens and I can get the informotions locally. Assuming the RAM would fill up, would a RAM expansion be enough? I had planned to go from 32 GB to 64 anyway. Quote Link to comment
trurl Posted August 26, 2022 Share Posted August 26, 2022 6 hours ago, Kulisch said: Assuming the RAM would fill up, would a RAM expansion be enough? More RAM is not the solution. If rootfs is filling you would need to figure out why. Nothing should be writing into rootfs except the OS. A common reason for filling rootfs is specifying a path that isn't mounted storage, such as a container host path. Quote Link to comment
Kulisch Posted September 7, 2022 Author Share Posted September 7, 2022 Everytime I use the VM for an hour, happens a kernel panic in unraid. So there is no chance to check anything after that. After disabling Docker and starting the VM the CPU goes on some CPU Threads on 100 %. (Windows 11 fresh install) root@unRAID:~# df -h Filesystem Size Used Avail Use% Mounted on rootfs 16G 2.0G 14G 13% / tmpfs 32M 420K 32M 2% /run /dev/sda1 15G 3.2G 12G 23% /boot overlay 16G 2.0G 14G 13% /lib/firmware overlay 16G 2.0G 14G 13% /lib/modules devtmpfs 8.0M 0 8.0M 0% /dev tmpfs 16G 0 16G 0% /dev/shm cgroup_root 8.0M 0 8.0M 0% /sys/fs/cgroup tmpfs 128M 476K 128M 1% /var/log tmpfs 1.0M 0 1.0M 0% /mnt/disks tmpfs 1.0M 0 1.0M 0% /mnt/remotes tmpfs 1.0M 0 1.0M 0% /mnt/rootshare /dev/md1 2.8T 1.3T 1.5T 48% /mnt/disk1 /dev/sdb1 466G 200G 264G 44% /mnt/cache shfs 2.8T 1.3T 1.5T 48% /mnt/user0 shfs 2.8T 1.3T 1.5T 48% /mnt/user /dev/loop3 50G 4.9M 50G 1% /etc/libvirt Forcing the shutdown doesn't work and after the 2nd try, Unraid crashes. After booting up, started the VM and it crashed after 1 hour again. root@unRAID:~# tail -f /var/log/syslog Sep 7 14:03:20 unRAID avahi-daemon[10293]: New relevant interface vnet0.IPv6 for mDNS. Sep 7 14:03:20 unRAID avahi-daemon[10293]: Registering new address record for fe80::fc54:ff:fe8d:ec92 on vnet0.*. Sep 7 14:03:20 unRAID flash_backup: adding task: /usr/local/emhttp/plugins/dynamix.my.servers/scripts/UpdateFlashBackup update Sep 7 14:05:59 unRAID ntpd[1724]: kernel reports TIME_ERROR: 0x41: Clock Unsynchronized Sep 7 14:06:51 unRAID sshd[15169]: Connection from 10.0.10.117 port 1579 on 10.0.10.10 port 22 rdomain "" Sep 7 14:06:51 unRAID sshd[15169]: Postponed keyboard-interactive for root from 10.0.10.117 port 1579 ssh2 [preauth] Sep 7 14:06:56 unRAID sshd[15169]: Postponed keyboard-interactive/pam for root from 10.0.10.117 port 1579 ssh2 [preauth] Sep 7 14:06:56 unRAID sshd[15169]: Accepted keyboard-interactive/pam for root from 10.0.10.117 port 1579 ssh2 Sep 7 14:06:56 unRAID sshd[15169]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0) Sep 7 14:06:56 unRAID sshd[15169]: Starting session: shell on pts/1 for root from 10.0.10.117 port 1579 id 0 Sep 7 14:07:55 unRAID avahi-daemon[10293]: Interface vnet0.IPv6 no longer relevant for mDNS. Sep 7 14:07:55 unRAID avahi-daemon[10293]: Leaving mDNS multicast group on interface vnet0.IPv6 with address fe80::fc54:ff:fe8d:ec92. Sep 7 14:07:55 unRAID kernel: br0: port 2(vnet0) entered disabled state Sep 7 14:07:55 unRAID kernel: device vnet0 left promiscuous mode Sep 7 14:07:55 unRAID kernel: br0: port 2(vnet0) entered disabled state Sep 7 14:07:55 unRAID avahi-daemon[10293]: Withdrawing address record for fe80::fc54:ff:fe8d:ec92 on vnet0. Sep 7 14:08:21 unRAID kernel: br0: port 2(vnet1) entered blocking state Sep 7 14:08:21 unRAID kernel: br0: port 2(vnet1) entered disabled state Sep 7 14:08:21 unRAID kernel: device vnet1 entered promiscuous mode Sep 7 14:08:21 unRAID kernel: br0: port 2(vnet1) entered blocking state Sep 7 14:08:21 unRAID kernel: br0: port 2(vnet1) entered forwarding state Sep 7 14:08:22 unRAID avahi-daemon[10293]: Joining mDNS multicast group on interface vnet1.IPv6 with address fe80::fc54:ff:fe8d:ec92. Sep 7 14:08:22 unRAID avahi-daemon[10293]: New relevant interface vnet1.IPv6 for mDNS. Sep 7 14:08:22 unRAID avahi-daemon[10293]: Registering new address record for fe80::fc54:ff:fe8d:ec92 on vnet1.*. Sep 7 14:08:39 unRAID nmbd[10250]: [2022/09/07 14:08:39.002205, 0] ../../source3/nmbd/nmbd_become_lmb.c:398(become_local_master_stage2) Sep 7 14:08:39 unRAID nmbd[10250]: ***** Sep 7 14:08:39 unRAID nmbd[10250]: Sep 7 14:08:39 unRAID nmbd[10250]: Samba name server UNRAID is now a local master browser for workgroup WORKGROUP on subnet 192.168.122.1 Sep 7 14:08:39 unRAID nmbd[10250]: Sep 7 14:08:39 unRAID nmbd[10250]: ***** Sep 7 14:11:00 unRAID root: Fix Common Problems Version 2022.08.29 Sep 7 14:11:07 unRAID root: Fix Common Problems: Warning: NerdPack.plg Not Compatible with Unraid version 6.11.0-rc4 Sep 7 14:11:10 unRAID root: Fix Common Problems: Warning: Syslog mirrored to flash Sep 7 14:11:10 unRAID root: Fix Common Problems: Warning: The plugin usb_manager_usbip_addon.plg is not known to Community Applications and is possibly incompatible with your server Sep 7 14:36:46 unRAID kernel: ------------[ cut here ]------------ Sep 7 14:36:46 unRAID kernel: WARNING: CPU: 3 PID: 57 at kernel/rcu/tree.c:2634 rcu_do_batch+0x3d8/0x46c Sep 7 14:36:46 unRAID kernel: Modules linked in: xt_CHECKSUM xt_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp ip6table_mangle ip6table_nat iptable_mangle vhost_net tun vhost vhost_iotlb tap vhci_hcd usbip_host usbip_core xfs md_mod efivarfs iptable_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha ip6table_filter ip6_tables iptable_filter ip_tables x_tables af_packet 8021q garp mrp bridge stp llc bonding tls ipv6 nvidia_drm(PO) nvidia_modeset(PO) nvidia(PO) drm_kms_helper drm btusb sr_mod btrtl cdrom btbcm kvm_amd kvm crct10dif_pclmul crc32_pclmul crc32c_intel wmi_bmof ghash_clmulni_intel aesni_intel crypto_simd btintel cryptd i2c_piix4 backlight rapl bluetooth ccp k10temp i2c_core igc syscopyarea sysfillrect ahci sysimgblt fb_sys_fops joydev libahci ecdh_generic ecc wmi button unix Sep 7 14:36:46 unRAID kernel: CPU: 3 PID: 57 Comm: rcuop/6 Tainted: P O 5.19.3-Unraid #1 Sep 7 14:36:46 unRAID kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./B550 Taichi, BIOS P1.80 02/22/2021 Sep 7 14:36:46 unRAID kernel: RIP: 0010:rcu_do_batch+0x3d8/0x46c Sep 7 14:36:46 unRAID kernel: Code: 00 00 00 75 49 48 85 d2 74 44 31 c0 48 89 83 f0 00 00 00 48 8b 05 71 c6 fe 00 48 89 83 00 01 00 00 48 8b 43 78 48 85 c0 74 4a <0f> 0b 48 8b 3c 24 e8 91 54 00 00 48 85 c0 74 02 0f 0b 48 8b 3c 24 Sep 7 14:36:46 unRAID kernel: RSP: 0018:ffffc9000039be60 EFLAGS: 00010086 Sep 7 14:36:46 unRAID kernel: RAX: ffff888101494000 RBX: ffff888122390d70 RCX: 0000000000000000 Sep 7 14:36:46 unRAID kernel: RDX: 0000000100000000 RSI: ffffffffffffffed RDI: ffff88881e9acdb8 Sep 7 14:36:46 unRAID kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: ffff88881e8ec070 Sep 7 14:36:46 unRAID kernel: R10: 0000000000000000 R11: ffff888100400058 R12: 0000000000000246 Sep 7 14:36:46 unRAID kernel: R13: 0000000000000013 R14: 000000000000000a R15: ffff888100a76e40 Sep 7 14:36:46 unRAID kernel: FS: 0000000000000000(0000) GS:ffff88881e8c0000(0000) knlGS:0000000000000000 Sep 7 14:36:46 unRAID kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Sep 7 14:36:46 unRAID kernel: CR2: 0000000000462cb8 CR3: 000000019f590000 CR4: 0000000000350ee0 Sep 7 14:36:46 unRAID kernel: Call Trace: Sep 7 14:36:46 unRAID kernel: <TASK> Sep 7 14:36:46 unRAID kernel: ? preempt_latency_start+0x2b/0x46 Sep 7 14:36:46 unRAID kernel: rcu_nocb_cb_kthread+0x10f/0x28c Sep 7 14:36:46 unRAID kernel: ? rcu_cpu_kthread+0x10a/0x10a Sep 7 14:36:46 unRAID kernel: kthread+0xe7/0xef Sep 7 14:36:46 unRAID kernel: ? kthread_complete_and_exit+0x1b/0x1b Sep 7 14:36:46 unRAID kernel: ret_from_fork+0x22/0x30 Sep 7 14:36:46 unRAID kernel: </TASK> Sep 7 14:36:46 unRAID kernel: ---[ end trace 0000000000000000 ]--- Sep 7 14:36:46 unRAID kernel: ------------[ cut here ]------------ Sep 7 14:36:46 unRAID kernel: WARNING: CPU: 3 PID: 57 at kernel/rcu/tree.c:2637 rcu_do_batch+0x3e8/0x46c Sep 7 14:36:46 unRAID kernel: Modules linked in: xt_CHECKSUM xt_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp ip6table_mangle ip6table_nat iptable_mangle vhost_net tun vhost vhost_iotlb tap vhci_hcd usbip_host usbip_core xfs md_mod efivarfs iptable_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha ip6table_filter ip6_tables iptable_filter ip_tables x_tables af_packet 8021q garp mrp bridge stp llc bonding tls ipv6 nvidia_drm(PO) nvidia_modeset(PO) nvidia(PO) drm_kms_helper drm btusb sr_mod btrtl cdrom btbcm kvm_amd kvm crct10dif_pclmul crc32_pclmul crc32c_intel wmi_bmof ghash_clmulni_intel aesni_intel crypto_simd btintel cryptd i2c_piix4 backlight rapl bluetooth ccp k10temp i2c_core igc syscopyarea sysfillrect ahci sysimgblt fb_sys_fops joydev libahci ecdh_generic ecc wmi button unix Sep 7 14:36:46 unRAID kernel: CPU: 3 PID: 57 Comm: rcuop/6 Tainted: P W O 5.19.3-Unraid #1 Sep 7 14:36:46 unRAID kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./B550 Taichi, BIOS P1.80 02/22/2021 Sep 7 14:36:46 unRAID kernel: RIP: 0010:rcu_do_batch+0x3e8/0x46c Sep 7 14:36:46 unRAID kernel: Code: 00 00 00 48 8b 05 71 c6 fe 00 48 89 83 00 01 00 00 48 8b 43 78 48 85 c0 74 4a 0f 0b 48 8b 3c 24 e8 91 54 00 00 48 85 c0 74 02 <0f> 0b 48 8b 3c 24 e8 81 54 00 00 48 85 c0 75 41 0f 0b eb 3d 48 2b Sep 7 14:36:46 unRAID kernel: RSP: 0018:ffffc9000039be60 EFLAGS: 00010002 Sep 7 14:36:46 unRAID kernel: RAX: 0000000000000023 RBX: ffff888122390d70 RCX: 0000000000000000 Sep 7 14:36:46 unRAID kernel: RDX: 0000000000000000 RSI: ffffffffffffffed RDI: ffff88881e9acdb8 Sep 7 14:36:46 unRAID kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: ffff88881e8ec070 Sep 7 14:36:46 unRAID kernel: R10: 0000000000000000 R11: ffff888100400058 R12: 0000000000000246 Sep 7 14:36:46 unRAID kernel: R13: 0000000000000013 R14: 000000000000000a R15: ffff888100a76e40 Sep 7 14:36:46 unRAID kernel: FS: 0000000000000000(0000) GS:ffff88881e8c0000(0000) knlGS:0000000000000000 Sep 7 14:36:46 unRAID kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Sep 7 14:36:46 unRAID kernel: CR2: 0000000000462cb8 CR3: 000000019f590000 CR4: 0000000000350ee0 Sep 7 14:36:46 unRAID kernel: Call Trace: Sep 7 14:36:46 unRAID kernel: <TASK> Sep 7 14:36:46 unRAID kernel: ? preempt_latency_start+0x2b/0x46 Sep 7 14:36:46 unRAID kernel: rcu_nocb_cb_kthread+0x10f/0x28c Sep 7 14:36:46 unRAID kernel: ? rcu_cpu_kthread+0x10a/0x10a Sep 7 14:36:46 unRAID kernel: kthread+0xe7/0xef Sep 7 14:36:46 unRAID kernel: ? kthread_complete_and_exit+0x1b/0x1b Sep 7 14:36:46 unRAID kernel: ret_from_fork+0x22/0x30 Sep 7 14:36:46 unRAID kernel: </TASK> Sep 7 14:36:46 unRAID kernel: ---[ end trace 0000000000000000 ]--- Sep 7 14:36:46 unRAID kernel: BUG: scheduling while atomic: rcuop/6/57/0x00000002 Sep 7 14:36:46 unRAID kernel: Modules linked in: xt_CHECKSUM xt_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp ip6table_mangle ip6table_nat iptable_mangle vhost_net tun vhost vhost_iotlb tap vhci_hcd usbip_host usbip_core xfs md_mod efivarfs iptable_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha ip6table_filter ip6_tables iptable_filter ip_tables x_tables af_packet 8021q garp mrp bridge stp llc bonding tls ipv6 nvidia_drm(PO) nvidia_modeset(PO) nvidia(PO) drm_kms_helper drm btusb sr_mod btrtl cdrom btbcm kvm_amd kvm crct10dif_pclmul crc32_pclmul crc32c_intel wmi_bmof ghash_clmulni_intel aesni_intel crypto_simd btintel cryptd i2c_piix4 backlight rapl bluetooth ccp k10temp i2c_core igc syscopyarea sysfillrect ahci sysimgblt fb_sys_fops joydev libahci ecdh_generic ecc wmi button unix Sep 7 14:36:46 unRAID kernel: Preemption disabled at: Sep 7 14:36:46 unRAID kernel: [<ffffffff810cf30d>] rcu_do_batch+0x2e6/0x46c Sep 7 14:36:46 unRAID kernel: CPU: 3 PID: 57 Comm: rcuop/6 Tainted: P W O 5.19.3-Unraid #1 Sep 7 14:36:46 unRAID kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./B550 Taichi, BIOS P1.80 02/22/2021 Sep 7 14:36:46 unRAID kernel: Call Trace: Sep 7 14:36:46 unRAID kernel: <TASK> Sep 7 14:36:46 unRAID kernel: dump_stack_lvl+0x44/0x5c Sep 7 14:36:46 unRAID kernel: ? rcu_do_batch+0x2e6/0x46c Sep 7 14:36:46 unRAID kernel: __schedule_bug+0xa3/0xb6 Sep 7 14:36:46 unRAID kernel: __schedule+0x59/0x5f6 Sep 7 14:36:46 unRAID kernel: ? preempt_latency_start+0x2b/0x46 Sep 7 14:36:46 unRAID kernel: schedule+0x8e/0xc3 Sep 7 14:36:46 unRAID kernel: rcu_nocb_cb_kthread+0x90/0x28c Sep 7 14:36:46 unRAID kernel: ? rcu_cpu_kthread+0x10a/0x10a Sep 7 14:36:46 unRAID kernel: kthread+0xe7/0xef Sep 7 14:36:46 unRAID kernel: ? kthread_complete_and_exit+0x1b/0x1b Sep 7 14:36:46 unRAID kernel: ret_from_fork+0x22/0x30 Sep 7 14:36:46 unRAID kernel: </TASK> Sep 7 15:08:57 unRAID kernel: BUG: kernel NULL pointer dereference, address: 000000000000009c Sep 7 15:08:57 unRAID kernel: #PF: supervisor read access in kernel mode Sep 7 15:08:57 unRAID kernel: #PF: error_code(0x0000) - not-present page Sep 7 15:08:57 unRAID kernel: PGD 0 P4D 0 Sep 7 15:08:57 unRAID kernel: Oops: 0000 [#1] PREEMPT SMP NOPTI Sep 7 15:08:57 unRAID kernel: CPU: 6 PID: 0 Comm: swapper/6 Tainted: P W O 5.19.3-Unraid #1 Sep 7 15:08:57 unRAID kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./B550 Taichi, BIOS P1.80 02/22/2021 Sep 7 15:08:57 unRAID kernel: RIP: 0010:update_cfs_rq_load_avg+0x22/0x189 Sep 7 15:08:57 unRAID kernel: Code: 43 80 5b e9 c2 b3 96 00 41 57 41 56 41 55 41 54 55 48 89 fd 53 48 89 f3 41 51 44 8b a6 c4 00 00 00 45 85 e4 0f 84 49 01 00 00 <8b> 86 9c 00 00 00 48 8d be c0 00 00 00 48 89 3c 24 44 8d a8 7e b6 Sep 7 15:08:57 unRAID kernel: RSP: 0018:ffffc90000187e90 EFLAGS: 00010006 Sep 7 15:08:57 unRAID kernel: RAX: ffff8881022c4f0f RBX: ffff88881e9abfc0 RCX: ffff88881e9aca28 Sep 7 15:08:57 unRAID kernel: RDX: ffff88881e9aca28 RSI: 0000000000000000 RDI: ffff88881e9abfc0 Sep 7 15:08:57 unRAID kernel: RBP: ffff8881022c4ec0 R08: 0000000000000000 R09: ffff88881e99f0e0 Sep 7 15:08:57 unRAID kernel: R10: 0000000000000003 R11: 0000000000000000 R12: ffff888100906e40 Sep 7 15:08:57 unRAID kernel: R13: 0000000000000000 R14: 0000000000000000 R15: ffffc90000187ed8 Sep 7 15:08:57 unRAID kernel: FS: 0000000000000000(0000) GS:ffff88881e980000(0000) knlGS:0000000000000000 Sep 7 15:08:57 unRAID kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Sep 7 15:08:57 unRAID kernel: CR2: 000000000000009c CR3: 00000004b0bb8000 CR4: 0000000000350ee0 Sep 7 15:08:57 unRAID kernel: Call Trace: Sep 7 15:08:57 unRAID kernel: <TASK> Sep 7 15:08:57 unRAID kernel: ? __schedule+0x1ba/0x5f6 Sep 7 15:08:57 unRAID kernel: ? schedule_idle+0x27/0x34 Sep 7 15:08:57 unRAID kernel: ? cpu_startup_entry+0x1d/0x1f Sep 7 15:08:57 unRAID kernel: ? start_secondary+0xeb/0xeb Sep 7 15:08:57 unRAID kernel: ? secondary_startup_64_no_verify+0xce/0xdb Sep 7 15:08:57 unRAID kernel: </TASK> Sep 7 15:08:57 unRAID kernel: Modules linked in: xt_CHECKSUM xt_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp ip6table_mangle ip6table_nat iptable_mangle vhost_net tun vhost vhost_iotlb tap vhci_hcd usbip_host usbip_core xfs md_mod efivarfs iptable_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha ip6table_filter ip6_tables iptable_filter ip_tables x_tables af_packet 8021q garp mrp bridge stp llc bonding tls ipv6 nvidia_drm(PO) nvidia_modeset(PO) nvidia(PO) drm_kms_helper drm btusb sr_mod btrtl cdrom btbcm kvm_amd kvm crct10dif_pclmul crc32_pclmul crc32c_intel wmi_bmof ghash_clmulni_intel aesni_intel crypto_simd btintel cryptd i2c_piix4 backlight rapl bluetooth ccp k10temp i2c_core igc syscopyarea sysfillrect ahci sysimgblt fb_sys_fops joydev libahci ecdh_generic ecc wmi button unix Sep 7 15:08:57 unRAID kernel: CR2: 000000000000009c Sep 7 15:08:57 unRAID kernel: ---[ end trace 0000000000000000 ]--- Sep 7 15:08:57 unRAID kernel: xhci_hcd 0000:01:00.0: xHCI host not responding to stop endpoint command Sep 7 15:08:57 unRAID kernel: xhci_hcd 0000:01:00.0: xHCI host controller not responding, assume dead Sep 7 15:08:57 unRAID kernel: xhci_hcd 0000:01:00.0: HC died; cleaning up Sep 7 15:08:57 unRAID kernel: usb 1-3: USB disconnect, device number 2 Sep 7 15:08:57 unRAID kernel: usb 1-5: USB disconnect, device number 3 Sep 7 15:08:57 unRAID kernel: sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x07 driverbyte=DRIVER_OK cmd_age=0s Sep 7 15:08:57 unRAID kernel: sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x2a 2a 00 01 3d 9c 92 00 00 01 00 Sep 7 15:08:57 unRAID kernel: I/O error, dev sda, sector 20814994 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0 Sep 7 15:08:58 unRAID kernel: sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x07 driverbyte=DRIVER_OK cmd_age=0s Sep 7 15:08:58 unRAID kernel: sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x2a 2a 00 00 00 08 01 00 00 01 00 Sep 7 15:08:58 unRAID kernel: I/O error, dev sda, sector 2049 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0 Sep 7 15:08:58 unRAID kernel: Buffer I/O error on dev sda1, logical block 1, lost async page write Sep 7 15:08:58 unRAID kernel: sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x07 driverbyte=DRIVER_OK cmd_age=0s Sep 7 15:08:58 unRAID kernel: sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x2a 2a 00 00 00 2f c4 00 00 01 00 Sep 7 15:08:58 unRAID kernel: I/O error, dev sda, sector 12228 op 0x1:(WRITE) flags 0x100000 phys_seg 1 prio class 0 Sep 7 15:08:58 unRAID kernel: Buffer I/O error on dev sda1, logical block 10180, lost async page write Sep 7 15:08:58 unRAID kernel: sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x07 driverbyte=DRIVER_OK cmd_age=0s Sep 7 15:08:58 unRAID kernel: sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x2a 2a 00 01 3d 9c a2 00 00 01 00 Sep 7 15:08:58 unRAID kernel: I/O error, dev sda, sector 20815010 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0 Sep 7 15:08:58 unRAID kernel: sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x07 driverbyte=DRIVER_OK cmd_age=0s Sep 7 15:08:58 unRAID kernel: sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x2a 2a 00 01 3d 9c c2 00 00 01 00 Sep 7 15:08:58 unRAID kernel: I/O error, dev sda, sector 20815042 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0 Sep 7 15:08:58 unRAID kernel: sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x07 driverbyte=DRIVER_OK cmd_age=0s Sep 7 15:08:58 unRAID kernel: sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x2a 2a 00 00 00 69 04 00 00 02 00 Sep 7 15:08:58 unRAID kernel: I/O error, dev sda, sector 26884 op 0x1:(WRITE) flags 0x100000 phys_seg 2 prio class 0 Sep 7 15:08:58 unRAID kernel: Buffer I/O error on dev sda1, logical block 24836, lost async page write Sep 7 15:08:58 unRAID kernel: Buffer I/O error on dev sda1, logical block 24837, lost async page write Sep 7 15:08:58 unRAID kernel: sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x07 driverbyte=DRIVER_OK cmd_age=0s Sep 7 15:08:58 unRAID kernel: sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x2a 2a 00 00 00 7a a2 00 00 01 00 Sep 7 15:08:58 unRAID kernel: I/O error, dev sda, sector 31394 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0 Sep 7 15:08:58 unRAID kernel: Buffer I/O error on dev sda1, logical block 29346, lost async page write Sep 7 15:08:58 unRAID kernel: sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x07 driverbyte=DRIVER_OK cmd_age=0s Sep 7 15:08:58 unRAID kernel: sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x2a 2a 00 00 00 85 b5 00 00 02 00 Sep 7 15:08:58 unRAID kernel: I/O error, dev sda, sector 34229 op 0x1:(WRITE) flags 0x0 phys_seg 2 prio class 0 Sep 7 15:08:58 unRAID kernel: Buffer I/O error on dev sda1, logical block 32181, lost async page write Sep 7 15:08:58 unRAID kernel: Buffer I/O error on dev sda1, logical block 32182, lost async page write Sep 7 15:08:58 unRAID kernel: sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x07 driverbyte=DRIVER_OK cmd_age=0s Sep 7 15:08:58 unRAID kernel: sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x2a 2a 00 00 00 2f c3 00 00 02 00 Sep 7 15:08:58 unRAID kernel: I/O error, dev sda, sector 12227 op 0x1:(WRITE) flags 0x0 phys_seg 2 prio class 0 Sep 7 15:08:58 unRAID kernel: Buffer I/O error on dev sda1, logical block 10179, lost async page write Sep 7 15:08:58 unRAID kernel: Buffer I/O error on dev sda1, logical block 10180, lost async page write Sep 7 15:08:58 unRAID kernel: sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x07 driverbyte=DRIVER_OK cmd_age=0s Sep 7 15:08:58 unRAID kernel: sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x28 28 00 00 00 85 b5 00 00 01 00 Sep 7 15:08:58 unRAID kernel: I/O error, dev sda, sector 34229 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Sep 7 15:08:58 unRAID kernel: FAT-fs (sda1): Directory bread(block 32181) failed Sep 7 15:08:58 unRAID kernel: FAT-fs (sda1): Directory bread(block 32182) failed Sep 7 15:08:58 unRAID kernel: FAT-fs (sda1): Directory bread(block 32181) failed Sep 7 15:08:58 unRAID kernel: FAT-fs (sda1): Directory bread(block 32182) failed Sep 7 15:08:58 unRAID kernel: FAT-fs (sda1): Directory bread(block 32181) failed I am thinking of creating a backup and reinstalling Unraid on the stick... Is there anything to consider before I do this? For example, the layout of the hard drives or saving the docker templates? syslog unraid-diagnostics-20220907-1336.zip Quote Link to comment
Kulisch Posted September 19, 2022 Author Share Posted September 19, 2022 I suspect that the problem is with the array itself. The system does not freeze when docker is completely off. The system freezes depending on which container is on. Using the VM under /mnt/user/domains/ causes problems, freezes the VM or Unraid itself. Under /mnt/cache/domains/ I have not noticed any of the mentioned problems. I have performed a RAM check again which was successful. Also a Read-Check. I also checked the S.M.A.R.T. values and everything looks fine there. Do I have to rsync all the data and move it back or is there an easy way to solve the problem? 1 hard disk for cache (pool) and one disk 1 (array). And one of my windows disks the partition layout is broken. It is only recognized as RAW. Does anyone know how I can fix this? Quote Link to comment
JorgeB Posted September 19, 2022 Share Posted September 19, 2022 1 hour ago, Kulisch said: And one of my windows disks the partition layout is broken. It is only recognized as RAW. Does anyone know how I can fix this? Try testdisk in Windows. Quote Link to comment
Kulisch Posted September 19, 2022 Author Share Posted September 19, 2022 After creating a Windows 11 VM and attaching the corrupted Harddrive it detected, the Problem was found during boot by the OS and repaired it. Now I only have to get the persmissions for my Harddrive. I only can list the files in powershell as a administrator. Explorer says the permissions are missing. Using explorer as admin doesnt change anything. Still, I have to repair the array. Is there something that I can use in Unraid to fix this problem. Setting permissions or moving the files? Or do I have delete everything and building up the array from beginning? Quote Link to comment
JorgeB Posted September 19, 2022 Share Posted September 19, 2022 What is the array problem? Please post the diagnostics. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.