Unraid not responsive (only ping)

[Kulisch]

Hello everyone,

 

my Unraid server is not accessible. (SSH, WebUI, Docker, VM,...)

 

But there is no timeout, no drop, no block, nothing. SSH (no connection reset) and HTTP requests (Just blank, no error code) taking forever without any response. 

 

Only pinging the server is possible. This happens randomly. (uptime: 2 days, 4 days, 10 days)

 

 

I can see that the server has a peak at power consumption in that time it happened. It "frozen" at 13:25 - 13:30 

 

 

It is an AMD Ryzen 7 3800X build, so I did some UEFI settings like deactivating C-States, VFIO-Passthrough, B550 Taichi, BIOS Update Ver 2.0, no Memory Overclock (default 2133 MHZ, 2 x 16 GB). Maybe I'm missing something...

 

Im having the feeling that this happens randomly during using a VM (Windows with GPU Passthrough). I do not remember any scenario where this happened and the VM was not started. I dont have a second GPU or a APU. 

 

Syslog to USB is activated and I can provide the logs after I restart the server.

 

But before I do this, is there anything I should try before shutting down the server by pushing the button or cutting off the power?

 

Thank you in advance

Edited July 14, 2022 by Kulisch

[JorgeB]

Enable the syslog server and post that together with the diagnostics after the next crash.

[Kulisch]

Thank you. There you go:

 

unraid-diagnostics-20220702-1242.zipusb-syslogremote-syslog

[JorgeB]

Nothing obvious, some OOM errors and kvm related call traces so you might see if the issue doesn't occur if the VM(s) is(are) not running.

[Kulisch]

After 19 hours it happened again, while using Remote Desktop. This time another VM (Win 11). Used it for 1 hour (Not passtroughing anything) and it froze at 11:18.

 

This time, system is completely frozen. Trying to access over SSH and HTTP give me timeout responses.

 

unraid-diagnostics-20220715-1144.zipserver-syslogusb-syslog

 

Is it possible to enable and save debug logs of a VM? Or other advices? Because I dont remember having a frozen system without running a VM.

Edited July 15, 2022 by Kulisch

[JorgeB]

Again nothing relevant logged, make sure you're following the Ryzen specif recommendations, but if it only happens when using VMs it's probably not that.

[Kulisch]

I checked the settings and changed from "Auto" to "typical current idle". After using a VM for several hours, unraid "freezes" again.

 

Reverted to BIOS Update from 2.0 to 1.8 because I know that this Version worked proberly. Of course I checked the BIOS Settings after Downgrade and made the ryzen specific changes.

 

Now I reverted Unraid 6.10.3 to 6.10.2.

 

This time I see the VM (newly created Windows 11, nothing changed but windows updates) is not responsive after est. 2 hours idling.

 

 

Settings VM:

 

 

Now I'm pretty sure, that this problem does only occurs using a VM. Unraid was running for 3 days without using a VM and nothing bad happened. Every time I start a machine, after 1 or 2 hours the server is not responsive. Just pingable.

 

unraid-diagnostics-20220722-1002.zipusb-syslogserver-syslog

Edited July 22, 2022 by Kulisch

[trurl]

Do you have an attached monitor and keyboard? 

[Kulisch]

Update: After the revert to 6.10.2 the Windows 10 and 11 machines are freezing after 1 hour. Not Unraid anymore. 👍

 

The Kali machine (without GUI) is online over 4 hours and nothing special is going wrong.

 

So I consider the Guest OS makes the trouble. 

 

Do I have to check the "Energy saving" options from Windows? (Both options are on "never")

 

 

37 minutes ago, trurl said:

Do you have an attached monitor and keyboard?

 

Yes both of them. But there are not used for the VM in that case.

 

Mouse and keyboard are passthrough(ed?) with the USB Controller.

 

Monitor is connected with the only GPU which is also passthrough(ed?).

[trurl]

2 hours ago, trurl said:

Do you have an attached monitor and keyboard? 

 

I wanted to know if you could access the Unraid command line or the Unraid webUI on the server itself instead of over the network.

[Kulisch]

21 minutes ago, trurl said:

 

I wanted to know if you could access the Unraid command line or the Unraid webUI on the server itself instead of over the network.

 

I could. I would have to remove the GPU and the USB controller from the passthrough, but that should not affect the error pattern. After all, it affected the VMs with and without passthrough.

 

Should I check something specific or should I just see if the system reacts at all when "freezing".

[trurl]

See what you get from command line with this

dh -h

Also might see if you can get diagnostics from command line

[Kulisch]

See what you get from command line with this

dh -h

Also might see if you can get diagnostics from command line

Ok, I will try that in the next days. Trank you


Gesendet von iPhone mit Tapatalk

[trurl]

Sorry, should be

df -h

 

[Kulisch]

So I updated to 6.10.3 again, took the IOMMU Groups back to Unraid, started a VM without any connected hardware device and it froze again.

 

After checking the monitor I saw a Kernel Panic happened. So I couldn't check the diagnostics or "disk free" (df -h).

 

 

 

 

Edit:

 

Forgot to give you the diagnostics and logs after reboot.

 

server-syslogusb-syslogunraid-diagnostics-20220726-0759.zip

 

After starting Unraid, this messages showed up:

Edited July 26, 2022 by Kulisch

[Kulisch]

1. I installed the RC2 to test if there are still the same problems.

 

After 18 Days and 5 hours uptime I started a Windows 11 and a Kali VM which I started installing (Stopped at user creation). Just to check if the problem appears again. I lost my RDP Session, tried to ping, nothing happened. This freeze happened after 20 - 30 minutes. 

 

2. i checked the Syslog and it increased a little bit. 42.5 mb...

 

Jul 27 08:06:18 unRAID kernel: vfio-pci 0000:0a:00.0: BAR 3: can't reserve [mem 0xe0000000-0xe1ffffff 64bit pref]
Jul 27 08:06:18 unRAID kernel: vfio-pci 0000:0a:00.0: BAR 3: can't reserve [mem 0xe0000000-0xe1ffffff 64bit pref]
Jul 27 08:06:18 unRAID kernel: vfio-pci 0000:0a:00.0: BAR 3: can't reserve [mem 0xe0000000-0xe1ffffff 64bit pref]
Jul 27 08:06:18 unRAID kernel: vfio-pci 0000:0a:00.0: BAR 3: can't reserve [mem 0xe0000000-0xe1ffffff 64bit pref]
Jul 27 08:06:18 unRAID kernel: vfio-pci 0000:0a:00.0: BAR 3: can't reserve [mem 0xe0000000-0xe1ffffff 64bit pref]
Jul 27 08:06:18 unRAID kernel: vfio-pci 0000:0a:00.0: BAR 3: can't reserve [mem 0xe0000000-0xe1ffffff 64bit pref]

 

This line shows 364433 times up.

 

I removed these lines and replaced it with ---

syslog-edit.log unraid-diagnostics-20220824-0756.zip

[Kulisch]

Yesterday I made a memtest which passed.

 

I'm running out of ideas. Does anyone have any suggestions what I could do? Because unfortunately this can not stay like this. I depend on the VMs and it is among other things the reason why I use Unraid at all.

[trurl]

The point of

df -h

was to see if you had filled rootfs, which is in RAM and is where the OS is stored. If you fill it the OS won't have any room to work in.

[Kulisch]

5 hours ago, trurl said:

The point of

df -h

was to see if you had filled rootfs, which is in RAM and is where the OS is stored. If you fill it the OS won't have any room to work in.

 

Thanks for the reply.

 

Then I take back the GPU from passtrough again, hoping that no crash happens and I can get the informotions locally. 

 

Assuming the RAM would fill up, would a RAM expansion be enough? I had planned to go from 32 GB to 64 anyway.

[trurl]

6 hours ago, Kulisch said:

Assuming the RAM would fill up, would a RAM expansion be enough?

More RAM is not the solution. If rootfs is filling you would need to figure out why. Nothing should be writing into rootfs except the OS. A common reason for filling rootfs is specifying a path that isn't mounted storage, such as a container host path.

[Kulisch]

Everytime I use the VM for an hour, happens a kernel panic in unraid. So there is no chance to check anything after that.

 

After disabling Docker and starting the VM the CPU goes on some CPU Threads on 100 %. (Windows 11 fresh install)

 

root@unRAID:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
rootfs           16G  2.0G   14G  13% /
tmpfs            32M  420K   32M   2% /run
/dev/sda1        15G  3.2G   12G  23% /boot
overlay          16G  2.0G   14G  13% /lib/firmware
overlay          16G  2.0G   14G  13% /lib/modules
devtmpfs        8.0M     0  8.0M   0% /dev
tmpfs            16G     0   16G   0% /dev/shm
cgroup_root     8.0M     0  8.0M   0% /sys/fs/cgroup
tmpfs           128M  476K  128M   1% /var/log
tmpfs           1.0M     0  1.0M   0% /mnt/disks
tmpfs           1.0M     0  1.0M   0% /mnt/remotes
tmpfs           1.0M     0  1.0M   0% /mnt/rootshare
/dev/md1        2.8T  1.3T  1.5T  48% /mnt/disk1
/dev/sdb1       466G  200G  264G  44% /mnt/cache
shfs            2.8T  1.3T  1.5T  48% /mnt/user0
shfs            2.8T  1.3T  1.5T  48% /mnt/user
/dev/loop3       50G  4.9M   50G   1% /etc/libvirt

 

 

Forcing the shutdown doesn't work and after the 2nd try, Unraid crashes.

 

After booting up, started the VM and it crashed after 1 hour again.

 

root@unRAID:~# tail -f /var/log/syslog
Sep  7 14:03:20 unRAID  avahi-daemon[10293]: New relevant interface vnet0.IPv6 for mDNS.
Sep  7 14:03:20 unRAID  avahi-daemon[10293]: Registering new address record for fe80::fc54:ff:fe8d:ec92 on vnet0.*.
Sep  7 14:03:20 unRAID flash_backup: adding task: /usr/local/emhttp/plugins/dynamix.my.servers/scripts/UpdateFlashBackup update
Sep  7 14:05:59 unRAID  ntpd[1724]: kernel reports TIME_ERROR: 0x41: Clock Unsynchronized
Sep  7 14:06:51 unRAID  sshd[15169]: Connection from 10.0.10.117 port 1579 on 10.0.10.10 port 22 rdomain ""
Sep  7 14:06:51 unRAID  sshd[15169]: Postponed keyboard-interactive for root from 10.0.10.117 port 1579 ssh2 [preauth]
Sep  7 14:06:56 unRAID  sshd[15169]: Postponed keyboard-interactive/pam for root from 10.0.10.117 port 1579 ssh2 [preauth]
Sep  7 14:06:56 unRAID  sshd[15169]: Accepted keyboard-interactive/pam for root from 10.0.10.117 port 1579 ssh2
Sep  7 14:06:56 unRAID  sshd[15169]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0)
Sep  7 14:06:56 unRAID  sshd[15169]: Starting session: shell on pts/1 for root from 10.0.10.117 port 1579 id 0
Sep  7 14:07:55 unRAID  avahi-daemon[10293]: Interface vnet0.IPv6 no longer relevant for mDNS.
Sep  7 14:07:55 unRAID  avahi-daemon[10293]: Leaving mDNS multicast group on interface vnet0.IPv6 with address fe80::fc54:ff:fe8d:ec92.
Sep  7 14:07:55 unRAID kernel: br0: port 2(vnet0) entered disabled state
Sep  7 14:07:55 unRAID kernel: device vnet0 left promiscuous mode
Sep  7 14:07:55 unRAID kernel: br0: port 2(vnet0) entered disabled state
Sep  7 14:07:55 unRAID  avahi-daemon[10293]: Withdrawing address record for fe80::fc54:ff:fe8d:ec92 on vnet0.
Sep  7 14:08:21 unRAID kernel: br0: port 2(vnet1) entered blocking state
Sep  7 14:08:21 unRAID kernel: br0: port 2(vnet1) entered disabled state
Sep  7 14:08:21 unRAID kernel: device vnet1 entered promiscuous mode
Sep  7 14:08:21 unRAID kernel: br0: port 2(vnet1) entered blocking state
Sep  7 14:08:21 unRAID kernel: br0: port 2(vnet1) entered forwarding state
Sep  7 14:08:22 unRAID  avahi-daemon[10293]: Joining mDNS multicast group on interface vnet1.IPv6 with address fe80::fc54:ff:fe8d:ec92.
Sep  7 14:08:22 unRAID  avahi-daemon[10293]: New relevant interface vnet1.IPv6 for mDNS.
Sep  7 14:08:22 unRAID  avahi-daemon[10293]: Registering new address record for fe80::fc54:ff:fe8d:ec92 on vnet1.*.
Sep  7 14:08:39 unRAID  nmbd[10250]: [2022/09/07 14:08:39.002205,  0] ../../source3/nmbd/nmbd_become_lmb.c:398(become_local_master_stage2)
Sep  7 14:08:39 unRAID  nmbd[10250]:   *****
Sep  7 14:08:39 unRAID  nmbd[10250]:
Sep  7 14:08:39 unRAID  nmbd[10250]:   Samba name server UNRAID is now a local master browser for workgroup WORKGROUP on subnet 192.168.122.1
Sep  7 14:08:39 unRAID  nmbd[10250]:
Sep  7 14:08:39 unRAID  nmbd[10250]:   *****
Sep  7 14:11:00 unRAID root: Fix Common Problems Version 2022.08.29
Sep  7 14:11:07 unRAID root: Fix Common Problems: Warning: NerdPack.plg Not Compatible with Unraid version 6.11.0-rc4
Sep  7 14:11:10 unRAID root: Fix Common Problems: Warning: Syslog mirrored to flash
Sep  7 14:11:10 unRAID root: Fix Common Problems: Warning: The plugin usb_manager_usbip_addon.plg is not known to Community Applications and is possibly incompatible with your server
Sep  7 14:36:46 unRAID kernel: ------------[ cut here ]------------
Sep  7 14:36:46 unRAID kernel: WARNING: CPU: 3 PID: 57 at kernel/rcu/tree.c:2634 rcu_do_batch+0x3d8/0x46c
Sep  7 14:36:46 unRAID kernel: Modules linked in: xt_CHECKSUM xt_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp ip6table_mangle ip6table_nat iptable_mangle vhost_net tun vhost vhost_iotlb tap vhci_hcd usbip_host usbip_core xfs md_mod efivarfs iptable_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha ip6table_filter ip6_tables iptable_filter ip_tables x_tables af_packet 8021q garp mrp bridge stp llc bonding tls ipv6 nvidia_drm(PO) nvidia_modeset(PO) nvidia(PO) drm_kms_helper drm btusb sr_mod btrtl cdrom btbcm kvm_amd kvm crct10dif_pclmul crc32_pclmul crc32c_intel wmi_bmof ghash_clmulni_intel aesni_intel crypto_simd btintel cryptd i2c_piix4 backlight rapl bluetooth ccp k10temp i2c_core igc syscopyarea sysfillrect ahci sysimgblt fb_sys_fops joydev libahci ecdh_generic ecc wmi button unix
Sep  7 14:36:46 unRAID kernel: CPU: 3 PID: 57 Comm: rcuop/6 Tainted: P           O      5.19.3-Unraid #1
Sep  7 14:36:46 unRAID kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./B550 Taichi, BIOS P1.80 02/22/2021
Sep  7 14:36:46 unRAID kernel: RIP: 0010:rcu_do_batch+0x3d8/0x46c
Sep  7 14:36:46 unRAID kernel: Code: 00 00 00 75 49 48 85 d2 74 44 31 c0 48 89 83 f0 00 00 00 48 8b 05 71 c6 fe 00 48 89 83 00 01 00 00 48 8b 43 78 48 85 c0 74 4a <0f> 0b 48 8b 3c 24 e8 91 54 00 00 48 85 c0 74 02 0f 0b 48 8b 3c 24
Sep  7 14:36:46 unRAID kernel: RSP: 0018:ffffc9000039be60 EFLAGS: 00010086
Sep  7 14:36:46 unRAID kernel: RAX: ffff888101494000 RBX: ffff888122390d70 RCX: 0000000000000000
Sep  7 14:36:46 unRAID kernel: RDX: 0000000100000000 RSI: ffffffffffffffed RDI: ffff88881e9acdb8
Sep  7 14:36:46 unRAID kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: ffff88881e8ec070
Sep  7 14:36:46 unRAID kernel: R10: 0000000000000000 R11: ffff888100400058 R12: 0000000000000246
Sep  7 14:36:46 unRAID kernel: R13: 0000000000000013 R14: 000000000000000a R15: ffff888100a76e40
Sep  7 14:36:46 unRAID kernel: FS:  0000000000000000(0000) GS:ffff88881e8c0000(0000) knlGS:0000000000000000
Sep  7 14:36:46 unRAID kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Sep  7 14:36:46 unRAID kernel: CR2: 0000000000462cb8 CR3: 000000019f590000 CR4: 0000000000350ee0
Sep  7 14:36:46 unRAID kernel: Call Trace:
Sep  7 14:36:46 unRAID kernel: <TASK>
Sep  7 14:36:46 unRAID kernel: ? preempt_latency_start+0x2b/0x46
Sep  7 14:36:46 unRAID kernel: rcu_nocb_cb_kthread+0x10f/0x28c
Sep  7 14:36:46 unRAID kernel: ? rcu_cpu_kthread+0x10a/0x10a
Sep  7 14:36:46 unRAID kernel: kthread+0xe7/0xef
Sep  7 14:36:46 unRAID kernel: ? kthread_complete_and_exit+0x1b/0x1b
Sep  7 14:36:46 unRAID kernel: ret_from_fork+0x22/0x30
Sep  7 14:36:46 unRAID kernel: </TASK>
Sep  7 14:36:46 unRAID kernel: ---[ end trace 0000000000000000 ]---
Sep  7 14:36:46 unRAID kernel: ------------[ cut here ]------------
Sep  7 14:36:46 unRAID kernel: WARNING: CPU: 3 PID: 57 at kernel/rcu/tree.c:2637 rcu_do_batch+0x3e8/0x46c
Sep  7 14:36:46 unRAID kernel: Modules linked in: xt_CHECKSUM xt_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp ip6table_mangle ip6table_nat iptable_mangle vhost_net tun vhost vhost_iotlb tap vhci_hcd usbip_host usbip_core xfs md_mod efivarfs iptable_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha ip6table_filter ip6_tables iptable_filter ip_tables x_tables af_packet 8021q garp mrp bridge stp llc bonding tls ipv6 nvidia_drm(PO) nvidia_modeset(PO) nvidia(PO) drm_kms_helper drm btusb sr_mod btrtl cdrom btbcm kvm_amd kvm crct10dif_pclmul crc32_pclmul crc32c_intel wmi_bmof ghash_clmulni_intel aesni_intel crypto_simd btintel cryptd i2c_piix4 backlight rapl bluetooth ccp k10temp i2c_core igc syscopyarea sysfillrect ahci sysimgblt fb_sys_fops joydev libahci ecdh_generic ecc wmi button unix
Sep  7 14:36:46 unRAID kernel: CPU: 3 PID: 57 Comm: rcuop/6 Tainted: P        W  O      5.19.3-Unraid #1
Sep  7 14:36:46 unRAID kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./B550 Taichi, BIOS P1.80 02/22/2021
Sep  7 14:36:46 unRAID kernel: RIP: 0010:rcu_do_batch+0x3e8/0x46c
Sep  7 14:36:46 unRAID kernel: Code: 00 00 00 48 8b 05 71 c6 fe 00 48 89 83 00 01 00 00 48 8b 43 78 48 85 c0 74 4a 0f 0b 48 8b 3c 24 e8 91 54 00 00 48 85 c0 74 02 <0f> 0b 48 8b 3c 24 e8 81 54 00 00 48 85 c0 75 41 0f 0b eb 3d 48 2b
Sep  7 14:36:46 unRAID kernel: RSP: 0018:ffffc9000039be60 EFLAGS: 00010002
Sep  7 14:36:46 unRAID kernel: RAX: 0000000000000023 RBX: ffff888122390d70 RCX: 0000000000000000
Sep  7 14:36:46 unRAID kernel: RDX: 0000000000000000 RSI: ffffffffffffffed RDI: ffff88881e9acdb8
Sep  7 14:36:46 unRAID kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: ffff88881e8ec070
Sep  7 14:36:46 unRAID kernel: R10: 0000000000000000 R11: ffff888100400058 R12: 0000000000000246
Sep  7 14:36:46 unRAID kernel: R13: 0000000000000013 R14: 000000000000000a R15: ffff888100a76e40
Sep  7 14:36:46 unRAID kernel: FS:  0000000000000000(0000) GS:ffff88881e8c0000(0000) knlGS:0000000000000000
Sep  7 14:36:46 unRAID kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Sep  7 14:36:46 unRAID kernel: CR2: 0000000000462cb8 CR3: 000000019f590000 CR4: 0000000000350ee0
Sep  7 14:36:46 unRAID kernel: Call Trace:
Sep  7 14:36:46 unRAID kernel: <TASK>
Sep  7 14:36:46 unRAID kernel: ? preempt_latency_start+0x2b/0x46
Sep  7 14:36:46 unRAID kernel: rcu_nocb_cb_kthread+0x10f/0x28c
Sep  7 14:36:46 unRAID kernel: ? rcu_cpu_kthread+0x10a/0x10a
Sep  7 14:36:46 unRAID kernel: kthread+0xe7/0xef
Sep  7 14:36:46 unRAID kernel: ? kthread_complete_and_exit+0x1b/0x1b
Sep  7 14:36:46 unRAID kernel: ret_from_fork+0x22/0x30
Sep  7 14:36:46 unRAID kernel: </TASK>
Sep  7 14:36:46 unRAID kernel: ---[ end trace 0000000000000000 ]---
Sep  7 14:36:46 unRAID kernel: BUG: scheduling while atomic: rcuop/6/57/0x00000002
Sep  7 14:36:46 unRAID kernel: Modules linked in: xt_CHECKSUM xt_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp ip6table_mangle ip6table_nat iptable_mangle vhost_net tun vhost vhost_iotlb tap vhci_hcd usbip_host usbip_core xfs md_mod efivarfs iptable_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha ip6table_filter ip6_tables iptable_filter ip_tables x_tables af_packet 8021q garp mrp bridge stp llc bonding tls ipv6 nvidia_drm(PO) nvidia_modeset(PO) nvidia(PO) drm_kms_helper drm btusb sr_mod btrtl cdrom btbcm kvm_amd kvm crct10dif_pclmul crc32_pclmul crc32c_intel wmi_bmof ghash_clmulni_intel aesni_intel crypto_simd btintel cryptd i2c_piix4 backlight rapl bluetooth ccp k10temp i2c_core igc syscopyarea sysfillrect ahci sysimgblt fb_sys_fops joydev libahci ecdh_generic ecc wmi button unix
Sep  7 14:36:46 unRAID kernel: Preemption disabled at:
Sep  7 14:36:46 unRAID kernel: [<ffffffff810cf30d>] rcu_do_batch+0x2e6/0x46c
Sep  7 14:36:46 unRAID kernel: CPU: 3 PID: 57 Comm: rcuop/6 Tainted: P        W  O      5.19.3-Unraid #1
Sep  7 14:36:46 unRAID kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./B550 Taichi, BIOS P1.80 02/22/2021
Sep  7 14:36:46 unRAID kernel: Call Trace:
Sep  7 14:36:46 unRAID kernel: <TASK>
Sep  7 14:36:46 unRAID kernel: dump_stack_lvl+0x44/0x5c
Sep  7 14:36:46 unRAID kernel: ? rcu_do_batch+0x2e6/0x46c
Sep  7 14:36:46 unRAID kernel: __schedule_bug+0xa3/0xb6
Sep  7 14:36:46 unRAID kernel: __schedule+0x59/0x5f6
Sep  7 14:36:46 unRAID kernel: ? preempt_latency_start+0x2b/0x46
Sep  7 14:36:46 unRAID kernel: schedule+0x8e/0xc3
Sep  7 14:36:46 unRAID kernel: rcu_nocb_cb_kthread+0x90/0x28c
Sep  7 14:36:46 unRAID kernel: ? rcu_cpu_kthread+0x10a/0x10a
Sep  7 14:36:46 unRAID kernel: kthread+0xe7/0xef
Sep  7 14:36:46 unRAID kernel: ? kthread_complete_and_exit+0x1b/0x1b
Sep  7 14:36:46 unRAID kernel: ret_from_fork+0x22/0x30
Sep  7 14:36:46 unRAID kernel: </TASK>
Sep  7 15:08:57 unRAID kernel: BUG: kernel NULL pointer dereference, address: 000000000000009c
Sep  7 15:08:57 unRAID kernel: #PF: supervisor read access in kernel mode
Sep  7 15:08:57 unRAID kernel: #PF: error_code(0x0000) - not-present page
Sep  7 15:08:57 unRAID kernel: PGD 0 P4D 0
Sep  7 15:08:57 unRAID kernel: Oops: 0000 [#1] PREEMPT SMP NOPTI
Sep  7 15:08:57 unRAID kernel: CPU: 6 PID: 0 Comm: swapper/6 Tainted: P        W  O      5.19.3-Unraid #1
Sep  7 15:08:57 unRAID kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./B550 Taichi, BIOS P1.80 02/22/2021
Sep  7 15:08:57 unRAID kernel: RIP: 0010:update_cfs_rq_load_avg+0x22/0x189
Sep  7 15:08:57 unRAID kernel: Code: 43 80 5b e9 c2 b3 96 00 41 57 41 56 41 55 41 54 55 48 89 fd 53 48 89 f3 41 51 44 8b a6 c4 00 00 00 45 85 e4 0f 84 49 01 00 00 <8b> 86 9c 00 00 00 48 8d be c0 00 00 00 48 89 3c 24 44 8d a8 7e b6
Sep  7 15:08:57 unRAID kernel: RSP: 0018:ffffc90000187e90 EFLAGS: 00010006
Sep  7 15:08:57 unRAID kernel: RAX: ffff8881022c4f0f RBX: ffff88881e9abfc0 RCX: ffff88881e9aca28
Sep  7 15:08:57 unRAID kernel: RDX: ffff88881e9aca28 RSI: 0000000000000000 RDI: ffff88881e9abfc0
Sep  7 15:08:57 unRAID kernel: RBP: ffff8881022c4ec0 R08: 0000000000000000 R09: ffff88881e99f0e0
Sep  7 15:08:57 unRAID kernel: R10: 0000000000000003 R11: 0000000000000000 R12: ffff888100906e40
Sep  7 15:08:57 unRAID kernel: R13: 0000000000000000 R14: 0000000000000000 R15: ffffc90000187ed8
Sep  7 15:08:57 unRAID kernel: FS:  0000000000000000(0000) GS:ffff88881e980000(0000) knlGS:0000000000000000
Sep  7 15:08:57 unRAID kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Sep  7 15:08:57 unRAID kernel: CR2: 000000000000009c CR3: 00000004b0bb8000 CR4: 0000000000350ee0
Sep  7 15:08:57 unRAID kernel: Call Trace:
Sep  7 15:08:57 unRAID kernel: <TASK>
Sep  7 15:08:57 unRAID kernel: ? __schedule+0x1ba/0x5f6
Sep  7 15:08:57 unRAID kernel: ? schedule_idle+0x27/0x34
Sep  7 15:08:57 unRAID kernel: ? cpu_startup_entry+0x1d/0x1f
Sep  7 15:08:57 unRAID kernel: ? start_secondary+0xeb/0xeb
Sep  7 15:08:57 unRAID kernel: ? secondary_startup_64_no_verify+0xce/0xdb
Sep  7 15:08:57 unRAID kernel: </TASK>
Sep  7 15:08:57 unRAID kernel: Modules linked in: xt_CHECKSUM xt_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp ip6table_mangle ip6table_nat iptable_mangle vhost_net tun vhost vhost_iotlb tap vhci_hcd usbip_host usbip_core xfs md_mod efivarfs iptable_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha ip6table_filter ip6_tables iptable_filter ip_tables x_tables af_packet 8021q garp mrp bridge stp llc bonding tls ipv6 nvidia_drm(PO) nvidia_modeset(PO) nvidia(PO) drm_kms_helper drm btusb sr_mod btrtl cdrom btbcm kvm_amd kvm crct10dif_pclmul crc32_pclmul crc32c_intel wmi_bmof ghash_clmulni_intel aesni_intel crypto_simd btintel cryptd i2c_piix4 backlight rapl bluetooth ccp k10temp i2c_core igc syscopyarea sysfillrect ahci sysimgblt fb_sys_fops joydev libahci ecdh_generic ecc wmi button unix
Sep  7 15:08:57 unRAID kernel: CR2: 000000000000009c
Sep  7 15:08:57 unRAID kernel: ---[ end trace 0000000000000000 ]---
Sep  7 15:08:57 unRAID kernel: xhci_hcd 0000:01:00.0: xHCI host not responding to stop endpoint command
Sep  7 15:08:57 unRAID kernel: xhci_hcd 0000:01:00.0: xHCI host controller not responding, assume dead
Sep  7 15:08:57 unRAID kernel: xhci_hcd 0000:01:00.0: HC died; cleaning up
Sep  7 15:08:57 unRAID kernel: usb 1-3: USB disconnect, device number 2
Sep  7 15:08:57 unRAID kernel: usb 1-5: USB disconnect, device number 3
Sep  7 15:08:57 unRAID kernel: sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x07 driverbyte=DRIVER_OK cmd_age=0s
Sep  7 15:08:57 unRAID kernel: sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x2a 2a 00 01 3d 9c 92 00 00 01 00
Sep  7 15:08:57 unRAID kernel: I/O error, dev sda, sector 20814994 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
Sep  7 15:08:58 unRAID kernel: sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x07 driverbyte=DRIVER_OK cmd_age=0s
Sep  7 15:08:58 unRAID kernel: sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x2a 2a 00 00 00 08 01 00 00 01 00
Sep  7 15:08:58 unRAID kernel: I/O error, dev sda, sector 2049 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
Sep  7 15:08:58 unRAID kernel: Buffer I/O error on dev sda1, logical block 1, lost async page write
Sep  7 15:08:58 unRAID kernel: sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x07 driverbyte=DRIVER_OK cmd_age=0s
Sep  7 15:08:58 unRAID kernel: sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x2a 2a 00 00 00 2f c4 00 00 01 00
Sep  7 15:08:58 unRAID kernel: I/O error, dev sda, sector 12228 op 0x1:(WRITE) flags 0x100000 phys_seg 1 prio class 0
Sep  7 15:08:58 unRAID kernel: Buffer I/O error on dev sda1, logical block 10180, lost async page write
Sep  7 15:08:58 unRAID kernel: sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x07 driverbyte=DRIVER_OK cmd_age=0s
Sep  7 15:08:58 unRAID kernel: sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x2a 2a 00 01 3d 9c a2 00 00 01 00
Sep  7 15:08:58 unRAID kernel: I/O error, dev sda, sector 20815010 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
Sep  7 15:08:58 unRAID kernel: sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x07 driverbyte=DRIVER_OK cmd_age=0s
Sep  7 15:08:58 unRAID kernel: sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x2a 2a 00 01 3d 9c c2 00 00 01 00
Sep  7 15:08:58 unRAID kernel: I/O error, dev sda, sector 20815042 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
Sep  7 15:08:58 unRAID kernel: sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x07 driverbyte=DRIVER_OK cmd_age=0s
Sep  7 15:08:58 unRAID kernel: sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x2a 2a 00 00 00 69 04 00 00 02 00
Sep  7 15:08:58 unRAID kernel: I/O error, dev sda, sector 26884 op 0x1:(WRITE) flags 0x100000 phys_seg 2 prio class 0
Sep  7 15:08:58 unRAID kernel: Buffer I/O error on dev sda1, logical block 24836, lost async page write
Sep  7 15:08:58 unRAID kernel: Buffer I/O error on dev sda1, logical block 24837, lost async page write
Sep  7 15:08:58 unRAID kernel: sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x07 driverbyte=DRIVER_OK cmd_age=0s
Sep  7 15:08:58 unRAID kernel: sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x2a 2a 00 00 00 7a a2 00 00 01 00
Sep  7 15:08:58 unRAID kernel: I/O error, dev sda, sector 31394 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
Sep  7 15:08:58 unRAID kernel: Buffer I/O error on dev sda1, logical block 29346, lost async page write
Sep  7 15:08:58 unRAID kernel: sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x07 driverbyte=DRIVER_OK cmd_age=0s
Sep  7 15:08:58 unRAID kernel: sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x2a 2a 00 00 00 85 b5 00 00 02 00
Sep  7 15:08:58 unRAID kernel: I/O error, dev sda, sector 34229 op 0x1:(WRITE) flags 0x0 phys_seg 2 prio class 0
Sep  7 15:08:58 unRAID kernel: Buffer I/O error on dev sda1, logical block 32181, lost async page write
Sep  7 15:08:58 unRAID kernel: Buffer I/O error on dev sda1, logical block 32182, lost async page write
Sep  7 15:08:58 unRAID kernel: sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x07 driverbyte=DRIVER_OK cmd_age=0s
Sep  7 15:08:58 unRAID kernel: sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x2a 2a 00 00 00 2f c3 00 00 02 00
Sep  7 15:08:58 unRAID kernel: I/O error, dev sda, sector 12227 op 0x1:(WRITE) flags 0x0 phys_seg 2 prio class 0
Sep  7 15:08:58 unRAID kernel: Buffer I/O error on dev sda1, logical block 10179, lost async page write
Sep  7 15:08:58 unRAID kernel: Buffer I/O error on dev sda1, logical block 10180, lost async page write
Sep  7 15:08:58 unRAID kernel: sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x07 driverbyte=DRIVER_OK cmd_age=0s
Sep  7 15:08:58 unRAID kernel: sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x28 28 00 00 00 85 b5 00 00 01 00
Sep  7 15:08:58 unRAID kernel: I/O error, dev sda, sector 34229 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Sep  7 15:08:58 unRAID kernel: FAT-fs (sda1): Directory bread(block 32181) failed
Sep  7 15:08:58 unRAID kernel: FAT-fs (sda1): Directory bread(block 32182) failed
Sep  7 15:08:58 unRAID kernel: FAT-fs (sda1): Directory bread(block 32181) failed
Sep  7 15:08:58 unRAID kernel: FAT-fs (sda1): Directory bread(block 32182) failed
Sep  7 15:08:58 unRAID kernel: FAT-fs (sda1): Directory bread(block 32181) failed

 

I am thinking of creating a backup and reinstalling Unraid on the stick...

 

Is there anything to consider before I do this? For example, the layout of the hard drives or saving the docker templates?

syslog unraid-diagnostics-20220907-1336.zip

[Kulisch]

I suspect that the problem is with the array itself.

 

The system does not freeze when docker is completely off. The system freezes depending on which container is on.

 

Using the VM under /mnt/user/domains/ causes problems, freezes the VM or Unraid itself. Under /mnt/cache/domains/ I have not noticed any of the mentioned problems.

 

I have performed a RAM check again which was successful. Also a Read-Check. I also checked the S.M.A.R.T. values and everything looks fine there. Do I have to rsync all the data and move it back or is there an easy way to solve the problem?

 

1 hard disk for cache (pool) and one disk 1 (array).

 

And one of my windows disks the partition layout is broken. It is only recognized as RAW. Does anyone know how I can fix this?

[JorgeB]

1 hour ago, Kulisch said:

And one of my windows disks the partition layout is broken. It is only recognized as RAW. Does anyone know how I can fix this?

Try testdisk in Windows.

[Kulisch]

After creating a Windows 11 VM and attaching the corrupted Harddrive it detected, the Problem was found during boot by the OS and repaired it.

 

Now I only have to get the persmissions for my Harddrive. I only can list the files in powershell as a administrator. Explorer says the permissions are missing. Using explorer as admin doesnt change anything.

 

Still, I have to repair the array. Is there something that I can use in Unraid to fix this problem. Setting permissions or moving the files? Or do I have delete everything and building up the array from beginning?

[JorgeB]

What is the array problem? Please post the diagnostics.