alkenerly

@JorgeB I do agree that is a very sensible troubleshooting step. The main reason I have not attempted it fully is because I would need to let the server run for at least 3-4 weeks until I can confidently say its "stable" in this configuration. Reenabling containers and VMs could take several months at that rate and I struggle to not have a fully functioning NAS for several months. My main hang-up is that I was under the impression that VMs and containers both serve to segment the running processes from each other and the greater system. e.g. a docker or VM crashing shouldn't cause unraid itself to crash. I suppose that may be my only route. Or transfer to totally new server hardware to rule out hardware.

Hey all, I just wanted to post an update on this topic. I have since upgraded to 7.0.1 but still experience the same issue. Each time the server runs correctly for 1-2 weeks and then all VMs, Docker Containers, webui, ssh suddenly crash at once. I can log in via the attached monitor and keyboard but no command functions properly. Even executing `shutdown` creates a lot of bus errors, seg faults, and never actually shuts down. Only option is a hard reset. Upon reboot, the server does a parity check and finds/fixes 5 errors (its always exactly 5) and runs as expected for another week or two. I disabled XMP in the bios, ran memtests, updated the BIOS, etc. but still had another crash today. I believe I have exhausted all of my troubleshooting ability. It definitely stinks that my unraid server is so unstable. It used to be rock solid! Does this seem like a hardware failure to anyone? Memory issue?

Ever since I made this forum post, I have been trying to figure out what has caused the occasional crash. It does not happen more than once per week, so it is not the kind of thing that can be quickly tested. I think I may have found a correlation with the crashes. I use vscode for remote development and will frequently attach my vscode window to my unraid server host. Doing that installs the vscode server on unraid itself. It works great! From there, I will sometimes remotely attach to a container. Either a development container that I have defined with a Dockerfile or another one of my running containers like nextcloud to edit configs, etc. @JorgeB, do you see a way that running vscode on the host and/or inside a container could cause this issue? Any way I could mitigate it?

Thanks for looking into it for me. This has happened twice so far and there was roughly one week of stability after the first instance before the second occurred. So in all likelihood it will take some time to narrow down a misbehaving container. After the most recent reboot, my snapcast failed to function correctly. The logs for that container are pretty messy, but it complains of a bad file descriptor 2025-01-23 08-56-21.267 [Error] (AirplayStream) Error opening metadata pipe, retrying in 500ms. Error: assign: Bad file descriptor [system:9 at /usr/include/boost/asio/detail/impl/reactive_descriptor_service.ipp:120 in function 'assign'] I'll keep snapcast turned off and see if that resolves the issue. I am little confused about how a misbehaving container could crash the whole system.

I have encountered a strange issue about once per week for the last two weeks. Seemingly randomly, all the services on my instance crash. Dockers, VMs, WebGUI, ssh, etc. My only interface to the server when this occurs is monitor/mouse/keyboard. I am able to perform some bash commands but not others. For example, I can tail the syslog. The syslog does not seem to show anything interesting. Last logs refer to an SSD trim command running. I suspect that processes are largely failing to write to the syslog when I am in this error state. Others commands give me a bash input/output error. For example root@WarrentonUnraid:/mnt/user# /etc/rc.d/rc.nginx status -bash: /etc/rc.d/rc.nginx: Input/output error I was also unable to perform: powerdown -r because of the same IO error. Doing a hard reboot of the server causes everything to come back fine. Of course it initiates a parity check as expected, but all the VMs/Dockers/etc. run fine. I do not know how to induce this error state again. Does anyone have any advice on things to check or look into to prevent this issue from coming back? warrentonunraid-diagnostics-20250123-1109.zip

Jorge, thanks for the reply. Sadly booting in safe mode yields the same result. Booting typically but with VMs and Dockers disabled also yields the same result. Do you have any intuition on what kind of hardware issue it could be? I suspected the power supply, but it clearly wasn't that. Is this what bad RAM crashes look like? Thanks for the help

Diagnostics file attached. Around 3 months ago, this server began to crash after being booted for about one hour. After a lot of rebooting and troubleshooting I was finally able to get it stable again by enabling "VFIO allow unsafe interrupts" on the VM Manger page. In the last few days, the symptoms have come back. Immediately before crashing it does not put anything into the syslog or any other log file indicating a crash. I kept looking for a stack trace or something, but could not find one. It is just as if someone pulled the power cord out of the server. This unraid server has been around for 6-7 years and has been mostly great. It suffered from that old Ryzen sleep state bug a few years ago. Around a year ago I added two GPUs into this system to passthrough to two windows VMs. The server worked great for several months following that change. I replaced the power supply in the server when I originally had this crashing problem a month or two ago. If anyone has any guesses or insight to what could be happening, please feel free to weigh in. I'm officially stumped. jesupunraid-diagnostics-20240905-1001.zip

I have a windows 11 VM running on Unraid 6.12.1 with my AMD rx 460 passed through. Everything almost works, but video decoding does not. I drop 30% of frames on any youtube video. VLC crashes if I try to play anything. The AMD GPU software in the tray also crashes if I try to open it. Since I use this PC as an HTPC, I really need the bells and whistles of decoding, HDR, etc. working. I have followed several forum posts, all the space invaders videos, etc. I have cycled through several different problems (black screen, no boot, Code 43, Code 10). The problem I currently have is what I'm stuck on. I think its been harder for me to google for since there is no explicit error code for it. Its so close to working! I have tried uninstalling and reinstalling several different versions of the AMD driver from its website. I tried booting the same disk in an equivalent VM that uses i440 instead of q35. I tried passing the GPU bios (that I got from GPUZ dump) and not. I have tried editing the GRUB command line a few different ways. Its hard to know what information will be needed for anyone here to help me debug, so I'll just throw in what I think could be relevant. ASUSTeK COMPUTER INC. TUF B450-PLUS GAMING (BIOS from 3/13/23) Ryzen 5 3600 Gigabyte AMD RX 460 2GB (only GPU, in slot 1 on mobo) I do not think there are meaning logs from the VM. The syslog shows these lines whenever the VM boots Jun 30 07:26:31 Tower kernel: vfio-pci 0000:09:00.0: vfio_ecap_init: hiding ecap 0x19@0x270 Jun 30 07:26:31 Tower kernel: vfio-pci 0000:09:00.0: vfio_ecap_init: hiding ecap 0x1b@0x2d0 Jun 30 07:26:31 Tower kernel: vfio-pci 0000:09:00.0: vfio_ecap_init: hiding ecap 0x1e@0x370 GRUB command line: append pcie_acs_override=downstream,multifunction video=efifb:off initrd=/bzroot IOMMU Groups: IOMMU group 24: [1912:0014] 05:00.0 USB controller: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller (rev 03) This controller is bound to vfio, connected USB devices are not visible. IOMMU group 25: [1002:67ef] 09:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Baffin [Radeon RX 460/560D / Pro 450/455/460/555/555X/560/560X] (rev cf) IOMMU group 26: [1002:aae0] 09:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Baffin HDMI/DP Audio [Radeon RX 550 640SP / RX 560/560X] <?xml version='1.0' encoding='UTF-8'?> <domain type='kvm' id='4'> <name>Windows 11 - 3</name> <uuid>b0f0dfb2-d8ea-a254-943a-a7139086f125</uuid> <metadata> <vmtemplate xmlns="unraid" name="Windows 11" icon="windows11.png" os="windowstpm"/> </metadata> <memory unit='KiB'>8388608</memory> <currentMemory unit='KiB'>8388608</currentMemory> <memoryBacking> <nosharepages/> </memoryBacking> <vcpu placement='static'>6</vcpu> <cputune> <vcpupin vcpu='0' cpuset='1'/> <vcpupin vcpu='1' cpuset='7'/> <vcpupin vcpu='2' cpuset='3'/> <vcpupin vcpu='3' cpuset='9'/> <vcpupin vcpu='4' cpuset='5'/> <vcpupin vcpu='5' cpuset='11'/> </cputune> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-q35-7.1'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi-tpm.fd</loader> <nvram>/etc/libvirt/qemu/nvram/b0f0dfb2-d8ea-a254-943a-a7139086f125_VARS-pure-efi-tpm.fd</nvram> </os> <features> <acpi/> <apic/> <hyperv mode='custom'> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='8191'/> <vendor_id state='on' value='none'/> </hyperv> </features> <cpu mode='host-passthrough' check='none' migratable='on'> <topology sockets='1' dies='1' cores='3' threads='2'/> <cache mode='passthrough'/> <feature policy='require' name='topoext'/> </cpu> <clock offset='localtime'> <timer name='hypervclock' present='yes'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/local/sbin/qemu</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='writeback'/> <source file='/mnt/user/domains/Windows 11 - 2/vdisk1.img' index='2'/> <backingStore/> <target dev='hdc' bus='virtio'/> <serial>vdisk1</serial> <boot order='1'/> <alias name='virtio-disk2'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/user/isos/virtio-win-0.1.229-1.iso' index='1'/> <backingStore/> <target dev='hdb' bus='sata'/> <readonly/> <alias name='sata0-0-1'/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <controller type='pci' index='0' model='pcie-root'> <alias name='pcie.0'/> </controller> <controller type='pci' index='1' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='1' port='0x10'/> <alias name='pci.1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='2' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='2' port='0x11'/> <alias name='pci.2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/> </controller> <controller type='pci' index='3' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='3' port='0x12'/> <alias name='pci.3'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/> </controller> <controller type='pci' index='4' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='4' port='0x13'/> <alias name='pci.4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/> </controller> <controller type='pci' index='5' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='5' port='0x14'/> <alias name='pci.5'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/> </controller> <controller type='pci' index='6' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='6' port='0x15'/> <alias name='pci.6'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/> </controller> <controller type='virtio-serial' index='0'> <alias name='virtio-serial0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </controller> <controller type='sata' index='0'> <alias name='ide'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <controller type='usb' index='0' model='ich9-ehci1'> <alias name='usb'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <alias name='usb'/> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <alias name='usb'/> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <alias name='usb'/> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/> </controller> <interface type='bridge'> <mac address='52:54:00:a4:4b:c4'/> <source bridge='br0'/> <target dev='vnet3'/> <model type='virtio-net'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </interface> <serial type='pty'> <source path='/dev/pts/1'/> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> <alias name='serial0'/> </serial> <console type='pty' tty='/dev/pts/1'> <source path='/dev/pts/1'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <channel type='unix'> <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-4-Windows 11 - 3/org.qemu.guest_agent.0'/> <target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='tablet' bus='usb'> <alias name='input0'/> <address type='usb' bus='0' port='1'/> </input> <input type='mouse' bus='ps2'> <alias name='input1'/> </input> <input type='keyboard' bus='ps2'> <alias name='input2'/> </input> <tpm model='tpm-tis'> <backend type='emulator' version='2.0' persistent_state='yes'/> <alias name='tpm0'/> </tpm> <audio id='1' type='none'/> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x09' slot='0x00' function='0x0'/> </source> <alias name='hostdev0'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0' multifunction='on'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x09' slot='0x00' function='0x1'/> </source> <alias name='hostdev1'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x1'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </source> <alias name='hostdev2'/> <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/> </hostdev> <memballoon model='none'/> </devices> <seclabel type='dynamic' model='dac' relabel='yes'> <label>+0:+100</label> <imagelabel>+0:+100</imagelabel> </seclabel> </domain> tower-diagnostics-20230630-1118.zip