All VM related sections freezes unRAID WebUI and GUI


Recommended Posts

Hi,

 

I've created a few VM's under 6.10.0 RC2 and they work fine with VNC, no problems at all but as soon I passthrough GPU the whole system crashes, and all VM-related sections freeze the unRAID server (WebUI and GUI). The only way to restart the system is by a hard reset (unclean).

 

Attached you can find diagnostics, for the first time I was able to pull the diag because previously when it has crashed I have not been able to extract it.

 

Even though the server is frozen, all dockers work as usual. SSH/telnet does not work either during this (from previous freezes but they are disabled because of security reasons).

 

My Specs:

  • CPU - Ryzen 3900X
  • Motherboard - X570 Taichi
  • GPU - RX 480 4GB
  • RAM - DDR4 64 GB (memtest shows no errors)

tower-diagnostics-20220122-1549.zip

Edited by SimpleDino
Link to comment

I have tried isolating as well and it still doesn't work. When isolated I couldn't even extract diagnostics or do anything else and I hate doing unclean shutdown :(.

Can I stop array then try troubleshooting VM's with different settings in case it freezes again? How will unclean affect (hard reset) the server then?

 

One thing that might stand out is that I had/have to fill out the vbios section manually, could it be that the vbios is simply not applied when manually input?

 

 

Link to comment

Okay, I've now managed to get VM's working with GPU passthrough on Fedora and windows 11. I will try MacOS soon also and see what results I get there.

 

Instead of passing through the whole USB controller for Mouse & Keyboard (un-binded) I'm instead passing through them as stand online usb devices. Made a difference.

 

The VM's are working now as they should no problems, no freezing (yet) but the logs show this message (vfio: Cannot reset device, no available reset) Is this normal?

Link to comment

About the freezing, maybe you passed through the usb controller to which your unraid usb stick was plugged in? That's not possible.

45 minutes ago, SimpleDino said:

but the logs show this message (vfio: Cannot reset device, no available reset)

It means that the device (which device??) has no reset mechanism and it cannot be resetted, so you should have issues after a shutdown/restart of the vm.

Link to comment
18 hours ago, ghost82 said:

About the freezing, maybe you passed through the usb controller to which your unraid usb stick was plugged in? That's not possible.

It means that the device (which device??) has no reset mechanism and it cannot be resetted, so you should have issues after a shutdown/restart of the vm.

No, first it's not possible and I try to be thorough!

 

It's this message that comes up on all VM's with GPU & sound part passed through:

2022-01-25T07:24:07.859000Z qemu-system-x86_64: vfio: Cannot reset device 0000:0f:00.1, no available reset mechanism.
2022-01-25T07:24:07.863951Z qemu-system-x86_64: vfio: Cannot reset device 0000:0f:00.1, no available reset mechanism.

 

But I've notice with the amd vendor plugin, when win11 sometimes crashes or does not get past uefi shell script POST install then I can still run the gpu in Fedora VM. So no need for @Squids fantastic script (this time) 🙂.

 

Edit: Fedora also crashes periodically and we are back to shell script!

Edited by SimpleDino
Link to comment
1 hour ago, ghost82 said:

Your amd gpu needs the vendor reset plugin to properly initialize.

So, does it work if you reboot the vm?

 

The only cases I read about issues with the amd patch and the audio part not being able to initialize properly is with some 5000 amd series card.

Yes, true. I do have the Vendor plugin installed. It worked during one reboot. But now I only get to Shell Script (I do type exit and continue) and no further. Something is wrong, maybe the order of BUS:s for the vdisk or something?

 

ErrorWarningSystemArrayLogin


-device pcie-root-port,port=0x8,chassis=6,id=pci.6,bus=pcie.0,multifunction=on,addr=0x1 \
-device pcie-root-port,port=0x9,chassis=7,id=pci.7,bus=pcie.0,addr=0x1.0x1 \
-device pcie-root-port,port=0xa,chassis=8,id=pci.8,bus=pcie.0,addr=0x1.0x2 \
-device pcie-root-port,port=0xe,chassis=9,id=pci.9,bus=pcie.0,addr=0x1.0x6 \
-device pcie-pci-bridge,id=pci.10,bus=pci.1,addr=0x0 \
-device ich9-usb-ehci1,id=usb,bus=pcie.0,addr=0x7.0x7 \
-device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pcie.0,multifunction=on,addr=0x7 \
-device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pcie.0,addr=0x7.0x1 \
-device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pcie.0,addr=0x7.0x2 \
-device virtio-serial-pci,id=virtio-serial0,bus=pci.2,addr=0x0 \
-blockdev '{"driver":"file","filename":"/mnt/user/domains/Windows 11/vdisk1.img","node-name":"libvirt-3-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-3-format","read-only":false,"cache":{"direct":false,"no-flush":false},"driver":"qcow2","file":"libvirt-3-storage","backing":null}' \
-device virtio-blk-pci,bus=pci.4,addr=0x0,drive=libvirt-3-format,id=virtio-disk2,bootindex=1,write-cache=on \
-blockdev '{"driver":"file","filename":"/mnt/user/isos/Windows/Win11_Swedish_x64.iso","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-2-format","read-only":true,"driver":"raw","file":"libvirt-2-storage"}' \
-device ide-cd,bus=ide.0,drive=libvirt-2-format,id=sata0-0-0,bootindex=2 \
-blockdev '{"driver":"file","filename":"/mnt/user/isos/Virtio/virtio-win-0.1.208-1.iso","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":true,"driver":"raw","file":"libvirt-1-storage"}' \
-device ide-cd,bus=ide.1,drive=libvirt-1-format,id=sata0-0-1 \
-netdev tap,fd=37,id=hostnet0 \
-device virtio-net,netdev=hostnet0,id=net0,mac=52:54:00:0f:e5:9e,bus=pci.5,addr=0x0 \
-chardev pty,id=charserial0 \
-device isa-serial,chardev=charserial0,id=serial0 \
-chardev socket,id=charchannel0,fd=38,server=on,wait=off \
-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 \
-tpmdev emulator,id=tpm-tpm0,chardev=chrtpm \
-chardev 'socket,id=chrtpm,path=/run/libvirt/qemu/swtpm/2-Windows 11-swtpm.sock' \
-device tpm-tis,tpmdev=tpm-tpm0,id=tpm0 \
-device usb-tablet,id=input0,bus=usb.0,port=1 \
-audiodev id=audio1,driver=none \
-device 'vfio-pci,host=0000:0f:00.0,id=hostdev0,bus=pci.3,multifunction=on,addr=0x0,romfile=/mnt/user/isos/vbios/rx 480v3.rom' \
-device vfio-pci,host=0000:0f:00.1,id=hostdev1,bus=pci.3,addr=0x0.0x1 \
-device usb-host,hostdevice=/dev/bus/usb/003/008,id=hostdev2,bus=usb.0,port=2 \
-device usb-host,hostdevice=/dev/bus/usb/005/003,id=hostdev3,bus=usb.0,port=3 \
-device usb-host,hostdevice=/dev/bus/usb/003/007,id=hostdev4,bus=usb.0,port=4 \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
char device redirected to /dev/pts/0 (label charserial0)
2022-01-25T09:09:22.231649Z qemu-system-x86_64: vfio: Cannot reset device 0000:0f:00.1, no available reset mechanism.
2022-01-25T09:09:22.235623Z qemu-system-x86_64: vfio: Cannot reset device 0000:0f:00.1, no available reset mechanism.

 

It's RX 480 GB.

Edited by SimpleDino
Link to comment

It does not work which is very odd. Because all the VMs work perfectly with VNC. Probably something wrong with the mappings.

Anyhow I will continue troubleshooting this odd behaviour hopefully I'll get figured out or just reinstall VM's. unRAID is very fragile when it comes to changes in vm (I've probably mentioned this earlier).

Link to comment
  • 1 year later...

Not to re-open a dead thread, but I am experiencing this exact behavior. The difference is I am using a Windows 11 VM and am on a Dell R530 platform, dual xeons and whatnot. Weird thing is everything was working until the other night when I powered off the server and turned it back on. Now whenever I try and start the VM the entire web ui crashes. Then if I SSH into the server and send a reboot command SSH hangs and does nothing. The only way I can recover control is to hard power down and reboot. I am passing through only a P620 GPU to this vm otherwise its all virtual hardware.

Link to comment
  • 5 months later...
On 1/27/2023 at 7:02 AM, squanchy2112 said:

Not to re-open a dead thread, but I am experiencing this exact behavior. The difference is I am using a Windows 11 VM and am on a Dell R530 platform, dual xeons and whatnot. Weird thing is everything was working until the other night when I powered off the server and turned it back on. Now whenever I try and start the VM the entire web ui crashes. Then if I SSH into the server and send a reboot command SSH hangs and does nothing. The only way I can recover control is to hard power down and reboot. I am passing through only a P620 GPU to this vm otherwise its all virtual hardware.

Hi @squanchy2112 sorry to re re-open a dead thread, but I have just experienced exactly what you describe, albeit on less advanced hardware. Did you ever manage to get a fix for this? 

Regards

Aleks

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.