VM Suddenly Won't Boot with Video Passthrough, Has Worked for Months

[harshl]

I recently implemented a VM backup script from JTok referenced here:

 

Three VMs were targeted for backup. Two with VNC graphics and one with a Vega 56 card being passed through. This VM with the Vega 56 has been running great for me since November of 2018. Once this script ran, it did not succeed on the gaming VM with the Vega card passed through. The logs/notifications seem to indicate it didn't shutdown in time, but I'm not 100% sure.

Upon going to the server to check on it, the video card was at maximum fan speed and the gaming VM was down.

I powered the server down to reset the Video and USB cards that are passed through and let it boot back up. The VM attempts to start and does not produce any errors when manually started, but it never produces any video and does not actually boot either.

I removed the video card and put a VNC adapter in the config and it boots right up with the USB card still passed through. So the video card is producing some kind of issue, but I don't know what. I don't see any obvious errors anywhere. The VM never does actually start with the video card passed through now, the 4 vCPUs that are assigned are all flatlined at 0 with the exception of the first core, which has an occasional 1% and I never see it appear on the network.

I have tried destroying the VM and recreating it and I get the exact same behavior.

Anonymized diagnostics are attached.

Here are the VM logs for the system with the issue:

-device pcie-root-port,port=0x12,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2 \
-device pcie-root-port,port=0x13,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3 \
-device pcie-root-port,port=0x14,chassis=5,id=pci.5,bus=pcie.0,addr=0x2.0x4 \
-device pcie-root-port,port=0x15,chassis=6,id=pci.6,bus=pcie.0,addr=0x2.0x5 \
-device pcie-root-port,port=0xa,chassis=7,id=pci.7,bus=pcie.0,addr=0x1.0x2 \
-device pcie-root-port,port=0xb,chassis=8,id=pci.8,bus=pcie.0,addr=0x1.0x3 \
-device pcie-pci-bridge,id=pci.9,bus=pci.1,addr=0x0 \
-device ich9-usb-ehci1,id=usb,bus=pcie.0,addr=0x7.0x7 \
-device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pcie.0,multifunction=on,addr=0x7 \
-device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pcie.0,addr=0x7.0x1 \
-device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pcie.0,addr=0x7.0x2 \
-device virtio-serial-pci,id=virtio-serial0,bus=pci.2,addr=0x0 \
-drive file=/mnt/user/domains/cubegaming/vdisk1.img,format=raw,if=none,id=drive-virtio-disk2,cache=none \
-device virtio-blk-pci,scsi=off,bus=pci.4,addr=0x0,drive=drive-virtio-disk2,id=virtio-disk2,bootindex=1,write-cache=on \
-drive file=/mnt/user/domains/cubegaming/vdisk2.img,format=raw,if=none,id=drive-virtio-disk3,cache=none \
-device virtio-blk-pci,scsi=off,bus=pci.5,addr=0x0,drive=drive-virtio-disk3,id=virtio-disk3,write-cache=on \
-drive file=/mnt/user/isos/en_windows_10_business_editions_version_1903_updated_july_2019_x64_dvd_97049592.iso,format=raw,if=none,id=drive-sata0-0-0,readonly=on \
-device ide-cd,bus=ide.0,drive=drive-sata0-0-0,id=sata0-0-0,bootindex=2 \
-drive file=/mnt/user/isos/virtio-win-0.1.160-1.iso,format=raw,if=none,id=drive-sata0-0-1,readonly=on \
-device ide-cd,bus=ide.1,drive=drive-sata0-0-1,id=sata0-0-1 \
-netdev tap,fd=32,id=hostnet0,vhost=on,vhostfd=33 \
-device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:b2:9f:1a,bus=pci.3,addr=0x0 \
-chardev pty,id=charserial0 \
-device isa-serial,chardev=charserial0,id=serial0 \
-chardev socket,id=charchannel0,fd=35,server,nowait \
-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 \
-device usb-tablet,id=input0,bus=usb.0,port=1 \
-vnc 0.0.0.0:3,websocket=5703 \
-k en-us \
-device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pci.9,addr=0x1 \
-device vfio-pci,host=03:00.0,id=hostdev0,bus=pci.6,addr=0x0 \
-device vfio-pci,host=03:00.1,id=hostdev1,bus=pci.7,addr=0x0 \
-device vfio-pci,host=0b:00.0,id=hostdev2,bus=pci.8,addr=0x0 \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
2019-09-07 18:22:37.573+0000: Domain id=6 is tainted: high-privileges
2019-09-07 18:22:37.573+0000: Domain id=6 is tainted: host-cpu
char device redirected to /dev/pts/3 (label charserial0)
2019-09-07T18:34:36.168070Z qemu-system-x86_64: terminating on signal 15 from pid 7305 (/usr/sbin/libvirtd)
2019-09-07 18:34:37.368+0000: shutting down, reason=destroyed

Any thoughts on what might be going on here? The video card is fine. I pulled it from my server and dropped it into another system and it worked fine.

 

I am creating a Linux VM and will try passing this through shortly as well and will report back what behavior I see with that when I am done testing.

 

Any guidance is appreciated!

-Landon

diagnostics-20190907-1840.zip

[harshl]

Well, this was unexpected, but the Linux VM booted right up and produced video. It produced a bunch of garbage on the screen and hung during the boot, but I suspect that is a different issue entirely.

 

So... what persists even through a VM deletion and recreation besides the disk that might be hanging this up?

 

I'm stumped!

-Landon

[harshl]

Well, I'm updating this post from Linux Mint with the video card passed through and everything working as it should in my Windows VM, which will no longer boot. I don't even know what to look at. Crazy that it will work fine in one VM and not in another.

 

If anyone has any wise ideas, let me know.

 

Thanks!