kennelm Posted April 22, 2018 Share Posted April 22, 2018 (edited) All, I run a CentOS physical server that I am trying to migrate to my unRAID server as a VM to simply things. I got the VM up and running just fine using VNC, but not when I use a graphics card passthrough. Because this is a media server that needs graphics card passthrough, I'll need to get that part working. I've encountered a weird problem where, if I do a full restart the entire UnRAID box, I can successfully launch the VM and the CentOS gnome desktop appears on my TV attached over HDMI to the nvidia graphics card. Media playback/decoding is good and everything seems to work. But, once the VM is already up, if I use the console to stop and restart the VM, I can't get the desktop to launch over the HDMI connection. Instead, I get a progress bar with the message "CentOS Linux 7[ OK ] Starting Virtualization daemon". The desktop never appears. Yet, If I force stop the VM, switch to VNC, the VM launches the desktop just fine. And, if I reboot the unRAID box completely, the VM launches the desktop over HDMI just fine. So, it seems it all works the first time, but not any subsequent VM restart. Here are my particulars: unRAID Plus version 6.4.0 Model: N/A M/B: ASRock - Z370 Pro4 CPU: Intel® Core™ i7-8700K CPU @ 3.70GHz HVM: Enabled IOMMU: Enabled Cache: 384 kB, 1536 kB, 12288 kB Memory: 16 GB (max. installable capacity 64 GB) Network: eth0: 1000 Mb/s, full duplex, mtu 1500 Kernel: Linux 4.14.13-unRAID x86_64 OpenSSL: 1.0.2n Nvidia GeForce 210 graphics card Here is the VM XML: <domain type='kvm'> <name>CentOS</name> <uuid>a23726d2-fd1e-bd96-b30c-65e4105ea468</uuid> <metadata> <vmtemplate xmlns="unraid" name="CentOS" icon="centos.png" os="centos"/> </metadata> <memory unit='KiB'>4194304</memory> <currentMemory unit='KiB'>4194304</currentMemory> <memoryBacking> <nosharepages/> </memoryBacking> <vcpu placement='static'>1</vcpu> <cputune> <vcpupin vcpu='0' cpuset='10'/> </cputune> <os> <type arch='x86_64' machine='pc-q35-2.10'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader> <nvram>/etc/libvirt/qemu/nvram/a23726d2-fd1e-bd96-b30c-65e4105ea468_VARS-pure-efi.fd</nvram> </os> <features> <acpi/> <apic/> </features> <cpu mode='host-passthrough' check='none'> <topology sockets='1' cores='1' threads='1'/> </cpu> <clock offset='utc'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/local/sbin/qemu</emulator> <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='writeback'/> <source file='/mnt/user/domains/CentOS/vdisk1.img'/> <target dev='hdc' bus='virtio'/> <boot order='1'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/user/larry/CentOS/CentOS-7-x86_64-DVD-1511 (1).iso'/> <target dev='hda' bus='sata'/> <readonly/> <boot order='2'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/> </controller> <controller type='sata' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <controller type='pci' index='0' model='pcie-root'/> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </controller> <controller type='pci' index='1' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='1' port='0x8'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='2' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='2' port='0x9'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='pci' index='3' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='3' port='0xa'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='4' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='4' port='0xb'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/> </controller> <controller type='pci' index='5' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='5' port='0xc'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x4'/> </controller> <controller type='pci' index='6' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='6' port='0xd'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x5'/> </controller> <controller type='pci' index='7' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='7' port='0xe'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x6'/> </controller> <controller type='pci' index='8' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='8' port='0xf'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x7'/> </controller> <filesystem type='mount' accessmode='passthrough'> <source dir='/mnt'/> <target dir='shares'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </filesystem> <interface type='bridge'> <mac address='52:54:00:e1:a1:4c'/> <source bridge='br0'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </interface> <serial type='pty'> <target port='0'/> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='unix'> <target type='virtio' name='org.qemu.guest_agent.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x04' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x0461'/> <product id='0x4d15'/> </source> <address type='usb' bus='0' port='1'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x046d'/> <product id='0xc31c'/> </source> <address type='usb' bus='0' port='2'/> </hostdev> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/> </memballoon> </devices> </domain> Anything here look wrong? Or might explain the weird issue when I restart the VM? Any insight would be appreciated. If I can't get past this problem, then there's no point in virtualizing this server. Edited April 22, 2018 by kennelm typo Quote Link to comment
1812 Posted April 25, 2018 Share Posted April 25, 2018 Could be the older card not wanting to reset completely after usage. Have you tried stubbing it yet? Quote Link to comment
kennelm Posted April 25, 2018 Author Share Posted April 25, 2018 21 hours ago, 1812 said: Could be the older card not wanting to reset completely after usage. Have you tried stubbing it yet? What is stubbing? I did notice in the unraid log that there were messages related to resetting low-speed USB devices for the mouse and keyboard. That only happened on the second launch of the VM but not on the 1st. Quote Link to comment
1812 Posted April 25, 2018 Share Posted April 25, 2018 1 minute ago, kennelm said: What is stubbing? I did notice in the unraid log that there were messages related to resetting low-speed USB devices for the mouse and keyboard. That only happened on the second launch of the VM but not on the 1st. Do me a favor. Try to start the vm the second time and after it fails, then post your full diagnostics zip (tools>diagnostics) Quote Link to comment
kennelm Posted April 25, 2018 Author Share Posted April 25, 2018 (edited) 1 hour ago, 1812 said: Do me a favor. Try to start the vm the second time and after it fails, then post your full diagnostics zip (tools>diagnostics) Thanks for offering to help. See attached. Additional info: root@Tower:~# lspci 00:00.0 Host bridge: Intel Corporation Device 3ec2 (rev 07) 00:14.0 USB controller: Intel Corporation 200 Series PCH USB 3.0 xHCI Controller 00:14.2 Signal processing controller: Intel Corporation 200 Series PCH Thermal Subsystem 00:16.0 Communication controller: Intel Corporation 200 Series PCH CSME HECI #1 00:17.0 SATA controller: Intel Corporation 200 Series PCH SATA controller [AHCI mode] 00:1b.0 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #17 (rev f0) 00:1b.2 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #19 (rev f0) 00:1c.0 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #1 (rev f0) 00:1c.4 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #5 (rev f0) 00:1d.0 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #9 (rev f0) 00:1f.0 ISA bridge: Intel Corporation Device a2c9 00:1f.2 Memory controller: Intel Corporation 200 Series PCH PMC 00:1f.3 Audio device: Intel Corporation 200 Series PCH HD Audio 00:1f.4 SMBus: Intel Corporation 200 Series PCH SMBus Controller 00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (2) I219-V 02:00.0 SATA controller: Marvell Technology Group Ltd. Device 9215 (rev 11) 04:00.0 VGA compatible controller: NVIDIA Corporation GT218 [GeForce 210] (rev a2) 04:00.1 Audio device: NVIDIA Corporation High Definition Audio Controller (rev a1) lspci -v (Video card only) 04:00.0 VGA compatible controller: NVIDIA Corporation GT218 [GeForce 210] (rev a2) (prog-if 00 [VGA controller]) Subsystem: eVga.com. Corp. GT218 [GeForce 210] Flags: bus master, fast devsel, latency 0, IRQ 125 Memory at de000000 (32-bit, non-prefetchable) Memory at c0000000 (64-bit, prefetchable) Memory at d0000000 (64-bit, prefetchable) I/O ports at d000 Expansion ROM at 000c0000 [disabled] Capabilities: [60] Power Management version 3 Capabilities: [68] MSI: Enable+ Count=1/1 Maskable- 64bit+ Capabilities: [78] Express Endpoint, MSI 00 Capabilities: [b4] Vendor Specific Information: Len=14 <?> Capabilities: [100] Virtual Channel Capabilities: [128] Power Budgeting <?> Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?> Kernel driver in use: vfio-pci 04:00.1 Audio device: NVIDIA Corporation High Definition Audio Controller (rev a1) Subsystem: eVga.com. Corp. High Definition Audio Controller Flags: bus master, fast devsel, latency 0, IRQ 17 Memory at df080000 (32-bit, non-prefetchable) Capabilities: [60] Power Management version 3 Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+ Capabilities: [78] Express Endpoint, MSI 00 Kernel driver in use: vfio-pci root@Tower:~# lsusb Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub Bus 001 Device 004: ID 051d:0002 American Power Conversion Uninterruptible Power Supply Bus 001 Device 005: ID 046d:c31c Logitech, Inc. Keyboard K120 Bus 001 Device 003: ID 0461:4d15 Primax Electronics, Ltd Dell Optical Mouse Bus 001 Device 002: ID 0781:5406 SanDisk Corp. Cruzer Micro U3 Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub tower-diagnostics-20180425-1933.zip Edited April 26, 2018 by kennelm added info Quote Link to comment
1812 Posted April 26, 2018 Share Posted April 26, 2018 (edited) 1 hour ago, kennelm said: What is stubbing? I did notice in the unraid log that there were messages related to resetting low-speed USB devices for the mouse and keyboard. That only happened on the second launch of the VM but not on the 1st. you can try ruling this out as a source problem by removing the usb for second launch This is ruled out by being able to start the vm using vnc. This thread references problems passing through at 210 as the only card in the system: https://lime-technology.com/forums/topic/47271-evga-nvidia-gt-210-windows-10-1-pcie-slot/ I do not know why enabling "unsafe interrupts" helped with the issue as I don't have this board, but I run that command in my syslinux.cfg on all my proliant servers without issue. Some people have had to dump the bios on the GPU on single card nvidia systems to get them to work, others have found that moving the card to the primary slot also helps. the following 2 vids may also be relevant in regards to that: https://www.youtube.com/watch?v=mM7ntkiUoPk https://www.youtube.com/watch?v=1IP-h9IKof0 if you have another gpu lying around and can toss it in and assign that as primary in the motherboard bios, and then it fixes the problem, then the video fix listed above will probably work if you want to return back to a single card system. If you can, I would start here first. Edited April 26, 2018 by 1812 Quote Link to comment
kennelm Posted April 26, 2018 Author Share Posted April 26, 2018 38 minutes ago, 1812 said: you can try ruling this out as a source problem by removing the usb for second launch This is ruled out by being able to start the vm using vnc. This thread references problems passing through at 210 as the only card in the system: https://lime-technology.com/forums/topic/47271-evga-nvidia-gt-210-windows-10-1-pcie-slot/ I do not know why enabling "unsafe interrupts" helped with the issue as I don't have this board, but I run that command in my syslinux.cfg on all my proliant servers without issue. Some people have had to dump the bios on the GPU on single card nvidia systems to get them to work, others have found that moving the card to the primary slot also helps. the following 2 vids may also be relevant in regards to that: https://www.youtube.com/watch?v=mM7ntkiUoPk https://www.youtube.com/watch?v=1IP-h9IKof0 if you have another gpu lying around and can toss it in and assign that as primary in the motherboard bios, and then it fixes the problem, then the video fix listed above will probably work if you want to return back to a single card system. Thanks. I'll dig into it. Meanwhile, my Asrock Z370 Pro4 motherboard has a robust onboard video capability. Can this serve as one of the GPUs and the nvidia as the other? Or do I need 2 PCIe GPUs? I assume the primary GPU is allocated to the unraid system and the guest machine gets the other GPU? Or do I have that backwards? Quote Link to comment
1812 Posted April 26, 2018 Share Posted April 26, 2018 7 minutes ago, kennelm said: Thanks. I'll dig into it. Meanwhile, my Asrock Z370 Pro4 motherboard has a robust onboard video capability. Can this serve as one of the GPUs and the nvidia as the other? Or do I need 2 PCIe GPUs? I assume the primary GPU is allocated to the unraid system and the guest machine gets the other GPU? Or do I have that backwards? Onboard can/should be assigned to unRaid. It didn’t show in the devices inventory (unless I missed it) Is it disabled in the bios? If so, enable it and set to primary, then reboot and retest. Quote Link to comment
kennelm Posted April 26, 2018 Author Share Posted April 26, 2018 20 hours ago, 1812 said: Onboard can/should be assigned to unRaid. It didn’t show in the devices inventory (unless I missed it) Is it disabled in the bios? If so, enable it and set to primary, then reboot and retest. Good catch. Somehow this got disabled in the BIOS. I've re-enabled the on-board GPU, but I'm not sure how to designate as "primary." Point being, the second launch of the VM still fails. Interestingly, before I launched the VM the first time, I noticed 3 choices in the VM graphics card drop-down box: VNC, Nvidia, and Intel. After the second launch failed, I checked again and the Intel on-board GPU was not in the list anymore.... Quote Link to comment
1812 Posted April 26, 2018 Share Posted April 26, 2018 1 hour ago, kennelm said: Good catch. Somehow this got disabled in the BIOS. I've re-enabled the on-board GPU, but I'm not sure how to designate as "primary." Point being, the second launch of the VM still fails. Interestingly, before I launched the VM the first time, I noticed 3 choices in the VM graphics card drop-down box: VNC, Nvidia, and Intel. After the second launch failed, I checked again and the Intel on-board GPU was not in the list anymore.... In the bios, the could be a section for video options or something similar and you'd select it there. You might just have to dig around to set which is primary or first use. When it boots, is the unraid loading screen outputting via the onboard now or nvida card? Quote Link to comment
kennelm Posted April 27, 2018 Author Share Posted April 27, 2018 51 minutes ago, 1812 said: In the bios, the could be a section for video options or something similar and you'd select it there. You might just have to dig around to set which is primary or first use. When it boots, is the unraid loading screen outputting via the onboard now or nvida card? Doh. I'm an idiot. I totally overlooked a BIOS selection to designate a primary video card. Now, the unraid screen appears on the onboard card, and the Vm display is on the NVIDIA card. Again, the VM works the first time around, but on the second try, I now get a ROM error: 2018-04-27T00:16:09.953404Z qemu-system-x86_64: vfio-pci: Cannot read device rom at 0000:04:00.0Device option ROM contents are probably invalid (check dmesg).Skip option ROM probe with rombar=0, or load from file with romfile= I'm going to guess that I need to dump the ROM per the video above. Quote Link to comment
1812 Posted April 27, 2018 Share Posted April 27, 2018 28 minutes ago, kennelm said: Doh. I'm an idiot. I totally overlooked a BIOS selection to designate a primary video card. Now, the unraid screen appears on the onboard card, and the Vm display is on the NVIDIA card. Again, the VM works the first time around, but on the second try, I now get a ROM error: 2018-04-27T00:16:09.953404Z qemu-system-x86_64: vfio-pci: Cannot read device rom at 0000:04:00.0Device option ROM contents are probably invalid (check dmesg).Skip option ROM probe with rombar=0, or load from file with romfile= I'm going to guess that I need to dump the ROM per the video above. give it a try or get the tech powerup one and modify the header. Quote Link to comment
kennelm Posted April 28, 2018 Author Share Posted April 28, 2018 (edited) On 4/26/2018 at 8:47 PM, 1812 said: give it a try or get the tech powerup one and modify the header. No joy in Mudville. Based on what I'm reading, this older G 210 card probably won't pass through correctly, so I just ordered the GT 710 per your post in this thread: Edited April 28, 2018 by kennelm Quote Link to comment
1812 Posted April 28, 2018 Share Posted April 28, 2018 9 hours ago, kennelm said: No joy in Mudville. Based on what I'm reading, this older G 210 card probably won't pass through correctly, so I just ordered the GT 710 per your post in this thread: Let us know if this fixes it! The 710's are great basic cards. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.