CentOS VM Problem

kennelm · April 22, 2018

All,

I run a CentOS physical server that I am trying to migrate to my unRAID server as a VM to simply things. I got the VM up and running just fine using VNC, but not when I use a graphics card passthrough. Because this is a media server that needs graphics card passthrough, I'll need to get that part working. I've encountered a weird problem where, if I do a full restart the entire UnRAID box, I can successfully launch the VM and the CentOS gnome desktop appears on my TV attached over HDMI to the nvidia graphics card. Media playback/decoding is good and everything seems to work. But, once the VM is already up, if I use the console to stop and restart the VM, I can't get the desktop to launch over the HDMI connection. Instead, I get a progress bar with the message "CentOS Linux 7[ OK ] Starting Virtualization daemon". The desktop never appears. Yet, If I force stop the VM, switch to VNC, the VM launches the desktop just fine. And, if I reboot the unRAID box completely, the VM launches the desktop over HDMI just fine. So, it seems it all works the first time, but not any subsequent VM restart.

Here are my particulars:

unRAID Plus version 6.4.0

Model: N/A

M/B: ASRock - Z370 Pro4

CPU: Intel® Core™ i7-8700K CPU @ 3.70GHz

HVM: Enabled

IOMMU: Enabled

Cache: 384 kB, 1536 kB, 12288 kB

Memory: 16 GB (max. installable capacity 64 GB)

Network: eth0: 1000 Mb/s, full duplex, mtu 1500

Kernel: Linux 4.14.13-unRAID x86_64

OpenSSL: 1.0.2n

Nvidia GeForce 210 graphics card

Here is the VM XML:

<domain type='kvm'>
  <name>CentOS</name>
  <uuid>a23726d2-fd1e-bd96-b30c-65e4105ea468</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="CentOS" icon="centos.png" os="centos"/>
  </metadata>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>1</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='10'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-q35-2.10'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/a23726d2-fd1e-bd96-b30c-65e4105ea468_VARS-pure-efi.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-passthrough' check='none'>
    <topology sockets='1' cores='1' threads='1'/>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/user/domains/CentOS/vdisk1.img'/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/larry/CentOS/CentOS-7-x86_64-DVD-1511 (1).iso'/>
      <target dev='hda' bus='sata'/>
      <readonly/>
      <boot order='2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </controller>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x8'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x9'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0xa'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0xb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0xc'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x4'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0xd'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x5'/>
    </controller>
    <controller type='pci' index='7' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='7' port='0xe'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x6'/>
    </controller>
    <controller type='pci' index='8' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='8' port='0xf'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x7'/>
    </controller>
    <filesystem type='mount' accessmode='passthrough'>
      <source dir='/mnt'/>
      <target dir='shares'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </filesystem>
    <interface type='bridge'>
      <mac address='52:54:00:e1:a1:4c'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x04' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x0461'/>
        <product id='0x4d15'/>
      </source>
      <address type='usb' bus='0' port='1'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x046d'/>
        <product id='0xc31c'/>
      </source>
      <address type='usb' bus='0' port='2'/>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
    </memballoon>
  </devices>
</domain>

Anything here look wrong? Or might explain the weird issue when I restart the VM? Any insight would be appreciated. If I can't get past this problem, then there's no point in virtualizing this server.

Edited April 22, 2018 by kennelm
typo

kennelm · April 25, 2018

Anyone?

2.6k · April 25, 2018

Could be the older card not wanting to reset completely after usage. Have you tried stubbing it yet?

kennelm · April 25, 2018

21 hours ago, 1812 said:

Could be the older card not wanting to reset completely after usage. Have you tried stubbing it yet?

What is stubbing?

I did notice in the unraid log that there were messages related to resetting low-speed USB devices for the mouse and keyboard. That only happened on the second launch of the VM but not on the 1st.

2.6k · April 25, 2018

1 minute ago, kennelm said:

What is stubbing?

I did notice in the unraid log that there were messages related to resetting low-speed USB devices for the mouse and keyboard. That only happened on the second launch of the VM but not on the 1st.

Do me a favor. Try to start the vm the second time and after it fails, then post your full diagnostics zip (tools>diagnostics)

kennelm · April 25, 2018

1 hour ago, 1812 said:

Do me a favor. Try to start the vm the second time and after it fails, then post your full diagnostics zip (tools>diagnostics)

Thanks for offering to help. See attached.

Additional info:

root@Tower:~# lspci
00:00.0 Host bridge: Intel Corporation Device 3ec2 (rev 07)
00:14.0 USB controller: Intel Corporation 200 Series PCH USB 3.0 xHCI Controller
00:14.2 Signal processing controller: Intel Corporation 200 Series PCH Thermal Subsystem
00:16.0 Communication controller: Intel Corporation 200 Series PCH CSME HECI #1
00:17.0 SATA controller: Intel Corporation 200 Series PCH SATA controller [AHCI mode]
00:1b.0 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #17 (rev f0)
00:1b.2 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #19 (rev f0)
00:1c.0 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #1 (rev f0)
00:1c.4 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #5 (rev f0)
00:1d.0 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #9 (rev f0)
00:1f.0 ISA bridge: Intel Corporation Device a2c9
00:1f.2 Memory controller: Intel Corporation 200 Series PCH PMC
00:1f.3 Audio device: Intel Corporation 200 Series PCH HD Audio
00:1f.4 SMBus: Intel Corporation 200 Series PCH SMBus Controller
00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (2) I219-V
02:00.0 SATA controller: Marvell Technology Group Ltd. Device 9215 (rev 11)
04:00.0 VGA compatible controller: NVIDIA Corporation GT218 [GeForce 210] (rev a2)
04:00.1 Audio device: NVIDIA Corporation High Definition Audio Controller (rev a1)

lspci -v (Video card only)

04:00.0 VGA compatible controller: NVIDIA Corporation GT218 [GeForce 210] (rev a2) (prog-if 00 [VGA controller])
Subsystem: eVga.com. Corp. GT218 [GeForce 210]
Flags: bus master, fast devsel, latency 0, IRQ 125
Memory at de000000 (32-bit, non-prefetchable)
Memory at c0000000 (64-bit, prefetchable)
Memory at d0000000 (64-bit, prefetchable)
I/O ports at d000
Expansion ROM at 000c0000 [disabled]
Capabilities: [60] Power Management version 3
Capabilities: [68] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [78] Express Endpoint, MSI 00
Capabilities: [b4] Vendor Specific Information: Len=14 <?>
Capabilities: [100] Virtual Channel
Capabilities: [128] Power Budgeting <?>
Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
Kernel driver in use: vfio-pci

04:00.1 Audio device: NVIDIA Corporation High Definition Audio Controller (rev a1)
Subsystem: eVga.com. Corp. High Definition Audio Controller
Flags: bus master, fast devsel, latency 0, IRQ 17
Memory at df080000 (32-bit, non-prefetchable)
Capabilities: [60] Power Management version 3
Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [78] Express Endpoint, MSI 00
Kernel driver in use: vfio-pci

root@Tower:~# lsusb
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 004: ID 051d:0002 American Power Conversion Uninterruptible Power Supply
Bus 001 Device 005: ID 046d:c31c Logitech, Inc. Keyboard K120
Bus 001 Device 003: ID 0461:4d15 Primax Electronics, Ltd Dell Optical Mouse
Bus 001 Device 002: ID 0781:5406 SanDisk Corp. Cruzer Micro U3
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

tower-diagnostics-20180425-1933.zip

Edited April 26, 2018 by kennelm
added info

2.6k · April 26, 2018

1 hour ago, kennelm said:

What is stubbing?

I did notice in the unraid log that there were messages related to resetting low-speed USB devices for the mouse and keyboard. That only happened on the second launch of the VM but not on the 1st.

~~you can try ruling this out as a source problem by removing the usb for second launch~~ This is ruled out by being able to start the vm using vnc.

This thread references problems passing through at 210 as the only card in the system: https://lime-technology.com/forums/topic/47271-evga-nvidia-gt-210-windows-10-1-pcie-slot/

I do not know why enabling "unsafe interrupts" helped with the issue as I don't have this board, but I run that command in my syslinux.cfg on all my proliant servers without issue.

Some people have had to dump the bios on the GPU on single card nvidia systems to get them to work, others have found that moving the card to the primary slot also helps.

the following 2 vids may also be relevant in regards to that:

https://www.youtube.com/watch?v=mM7ntkiUoPk

https://www.youtube.com/watch?v=1IP-h9IKof0

if you have another gpu lying around and can toss it in and assign that as primary in the motherboard bios, and then it fixes the problem, then the video fix listed above will probably work if you want to return back to a single card system. If you can, I would start here first.

Edited April 26, 2018 by 1812

kennelm · April 26, 2018

38 minutes ago, 1812 said:

~~you can try ruling this out as a source problem by removing the usb for second launch~~ This is ruled out by being able to start the vm using vnc.

This thread references problems passing through at 210 as the only card in the system: https://lime-technology.com/forums/topic/47271-evga-nvidia-gt-210-windows-10-1-pcie-slot/

I do not know why enabling "unsafe interrupts" helped with the issue as I don't have this board, but I run that command in my syslinux.cfg on all my proliant servers without issue.

Some people have had to dump the bios on the GPU on single card nvidia systems to get them to work, others have found that moving the card to the primary slot also helps.

the following 2 vids may also be relevant in regards to that:

https://www.youtube.com/watch?v=mM7ntkiUoPk

https://www.youtube.com/watch?v=1IP-h9IKof0

if you have another gpu lying around and can toss it in and assign that as primary in the motherboard bios, and then it fixes the problem, then the video fix listed above will probably work if you want to return back to a single card system.

Thanks. I'll dig into it. Meanwhile, my Asrock Z370 Pro4 motherboard has a robust onboard video capability. Can this serve as one of the GPUs and the nvidia as the other? Or do I need 2 PCIe GPUs? I assume the primary GPU is allocated to the unraid system and the guest machine gets the other GPU? Or do I have that backwards?

2.6k · April 26, 2018

7 minutes ago, kennelm said:

Thanks. I'll dig into it. Meanwhile, my Asrock Z370 Pro4 motherboard has a robust onboard video capability. Can this serve as one of the GPUs and the nvidia as the other? Or do I need 2 PCIe GPUs? I assume the primary GPU is allocated to the unraid system and the guest machine gets the other GPU? Or do I have that backwards?

Onboard can/should be assigned to unRaid. It didn’t show in the devices inventory (unless I missed it) Is it disabled in the bios? If so, enable it and set to primary, then reboot and retest.

kennelm · April 26, 2018

20 hours ago, 1812 said:

Onboard can/should be assigned to unRaid. It didn’t show in the devices inventory (unless I missed it) Is it disabled in the bios? If so, enable it and set to primary, then reboot and retest.

Good catch. Somehow this got disabled in the BIOS. I've re-enabled the on-board GPU, but I'm not sure how to designate as "primary."

Point being, the second launch of the VM still fails. Interestingly, before I launched the VM the first time, I noticed 3 choices in the VM graphics card drop-down box: VNC, Nvidia, and Intel. After the second launch failed, I checked again and the Intel on-board GPU was not in the list anymore....

2.6k · April 26, 2018

1 hour ago, kennelm said:

Good catch. Somehow this got disabled in the BIOS. I've re-enabled the on-board GPU, but I'm not sure how to designate as "primary."

Point being, the second launch of the VM still fails. Interestingly, before I launched the VM the first time, I noticed 3 choices in the VM graphics card drop-down box: VNC, Nvidia, and Intel. After the second launch failed, I checked again and the Intel on-board GPU was not in the list anymore....

In the bios, the could be a section for video options or something similar and you'd select it there. You might just have to dig around to set which is primary or first use.

When it boots, is the unraid loading screen outputting via the onboard now or nvida card?

kennelm · April 27, 2018

51 minutes ago, 1812 said:

In the bios, the could be a section for video options or something similar and you'd select it there. You might just have to dig around to set which is primary or first use.

When it boots, is the unraid loading screen outputting via the onboard now or nvida card?

Doh. I'm an idiot. I totally overlooked a BIOS selection to designate a primary video card. Now, the unraid screen appears on the onboard card, and the Vm display is on the NVIDIA card. Again, the VM works the first time around, but on the second try, I now get a ROM error:

2018-04-27T00:16:09.953404Z qemu-system-x86_64: vfio-pci: Cannot read device rom at 0000:04:00.0
Device option ROM contents are probably invalid (check dmesg).
Skip option ROM probe with rombar=0, or load from file with romfile=

I'm going to guess that I need to dump the ROM per the video above.

2.6k · April 27, 2018

28 minutes ago, kennelm said:

Doh. I'm an idiot. I totally overlooked a BIOS selection to designate a primary video card. Now, the unraid screen appears on the onboard card, and the Vm display is on the NVIDIA card. Again, the VM works the first time around, but on the second try, I now get a ROM error:

2018-04-27T00:16:09.953404Z qemu-system-x86_64: vfio-pci: Cannot read device rom at 0000:04:00.0
Device option ROM contents are probably invalid (check dmesg).
Skip option ROM probe with rombar=0, or load from file with romfile=

I'm going to guess that I need to dump the ROM per the video above.

give it a try or get the tech powerup one and modify the header.

kennelm · April 28, 2018

On 4/26/2018 at 8:47 PM, 1812 said:

give it a try or get the tech powerup one and modify the header.

No joy in Mudville. Based on what I'm reading, this older G 210 card probably won't pass through correctly, so I just ordered the GT 710 per your post in this thread:

Edited April 28, 2018 by kennelm

2.6k · April 28, 2018

9 hours ago, kennelm said:

No joy in Mudville. Based on what I'm reading, this older G 210 card probably won't pass through correctly, so I just ordered the GT 710 per your post in this thread:

Let us know if this fixes it! The 710's are great basic cards.

CentOS VM Problem

Recommended Posts

kennelm

Link to comment

kennelm

Link to comment

1812

Link to comment

kennelm

Link to comment

1812

Link to comment

kennelm

Link to comment

1812

Link to comment

kennelm

Link to comment

1812

Link to comment

kennelm

Link to comment

1812

Link to comment

kennelm

Link to comment

1812

Link to comment

kennelm

Link to comment

1812

Link to comment

Join the conversation