Jump to content

Starting a Windows VM problem--vfio error: 0000:00:1f.3: group 7 is not viable

Sharc

By Sharc
in General Support

Recommended Posts

Sharc Noob

Sharc

Posted
Posted

I wanted to use a ssd passthrough to a windows VM. It has already got windows 10 installed (I don't want to install from scratch). 

 

Now I have configured the VM following the instructions in https://wiki.unraid.net/UnRAID_6/VM_Management#Using_a_physical_disk_in_a_VM.2C_including_one_with_an_OS_already_installed. 

 

I want to start the VM and encountered below error: 

 

2018-11-25 06:16:55.594+0000: shutting down, reason=failed
2018-11-25 06:18:52.954+0000: starting up libvirt version: 4.7.0, qemu version: 3.0.0, kernel: 4.18.15-unRAID, hostname: Tower
LC_ALL=C PATH=/bin:/sbin:/usr/bin:/usr/sbin HOME=/ QEMU_AUDIO_DRV=none /usr/local/sbin/qemu -name guest=nvme_win10,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-4-nvme_win10/master-key.aes -machine pc-i440fx-3.0,accel=kvm,usb=off,dump-guest-core=off,mem-merge=off -cpu host,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff,hv_vendor_id=none -m 12288 -realtime mlock=off -smp 3,sockets=1,cores=3,threads=1 -uuid 08e05666-9354-9e10-2516-7b4288dfa320 -no-user-config -nodefaults -chardev socket,id=charmonitor,fd=25,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime -no-hpet -no-shutdown -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x7.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x7 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x7.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x7.0x2 -device ahci,id=sata=usb.0,port=3 -vnc 0.0.0.0:0,websocket=5700 -k en-us -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pci.0,addr=0x2 -device vfio-pci,host=00:1f.3,id=hostdev0,bus=pci.0,addr=0x9 -device usb-host,hostbus=1,hostaddr=3,id=hostdev1,bus=usb.0,port=1 -device usb-host,hostbus=1,hostaddr=2,id=hostdev2,bus=usb.0,port=2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on
2018-11-25 06:18:52.954+0000: Domain id=4 is tainted: high-privileges
2018-11-25 06:18:52.954+0000: Domain id=4 is tainted: host-cpu
2018-11-25T06:18:53.023294Z qemu-system-x86_64: -chardev pty,id=charserial0: char device redirected to /dev/pts/0 (label charserial0)
2018-11-25T06:18:53.058432Z qemu-system-x86_64: -device vfio-pci,host=00:1f.3,id=hostdev0,bus=pci.0,addr=0x9: vfio error: 0000:00:1f.3: group 7 is not viable
Please ensure all devices within the iommu_group are bound to their vfio bus driver.
2018-11-25 06:18:53.079+0000: shutting down, reason=failed

 

 

 

Here are the iommu devices info: 

 

IOMMU group 0:[8086:590f] 00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v6/7th Gen Core Processor Host Bridge/DRAM Registers (rev 05)

IOMMU group 1:[8086:5902] 00:02.0 VGA compatible controller: Intel Corporation HD Graphics 610 (rev 04)

IOMMU group 2:[8086:a12f] 00:14.0 USB controller: Intel Corporation 100 Series/C230 Series Chipset Family USB 3.0 xHCI Controller (rev 31)

IOMMU group 3:[8086:a13a] 00:16.0 Communication controller: Intel Corporation 100 Series/C230 Series Chipset Family MEI Controller #1 (rev 31)

IOMMU group 4:[8086:a102] 00:17.0 SATA controller: Intel Corporation Q170/Q150/B150/H170/H110/Z170/CM236 Chipset SATA Controller [AHCI Mode] (rev 31)

IOMMU group 5:[8086:a114] 00:1c.0 PCI bridge: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #5 (rev f1)

IOMMU group 6:[8086:a118] 00:1d.0 PCI bridge: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #9 (rev f1)

IOMMU group 7:[8086:a143] 00:1f.0 ISA bridge: Intel Corporation H110 Chipset LPC/eSPI Controller (rev 31)

[8086:a121] 00:1f.2 Memory controller: Intel Corporation 100 Series/C230 Series Chipset Family Power Management Controller (rev 31)

[8086:a170] 00:1f.3 Audio device: Intel Corporation 100 Series/C230 Series Chipset Family HD Audio Controller (rev 31)

[8086:a123] 00:1f.4 SMBus: Intel Corporation 100 Series/C230 Series Chipset Family SMBus (rev 31)

IOMMU group 8:[8086:15b8] 00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (2) I219-V (rev 31)

IOMMU group 9:[14a4:23f1] 01:00.0 Non-Volatile memory controller: Lite-On Technology Corporation Device 23f1 (rev 01)

 

 

Here is my VM setting xml: 

 

<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm'>
  <name>nvme_win10</name>
  <uuid>08e05666-9354-9e10-2516-7b4288dfa320</uuid>
  <description>passthrough to disk</description>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/>
  </metadata>
  <memory unit='KiB'>12582912</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>3</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='2'/>
    <vcpupin vcpu='1' cpuset='1'/>
    <vcpupin vcpu='2' cpuset='3'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-i440fx-3.0'>hvm</type>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='none'/>
    </hyperv>
  </features>
  <cpu mode='host-passthrough' check='none'>
    <topology sockets='1' cores='3' threads='1'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source dev='/dev/nvme0n1'/>
      <target dev='hdc' bus='sata'/>
      <boot order='1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='2'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/user/domains/nvme_win10/vdisk2.img'/>
      <target dev='hdd' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/virtio-win-0.1.160-1.iso'/>
      <target dev='hdb' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:6c:f9:9e'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='3'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' websocket='-1' listen='0.0.0.0' keymap='en-us'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x00' slot='0x1f' function='0x3'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x049f'/>
        <product id='0x0051'/>
      </source>
      <address type='usb' bus='0' port='1'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x17ef'/>
        <product id='0x600e'/>
      </source>
      <address type='usb' bus='0' port='2'/>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </memballoon>
  </devices>
</domain>
 

 

 

Please help point out what I have done wrong. thanks 

 

 

Cheers

Sharc

Link to comment
John_M Experienced

John_M

Posted
Posted
35 minutes ago, Sharc said:

2018-11-25T06:18:53.058432Z qemu-system-x86_64: -device vfio-pci,host=00:1f.3,id=hostdev0,bus=pci.0,addr=0x9: vfio error: 0000:00:1f.3: group 7 is not viable
Please ensure all devices within the iommu_group are bound to their vfio bus driver.

Is the problem. You're trying to pass through an audio controller, which is in the same group as two other devices.

35 minutes ago, Sharc said:

OMMU group 7:[8086:a143] 00:1f.0 ISA bridge: Intel Corporation H110 Chipset LPC/eSPI Controller (rev 31)

[8086:a121] 00:1f.2 Memory controller: Intel Corporation 100 Series/C230 Series Chipset Family Power Management Controller (rev 31)

[8086:a170] 00:1f.3 Audio device: Intel Corporation 100 Series/C230 Series Chipset Family HD Audio Controller (rev 31)

[8086:a123] 00:1f.4 SMBus: Intel Corporation 100 Series/C230 Series Chipset Family SMBus (rev 31)

Isn't this the device you want?

Quote

IOMMU group 4:[8086:a102] 00:17.0 SATA controller: Intel Corporation Q170/Q150/B150/H170/H110/Z170/CM236 Chipset SATA Controller [AHCI Mode] (rev 31)

 

Link to comment
Sharc Noob

Sharc

Posted
  • Author
Posted

Hi John_M, 

 

Thanks for pointing that out. 

 

I can temporarily disable the passthrough of the audio controller and see how it works. But as for future use in Windows, I actually need the Audio Device to be passed through to the Windows VM. Anyway I can get it working ? 

 

 

Cheers

Sharc

Link to comment
Sharc Noob

Sharc

Posted
  • Author
Posted

Alright, I am giving up trying to move to unRaid. Pain point is making the windows VM (passthrough to NVME drive) to work properly. I might be able to get it working, but needing too much effort is my problem. I don't have that much time to deal with all these un-expected (or are they expected? ) issues. 

 

I am going with Windows with Hyper-V or Virtualbox. 

Link to comment
saarg Mentor

saarg

Posted
Posted

The issue is using consumer hardware. A lot of it isn't made for using as a virtual server. And some don't care to implement seperation of devices correctly. The ACS override is a hack to get hardware pass through to work on more than server hardware. 

Have you tried playing with the different ACS overrides? 

 

Which motherboard do you use? 

Link to comment
saarg Mentor

saarg

Posted
Posted
19 minutes ago, Sharc said:

Yes it is consumer hardware. If I would pay for the server hardware I might have went for NAS hardware products already. 

 

My motherboard is Asus H110 m-a/M.2

 

Nas hardware doesn't mean you get hardware pass through working. I would assume it's less chance on a Nas. 

Link to comment
Hoopster Experienced

Hoopster

Posted
Posted
1 hour ago, Sharc said:

Yes it is consumer hardware. If I would pay for the server hardware I might have went for NAS hardware products already. 

 

My motherboard is Asus H110 m-a/M.2

The advantage of unRAID over NAS hardware products such as QNAP, Asustor, Synology, etc. is that you can choose the hardware that meets your needs as unRAID is just the OS and it runs on virtually any hardware.

 

For the same price as many QNAP or Synology units, you could build a more powerful NAS and run unRAID.  Most commercial NAS boxes focus on file serving and sharing.  Some can run dockers OK, but, very, very few are suitable for VMs.

 

The bottom line is that your motherboard is not the best for VMs with hardware pass through because of its IOMMU limitations.

 

Obviously, you need to select the platform that best suits your needs, but, as already mentioned, Virtualbox and Hyper-V are unlikely to solve your problem since your motherboard is the limiting factor, not the unRAID VM framework.  Both of the hypervisor solutions you mentioned still rely on IOMMU for discrete device assignment.

 

Have you tried a PCIe sound card and passing that through? Perhaps that would be in a separate IOMMU group.

Link to comment
Hoopster Experienced

Hoopster

Posted
Posted

For comparison sake (both boards are from the same manufacturer):

 

Here are my IOMMU groups on my "server-grade" motherboard.  Note how almost everything is in a separate group.

 

IOMMU group 0:[8086:0c00] 00:00.0 Host bridge: Intel Corporation 4th Gen Core Processor DRAM Controller (rev 06)

 

IOMMU group 1:[8086:0412] 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller (rev 06)

 

IOMMU group 2:[8086:0c0c] 00:03.0 Audio device: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller (rev 06)

 

IOMMU group 3:[8086:8c31] 00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 05)

 

IOMMU group 4:[8086:8c3a] 00:16.0 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 (rev 04)

[8086:8c3b] 00:16.1 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #2 (rev 04)

 

IOMMU group 5:[8086:8c2d] 00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 (rev 05)

 

IOMMU group 6:[8086:8c10] 00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d5)

 

IOMMU group 7:[8086:8c14] 00:1c.2 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #3 (rev d5)

 

IOMMU group 8:[8086:8c16] 00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 (rev d5)

 

IOMMU group 9:[8086:8c1e] 00:1c.7 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #8 (rev d5)

 

IOMMU group 10:[8086:8c26] 00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 05)

 

IOMMU group 11:[8086:8c56] 00:1f.0 ISA bridge: Intel Corporation C226 Series Chipset Family Server Advanced SKU LPC Controller (rev 05)

[8086:8c02] 00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05)

[8086:8c22] 00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 05)

 

IOMMU group 12:[8086:1533] 02:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)

 

IOMMU group 13:[8086:1533] 03:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)

 

IOMMU group 14:[1a03:1150] 04:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 02)

[1a03:2000] 05:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 21)

 

And here are my IOMMU groups on my "consumer/workstation" motherboard (It looks very similar to yours):

 

IOMMU group 0:[8086:1918] 00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor Host Bridge/DRAM Registers (rev 07)

 

IOMMU group 1:[8086:191d] 00:02.0 VGA compatible controller: Intel Corporation HD Graphics P530 (rev 06)

 

IOMMU group 2:[8086:a12f] 00:14.0 USB controller: Intel Corporation 100 Series/C230 Series Chipset Family USB 3.0 xHCI Controller (rev 31)

[8086:a131] 00:14.2 Signal processing controller: Intel Corporation 100 Series/C230 Series Chipset Family Thermal Subsystem (rev 31)

 

IOMMU group 3:[8086:a13a] 00:16.0 Communication controller: Intel Corporation 100 Series/C230 Series Chipset Family MEI Controller #1 (rev 31)

[8086:a13d] 00:16.3 Serial controller: Intel Corporation 100 Series/C230 Series Chipset Family KT Redirection (rev 31)

 

IOMMU group 4:[8086:a102] 00:17.0 SATA controller: Intel Corporation Q170/Q150/B150/H170/H110/Z170/CM236 Chipset SATA Controller [AHCI Mode] (rev 31)

 

IOMMU group 5:[8086:a110] 00:1c.0 PCI bridge: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #1 (rev f1)

 

IOMMU group 6:[8086:a117] 00:1c.7 PCI bridge: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #8 (rev f1)

 

IOMMU group 7:[8086:a149] 00:1f.0 ISA bridge: Intel Corporation C236 Chipset LPC/eSPI Controller (rev 31)

[8086:a121] 00:1f.2 Memory controller: Intel Corporation 100 Series/C230 Series Chipset Family Power Management Controller (rev 31)

[8086:a170] 00:1f.3 Audio device: Intel Corporation 100 Series/C230 Series Chipset Family HD Audio Controller (rev 31)

[8086:a123] 00:1f.4 SMBus: Intel Corporation 100 Series/C230 Series Chipset Family SMBus (rev 31)

 

IOMMU group 8:[8086:15b7] 00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (2) I219-LM (rev 31)

 

IOMMU group 9:[8086:1533] 02:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)

 

Link to comment
Sharc Noob

Sharc

Posted
  • Author
Posted

I have this PC for years, running Windows 10. Hardware configurations are like below:

Motherboard:    Asus H110M-A/M.2

CPU:                Intel Pentium G4560 (2 cores, 4 threads)

Mem:               16GB DDR4-2133 (forgot the speed?)

Harddrive 1:        Plextor M9Peg (nvme 256G), with Windows 10 already running on it (tentative to use it passthrough to Windows VM in unRaid, in order to get smooth experience)

Harddrive 2:       WD Red 4T (tentative to use it as Parity drive)

Harddrive 3:       Seagate IronWolf 4T 

Harddrive 4:       Seagate IronWolf 4T 

Harddrive 5:       (Old) Micron M4 64G SSD  (tentative to use it as cache drive)

Graphic card:     Motherboard embedded Intel

NIC:                   Motherboard embedded I217 (or something similar)

 

Now I am looking at below options to provide me a home NAS: 

1. Buying an NAS hardware, i.e. QNAP. I found a fairly basic product with 4 HDD slots. So this means to run QNAP as NAS only, and PC as windows desktop.

2. Installing some virtualized VM to run some NAS service. Which means running PC as Windows desktop, and as well VM Host. Using HyperV or VirtualBox as the VM software. 

3. Convert my PC to unRaid, and run Windows VM (with passthrough audio, NVME drive, etc) on top of it, and another VM for Synology service in order for internet access of my photos

 

I was fancying using unRaid given it's way of array works there. But realized that it might take me too much time/effort to make it really work for me. 

 

I didn't want to spend much more money buying extra Graphic/Audio cards, or upgrading CPUs

 

Correct me if I made some wrong assumptions.

Link to comment
saarg Mentor

saarg

Posted
Posted

The first thing to figure out is if your graphics card can be passed through. You never answered if you had tried all the different ACS overrides to separate devices to their own groups. 

If you get that sorted, the audio device is next. If you can't pass through the analog audio, you could use the hdmi for audio? 

How is your screen audio setup? 

 

Do you really need to pass through the whole nvme disk? You can use it as your cache disk and store the VM disk on it. Your issue wasn't with passing through the nvme disk, but your audio device, so you should be able to pass through the nvme disk. 

Link to comment

Archived

This topic is now archived and is closed to further replies.

Go to topic listing
×
×
  • Create New...