Passthrough new Video Card

[HarryHeck]

I have a windows 10 gaming vm that ive pass though a 1070 for quite while. This has worked with some success about 3 years. One time I had to re-seat the card an mess around to get the 1070 working again. When the card does this if you boot your vm the video card fans stop spinning and no video out.

 

I bought a 3060 to passthrough. I used space invaders script to dump rom. Its at default location. I cannot get video out of 3060.  the windows vm boots because I can add second vnc card.

 

I was going to post my xml I cannot copy from that browser.

 

9900k cpu

AsRock z390 Taichi Ultimate latest bios

3060 gigabyte card

 

[HarryHeck]

Boot logs

 

-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=31,server,nowait \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=localtime,driftfix=slew \
-global kvm-pit.lost_tick_policy=delay \
-no-hpet \
-no-shutdown \
-boot strict=on \
-device nec-usb-xhci,p2=15,p3=15,id=usb,bus=pci.0,addr=0x7 \
-device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x3 \
-blockdev '{"driver":"file","filename":"/mnt/user/domains/Windows 10/vdisk1.img","node-name":"libvirt-4-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-4-format","read-only":false,"cache":{"direct":false,"no-flush":false},"driver":"raw","file":"libvirt-4-storage"}' \
-device virtio-blk-pci,bus=pci.0,addr=0x5,drive=libvirt-4-format,id=virtio-disk2,bootindex=1,write-cache=on \
-blockdev '{"driver":"file","filename":"/mnt/user/domains/Windows 10/vdisk2.img","node-name":"libvirt-3-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-3-format","read-only":false,"cache":{"direct":false,"no-flush":false},"driver":"raw","file":"libvirt-3-storage"}' \
-device virtio-blk-pci,bus=pci.0,addr=0x6,drive=libvirt-3-format,id=virtio-disk3,write-cache=on \
-blockdev '{"driver":"file","filename":"/mnt/user/isos/Win10_1903_V1_English_x64.iso","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-2-format","read-only":true,"driver":"raw","file":"libvirt-2-storage"}' \
-device ide-cd,bus=ide.0,unit=0,drive=libvirt-2-format,id=ide0-0-0,bootindex=2 \
-blockdev '{"driver":"file","filename":"/mnt/user/isos/virtio-win-0.1.160-1.iso","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":true,"driver":"raw","file":"libvirt-1-storage"}' \
-device ide-cd,bus=ide.0,unit=1,drive=libvirt-1-format,id=ide0-0-1 \
-netdev tap,fd=33,id=hostnet0,vhost=on,vhostfd=34 \
-device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:70:23:b1,bus=pci.0,addr=0x4 \
-chardev pty,id=charserial0 \
-device isa-serial,chardev=charserial0,id=serial0 \
-chardev socket,id=charchannel0,fd=35,server,nowait \
-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 \
-device usb-tablet,id=input0,bus=usb.0,port=1 \
-vnc 0.0.0.0:0,websocket=5700 \
-k en-us \
-device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pci.0,addr=0x2 \
-device vfio-pci,host=0000:01:00.0,id=hostdev0,bus=pci.0,addr=0x8 \
-device vfio-pci,host=0000:01:00.1,id=hostdev1,bus=pci.0,addr=0x9 \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
2022-03-31 03:16:43.920+0000: Domain id=3 is tainted: high-privileges
2022-03-31 03:16:43.920+0000: Domain id=3 is tainted: host-cpu
char device redirected to /dev/pts/0 (label charserial0)

[ghost82]

No useful info here, post diagnostics

[HarryHeck]

Thanks for your reply ghost. Here is my diagnostics. Any thing else I need to be moar helpful here?

tower-diagnostics-20220331-1535.zip

[ghost82]

You don't have that 3060 installed, you have the 'old' 1070, so I cannot say anything about the 3060.

For the 1070:

12 hours ago, HarryHeck said:

One time I had to re-seat the card an mess around to get the 1070 working again. When the card does this if you boot your vm the video card fans stop spinning and no video out.

If you didn't change anything this could be hardware related (?)

 

Unfortunately there's nothing useful in the logs.

Things to try:

1. check that the gpu is fully functional in bare metal

2. set the gpu as multifunction, from this:

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/user/isos/vbios/1070gpu vbios.rom'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </hostdev>

to this:

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/user/isos/vbios/1070gpu vbios.rom'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x1'/>
    </hostdev>

 

3. bind to vfio at boot iommu group 1: you will have no video output for unraid, the screen will seem frozen, connect from another device on your lan

4. change the physical pcie slot of the gpu (top slot for vfio doesn't work well with some bios/mb)

5. change machine type to q35, it better supports pcie passthrough

6. make sure '1070gpu vbios.rom' is working for your gpu

7. for black screen issues set vm with vnc and enable remote desktop inside the vm, then set again gpu passthrough and connect to the vm with remote desktop and look at system devices

8. for black screen issue do '7' and install latest nvidia drivers from remote desktop

9. check for bios updates for your motherboard

[HarryHeck]

You caught me (= . Since I was was having issues with 3060 i re added my 1070 figured If i Got the card I had working at one time I could just swap cards then. Granted it would be nice to get both cards working with onboard igpu as well.

 

1. I will test the new gpu.

2. I will try and make this change.

3. You lost me here i'm confused. Sorry I'm somewhat of a newb. I am using onboard video via hdmi using the onboard as primary gpu via 9900k intel igpu.

4. I have been moving the card/cards around I will do this again.

5. OK q35 I assume I need to make a new VM to play around with that. Good to know.

6. I have two .rom files on from space invader script one I manually edited via hex editor.

7. I am currently using moonlight to connect to PC and VNC I will try rdp.

8. Drivers may be a little out of date 1070 or even 3060 for sure.

9. I am fully up to date on MB bios. It changed some of my settings once upgraded. I think I have all the proper sv-io and other settings set properly. Been a while )= although.

[HarryHeck]

You caught me (= . Since I was was having issues with 3060 i re added my 1070 figured If i Got the card I had working at one time I could just swap cards then. Granted it would be nice to get both cards working with onboard igpu as well.

 

1. I will test the new gpu.

2. I will try and make this change.

3. You lost me here i'm confused. Sorry I'm somewhat of a newb. I am using onboard video via hdmi using the onboard as primary gpu via 9900k intel igpu.

4. I have been moving the card/cards around I will do this again.

5. OK q35 I assume I need to make a new VM to play around with that. Good to know.

6. I have two .rom files on from space invader script one I manually edited via hex editor.

7. I am currently using moonlight to connect to PC and VNC I will try rdp.

8. Drivers may be a little out of date 1070 or even 3060 for sure.

9. I am fully up to date on MB bios. It changed some of my settings once upgraded. I think I have all the proper sv-io and other settings set properly. Been a while )= although.

[HarryHeck]

OK Moved my card to another slot. I am unable to change the machine to q35 on my current domain vm.

 

I updated my drivers to the latest. I got rdp working. I can see 1070 in device manager so I'm pretty sure the new drivers took.

 

Selecting Q35-5.1 and trying to save the VM. I get XML error PCI controller with index='0' must be model='pci-root' for this machine type, but model='pci-root' was found instead.

 

New diagnogotics

tower-diagnostics-20220331-2345.zip

[ghost82]

8 hours ago, HarryHeck said:

I get XML error PCI controller with index='0' must be model='pci-root'

Layout of q35 is different than i440fx, you cannot just change machine type: faster option is to create a new vm pointing to the same vdisk(s).

[HarryHeck]

IS ti OK Moved my card to another slot. I am unable to change the machine to q35 on my current domain vm.

 

I updated my drivers to the latest. I got rdp working. I can see 1070 in device manager so I'm pretty sure the new drivers took.

 

Selecting Q35-5.1 and trying to save the VM. I get XML error PCI controller with index='0' must be model='pci-root' for this machine type, but model='pci-root' was found instead.

 

New diagnogotics

tower-diagnostics-20220331-2345.zip

[HarryHeck]

IS it ok if my video card and audio are in different iommu groups is that preferred.

[ghost82]

4 hours ago, HarryHeck said:

IS it ok if my video card and audio are in different iommu groups is that preferred

It should be same, you can have them in separate iommu groups (alone) or in the same iommu group (alone, without any other device, apart bridges).

Make sure to setup multifunction for audio/video in the xml.

[HarryHeck]

OK made a new machine. Called it game. Swapped back to 3060 since the 1070 isn't helping.

 

Friend tested 3060 on his machine display port works just fine.

 

Video card is in its own iommu. I opened xml changed stuff to multifunction. 

 

Still no dice black screen )=

tower-diagnostics-20220405-2259.zip

[HarryHeck]

OK made a new machine. Called it game. Swapped back to 3060 since the 1070 isn't helping.

 

Friend tested 3060 on his machine display port works just fine.

 

Video card is in its own iommu. I opened xml changed stuff to multifunction. 

 

Still no dice black screen )=

[HarryHeck]

When trying to boot q35 vm without a card it says. Guest has not initialized the display (yet). It never does. So I don't think my q35 ever boots.

[HarryHeck]

OK q35-51 machine booted multifunction='on'. Had to swap seabios.  Still having issues seeing video out. I am just at a loss. This literally used to work. 3060 card works other wise. I just don't get it.

tower-diagnostics-20220410-0437.zip

[HarryHeck]

I was wrong machine is not booting. As it stands now I cannot get machine to boot. Video is now not the issue its disks. Even if I try to install windows 10 again. I cannot get the driver to load on a single disk. I can get the .inf e:\ballon\win10\amd64 to load but it doesnt see the disk. What is the proper disk setup for q35? Can i use the .img files form my old vm?