UHD 750 Passthrough Android TV VM

[BeardElk]

Alright, I´ve got a problem.

Got an Android TV x64 working in vnc, but trying to use igpu, i´ve gotten stuck. 

Tried stubbing igpu, no luck, setting pcie2 as primary in bios, no luck. (Luck = black screen)

Now i got this error message =

"VM creation error

internal error: qemu unexpectedly closed the monitor: 2021-12-17T21:40:31.651900Z qemu-system-x86_64: -device vfio-pci,host=0000:00:02.0,id=hostdev0,bus=pcie.0,addr=0x2: IGD device 0000:00:02.0 cannot support legacy mode due to existing devices at address 1f.0
2021-12-17T21:40:31.652004Z qemu-system-x86_64: -device vfio-pci,host=0000:00:1f.3,id=hostdev1,bus=pci.5,addr=0x1: vfio 0000:00:1f.3: group 13 is not viable
Please ensure all devices within the iommu_group are bound to their vfio bus driver."

when trying to create the vm from existing vdisk, and I don´t know whats wrong. 

I get a "guest has not initialized the display (yet)" every time I try something new, so just end up deleting the vm (not image) and trying again.
Please Help.

[Squid]

34 minutes ago, BeardElk said:

group 13 is not viable
Please ensure all devices within the iommu_group are bound to their vfio bus driver.

In order to do passthrough, the device has to be isolated from the other devices (ie: enable one of the ACS override modes in VM Settings)

 

But, since this is presumably an Intel system, the Intel-GVT-g app might help you out.

[SimonF]

11 hours ago, BeardElk said:

Please Help.

Which version of Unraid are you running as 6.10rc2 loads the i915 driver so you may need to black list it aswell as isolating the IOMMG group as Squid suggested.

 

echo "blacklist i915" > /boot/config/modprobe.d/i915.conf

 

11 hours ago, BeardElk said:

IGD device 0000:00:02.0 cannot support legacy mode

 

Looks like you IGD is isolated, but looks like may be the audio or other PCI device can you provide a screen print of PCI Devices in System Devices.

 

IOMMU group 1:[8086:4680] 00:02.0 VGA compatible controller: Intel Corporation AlderLake-S GT1 (rev 0c)

 

Also which machine type and Bios are you using?

[BeardElk]

On 12/18/2021 at 10:13 AM, SimonF said:

Looks like you IGD is isolated, but looks like may be the audio or other PCI device can you provide a screen print of PCI Devices in System Devices.

 

IOMMU group 1:[8086:4680] 00:02.0 VGA compatible controller: Intel Corporation AlderLake-S GT1 (rev 0c)

 

Also which machine type and Bios are you using?

 

I´m on unraid 6.10rc2, with an intel 11600k on a gigabyte z590 motherboard with the F5 bios, gonna update to F6 shorty. 

 

Print of pci-e devices : 

 

Gonna try all the tips i´ve gotten and come back to you! 

 

 

[BeardElk]

On 12/17/2021 at 11:23 PM, Squid said:

In order to do passthrough, the device has to be isolated from the other devices (ie: enable one of the ACS override modes in VM Settings)

 

But, since this is presumably an Intel system, the Intel-GVT-g app might help you out.

Intel-GVT-g app does not support 11th gen Intel yet, unfortunately.

[SimonF]

4 hours ago, BeardElk said:

 

I´m on unraid 6.10rc2, with an intel 11600k on a gigabyte z590 motherboard with the F5 bios, gonna update to F6 shorty. 

 

Print of pci-e devices : 

 

Gonna try all the tips i´ve gotten and come back to you! 

 

 

Are you using OVMF for the bios on the VM?

[BeardElk]

1 hour ago, SimonF said:

Are you using OVMF for the bios on the VM?

Yes.

[BeardElk]

So, i´ve done a bunch of testing, fucking around in bios, trying different things and with csm enabled etc, (my unraid wont even boot with csm enabled, doesn´t mater if i set it to uefi only or uefi/legacy etc). 

And with iGPU @ enabled, I now get "XML error: Attempted double use of PCI Address 0000:00:02.0", and with iGPU @ Auto, i get problems with my other VM, and unraid don´t even recognize the iGPU. (its not even an option to select, its just gone). 

I´ve stubed everything associated with intel graphics, an this is where i get "Attempted double use of PCI" etc etc. 

[BeardElk]

Alright, now that I´ve stubed the right boot option (kill me) I´m back to this :

"pci,host=0000:00:02.0,id=hostdev0,bus=pcie.0,multifunction=on,addr=0x2: IGD device 0000:00:02.0 cannot support legacy mode due to existing devices at address 1f.0
2021-12-21T21:58:06.855931Z qemu-system-x86_64: vfio: Cannot reset device 0000:00:16.0, no available reset mechanism.
2021-12-21T21:58:06.855983Z qemu-system-x86_64: vfio: Cannot reset device 0000:00:1f.5, no available reset mechanism.
2021-12-21T21:58:06.859922Z qemu-system-x86_64: vfio: Cannot reset device 0000:00:1f.3, no available reset mechanism.
2021-12-21T21:58:06.971980Z qemu-system-x86_64: vfio: Cannot reset device 0000:00:1f.5, no available reset mechanism.
2021-12-21T21:58:06.976926Z qemu-system-x86_64: vfio: Cannot reset device 0000:00:16.0, no available reset mechanism.
2021-12-21T21:58:06.980924Z qemu-system-x86_64: vfio: Cannot reset device 0000:00:1f.3, no available reset mechanism.
2021-12-21T21:58:09.090715Z qemu-system-x86_64: vfio-pci: Cannot read device rom at 0000:00:02.0"

 

And in realvnc the screen is just black. 
Hdmi output says "no input" and realvnc is just black. 

Full vm log in "test vm log.txt"

test vm log.txt

Edited December 21, 2021 by BeardElk

[BeardElk]

I´ve got a 1050ti in the server as well, for plex transcoding (haven´t gotten hdr tone mapping to work with uhd750), but the 1050ti is putting up a fucking fight as well. 
Not getting anything to work atm........... 
Sorry for my language, I´m just getting frustrated. 

[BeardElk]

So i´ve dropped the idee of using the igpu for my second vm, it works more then fine now as a plex transcoder. 

My already working vm got a gtx1660ti passthrough, and thats working fine. 
Trying to passthrough the gtx1050ti to the second vm does not work fine, in fact, does not work at all. 
I get No errors what so ever in the log, but screen output is zero and realvnc is just black. 

Going back to VNC as a gpu and its back to working.
Everything is bound at vfio, have not tried stubbing in the go file. 

 

Current devices page: 

 

Have not seen a difference in behaviour selecting group 16 & 17, or unselecting them. 
The VM behaves the same either way.

Edited December 22, 2021 by BeardElk

[BeardElk]

Log for the second VM with the gtx 1050ti passthrough, and black screen for both hdmi output and realvnc.

 

 

 

#Edit

I´ve tried to add VBios rom but everything is the same.

I´ve also tried with no vm running, ie just trying to start that vm, and it´s still just black screen. 

Also tried both Q35 and I440FX, both OVMF.

 

I´ve now also tried Spaceinvaders "rom hack", dump vbios and edit out the nvidia lock shit, still not working.

Edited December 22, 2021 by BeardElk

[BeardElk]

Now I get this :
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

But first VM still works. 
Second VM is still black.

Edit:


It worked earlier, don´t have a fucking clue what i´ve changed to break it. 
All my config now is the same as 12h ago. 

Edited December 22, 2021 by BeardElk

[ghost82]

Freeze your config, start the vm and attach diagnostics, it's impossible to help and track your changes at the same time.

Edited December 22, 2021 by ghost82

[BeardElk]

On 12/22/2021 at 8:51 AM, ghost82 said:

Freeze your config, start the vm and attach diagnostics, it's impossible to help and track your changes at the same time.

 

Sorry, my fucking add takes over sometimes, and I wont stop until I´ve got it working.... 

The vm i´m trying to get working right now is just a random W10 vm, since it uses the same config as the android tv one, just a different os (that supports realvnc) i figured it would be easiest to try get working. 
The one thing I´ve just now noticed, is this :

"-display none \
-no-user-config \
-nodefaults \" 
 

My working VM does not have those lines in the log. 
Diag and everything is in this post. 

 

belk-diagnostics-20211223-2227.zip NOT WORKING VM log.txt vfio log.txt WORKING VM log.txt

 

Edited December 23, 2021 by BeardElk

[BeardElk]

Just a small update. 
I haven´t changed everything etc etc. I´ve just followed spaceinvaders guide do dump vBios, and then try to add it to the vm. 
One problem I have is that I cannot browse for the file, its just blank when I click on it (unraid 6.10rc2). 
And even if I write the right location of the vBios file I still get "-display none" in the log when running the vm. (And no display output, just like before).

So I don´t know if its a rc2 thing or my hardware wont play nice thing. 

Thats it! 
Hope you all had a nice christmas!

Edited December 27, 2021 by BeardElk

[BeardElk]

I´m really f-in stuck! 
In the VM log i get this at the start of the log : 
"-smp 4,sockets=1,dies=1,cores=2,threads=2 \
-uuid 9316829c-37bd-f9eb-a6d6-f67bc80b3b09 \
-display none \
-no-user-config \
-nodefaults \"
which is not present in the working VM with a different gpu passthrough. 

Please Help!

[BeardElk]

So,  by some freaking miracle I got it working! 
Over the last couple of weeks I´ve tried different setups and hardware configs, and for some reason I could not get my 1050ti to passthrough, so that was the first problem I had to solve. 
Followed every guide and everything to try getting it to work... Dumped my own bios, edited it as by Spaceinvaders guide, nothing. 
Added video=efifb:off to my go file, and still nothing. 
Both my Android TV vm and Windows 10/11 vm did boot, but both gave just a black screen on both realvnc and on hdmi output. 
The screen reqognized there was "some" output, because it did not say "no hdmi input", but the screen was black non the less. 
Last night before I went to bed I read something about the 1050ti and unraid with csm, so i went into bios and enabled CSM support as the last thing I did before I went to bed, but still the same black screen. 
Today, I re-did the hole vm, never started from vnc, just selected the vdisk and set the 1050ti as primary gpu, and it fucking worked. 
By some fucking miracle, things decided they were done fucking with me, and just worked! 

I don´t know if this will help anyone but, wanna at least share what I "did" to get it to work. 
Another thing is that it would now refuse to boot if I had a keyboard connected, the google boot screen would come on and then it would just shut off, so the only control I have is via the flirc usb, but for me, thats how I wanted it anyway! 
Gonna mark this as resolved now! Thanks everyone who responded and gave me tips!  

[BeardElk]

I`m gonna restart this post myself. 

Got it working with my gtx150ti as gpu passthrough, thinking my igpu would be enough for plex transcoding, and it Was. for myself, adding users and uhd750 supports MAX 3 1080p transcoding streams, and that just dont cut it for me. 

So now I´m again at your mercy. 

 

I´ve got video=efifb:off in my go file, and no service are using my igpu other than unraid. 
So atm my thinking is that unraid is somehow using my igpu, although im telling it not to, and when I´m trying to add it to a vm, I get = "XML error: Attempted double use of PCI Address 0000:00:02.0"

I dont get why its "0000:00"?

In my vm xml, graphic card (uhd750) has 00:02.0, but audio has 00:1f:3 , so thats a obvious problem, but the fact that XML error is 0000:00:02.0 I dont get, and don´t know how to fix. 

 

And yes, they are stubbed in "tools=systemdevices"

Edited January 29, 2022 by BeardElk

[BeardElk]

In bios initial gpu is set to gpu2 (pcie1 is 1660ti pcie2 is 1050ti) , in the go file, i got "i915.force_probe=4c8a", also got "video=efifb:off" in syslinux.cfg

[SimonF]

6 hours ago, BeardElk said:

I dont get why its "0000:00"?

This is the full PCI Device, first number is the domain and mostly they are 0,

 

if you are on RC2 i915 drivers are loaded by default and will take the iGPU, run the command below and you should see the driver in use. -D also shows the domain.

 

You should black list the i915 driver and remove anything from the go file for i915. Also remove Intel_gpu_top if you have it installed as it will load the driver also.

 

root@computenode:/etc/libvirt/hooks/qemu.d# lspci -Dvs 00:02.0
0000:00:02.0 VGA compatible controller: Intel Corporation AlderLake-S GT1 (rev 0c) (prog-if 00 [VGA controller])
        DeviceName: Onboard - Video
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device 7d25
        Flags: bus master, fast devsel, latency 0, IRQ 177, IOMMU group 1
        Memory at 6001000000 (64-bit, non-prefetchable) [size=16M]
        Memory at 4000000000 (64-bit, prefetchable) [size=256M]
        I/O ports at 5000 [size=64]
        Expansion ROM at 000c0000 [virtual] [disabled] [size=128K]
        Capabilities: [40] Vendor Specific Information: Len=0c <?>
        Capabilities: [70] Express Root Complex Integrated Endpoint, MSI 00
        Capabilities: [ac] MSI: Enable+ Count=1/1 Maskable+ 64bit-
        Capabilities: [d0] Power Management version 2
        Capabilities: [100] Process Address Space ID (PASID)
        Capabilities: [200] Address Translation Service (ATS)
        Capabilities: [300] Page Request Interface (PRI)
        Capabilities: [320] Single Root I/O Virtualization (SR-IOV)
        Kernel driver in use: i915
        Kernel modules: i915

 

Edited January 29, 2022 by SimonF

[BeardElk]

On 1/29/2022 at 11:31 AM, SimonF said:

This is the full PCI Device, first number is the domain and mostly they are 0,

 

if you are on RC2 i915 drivers are loaded by default and will take the iGPU, run the command below and you should see the driver in use. -D also shows the domain.

 

You should black list the i915 driver and remove anything from the go file for i915. Also remove Intel_gpu_top if you have it installed as it will load the driver also.

 

root@computenode:/etc/libvirt/hooks/qemu.d# lspci -Dvs 00:02.0
0000:00:02.0 VGA compatible controller: Intel Corporation AlderLake-S GT1 (rev 0c) (prog-if 00 [VGA controller])
        DeviceName: Onboard - Video
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device 7d25
        Flags: bus master, fast devsel, latency 0, IRQ 177, IOMMU group 1
        Memory at 6001000000 (64-bit, non-prefetchable) [size=16M]
        Memory at 4000000000 (64-bit, prefetchable) [size=256M]
        I/O ports at 5000 [size=64]
        Expansion ROM at 000c0000 [virtual] [disabled] [size=128K]
        Capabilities: [40] Vendor Specific Information: Len=0c <?>
        Capabilities: [70] Express Root Complex Integrated Endpoint, MSI 00
        Capabilities: [ac] MSI: Enable+ Count=1/1 Maskable+ 64bit-
        Capabilities: [d0] Power Management version 2
        Capabilities: [100] Process Address Space ID (PASID)
        Capabilities: [200] Address Translation Service (ATS)
        Capabilities: [300] Page Request Interface (PRI)
        Capabilities: [320] Single Root I/O Virtualization (SR-IOV)
        Kernel driver in use: i915
        Kernel modules: i915

 

Thanks! 
Just one quick question! How do i blacklist i915? 
I´ve blacklisted something before, but I can´t remeber what or how at the moment, im drawing a blank. 
Google has not been my fiend either.

[SimonF]

3 minutes ago, BeardElk said:

Thanks! 
Just one quick question! How do i blacklist i915? 
I´ve blacklisted something before, but I can´t remeber what or how at the moment, im drawing a blank. 
Google has not been my fiend either.

Run this on the command line.

 

echo "blacklist i915" > /boot/config/modprobe.d/i915.conf and then reboot.

[BeardElk]

25 minutes ago, SimonF said:

Run this on the command line.

 

echo "blacklist i915" > /boot/config/modprobe.d/i915.conf and then reboot.

Thanks! 
Still get XML error: Attempted double use of PCI Address 0000:00:02.0 with i915 blacklisted, lspci -dvs 00:02.0 shows :

root@BelK:~# lspci -Dvs 00:02.0
0000:00:02.0 VGA compatible controller: Intel Corporation RocketLake-S GT1 [UHD Graphics 750] (rev 04) (prog-if 00 [VGA controller])
        DeviceName: Onboard - Video
        Subsystem: Gigabyte Technology Co., Ltd Device d000
        Flags: bus master, fast devsel, latency 0, IRQ 11, IOMMU group 2
        Memory at 82000000 (64-bit, non-prefetchable) [size=16M]
        Memory at 40000000 (64-bit, prefetchable) [size=256M]
        I/O ports at 6000 [size=64]
        Expansion ROM at 000c0000 [virtual] [disabled] [size=128K]
        Capabilities: [40] Vendor Specific Information: Len=0c <?>
        Capabilities: [70] Express Root Complex Integrated Endpoint, MSI 00
        Capabilities: [ac] MSI: Enable- Count=1/1 Maskable+ 64bit-
        Capabilities: [d0] Power Management version 2
        Capabilities: [100] Process Address Space ID (PASID)
        Capabilities: [200] Address Translation Service (ATS)
        Capabilities: [300] Page Request Interface (PRI)
        Kernel modules: i915

 

was unable to use "/etc/libvirt/hooks/qemu.d# :

root@BelK:~# /etc/libvirt/hooks/qemu.d# lspci -Dvs 00:02.0
bash: /etc/libvirt/hooks/qemu.d#: No such file or directory

 

also unable to bind igpu vga with audio, then I would get "Kernel driver in use: vfio-pci"

[SimonF]

4 minutes ago, BeardElk said:

/etc/libvirt/hooks/qemu.d

That was just my path when i ran the command. Can you post diags?

 

do you intel gpu top loaded in plugins?