January 9, 20224 yr Hello, I can't passthrough my P400 GPU to VM when start VM(Windows 10) then screen remains black. When i put one more card then P400 can passthrough to my VM. I want to use only one GPU (P400) for VM (Windows 10) and Unraid. Are there any suggestions? Thanks!
January 10, 20224 yr Make sure the gpu is not in use by unraid when it is passed through. Usually, if the vm runs without issues with a black screen, the gpu is in use by efifb, if this is the case disable it.
January 10, 20224 yr Author 3 hours ago, ghost82 said: Make sure the gpu is not in use by unraid when it is passed through. Usually, if the vm runs without issues with a black screen, the gpu is in use by efifb, if this is the case disable it. Do you mean add this command after append "video=efifb:off"?
January 10, 20224 yr Author Okay, what information you need to know. I tried efifb:off but that did not work. VM starts normaly but screen is black. There are log form VM. Quote -rtc base=localtime \ -no-hpet \ -no-shutdown \ -boot strict=on \ -device pcie-root-port,port=0x8,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x1 \ -device pcie-root-port,port=0x9,chassis=2,id=pci.2,bus=pcie.0,addr=0x1.0x1 \ -device pcie-root-port,port=0xa,chassis=3,id=pci.3,bus=pcie.0,addr=0x1.0x2 \ -device pcie-root-port,port=0xb,chassis=4,id=pci.4,bus=pcie.0,addr=0x1.0x3 \ -device pcie-root-port,port=0xc,chassis=5,id=pci.5,bus=pcie.0,addr=0x1.0x4 \ -device pcie-root-port,port=0xd,chassis=6,id=pci.6,bus=pcie.0,addr=0x1.0x5 \ -device pcie-root-port,port=0xe,chassis=7,id=pci.7,bus=pcie.0,addr=0x1.0x6 \ -device pcie-root-port,port=0xf,chassis=8,id=pci.8,bus=pcie.0,addr=0x1.0x7 \ -device pcie-pci-bridge,id=pci.9,bus=pci.1,addr=0x0 \ -device ich9-usb-ehci1,id=usb,bus=pcie.0,addr=0x7.0x7 \ -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pcie.0,multifunction=on,addr=0x7 \ -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pcie.0,addr=0x7.0x1 \ -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pcie.0,addr=0x7.0x2 \ -device virtio-serial-pci,id=virtio-serial0,bus=pci.2,addr=0x0 \ -blockdev '{"driver":"file","filename":"/mnt/user/domains/Windows 10/vdisk1.img","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-1-format","read-only":false,"cache":{"direct":false,"no-flush":false},"driver":"raw","file":"libvirt-1-storage"}' \ -device virtio-blk-pci,bus=pci.4,addr=0x0,drive=libvirt-1-format,id=virtio-disk2,bootindex=1,write-cache=on \ -netdev tap,fd=33,id=hostnet0 \ -device virtio-net,netdev=hostnet0,id=net0,mac=52:54:00:1c:94:0b,bus=pci.3,addr=0x0 \ -chardev pty,id=charserial0 \ -device isa-serial,chardev=charserial0,id=serial0 \ -chardev socket,id=charchannel0,fd=34,server,nowait \ -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 \ -device usb-tablet,id=input0,bus=usb.0,port=4 \ -device vfio-pci,host=0000:0b:00.0,id=hostdev0,bus=pci.5,multifunction=on,addr=0x0 \ -device vfio-pci,host=0000:0b:00.1,id=hostdev1,bus=pci.5,addr=0x0.0x1 \ -device vfio-pci,host=0000:06:00.0,id=hostdev2,bus=pci.7,addr=0x0 \ -device usb-host,hostbus=5,hostaddr=5,id=hostdev3,bus=usb.0,port=1 \ -device usb-host,hostbus=3,hostaddr=2,id=hostdev4,bus=usb.0,port=2 \ -device usb-host,hostbus=5,hostaddr=2,id=hostdev5,bus=usb.0,port=3 \ -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \ -msg timestamp=on 2022-01-10 10:59:23.103+0000: Domain id=1 is tainted: high-privileges 2022-01-10 10:59:23.103+0000: Domain id=1 is tainted: host-cpu char device redirected to /dev/pts/0 (label charserial0) 2022-01-10T10:59:24.424451Z qemu-system-x86_64: -device vfio-pci,host=0000:0b:00.0,id=hostdev0,bus=pci.5,multifunction=on,addr=0x0: Failed to mmap 0000:0b:00.0 BAR 3. Performance may be slow 13 hours ago, ghost82 said: Make sure the gpu is not in use by unraid when it is passed through. Usually, if the vm runs without issues with a black screen, the gpu is in use by efifb, if this is the case disable it. Do you mean add this command after append "video=efifb:off"?
January 10, 20224 yr Attach diagnostics please. 19 minutes ago, Gangdog said: Failed to mmap 0000:0b:00.0 BAR 3. Performance may be slow This suggests that the gpu is in use by something else, are you sure you applied video=efifb:off in syslinux config? Apply it, start the vm and attach diagnostics here. Edited January 10, 20224 yr by ghost82
January 11, 20224 yr Author Diagnostics with started VM and passthrough Quadro P400. tower-diagnostics-20220111-1022.zip
January 11, 20224 yr You have nvidia drivers installed in unraid, I think the host is using the gpu because of this. To avoid this issue attach to vfio at boot iommu group 27. After putting a tick to that group, save, reboot unraid and start the vm. Note: you won't have any video output for unraid, the system will seem frozen, but it will not be the case. Connect to unraid from a second device in the lan to manage it. I would pass also the dumped rom file. Edited January 11, 20224 yr by ghost82
January 11, 20224 yr Author Bind IOMMU group 27 restart Unraid but same situation - black screen. VM can stop only with "force stop". What else i can try? tower-diagnostics-20220111-2142.zip
January 11, 20224 yr No more ideas from the logs..nvidia is not loading the driver, the gpu is isolated at boot, efifb is not using the gpu, so everything should be setup for a correct passthrough (except that you didn't apply the multifunction, probably when you added the rom file, but I don't think it will make any difference). Last thing I can think is to move the gpu in another slot, especially if you have it now in the top slot, and perform bios update if it's available.
January 12, 20224 yr I noticed this in the log: Jan 11 10:58:22 Tower kernel: pci 0000:0b:00.0: BAR 3: assigned to efifb although the cmdline seems correct with video=efifb:off BOOT_IMAGE=/bzimage initrd=/bzroot video=efifb:off It seems the gpu is assigned to efifb even if you tell it to not do it (????) From unraid terminal, can you attach the output of: cat /proc/iomem
January 12, 20224 yr Author 3 hours ago, ghost82 said: I noticed this in the log: Jan 11 10:58:22 Tower kernel: pci 0000:0b:00.0: BAR 3: assigned to efifb although the cmdline seems correct with video=efifb:off BOOT_IMAGE=/bzimage initrd=/bzroot video=efifb:off It seems the gpu is assigned to efifb even if you tell it to not do it (????) From unraid terminal, can you attach the output of: Quote cat /proc/iomem root@Tower:~# cat /proc/iomem 00000000-00000fff : Reserved 00001000-0009ffff : System RAM 000a0000-000fffff : Reserved 000a0000-000bffff : PCI Bus 0000:00 000c0000-000dffff : PCI Bus 0000:00 000c0000-000ce5ff : Video ROM 000f0000-000fffff : System ROM 00100000-09c3efff : System RAM 04000000-04a00816 : Kernel code 04c00000-04e4afff : Kernel rodata 05000000-05127f7f : Kernel data 05471000-055fffff : Kernel bss 09c3f000-09ffffff : Reserved 0a000000-0a1fffff : System RAM 0a200000-0a210fff : ACPI Non-volatile Storage 0a211000-ac44ffff : System RAM ac450000-ac450fff : Reserved ac451000-ac48cfff : System RAM ac48d000-ac48dfff : Reserved ac48e000-b519c017 : System RAM b519c018-b51bca57 : System RAM b51bca58-b51bd017 : System RAM b51bd018-b51cc057 : System RAM b51cc058-b7aa5fff : System RAM b7aa6000-b7aa6fff : Reserved b7aa7000-b9a3ffff : System RAM b9a40000-bc48efff : Reserved bc470000-bc473fff : MSFT0101:00 bc474000-bc477fff : MSFT0101:00 bc48f000-bc4cdfff : ACPI Tables bc4ce000-bcbc1fff : ACPI Non-volatile Storage bcbc2000-bddfefff : Reserved bddff000-beffffff : System RAM bf000000-bfffffff : Reserved c0000000-fec02fff : PCI Bus 0000:00 d0000000-e1ffffff : PCI Bus 0000:0b d0000000-dfffffff : 0000:0b:00.0 e0000000-e1ffffff : 0000:0b:00.0 f0000000-f7ffffff : PCI MMCONFIG 0000 [bus 00-7f] f0000000-f7ffffff : Reserved f0000000-f7ffffff : pnp 00:00 fb000000-fc0fffff : PCI Bus 0000:0b fb000000-fbffffff : 0000:0b:00.0 fc080000-fc083fff : 0000:0b:00.1 fc200000-fcafffff : PCI Bus 0000:02 fc200000-fcafffff : PCI Bus 0000:03 fc200000-fc3fffff : PCI Bus 0000:08 fc200000-fc2fffff : 0000:08:00.3 fc200000-fc2fffff : xhci-hcd fc300000-fc3fffff : 0000:08:00.1 fc300000-fc3fffff : xhci-hcd fc400000-fc5fffff : PCI Bus 0000:07 fc400000-fc4fffff : 0000:07:00.0 fc400000-fc4fffff : igc fc500000-fc503fff : 0000:07:00.0 fc500000-fc503fff : igc fc600000-fc6fffff : PCI Bus 0000:0a fc600000-fc6007ff : 0000:0a:00.0 fc600000-fc6007ff : ahci fc700000-fc7fffff : PCI Bus 0000:09 fc700000-fc7007ff : 0000:09:00.0 fc700000-fc7007ff : ahci fc800000-fc8fffff : PCI Bus 0000:06 fc800000-fc803fff : 0000:06:00.0 fc900000-fc9fffff : PCI Bus 0000:05 fc900000-fc907fff : 0000:05:00.0 fc900000-fc907fff : xhci-hcd fca00000-fcafffff : PCI Bus 0000:04 fca00000-fca03fff : 0000:04:00.0 fca00000-fca03fff : nvme fca04000-fca040ff : 0000:04:00.0 fca04000-fca040ff : nvme fcb00000-fcdfffff : PCI Bus 0000:0d fcb00000-fcbfffff : 0000:0d:00.3 fcb00000-fcbfffff : xhci-hcd fcc00000-fccfffff : 0000:0d:00.1 fcc00000-fccfffff : ccp fcd00000-fcd07fff : 0000:0d:00.4 fcd08000-fcd09fff : 0000:0d:00.1 fcd08000-fcd09fff : ccp fce00000-fcefffff : PCI Bus 0000:01 fce00000-fce1ffff : 0000:01:00.0 fce20000-fce2ffff : 0000:01:00.0 fce20000-fce2ffff : nvme fce30000-fce33fff : 0000:01:00.0 fce30000-fce33fff : nvme fce34000-fceb3fff : 0000:01:00.0 fd200000-fd2fffff : Reserved fd200000-fd2fffff : pnp 00:01 fd380000-fd3fffff : amd_iommu fd400000-fd5fffff : Reserved fe000000-fe01ffff : pnp 00:03 fea00000-fea0ffff : Reserved feb80000-fec01fff : Reserved fec00000-fec003ff : IOAPIC 0 fec01000-fec013ff : IOAPIC 1 fec10000-fec10fff : Reserved fec10000-fec10fff : pnp 00:05 fed00000-fed00fff : Reserved fed00000-fed003ff : HPET 0 fed00000-fed003ff : PNP0103:00 fed40000-fed44fff : Reserved fed80000-fed8ffff : Reserved fed81500-fed818ff : AMDI0030:00 fedc0000-fedc0fff : pnp 00:05 fedc2000-fedcffff : Reserved fedd4000-fedd5fff : Reserved fee00000-ffffffff : PCI Bus 0000:00 fee00000-fee00fff : Local APIC fee00000-fee00fff : pnp 00:05 ff000000-ffffffff : Reserved ff000000-ffffffff : pnp 00:05 100000000-103f2fffff : System RAM 103f300000-103fffffff : Reserved
January 12, 20224 yr Author 15 hours ago, ghost82 said: No more ideas from the logs..nvidia is not loading the driver, the gpu is isolated at boot, efifb is not using the gpu, so everything should be setup for a correct passthrough (except that you didn't apply the multifunction, probably when you added the rom file, but I don't think it will make any difference). Last thing I can think is to move the gpu in another slot, especially if you have it now in the top slot, and perform bios update if it's available. I tried on different PCI slots but same situation. I tried to dump BIOS form P400 but GPU-Z says that can't read. But when I tried start VM with two GPU then VM works good with P400 even without using a bios dump file.
November 12, 20223 yr Were you able to get this to work? I am thinking of getting a P400 for passthrough to a Windows 10 VM and I'm trying to figure out if it'll work.
November 12, 20223 yr The p400 should work without any issue, I think there are some reports here of users running it successfully. The issue in this post was that gpu was in use by efifb on the host side (at least, the syslog was reporting this) even with the kernel parameter video=efifb:off. The issue could be: 1. a kernel bug, which could have been resolved in recent kernels 2. a bios issue: I read about a similar report (BAR 3 assigned to efifb with video=efifb:off); that user had a x79 motherboard with ami bios and he was booting the host in uefi mode, and he had to force expose uefi on pcie lanes to fix this, i.e. enable cms in bios with "uefi only" options set in all subfields. Edited November 12, 20223 yr by ghost82
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.