August 5, 201510 yr Hi Guys , I will need some help here, here some info of my setup Mobo: Intel DX58SO CPU : Intel® Core™ i7 CPU 940 @ 2.93GHz Memory : 20480 MB GPU 1 : MSI GTX 970 GPU 2 : Zotac GTX 560 HD : 14 TB So I successfuly run two VM's , Kodibuutu(GTX560) and Windows 10(GTX970). Both work with PCIe passthrough My problem happen when I shutdown/reboot any VM's that use de GTX560. There's what I got : 2015-08-04 22:19:11.135+0000: starting up libvirt version: 1.2.15, qemu version: 2.3.0 LC_ALL=C PATH=/bin:/sbin:/usr/bin:/usr/sbin HOME=/ QEMU_AUDIO_DRV=none /usr/bin/qemu-system-x86_64 -name kodibuuntu -S -machine pc-q35-2.3,accel=kvm,usb=off,mem-merge=off -m 6144 -realtime mlock=on -smp 2,sockets=1,cores=2,threads=1 -uuid 84c1b75d-2617-16c2-7474-ee1a1d5757eb -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/kodibuuntu.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on -device i82801b11-bridge,id=pci.1,bus=pcie.0,addr=0x1e -device pci-bridge,chassis_nr=2,id=pci.2,bus=pci.1,addr=0x1 -device ich9-usb-ehci1,id=usb,bus=pci.2,addr=0x3.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.2,multifunction=on,addr=0x3 -device virtio-serial-pci,id=virtio-serial0,bus=pci.2,addr=0x4 -drive file=/mnt/disk5/appdata/kodibuuntu/vdisk1.img,if=none,id=drive-virtio-disk1,format=raw,cache=writeback -deviport=2,chassis=1,id=root.1 -device vfio-pci,host=03:00.0,bus=pcie.0,multifunction=on,x-vga=on -device vfio-pci,host=03:00.1,bus=pcie.0 -msg timestamp=on Domain id=2 is tainted: high-privileges Domain id=2 is tainted: custom-argv char device redirected to /dev/pts/0 (label charserial0) qemu: terminating on signal 15 from pid 2688 2015-08-04 23:41:26.155+0000: shutting down 2015-08-04 23:42:42.148+0000: starting up libvirt version: 1.2.15, qemu version: 2.3.0 LC_ALL=C PATH=/bin:/sbin:/usr/bin:/usr/sbin HOME=/ QEMU_AUDIO_DRV=none /usr/bin/qemu-system-x86_64 -name kodibuuntu -S -machine pc-q35-2.3,accel=kvm,usb=off,mem-merge=off -m 6144 -realtime mlock=on -smp 2,sockets=1,cores=2,threads=1 -uuid 84c1b75d-2617-16c2-7474-ee1a1d5757eb -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/kodibuuntu.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on -device i82801b11-bridge,id=pci.1,bus=pcie.0,addr=0x1e -device pci-bridge,chassis_nr=2,id=pci.2,bus=pci.1,addr=0x1 -device ich9-usb-ehci1,id=usb,bus=pci.2,addr=0x3.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.2,multifunction=on,addr=0x3 -device virtio-serial-pci,id=virtio-serial0,bus=pci.2,addr=0x4 -drive file=/mnt/disk5/appdata/kodibuuntu/vdisk1.img,if=none,id=drive-virtio-disk1,format=raw,cache=writeback -deviport=2,chassis=1,id=root.1 -device vfio-pci,host=03:00.0,bus=pcie.0,multifunction=on,x-vga=on -device vfio-pci,host=03:00.1,bus=pcie.0 -msg timestamp=on Domain id=3 is tainted: high-privileges Domain id=3 is tainted: custom-argv char device redirected to /dev/pts/0 (label charserial0) 2015-08-04T23:42:48.891177Z qemu-system-x86_64: vfio-pci: Cannot read device rom at 0000:03:00.0 Device option ROM contents are probably invalid (check dmesg). Skip option ROM probe with rombar=0, or load from file with romfile= My XML for this: <domain type='kvm' id='2' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>kodibuuntu</name> <uuid>84c1b75d-2617-16c2-7474-ee1a1d5757eb</uuid> <metadata> <vmtemplate name="Custom" icon="ubuntu.png" os="ubuntu"/> </metadata> <memory unit='KiB'>6291456</memory> <currentMemory unit='KiB'>6291456</currentMemory> <memoryBacking> <nosharepages/> <locked/> </memoryBacking> <vcpu placement='static'>2</vcpu> <cputune> <vcpupin vcpu='0' cpuset='2'/> <vcpupin vcpu='1' cpuset='3'/> </cputune> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-q35-2.3'>hvm</type> </os> <features> <acpi/> <apic/> </features> <cpu> <topology sockets='1' cores='2' threads='1'/> </cpu> <clock offset='utc'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/bin/qemu-system-x86_64</emulator> <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='writeback'/> <source file='/mnt/disk5/appdata/kodibuuntu/vdisk1.img'/> <backingStore/> <target dev='hdb' bus='virtio'/> <boot order='1'/> <alias name='virtio-disk1'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x05' function='0x0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/user/ISO/PC/kodibuntu-14.0~helix_amd64.iso'/> <backingStore/> <target dev='hda' bus='sata'/> <readonly/> <boot order='2'/> <alias name='sata0-0-0'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <alias name='usb0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <alias name='usb0'/> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0' multifunction='on'/> </controller> <controller type='sata' index='0'> <alias name='sata0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <controller type='pci' index='0' model='pcie-root'> <alias name='pcie.0'/> </controller> <controller type='pci' index='1' model='dmi-to-pci-bridge'> <alias name='pci.1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/> </controller> <controller type='pci' index='2' model='pci-bridge'> <alias name='pci.2'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/> </controller> <controller type='virtio-serial' index='0'> <alias name='virtio-serial0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x04' function='0x0'/> </controller> <filesystem type='mount' accessmode='passthrough'> <source dir='/mnt/user/'/> <target dir='user'/> <alias name='fs0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/> </filesystem> <interface type='bridge'> <mac address='52:54:00:24:3a:01'/> <source bridge='br0'/> <target dev='vnet0'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/> </interface> <serial type='pty'> <source path='/dev/pts/0'/> <target port='0'/> <alias name='serial0'/> </serial> <console type='pty' tty='/dev/pts/0'> <source path='/dev/pts/0'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <channel type='unix'> <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/kodibuuntu.org.qemu.guest_agent.0'/> <target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x1241'/> <product id='0x1603'/> <address bus='3' device='2'/> </source> <alias name='hostdev0'/> </hostdev> <memballoon model='virtio'> <alias name='balloon0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x06' function='0x0'/> </memballoon> </devices> <qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=2,chassis=1,id=root.1'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=03:00.0,bus=pcie.0,multifunction=on,x-vga=on'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=03:00.1,bus=pcie.0'/> </qemu:commandline> </domain> I try many different thing and I can tell that the error vfio-pci: Cannot read device rom at 0000:0X:00.0 happen only with the GTX 560 , never happen with the GTX 970. I switch the GTX 560 to another PCIe port , also I made it through a ubuntu and windows vm, always get the same result when I restart the vm (except PCIe device id for sure). If I want to re-use this GPU I need to reboot the physical machine. Can anybody know a solution for that ?
August 5, 201510 yr You can try passing it the rom file in order to help with this situation. It's a pretty easy thing to do. Basically download the rom (look in guide), place it somewhere on your server, add to xml the location, start. Detailed here http://lime-technology.com/wiki/index.php/UnRAID_Manual_6#Edit_XML_for_VM_to_supply_GPU_ROM_manually Unfortunately (in my experience) if this doesn't work, you don't have too many other options. You can see if that card has a firmware update from the manufacturer that may help alleviate this. You can try OVMF instead of SeaBIOS (I didn't look specifically, but I assume that is what you're using) as it may help also. This change may require you to make a new XML, as (for example) Windows can't be switched from SeaBIOS to OVMF all willy nilly.. If none of this reliably fixes the issue, there's not much that I know you can do to help. Some cards work very well, others, not so much.
August 5, 201510 yr You can try passing it the rom file in order to help with this situation. It's a pretty easy thing to do. Basically download the rom (look in guide), place it somewhere on your server, add to xml the location, start. Detailed here http://lime-technology.com/wiki/index.php/UnRAID_Manual_6#Edit_XML_for_VM_to_supply_GPU_ROM_manually Unfortunately (in my experience) if this doesn't work, you don't have too many other options. You can see if that card has a firmware update from the manufacturer that may help alleviate this. You can try OVMF instead of SeaBIOS (I didn't look specifically, but I assume that is what you're using) as it may help also. This change may require you to make a new XML, as (for example) Windows can't be switched from SeaBIOS to OVMF all willy nilly.. If none of this reliably fixes the issue, there's not much that I know you can do to help. Some cards work very well, others, not so much. This is the right advice. The issue isn't unRAID, its the gtx 560 GPU. Some GPUs just aren't well behaved enough for pass through. Try the rom file as suggested though and report back if that changes anything for you.
August 5, 201510 yr Author Perfect guys, i'll try to look if there any firmware update available for my card ,pretty sure it is not up to date. Secondly , I will try with the rom file. I will keep you update of my results. BTW , FYI - I read on some post and some people getting a black screen when they try to passthrough . I can tell that with my motherboard, I do not have on-board GPU and when the bios boot it will take the first PCIe slot with a video cards by default and when you want to pass through it. You can see that the VM try to take control of it then screen comes black with no error and the vm boot well except that the GPU will not be listed in the hardware from that VM. Maybe it could be specified somewhere in the wiki of KVM
July 9, 20187 yr Hi, Have you solved this problem? I have a similar error after rebooting the VM: (qemu) qemu-system-x86_64: vfio-pci: Cannot read device rom at 0000:02:00.0 Device option ROM contents are probably invalid (check dmesg). Skip option ROM probe with rombar=0, or load from file with romfile= My vm works only for once after each host reboot.
March 3, 20206 yr I have this issue with retro Nvidia 6600 with KVM vanilla on Z170.. its there any other solution than reboot whole machine? Can someone confirm that at least for some cards, rom file is solution? Edited March 3, 20206 yr by ruthan
Archived
This topic is now archived and is closed to further replies.