united418 Posted August 5, 2015 Share Posted August 5, 2015 Hi Guys , I will need some help here, here some info of my setup Mobo: Intel DX58SO CPU : Intel® Core™ i7 CPU 940 @ 2.93GHz Memory : 20480 MB GPU 1 : MSI GTX 970 GPU 2 : Zotac GTX 560 HD : 14 TB So I successfuly run two VM's , Kodibuutu(GTX560) and Windows 10(GTX970). Both work with PCIe passthrough My problem happen when I shutdown/reboot any VM's that use de GTX560. There's what I got : 2015-08-04 22:19:11.135+0000: starting up libvirt version: 1.2.15, qemu version: 2.3.0 LC_ALL=C PATH=/bin:/sbin:/usr/bin:/usr/sbin HOME=/ QEMU_AUDIO_DRV=none /usr/bin/qemu-system-x86_64 -name kodibuuntu -S -machine pc-q35-2.3,accel=kvm,usb=off,mem-merge=off -m 6144 -realtime mlock=on -smp 2,sockets=1,cores=2,threads=1 -uuid 84c1b75d-2617-16c2-7474-ee1a1d5757eb -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/kodibuuntu.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on -device i82801b11-bridge,id=pci.1,bus=pcie.0,addr=0x1e -device pci-bridge,chassis_nr=2,id=pci.2,bus=pci.1,addr=0x1 -device ich9-usb-ehci1,id=usb,bus=pci.2,addr=0x3.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.2,multifunction=on,addr=0x3 -device virtio-serial-pci,id=virtio-serial0,bus=pci.2,addr=0x4 -drive file=/mnt/disk5/appdata/kodibuuntu/vdisk1.img,if=none,id=drive-virtio-disk1,format=raw,cache=writeback -deviport=2,chassis=1,id=root.1 -device vfio-pci,host=03:00.0,bus=pcie.0,multifunction=on,x-vga=on -device vfio-pci,host=03:00.1,bus=pcie.0 -msg timestamp=on Domain id=2 is tainted: high-privileges Domain id=2 is tainted: custom-argv char device redirected to /dev/pts/0 (label charserial0) qemu: terminating on signal 15 from pid 2688 2015-08-04 23:41:26.155+0000: shutting down 2015-08-04 23:42:42.148+0000: starting up libvirt version: 1.2.15, qemu version: 2.3.0 LC_ALL=C PATH=/bin:/sbin:/usr/bin:/usr/sbin HOME=/ QEMU_AUDIO_DRV=none /usr/bin/qemu-system-x86_64 -name kodibuuntu -S -machine pc-q35-2.3,accel=kvm,usb=off,mem-merge=off -m 6144 -realtime mlock=on -smp 2,sockets=1,cores=2,threads=1 -uuid 84c1b75d-2617-16c2-7474-ee1a1d5757eb -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/kodibuuntu.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on -device i82801b11-bridge,id=pci.1,bus=pcie.0,addr=0x1e -device pci-bridge,chassis_nr=2,id=pci.2,bus=pci.1,addr=0x1 -device ich9-usb-ehci1,id=usb,bus=pci.2,addr=0x3.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.2,multifunction=on,addr=0x3 -device virtio-serial-pci,id=virtio-serial0,bus=pci.2,addr=0x4 -drive file=/mnt/disk5/appdata/kodibuuntu/vdisk1.img,if=none,id=drive-virtio-disk1,format=raw,cache=writeback -deviport=2,chassis=1,id=root.1 -device vfio-pci,host=03:00.0,bus=pcie.0,multifunction=on,x-vga=on -device vfio-pci,host=03:00.1,bus=pcie.0 -msg timestamp=on Domain id=3 is tainted: high-privileges Domain id=3 is tainted: custom-argv char device redirected to /dev/pts/0 (label charserial0) 2015-08-04T23:42:48.891177Z qemu-system-x86_64: vfio-pci: Cannot read device rom at 0000:03:00.0 Device option ROM contents are probably invalid (check dmesg). Skip option ROM probe with rombar=0, or load from file with romfile= My XML for this: <domain type='kvm' id='2' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>kodibuuntu</name> <uuid>84c1b75d-2617-16c2-7474-ee1a1d5757eb</uuid> <metadata> <vmtemplate name="Custom" icon="ubuntu.png" os="ubuntu"/> </metadata> <memory unit='KiB'>6291456</memory> <currentMemory unit='KiB'>6291456</currentMemory> <memoryBacking> <nosharepages/> <locked/> </memoryBacking> <vcpu placement='static'>2</vcpu> <cputune> <vcpupin vcpu='0' cpuset='2'/> <vcpupin vcpu='1' cpuset='3'/> </cputune> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-q35-2.3'>hvm</type> </os> <features> <acpi/> <apic/> </features> <cpu> <topology sockets='1' cores='2' threads='1'/> </cpu> <clock offset='utc'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/bin/qemu-system-x86_64</emulator> <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='writeback'/> <source file='/mnt/disk5/appdata/kodibuuntu/vdisk1.img'/> <backingStore/> <target dev='hdb' bus='virtio'/> <boot order='1'/> <alias name='virtio-disk1'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x05' function='0x0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/user/ISO/PC/kodibuntu-14.0~helix_amd64.iso'/> <backingStore/> <target dev='hda' bus='sata'/> <readonly/> <boot order='2'/> <alias name='sata0-0-0'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <alias name='usb0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <alias name='usb0'/> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0' multifunction='on'/> </controller> <controller type='sata' index='0'> <alias name='sata0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <controller type='pci' index='0' model='pcie-root'> <alias name='pcie.0'/> </controller> <controller type='pci' index='1' model='dmi-to-pci-bridge'> <alias name='pci.1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/> </controller> <controller type='pci' index='2' model='pci-bridge'> <alias name='pci.2'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/> </controller> <controller type='virtio-serial' index='0'> <alias name='virtio-serial0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x04' function='0x0'/> </controller> <filesystem type='mount' accessmode='passthrough'> <source dir='/mnt/user/'/> <target dir='user'/> <alias name='fs0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/> </filesystem> <interface type='bridge'> <mac address='52:54:00:24:3a:01'/> <source bridge='br0'/> <target dev='vnet0'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/> </interface> <serial type='pty'> <source path='/dev/pts/0'/> <target port='0'/> <alias name='serial0'/> </serial> <console type='pty' tty='/dev/pts/0'> <source path='/dev/pts/0'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <channel type='unix'> <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/kodibuuntu.org.qemu.guest_agent.0'/> <target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x1241'/> <product id='0x1603'/> <address bus='3' device='2'/> </source> <alias name='hostdev0'/> </hostdev> <memballoon model='virtio'> <alias name='balloon0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x06' function='0x0'/> </memballoon> </devices> <qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=2,chassis=1,id=root.1'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=03:00.0,bus=pcie.0,multifunction=on,x-vga=on'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=03:00.1,bus=pcie.0'/> </qemu:commandline> </domain> I try many different thing and I can tell that the error vfio-pci: Cannot read device rom at 0000:0X:00.0 happen only with the GTX 560 , never happen with the GTX 970. I switch the GTX 560 to another PCIe port , also I made it through a ubuntu and windows vm, always get the same result when I restart the vm (except PCIe device id for sure). If I want to re-use this GPU I need to reboot the physical machine. Can anybody know a solution for that ? Quote Link to comment
bungee91 Posted August 5, 2015 Share Posted August 5, 2015 You can try passing it the rom file in order to help with this situation. It's a pretty easy thing to do. Basically download the rom (look in guide), place it somewhere on your server, add to xml the location, start. Detailed here http://lime-technology.com/wiki/index.php/UnRAID_Manual_6#Edit_XML_for_VM_to_supply_GPU_ROM_manually Unfortunately (in my experience) if this doesn't work, you don't have too many other options. You can see if that card has a firmware update from the manufacturer that may help alleviate this. You can try OVMF instead of SeaBIOS (I didn't look specifically, but I assume that is what you're using) as it may help also. This change may require you to make a new XML, as (for example) Windows can't be switched from SeaBIOS to OVMF all willy nilly.. If none of this reliably fixes the issue, there's not much that I know you can do to help. Some cards work very well, others, not so much. Quote Link to comment
jonp Posted August 5, 2015 Share Posted August 5, 2015 You can try passing it the rom file in order to help with this situation. It's a pretty easy thing to do. Basically download the rom (look in guide), place it somewhere on your server, add to xml the location, start. Detailed here http://lime-technology.com/wiki/index.php/UnRAID_Manual_6#Edit_XML_for_VM_to_supply_GPU_ROM_manually Unfortunately (in my experience) if this doesn't work, you don't have too many other options. You can see if that card has a firmware update from the manufacturer that may help alleviate this. You can try OVMF instead of SeaBIOS (I didn't look specifically, but I assume that is what you're using) as it may help also. This change may require you to make a new XML, as (for example) Windows can't be switched from SeaBIOS to OVMF all willy nilly.. If none of this reliably fixes the issue, there's not much that I know you can do to help. Some cards work very well, others, not so much. This is the right advice. The issue isn't unRAID, its the gtx 560 GPU. Some GPUs just aren't well behaved enough for pass through. Try the rom file as suggested though and report back if that changes anything for you. Quote Link to comment
united418 Posted August 5, 2015 Author Share Posted August 5, 2015 Perfect guys, i'll try to look if there any firmware update available for my card ,pretty sure it is not up to date. Secondly , I will try with the rom file. I will keep you update of my results. BTW , FYI - I read on some post and some people getting a black screen when they try to passthrough . I can tell that with my motherboard, I do not have on-board GPU and when the bios boot it will take the first PCIe slot with a video cards by default and when you want to pass through it. You can see that the VM try to take control of it then screen comes black with no error and the vm boot well except that the GPU will not be listed in the hardware from that VM. Maybe it could be specified somewhere in the wiki of KVM Quote Link to comment
DaLaoWang Posted July 9, 2018 Share Posted July 9, 2018 Hi, Have you solved this problem? I have a similar error after rebooting the VM: (qemu) qemu-system-x86_64: vfio-pci: Cannot read device rom at 0000:02:00.0 Device option ROM contents are probably invalid (check dmesg). Skip option ROM probe with rombar=0, or load from file with romfile= My vm works only for once after each host reboot. Quote Link to comment
ruthan Posted March 3, 2020 Share Posted March 3, 2020 (edited) I have this issue with retro Nvidia 6600 with KVM vanilla on Z170.. its there any other solution than reboot whole machine? Can someone confirm that at least for some cards, rom file is solution? Edited March 3, 2020 by ruthan Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.