Simpler / Easier PCI Device Pass Through for NON-GPUs


jonp

Recommended Posts

If you have a non-GPU device that you wish to assign to a guest in KVM and are using the machine type of PC (not Q35), you can use this XML and skip the <qemu:cmd> arguments at the end.  This goes in between the <devices> and </devices> tags in your domain XML.

 

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x02' slot='0x00' function='0x1'/>
      </source>
    </hostdev>

 

For "bus", "slot", and "function", this should be your PCI device address.  The above XML will pass through PCI device 02:00.1.

 

Again, this only applies to non-graphics PCI devices (NICs, USB controllers, capture cards, tuners, etc.)

 

Happy virtualizing!!

  • Like 1
  • Thanks 1
Link to comment

Is there a particular reason that there are two different machine types available?  I can see this creating a lot of confusion if certain XML elements apply to one but not the other.

 

John

 

Yes.  Q35 is a more modern machine type, but still heavy in development and not fully supported by libvirt yet.  Some folks who have attempted GPU pass through with the PC machine type and failed had success when switching to Q35.  That said, it's advised that new users as of the latest beta first try the PC type before attempting Q35.  The fate of the Q35 machine type longer term is yet to be determined.

Link to comment

Is there a particular reason that there are two different machine types available?  I can see this creating a lot of confusion if certain XML elements apply to one but not the other.

 

John

 

Yes.  Q35 is a more modern machine type, but still heavy in development and not fully supported by libvirt yet.  Some folks who have attempted GPU pass through with the PC machine type and failed had success when switching to Q35.  That said, it's advised that new users as of the latest beta first try the PC type before attempting Q35.  The fate of the Q35 machine type longer term is yet to be determined.

 

Is there any way to switch a machine type from q35 to pc?

Link to comment

Is there a particular reason that there are two different machine types available?  I can see this creating a lot of confusion if certain XML elements apply to one but not the other.

 

John

 

Yes.  Q35 is a more modern machine type, but still heavy in development and not fully supported by libvirt yet.  Some folks who have attempted GPU pass through with the PC machine type and failed had success when switching to Q35.  That said, it's advised that new users as of the latest beta first try the PC type before attempting Q35.  The fate of the Q35 machine type longer term is yet to be determined.

 

Is there any way to switch a machine type from q35 to pc?

Yes. With windows vms that are already installed though, this will cause you to need to reactivate your Windows license as this will look like you're changing the entire motherboard.

 

For VMs that utilize pci pass through, GPUs or otherwise, you'll need to adjust your qemu command line section of the xml to not use pcie. Share your xml and I'll reply with the edits to make.

Link to comment

Is there a particular reason that there are two different machine types available?  I can see this creating a lot of confusion if certain XML elements apply to one but not the other.

 

John

 

Yes.  Q35 is a more modern machine type, but still heavy in development and not fully supported by libvirt yet.  Some folks who have attempted GPU pass through with the PC machine type and failed had success when switching to Q35.  That said, it's advised that new users as of the latest beta first try the PC type before attempting Q35.  The fate of the Q35 machine type longer term is yet to be determined.

 

Is there any way to switch a machine type from q35 to pc?

Yes. With windows vms that are already installed though, this will cause you to need to reactivate your Windows license as this will look like you're changing the entire motherboard.

 

For VMs that utilize pci pass through, GPUs or otherwise, you'll need to adjust your qemu command line section of the xml to not use pcie. Share your xml and I'll reply with the edits to make.

 

Thanks for clearing that up.  Here's my xml, I'll be interested to know what changes are required.

 

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <name>Argus-TV-Win7</name>
  <uuid>c0e55720-d29b-52cc-1e31-5730c8ad65f1</uuid>
  <metadata>
    <snapshot1422571432>Fresh Install, Drivers installed, Argus Not Installed</snapshot1422571432>
    <snapshots/>
  </metadata>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <vcpu placement='static'>1</vcpu>
  <os>
    <type arch='x86_64' machine='pc-q35-2.1'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/mnt/disk/KVM/VM/Argus-TV-Win7/Argus-TV-Win7.qcow2'/>
      <target dev='hda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/disk/KVM/Media/Win7AIO.iso'/>
      <target dev='hdc' bus='sata'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='2'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:a4:58:e5'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <input type='tablet' bus='usb'/>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='5910' autoport='no' websocket='5510' listen='0.0.0.0' keymap='en-gb'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='vmvga' vram='16384' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x04' function='0x0'/>
    </memballoon>
  </devices>
  <qemu:commandline>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=02:00.0,bus=pcie.0'/>
  </qemu:commandline>
</domain>

Link to comment

There are two things to change.  The first is the machine type setting and the second is in the qemu command line arguments.  I've highlighted the parts that should be changed below:

 

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>

  <name>Argus-TV-Win7</name>

  <uuid>c0e55720-d29b-52cc-1e31-5730c8ad65f1</uuid>

  <metadata>

    <snapshot1422571432>Fresh Install, Drivers installed, Argus Not Installed</snapshot1422571432>

    <snapshots/>

  </metadata>

  <memory unit='KiB'>4194304</memory>

  <currentMemory unit='KiB'>4194304</currentMemory>

  <vcpu placement='static'>1</vcpu>

  <os>

    <type arch='x86_64' machine='pc'>hvm</type>

    <boot dev='hd'/>

  </os>

  <features>

    <acpi/>

    <apic/>

  </features>

  <clock offset='localtime'>

    <timer name='hypervclock' present='yes'/>

    <timer name='hpet' present='no'/>

  </clock>

  <on_poweroff>destroy</on_poweroff>

  <on_reboot>restart</on_reboot>

  <on_crash>restart</on_crash>

  <devices>

    <emulator>/usr/bin/qemu-system-x86_64</emulator>

    <disk type='file' device='disk'>

      <driver name='qemu' type='qcow2'/>

      <source file='/mnt/disk/KVM/VM/Argus-TV-Win7/Argus-TV-Win7.qcow2'/>

      <target dev='hda' bus='virtio'/>

      <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0'/>

    </disk>

    <disk type='file' device='cdrom'>

      <driver name='qemu' type='raw'/>

      <source file='/mnt/disk/KVM/Media/Win7AIO.iso'/>

      <target dev='hdc' bus='sata'/>

      <readonly/>

      <address type='drive' controller='0' bus='0' target='0' unit='2'/>

    </disk>

    <controller type='usb' index='0' model='ich9-ehci1'>

      <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x7'/>

    </controller>

    <controller type='usb' index='0' model='ich9-uhci1'>

      <master startport='0'/>

      <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0' multifunction='on'/>

    </controller>

    <controller type='sata' index='0'>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>

    </controller>

    <controller type='pci' index='0' model='pcie-root'/>

    <controller type='pci' index='1' model='dmi-to-pci-bridge'>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>

    </controller>

    <controller type='pci' index='2' model='pci-bridge'>

      <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/>

    </controller>

    <interface type='bridge'>

      <mac address='52:54:00:a4:58:e5'/>

      <source bridge='br0'/>

      <model type='virtio'/>

      <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/>

    </interface>

    <serial type='pty'>

      <target port='0'/>

    </serial>

    <console type='pty'>

      <target type='serial' port='0'/>

    </console>

    <input type='tablet' bus='usb'/>

    <input type='mouse' bus='ps2'/>

    <input type='keyboard' bus='ps2'/>

    <graphics type='vnc' port='5910' autoport='no' websocket='5510' listen='0.0.0.0' keymap='en-gb'>

      <listen type='address' address='0.0.0.0'/>

    </graphics>

    <video>

      <model type='vmvga' vram='16384' heads='1'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>

    </video>

    <memballoon model='virtio'>

      <address type='pci' domain='0x0000' bus='0x02' slot='0x04' function='0x0'/>

    </memballoon>

  </devices>

  <qemu:commandline>

    <qemu:arg value='-device'/>

    <qemu:arg value='vfio-pci,host=02:00.0,bus=root.1,addr=00.1'/>

  </qemu:commandline>

</domain>

 

However, if you are going to use the machine type of PC, I would actually first try to eliminate the QEMU arguments altogether and use the method I posted at the beginning of this thread.  The machine type of PC still needs to be set as I indicated above, but you can then remove the <qemu:commandline> section of the XML.  Instead, before </devices> you would insert this:

 

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
      </source>
    </hostdev>

 

That above code is accurate for your pci device of 2:00.0

Link to comment

In the end I just recreated my VM and used the snippet you posted.

 

Main reason was to migrate from an install on WHS2011 to Windows 7.

 

I was using WHS2011 as I wanted the IIS server, but now with smdions reverse proxy I've got most of the stuff I was interested in working.

 

Just got to figure out how to setup OwnCloud & reverse proxy the Argus TV web server and I'm then away from Windows other than for my PVR.

 

One other question, can I now remove this from my go file?

 

#bindpci
#DVB-T2
/usr/local/sbin/vfio-bind 0000:00:1c.4
/usr/local/sbin/vfio-bind 0000:02:00.0

Link to comment

In the end I just recreated my VM and used the snippet you posted.

 

Main reason was to migrate from an install on WHS2011 to Windows 7.

 

I was using WHS2011 as I wanted the IIS server, but now with smdions reverse proxy I've got most of the stuff I was interested in working.

 

Just got to figure out how to setup OwnCloud & reverse proxy the Argus TV web server and I'm then away from Windows other than for my PVR.

 

One other question, can I now remove this from my go file?

 

#bindpci
#DVB-T2
/usr/local/sbin/vfio-bind 0000:00:1c.4
/usr/local/sbin/vfio-bind 0000:02:00.0

Give it a shot!!  Just put a # in front of that line in your go file and save. This way if it doesn't work, you just remove the # and its back to as it was. You should be able to remove it.

Link to comment

 

One other question, can I now remove this from my go file?

 

#bindpci
#DVB-T2
/usr/local/sbin/vfio-bind 0000:00:1c.4
/usr/local/sbin/vfio-bind 0000:02:00.0

Give it a shot!!  Just put a # in front of that line in your go file and save. This way if it doesn't work, you just remove the # and its back to as it was. You should be able to remove it.

 

Works like a treat, my go file is now very much neater.  Cheers jon for walking me through this, made it much easier than the trial and error rigmarole I would have had to go through by myself, much appreciated once again.

Link to comment

 

One other question, can I now remove this from my go file?

 

#bindpci
#DVB-T2
/usr/local/sbin/vfio-bind 0000:00:1c.4
/usr/local/sbin/vfio-bind 0000:02:00.0

Give it a shot!!  Just put a # in front of that line in your go file and save. This way if it doesn't work, you just remove the # and its back to as it was. You should be able to remove it.

 

Works like a treat, my go file is now very much neater.  Cheers jon for walking me through this, made it much easier than the trial and error rigmarole I would have had to go through by myself, much appreciated once again.

No problem!  Glad we got you squared away and that you validated this method works solid.

Link to comment
  • 3 weeks later...
  • 1 month later...

So, I've been helping a fellow forumite and wrote up this guide in another thread.  Thought it might be useful here for those new to KVM as there is a lot of confusion surrounding passthrough. 

 

jonp - If you feel this doesn't belong here then please feel free to delete it.

 

This is how I passthrough my TV Card on a Intel system.

 

Prerequisites: VT-d or equivalent for AMD must be enabled in BIOS and your hardware must be VT-d compatible.

 

Before you start, check the settings for PCIe ACS Override are as below.

EDIT: Apparently there may be some issues with PCIe ACS Override so ensure this is set to off, I haven't yet had chance to investigate with my own machine but the info comes from a very knowledgeable forum user.  I have also removed references to this in syslinux.cfg - the updated config is posted below.

 

EDIT2: Now had a chance to check this and the below config is working as far as I can tell.  There is no need to edit your syslinux.cfg and PCIe ACS Override should be set to off.  I have modified the text and pictures below so it is all current and correct as far as I understand things.

I can confirm this method works with beta14b and beta15

maBRK37.jpg

 

My syslinux.cfg

default /syslinux/menu.c32
menu title Lime Technology
prompt 0
timeout 50
label unRAID OS
  menu default
  kernel /bzimage
  append initrd=/bzroot
label unRAID OS Safe Mode (no plugins)
  kernel /bzimage
  append initrd=/bzroot unraidsafemode
label Memtest86+
  kernel /memtest
label Xen/unRAID OS
  kernel /syslinux/mboot.c32
  append /xen --- /bzimage --- /bzroot
label Xen/unRAID OS Safe Mode (no plugins)
  kernel /syslinux/mboot.c32
  append /xen --- /bzimage --- /bzroot unraidsafemode

 

My go file

#!/bin/bash
# Start the Management Utility
/usr/local/sbin/emhttp &

 

lspci -knn (relevant bit only)

02:00.0 Multimedia controller [0480]: Philips Semiconductors SAA7160 [1131:7160] (rev 02)
        Subsystem: Device [6285:0001]

 

My VM setup page - Note my bridge setup maybe different to yours.

BUJsMqk.jpg

 

The resulting xml

<domain type='kvm' id='3'>
  <name>Test Argus</name>
  <uuid>a0fe523e-4438-9b12-43f3-4cbf048a442e</uuid>
  <memory unit='KiB'>2097152</memory>
  <currentMemory unit='KiB'>2097152</currentMemory>
  <memoryBacking>
    <nosharepages/>
    <locked/>
  </memoryBacking>
  <vcpu placement='static'>1</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='0'/>
  </cputune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.2'>hvm</type>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
    </hyperv>
  </features>
  <cpu mode='host-passthrough'>
    <topology sockets='1' cores='1' threads='1'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source file='/mnt/disk/KVM/VM/Test Argus/vdisk1.img'/>
      <backingStore/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <alias name='virtio-disk2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/disk/KVM/Media/Win7AIO.iso'/>
      <backingStore/>
      <target dev='hda' bus='ide'/>
      <readonly/>
      <boot order='2'/>
      <alias name='ide0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/disk/KVM/Media/virtio-win-0.1-100.iso'/>
      <backingStore/>
      <target dev='hdb' bus='ide'/>
      <readonly/>
      <alias name='ide0-0-1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0'>
      <alias name='usb0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>
    <controller type='ide' index='0'>
      <alias name='ide0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:f5:0a:c6'/>
      <source bridge='br0'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/0'/>
      <target port='0'/>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/0'>
      <source path='/dev/pts/0'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <input type='tablet' bus='usb'>
      <alias name='input0'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='5900' autoport='yes' websocket='5700' listen='0.0.0.0' keymap='en-gb'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='vmvga' vram='16384' heads='1'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </memballoon>
  </devices>
</domain>

 

Now to add in my card I paste in my passthrough config..

 

Generically, the address is

aa:bb:c

So I need to paste into

 

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0xaa' slot='0xbb' function='0xc'/>
      </source>
    </hostdev>

 

My Card's address is

02:00.0

Which becomes

 

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
      </source>
    </hostdev>

 

This goes above these last two lines in your xml, ie in the devices section.

  </devices>
</domain>

 

Once you've pasted it in and saved it then you may notice two extra lines are added automatically.  This is the KVM assigning your passthrough card a slot on the virtual machine, so don't worry about that.  In my case that was

<alias name='hostdev0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>

 

So the relevant part becomes:

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </hostdev>

 

Install the OS and boot up once it's completed, my device manager shows this:

CIFFnnR.jpg

 

Install my drivers:

QBEYosN.jpg

 

For completeness here's my finished xml, note that KVM manager does reorder things a little bit, the important thing to avoid errors is to paste it into the correct place to start with (above the last two lines)

<domain type='kvm' id='3'>
  <name>Test Argus</name>
  <uuid>a0fe523e-4438-9b12-43f3-4cbf048a442e</uuid>
  <memory unit='KiB'>2097152</memory>
  <currentMemory unit='KiB'>2097152</currentMemory>
  <memoryBacking>
    <nosharepages/>
    <locked/>
  </memoryBacking>
  <vcpu placement='static'>1</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='0'/>
  </cputune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.2'>hvm</type>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
    </hyperv>
  </features>
  <cpu mode='host-passthrough'>
    <topology sockets='1' cores='1' threads='1'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source file='/mnt/disk/KVM/VM/Test Argus/vdisk1.img'/>
      <backingStore/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <alias name='virtio-disk2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/disk/KVM/Media/Win7AIO.iso'/>
      <backingStore/>
      <target dev='hda' bus='ide'/>
      <readonly/>
      <boot order='2'/>
      <alias name='ide0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/disk/KVM/Media/virtio-win-0.1-100.iso'/>
      <backingStore/>
      <target dev='hdb' bus='ide'/>
      <readonly/>
      <alias name='ide0-0-1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0'>
      <alias name='usb0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>
    <controller type='ide' index='0'>
      <alias name='ide0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:f5:0a:c6'/>
      <source bridge='br0'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/0'/>
      <target port='0'/>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/0'>
      <source path='/dev/pts/0'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <input type='tablet' bus='usb'>
      <alias name='input0'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='5900' autoport='yes' websocket='5700' listen='0.0.0.0' keymap='en-gb'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='vmvga' vram='16384' heads='1'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </hostdev>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </memballoon>
  </devices>
</domain>

 

For Linux

 

My Ubuntu server setup in KVM

LONAGoQ.jpg

 

Pasting in

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
      </source>
    </hostdev>

 

Which the KVM plugin turns to (allocating a PCI domain on the ubuntu VM)

 

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </hostdev>

 

Then I installed the ubuntu iso & booted.

Ran

lspci -knn

 

And saw

26L6ex9.jpg

 

Note the address now matches

<address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>

 

Rather than

02:00.0 Multimedia controller [0480]: Philips Semiconductors SAA7160 [1131:7160] (rev 02)

        Subsystem: Device [6285:0001]

which was the address on the Unraid host OS.

 

Here's my complete XML file for reference

<domain type='kvm'>
  <name>TVHeadEnd</name>
  <uuid>52c64f2e-ffcd-14e4-0876-824e00c83f65</uuid>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>1048576</currentMemory>
  <memoryBacking>
    <nosharepages/>
    <locked/>
  </memoryBacking>
  <vcpu placement='static'>1</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='0'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.2'>hvm</type>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-passthrough'>
    <topology sockets='1' cores='1' threads='1'/>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source file='/mnt/disk/KVM/VM/TVHeadEnd/vdisk1.img'/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/disk/KVM/Media/ubuntu-14.10-server-amd64.iso'/>
      <target dev='hda' bus='ide'/>
      <readonly/>
      <boot order='2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <filesystem type='mount' accessmode='passthrough'>
      <source dir='/mnt/cache/Recorded TV/'/>
      <target dir='recordings'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </filesystem>
    <interface type='bridge'>
      <mac address='52:54:00:4a:ad:2c'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <input type='tablet' bus='usb'/>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' websocket='-1' listen='0.0.0.0' keymap='en-gb'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='vmvga' vram='16384' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </memballoon>
  </devices>
</domain>

 

If you encounter any problems then run

 

virsh start name

 

Where name is what you find in the second line of the XML file, in my XML file for example

<name>Test Argus</name>

 

So I would enter

virsh start "Test Argus"

 

The quotation marks are only necessary if there are spaces in your VM name.

 

Then post the results.

 

EDIT:  Added virsh commands to help solve any problems and get error messages.

  • Thanks 1
Link to comment

Attempting passthrough but no joy.

 

Here's my base XML which works;

 

<domain type='kvm'>
  <name>MediaServer</name>
  <uuid>52774a80-94bc-1c66-8317-bffa1cda7c86</uuid>
  <memory unit='KiB'>12582912</memory>
  <currentMemory unit='KiB'>12582912</currentMemory>
  <memoryBacking>
    <nosharepages/>
    <locked/>
  </memoryBacking>
  <vcpu placement='static'>4</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='0'/>
    <vcpupin vcpu='1' cpuset='1'/>
    <vcpupin vcpu='2' cpuset='2'/>
    <vcpupin vcpu='3' cpuset='3'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.2'>hvm</type>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
    </hyperv>
  </features>
  <cpu mode='host-passthrough'>
    <topology sockets='1' cores='4' threads='1'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source file='/mnt/cache/domains/TVserver/TVserver.img'/>
      <target dev='hda' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/cache/domains/virtio-win-0.1-100.iso'/>
      <target dev='hda' bus='ide'/>
      <readonly/>
      <boot order='2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:f9:a5:e0'/>
      <source bridge='virbr0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <input type='tablet' bus='usb'/>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' websocket='-1' listen='0.0.0.0'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='vmvga' vram='16384' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </memballoon>
  </devices>
</domain>

 

 

here's my PCIe device;

 

0e:00.0 Multimedia controller: Digital Devices GmbH Octopus DVB Adapter

 

 

here's the XML I added;

 

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0e' slot='0x00' function='0x0'/>
      </source>
    </hostdev>

 

 

And the resulting XML file, but when I try to launch the VM, nothing happens (no entries in log file, no errors displayed on screen);

 

 

<domain type='kvm'>
  <name>MediaServer</name>
  <uuid>52774a80-94bc-1c66-8317-bffa1cda7c86</uuid>
  <memory unit='KiB'>12582912</memory>
  <currentMemory unit='KiB'>12582912</currentMemory>
  <memoryBacking>
    <nosharepages/>
    <locked/>
  </memoryBacking>
  <vcpu placement='static'>4</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='0'/>
    <vcpupin vcpu='1' cpuset='1'/>
    <vcpupin vcpu='2' cpuset='2'/>
    <vcpupin vcpu='3' cpuset='3'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.2'>hvm</type>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
    </hyperv>
  </features>
  <cpu mode='host-passthrough'>
    <topology sockets='1' cores='4' threads='1'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source file='/mnt/cache/domains/TVserver/TVserver.img'/>
      <target dev='hda' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/cache/domains/virtio-win-0.1-100.iso'/>
      <target dev='hda' bus='ide'/>
      <readonly/>
      <boot order='2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:f9:a5:e0'/>
      <source bridge='virbr0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <input type='tablet' bus='usb'/>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' websocket='-1' listen='0.0.0.0'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='vmvga' vram='16384' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0e' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </memballoon>
  </devices>
</domain>

 

 

Link to comment

Attempting passthrough but no joy.

 

Here's my base XML which works;

 

<domain type='kvm'>
  <name>MediaServer</name>
  <uuid>52774a80-94bc-1c66-8317-bffa1cda7c86</uuid>
  <memory unit='KiB'>12582912</memory>
  <currentMemory unit='KiB'>12582912</currentMemory>
  <memoryBacking>
    <nosharepages/>
    <locked/>
  </memoryBacking>
  <vcpu placement='static'>4</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='0'/>
    <vcpupin vcpu='1' cpuset='1'/>
    <vcpupin vcpu='2' cpuset='2'/>
    <vcpupin vcpu='3' cpuset='3'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.2'>hvm</type>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
    </hyperv>
  </features>
  <cpu mode='host-passthrough'>
    <topology sockets='1' cores='4' threads='1'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source file='/mnt/cache/domains/TVserver/TVserver.img'/>
      <target dev='hda' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/cache/domains/virtio-win-0.1-100.iso'/>
      <target dev='hda' bus='ide'/>
      <readonly/>
      <boot order='2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:f9:a5:e0'/>
      <source bridge='virbr0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <input type='tablet' bus='usb'/>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' websocket='-1' listen='0.0.0.0'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='vmvga' vram='16384' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </memballoon>
  </devices>
</domain>

 

 

here's my PCIe device;

 

0e:00.0 Multimedia controller: Digital Devices GmbH Octopus DVB Adapter

 

 

here's the XML I added;

 

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0e' slot='0x00' function='0x0'/>
      </source>
    </hostdev>

 

 

And the resulting XML file, but when I try to launch the VM, nothing happens (no entries in log file, no errors displayed on screen);

 

 

<domain type='kvm'>
  <name>MediaServer</name>
  <uuid>52774a80-94bc-1c66-8317-bffa1cda7c86</uuid>
  <memory unit='KiB'>12582912</memory>
  <currentMemory unit='KiB'>12582912</currentMemory>
  <memoryBacking>
    <nosharepages/>
    <locked/>
  </memoryBacking>
  <vcpu placement='static'>4</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='0'/>
    <vcpupin vcpu='1' cpuset='1'/>
    <vcpupin vcpu='2' cpuset='2'/>
    <vcpupin vcpu='3' cpuset='3'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.2'>hvm</type>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
    </hyperv>
  </features>
  <cpu mode='host-passthrough'>
    <topology sockets='1' cores='4' threads='1'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source file='/mnt/cache/domains/TVserver/TVserver.img'/>
      <target dev='hda' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/cache/domains/virtio-win-0.1-100.iso'/>
      <target dev='hda' bus='ide'/>
      <readonly/>
      <boot order='2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:f9:a5:e0'/>
      <source bridge='virbr0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <input type='tablet' bus='usb'/>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' websocket='-1' listen='0.0.0.0'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='vmvga' vram='16384' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0e' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </memballoon>
  </devices>
</domain>

 

Ok, go to command line for me and type the following:

 

virsh start MediaServer

 

Copy and paste the output returned here.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.