***GUIDE*** Passthrough Entire PCI USB Controller


archedraft

511 posts in this topic Last Reply

Recommended Posts

6 hours ago, darrenyorston said:

Yes the card works when not passed through to a VM.

 

I think I may have solved the problem. Seems the power cable might have had a problem. I have changed out to a different Molex cable and now the ports are active. Maybe a bad connection though, the same connectors run my hot swap-able drive bays fine.

 

Now I am getting "DRIVER_IRQL_NOT_LESS_OR_EQUAL" on astkmd.sys

 

You are still stubbing the card? If you are, you could try to change the pcie slot to see if it makes a difference. 

Link to post
  • Replies 510
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Popular Posts

Intent: This guide will take you through the process of passing through an entire PCI USB Controller instead of doing it individually. One benefit is that USB will be plug n play on your virtual mac

I automated steps 1-9 of this guide.   The output shows USB bus #, PCI Address, and IOMMU group #   Bus 1 PCI 0000:00:1a.0 IOMMU 20     Intel Corp.     Linux Foundation 2.0 root hub Bus 2

I believe users with this specific card (Startech PEXUSB3S44V) have no issues passing them to the machine, but have issues once passed to the machine inside the VM. Mine throws a Code 10 error on the

Posted Images

Still doing it the same as listed previously. I can't change the slot unfortunately. The Supermicro board only has two slots and the other one is occupied by my 4 port NIC. It would be a bitch to change over as the 4 port NIC is passed through to my PFSense VM.

 

Im thinking of deleting the VM and making it anew. Maybe something is wrong with it.

Link to post

I have tried "stubbing the devices", according to "gridrunner", using the new method "vfio-pci".

 

Syslinux Configuration: 

  append vfio-pci.ids=1b73:1100 pcie_acs_override=downstream initrd=/bzroot

PCI Devices and IOMMU Groups:

IOMMU group 18
	[1b73:1100] 03:00.0 USB controller: Fresco Logic FL1100 USB 3.0 Host Controller (rev ff)

USB Devices:

Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 002: ID 03f0:5607 Hewlett-Packard 
Bus 001 Device 003: ID 045b:0209 Hitachi, Ltd 
Bus 001 Device 004: ID 1b1c:1c00 Corsair 
Bus 001 Device 005: ID 046d:c332 Logitech, Inc. 
Bus 001 Device 006: ID 1b1c:1b13 Corsair 
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 002 Device 002: ID 045b:0210 Hitachi, Ltd 

However, I am getting this error when I start my Windows 10 Virtual Machines.

Execution error

internal error: qemu unexpectedly closed the monitor: 

2017-05-10T08:04:34.456440Z qemu-system-x86_64: 

-device vfio-pci,host=03:00.0,id=hostdev2,bus=pci.0,addr=0x9: vfio: Error: Failed to setup INTx fd: Device or resource busy 

2017-05-10T08:04:36.333862Z qemu-system-x86_64: 

-device vfio-pci,host=03:00.0,id=hostdev2,bus=pci.0,addr=0x9: Device initialization failed

Should,  I edit my VM XML?

<domain type='kvm'>
  <name>Windows 10 VM2 (Opperational)</name>
  <uuid>0ceecee2-196e-7ba7-ef43-5cd95ee4143b</uuid>
  <description>Windows 10 Enterprise X64 1703 RS2</description>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/>
  </metadata>
  <memory unit='KiB'>12582912</memory>
  <currentMemory unit='KiB'>12582912</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>2</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='3'/>
    <vcpupin vcpu='1' cpuset='7'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.7'>hvm</type>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='none'/>
    </hyperv>
  </features>
  <cpu mode='host-passthrough'>
    <topology sockets='1' cores='1' threads='2'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/user/domains/Windows 10 VM2 (Opperational)/vdisk1.img'/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/user/domains/Windows 10 VM2 (Opperational)/vdisk2.img'/>
      <target dev='hdd' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/iso/windows/en_windows_10_enterprise_version_1703_updated_march_2017_x64_dvd_10189290.iso'/>
      <target dev='hda' bus='ide'/>
      <readonly/>
      <boot order='2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/latest-virtio/virtio-win/virtio-win-0.1.135-2/virtio-win-0.1.135.iso'/>
      <target dev='hdb' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='nec-xhci'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:5c:f3:9a'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <hostdev mode='subsystem' type='pci' managed='yes' xvga='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x04' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </memballoon>
  </devices>
</domain>

 

Link to post
On 2017-5-10 at 9:07 AM, Awkward said:

I have tried "stubbing the devices", according to "gridrunner", using the new method "vfio-pci".

 

Syslinux Configuration: 


  append vfio-pci.ids=1b73:1100 pcie_acs_override=downstream initrd=/bzroot

PCI Devices and IOMMU Groups:


IOMMU group 18
	[1b73:1100] 03:00.0 USB controller: Fresco Logic FL1100 USB 3.0 Host Controller (rev ff)

USB Devices:


Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 002: ID 03f0:5607 Hewlett-Packard 
Bus 001 Device 003: ID 045b:0209 Hitachi, Ltd 
Bus 001 Device 004: ID 1b1c:1c00 Corsair 
Bus 001 Device 005: ID 046d:c332 Logitech, Inc. 
Bus 001 Device 006: ID 1b1c:1b13 Corsair 
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 002 Device 002: ID 045b:0210 Hitachi, Ltd 

However, I am getting this error when I start my Windows 10 Virtual Machines.


Execution error

internal error: qemu unexpectedly closed the monitor: 

2017-05-10T08:04:34.456440Z qemu-system-x86_64: 

-device vfio-pci,host=03:00.0,id=hostdev2,bus=pci.0,addr=0x9: vfio: Error: Failed to setup INTx fd: Device or resource busy 

2017-05-10T08:04:36.333862Z qemu-system-x86_64: 

-device vfio-pci,host=03:00.0,id=hostdev2,bus=pci.0,addr=0x9: Device initialization failed

Should,  I edit my VM XML?


<domain type='kvm'>
  <name>Windows 10 VM2 (Opperational)</name>
  <uuid>0ceecee2-196e-7ba7-ef43-5cd95ee4143b</uuid>
  <description>Windows 10 Enterprise X64 1703 RS2</description>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/>
  </metadata>
  <memory unit='KiB'>12582912</memory>
  <currentMemory unit='KiB'>12582912</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>2</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='3'/>
    <vcpupin vcpu='1' cpuset='7'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.7'>hvm</type>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='none'/>
    </hyperv>
  </features>
  <cpu mode='host-passthrough'>
    <topology sockets='1' cores='1' threads='2'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/user/domains/Windows 10 VM2 (Opperational)/vdisk1.img'/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/user/domains/Windows 10 VM2 (Opperational)/vdisk2.img'/>
      <target dev='hdd' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/iso/windows/en_windows_10_enterprise_version_1703_updated_march_2017_x64_dvd_10189290.iso'/>
      <target dev='hda' bus='ide'/>
      <readonly/>
      <boot order='2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/latest-virtio/virtio-win/virtio-win-0.1.135-2/virtio-win-0.1.135.iso'/>
      <target dev='hdb' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='nec-xhci'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:5c:f3:9a'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <hostdev mode='subsystem' type='pci' managed='yes' xvga='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x04' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </memballoon>
  </devices>
</domain>

 

 

3
 

 

Quote

 

 

Try this

append vfio-pci.ids=1b73 vfio_iommu_type1.allow_unsafe_interrupts=1 iommu=pt pcie_acs_override=downstream initrd=/bzroot

Edited by gridrunner
Link to post
On 5/10/2017 at 6:27 PM, gridrunner said:

 

 

Try this

append vfio-pci.ids=1b73 vfio_iommu_type1.allow_unsafe_interrupts=1 iommu=pt pcie_acs_override=downstream initrd=/bzroot

 

We use "vfio-pci.ids=1b73" and omitting the ":1100" since we are using "vfio_iommu_type1.allow_unsafe_interrupts=1 iommu=pt".

 

Thank you for your assistance, Gridrunner, I will give that a try.

 

Update:

 

I have noticed that in the VM setting, "Other PCI Devices" no longer see the "USB controller: Fresco Logic FL1100 USB 3.0 Host Controller" as it is now integrated into the "USB Devices", which is now hot pluggable.

 

 

Edited by Awkward
Updating Progress
Link to post
3 hours ago, Awkward said:

 

We use "vfio-pci.ids=1b73" and omitting the ":1100" since we are using "vfio_iommu_type1.allow_unsafe_interrupts=1 iommu=pt".

 

Thank you for your assistance, Gridrunner, I will give that a try.

 

Update:

 

I have noticed that in the VM setting, "Other PCI Devices" no longer see the "USB controller: Fresco Logic FL1100 USB 3.0 Host Controller" as it is now integrated into the "USB Devices", which is now hot pluggable.

 

 

 

oopps no my bad. I left out the :1100 in error

should be

append vfio-pci.ids=1b73:1100 vfio_iommu_type1.allow_unsafe_interrupts=1 iommu=pt pcie_acs_override=downstream initrd=/bzroot

 

Link to post

Hey,

started the routine but got confused O.o

 

I am using:

USB 2.0 on board(2 ports),

USB 3.0 on board(4 ports),

USB 3.0 Header (2 ports)

 

my output:

~#lspci | grep USB
00:14.0 USB controller: Intel Corporation Sunrise Point-H USB 3.0 xHCI Controller (rev 31)

~#lsusb
Bus 002 Device 002: ID 0951:1666 Kingston Technology DataTraveler G4
Bus 002 Device 003: ID 0781:5590 SanDisk Corp.
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 005: ID 04b3:3025 IBM Corp. NetVista Full Width Keyboard
Bus 001 Device 004: ID 03f0:2b17 Hewlett-Packard LaserJet 1020
Bus 001 Device 003: ID 045e:0083 Microsoft Corp. Basic Optical Mouse
Bus 001 Device 002: ID 046d:c52b Logitech, Inc. Unifying Receiver
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

IOMMU group 3
	[8086:a12f] 00:14.0 USB controller: Intel Corporation Sunrise Point-H USB 3.0 xHCI Controller (rev 31)

well,

looks like the 'USB 2.0 onboard' and the 'USB 3.0 Header',

sharing the same bus (BUS 001).


I only want to passthru all the devices from Bus 001,
how do I do that?

Edited by amstel
Link to post
  • 3 weeks later...
On 5/20/2017 at 1:45 AM, amstel said:

Hey,

started the routine but got confused O.o

 

I am using:

USB 2.0 on board(2 ports),

USB 3.0 on board(4 ports),

USB 3.0 Header (2 ports)

 

my output:


~#lspci | grep USB
00:14.0 USB controller: Intel Corporation Sunrise Point-H USB 3.0 xHCI Controller (rev 31)

~#lsusb
Bus 002 Device 002: ID 0951:1666 Kingston Technology DataTraveler G4
Bus 002 Device 003: ID 0781:5590 SanDisk Corp.
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 005: ID 04b3:3025 IBM Corp. NetVista Full Width Keyboard
Bus 001 Device 004: ID 03f0:2b17 Hewlett-Packard LaserJet 1020
Bus 001 Device 003: ID 045e:0083 Microsoft Corp. Basic Optical Mouse
Bus 001 Device 002: ID 046d:c52b Logitech, Inc. Unifying Receiver
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

IOMMU group 3
	[8086:a12f] 00:14.0 USB controller: Intel Corporation Sunrise Point-H USB 3.0 xHCI Controller (rev 31)

well,

looks like the 'USB 2.0 onboard' and the 'USB 3.0 Header',

sharing the same bus (BUS 001).


I only want to passthru all the devices from Bus 001,
how do I do that?

 

I had similar issue..  and when in XHCI you cannot seem to separate the Intel controllers to pass them through. I had to make a BIOS setting change (relating to EHCI/OHCI/XHCI - varies on mobo vendor) which effectively makes them USB 2.0 only but splits the controllers. Then you can pass through one.     

Link to post

Does anyone have any insight if I could pass though group 16 to a VM for the USB controller without too much hassle?

This is with ACS override on, I'll try rebooting later to see what the groups are like without it on again but they are not great for anything other than GPU's.

 

IOMMU group 14
	[10de:1189] 0c:00.0 VGA compatible controller: NVIDIA Corporation GK104 [GeForce GTX 670] (rev a1)
	[10de:0e0a] 0c:00.1 Audio device: NVIDIA Corporation GK104 HDMI Audio Controller (rev a1)
IOMMU group 15
	[1002:679a] 0d:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti PRO [Radeon HD 7950/8950 OEM / R9 280]
	[1002:aaa0] 0d:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti HDMI Audio [Radeon HD 7870 XT / 7950/7970]
IOMMU group 16
	[1022:145a] 11:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device 145a
	[1022:1456] 11:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Device 1456
	[1022:145c] 11:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] USB3 Host Controller
IOMMU group 17
	[1022:1455] 12:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device 1455
	[1022:7901] 12:00.2 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
	[1022:1457] 12:00.3 Audio device: Advanced Micro Devices, Inc. [AMD] Device 1457

 

Link to post
On 6/7/2017 at 2:59 AM, methanoid said:

Q: Without boring for reason as to WHY I need to but if I want add another USB 3.0 controller card does it have to be a different chipset (NEC/Etron/VIA etc) OR can I put an identical card in and pass that through as well?

This should work.  I have two Fresco 1100 USB cards in my rig.  I run two gaming VMs in paralell running Oculus and Vive.  My setup isn't perfect.  I get occasional hangs, especially when playing "switchboard operator" on the USB ports.  I think this is due to my own problems with PCI initialization.  My cards reside at 4e and 54.  Here's where they are discovered in syslog.txt:

Quote

Jun  8 18:26:09 Yggdrasil kernel: pci 0000:4e:00.0: PME# supported from D0 D1 D3hot
Jun  8 18:26:09 Yggdrasil kernel: pci 0000:4c:09.0: PCI bridge to [bus 4e]
Jun  8 18:26:09 Yggdrasil kernel: pci 0000:4c:09.0:   bridge window [mem 0xcb200000-0xcb2fffff]
Jun  8 18:26:09 Yggdrasil kernel: pci 0000:4f:00.0: [10de:1b81] type 00 class 0x030000
Jun  8 18:26:09 Yggdrasil kernel: pci 0000:4f:00.0: reg 0x10: [mem 0xca000000-0xcaffffff]
Jun  8 18:26:09 Yggdrasil kernel: pci 0000:4f:00.0: reg 0x14: [mem 0xb0000000-0xbfffffff 64bit pref]
Jun  8 18:26:09 Yggdrasil kernel: pci 0000:4f:00.0: reg 0x1c: [mem 0xc0000000-0xc1ffffff 64bit pref]
Jun  8 18:26:09 Yggdrasil kernel: pci 0000:4f:00.0: reg 0x24: [io  0xb000-0xb07f]
Jun  8 18:26:09 Yggdrasil kernel: pci 0000:4f:00.0: reg 0x30: [mem 0xcb000000-0xcb07ffff pref]
Jun  8 18:26:09 Yggdrasil kernel: pci 0000:4f:00.1: [10de:10f0] type 00 class 0x040300
Jun  8 18:26:09 Yggdrasil kernel: pci 0000:4f:00.1: reg 0x10: [mem 0xcb080000-0xcb083fff]
Jun  8 18:26:09 Yggdrasil kernel: pci 0000:4c:10.0: PCI bridge to [bus 4f]
Jun  8 18:26:09 Yggdrasil kernel: pci 0000:4c:10.0:   bridge window [io  0xb000-0xbfff]
Jun  8 18:26:09 Yggdrasil kernel: pci 0000:4c:10.0:   bridge window [mem 0xca000000-0xcb0fffff]
Jun  8 18:26:09 Yggdrasil kernel: pci 0000:4c:10.0:   bridge window [mem 0xb0000000-0xc1ffffff 64bit pref]

Similar stuff on 54.  Later, it is revisited at 4c (which is the PCI bridge chip on MOBO)

Quote

Jun  8 18:26:09 Yggdrasil kernel: pci 0000:02:00.0: bridge window [mem 0x0010_0000-0x002f_ffff] to [bus 03-07] add_size 400000 add_align 100000
Jun  8 18:26:09 Yggdrasil kernel: pci 0000:4c:09.0: bridge window [io  0x1000-0x0fff] to [bus 4e] add_size 1000
Jun  8 18:26:09 Yggdrasil kernel: pci 0000:4c:09.0: bridge window [mem 0x0010_0000-0x000f_ffff 64bit pref] to [bus 4e] add_size 20_0000 add_align 10_0000
Jun  8 18:26:09 Yggdrasil kernel: pci 0000:00:1c.0: bridge window [io  0x1000-0x0fff] to [bus 52] add_size 1000
Jun  8 18:26:09 Yggdrasil kernel: pci 0000:00:1c.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 52] add_size 20_0000 add_align 10_0000
Jun  8 18:26:09 Yggdrasil kernel: pci 0000:00:1c.0: bridge window [mem 0x00100000-0x000fffff] to [bus 52] add_size 200000 add_align 100000

And then I get a non-fatal error:

Quote

Jun  8 18:26:09 Yggdrasil kernel: pci 0000:00:01.1: PCI bridge to [bus 02-4a]
Jun  8 18:26:09 Yggdrasil kernel: pci 0000:00:01.1:   bridge window [io  0x4000-0x7fff]
Jun  8 18:26:09 Yggdrasil kernel: pci 0000:00:01.1:   bridge window [mem 0xcf200000-0xdf1fffff]
Jun  8 18:26:09 Yggdrasil kernel: pci 0000:00:01.1:   bridge window [mem 0xc2100000-0xc40fffff 64bit pref]
Jun  8 18:26:09 Yggdrasil kernel: pci 0000:4c:09.0: res[15]=[mem 0x0010_0000-0x000f_ffff 64bit pref] res_to_dev_res add_size 20_0000 min_align 10_0000
Jun  8 18:26:09 Yggdrasil kernel: pci 0000:4c:09.0: res[15]=[mem 0x0010_0000-0x002f_ffff 64bit pref] res_to_dev_res add_size 20_0000 min_align 10_0000
Jun  8 18:26:09 Yggdrasil kernel: pci 0000:4c:09.0: res[13]=[io  0x1000-0x0fff] res_to_dev_res add_size 1000 min_align 1000
Jun  8 18:26:09 Yggdrasil kernel: pci 0000:4c:09.0: res[13]=[io  0x1000-0x1fff] res_to_dev_res add_size 1000 min_align 1000
Jun  8 18:26:09 Yggdrasil kernel: pci 0000:4c:09.0: BAR 15: no space for [mem size 0x00200000 64bit pref]
Jun  8 18:26:09 Yggdrasil kernel: pci 0000:4c:09.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
Jun  8 18:26:09 Yggdrasil kernel: pci 0000:4c:09.0: BAR 13: no space for [io  size 0x1000]
Jun  8 18:26:09 Yggdrasil kernel: pci 0000:4c:09.0: BAR 13: failed to assign [io  size 0x1000]
Jun  8 18:26:09 Yggdrasil kernel: pci 0000:4c:09.0: BAR 15: no space for [mem size 0x00200000 64bit pref]
Jun  8 18:26:09 Yggdrasil kernel: pci 0000:4c:09.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
Jun  8 18:26:09 Yggdrasil kernel: pci 0000:4c:09.0: BAR 13: no space for [io  size 0x1000]
Jun  8 18:26:09 Yggdrasil kernel: pci 0000:4c:09.0: BAR 13: failed to assign [io  size 0x1000]
Jun  8 18:26:09 Yggdrasil kernel: pci 0000:4c:08.0: PCI bridge to [bus 4d]
Jun  8 18:26:09 Yggdrasil kernel: pci 0000:4c:08.0:   bridge window [mem 0xcb300000-0xcb3fffff]
Jun  8 18:26:09 Yggdrasil kernel: pci 0000:4c:09.0: PCI bridge to [bus 4e]
Jun  8 18:26:09 Yggdrasil kernel: pci 0000:4c:09.0:   bridge window [mem 0xcb200000-0xcb2fffff]
Jun  8 18:26:09 Yggdrasil kernel: pci 0000:4c:10.0: PCI bridge to [bus 4f]
 

Then another non-fatal error:

Quote

Jun  8 18:26:09 Yggdrasil kernel: DMAR: [Firmware Bug]: RMRR entry for device 4e:00.0 is broken - applying workaround
Jun  8 18:26:09 Yggdrasil kernel: DMAR: [Firmware Bug]: RMRR entry for device 54:00.0 is broken - applying workaround

And then this message:

Quote

Jun  8 18:26:09 Yggdrasil kernel: DMAR: Setting RMRR:
Jun  8 18:26:09 Yggdrasil kernel: DMAR: Ignoring identity map for HW passthrough device 0000:00:14.0 [0x66e06000 - 0x66e15fff]
Jun  8 18:26:09 Yggdrasil kernel: DMAR: Ignoring identity map for HW passthrough device 0000:00:1a.0 [0x66e06000 - 0x66e15fff]
Jun  8 18:26:09 Yggdrasil kernel: DMAR: Ignoring identity map for HW passthrough device 0000:00:1d.0 [0x66e06000 - 0x66e15fff]
Jun  8 18:26:09 Yggdrasil kernel: DMAR: Ignoring identity map for HW passthrough device 0000:4e:00.0 [0x66e06000 - 0x66e15fff]
Jun  8 18:26:09 Yggdrasil kernel: DMAR: Ignoring identity map for HW passthrough device 0000:54:00.0 [0x66e06000 - 0x66e15fff]
Jun  8 18:26:09 Yggdrasil kernel: DMAR: Prepare 0-16MiB unity mapping for LPC
Jun  8 18:26:09 Yggdrasil kernel: DMAR: Ignoring identity map for HW passthrough device 0000:00:1f.0 [0x0 - 0xffffff]
Jun  8 18:26:09 Yggdrasil kernel: DMAR: Intel(R) Virtualization Technology for Directed I/O

And finally this message:

Quote

Jun  8 18:26:09 Yggdrasil kernel: hub 5-0:1.0: USB hub found
Jun  8 18:26:09 Yggdrasil kernel: hub 5-0:1.0: 2 ports detected
Jun  8 18:26:09 Yggdrasil kernel: xhci_hcd 0000:06:00.0: xHCI Host Controller
Jun  8 18:26:09 Yggdrasil kernel: xhci_hcd 0000:06:00.0: new USB bus registered, assigned bus number 6
Jun  8 18:26:09 Yggdrasil kernel: hub 6-0:1.0: USB hub found
Jun  8 18:26:09 Yggdrasil kernel: hub 6-0:1.0: 2 ports detected
Jun  8 18:26:09 Yggdrasil kernel: xhci_hcd 0000:4e:00.0: xHCI Host Controller
Jun  8 18:26:09 Yggdrasil kernel: xhci_hcd 0000:4e:00.0: new USB bus registered, assigned bus number 7
Jun  8 18:26:09 Yggdrasil kernel: xhci_hcd 0000:4e:00.0: hcc params 0x200071e1 hci version 0x100 quirks 0x00000010
Jun  8 18:26:09 Yggdrasil kernel: hub 7-0:1.0: USB hub found
Jun  8 18:26:09 Yggdrasil kernel: hub 7-0:1.0: 4 ports detected
Jun  8 18:26:09 Yggdrasil kernel: xhci_hcd 0000:4e:00.0: xHCI Host Controller
Jun  8 18:26:09 Yggdrasil kernel: xhci_hcd 0000:4e:00.0: new USB bus registered, assigned bus number 8
Jun  8 18:26:09 Yggdrasil kernel: usb usb8: We don't know the algorithms for LPM for this host, disabling LPM.
Jun  8 18:26:09 Yggdrasil kernel: hub 8-0:1.0: USB hub found
Jun  8 18:26:09 Yggdrasil kernel: hub 8-0:1.0: 4 ports detected
Jun  8 18:26:09 Yggdrasil kernel: xhci_hcd 0000:54:00.0: xHCI Host Controller
Jun  8 18:26:09 Yggdrasil kernel: xhci_hcd 0000:54:00.0: new USB bus registered, assigned bus number 9
Jun  8 18:26:09 Yggdrasil kernel: xhci_hcd 0000:54:00.0: hcc params 0x200071e1 hci version 0x100 quirks 0x00000010
Jun  8 18:26:09 Yggdrasil kernel: hub 9-0:1.0: USB hub found
Jun  8 18:26:09 Yggdrasil kernel: hub 9-0:1.0: 4 ports detected
Jun  8 18:26:09 Yggdrasil kernel: xhci_hcd 0000:54:00.0: xHCI Host Controller
Jun  8 18:26:09 Yggdrasil kernel: xhci_hcd 0000:54:00.0: new USB bus registered, assigned bus number 10
Jun  8 18:26:09 Yggdrasil kernel: usb usb10: We don't know the algorithms for LPM for this host, disabling LPM.
Jun  8 18:26:09 Yggdrasil kernel: hub 10-0:1.0: USB hub found
Jun  8 18:26:09 Yggdrasil kernel: hub 10-0:1.0: 4 ports detected
Jun  8 18:26:09 Yggdrasil kernel: usbcore: registered new interface driver usb-storage

 

USB devices seem to work fine in Unraid after this point.  I can even boot from the card (better than my mobo usb, actually).  However, when I launch my VM, I get one more non-fatal error:

Quote

Jun  8 18:26:48 Yggdrasil acpid: input device has been disconnected, fd 10
Jun  8 18:26:48 Yggdrasil kernel: xhci_hcd 0000:4e:00.0: USB bus 7 deregistered
Jun  8 18:26:48 Yggdrasil kernel: br0: port 2(vnet0) entered blocking state
Jun  8 18:26:48 Yggdrasil kernel: br0: port 2(vnet0) entered disabled state
Jun  8 18:26:48 Yggdrasil kernel: device vnet0 entered promiscuous mode
Jun  8 18:26:48 Yggdrasil kernel: br0: port 2(vnet0) entered blocking state
Jun  8 18:26:48 Yggdrasil kernel: br0: port 2(vnet0) entered forwarding state
Jun  8 18:26:52 Yggdrasil kernel: vfio-pci 0000:50:00.0: enabling device (0100 -> 0103)
Jun  8 18:26:52 Yggdrasil kernel: vfio_ecap_init: 0000:50:00.0 hiding ecap 0x19@0x900
Jun  8 18:26:52 Yggdrasil kernel: pmd_set_huge: Cannot satisfy [mem 0x90000000-0x90200000] with a huge-page mapping due to MTRR override.
Jun  8 18:26:52 Yggdrasil kernel: vfio-pci 0000:06:00.0: enabling device (0000 -> 0002)
Jun  8 18:26:52 Yggdrasil kernel: vfio_ecap_init: 0000:06:00.0 hiding ecap 0x19@0x700
Jun  8 18:27:17 Yggdrasil kernel: kvm: zapping shadow pages for mmio generation wraparound
Jun  8 18:27:17 Yggdrasil kernel: kvm: zapping shadow pages for mmio generation wraparound
 

There's a warning in the VM log file as well:

Quote

2017-06-09T01:26:52.163119Z qemu-system-x86_64: -device vfio-pci,host=4e:00.0,id=hostdev3,bus=pci.0,addr=0x9: Failed to mmap 0000:4e:00.0 BAR 2. Performance may be slow

After this point, USB mostly works.  But I get flaky behavior that I think is related to the non-fatal errors.  Some of the flaky behavior results in hard hangs, which is not nice at all!

 

I've been doing the research on how PCI enumeration works so that I can steer things into a more predictable and stable configuration.  This thread is pure gold!

 

--Harper

Link to post
  • 2 weeks later...

I have one USB3 PCIE card and was wondering how to best split the resources between two W10 virtual machines.  I have two USB3 hubs and I would like to assign 1 to each virtual machine.  I was wondering if anyone has successfully passed a USB hub to a virtual machine?

 

Thanks in advance

Edited by csmccarron
Link to post
1 hour ago, csmccarron said:

I have one USB3 PCIE card and was wondering how to best split the resources between two W10 virtual machines.  I have two USB3 hubs and I would like to assign 1 to each virtual machine.  I was wondering if anyone has successfully passed a USB hub to a virtual machine?

 

Thanks in advance

 

You can pass controller but not individual ports/hubs...  

Link to post
2 minutes ago, methanoid said:

 

You can pass controller but not individual ports/hubs...  

 

That's what I was afraid I would be told :(.  I guess I will need to buy another card and riser cable to get around the GPU's.

 

Thanks for the reply

Link to post

On another thread @Siberys had a StarTech PEXUSB3S44V USB3 card that has 4 separate controllers.  If I get one of these cards, in theory I should be able to pass each controller to a separate virtual machine, is that correct?

 

08:00.0 USB controller [0c03]: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller [1912:0015] (rev 02) (A)

09:00.0 USB controller [0c03]: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller [1912:0015] (rev 02) (B)

0a:00.0 USB controller [0c03]: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller [1912:0015] (rev 02) ©

0b:00.0 USB controller [0c03]: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller [1912:0015] (rev 02) (D)

 

/sys/kernel/iommu_groups/42/devices/0000:08:00.0

/sys/kernel/iommu_groups/43/devices/0000:09:00.0

/sys/kernel/iommu_groups/44/devices/0000:0a:00.0

/sys/kernel/iommu_groups/45/devices/0000:0b:00.0

 

UPDATE:  :(

Based on what I have read about UASP (USB Attach SCSI Protocal) this needs to be enabled in the kernel (CONFIG_USB_UAS) which it is not.

Edited by csmccarron
Link to post
8 hours ago, csmccarron said:

On another thread @Siberys had a StarTech PEXUSB3S44V USB3 card that has 4 separate controllers.  If I get one of these cards, in theory I should be able to pass each controller to a separate virtual machine, is that correct?

 

08:00.0 USB controller [0c03]: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller [1912:0015] (rev 02) (A)

09:00.0 USB controller [0c03]: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller [1912:0015] (rev 02) (B)

0a:00.0 USB controller [0c03]: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller [1912:0015] (rev 02) ©

0b:00.0 USB controller [0c03]: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller [1912:0015] (rev 02) (D)

 

/sys/kernel/iommu_groups/42/devices/0000:08:00.0

/sys/kernel/iommu_groups/43/devices/0000:09:00.0

/sys/kernel/iommu_groups/44/devices/0000:0a:00.0

/sys/kernel/iommu_groups/45/devices/0000:0b:00.0

 

UPDATE:  :(

Based on what I have read about UASP (USB Attach SCSI Protocal) this needs to be enabled in the kernel (CONFIG_USB_UAS) which it is not.

 

I don't see how enabling something in the kernel will help passing the card through. When passing a device through to a VM, the VM needs all the drivers for the device. In the host (unraid) the device is bound to the vfio driver.

Link to post
2 hours ago, saarg said:

 

I don't see how enabling something in the kernel will help passing the card through. When passing a device through to a VM, the VM needs all the drivers for the device. In the host (unraid) the device is bound to the vfio driver.

 

True, I have one on the way and have built a custom 6.4 RC6 kernel to test with.  This particular card provides 4 ports and each port is its own controller.  It is perfect for the individual that has multiple virtual machines but only has room for a single USB3 expansion card like me.  This card requires the kernel to have USB attached SCSI Protocal (UASP) enabled to work if you just want to use it on any other Linux operating system.  I am thinking that even though it is bound to the vfio driver, maybe the kernel still need UASP support to communicate to the controller.  It can be compiled as a module so if it is not needed it will not be loaded.  I will report back my findings on the UASP Feature Request thread.

 

Hopefully I will have more information after July 1 when I get back home.

Link to post
  • 4 weeks later...
On 6/24/2017 at 7:55 PM, saarg said:

 

I don't see how enabling something in the kernel will help passing the card through. When passing a device through to a VM, the VM needs all the drivers for the device. In the host (unraid) the device is bound to the vfio driver.

 

I believe users with this specific card (Startech PEXUSB3S44V) have no issues passing them to the machine, but have issues once passed to the machine inside the VM. Mine throws a Code 10 error on the USB hub that the device "cannot start" when passed to a Windows 10 VM using either provided drivers or Microsoft drivers. It also does not function with an Ubuntu 16.04 VM, in my experience, which has native UAS/UASP support if I'm not mistaken.

Either we need some sort of kernel patch to get it to work, or Startech's implementation on this card is subject, which may be the case either way. The odd thing about this card specifically is that it uses the same Renesas µPD720202 controllers that I know work passed to VM in unRAID and ESXI, at least without a PCI Express Switch (PLX) in-between it and the host. This points to a potential botched implementation on Startech's part, unless they use some sort of custom firmware on the controllers themselves.

 

However, I have record of this specific card working as expected in ESXI 6.0 U3 and ESXI 6.5.0d. I believe VMware added support for UAS/UASP standards with a patch in ESXI 5.5 back in 2014, and the Code 10 error in a Windows VM was what you would get before that patch. I know this in no way means unRAID needs explicit UAS/UASP support to get this card to work, as ESXI =/= KVM, but it it worth trying out.

Thank you for the help!

Edited by Prograde
Link to post
  • 1 month later...

I am still having issues getting two add in USB controllers to work with 2 different virtual machines.  So currently here are my two cards:

 

IOMMU group 44 [1b6f:7052] 82:00.0 USB controller: Etron Technology, Inc. EJ188/EJ198 USB 3.0 Host Controller (rev ff)

IOMMU group 45 [1b6f:7052] 84:00.0 USB controller: Etron Technology, Inc. EJ188/EJ198 USB 3.0 Host Controller

 

syslinux.cfg

 vfio-pci.ids=1b21:1242,1b6f:7052,10de:1b81,10de:10f0,10de:13c2,10de:0fbb

 

VM1

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x84' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </hostdev>
 

VM2

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x82' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </hostdev>
 

So basically when I boot 1 VM, the other VM does not work correctly.  Its like the 2 cards are being used by the first virtual machine to boot or the driver is really confused.  So how can I bind vfio-pci to each card separately so both virtual machines can use their assigned USB card without conflicting with each other?  The second VM will boot, it just is either really slow taking input from keyboard or it locks up.

 

Does anyone know how this works?  

 

"A further trick we can use is to craft an ids list using the advanced parsing of PCI vendor and class attributes to create

 an option list that will claim any Nvidia or AMD GPU or audio device:  vfio-pci.ids=1002:ffffffff:ffffffff:ffffffff:00030000:ffff00ff..."

 

Thanks

Link to post
5 hours ago, saarg said:

I use this instead of vfio-pci.ids.

 


xen-pciback.hide=(09:00.0)(00:1d.0)

 


Partial Device listing

IOMMU group 18
	[10de:13c2] 03:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 970] (rev a1)
	[10de:0fbb] 03:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller (rev a1)
IOMMU group 19
	[10b5:8518] 04:00.0 PCI bridge: PLX Technology, Inc. PEX 8518 16-lane, 5-port PCI Express Switch (rev ac)
IOMMU group 20
	[10b5:8518] 05:01.0 PCI bridge: PLX Technology, Inc. PEX 8518 16-lane, 5-port PCI Express Switch (rev ac)
	[8086:10c9] 06:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
	[8086:10c9] 06:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
IOMMU group 21
	[10b5:8518] 05:02.0 PCI bridge: PLX Technology, Inc. PEX 8518 16-lane, 5-port PCI Express Switch (rev ac)
	[8086:10c9] 08:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
	[8086:10c9] 08:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
IOMMU group 22
	[10de:1b81] 0a:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070] (rev a1)
	[10de:10f0] 0a:00.1 Audio device: NVIDIA Corporation GP104 High Definition Audio Controller (rev a1)
IOMMU group 44
	[1b6f:7052] 82:00.0 USB controller: Etron Technology, Inc. EJ188/EJ198 USB 3.0 Host Controller
IOMMU group 45
	[1b6f:7052] 84:00.0 USB controller: Etron Technology, Inc. EJ188/EJ198 USB 3.0 Host Controller

So would I just do this for the USB3 cards only or should I also do it for the GPU's? 

Is this another way to bind vfio-pci driver to passthru devices?

 

Just USB3

xen-pciback.hide=(82:00.0)(84:00.0)

 

Both USB3 and GPUs  

xen-pciback.hide=(82:00.0)(84:00.0)(03:00.0)(03:00.1)(0a:00.0)(oa:00.1)

 

Currently, I am not blacklisting the cards and they are working, but I would rather blacklist them if possible.

 

Edited by csmccarron
Link to post
7 hours ago, saarg said:

I use this instead of vfio-pci.ids.

 


xen-pciback.hide=(09:00.0)(00:1d.0)

 

 

Does this work for KVM in exactly the same way as vfio-pci.ids?

 

I am trying to isolate for VMs 3 USB controllers and a CMI 8738 audio card

 

8086:8d26 USB2 onboard - own IOMMU group
8086:8d2d USB onboard- own IOMMU group
1912:0015 USB3 card- own IOMMU group
 
8086:244e PCI Bridge
1b21:1080 ASM1083 Bridge - all 3 in one group
13f6:0111 CMI8738
 

using:    append isolcpus=4-13,18-27 vfio-pci.ids= 8086:8d26,8086:8d2d,1912:0015,8086:244e,1b21:1080,13f6:0111 initrd=/bzroot

 

But although the USB2 seems to get to the VM (I am typing on it now!) it does not show in the dropdown list of available devices to passthrough for VMs. Of all those devices only the Renesas USB3 card shows??! The two USB2 controllers used to show before I tried 6.4 betas and rebuilt my unRAID USB after it mucked up.

 

What am I doing wrong? Any ideas?

Link to post

xen-pciback.hide= is just another method to hide the devices so it's not used by any other driver, but instead of using the ID it uses the PCI number. This is a better way if you have to devices with the same ID, but only want to hide one of them. 

Using this will not make the device show up in the other devices list as it's not bound to the vfio driver. 

You can probably use both xen-pciback.hide and vfio-pci.ids at the same time. 

 

@methanoid

Looks like you have a space after vfio-pci.ids=

Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.