onboard Firewire 1394 passthrough


Recommended Posts

hi guys

I have the FireWire device on board which I want to passthrough to a VM.

in my case its OSX. I managed to pass through the GTX560Ti, onboard C600/x79 HD audio device, and the USB ports etc  

So its not that I am very knowledgeable but I managed to do the above. 

 

The 1394device is above my "paygrade".

When I start the VM I get the error of :  -device vfio-pci,host=0a:02.0,id=hostdev4,bus=pci.2,addr=0x6: vfio: Error: Failed to setup INTx fd: Device or resource busy

I have isolated the device in the syslinux. its the 104c:8023

" append vfio-pci.ids=1033:0194,1b73:1100,104c:8023 isolcpus=2-13,18-29 initrd=/bzroot "

I see in the /var/log/syslog the following lines:

Mar 20 19:16:09 Tower kernel: vfio-pci 0000:0a:02.0: enabling device (0000 -> 0002)

Mar 20 19:16:09 Tower kernel: genirq: Flags mismatch irq 18. 00000000 (vfio-intx(0000:0a:02.0)) vs. 00000080 (i801_smbus)

What I understand is that my 1394 device uses the same IRQ as the SMBus onboard device.

So at this point I am not really sure what I can do. as the SMbus has a bunch of "friends" with him in the IMMU group, so I cant take the whole group.

PCI Devices

00:00.0 Host bridge [0600]: Intel Corporation Xeon E5/Core i7 DMI2 [8086:3c00] (rev 07)
00:01.0 PCI bridge [0604]: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 1a [8086:3c02] (rev 07)
00:01.1 PCI bridge [0604]: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 1b [8086:3c03] (rev 07)
00:02.0 PCI bridge [0604]: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 2a [8086:3c04] (rev 07)
00:03.0 PCI bridge [0604]: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 3a in PCI Express Mode [8086:3c08] (rev 07)
00:04.0 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 DMA Channel 0 [8086:3c20] (rev 07)
00:04.1 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 DMA Channel 1 [8086:3c21] (rev 07)
00:04.2 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 DMA Channel 2 [8086:3c22] (rev 07)
00:04.3 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 DMA Channel 3 [8086:3c23] (rev 07)
00:04.4 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 DMA Channel 4 [8086:3c24] (rev 07)
00:04.5 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 DMA Channel 5 [8086:3c25] (rev 07)
00:04.6 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 DMA Channel 6 [8086:3c26] (rev 07)
00:04.7 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 DMA Channel 7 [8086:3c27] (rev 07)
00:05.0 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Address Map, VTd_Misc, System Management [8086:3c28] (rev 07)
00:05.2 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Control Status and Global Errors [8086:3c2a] (rev 07)
00:05.4 PIC [0800]: Intel Corporation Xeon E5/Core i7 I/O APIC [8086:3c2c] (rev 07)
00:11.0 PCI bridge [0604]: Intel Corporation C600/X79 series chipset PCI Express Virtual Root Port [8086:1d3e] (rev 06)
00:16.0 Communication controller [0780]: Intel Corporation C600/X79 series chipset MEI Controller #1 [8086:1d3a] (rev 05)
00:16.1 Communication controller [0780]: Intel Corporation C600/X79 series chipset MEI Controller #2 [8086:1d3b] (rev 05)
00:1a.0 USB controller [0c03]: Intel Corporation C600/X79 series chipset USB2 Enhanced Host Controller #2 [8086:1d2d] (rev 06)
00:1b.0 Audio device [0403]: Intel Corporation C600/X79 series chipset High Definition Audio Controller [8086:1d20] (rev 06)
00:1c.0 PCI bridge [0604]: Intel Corporation C600/X79 series chipset PCI Express Root Port 1 [8086:1d10] (rev b6)
00:1c.5 PCI bridge [0604]: Intel Corporation C600/X79 series chipset PCI Express Root Port 6 [8086:1d1a] (rev b6)
00:1c.6 PCI bridge [0604]: Intel Corporation C600/X79 series chipset PCI Express Root Port 7 [8086:1d1c] (rev b6)
00:1d.0 USB controller [0c03]: Intel Corporation C600/X79 series chipset USB2 Enhanced Host Controller #1 [8086:1d26] (rev 06)
00:1e.0 PCI bridge [0604]: Intel Corporation 82801 PCI Bridge [8086:244e] (rev a6)
00:1f.0 ISA bridge [0601]: Intel Corporation C600/X79 series chipset LPC Controller [8086:1d41] (rev 06)
00:1f.2 SATA controller [0106]: Intel Corporation C600/X79 series chipset 6-Port SATA AHCI Controller [8086:1d02] (rev 06)
00:1f.3 SMBus [0c05]: Intel Corporation C600/X79 series chipset SMBus Host Controller [8086:1d22] (rev 06)
00:1f.6 Signal processing controller [1180]: Intel Corporation C600/X79 series chipset Thermal Management Controller [8086:1d24] (rev 06)
04:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480] [1002:67df] (rev c7)
04:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:aaf0]
05:00.0 Serial Attached SCSI controller [0107]: Intel Corporation C602 chipset 4-Port SATA Storage Control Unit [8086:1d6b] (rev 06)
06:00.0 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
06:00.1 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
08:00.0 USB controller [0c03]: NEC Corporation uPD720200 USB 3.0 Host Controller [1033:0194] (rev 04)
09:00.0 USB controller [0c03]: NEC Corporation uPD720200 USB 3.0 Host Controller [1033:0194] (rev 04)
0a:02.0 FireWire (IEEE 1394) [0c00]: Texas Instruments TSB43AB22A IEEE-1394a-2000 Controller (PHY/Link) [iOHCI-Lynx] [104c:8023]
7f:08.0 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 QPI Link 0 [8086:3c80] (rev 07)
7f:08.3 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 QPI Link Reut 0 [8086:3c83] (rev 07)
7f:08.4 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 QPI Link Reut 0 [8086:3c84] (rev 07)
7f:09.0 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 QPI Link 1 [8086:3c90] (rev 07)
7f:09.3 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 QPI Link Reut 1 [8086:3c93] (rev 07)
7f:09.4 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 QPI Link Reut 1 [8086:3c94] (rev 07)
7f:0a.0 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Power Control Unit 0 [8086:3cc0] (rev 07)
7f:0a.1 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Power Control Unit 1 [8086:3cc1] (rev 07)
7f:0a.2 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Power Control Unit 2 [8086:3cc2] (rev 07)
7f:0a.3 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Power Control Unit 3 [8086:3cd0] (rev 07)
7f:0b.0 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Interrupt Control Registers [8086:3ce0] (rev 07)
7f:0b.3 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Semaphore and Scratchpad Configuration Registers [8086:3ce3] (rev 07)
7f:0c.0 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Unicast Register 0 [8086:3ce8] (rev 07)
7f:0c.1 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Unicast Register 0 [8086:3ce8] (rev 07)
7f:0c.2 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Unicast Register 0 [8086:3ce8] (rev 07)
7f:0c.3 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Unicast Register 0 [8086:3ce8] (rev 07)
7f:0c.6 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller System Address Decoder 0 [8086:3cf4] (rev 07)
7f:0c.7 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 System Address Decoder [8086:3cf6] (rev 07)
7f:0d.0 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Unicast Register 0 [8086:3ce8] (rev 07)
7f:0d.1 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Unicast Register 0 [8086:3ce8] (rev 07)
7f:0d.2 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Unicast Register 0 [8086:3ce8] (rev 07)
7f:0d.3 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Unicast Register 0 [8086:3ce8] (rev 07)
7f:0d.6 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller System Address Decoder 1 [8086:3cf5] (rev 07)
7f:0e.0 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Processor Home Agent [8086:3ca0] (rev 07)
7f:0e.1 Performance counters [1101]: Intel Corporation Xeon E5/Core i7 Processor Home Agent Performance Monitoring [8086:3c46] (rev 07)
7f:0f.0 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Registers [8086:3ca8] (rev 07)
7f:0f.1 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller RAS Registers [8086:3c71] (rev 07)
7f:0f.2 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 0 [8086:3caa] (rev 07)
7f:0f.3 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 1 [8086:3cab] (rev 07)
7f:0f.4 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 2 [8086:3cac] (rev 07)
7f:0f.5 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 3 [8086:3cad] (rev 07)
7f:0f.6 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 4 [8086:3cae] (rev 07)
7f:10.0 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 0 [8086:3cb0] (rev 07)
7f:10.1 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 1 [8086:3cb1] (rev 07)
7f:10.2 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 0 [8086:3cb2] (rev 07)
7f:10.3 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 1 [8086:3cb3] (rev 07)
7f:10.4 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 2 [8086:3cb4] (rev 07)
7f:10.5 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 3 [8086:3cb5] (rev 07)
7f:10.6 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 2 [8086:3cb6] (rev 07)
7f:10.7 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 3 [8086:3cb7] (rev 07)
7f:11.0 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 DDRIO [8086:3cb8] (rev 07)
7f:13.0 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 R2PCIe [8086:3ce4] (rev 07)
7f:13.1 Performance counters [1101]: Intel Corporation Xeon E5/Core i7 Ring to PCI Express Performance Monitor [8086:3c43] (rev 07)
7f:13.4 Performance counters [1101]: Intel Corporation Xeon E5/Core i7 QuickPath Interconnect Agent Ring Registers [8086:3ce6] (rev 07)
7f:13.5 Performance counters [1101]: Intel Corporation Xeon E5/Core i7 Ring to QuickPath Interconnect Link 0 Performance Monitor [8086:3c44] (rev 07)
7f:13.6 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Ring to QuickPath Interconnect Link 1 Performance Monitor [8086:3c45] (rev 07)
80:01.0 PCI bridge [0604]: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 1a [8086:3c02] (rev 07)
80:02.0 PCI bridge [0604]: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 2a [8086:3c04] (rev 07)
80:03.0 PCI bridge [0604]: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 3a in PCI Express Mode [8086:3c08] (rev 07)
80:04.0 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 DMA Channel 0 [8086:3c20] (rev 07)
80:04.1 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 DMA Channel 1 [8086:3c21] (rev 07)
80:04.2 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 DMA Channel 2 [8086:3c22] (rev 07)
80:04.3 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 DMA Channel 3 [8086:3c23] (rev 07)
80:04.4 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 DMA Channel 4 [8086:3c24] (rev 07)
80:04.5 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 DMA Channel 5 [8086:3c25] (rev 07)
80:04.6 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 DMA Channel 6 [8086:3c26] (rev 07)
80:04.7 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 DMA Channel 7 [8086:3c27] (rev 07)
80:05.0 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Address Map, VTd_Misc, System Management [8086:3c28] (rev 07)
80:05.2 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Control Status and Global Errors [8086:3c2a] (rev 07)
80:05.4 PIC [0800]: Intel Corporation Xeon E5/Core i7 I/O APIC [8086:3c2c] (rev 07)
82:00.0 VGA compatible controller [0300]: NVIDIA Corporation GF114 [GeForce GTX 560 Ti] [10de:1200] (rev a1)
82:00.1 Audio device [0403]: NVIDIA Corporation GF114 HDMI Audio Controller [10de:0e0c] (rev a1)
ff:08.0 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 QPI Link 0 [8086:3c80] (rev 07)
ff:08.3 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 QPI Link Reut 0 [8086:3c83] (rev 07)
ff:08.4 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 QPI Link Reut 0 [8086:3c84] (rev 07)
ff:09.0 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 QPI Link 1 [8086:3c90] (rev 07)
ff:09.3 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 QPI Link Reut 1 [8086:3c93] (rev 07)
ff:09.4 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 QPI Link Reut 1 [8086:3c94] (rev 07)
ff:0a.0 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Power Control Unit 0 [8086:3cc0] (rev 07)
ff:0a.1 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Power Control Unit 1 [8086:3cc1] (rev 07)
ff:0a.2 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Power Control Unit 2 [8086:3cc2] (rev 07)
ff:0a.3 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Power Control Unit 3 [8086:3cd0] (rev 07)
ff:0b.0 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Interrupt Control Registers [8086:3ce0] (rev 07)
ff:0b.3 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Semaphore and Scratchpad Configuration Registers [8086:3ce3] (rev 07)
ff:0c.0 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Unicast Register 0 [8086:3ce8] (rev 07)
ff:0c.1 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Unicast Register 0 [8086:3ce8] (rev 07)
ff:0c.2 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Unicast Register 0 [8086:3ce8] (rev 07)
ff:0c.3 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Unicast Register 0 [8086:3ce8] (rev 07)
ff:0c.6 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller System Address Decoder 0 [8086:3cf4] (rev 07)
ff:0c.7 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 System Address Decoder [8086:3cf6] (rev 07)
ff:0d.0 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Unicast Register 0 [8086:3ce8] (rev 07)
ff:0d.1 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Unicast Register 0 [8086:3ce8] (rev 07)
ff:0d.2 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Unicast Register 0 [8086:3ce8] (rev 07)
ff:0d.3 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Unicast Register 0 [8086:3ce8] (rev 07)
ff:0d.6 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller System Address Decoder 1 [8086:3cf5] (rev 07)
ff:0e.0 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Processor Home Agent [8086:3ca0] (rev 07)
ff:0e.1 Performance counters [1101]: Intel Corporation Xeon E5/Core i7 Processor Home Agent Performance Monitoring [8086:3c46] (rev 07)
ff:0f.0 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Registers [8086:3ca8] (rev 07)
ff:0f.1 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller RAS Registers [8086:3c71] (rev 07)
ff:0f.2 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 0 [8086:3caa] (rev 07)
ff:0f.3 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 1 [8086:3cab] (rev 07)
ff:0f.4 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 2 [8086:3cac] (rev 07)
ff:0f.5 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 3 [8086:3cad] (rev 07)
ff:0f.6 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 4 [8086:3cae] (rev 07)
ff:10.0 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 0 [8086:3cb0] (rev 07)
ff:10.1 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 1 [8086:3cb1] (rev 07)
ff:10.2 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 0 [8086:3cb2] (rev 07)
ff:10.3 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 1 [8086:3cb3] (rev 07)
ff:10.4 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 2 [8086:3cb4] (rev 07)
ff:10.5 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 3 [8086:3cb5] (rev 07)
ff:10.6 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 2 [8086:3cb6] (rev 07)
ff:10.7 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 3 [8086:3cb7] (rev 07)
ff:11.0 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 DDRIO [8086:3cb8] (rev 07)
ff:13.0 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 R2PCIe [8086:3ce4] (rev 07)
ff:13.1 Performance counters [1101]: Intel Corporation Xeon E5/Core i7 Ring to PCI Express Performance Monitor [8086:3c43] (rev 07)
ff:13.4 Performance counters [1101]: Intel Corporation Xeon E5/Core i7 QuickPath Interconnect Agent Ring Registers [8086:3ce6] (rev 07)
ff:13.5 Performance counters [1101]: Intel Corporation Xeon E5/Core i7 Ring to QuickPath Interconnect Link 0 Performance Monitor [8086:3c44] (rev 07)
ff:13.6 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Ring to QuickPath Interconnect Link 1 Performance Monitor [8086:3c45] (rev 07)
IOMMU Groups

/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/1/devices/0000:00:01.0
/sys/kernel/iommu_groups/2/devices/0000:00:01.1
/sys/kernel/iommu_groups/3/devices/0000:00:02.0
/sys/kernel/iommu_groups/4/devices/0000:00:03.0
/sys/kernel/iommu_groups/5/devices/0000:00:04.0
/sys/kernel/iommu_groups/5/devices/0000:00:04.1
/sys/kernel/iommu_groups/5/devices/0000:00:04.2
/sys/kernel/iommu_groups/5/devices/0000:00:04.3
/sys/kernel/iommu_groups/5/devices/0000:00:04.4
/sys/kernel/iommu_groups/5/devices/0000:00:04.5
/sys/kernel/iommu_groups/5/devices/0000:00:04.6
/sys/kernel/iommu_groups/5/devices/0000:00:04.7
/sys/kernel/iommu_groups/6/devices/0000:00:05.0
/sys/kernel/iommu_groups/6/devices/0000:00:05.2
/sys/kernel/iommu_groups/6/devices/0000:00:05.4
/sys/kernel/iommu_groups/7/devices/0000:00:11.0
/sys/kernel/iommu_groups/8/devices/0000:00:16.0
/sys/kernel/iommu_groups/8/devices/0000:00:16.1
/sys/kernel/iommu_groups/9/devices/0000:00:1a.0
/sys/kernel/iommu_groups/10/devices/0000:00:1b.0
/sys/kernel/iommu_groups/11/devices/0000:00:1c.0
/sys/kernel/iommu_groups/12/devices/0000:00:1c.5
/sys/kernel/iommu_groups/13/devices/0000:00:1c.6
/sys/kernel/iommu_groups/14/devices/0000:00:1d.0
/sys/kernel/iommu_groups/15/devices/0000:00:1e.0
/sys/kernel/iommu_groups/15/devices/0000:0a:02.0
/sys/kernel/iommu_groups/16/devices/0000:00:1f.0
/sys/kernel/iommu_groups/16/devices/0000:00:1f.2
/sys/kernel/iommu_groups/16/devices/0000:00:1f.3
/sys/kernel/iommu_groups/16/devices/0000:00:1f.6
/sys/kernel/iommu_groups/17/devices/0000:04:00.0
/sys/kernel/iommu_groups/17/devices/0000:04:00.1
/sys/kernel/iommu_groups/18/devices/0000:05:00.0
/sys/kernel/iommu_groups/19/devices/0000:06:00.0
/sys/kernel/iommu_groups/20/devices/0000:06:00.1
/sys/kernel/iommu_groups/21/devices/0000:08:00.0
/sys/kernel/iommu_groups/22/devices/0000:09:00.0
/sys/kernel/iommu_groups/23/devices/0000:7f:08.0
/sys/kernel/iommu_groups/23/devices/0000:7f:08.3
/sys/kernel/iommu_groups/23/devices/0000:7f:08.4
/sys/kernel/iommu_groups/24/devices/0000:7f:09.0
/sys/kernel/iommu_groups/24/devices/0000:7f:09.3
/sys/kernel/iommu_groups/24/devices/0000:7f:09.4
/sys/kernel/iommu_groups/25/devices/0000:7f:0a.0
/sys/kernel/iommu_groups/25/devices/0000:7f:0a.1
/sys/kernel/iommu_groups/25/devices/0000:7f:0a.2
/sys/kernel/iommu_groups/25/devices/0000:7f:0a.3
/sys/kernel/iommu_groups/26/devices/0000:7f:0b.0
/sys/kernel/iommu_groups/26/devices/0000:7f:0b.3
/sys/kernel/iommu_groups/27/devices/0000:7f:0c.0
/sys/kernel/iommu_groups/27/devices/0000:7f:0c.1
/sys/kernel/iommu_groups/27/devices/0000:7f:0c.2
/sys/kernel/iommu_groups/27/devices/0000:7f:0c.3
/sys/kernel/iommu_groups/27/devices/0000:7f:0c.6
/sys/kernel/iommu_groups/27/devices/0000:7f:0c.7
/sys/kernel/iommu_groups/28/devices/0000:7f:0d.0
/sys/kernel/iommu_groups/28/devices/0000:7f:0d.1
/sys/kernel/iommu_groups/28/devices/0000:7f:0d.2
/sys/kernel/iommu_groups/28/devices/0000:7f:0d.3
/sys/kernel/iommu_groups/28/devices/0000:7f:0d.6
/sys/kernel/iommu_groups/29/devices/0000:7f:0e.0
/sys/kernel/iommu_groups/29/devices/0000:7f:0e.1
/sys/kernel/iommu_groups/30/devices/0000:7f:0f.0
/sys/kernel/iommu_groups/30/devices/0000:7f:0f.1
/sys/kernel/iommu_groups/30/devices/0000:7f:0f.2
/sys/kernel/iommu_groups/30/devices/0000:7f:0f.3
/sys/kernel/iommu_groups/30/devices/0000:7f:0f.4
/sys/kernel/iommu_groups/30/devices/0000:7f:0f.5
/sys/kernel/iommu_groups/30/devices/0000:7f:0f.6
/sys/kernel/iommu_groups/31/devices/0000:7f:10.0
/sys/kernel/iommu_groups/31/devices/0000:7f:10.1
/sys/kernel/iommu_groups/31/devices/0000:7f:10.2
/sys/kernel/iommu_groups/31/devices/0000:7f:10.3
/sys/kernel/iommu_groups/31/devices/0000:7f:10.4
/sys/kernel/iommu_groups/31/devices/0000:7f:10.5
/sys/kernel/iommu_groups/31/devices/0000:7f:10.6
/sys/kernel/iommu_groups/31/devices/0000:7f:10.7
/sys/kernel/iommu_groups/32/devices/0000:7f:11.0
/sys/kernel/iommu_groups/33/devices/0000:7f:13.0
/sys/kernel/iommu_groups/33/devices/0000:7f:13.1
/sys/kernel/iommu_groups/33/devices/0000:7f:13.4
/sys/kernel/iommu_groups/33/devices/0000:7f:13.5
/sys/kernel/iommu_groups/33/devices/0000:7f:13.6
/sys/kernel/iommu_groups/34/devices/0000:80:01.0
/sys/kernel/iommu_groups/35/devices/0000:80:02.0
/sys/kernel/iommu_groups/36/devices/0000:80:03.0
/sys/kernel/iommu_groups/37/devices/0000:80:04.0
/sys/kernel/iommu_groups/37/devices/0000:80:04.1
/sys/kernel/iommu_groups/37/devices/0000:80:04.2
/sys/kernel/iommu_groups/37/devices/0000:80:04.3
/sys/kernel/iommu_groups/37/devices/0000:80:04.4
/sys/kernel/iommu_groups/37/devices/0000:80:04.5
/sys/kernel/iommu_groups/37/devices/0000:80:04.6
/sys/kernel/iommu_groups/37/devices/0000:80:04.7
/sys/kernel/iommu_groups/38/devices/0000:80:05.0
/sys/kernel/iommu_groups/38/devices/0000:80:05.2
/sys/kernel/iommu_groups/38/devices/0000:80:05.4
/sys/kernel/iommu_groups/39/devices/0000:82:00.0
/sys/kernel/iommu_groups/39/devices/0000:82:00.1
/sys/kernel/iommu_groups/40/devices/0000:ff:08.0
/sys/kernel/iommu_groups/40/devices/0000:ff:08.3
/sys/kernel/iommu_groups/40/devices/0000:ff:08.4
/sys/kernel/iommu_groups/41/devices/0000:ff:09.0
/sys/kernel/iommu_groups/41/devices/0000:ff:09.3
/sys/kernel/iommu_groups/41/devices/0000:ff:09.4
/sys/kernel/iommu_groups/42/devices/0000:ff:0a.0
/sys/kernel/iommu_groups/42/devices/0000:ff:0a.1
/sys/kernel/iommu_groups/42/devices/0000:ff:0a.2
/sys/kernel/iommu_groups/42/devices/0000:ff:0a.3
/sys/kernel/iommu_groups/43/devices/0000:ff:0b.0
/sys/kernel/iommu_groups/43/devices/0000:ff:0b.3
/sys/kernel/iommu_groups/44/devices/0000:ff:0c.0
/sys/kernel/iommu_groups/44/devices/0000:ff:0c.1
/sys/kernel/iommu_groups/44/devices/0000:ff:0c.2
/sys/kernel/iommu_groups/44/devices/0000:ff:0c.3
/sys/kernel/iommu_groups/44/devices/0000:ff:0c.6
/sys/kernel/iommu_groups/44/devices/0000:ff:0c.7
/sys/kernel/iommu_groups/45/devices/0000:ff:0d.0
/sys/kernel/iommu_groups/45/devices/0000:ff:0d.1
/sys/kernel/iommu_groups/45/devices/0000:ff:0d.2
/sys/kernel/iommu_groups/45/devices/0000:ff:0d.3
/sys/kernel/iommu_groups/45/devices/0000:ff:0d.6
/sys/kernel/iommu_groups/46/devices/0000:ff:0e.0
/sys/kernel/iommu_groups/46/devices/0000:ff:0e.1
/sys/kernel/iommu_groups/47/devices/0000:ff:0f.0
/sys/kernel/iommu_groups/47/devices/0000:ff:0f.1
/sys/kernel/iommu_groups/47/devices/0000:ff:0f.2
/sys/kernel/iommu_groups/47/devices/0000:ff:0f.3
/sys/kernel/iommu_groups/47/devices/0000:ff:0f.4
/sys/kernel/iommu_groups/47/devices/0000:ff:0f.5
/sys/kernel/iommu_groups/47/devices/0000:ff:0f.6
/sys/kernel/iommu_groups/48/devices/0000:ff:10.0
/sys/kernel/iommu_groups/48/devices/0000:ff:10.1
/sys/kernel/iommu_groups/48/devices/0000:ff:10.2
/sys/kernel/iommu_groups/48/devices/0000:ff:10.3
/sys/kernel/iommu_groups/48/devices/0000:ff:10.4
/sys/kernel/iommu_groups/48/devices/0000:ff:10.5
/sys/kernel/iommu_groups/48/devices/0000:ff:10.6
/sys/kernel/iommu_groups/48/devices/0000:ff:10.7
/sys/kernel/iommu_groups/49/devices/0000:ff:11.0
/sys/kernel/iommu_groups/50/devices/0000:ff:13.0
/sys/kernel/iommu_groups/50/devices/0000:ff:13.1
/sys/kernel/iommu_groups/50/devices/0000:ff:13.4
/sys/kernel/iommu_groups/50/devices/0000:ff:13.5
/sys/kernel/iommu_groups/50/devices/0000:ff:13.6
CPU Thread Pairings

cpu 0 <===> cpu 16
cpu 1 <===> cpu 17
cpu 2 <===> cpu 18
cpu 3 <===> cpu 19
cpu 4 <===> cpu 20
cpu 5 <===> cpu 21
cpu 6 <===> cpu 22
cpu 7 <===> cpu 23
cpu 8 <===> cpu 24
cpu 9 <===> cpu 25
cpu 10 <===> cpu 26
cpu 11 <===> cpu 27
cpu 12 <===> cpu 28
cpu 13 <===> cpu 29
cpu 14 <===> cpu 30
cpu 15 <===> cpu 31
USB Devices

Bus 002 Device 005: ID 090c:1000 Silicon Motion, Inc. - Taiwan (formerly Feiya Technology Corp.) Flash Drive
Bus 002 Device 004: ID 0a5c:21e8 Broadcom Corp. BCM20702A0 Bluetooth 4.0
Bus 002 Device 003: ID 03f0:8c11 Hewlett-Packard Deskjet F4500 series
Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
SCSI Devices

[0:0:0:0]    disk    Storage  Xtreamer         0001  /dev/sda   8.09GB
[1:0:0:0]    disk    ATA      WDC WD10EAVS-00D 1A01  /dev/sde   1.00TB
[1:0:1:0]    disk    ATA      WDC WD40EFRX-68N 0A82  /dev/sdf   4.00TB
[1:0:2:0]    disk    ATA      WDC WD40EFRX-68N 0A82  /dev/sdg   4.00TB
[2:0:0:0]    disk    ATA      Samsung SSD 750  1B6Q  /dev/sdb    500GB
[3:0:0:0]    disk    ATA      WDC WD20EZRX-00D 0A80  /dev/sdc   2.00TB
[6:0:0:0]    disk    ATA      WDC WD20EZRX-00D 0A80  /dev/sdd   2.00TB

 

Link to comment

try adding this to your syslinux.cfg instead: pcie_acs_override=id:104c:8023

 

it will attempt to remove it from the IOMMU grouping and isolate it.

 

It will probably no longer be stubbed so you may have to initially edit the xml manually to add it. (or if you're smart, copy the xml with the device now before you reboot.) - actually, it "might" retain it, but just make a copy of the xml for now incase you need it.

 

I don't know if this will fix your problem, but it may solve 1 of the issues of device isolation.

Link to comment

unfortunately the above didnt work. it gave the same error and the device was not separated from the IOMMU group 15.

I tried multiple variations of pcie_acs_override. for the device for the other pci bridge its with in the group and even both.  probably it was stupid as the pci bridge is used by something (dont know who..).

so I am lost.

I though about adding a separate PCI board for the Firewire, but its full PCI16 and I dont have free slots as all used by GPUs.

 

anyone can think about any other idea ?

 

Link to comment

The device i am trying to pass is the 1394

a:02.0 FireWire (IEEE 1394) [0c00]: Texas Instruments TSB43AB22A IEEE-1394a-2000 Controller (PHY/Link) [iOHCI-Lynx] [104c:8023]

 

I see in the /var/log/syslog :

 "Tower kernel: genirq: Flags mismatch irq 18. 00000000 (vfio-intx(0000:0a:02.0)) vs. 00000080 (i801_smbus)"

 

should I understand that indeed the pc bridge in the IOMMU group is the "culprit" ?

 the supermicro motherboard puts the FireWire (IEEE 1394)  behind some pci-to-pci bridge ?

and the path to solution is to try to separate the devices into individual IOMMU groups ?

this is the only path? 

what does the above "00000080 (i801_smbus" mean ?   

 

when I am trying to passthrough the bridge as well,  the VM shows error that the device 00:1e.0  does not exist.

how to make it recognized and maybe to passthrough both devices ?

 

stubing  (pci-stub.ids=104c:8023,8086:244e) didnt result in anything as well.

 

I am trying to learn from your answers guys

 

digging around I found various commands that I am not clear how to use in the syslinux

Maybe some combination of them may help.

vfio_iommu_type1.allow_unsafe_interrupts=1

iommu=pt

intel_iommu=on

intel_pstate=disable

 

 

 

 

Edited by dadarara
Link to comment

I'v been working with another member on the forum with a similar issue where the device they want is in an iommu group with a pci bridge. so far no good results.

 

 

post your current syslinux.cfg and we can try a couple different modifications. Looking around on the web i haven't found a working

solution yet.

 

---edit

 

actually just post your whole diagnostics zip

Edited by 1812
Link to comment

Try the following in syslinux.cfg

 

intel_iommu=on vfio-pci.ids=1033:0194,1b73:1100,104c:8023,8086:244e vfio_iommu_type1.allow_unsafe_interrupts=1  isolcpus=2-13,18-29

 

your iommu groupings are on, and active, but this loads them first, according to some reading I was doing. once booted, look at your iommu groupings and see if the firewire and pci bridge are now separated.

 

If it shows a different error when trying to boot the vm than first reported, please report that.

 

if that doesn't work try this variation:

 

intel_iommu=on pcie_acs_override=1033:0194,1b73:1100,104c:8023,8086:244e vfio_iommu_type1.allow_unsafe_interrupts=1  isolcpus=2-13,18-29

 

once booted, look at your iommu groupings and see if the firewire and pci bridge are now separated.

 

If it shows a different error when trying to boot the vm than first reported, please report that.

 

 

in either case, you should only be passing the firewire device and not the pci bridge to the vm.

 

Link to comment

no luck.  the IOMMU group 15 is not separated.

 

option1:

intel_iommu=on vfio-pci.ids=1033:0194,1b73:1100,104c:8023,8086:244e vfio_iommu_type1.allow_unsafe_interrupts=1  isolcpus=2-13,18-29

Execution error

internal error: early end of file from monitor, possible problem: 2017-03-29T07:12:44.125465Z qemu-system-x86_64: -device vfio-pci,host=0a:02.0,id=hostdev1,bus=pci.0,addr=0x9: vfio: Error: Failed to setup INTx fd: Device or resource busy
2017-03-29T07:12:44.126111Z qemu-system-x86_64: -device vfio-pci,host=0a:02.0,id=hostdev1,bus=pci.0,addr=0x9: Device initialization failed

IOMMU groups:

/sys/kernel/iommu_groups/15/devices/0000:00:1e.0

/sys/kernel/iommu_groups/15/devices/0000:0a:02.0

 

Option2:

intel_iommu=on pcie_acs_override=1033:0194,1b73:1100,104c:8023,8086:244e vfio_iommu_type1.allow_unsafe_interrupts=1  isolcpus=2-13,18-29

 

Execution error

internal error: process exited while connecting to monitor: 2017-03-29T08:07:07.008431Z qemu-system-x86_64: -device vfio-pci,host=0a:02.0,id=hostdev1,bus=pci.0,addr=0x9: vfio: Error: Failed to setup INTx fd: Device or resource busy
2017-03-29T08:07:07.009009Z qemu-system-x86_64: -device vfio-pci,host=0a:02.0,id=hostdev1,bus=pci.0,addr=0x9: Device initialization failed

IOMMU groups:

sys/kernel/iommu_groups/15/devices/0000:00:1e.0

/sys/kernel/iommu_groups/15/devices/0000:0a:02.0

Link to comment
17 minutes ago, dadarara said:

I found this related topic.  https://lists.gt.net/xen/devel/320085

I think something similar is happening with me .

When I attached the pci bridge together with the firewire 1392a device .  I get an error saying  that the  00:1e.0 PCI bridge device does not exist.

but I do see it in the device list.

can something similar to the patch discussed in the post, be done in unRAID ?


that's a good question...for someone else. I don't patch or compile anything for unRaid on my servers. You might try contacting one of the LT admins on here or search to see if others have done it? or if you're handy and knowledgeable, try it yourself?

Link to comment
  • 2 weeks later...

What would be the possible feature / software change that would allow to pass through the device behind a pci-2-pci bridge

and that is refuses to be separated into individual IOMMU group

 

I dont know how to formulate the feature request for such a situation.

help please 

Link to comment

for sure the existence of a PCI bridge hiding the device I wish to passthrough is the specifics of the MOBO.

the situation such as I have exists in many motherboards.  its just that the common issues people have are with GPUs , USB, Network.  These devices are not hiding behind a dedicated PCI bridge chip.   the Firewire is one of the unlucky features that utilize a chip that does not have PCI bus, and thus need to be coupled with PCI-2-PCI bridge.

 

I am looking for a workaround to handle such a situation with the IOMMU setup. 

I have seen some solution  https://lists.gt.net/xen/devel/320085 

what I am looking for is for some official built-in change in the unRAID distribution that will handle such situations.

 

I just dont know enough technical terms to explain what needs to be done.

Link to comment

what can I say people...   I probably didnt do the right things , but now I have the firewire passed through and working.

(apart from the fact that my camera's dv firewire output is faulty.  

 

anyways my  SysLinux.cfg looks like this now:

 

 

default /syslinux/menu.c32

menu title Lime Technology, Inc.

prompt 0

timeout 50

label unRAID OS

  menu default

  kernel /bzimage

  append intel_iommu=on pcie_acs_override=1033:0194,1b73:1100 vfio_iommu_type1.allow_unsafe_interrupts=1  isolcpus=2-13,18-29 vfio-pci.ids=8086:244e modprobe.blacklist=i2c_i801,i2c_smbus initrd=/bzroot

label unRAID OS GUI Mode

  kernel /bzimage

  append initrd=/bzroot /bzroot-gui

label unRAID OS Safe Mode (no plugins, no GUI)

  kernel /bzimage

  append initrd=/bzroot unraidsafemode

label unRAID OS GUI Safe Mode (no plugins)

  kernel /bzimage

  append initrd=/bzroot,/bzroot-gui unraidsafemode

label Memtest86+

  kernel /memtest

Edited by dadarara
Solved
Link to comment

Hi guys

I am doing trial and error here. gathering stuff from other people's successes.

I am not really clearly understand it all. and might be I am explaining this wrongly but lets try for the sake of other people:

 

1) pcie_acs_override=1033:0194,1b73:1100    : ACS override patches, thats for the 2 identical USB hubs I have. (and the USB hub PCI extension card for which I am not sure I need to do it though)  it supposed to override the IOMMU assignments and separate them into two distinct groups. you need to specify the device id. Again, not sure this is the exact explanation, but it works :)

IOMMU group 21
	[1033:0194] 08:00.0 USB controller: NEC Corporation uPD720200 USB 3.0 Host Controller (rev 04)
IOMMU group 22
	[1033:0194] 09:00.0 USB controller: NEC Corporation uPD720200 USB 3.0 Host Controller (rev 04)

2) vfio_iommu_type1.allow_unsafe_interrupts=1 :  REALLY dont understand what exactly it does. I do know what interrupt is, I dont know what is unsafe one. seen it a lot so decided to use it as well...

 

3) isolcpus=2-13,18-29    thats for the isolation of the CPU cores I want to use for the unraid, dockers and stuff

 

4) vfio-pci.ids=8086:244e    and the     modprobe.blacklist=i2c_i801,i2c_smbus

here it starts to be tricky .    at first I was trying to assign both the PCI bridge and the Firewire device to the vfio , but after some trial left only the bridge in the command.

 

regarding the blacklisting,

I saw in the /var/log/syslog the following lines: 

Mar 20 19:16:09 Tower kernel: vfio-pci 0000:0a:02.0: enabling device (0000 -> 0002)

Mar 20 19:16:09 Tower kernel: genirq: Flags mismatch irq 18. 00000000 (vfio-intx(0000:0a:02.0)) vs. 00000080 (i801_smbus)

 

 

 I got the idea that the smbus of my motherboard gets in the way of the IRQ assignments and I have stumbled upon some entry that was dealing with the exact description of my smbus.  I didnt know how to define the smbus in the blacklisting until I saw that post (somewhere...)

 

another step was to ONLY pass through the firewire device and NOT the PCI bridge.  passing both I get the error that the bridge couldn't be reseted. and indeed it does not have this feature.   I would really like someone to confirm that I understood the meaning of  ">Reset- "  as CAN NOT accept reset.  or I am totally wrong here ... :)  see bellow.

Also, I didnt understand, why the system couldnt send a reset to the whole PCI bridge and everything behind it. but it doesnt matter now, as it works.

 

root@Tower:~# lspci -vvvs 0:1e.0

00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a6) (prog-if 01 [Subtractive decode])

Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-

Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

Latency: 0

NUMA node: 0

Bus: primary=00, secondary=0a, subordinate=0a, sec-latency=64

I/O behind bridge: 0000f000-00000fff

Memory behind bridge: df900000-df9fffff

Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff

Secondary status: 66MHz- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ <SERR- <PERR-

BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-

PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-

Capabilities: [50] Subsystem: Super Micro Computer Inc 82801 PCI Bridge

 

so now the FireWire works in the windows VM and I can capture the DV camera.  

(you will not FU#$@#ing believe what I was trying to do to make it work.  as I have 3 video cards in my rig and I have some physical problem to put an addon card, I had to make it work.

so I tryed to install TVheadend and SageTV dockers and make them work with the FW0/FW1 devices the unraid recognized the firewire and the camera. and install some dvgrab utility to grab the video.  I almost succeeded. I could control the camera but no capture of video stream.  BUT what the hell. I learned a lot.  and now it works

 

I want to thank all the guys here in the forum for the help. this unRAID is really something.  I have now replaced 5 computers. my son is really happy with his Gaming capabilities. fast games and excellent video quality. and the whole rig takes 200-250W . this was almost the power usage of only one computer before.

 

 

2017-03-01 21.14.30-resized.jpg

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.