***GUIDE*** Passthrough Entire PCI USB Controller


archedraft

511 posts in this topic Last Reply

Recommended Posts

8 hours ago, darrenyorston said:

I don't know.

 

"pci-stub .ids=8086:1521" is my 4 port gigabit ethernet NIC. It seems to work with that space so I have no idea as to whether it should have it or not.

 

Looking over other usage of pci-stub on these forums it always seems to have a space between the b and the period mark.

 

I have no idea how you add multiple pci devices in the syslinux config so I preumed you just put multiple entries.

 

 

 

 

 

Also, you are using 2 different ways of "stubbing the devices"

The new way is using vfio-pci and the older way using pci-stub

Just use  vfio-pci

For more than one device separate with a comma

ie

append  vfio-pci.ids=1b73:1100,8086:15a1 initrd=/bzroot  

 

Link to post
  • Replies 510
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Popular Posts

Intent: This guide will take you through the process of passing through an entire PCI USB Controller instead of doing it individually. One benefit is that USB will be plug n play on your virtual mac

I automated steps 1-9 of this guide.   The output shows USB bus #, PCI Address, and IOMMU group #   Bus 1 PCI 0000:00:1a.0 IOMMU 20     Intel Corp.     Linux Foundation 2.0 root hub Bus 2

I believe users with this specific card (Startech PEXUSB3S44V) have no issues passing them to the machine, but have issues once passed to the machine inside the VM. Mine throws a Code 10 error on the

Posted Images

5 hours ago, gridrunner said:

 

Also, you are using 2 different ways of "stubbing the devices"

The new way is using vfio-pci and the older way using pci-stub

Just use  vfio-pci

For more than one device separate with a comma

ie


append  vfio-pci.ids=1b73:1100,8086:15a1 initrd=/bzroot  

 

Many thanks mate!

Link to post

I have adjusted my syslinux config to read as follows:

 

"default /syslinux/menu.c32
menu title Lime Technology, Inc.
prompt 0
timeout 50
label unRAID OS
  menu default
  kernel /bzimage
  append vfio-pci.ids=8086:1521,1106:3483 initrd=/bzroot
label unRAID OS GUI Mode
  kernel /bzimage
  append initrd=/bzroot,/bzroot-gui
label unRAID OS Safe Mode (no plugins, no GUI)
  kernel /bzimage
  append initrd=/bzroot unraidsafemode
label unRAID OS GUI Safe Mode (no plugins)
  kernel /bzimage
  append initrd=/bzroot,/bzroot-gui unraidsafemode
label Memtest86+
  kernel /memtest"

 

When I try to start the VM with the the USB controller selected in "Other PCI devices" I still receive the following error"

 

"internal error: qemu unexpectedly closed the monitor: 2017-04-18T01:05:05.332604Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,id=hostdev1,bus=pci.0,addr=0x6: vfio: error, group 1 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver.
2017-04-18T01:05:05.332636Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,id=hostdev1,bus=pci.0,addr=0x6: vfio: failed to get group 1
2017-04-18T01:05:05.332647Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,id=hostdev1,bus=pci.0,addr=0x6: Device initialization failed"

 

The PCI card I added is in IOMMU group 1, along with the PCI bridge. I have tried a different card and it also gets put into IOMMU group one.

 

In system devices my PCI Devices and IOMMU groups are:

 

"

IOMMU group 0
	[8086:0c08] 00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v3 Processor DRAM Controller (rev 06)
IOMMU group 1
	[8086:0c01] 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06)
	[8086:0c05] 00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x8 Controller (rev 06)
	[1106:3483] 01:00.0 USB controller: VIA Technologies, Inc. VL805 USB 3.0 Host Controller (rev 01)
	[1000:0086] 02:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2308 PCI-Express Fusion-MPT SAS-2 (rev 05)
IOMMU group 2
	[8086:8c31] 00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 05)
IOMMU group 3
	[8086:8c2d] 00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 (rev 05)
IOMMU group 4
	[8086:8c10] 00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d5)
IOMMU group 5
	[8086:8c14] 00:1c.2 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #3 (rev d5)
IOMMU group 6
	[8086:8c16] 00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 (rev d5)
IOMMU group 7
	[8086:8c18] 00:1c.4 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #5 (rev d5)
IOMMU group 8
	[8086:8c26] 00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 05)
IOMMU group 9
	[8086:8c52] 00:1f.0 ISA bridge: Intel Corporation C222 Series Chipset Family Server Essential SKU LPC Controller (rev 05)
	[8086:8c02] 00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05)
	[8086:8c22] 00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 05)
	[8086:8c24] 00:1f.6 Signal processing controller: Intel Corporation 8 Series Chipset Family Thermal Management Controller (rev 05)
IOMMU group 10
	[1a03:1150] 03:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 03)
	[1a03:2000] 04:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 30)
IOMMU group 11
	[8086:1533] 05:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
IOMMU group 12
	[8086:1533] 06:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
IOMMU group 13
	[8086:1521] 07:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
IOMMU group 14
	[8086:1521] 07:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
IOMMU group 15
	[8086:1521] 07:00.2 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
IOMMU group 16
	[8086:1521] 07:00.3 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)"

 

I am presuming the issue is that the card is not in its own IOMMU group. So how do I isolate it so that it is?

Link to post
12 hours ago, darrenyorston said:

I have adjusted my syslinux config to read as follows:

 

"default /syslinux/menu.c32
menu title Lime Technology, Inc.
prompt 0
timeout 50
label unRAID OS
  menu default
  kernel /bzimage
  append vfio-pci.ids=8086:1521,1106:3483 initrd=/bzroot
label unRAID OS GUI Mode
  kernel /bzimage
  append initrd=/bzroot,/bzroot-gui
label unRAID OS Safe Mode (no plugins, no GUI)
  kernel /bzimage
  append initrd=/bzroot unraidsafemode
label unRAID OS GUI Safe Mode (no plugins)
  kernel /bzimage
  append initrd=/bzroot,/bzroot-gui unraidsafemode
label Memtest86+
  kernel /memtest"

 

When I try to start the VM with the the USB controller selected in "Other PCI devices" I still receive the following error"

 

"internal error: qemu unexpectedly closed the monitor: 2017-04-18T01:05:05.332604Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,id=hostdev1,bus=pci.0,addr=0x6: vfio: error, group 1 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver.
2017-04-18T01:05:05.332636Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,id=hostdev1,bus=pci.0,addr=0x6: vfio: failed to get group 1
2017-04-18T01:05:05.332647Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,id=hostdev1,bus=pci.0,addr=0x6: Device initialization failed"

 

The PCI card I added is in IOMMU group 1, along with the PCI bridge. I have tried a different card and it also gets put into IOMMU group one.

 

In system devices my PCI Devices and IOMMU groups are:

 

"


IOMMU group 0
	[8086:0c08] 00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v3 Processor DRAM Controller (rev 06)
IOMMU group 1
	[8086:0c01] 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06)
	[8086:0c05] 00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x8 Controller (rev 06)
	[1106:3483] 01:00.0 USB controller: VIA Technologies, Inc. VL805 USB 3.0 Host Controller (rev 01)
	[1000:0086] 02:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2308 PCI-Express Fusion-MPT SAS-2 (rev 05)
IOMMU group 2
	[8086:8c31] 00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 05)
IOMMU group 3
	[8086:8c2d] 00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 (rev 05)
IOMMU group 4
	[8086:8c10] 00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d5)
IOMMU group 5
	[8086:8c14] 00:1c.2 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #3 (rev d5)
IOMMU group 6
	[8086:8c16] 00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 (rev d5)
IOMMU group 7
	[8086:8c18] 00:1c.4 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #5 (rev d5)
IOMMU group 8
	[8086:8c26] 00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 05)
IOMMU group 9
	[8086:8c52] 00:1f.0 ISA bridge: Intel Corporation C222 Series Chipset Family Server Essential SKU LPC Controller (rev 05)
	[8086:8c02] 00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05)
	[8086:8c22] 00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 05)
	[8086:8c24] 00:1f.6 Signal processing controller: Intel Corporation 8 Series Chipset Family Thermal Management Controller (rev 05)
IOMMU group 10
	[1a03:1150] 03:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 03)
	[1a03:2000] 04:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 30)
IOMMU group 11
	[8086:1533] 05:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
IOMMU group 12
	[8086:1533] 06:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
IOMMU group 13
	[8086:1521] 07:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
IOMMU group 14
	[8086:1521] 07:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
IOMMU group 15
	[8086:1521] 07:00.2 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
IOMMU group 16
	[8086:1521] 07:00.3 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)"

 

I am presuming the issue is that the card is not in its own IOMMU group. So how do I isolate it so that it is?

 

Hi Darren,

Yes, you are getting this error due to the card not being isolated.

You will need to enable acs override patch. To do this please goto settings, then VM settings, toggle to advanced view at the top right. Then enable acs override.

You will then need to restart the server for this to take effect. Then check your iommu groups and hopefully, the USB controller will be in its own lovely iommu group !!

Link to post

I have enabled ACS override and the PCI USB controller is now in its own IOMMU group.

 

I have edited my Syslinux config and added the device ID.

 

After restarting my server, I was able to select the USB controller card in the VM GUI.

 

The VM now boots fine and the controller card shows up under Windows Device Manager however nothing happens when I plug a device in. Device Manager reports the device is working correctly.

 

The USB card is an Orico PVU3-202I

Link to post
9 hours ago, darrenyorston said:

I have enabled ACS override and the PCI USB controller is now in its own IOMMU group.

 

I have edited my Syslinux config and added the device ID.

 

After restarting my server, I was able to select the USB controller card in the VM GUI.

 

The VM now boots fine and the controller card shows up under Windows Device Manager however nothing happens when I plug a device in. Device Manager reports the device is working correctly.

 

The USB card is an Orico PVU3-202I

 

Do you maybe need to install drivers? 

Link to post
15 hours ago, darrenyorston said:

I thought that also. The card does have Windows drivers so I installed them. Still doesn't work though. Device Manager is reporting the card is working properly however when I plug a device in nothing happens.

 

Did you have anything plugged in when starting the vm? 

Link to post

It seems the card isnt functional via Linux. Nothing I plug into the card is detected when I run:

 

for usb_ctrl in $(find /sys/bus/usb/devices/usb* -maxdepth 0 -type l); do pci_path="$(dirname "$(realpath "${usb_ctrl}")")"; echo "Bus $(cat "${usb_ctrl}/busnum") --> $(basename $pci_path) (IOMMU group $(basename $(realpath $pci_path/iommu_group)))"; lsusb -s "$(cat "${usb_ctrl}/busnum"):"; echo; done

 

I have pulled the card out and put it into my Windows 10 machine and it works fine. Put it back into my unRAID machine and whilst the card is detected nothing happens when I plug a device into it.

Link to post
9 hours ago, darrenyorston said:

It seems the card isnt functional via Linux. Nothing I plug into the card is detected when I run:

 

for usb_ctrl in $(find /sys/bus/usb/devices/usb* -maxdepth 0 -type l); do pci_path="$(dirname "$(realpath "${usb_ctrl}")")"; echo "Bus $(cat "${usb_ctrl}/busnum") --> $(basename $pci_path) (IOMMU group $(basename $(realpath $pci_path/iommu_group)))"; lsusb -s "$(cat "${usb_ctrl}/busnum"):"; echo; done

 

I have pulled the card out and put it into my Windows 10 machine and it works fine. Put it back into my unRAID machine and whilst the card is detected nothing happens when I plug a device into it.

 

If you stub the card it will not work in unraid. That is the point of stubbing the card ;-) Remove the stubbing to check if it's working in unraid. 

Link to post

I have removed it. I deleted the entry from my Syslinux and restarted.

 

default /syslinux/menu.c32
menu title Lime Technology, Inc.
prompt 0
timeout 50
label unRAID OS
  menu default
  kernel /bzimage
  append pcie_acs_override=downstream vfio-pci.ids=8086:1521 initrd=/bzroot
label unRAID OS GUI Mode
  kernel /bzimage
  append pcie_acs_override=downstream initrd=/bzroot,/bzroot-gui
label unRAID OS Safe Mode (no plugins, no GUI)
  kernel /bzimage
  append pcie_acs_override=downstream initrd=/bzroot unraidsafemode
label unRAID OS GUI Safe Mode (no plugins)
  kernel /bzimage
  append pcie_acs_override=downstream initrd=/bzroot,/bzroot-gui unraidsafemode
label Memtest86+
  kernel /memtest

 

The vfio-pcs.ids entry is my four port ethernet card.

 

Maybe unRAID isn't compatable with the PVU3-202I PCI express cards.

 

The other options I have for PCI USB controllers are Shintaro, Skymaster, Sunix, Welland, or Vantec. Dont really want to buy another card and hope it works though.

Edited by darrenyorston
Link to post
19 hours ago, darrenyorston said:

I have removed it. I deleted the entry from my Syslinux and restarted.

 

default /syslinux/menu.c32
menu title Lime Technology, Inc.
prompt 0
timeout 50
label unRAID OS
  menu default
  kernel /bzimage
  append pcie_acs_override=downstream vfio-pci.ids=8086:1521 initrd=/bzroot
label unRAID OS GUI Mode
  kernel /bzimage
  append pcie_acs_override=downstream initrd=/bzroot,/bzroot-gui
label unRAID OS Safe Mode (no plugins, no GUI)
  kernel /bzimage
  append pcie_acs_override=downstream initrd=/bzroot unraidsafemode
label unRAID OS GUI Safe Mode (no plugins)
  kernel /bzimage
  append pcie_acs_override=downstream initrd=/bzroot,/bzroot-gui unraidsafemode
label Memtest86+
  kernel /memtest

 

The vfio-pcs.ids entry is my four port ethernet card.

 

Maybe unRAID isn't compatable with the PVU3-202I PCI express cards.

 

The other options I have for PCI USB controllers are Shintaro, Skymaster, Sunix, Welland, or Vantec. Dont really want to buy another card and hope it works though.

 

And if you plug in a USB drive does it show up in unraid now? 

Link to post
01:00.0 USB controller: VIA Technologies, Inc. VL805 USB 3.0 Host Controller (re                                                                                v 01) (prog-if 30 [XHCI])
        Subsystem: VIA Technologies, Inc. VL805 USB 3.0 Host Controller
        Flags: bus master, fast devsel, latency 0, IRQ 28
        Memory at f7c00000 (64-bit, non-prefetchable) [size=4K]
        Capabilities: [80] Power Management version 3
        Capabilities: [90] MSI: Enable+ Count=1/4 Maskable- 64bit+
        Capabilities: [c4] Express Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting
        Kernel driver in use: xhci_hcd

 

Edited by darrenyorston
added code
Link to post
6 hours ago, darrenyorston said:

Sorry about the paste..doesnt seem to like posting copied text from Putty.

 

You need to add it in a code tag (To the left of the smileys) and only supply the section for the card, not everything.

Edit your post and I'll look at it ;-)

Link to post

root@UNRAID:~# lspci | grep USB
00:12.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
00:12.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller
00:13.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
00:13.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller
00:14.5 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI2 Controller
00:16.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
00:16.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller
03:00.0 USB controller: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller  **** USB 3 ports front
04:00.0 USB controller: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller  **** USB 3 ports back motherboard

 

Bus 009 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 008 Device 007: ID 13fe:1d00 Kingston Technology Company Inc. DataTraveler 2.0 1GB/4GB Flash Drive / Patriot Xporter 4GB Flash Drive 
Bus 008 Device 006: ID 05e3:0723 Genesys Logic, Inc. GL827L SD/MMC/MS Flash Card Reader  
Bus 008 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

Want hot plugging for VM on the 2 front USB 3 plugs

 

root@UNRAID:~# readlink /sys/bus/usb/devices/usb8
../../../devices/pci0000:00/0000:00:05.0/0000:03:00.0/usb8

root@UNRAID:~# readlink /sys/bus/usb/devices/usb9
../../../devices/pci0000:00/0000:00:05.0/0000:03:00.0/usb9

 

So because they have the same pci0000:00/0000:00:05.0/0000:03:00.0 I would need to pass bus 8 & 9?

 

I assume I am looking for 03:00.0?

 

Step 9 is listed differently now in unRAID 6.3.3:

IOMMU group 16 *** all by itself
    [1b21:1142] 03:00.0 USB controller: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller

 

You then have to modify some parts of it to get it to work. The good thing is that you do not have to care about which bus and address it's supposed to have in the VM. You only need to find out the host PCI address.

The part you change is bus, slot and function. In your case it's 03:00.0.

Let's brake it down.

03 is the bus. You simply exchange the two numbers after the 0x.

00 is the slot. Same method as above.

0 is the function. Her it's also the same method as above.

So in your case the full device tag would be like this:

 

 

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>
    </hostdev>

 

 

 

 

Link to post
On 29.4.2017 at 4:27 AM, darrenyorston said:

01:00.0 USB controller: VIA Technologies, Inc. VL805 USB 3.0 Host Controller (re                                                                                v 01) (prog-if 30 [XHCI])
        Subsystem: VIA Technologies, Inc. VL805 USB 3.0 Host Controller
        Flags: bus master, fast devsel, latency 0, IRQ 28
        Memory at f7c00000 (64-bit, non-prefetchable) [size=4K]
        Capabilities: [80] Power Management version 3
        Capabilities: [90] MSI: Enable+ Count=1/4 Maskable- 64bit+
        Capabilities: [c4] Express Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting
        Kernel driver in use: xhci_hcd

 

 

Looks like unraid loads drivers for the card. 

Which devices did you try? Does a keyboard or a mouse light up if you plug it in? 

Link to post
15 hours ago, darrenyorston said:

I have tried a USB keyboard, USB mouse and the three USB thumb drives. Also the VM keeps crashing. It will blue screen then reboot. Once I remove the card from the VM it is stable.

 

Connect the devices without passing it through to a VM. Just trying to check if the card works or not in unraid.

Link to post

Yes the card works when not passed through to a VM.

 

I think I may have solved the problem. Seems the power cable might have had a problem. I have changed out to a different Molex cable and now the ports are active. Maybe a bad connection though, the same connectors run my hot swap-able drive bays fine.

 

Now I am getting "DRIVER_IRQL_NOT_LESS_OR_EQUAL" on astkmd.sys

Edited by darrenyorston
update for error fixed
Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.