***GUIDE*** Passing Through Network Controllers to unRAID 6 Virtual Machines


jonp

193 posts in this topic Last Reply

Recommended Posts

Hi All

 

I have a few minutes to look at my XML file and the layout is below.

 

I do not have a clue what to do next as this appears to be correct.

 <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x07' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x08' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </hostdev>

Do I need to change anything as the address is correct as per my IOMMU groups.

IOMMU group 20:	[8086:10bc] 07:00.0 Ethernet controller: Intel Corporation 82571EB/82571GB Gigabit Ethernet Controller (Copper) (rev 06)
[8086:10bc] 07:00.1 Ethernet controller: Intel Corporation 82571EB/82571GB Gigabit Ethernet Controller (Copper) (rev 06)
IOMMU group 21:	[8086:10bc] 08:00.0 Ethernet controller: Intel Corporation 82571EB/82571GB Gigabit Ethernet Controller (Copper) (rev 06)
[8086:10bc] 08:00.1 Ethernet controller: Intel Corporation 82571EB/82571GB Gigabit Ethernet Controller (Copper) (rev 06)

 

Link to post
  • Replies 192
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Popular Posts

In a recent reply to a post by another forum member (archedraft), I provided this guide to help him assign one of his NIC devices to a virtual machine, leaving the other for host networking (unRAID OS

I’ll write up a quick guide when I finish work. Sent from my iPhone using Tapatalk

After adding this to the syslinux confi now it works   label unRAID OS   menu default   kernel /bzimage   append pcie_acs_override=downstream,multifunction iommu=pt vfio-pci.ids=8086:1

Posted Images

12 minutes ago, Beaker69 said:

 


 <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x07' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x08' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </hostdev>

 

 

 

Try changing the guest address so both pci devices are on the same bus - same slot 
Add multifunction on the 1st device and change the second device to function 0x1
Same on the 3rd and 4th just change bus number.
Each pair should be on one bus.

Just copy paste this over your XML for those 4 devices. 
Should work.

 

 


DISCLAIMER! I don't know if its the right way to do this. But this was what fixed it for me!
I have very small idea on what i'm doing in KVM!
 

 <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x07' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
    </hostdev>
 <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x08' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x1'/>
    </hostdev>

 

Link to post
Just now, Beaker69 said:

It has semi worked. I now have 2 ports showing, it seems as if each pair has been bundled together.

 

I has given me a starting point on setting up Pfsense.

 

Any more thoughts on this would be most appreciated.

 <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x07' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
    </hostdev>
 <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x2'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x08' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x3'/>
    </hostdev>

This should work now. 
I just tricked it to see the whole nic as one pci device and multiple functions.
It was one of the ways that worked for me. Haven't tested yet neither setting pfsense or speed. but this made mine show up

Link to post

The same problem persists only two show up.

 

After starting Pfsense and confirming only 2 ports were available, I wnet back in to the XML and found the file below.

 <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x07' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x2'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x08' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x3'/>
    </hostdev>

I could find no way to remove the multifunction on command.

Link to post
 <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0b' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0b' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0c' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0c' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x1'/>
    </hostdev>

Here is how my XML is now and pfsense sees all 4 ports 


Here is my IOMMU

IOMMU group 18:	[111d:8018] 0a:02.0 PCI bridge: Microsemi / PMC / IDT PES12N3A 12-lane 3-Port PCI Express Switch (rev 0e)
				[8086:10bc] 0b:00.0 Ethernet controller: Intel Corporation 82571EB/82571GB Gigabit Ethernet Controller (Copper) (rev 06)
				[8086:10bc] 0b:00.1 Ethernet controller: Intel Corporation 82571EB/82571GB Gigabit Ethernet Controller (Copper) (rev 06)
IOMMU group 19:	[111d:8018] 0a:04.0 PCI bridge: Microsemi / PMC / IDT PES12N3A 12-lane 3-Port PCI Express Switch (rev 0e)
				[8086:10bc] 0c:00.0 Ethernet controller: Intel Corporation 82571EB/82571GB Gigabit Ethernet Controller (Copper) (rev 06)
				[8086:10bc] 0c:00.1 Ethernet controller: Intel Corporation 82571EB/82571GB Gigabit Ethernet Controller (Copper) (rev 06)

I dont know what else you could try. ill leave it to the pros.

Link to post
  • 3 weeks later...

Hi All

 

I also purchased myself a Intel 4 NIC for a pfsense but I cannot get it removed from Unraid and made visible on the VM

 

The netcard is in these two groups:

image.png.cfe426803f3262bfbd2ae0a3885ae43f.png

 

0a:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
0a:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
0b:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
0b:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)


0a:00.0 0200: 8086:10e8 (rev 01)
0a:00.1 0200: 8086:10e8 (rev 01)
0b:00.0 0200: 8086:10e8 (rev 01)
0b:00.1 0200: 8086:10e8 (rev 01)

 

I have tried both:

vfio-pci.ids=8086:10e8

pci-stub.ids=8086:10e8

 

I have read through many post and I keep getting this under network...

The two first is on the MB and the 4 NIC is shown like this "Shutdown (Inactive) shouldn't they be gone?

Also I can't see them show up when creating a VM like people show on their youtube videos?

 

(The card works because I used it before in Unraid, and I have removed it from the Bridge and network conf)

 

 

image.thumb.png.590f27b869ca87fe28091b0ccd8612b7.png

 

 

Link to post

Had been using @L0rdRaiden's video to pass through a nic to my CentOS vm. 

Since upgrading to 6.8.1 that guide no longer appears to work.   My XML is here:

 

    <interface type='bridge'>
      <mac address='52:54:00:fd:f4:bb'/>
      <source bridge='br1'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' websocket='-1' listen='0.0.0.0' keymap='en-us'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </video>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='dynamic' model='dac' relabel='yes'/>
</domain>

What appears most obvious to me is the 'hostdev' tag is no longer in use on the xml. 

How to fix?

Edited by eds
edit xml
Link to post
On 1/21/2020 at 4:15 PM, Beaker69 said:

Just an quick update. 

 

It is all working now and I am on the way to using Pfsense.

 

I would like to thank evryone for the advice.

Wish everyone could be so lucky.   Still crickets for my issue. 

Link to post
  • 2 weeks later...

I've been trying to pass though my HP NC365T 4 port NIC to my OPNsense VM all day, every time I boot, it acts like I have no interfaces. Here is my current setup.

 

IOMMU group 30:	[8086:150e] 03:00.0 Ethernet controller: Intel Corporation 82580 Gigabit Network Connection (rev 01)
IOMMU group 31:	[8086:150e] 03:00.1 Ethernet controller: Intel Corporation 82580 Gigabit Network Connection (rev 01)
IOMMU group 32:	[8086:150e] 03:00.2 Ethernet controller: Intel Corporation 82580 Gigabit Network Connection (rev 01)
IOMMU group 33:	[8086:150e] 03:00.3 Ethernet controller: Intel Corporation 82580 Gigabit Network Connection (rev 01)
Unraid OS
kernel /bzimage
append vfio-pci.ids=8086:150e initrd=/bzroot
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x2'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x3'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
    </hostdev>

 

Link to post
  • 2 weeks later...
On 12/21/2019 at 5:59 PM, skois said:

I have stubbed the NIC and it appears on the VM editor. I'm selecting all 4 ports and boot the VM. 
pfSense only sees 1 port. (em0).

The only way i managed to make pfSense to see all 4 ports was to manually change XML to this:
By changing guest nic's addresses to:
01:00.0
01:00.1
01:00.2

01:00.3

 

from

01:00.0

03:00.0

04:00.0

05:00.0
 

Is this correct way to do it? Why with the default XML it only lists 1 port?

 

Edited XML Part:


    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0b' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0b' slot='0x00' function='0x1'/>
      </source>
      <alias name='hostdev1'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0c' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev2'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x2'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0c' slot='0x00' function='0x1'/>
      </source>
      <alias name='hostdev3'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x3'/>
    </hostdev>

i have been trying for days literally following multiple guides, videos, etc to get my 4 port intel nic recognized in ANY vm with more than 1 port, this is the only thing on hundreds of pages of google results as well as posts here that got it working with not just more than 1 port but all 4 work without issue 

Link to post
15 hours ago, ibelouis said:

i have been trying for days literally following multiple guides, videos, etc to get my 4 port intel nic recognized in ANY vm with more than 1 port, this is the only thing on hundreds of pages of google results as well as posts here that got it working with not just more than 1 port but all 4 work without issue 

I'm happy that this helped you! 
But i REALLY would like an explanation why it is working by doing a manually edit like this.
Why the VM form creator doen't make it by it self.
And how come none of the "geniuses" around here give that kind of solution. Does the auto xml creation work on others? Is this happening only on specific systems? I remember getting the exact same NIC card as SI just to avoid any problems. and also remember that i had a simillar system as SI. 
I'm totally noob in KVM stuff and i only figured this while i was watching a SI's video about passing a video card and he mentions about the multifunction thing.


That was a lot of questions heh? :D

SI = SpaceInvader

Link to post
9 hours ago, skois said:

I'm happy that this helped you! 
But i REALLY would like an explanation why it is working by doing a manually edit like this.
Why the VM form creator doen't make it by it self.
And how come none of the "geniuses" around here give that kind of solution. Does the auto xml creation work on others? Is this happening only on specific systems? I remember getting the exact same NIC card as SI just to avoid any problems. and also remember that i had a simillar system as SI. 
I'm totally noob in KVM stuff and i only figured this while i was watching a SI's video about passing a video card and he mentions about the multifunction thing.


That was a lot of questions heh? :D

SI = SpaceInvader

This issue is pretty ridiculous.  I had virtual machine capability on my qnap nas and it didn't have any of these issues.

I thought Unraid would be much better in the  implementation.  Go figure it's worse!

And the video in this thread which used to work is now useless. 

Link to post
  • 2 weeks later...

Hi all

 

I seem to be having the same issues as others are having, but I'm using a 2 port SFP+ NIC. Might be a little different.

 

Here's my xml:

 

<hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x81' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x81' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </hostdev>

 

If anybody could help me id be sooo happy.

 

2020-02-26_11-24-54.png

Link to post
Just now, Partial said:

Hi all

 

I seem to be having the same issues as others are having, but I'm using a 2 port SFP+ NIC. Might be a little different.

 

Here's my xml:

 


<hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x81' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x81' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </hostdev>

 

If anybody could help me id be sooo happy.

 

2020-02-26_11-24-54.png

Change the second adress tags to 

<address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0' multifunction='on'/>
<address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x1'/>

 

So it will be in the same virtual bus 

Now you should see 2 ports when booted

Link to post

Hey guys,

 

i try to passthrough my nic to a win 10 vm.

 

2b:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection

8086:10d3

These are the details about my nic.

 

When I start the VM there is the following error:

 

Quote

internal error: Unknown PCI header type '127' for device '0000:2b:00.0'

 

What can I do? 

 

MB ist an ASRock Rack X470D4U with a Ryzen 7 2700.

 

Thank you very much for your help.

Edited by willi_jan9393
Link to post
On 2/26/2020 at 4:23 PM, skois said:

Change the second adress tags to 

<address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0' multifunction='on'/>
<address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x1'/>

 

So it will be in the same virtual bus 

Now you should see 2 ports when booted

❤️ That worked! TY!

Link to post
  • 1 month later...
26 minutes ago, shaunmccloud said:

This might be answered (didn't see it but I could be wrong), if I have a dual port i350 NIC built into my server I cannot pass just one of them through to a VM correct?

 

If this has been solved, I haven't seen it. 

Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.