Ryzen 3000 series build / Struggling with PCI-e lanes


Recommended Posts

15 minutes ago, johnnie.black said:

I believe so.

 

Yes, assuming it's using a x4 lane for the U.2 connection, though it might be shared with something else, that's usually mentioned din the manual if that's the case, shame one thing usually missing from consumer board's manual, a system block diagram, those make connections/bandwidth much clearer.

 

I think it's missing in the manual, only this is said about the U.2 connector :

 

2050421298_E15611_Pro_WS_X570-ACE_UM_V2_WEB.pdf2019-07-1111-09-35.png.23074cd830f9f0b5448e6925722ba744.png

 

The marketing page say :

 

Ultra-fast storage: Up to 64Gbps M.2 & 32Gbps U.2, plus up to 10Gbps back-panel USB 3.2 Gen 2 with Type-A and Type-C ports

Speed up with onboard M.2 & U.2

With x4 PCI Express® 4.0 bandwidth, Pro WS X570-ACE features up to 64Gbps M.2 and U.2 for faster data-transfer speeds — ideal for an operating system or application drive.

 

Also, if I understand right, I might put the Cards and Disks like this to get best performances :

 

Expansions cards :

PCIE_X16_1 =  EVGA GTX 1080 ti (for VM passthru)

PCIE_X16_2 = HBA #1 (9211-8i)

PCIE_X16_2 = HBA #2 (9211-8i)

PCIE_X1 = riser to a X16 for EVGA GTX 1060 6GB (for PLEX docker nvenc/nvdec)

 

Disks :

M2_1 = Corsair MP600 NVMe Gen4 (Cache drive)

SATA1 to SATA4 = 4 disks in Cage_1

HBA #1 port 1 = 5th disk in Cage_1

HBA #1 Port 2-6 = 5 disks in Cage_2

HBA #1 Port 7-8 = 2 first disks in Cage_3

HBA #2 Port 1-3 = 3 last disks in Cage_3

HBA #2 Port 4-8 = 5 disks in Cage 4

U.2 = Future Use ?  (Maybe re-use my Samsung EVO 840 500GB SATA SSD for VM Passthru?)

 

Question : Would you use U.2 4 SATA instead of SATA1 to SATA4 on the board instead of what I proposed here?

 

 

 

Edited by Pducharme
  • Like 1
Link to comment

I just read on https://www.guru3d.com/articles-pages/tech-preview-amd-ryzen-with-ryzen-3950x,5.html that Ryzen 3000 has 24 lanes and coupled with the X570 (16 lanes), do a total of 40 lanes!  that's a hell lot of PCIe lanes, and all PCIe Gen4, a x8 PCIe  Gen4 lane = a x16 PCIe Gen3 lane.

 

Ryzen 3000 series CPU :

  • 4 / 24 = Interconnect with X570 PCH.
  • 16 / 24 = for GPUs ( Single x16 or dual x8 )
  • 4 / 24 = NVMe Gen4 
  • Also include USB 3.0 Gen2 in the CPU, giving 4 x 10Gbps USB SuperSpeed ports

X570 Chipset :

  • 4 x 4 lanes phyical interfaces (16 lanes total)
  • fully configurable in PCIe x16, x8, x4, x2, x1, and SATA modes

 

Link to comment
Just now, johnnie.black said:

Yes, but those 16 lanes still share a x4 connection from the chipset to the CPU.

Ok so it’s like a lane « splitter ». I suppose it’s why the fastest bandwidth hungry stuff are on cpu lanes and that other less hungry are on the PCH.? 

Link to comment
38 minutes ago, fr05ty said:

@johnnie.black do you know if the ds4243 will benefit from 2 cables from the HBA to iom6 controllers in bay 1 and 3 of the disk shelf, or are they just for redundancy, my google-fu isn't working out well. I have a 9207-8e, all the disks i am going to populate it with are sata3 drives

No, never had any NetApp gear, but you can check link width with:

cat /sys/class/sas_host/host1/device/port-1\:0/sas_port/port-1\:0/num_phys

Might need to replace 1s with correct number, output of 4 means single link ( 4 phys in use), output of 8 means dual link.

 

  • Upvote 1
Link to comment

UPDATE !!  I got my Motherboard and CPU :) I assembled my test bench with :

 

Corsair MP600 1TB

3 x Western Red 3TB

1 x EVGA 1080ti

2 x 8GB Corsair DDR4 3200Mhz (Temporary until Tuesday when I’ll get my 2 x 16GB Corsair DDR4 3200Mhz)

 

i did put the Unraid USB key I just created for the purpose of this test, powered ON and it booted to Unraid!

 

i only have a trial key but that should be enough ;)

 

i still need to Update the motherboard BIOS because it came in on a very old BIOS of May 2019!  I haven’t tried Installing Windows 10 either. It was just an initial boot to check if the platform would at least boot!

This is what I plan to test :

 

1. Update BIOS and enable IOMMU

2. Install Community Apps ans Unraid-NVIDIA build.

3. Create an array of 1 Parity, 2 data disk and 1 cache disk. 
4. Create a Win10 VM and passThrough a Keyboard/Mouse and the 1080ti

5. Install Plex docker and add couple of movies to test Hardware decoding with the 1080ti (will be the 1060 6GB in my real system)

 

any other tests people want me to do on the platform?  How can I test stability?

E65ECF26-9A7E-480E-AFB2-4A9A78E6342D.jpeg

Edited by Pducharme
  • Upvote 1
Link to comment

Here is the IOMMU groups for the ASUS WS Pro X570-ACE w/BIOS 0702 (current) :

 

IOMMU group 0:	[1022:1482] 00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 1:	[1022:1483] 00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
IOMMU group 2:	[1022:1483] 00:01.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
IOMMU group 3:	[1022:1482] 00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 4:	[1022:1482] 00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 5:	[1022:1483] 00:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
IOMMU group 6:	[1022:1482] 00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 7:	[1022:1482] 00:05.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 8:	[1022:1482] 00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
[1022:1484] 00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
[1022:148a] 0a:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
IOMMU group 9:	[1022:1482] 00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
[1022:1484] 00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
[1022:1484] 00:08.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
[1022:1484] 00:08.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
[1022:1485] 0b:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
[1022:1486] 0b:00.1 Encryption controller: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP
[1022:149c] 0b:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
[1022:7901] 0c:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
[1022:7901] 0d:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
IOMMU group 10:	[1022:790b] 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 61)
[1022:790e] 00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)
IOMMU group 11:	[1022:1440] 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 0
[1022:1441] 00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 1
[1022:1442] 00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 2
[1022:1443] 00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 3
[1022:1444] 00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 4
[1022:1445] 00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 5
[1022:1446] 00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 6
[1022:1447] 00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 7
IOMMU group 12:	[1987:5016] 01:00.0 Non-Volatile memory controller: Phison Electronics Corporation Device 5016 (rev 01)
IOMMU group 13:	[1022:57ad] 02:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57ad
IOMMU group 14:	[1022:57a3] 03:03.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57a3
IOMMU group 15:	[1022:57a3] 03:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57a3
IOMMU group 16:	[1022:57a4] 03:08.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57a4
[1022:1485] 06:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
[1022:149c] 06:00.1 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
[1022:149c] 06:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
IOMMU group 17:	[1022:57a4] 03:09.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57a4
[1022:7901] 07:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
IOMMU group 18:	[1022:57a4] 03:0a.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57a4
[1022:7901] 08:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
IOMMU group 19:	[8086:1539] 04:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)
IOMMU group 20:	[10ec:816e] 05:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. Device 816e (rev 1a)
[10ec:8168] 05:00.1 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 1a)
[10ec:816a] 05:00.2 Serial controller: Realtek Semiconductor Co., Ltd. Device 816a (rev 1a)
[10ec:816c] 05:00.7 IPMI Interface: Realtek Semiconductor Co., Ltd. Device 816c (rev 1a)
IOMMU group 21:	[10de:1b06] 09:00.0 VGA compatible controller: NVIDIA Corporation GP102 [GeForce GTX 1080 Ti] (rev a1)
[10de:10ef] 09:00.1 Audio device: NVIDIA Corporation GP102 HDMI Audio Controller (rev a1)

 

Link to comment
Just now, phat_cow said:

You have a usb controller in group 9 and 16. Does that mean one of the usb controllers can be passed through to a VM without affecting the unraid flash drive?

I don't know that as of yet.  I only check what the groups are.  In the BIOS, I selected "Enabled" and not "auto" as I read it's better this way.   I never created a VM that I did passthrough to it on my current server, so i'll have to check tutorials on how to do it.  I'll have to build a new Win10 VM on this server to test that.

Link to comment
1 minute ago, fr05ty said:

did you have to disable the global c-state in the bios?i think that i had to do that with my 1700, i'm still waiting for my M/B to turn up it's a bit depressing just looking at a cpu just sitting alone

 

I didn't change that value.  How to know if I need to do it? what was the issue when you did not change it ?

Link to comment
1 minute ago, david279 said:

C states stuff is only for the first gen ryzen don't mess with it for 2nd and probably 3rd gen.

Ok thanks.  i'll leave that alone.  

 

I can see my GTX 1080 ti in the VM creation, and only 1 USB keyboard.  What else do I need to do to see the IOMMU groups ?

Link to comment

@fr05ty I suppose if I'm reading right my Group, that I could probably passthru USB controller in group 9?

 

Bus 1 --> 0000:06:00.1 (IOMMU group 16)
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

Bus 2 --> 0000:06:00.1 (IOMMU group 16)
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub

Bus 3 --> 0000:06:00.3 (IOMMU group 16)
Bus 003 Device 003: ID 0781:5571 SanDisk Corp. Cruzer Fit
Bus 003 Device 002: ID 0bda:5411 Realtek Semiconductor Corp. 
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

Bus 4 --> 0000:06:00.3 (IOMMU group 16)
Bus 004 Device 002: ID 0bda:0411 Realtek Semiconductor Corp. 
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub

Bus 5 --> 0000:0b:00.3 (IOMMU group 9)
Bus 005 Device 002: ID 045e:00dd Microsoft Corp. Comfort Curve Keyboard 2000 V1.0
Bus 005 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

Bus 6 --> 0000:0b:00.3 (IOMMU group 9)
Bus 006 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub

 

Here is the breakdown :

 

IOMMU group 17
[RESET] 03:09.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57a4]
[RESET] 07:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU group 7
        00:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 15
[RESET] 03:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57a3]
IOMMU group 5
[RESET] 00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU group 13
[RESET] 02:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57ad]
IOMMU group 3
        00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 21
[RESET] 09:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP102 [GeForce GTX 1080 Ti] [10de:1b06] (rev a1)
        09:00.1 Audio device [0403]: NVIDIA Corporation GP102 HDMI Audio Controller [10de:10ef] (rev a1)
IOMMU group 11
        00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 0 [1022:1440]
        00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 1 [1022:1441]
        00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 2 [1022:1442]
        00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 3 [1022:1443]
        00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 4 [1022:1444]
        00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 5 [1022:1445]
        00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 6 [1022:1446]
        00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 7 [1022:1447]
IOMMU group 1
[RESET] 00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU group 18
[RESET] 03:0a.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57a4]
[RESET] 08:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU group 8
        00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
[RESET] 00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
[RESET] 0a:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
IOMMU group 16
[RESET] 03:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57a4]
[RESET] 06:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
        06:00.1 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
[RESET] 06:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
IOMMU group 6
        00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 14
[RESET] 03:03.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57a3]
IOMMU group 4
        00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 12
[RESET] 01:00.0 Non-Volatile memory controller [0108]: Phison Electronics Corporation Device [1987:5016] (rev 01)
IOMMU group 2
[RESET] 00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU group 20
[RESET] 05:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. Device [10ec:816e] (rev 1a)
        05:00.1 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 1a)
        05:00.2 Serial controller [0700]: Realtek Semiconductor Co., Ltd. Device [10ec:816a] (rev 1a)
        05:00.7 IPMI Interface [0c07]: Realtek Semiconductor Co., Ltd. Device [10ec:816c] (rev 1a)
IOMMU group 10
        00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 61)
        00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU group 0
        00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 19
[RESET] 04:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03)
IOMMU group 9
        00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
[RESET] 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
[RESET] 00:08.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
[RESET] 00:08.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
[RESET] 0b:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
[RESET] 0b:00.1 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP [1022:1486]
[RESET] 0b:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
[RESET] 0c:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
[RESET] 0d:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)

 

Edited by Pducharme
Link to comment

All the usb controllers are grouped with other devices. Not a good look. You could try the ACS patch to split it up and clean up the groups a bit. Just make sure you are not trying to pass thru a USB controller with the unRAID usb in it. Group 16 has 2 controllers in it itself be nice to break that up.

Link to comment
22 minutes ago, david279 said:

All the usb controllers are grouped with other devices. Not a good look. You could try the ACS patch to split it up and clean up the groups a bit. Just make sure you are not trying to pass thru a USB controller with the unRAID usb in it. Group 16 has 2 controllers in it itself be nice to break that up.

Yeah... I think i'll leave that without USB controller Passthrough... I don't understand too much how and the Video I watch said that it add instability on the host.  Here the WAF (Wife Acceptance Factor) is already pretty low, so if the new server is unstable, I prefer not to thing what coud happen :)

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.