Jump to content
betaman

First time build - What to do AFTER the UnRAID server is built?!

38 posts in this topic Last Reply

Recommended Posts

Ok, so I've been reading like a maniac since I first joined this fabulous community and now I want to make sure that I've separated "dated" info from the most current and I setup my first UnRAID server in the best possible configuration for media streaming.  I've been through the wiki several times but some of the info is pretty old so I want to make sure I don't miss any important info from other posts.

 

Hardware Profile:

ASUS P5Q Premium LGA 775 Intel P45 ATX Intel Motherboard

Intel Core 2 Duo E8400 Wolfdale 3.00GHz LGA 775 65W Dual-Core Processor (overkill but I got to upgrade my desktop to a Quad Core in the process!)

G.SKILL 2GB (2 x 1GB) 240-Pin DDR2 SDRAM DDR2 1066 (PC2 8500) Dual Channel Kit Desktop Memory Model F2-8500CL5D-2GBPK

CORSAIR CMPSU-850TX 850W ATX12V 2.2 (again overkill but I got a good price and I may eventually have 20 drives!)

ASUS EN8400GS SILENT/HTP/512M GeForce 8400 GS 512MB 64-bit GDDR2 PCI Express x16 HDCP Ready Video Card

LIAN LI PC-P80 Black Aluminum ATX Full Tower Computer Case

4x Lian Li EX-H34 HDD 4in3 enclosures

3x Seagate Barracuda LP ST32000542AS 2TB 5900 RPM SATA 3.0Gb/s 3.5" Hard Drive

Kingston DataTraveler 1GB - still haven't figured out if I can use this stick or not but I have no issue to get a newer 4GB stick especially if I upgrade to the Pro version

 

My current needs are only about 4TB of data so I was going to manage a 3 drive configuration with a 2TB parity drive just to get my feet wet with the free version of UnRAID.  I see myself quickly adding more drives to convert my Blu-Ray collection as well as freeing up some 1 TB drives I'm currently using but I really wanted to familiarize myself with the OS before taking the big plunge.  I've already got the 3 drives but still waiting on the other hardware.  So here are my questions once I get the server built:

 

1.  Should I wait to format and test the drives with Joe L.'s preclear_disk.sh utility until I have the server built (it will be a couple of days and I didn't know if I could/should test the drives some other way in a Vista x64 environment in case there's any issues?  I don't want to lose any time with an RMA depending on how long I should test the 3 drives before writing data.

2.  Should I install 4.5 beta6 even though it's a beta?  Seems that most people feel it is stable enough as I'm not too concerned about the additional capacity or Active Directory working properly right now.

3.  User shares?  Should I use these since my main focus is media storage?  I like the concept but I'm not entirely clear on using them for reading and fixed disks for writing to control where the contents are located.  I've got an NMT Popcorn Hour A-110 so I just want to be able to point my NMT to one share location and have it see all of my UnRAID drive content.  User shares seems like the way to go but I'm not really sure if I should commit 100% to this (read/write) or how easy it is to switch back and forth depending on what I'm trying to do (i.e. stream a movie vs. writing new content to the server)

4.  What other apps are "must haves" for a noob like myself?  UnMenu seems like an obvious choice but I wasn't sure if any changes had been made in 4.5 beta6 that might preclude installing this customization?

 

I guess I should qualify all of this with the fact that I'm a relative novice when it comes to networking (have figured out SMB shares...woohoo!) and all things Linux based so I'm really looking to just get in the game with a good media server that will stream HD content to my NMT A-110 without any studdering video.  UnRAID seems to be the perfect fit for what I'm looking to do so I just don't want to screw it up from the beginning (i.e. question 3 from above)!

 

Thanks in advance to anyone reading this post and willing to offer some insight!

 

Share this post


Link to post

Hi betaman, I'm new to unRaid too and building my first server this weekend.

 

Hardware Profile:

ASUS EN8400GS SILENT/HTP/512M GeForce 8400 GS 512MB 64-bit GDDR2 PCI Express x16 HDCP Ready Video Card

LIAN LI PC-P80 Black Aluminum ATX Full Tower Computer Case

Wow, pretty case and nice GFX card for something that could live in a closet.  ::)

 

2.  Should I install 4.5 beta6 even though it's a beta? 

I'm going to start with a stable version at first. I'll install unMENU and upgrade once I have it all working. One less variable to worry about when things are done in steps. Thoughts?

 

3.  User shares?  Should I use these since my main focus is media storage?

I thought user shares were the only way to read/write data to the array.

Share this post


Link to post

Wow, pretty case and nice GFX card for something that could live in a closet.  ::)

 

Yeah, I bought the case as I figure it will be the last case I'll ever need and the GFX card was only $20 so thought I'd splurge!  So what are the specs of your build if you don't mind me asking?

 

I'm going to start with a stable version at first. I'll install unMENU and upgrade once I have it all working. One less variable to worry about when things are done in steps. Thoughts? 

I thought so too but after reading so much about some of the improvements in 4.5.x, I think Beta 6 is a relatively stable release.  I'd rather not have to learn backwards if at all possible.

 

I thought user shares were the only way to read/write data to the array.

Check out this thread http://lime-technology.com/forum/index.php?topic=4226.0

 

Share this post


Link to post

1.  Should I wait to format and test the drives with Joe L.'s preclear_disk.sh utility until I have the server built (it will be a couple of days and I didn't know if I could/should test the drives some other way in a Vista x64 environment in case there's any issues?  I don't want to lose any time with an RMA depending on how long I should test the 3 drives before writing data.

2.  Should I install 4.5 beta6 even though it's a beta?  Seems that most people feel it is stable enough as I'm not too concerned about the additional capacity or Active Directory working properly right now.

3.  User shares?  Should I use these since my main focus is media storage?  I like the concept but I'm not entirely clear on using them for reading and fixed disks for writing to control where the contents are located.  I've got an NMT Popcorn Hour A-110 so I just want to be able to point my NMT to one share location and have it see all of my UnRAID drive content.  User shares seems like the way to go but I'm not really sure if I should commit 100% to this (read/write) or how easy it is to switch back and forth depending on what I'm trying to do (i.e. stream a movie vs. writing new content to the server)

4.  What other apps are "must haves" for a noob like myself?  UnMenu seems like an obvious choice but I wasn't sure if any changes had been made in 4.5 beta6 that might preclude installing this customization?

 

1 - There would be no harm in formatting them (long format, not quick) in vista etc just to make sure they're not DOA. There may be windows tools to emulate a similar test to Joes script. However, once your server is ready I would still run the pre clear script over them. If nothing else it's a good habit to get into and the script really is very good.

 

2 - beta6 is certainly stable enough (I've been running it since its release with only one crash which I've never been able to reproduce - probably hardware related.) However there are some bugs - the one that springs to mind is cache drive performance. There is a thread on the beta release in the announcements sub forum a browse through that will highlight any other issues.

 

3 - Once you've set up a user share you can continue to share your individual drives. You can then read / write to them via the user share or direct to the drive shares at your discretion. Any new data written via one method will be immediately available to access via the other. I personally use user shares for much the same reason you've highlighted (one big share) but many people prefer to manage their data manually and copy to the individual drives. As above, you can try both with no penalty.

 

4 - unMenu is definitely a must and still fits with beta6 nicely. Depending on your setup and wants I'd also suggest the cache_dirs script and one of the many unraid email monitor scripts to alert you with any array or hardware problems.

Share this post


Link to post

Kingston DataTraveler 1GB - still haven't figured out if I can use this stick or not but I have no issue to get a newer 4GB stick especially if I upgrade to the Pro version

 

This stick will work fine but I would recommend getting a new stick if you’re going to a licensed version.

 

1.  Should I wait to format and test the drives with Joe L.'s preclear_disk.sh utility until I have the server built (it will be a couple of days and I didn't know if I could/should test the drives some other way in a Vista x64 environment in case there's any issues?  I don't want to lose any time with an RMA depending on how long I should test the 3 drives before writing data.

 

wait..use preclear

 

2.  Should I install 4.5 beta6 even though it's a beta?  Seems that most people feel it is stable enough as I'm not too concerned about the additional capacity or Active Directory working properly right now.

 

yes install beta6

 

3.  User shares?  Should I use these since my main focus is media storage?  I like the concept but I'm not entirely clear on using them for reading and fixed disks for writing to control where the contents are located.  I've got an NMT Popcorn Hour A-110 so I just want to be able to point my NMT to one share location and have it see all of my UnRAID drive content.  User shares seems like the way to go but I'm not really sure if I should commit 100% to this (read/write) or how easy it is to switch back and forth depending on what I'm trying to do (i.e. stream a movie vs. writing new content to the server)

 

Depends on content and if you’re using a jukebox.  If you’re using isos you will need to use some form of shares. If you have avis and mkv I would recommend streaming via http. I stream using Lighttp and rarely have issues. When I do it is minor pixilation with action sequences on 1080p content.

The good thing here is you can have both. Switching is easy and welcome fellow PCH user :)

 

4.  What other apps are "must haves" for a noob like myself?  UnMenu seems like an obvious choice but I wasn't sure if any changes had been made in 4.5 beta6 that might preclude installing this customization?

 

Only install what is needed. Everyone is a little different so you will know what you must have soon enough.

  - I have Monthly Parity Check and unRaid-Web for the Lighttp installation(http streaming).  I also have a host name fix but that’s pretty much it. I will probably add SABnzbd at some point. 

 

Cheers db

 

Share this post


Link to post

Thanks for the feedback boof and db!  Glad to hear that you can seamlessly switch between user shares and direct shares.  I thought about trying to run the beta 6 software on my desktop machine just to familiarize myself with the interface but I'm only a day away from receiving all of my hardware so I'll just wait at this point.

 

@boof:  what exactly is the cache_dirs script?  I've seen some of the e-mail scripts available and wondered when that will be part of the UnRAID software distribution as it seems to be something most people would want.

 

@db:  I've really been contemplating what streaming method to use for my setup (i.e. smb, nfs or http).  I was hoping that smb would work well enough that I could use it for all media.  Most of my media is 720p or 1080p h.264 mkv's but I do have a few DVD folder structures and iso's.  Not a very big avi collection as most of my stuff has DD 5.1 or DTS as well as higher bitrate video streams.

 

I may start the build tomorrow evening but I'm sure by the weekend I'll have plenty more questions!  Thanks again!

Share this post


Link to post

Wow, pretty case and nice GFX card for something that could live in a closet.  ::)

 

Yeah, I bought the case as I figure it will be the last case I'll ever need and the GFX card was only $20 so thought I'd splurge!  So what are the specs of your build if you don't mind me asking?

My Build Thread. I just recieve the last box from newegg and I plan to start the build tomorrow. I hope the MB works out, but I've learned when dealing with compters, things rarely go as smooth and you think.

 

I'm going to start with a stable version at first. I'll install unMENU and upgrade once I have it all working. One less variable to worry about when things are done in steps. Thoughts? 

 

I thought so too but after reading so much about some of the improvements in 4.5.x, I think Beta 6 is a relatively stable release.  I'd rather not have to learn backwards if at all possible.

I agree. Beta 6 seems to be stable and I've already loaded my USB stick with it  :)

Share this post


Link to post

Thanks for the feedback boof and db!  Glad to hear that you can seamlessly switch between user shares and direct shares.  I thought about trying to run the beta 6 software on my desktop machine just to familiarize myself with the interface but I'm only a day away from receiving all of my hardware so I'll just wait at this point.

 

@boof:  what exactly is the cache_dirs script?  I've seen some of the e-mail scripts available and wondered when that will be part of the UnRAID software distribution as it seems to be something most people would want.

 

@db:  I've really been contemplating what streaming method to use for my setup (i.e. smb, nfs or http).  I was hoping that smb would work well enough that I could use it for all media.  Most of my media is 720p or 1080p h.264 mkv's but I do have a few DVD folder structures and iso's.  Not a very big avi collection as most of my stuff has DD 5.1 or DTS as well as higher bitrate video streams.

 

I may start the build tomorrow evening but I'm sure by the weekend I'll have plenty more questions!  Thanks again!

 

The cache_dirs script uses the system 'find' command to keep a copy (or as much of a copy as your memory allows!) of your shares directory listings in memory. This means when you browse shares from a client all the info is cached and it doesn't (potentially) need to spin up the actual disks - which could be several if using user shares - to read the directory listings.

 

As well as allowing your disks to stay spun down far more effectively it also hugely increases the speed and responsiveness of share browsing. I think everyone who has used it has universally sung it's praises and noticed a huge difference.

 

http://www.lime-technology.com/wiki/index.php/Improving_unRAID_Performance#Keep_directory_entries_cached

 

explains it much better than me ;)

 

To butt in on your transport question - this will largely depend on your playback client. If it helps I exclusively use samba for everything and my playback requirements encompass the same requirements you have.

 

Playback is either from windows via a mapped share -> mediaportal or XBMC running under ubuntu via samba.

 

Both are fine, I've also streamed full bit rate blu rays with no issues. So depends on your client and I guess also your network - I'd strongly recommend a wired gigabit network!

Share this post


Link to post

I've really been contemplating what streaming method to use for my setup (i.e. smb, nfs or http).  I was hoping that smb would work well enough that I could use it for all media.  Most of my media is 720p or 1080p h.264 mkv's but I do have a few DVD folder structures and iso's.  Not a very big avi collection as most of my stuff has DD 5.1 or DTS as well as higher bitrate video streams.

 

While SMB support on the NMTs has improved greatly I still find http the best option.

 

Since you have some DVD structures as well as ISOs you may have to use SMB or NFS. Using NFS on unRaid works good but its still relatively new. I started out with NFS myself. It just didn't work out for me.

 

If you have trouble will present itself in the high action sequences on 1080p stuff. As an example..I have the Aliens collection. Some of the darker action scenes can stutter/pixalize slightly. You just have to try it and see if it works for you. I have used SMB for both 720p and 1080p successfully so it really depends on the movie, the encoding and the quality of your network. My video network is isolated, hardwired and all GB.

 

If your interested in streaming using http look at llink and unRaid web. Both are very easy to run on unRaid and just as easy to remove if you decide its not for you.

 

Good Luck, db

 

 

Share this post


Link to post

Ok, so the build went fairly well.  Two minor issues: First, in my efforts to keep the cabling as clean as possible, I forgot to plug in the second MB power adapter (it was tucked neatly away!) on the first boot so there was no beeps or anything!  Momentary panic!  :o  Then, I had some issues getting my router to assign an IP address to the tower.  After about an hour of troubleshooting, I rebooted the PC I was trying to view the http://tower info on and all was well (except I have to use the IP address in place of "tower" but that doesn't really concern me).

 

Ok, so now I'm getting ready to use Joe's preclear_disk.sh utility and I'm not sure about a few things:

 

1.  If I'm using the free version of UnRAID then it appears as though I must assign one of the disks as parity.  Should I only assign two disks for now and pre-clear those disks first, then copy data to those drives and then assign the 3rd drive as parity and do a pre-clear on it?  Sounds like copying over my data on the first run without a parity drive in the array will save some time and I do have the data on other disks currently.

 

2.  All of the data is on external NTFS formatted drives.  I've got a Gigabit switch coming but I'm wondering if that would be the fastest way to transfer data?  I think I saw a thread somewhere specifically related to NTFS formatted data so I'll have a look again.  It would seem that I've got a couple days until the pre-clear is finished on 3x 2TB drives.

 

3.  If I install the Gigabit switch into my network then I presume it should be the link between desktop PC and tower for the fastest possible file transfer; however, I've heard there are issues streaming to my Popcorn Hour NMT A-110 with a Gigabit network and most people add a 10/100 switch between their NMT and the Gigabit switch.  Is it also possible for me to just plug my A-110 into my Netgear router directly?  Then would I just plug everything else (Slingbox, desktop PC, UnRAID server and wireless router) into the Gigabit switch?  Would this serve the same purpose?

Share this post


Link to post

Preclear is used before assigning the disks to the array. The idea is not to tie up a working array formating and clearing a disk. The array is not mounted until the process is completed.  In your case using the pre_clear has limited value if you have already mounted the drives in unRaid.

Share this post


Link to post

Preclear is used before assigning the disks to the array. The idea is not to tie up a working array formating and clearing a disk. The array is not mounted until the process is completed.  In your case using the pre_clear has limited value if you have already mounted the drives in unRaid.

 

Yeah, I just figured that out!  Luckily, I hadn't mounted the drives yet.  Running Pre-Clear now on one disk.  I already asked in a different thread but could/should I run the Pre-Clear script on multiple drives simultaneously??

Share this post


Link to post

Preclear is used before assigning the disks to the array. The idea is not to tie up a working array formating and clearing a disk. The array is not mounted until the process is completed.  In your case using the pre_clear has limited value if you have already mounted the drives in unRaid.

 

Yeah, I just figured that out!  Luckily, I hadn't mounted the drives yet.  Running Pre-Clear now on one disk.  I already asked in a different thread but could/should I run the Pre-Clear script on multiple drives simultaneously??

Yes, just open up multiple telnet sessions, run one pre-clear in each, or switch to alternate system consoles if you logged in using the system console. (Alt-F1 through Alt-F6 switch between 6 virtual consoles) Run one pre-clear in each console.

 

Joe L.

Share this post


Link to post

Yes, just open up multiple telnet sessions, run one pre-clear in each, or switch to alternate system consoles if you logged in using the system console. (Alt-F1 through Alt-F6 switch between 6 virtual consoles) Run one pre-clear in each console.

 

Joe L.

 

Thanks Joe!  All 3 2TB drives are currently in Post Read Step 10 of 10 with over 24 hrs of runtime for one pass!  Does that seem too long?  I noticed one of the drives has significantly lower write throughput than the other 2 (85 MB/s as compared to 110-115 MB/s).  Seems to also be reading slower than the other 2 (69.2 MB/s as compared to 82.5 MB/s).

 

EDIT:  I've attached a syslog from the first disk to finish pre_clear.  Any help interpreting the results would be greatly appreciated!

Share this post


Link to post

I pre-cleared a Seagate 2TB last week & it took right at 21 hours to complete one pass. I also pre-cleared a 2TB WD & it took 21 hours 20 minutes. This was only pre-clearing one drive at a time. I've never tried to pre-clear multiple drives at the same time.

 

You're not to far past that at 24 hours..

Share this post


Link to post

Thanks for the info PhilH.  I was reviewing my syslog and noticed some issues related to unMenu.  I tried to edit my go script to load unMenu at startup:

 

#!/bin/bash

# Start the Management Utility

/usr/local/sbin/emhttp &

/boot/unmenu/uu

 

Does this look correct if I have unMenu installed in the root of my flash drive in a directory called unmenu?  Also, can I just use "Notepad" in Windows Vista x64 to edit this file without issue?

Share this post


Link to post

Yes, just open up multiple telnet sessions, run one pre-clear in each, or switch to alternate system consoles if you logged in using the system console. (Alt-F1 through Alt-F6 switch between 6 virtual consoles) Run one pre-clear in each console.

 

Joe L.

 

Thanks Joe!  All 3 2TB drives are currently in Post Read Step 10 of 10 with over 24 hrs of runtime for one pass!  Does that seem too long?  I noticed one of the drives has significantly lower write throughput than the other 2 (85 MB/s as compared to 110-115 MB/s).  Seems to also be reading slower than the other 2 (69.2 MB/s as compared to 82.5 MB/s).

 

EDIT:  I've attached a syslog from the first disk to finish pre_clear.  Any help interpreting the results would be greatly appreciated!

That disk looks pretty good to me. 

 

This line in the syslog seems to indicate you are attempting to start unMENU when it is already running.

Sep 28 21:08:10 Tower unmenu[15205]: awk: ./unmenu.awk:256: fatal: can't open two way socket `/inet/tcp/8080/0/0' for input/output (No such file or directory)

(Only one process can open port 8080 at a time.  Since one unMENU instance is already running, the port cannot be opened again)

 

Looks like you are on your way... Biggest drive I own is a 1.5 TB, and it takes over 16 hours to do one cycle.  20 to 24 for a 2TB sounds just fine.

 

Joe L.

Share this post


Link to post

Thanks Joe!  I started unmenu from the root@Tower:/boot# prompt and then edited my go script.  Does the pre_clear script reload the go script once it completes a cycle?  Perhaps that's where the second instance of unmenu is coming from.  Anyway, I'll post the other 2 drives once they complete.  I'm wondering if running 3 scripts of pre_clear slows down the tests at all?  All in all, 3 tests in approx. 27 hours is much better than running them successively for 20 hrs each!

Share this post


Link to post

Ok, here's the syslog after the last two disks finished pre_clear.  I think they are fine but wanted to confirm.  Thanks again for all of the help!

Share this post


Link to post

Ok, here's the syslog after the last two disks finished pre_clear.  I think they are fine but wanted to confirm.  Thanks again for all of the help!

I think you are right... It looks good.

 

I did notice the syslog had these three lines

Sep 28 18:59:49 Tower kernel: md: disk0 removed

Sep 28 18:59:49 Tower kernel: md: disk1 missing

Sep 28 18:59:49 Tower kernel: md: disk2 missing

 

It looks as if you assigned the drives at one point in time, but then un-assigned them.

 

To un-raid, un-assigning a drive is exactly the same as if it has failed.  Right now, this is harmless.  Later, once you assign drives, especially if you only assign one at a time it might complain about the missing/un-assigned drive and not let you start the array until it sees all the drives.  If you run into a snag, let us know, as it will help other in the future to know what to expect.

 

Running three pre-clear scripts in parallel is moving three times the data across your system.  Even the best system will not be able to do that in the same time as 1/3rd the data.  The pre-clear script starts many thousands of processes during the read phases... three processes will start three times as many.

 

Yes, three pre-clear cycles in 27 hours is a lot better than 60 hours for 3 in sequence.

 

Joe L.

Share this post


Link to post

I had 5 telnet sessions open running preclear.sh  ::) I didn't check the CPU usage during this. The largest drive was a 1.5TB WD Green and it took 24hrs to clear.

Share this post


Link to post

Ok, here's the syslog after the last two disks finished pre_clear.  I think they are fine but wanted to confirm.  Thanks again for all of the help!

I think you are right... It looks good.

 

I did notice the syslog had these three lines

Sep 28 18:59:49 Tower kernel: md: disk0 removed

Sep 28 18:59:49 Tower kernel: md: disk1 missing

Sep 28 18:59:49 Tower kernel: md: disk2 missing

 

It looks as if you assigned the drives at one point in time, but then un-assigned them.

 

To un-raid, un-assigning a drive is exactly the same as if it has failed.  Right now, this is harmless.  Later, once you assign drives, especially if you only assign one at a time it might complain about the missing/un-assigned drive and not let you start the array until it sees all the drives.   If you run into a snag, let us know, as it will help other in the future to know what to expect.

 

Joe L.

 

That's exactly what happened.  I will be assigning the drives to the array later tonight so I'll keep you posted.

Share this post


Link to post

Well, I think I was able to reassign the drives without any issue.  Syslog attached for review just in case!  Ignore the unmenu error as I forgot to use http://tower:8080 and tried to execute unmenu manually when I thought my go script edit failed.  When I use http://192.198.1.3 in my browser then I get the standard UnRAID interface.  Is this normal?

 

Next step is to start copying over data but I'm still researching the best method as well as using user shares for streaming to my PCH NMT A-110.  I'm thinking of trying to copy over the data "manually" and then setting up the user shares.  Does this make sense?  Also, I'm leaving out my parity drive for the initial data exchange and will activate this drive once I move over the bulk of the data (approx. 3 TB worth).

Share this post


Link to post

Disk 1 and Disk 2 are assigned at the end of the syslog, and look OK, but the array was not started.  Your syslog looks fine, although it indicates that your gigabit chipset is only connecting at 100Mbps.  That will severely impact your transfer speed, limit it to about 10MB/s, and take about 3 times as long if you were thinking of transferring the 3 TB across the network.

Share this post


Link to post

Disk 1 and Disk 2 are assigned at the end of the syslog, and look OK, but the array was not started. Your syslog looks fine, although it indicates that your gigabit chipset is only connecting at 100Mbps. That will severely impact your transfer speed, limit it to about 10MB/s, and take about 3 times as long if you were thinking of transferring the 3 TB across the network.

 

Thanks Rob! I haven't started the array because I haven't installed by Gigabit switch yet. Still trying to figure out how to assign all of the IP addresses. See Spinbot's thread:

 

http://lime-technology.com/forum/index.php?topic=4426.0

 

EDIT: Ok, I installed the Gigabit switch and the IP addresses are the same as before the switch (all DHCP assigned except my desktop machine) except now http://tower:8080 no longer works. http://192.168.1.3 still gets me to the regular menu. Do I need to change something to get unmenu to work now that the switch is installed??

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.