Unraid Hard Crashes Every 2-3 Days


Recommended Posts

Hey All,

 

I have been using Unraid now for almost 3 years, and, for the most part, have had very little to deal with in the way of issues. With that said, about two months ago I started having an issue with the server locking up and, i cannot, for the life of me, figure out why. Basically, every 2-3 days my entire server just comes to a grinding halt. I can't log into the webGUI, I can't ssh in using putty, and if I happen to have the GUI up, it is completely unresponsive. The only way to get the system back is by doing a hard reset which makes me cringe and then it runs a parity check on reboot. I am running the system headless since it is in a rack mounted case in my network closet.

 

I just replaced the thumb drive containing the OS and also downloaded a fresh copy of unraid to rule out either of those issues, but no change. I have also tried keeping all of my dockers disabled, except for Plex, and that has made no difference. I was using the Plex docker by linuxserver (i think that's what it was called) but while trying to TS i saw that Plex now has an official container so I switched over to that. I have SickRage installed and usually running, a FileBot container, and PlexPy. That's it. I don't run any VM's or anything crazy. This is purely a media streaming server. Other than adding new HDDs and the HBA card, I have not made any changes to the hardware of the server since it was built. The HBA card was added about a year ago and hasn't given me any issues that i know of (once it was flashed to IT mode).

 

I have posted my log files, however, I am never able to retrieve log files prior to the server crashing since I get locked out once it happens. As of this post the server has been up for about 24 hours.

 

My system specs are the following:

CPU: AMD 8350 w/ a 240mm AIO

MoBo: Gigabyte 990FX UD3 rev. 4

RAM: G.Skill - 16GB

NIC1: Onboard Realtek GbE

NIC2: Mellanox 10G SFP+ Adapter (only used to transfer files between my desktop and server)

HBA Card: 9211-8i (has been flashed to newest firmware and changed to IT mode)

Hard Drives: 2 x 8TB Seagate Ironwolf (Dual Parity), 3 x 3TB WD Reds, 1 x 6TB WD Red, 1 x 6TB Seagate Ironwolf, 1 x Crucial 750GB Cache SSD

PSU: EVGA T2 850W 80+ Titanium

Case: Rosewill RSV-L4500 w/ corsair fans

 

TIA!

 

media_server-diagnostics-20180108-1002.zip

Edited by TechnoBabble28
Link to comment

There's a checksum error on your cache device:

 

Jan  8 04:45:32 Media kernel: BTRFS warning (device sdb1): csum failed ino 1322318 off 2701209600 csum 3261635688 expected csum 1375544623

 

You should run a scrub to fin the affected file(s).

 

There's also a segfault with Plex

 

Jan  8 05:07:23 Media kernel: Plex Transcoder[2174]: segfault at 2b562c6d44ec ip 00002b580ab46099 sp 00002b580eedeacc error 4 in libh264_decoder.so[2b580aa80000+154000]

You should run a memtest to see if any RAM issues are detected  and/or try running the server with Plex disable to see if still crashes.

Link to comment

Thanks for the quick reply!

 

I just ran a scrub on sdb1 which is my cache drive but it returned no errors so im not sure what to make of that. Would switching back to XFS for the cache drive make any difference? I do plan on readding a second cache drive in the next few months.

 

btrfs scrub start -BR /mnt/cache
scrub done for b9191c01-a51f-48d1-9951-2a4bd18c831a
        scrub started at Mon Jan  8 11:04:25 2018 and finished after 00:00:48
        data_extents_scrubbed: 375708
        tree_extents_scrubbed: 10835
        data_bytes_scrubbed: 20367876096
        tree_bytes_scrubbed: 177520640
        read_errors: 0
        csum_errors: 0
        verify_errors: 0
        no_csum: 4095617
        csum_discards: 0
        super_errors: 0
        malloc_errors: 0
        uncorrectable_errors: 0
        unverified_errors: 0
        corrected_errors: 0
        last_physical: 530449432576

I will run a memtest when I get home later today and see if it comes back with any errors.

Edited by TechnoBabble28
Link to comment

Well, memtest just finished after about 5.5 hours with no errors. I didn't know what tests to run so i just let it run through its default set of 13 tests and it looks like it did 4 complete passes. Again though, everything came back fine.

 

Not sure where to go from here. I can hit the server with multiple Plex streams all day long but then go to bed and it's non-responsive by the following morning. I don't think it is Plex related, but I can keep all the dockers off for a few days to verify.

Edited by TechnoBabble28
Link to comment

Just a FYI, I ran memtest again overnight just to make sure it was ok and the results were the same.

 

8 hours ago, johnnie.black said:

It's likely hardware related as Plex uses a lot of CPU, it may even be heat related.

 

I can, and routinely have, run 4-5 transcoded streams on the server as i share with family. It has never been an issue before. I doubt it is a heat issue either. I have a Corsair H100 or 110 AIO connected to the CPU. I don't remember the exact model but it's a 240mm radiator.

 

2 hours ago, Leifgg said:

 One thing to check with Plex is how transcoding is setup. Is it to RAM or to disk?

 

I have run it both ways while trying to TS. I believe, at the moment, it is set to the docker default which is disk.

Link to comment
3 minutes ago, TechnoBabble28 said:

Just a FYI, I ran memtest again overnight just to make sure it was ok and the results were the same.

 

 

I can, and routinely have, run 4-5 transcoded streams on the server as i share with family. It has never been an issue before. I doubt it is a heat issue either. I have a Corsair H100 or 110 AIO connected to the CPU. I don't remember the exact model but it's a 240mm radiator.

 

 

I have run it both ways while trying to TS. I believe, at the moment, it is set to the docker default which is disk.

 

I am using same Plex docker as you are and have previously used transcoding to disk so that should work as long as there is enough space. Currently I am transcoding to RAM and that works fine as well but I have around 20 GByte available for Plex. Have also been using the linuxserver Plex docker and that has been working fine as well.

Link to comment
8 hours ago, Leifgg said:

 

I am using same Plex docker as you are and have previously used transcoding to disk so that should work as long as there is enough space. Currently I am transcoding to RAM and that works fine as well but I have around 20 GByte available for Plex. Have also been using the linuxserver Plex docker and that has been working fine as well.

 

This issue started while I was on the linuxserver Plex docker but I have been on the official Plex docker for a couple weeks and it has honestly made no difference. There is something else going on here and I just can't seem to put my finger on it. I don't think it is Docker related but I could be wrong.

 

I was about to say I wish there was a way for UnRAID to dump log files to the flash drive automatically but it looks like it already does that. I have attached a some of my log files from the last 4 files it automatically created. I have no idea what I am looking for so hopefully you guys can pick up on something. I am not sure if they are anonymous or not, but at this point, IDC.

media_server-diagnostics-20171229-0934.zip

media_server-diagnostics-20171231-1505.zip

media_server-diagnostics-20180104-0722.zip

media_server-diagnostics-20171224-1001.zip

Link to comment

More crc errors (in multiple devices) and segfaults, still looks to me like a hardware problem, most likely RAM or board, just took a quick glance at the syslogs, they might be more:


 

Dec 29 00:48:20 Media kernel: BTRFS warning (device sdb1): csum failed ino 806439 off 3295334400 csum 1504564530 expected csum 2921189641
Dec 29 00:48:37 Media kernel: php[16078]: segfault at 62e89ad8 ip 0000000000626490 sp 00007ffe3a553130 error 4 in php[400000+724000]
Dec 29 00:49:43 Media kernel: BTRFS warning (device sdb1): csum failed ino 806439 off 4021080064 csum 441696647 expected csum 984234305

Dec 29 23:13:40 Media kernel: BTRFS warning (device sdb1): csum failed ino 766591 off 615776256 csum 3719539788 expected csum 4111960129
Dec 29 23:18:26 Media kernel: BTRFS warning (device sdb1): csum failed ino 757499 off 1205260288 csum 1830344599 expected csum 503435445
Dec 29 23:23:12 Media kernel: BTRFS warning (device sdb1): csum failed ino 781994 off 1038303232 csum 736970172 expected csum 3116404328
Dec 29 23:24:17 Media kernel: BTRFS warning (device sdb1): csum failed ino 796151 off 1569144832 csum 2155416739 expected csum 3244027644
Dec 29 23:24:23 Media kernel: BTRFS warning (device sdb1): csum failed ino 796870 off 91152384 csum 795294195 expected csum 558944166
Dec 29 23:24:42 Media kernel: BTRFS warning (device sdb1): csum failed ino 797858 off 1382805504 csum 2550106591 expected csum 1259201795
Dec 29 23:27:35 Media kernel: BTRFS warning (device sdb1): csum failed ino 812845 off 871235584 csum 395572638 expected csum 547124521

Dec 30 00:55:49 Media kernel: BTRFS warning (device md5): csum failed ino 367 off 367562752 csum 3121216539 expected csum 3109126183
Dec 30 00:55:49 Media kernel: BTRFS warning (device md5): csum failed ino 367 off 367562752 csum 3121216539 expected csum 3109126183

 

Link to comment
7 hours ago, johnnie.black said:

More crc errors (in multiple devices) and segfaults, still looks to me like a hardware problem, most likely RAM or board, just took a quick glance at the syslogs, they might be more:


 


Dec 29 00:48:20 Media kernel: BTRFS warning (device sdb1): csum failed ino 806439 off 3295334400 csum 1504564530 expected csum 2921189641
Dec 29 00:48:37 Media kernel: php[16078]: segfault at 62e89ad8 ip 0000000000626490 sp 00007ffe3a553130 error 4 in php[400000+724000]
Dec 29 00:49:43 Media kernel: BTRFS warning (device sdb1): csum failed ino 806439 off 4021080064 csum 441696647 expected csum 984234305

Dec 29 23:13:40 Media kernel: BTRFS warning (device sdb1): csum failed ino 766591 off 615776256 csum 3719539788 expected csum 4111960129
Dec 29 23:18:26 Media kernel: BTRFS warning (device sdb1): csum failed ino 757499 off 1205260288 csum 1830344599 expected csum 503435445
Dec 29 23:23:12 Media kernel: BTRFS warning (device sdb1): csum failed ino 781994 off 1038303232 csum 736970172 expected csum 3116404328
Dec 29 23:24:17 Media kernel: BTRFS warning (device sdb1): csum failed ino 796151 off 1569144832 csum 2155416739 expected csum 3244027644
Dec 29 23:24:23 Media kernel: BTRFS warning (device sdb1): csum failed ino 796870 off 91152384 csum 795294195 expected csum 558944166
Dec 29 23:24:42 Media kernel: BTRFS warning (device sdb1): csum failed ino 797858 off 1382805504 csum 2550106591 expected csum 1259201795
Dec 29 23:27:35 Media kernel: BTRFS warning (device sdb1): csum failed ino 812845 off 871235584 csum 395572638 expected csum 547124521

Dec 30 00:55:49 Media kernel: BTRFS warning (device md5): csum failed ino 367 off 367562752 csum 3121216539 expected csum 3109126183
Dec 30 00:55:49 Media kernel: BTRFS warning (device md5): csum failed ino 367 off 367562752 csum 3121216539 expected csum 3109126183

 

Damn. Well I have a two options:

 

I have a spare AM3+ board and a second 8350 just sitting in my garage. I can swap those in and see if that makes any difference. I do not have any extra DDR3 laying around though, so short of pulling memory out of my desktop, I won't have anyway to rule that out.

 

or

 

I can spend ~$500 and make the upgrade to a Ryzen 1700 like i have been wanting to do for a while now. AMD just announced price cuts on all of their Ryzen lineup at CES yesterday so prices should come down even more here fairly shortly. Ive already upgraded the PSU, case, OS drive, and increased my storage so this is really the last big step, at least for a while. What are the chances that this issue remains after replacing mobo, CPU, and RAM?

Edited by TechnoBabble28
Link to comment

Then I will head over to Micro Center in the next few days and pick up some new components. Fingers crossed that takes care of that issue.

 

48 minutes ago, johnnie.black said:

IMO very low.

 

I know there was a similar crashing issue with Ryzen at one point but I need to shoot over to that thread and read up on what exactly that was and if/how i'll need to fix it.

 

Thanks for all the help, johnnie.black! I'll post in a few days if I come across any other problems.

Link to comment

Before you go replacing hardware, I'd start by disabling everything I can in the BIOS. Shut off C1 and C6, disable HPC, disable onboard sound, disable serial port, disable LAN booting... just turn off everything you can find that isn't necessary for your server. See if that has any affect. If that fixes it, that's a much easier solution than ripping your whole server apart.

Link to comment
11 minutes ago, danejaho said:

Before you go replacing hardware, I'd start by disabling everything I can in the BIOS. Shut off C1 and C6, disable HPC, disable onboard sound, disable serial port, disable LAN booting... just turn off everything you can find that isn't necessary for your server. See if that has any affect. If that fixes it, that's a much easier solution than ripping your whole server apart.

 

Most of that has already been disabled. Will need to check on C-states and double check the rest but I will give it a try!

Link to comment

Just an update. On Wednesday afternoon I had double checked BIOS settings to make sure all the unnecessary stuff was turned off and it was. I rebooted the server and left it alone till Thursday morning, where I discovered it had completely locked up on me again, this time after maybe 16 hours. I got tired of dealing with it so I drove to Micro Center and bought a Gigabyte Aorus AX370 Gaming K5 Mobo and a Ryzen 7 1700. I happened to have an 8 GB set of DDR4 2400 laying around already. I built the system and ran it for 24 hours just as a burn in with no drives or OS attached. I did update the MoBo BIOS to F10 which was released in mid December. I'll be adding an additional 16GB of DDR4 next week if everything goes well.

 

As of right now, I have just booted up UnRaid 6.3.5 on the new system for the first time and it recognizes all the sata drives, HBA card, and mellanox card without issue (not surprising). I will be upgrading to the latest RC of UnRaid here in a moment to hopefully prevent some of the issues people have had with Ryzen chips. Once I do that I am going to do a parity check since I have not been able to complete one since I started having my original issue.

 

EDIT: I just upgraded to 6.4 RC21b. Was uneventful as always. However, the GUI is now 100% width which is odd. Not sure if that is by design or just a bug on my end. Anyway, just started a parity check. It usually takes 48 hours to complete so I will check back in at the end of the weekend with an update. I also posted my IOMMU groupings below just in case anyone was curious.

Edited by TechnoBabble28
Link to comment

Here is my IOMMU Grouping

 

PCI Devices and IOMMU Groups

IOMMU group 0:	[1022:1452] 00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge
IOMMU group 1:	[1022:1453] 00:01.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge
IOMMU group 2:	[1022:1452] 00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge
IOMMU group 3:	[1022:1452] 00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge
IOMMU group 4:	[1022:1453] 00:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge
IOMMU group 5:	[1022:1453] 00:03.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge
IOMMU group 6:	[1022:1452] 00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge
IOMMU group 7:	[1022:1452] 00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge
IOMMU group 8:	[1022:1454] 00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B
IOMMU group 9:	[1022:1452] 00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge
IOMMU group 10:	[1022:1454] 00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B
IOMMU group 11:	[1022:790b] 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 59)
	[1022:790e] 00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)
IOMMU group 12:	[1022:1460] 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 0
	[1022:1461] 00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 1
	[1022:1462] 00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 2
	[1022:1463] 00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 3
	[1022:1464] 00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 4
	[1022:1465] 00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 5
	[1022:1466] 00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric Device 18h Function 6
	[1022:1467] 00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 7
IOMMU group 13:	[1022:43b9] 01:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] Device 43b9 (rev 02)
	[1022:43b5] 01:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] Device 43b5 (rev 02)
	[1022:43b0] 01:00.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b0 (rev 02)
	[1022:43b4] 02:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02)
	[1022:43b4] 02:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02)
	[1022:43b4] 02:03.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02)
	[1022:43b4] 02:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02)
	[1b21:1343] 03:00.0 USB controller: ASMedia Technology Inc. Device 1343
	[8086:1539] 04:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)
IOMMU group 14:	[1000:0072] 07:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 02)
IOMMU group 15:	[15b3:6750] 08:00.0 Ethernet controller: Mellanox Technologies MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0)
IOMMU group 16:	[1022:145a] 09:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device 145a
IOMMU group 17:	[1022:1456] 09:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor
IOMMU group 18:	[1022:145c] 09:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) USB 3.0 Host Controller
IOMMU group 19:	[1022:1455] 0a:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device 1455
IOMMU group 20:	[1022:7901] 0a:00.2 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)


CPU Thread Pairings

Pair 1:	cpu 0 / cpu 1
Pair 2:	cpu 2 / cpu 3
Pair 3:	cpu 4 / cpu 5
Pair 4:	cpu 6 / cpu 7
Pair 5:	cpu 8 / cpu 9
Pair 6:	cpu 10 / cpu 11
Pair 7:	cpu 12 / cpu 13
Pair 8:	cpu 14 / cpu 15


USB Devices

Bus 001 Device 001:	ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 001:	ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 003 Device 001:	ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 004 Device 001:	ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 005 Device 001:	ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 005 Device 002:	ID 0781:5571 SanDisk Corp. Cruzer Fit
Bus 006 Device 001:	ID 1d6b:0003 Linux Foundation 3.0 root hub


SCSI Devices

[0:0:0:0]	disk    SanDisk  Cruzer Fit       1.00  /dev/sda   15.6GB
[1:0:0:0]	disk    ATA      ST8000VN0022-2EL SC61  /dev/sdb   8.00TB
[2:0:0:0]	disk    ATA      ST8000DM004-2CX1 0001  /dev/sdc   8.00TB
[3:0:0:0]	disk    ATA      Crucial_CT750MX3 R040  /dev/sdd    750GB
[4:0:0:0]	disk    ATA      WDC WD5000AZLX-0 1A01  /dev/sde    500GB
[11:0:0:0]	disk    ATA      ST6000VN0041-2EL SC61  /dev/sdf   6.00TB
[11:0:1:0]	disk    ATA      WDC WD30EZRX-00D 0A80  /dev/sdg   3.00TB
[11:0:2:0]	disk    ATA      WDC WD30EZRX-00S 0A80  /dev/sdh   3.00TB
[11:0:3:0]	disk    ATA      WDC WD60EFRX-68M 0A82  /dev/sdi   6.00TB
[11:0:4:0]	disk    ATA      WDC WD30EZRX-00S 0A80  /dev/sdj   3.00TB

 

Link to comment

Just reporting back to say that the system did lock up on me initially, but only due to an error on my part. I forgot to disable c-states on the new mobo. I have since corrected that error, successfully completed a parity-sync (which i have not been able to do since the original problem surfaced), and have hammered the system with 10 simultaneous transcodes. This new system hasn't even stuttered. I then tried a 78GB 4K Blu-Ray rip and even then it played with no issue and was only pegging the CPU at ~65%. I am beyond pleased at this point.

 

I did update to 6.4 stable the morning it was released and the system has been up now for 2 days and 12 hours. I am not completely out of the woods yet but this is the most confident I have been in months.

 

Thanks again for everyone's input! I know in the end I just swapped components but I think was about time anyway.

 

Cheers!

Chase

Link to comment
  • 6 months later...

Thanks for the update Chase. I hope it's still working fine now. I found your thread because the last months my server is not as reliable as it used to be and I'm running out of ideas. So after reading yours now I wonder if I also have a hardware issue. I'll see, but I would appreciate if after a few months of using it, you can still confirm the new hardware solved the issue :)

 

Cheers,

Alberto

Link to comment
28 minutes ago, almarma said:

Thanks for the update Chase. I hope it's still working fine now. I found your thread because the last months my server is not as reliable as it used to be and I'm running out of ideas. So after reading yours now I wonder if I also have a hardware issue. I'll see, but I would appreciate if after a few months of using it, you can still confirm the new hardware solved the issue :)

 

Cheers,

Alberto

 

I have been running on 6.5 for a while now but have had no issues with the new system other than a stick of RAM that went bad on me. I do not use the Zen state options but I also don't use any VMs so it's not really an issue. The system has never locked up on me since switching to the new components and has only been down when I am messing with something or doing an update. I've been very happy with everything!

  • Like 1
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.