Frequent loop2 corruptions (XFS)


Recommended Posts

My loop2 (docker image) gets corrupted every month or more frequently, I have as well 1 VM in cache drive but it always affects to loop2. What I always do is to rebuild the docker image but I'm getting "tired".

 

I have already checked my RAM twice, and lower the freq in the BIOS, is not ECC.

 

I need some help understanting this piece of log the call trace and the BTFRS errors, although I use XFS in cache and array.

 

Considering the errors and warnings in the log, and the diagnistic file could someone please help me to troubleshoot this?

 

Thanks in adance

May 16 17:39:26 Unraid kernel: ------------[ cut here ]------------
May 16 17:39:26 Unraid kernel: WARNING: CPU: 3 PID: 31618 at net/netfilter/nf_conntrack_core.c:976 __nf_conntrack_confirm+0x9e/0x223
May 16 17:39:26 Unraid kernel: Modules linked in: veth xt_nat xt_CHECKSUM ipt_REJECT ip6table_mangle ip6table_nat iptable_mangle macvlan ip6table_filter ip6_tables vhost_net tun vhost tap xt_MASQUERADE iptable_filter iptable_nat nf_nat ip_tables xfs md_mod wireguard ip6_udp_tunnel udp_tunnel edac_mce_amd kvm_amd ccp kvm wmi_bmof crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel crypto_simd cryptd igb glue_helper r8169 i2c_piix4 k10temp nvme i2c_algo_bit i2c_core realtek nvme_core ahci video libahci backlight button wmi acpi_cpufreq
May 16 17:39:26 Unraid kernel: CPU: 3 PID: 31618 Comm: kworker/3:1 Not tainted 5.5.8-Unraid #1
May 16 17:39:26 Unraid kernel: Hardware name: Gigabyte Technology Co., Ltd. AB350M-Gaming 3/AB350M-Gaming 3-CF, BIOS F50a 11/27/2019
May 16 17:39:26 Unraid kernel: Workqueue: events macvlan_process_broadcast [macvlan]
May 16 17:39:26 Unraid kernel: RIP: 0010:__nf_conntrack_confirm+0x9e/0x223
May 16 17:39:26 Unraid kernel: Code: e8 b0 e0 ff ff 44 89 fa 89 c6 41 89 c4 48 c1 eb 20 89 df 41 89 de e8 6e de ff ff 84 c0 75 bc 48 8b 85 80 00 00 00 a8 08 74 18 <0f> 0b 89 df 44 89 e6 31 db e8 ce dc ff ff e8 e3 dd ff ff e9 5c 01
May 16 17:39:26 Unraid kernel: RSP: 0018:ffffc9000029cdd8 EFLAGS: 00010202
May 16 17:39:26 Unraid kernel: RAX: 0000000000000188 RBX: 0000000000004b2a RCX: 00000000ffab0327
May 16 17:39:26 Unraid kernel: RDX: 0000000000000001 RSI: ffffc9000029cd9c RDI: ffffffff820094a8
May 16 17:39:26 Unraid kernel: RBP: ffff888399eeed00 R08: 0000000005d17001 R09: ffff888554cf0660
May 16 17:39:26 Unraid kernel: R10: 0000000000000000 R11: ffff8887f7311000 R12: 00000000000052ed
May 16 17:39:26 Unraid kernel: R13: ffffffff8208fb80 R14: 0000000000004b2a R15: 0000000000000000
May 16 17:39:26 Unraid kernel: FS:  0000000000000000(0000) GS:ffff8887feac0000(0000) knlGS:0000000000000000
May 16 17:39:26 Unraid kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 16 17:39:26 Unraid kernel: CR2: 00005622b1c1ac78 CR3: 00000007f7bc4000 CR4: 00000000003406e0
May 16 17:39:26 Unraid kernel: Call Trace:
May 16 17:39:26 Unraid kernel: <IRQ>
May 16 17:39:26 Unraid kernel: nf_conntrack_confirm+0x2f/0x36
May 16 17:39:26 Unraid kernel: nf_hook_slow+0x39/0x8e
May 16 17:39:26 Unraid kernel: nf_hook.constprop.0+0xa5/0xc8
May 16 17:39:26 Unraid kernel: ? ip_protocol_deliver_rcu+0x10d/0x10d
May 16 17:39:26 Unraid kernel: ip_local_deliver+0x49/0x75
May 16 17:39:26 Unraid kernel: __netif_receive_skb_one_core+0x6f/0x90
May 16 17:39:26 Unraid kernel: process_backlog+0x9b/0x12e
May 16 17:39:26 Unraid kernel: net_rx_action+0xf3/0x277
May 16 17:39:26 Unraid kernel: __do_softirq+0xc4/0x1c2
May 16 17:39:26 Unraid kernel: do_softirq_own_stack+0x2a/0x40
May 16 17:39:26 Unraid kernel: </IRQ>
May 16 17:39:26 Unraid kernel: do_softirq+0x3a/0x44
May 16 17:39:26 Unraid kernel: netif_rx_ni+0x1c/0x22
May 16 17:39:26 Unraid kernel: macvlan_broadcast+0x112/0x140 [macvlan]
May 16 17:39:26 Unraid kernel: macvlan_process_broadcast+0xea/0x128 [macvlan]
May 16 17:39:26 Unraid kernel: process_one_work+0x13c/0x1d5
May 16 17:39:26 Unraid kernel: worker_thread+0x19a/0x23e
May 16 17:39:26 Unraid kernel: ? rescuer_thread+0x28a/0x28a
May 16 17:39:26 Unraid kernel: kthread+0xdf/0xe4
May 16 17:39:26 Unraid kernel: ? kthread_unpark+0x52/0x52
May 16 17:39:26 Unraid kernel: ret_from_fork+0x22/0x40
May 16 17:39:26 Unraid kernel: ---[ end trace 58ec1f435fdebd94 ]---
May 16 20:23:07 Unraid kernel: mdcmd (46): spindown 2
....
....
....
May 21 03:40:01 Unraid crond[2042]: exit status 1 from user root /usr/local/sbin/mover &> /dev/null
May 21 04:00:02 Unraid Docker Auto Update: Community Applications Docker Autoupdate running
May 21 04:00:02 Unraid Docker Auto Update: Checking for available updates
May 21 04:04:17 Unraid Docker Auto Update: Stopping Bazarr
May 21 04:04:24 Unraid kernel: vethfdf0fa1: renamed from eth0
May 21 04:04:24 Unraid Docker Auto Update: Stopping lidarr
May 21 04:04:27 Unraid kernel: vethafefe98: renamed from eth0
May 21 04:04:28 Unraid Docker Auto Update: Stopping Nextcloud
May 21 04:04:31 Unraid kernel: vetha7a7097: renamed from eth0
May 21 04:04:31 Unraid Docker Auto Update: Stopping Jackett
May 21 04:04:35 Unraid kernel: veth72ca047: renamed from eth0
May 21 04:04:35 Unraid Docker Auto Update: Stopping Netdata
May 21 04:04:36 Unraid kernel: veth2785cf5: renamed from eth0
May 21 04:04:36 Unraid Docker Auto Update: Stopping Plex
May 21 04:04:40 Unraid kernel: veth7321982: renamed from eth0
May 21 04:04:40 Unraid Docker Auto Update: Installing Updates for Bazarr lidarr Nextcloud Jackett Netdata Plex Statping
May 21 04:04:57 Unraid kernel: ------------[ cut here ]------------
May 21 04:04:57 Unraid kernel: WARNING: CPU: 6 PID: 6922 at fs/btrfs/extent-tree.c:897 lookup_inline_extent_backref+0x19a/0x48e
May 21 04:04:57 Unraid kernel: Modules linked in: veth xt_nat xt_CHECKSUM ipt_REJECT ip6table_mangle ip6table_nat iptable_mangle macvlan ip6table_filter ip6_tables vhost_net tun vhost tap xt_MASQUERADE iptable_filter iptable_nat nf_nat ip_tables xfs md_mod wireguard ip6_udp_tunnel udp_tunnel edac_mce_amd kvm_amd ccp kvm wmi_bmof crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel crypto_simd cryptd igb glue_helper r8169 i2c_piix4 k10temp nvme i2c_algo_bit i2c_core realtek nvme_core ahci video libahci backlight button wmi acpi_cpufreq
May 21 04:04:57 Unraid kernel: CPU: 6 PID: 6922 Comm: dockerd Tainted: G        W         5.5.8-Unraid #1
May 21 04:04:57 Unraid kernel: Hardware name: Gigabyte Technology Co., Ltd. AB350M-Gaming 3/AB350M-Gaming 3-CF, BIOS F50a 11/27/2019
May 21 04:04:57 Unraid kernel: RIP: 0010:lookup_inline_extent_backref+0x19a/0x48e
May 21 04:04:57 Unraid kernel: Code: 7c 24 5f a8 75 ba 4c 39 6c 24 60 75 b3 31 db 45 31 e4 45 85 e4 74 20 f6 84 24 c0 00 00 00 01 0f 84 bd 02 00 00 45 85 e4 74 0d <0f> 0b 41 bc fb ff ff ff e9 8e 02 00 00 4c 8b 6d 00 8b 75 40 4c 89
May 21 04:04:57 Unraid kernel: RSP: 0018:ffffc90000ea7898 EFLAGS: 00010202
May 21 04:04:57 Unraid kernel: RAX: 0000000000000001 RBX: 0000000000000000 RCX: 0000000000000000
May 21 04:04:57 Unraid kernel: RDX: 0000000000000001 RSI: 0000000000000001 RDI: 000000041d2c7000
May 21 04:04:57 Unraid kernel: RBP: ffff88855c6de770 R08: 0000000000000000 R09: 0000000000000000
May 21 04:04:57 Unraid kernel: R10: 0000000000000003 R11: 0000000000000004 R12: 0000000000000001
May 21 04:04:57 Unraid kernel: R13: 000000000000ffff R14: ffff8887b0c00c30 R15: ffff8887cbc00000
May 21 04:04:57 Unraid kernel: FS:  00001535de3f1700(0000) GS:ffff8887feb80000(0000) knlGS:0000000000000000
May 21 04:04:57 Unraid kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 21 04:04:57 Unraid kernel: CR2: 00000000f54ac480 CR3: 00000007ae8bc000 CR4: 00000000003406e0
May 21 04:04:57 Unraid kernel: Call Trace:
May 21 04:04:57 Unraid kernel: insert_inline_extent_backref+0x4d/0xd0
May 21 04:04:57 Unraid kernel: __btrfs_inc_extent_ref+0x87/0x1fb
May 21 04:04:57 Unraid kernel: __btrfs_run_delayed_refs+0x643/0xab4
May 21 04:04:57 Unraid kernel: btrfs_run_delayed_refs+0x56/0x124
May 21 04:04:57 Unraid kernel: btrfs_commit_transaction+0x57/0x81b
May 21 04:04:57 Unraid kernel: ? start_transaction+0x2f0/0x393
May 21 04:04:57 Unraid kernel: btrfs_mksubvol+0x342/0x454
May 21 04:04:57 Unraid kernel: ? step_into+0x5d/0x1b2
May 21 04:04:57 Unraid kernel: btrfs_ioctl_snap_create_transid+0x15c/0x18d
May 21 04:04:57 Unraid kernel: btrfs_ioctl_snap_create_v2+0x111/0x173
May 21 04:04:57 Unraid kernel: btrfs_ioctl+0x84b/0x2a8a
May 21 04:04:57 Unraid kernel: ? _cond_resched+0x1b/0x1e
May 21 04:04:57 Unraid kernel: ? slab_pre_alloc_hook+0x2c/0x53
May 21 04:04:57 Unraid kernel: ? _copy_to_user+0x2d/0x34
May 21 04:04:57 Unraid kernel: ? cp_new_stat+0x14b/0x17a
May 21 04:04:57 Unraid kernel: ? vfs_ioctl+0x19/0x26
May 21 04:04:57 Unraid kernel: vfs_ioctl+0x19/0x26
May 21 04:04:57 Unraid kernel: do_vfs_ioctl+0x577/0x5a1
May 21 04:04:57 Unraid kernel: ? __do_sys_newfstat+0x3c/0x5f
May 21 04:04:57 Unraid kernel: ksys_ioctl+0x37/0x56
May 21 04:04:57 Unraid kernel: __x64_sys_ioctl+0x11/0x14
May 21 04:04:57 Unraid kernel: do_syscall_64+0x7a/0x87
May 21 04:04:57 Unraid kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
May 21 04:04:57 Unraid kernel: RIP: 0033:0x4b3490
May 21 04:04:57 Unraid kernel: Code: 8b 7c 24 10 48 8b 74 24 18 48 8b 54 24 20 49 c7 c2 00 00 00 00 49 c7 c0 00 00 00 00 49 c7 c1 00 00 00 00 48 8b 44 24 08 0f 05 <48> 3d 01 f0 ff ff 76 20 48 c7 44 24 28 ff ff ff ff 48 c7 44 24 30
May 21 04:04:57 Unraid kernel: RSP: 002b:000000c001cc0d08 EFLAGS: 00000216 ORIG_RAX: 0000000000000010
May 21 04:04:57 Unraid kernel: RAX: ffffffffffffffda RBX: 000000c000058000 RCX: 00000000004b3490
May 21 04:04:57 Unraid kernel: RDX: 000000c0013f2000 RSI: 0000000050009417 RDI: 000000000000004a
May 21 04:04:57 Unraid kernel: RBP: 000000c001cc0d90 R08: 0000000000000000 R09: 0000000000000000
May 21 04:04:57 Unraid kernel: R10: 0000000000000000 R11: 0000000000000216 R12: 0000000000000000
May 21 04:04:57 Unraid kernel: R13: 0000000000000002 R14: 0000000000000002 R15: 0000000000000002
May 21 04:04:57 Unraid kernel: ---[ end trace 58ec1f435fdebd95 ]---
May 21 04:04:57 Unraid kernel: BTRFS: error (device loop2) in btrfs_run_delayed_refs:2209: errno=-5 IO failure
May 21 04:04:57 Unraid kernel: BTRFS info (device loop2): forced readonly
May 21 04:05:11 Unraid Docker Auto Update: Restarting Bazarr
May 21 04:05:11 Unraid Docker Auto Update: Restarting lidarr
May 21 04:05:11 Unraid Docker Auto Update: Restarting Nextcloud
May 21 04:05:11 Unraid Docker Auto Update: Restarting Jackett
May 21 04:05:11 Unraid Docker Auto Update: Restarting Netdata
May 21 04:05:11 Unraid Docker Auto Update: Restarting Plex
May 21 04:05:22 Unraid Docker Auto Update: Community Applications Docker Autoupdate finished
May 21 06:00:01 Unraid Plugin Auto Update: Checking for available plugin updates
May 21 06:00:58 Unraid Plugin Auto Update: Community Applications Plugin Auto Update finished

 

unraid-diagnostics-20200521-1601.zip

Link to comment
1 hour ago, johnnie.black said:

See here, you are overclocking the RAM, and most likely what's casing the problem, it's a known issue with Ryzen, even if no errors on memtest it can still corrupt data, usually also noticeable when you do a parity check by finding sync errors.

The memory runs at stock at 3000Mhz now it was running at 2933Mhz

New bios versions (I have the latest) improved the memory compatibility and basically any board with a recent bios can run memories up to 3200Mhz.

Anyway to will under clock it to 2400Mhz to see if it improves something.

 

In any case can you identify in the logs something that may indicate that the problem might not be the memory? I have been copy pasting sections of it in google but I can't get any clear conclusion.

Edited by L0rdRaiden
Link to comment

Yes I have read it and I knew it but I'm not overcloking my RAM or my CPU

But maybe operating memory at a frequency higher than the infinity fabric of chip may cause problems.  https://www.tomshardware.com/reviews/amd-ryzen-5-1600x-cpu-review,5014-2.html

But nothing in my setup is being overclocked or over voltage.

As I said I have lower it down to 2400 to see how it goes.

 

Edited by L0rdRaiden
Link to comment
4 hours ago, johnnie.black said:

It's running above the officially supported speed, I consider that overclocking, ant it's a known issue, so regardless of what you call it it's where I would start.

 

 

Memory now is running at 2400Mhz and still getting call trace errors, is there a way to troubleshoot more and find the root cause?

 

imagen.thumb.png.8b65be444d17b81d8f16456bafa8c12f.png

Link to comment
  • 3 weeks later...

Same problem again, MEMTEST 86 from passmark says RAM is OK. RAM Freq now is at 2400Mhz

 

 

Jun  9 12:01:09 Unraid webGUI: Successful login user root from 192.168.1.30
Jun  9 12:01:16 Unraid emhttpd: cmd: /usr/local/emhttp/plugins/dynamix/scripts/tail_log syslog
Jun  9 12:01:53 Unraid kernel: veth6af80b0: renamed from eth0
Jun  9 12:01:53 Unraid kernel: eth0: renamed from veth4bba7ca
Jun  9 13:02:16 Unraid kernel: BUG: unable to handle kernel paging request at fffff8ef07828908
Jun  9 13:02:16 Unraid kernel: PGD 0 P4D 0
Jun  9 13:02:16 Unraid kernel: Oops: 0000 [#1] SMP NOPTI
Jun  9 13:02:16 Unraid kernel: CPU: 7 PID: 26153 Comm: jackett Tainted: G        W  O      4.19.107-Unraid #1
Jun  9 13:02:16 Unraid kernel: Hardware name: Gigabyte Technology Co., Ltd. AB350M-Gaming 3/AB350M-Gaming 3-CF, BIOS F50a 11/27/2019
Jun  9 13:02:16 Unraid kernel: RIP: 0010:unmap_page_range+0x6ac/0x931
Jun  9 13:02:16 Unraid kernel: Code: 0c ff 8c 24 90 00 00 00 e9 8d 00 00 00 48 b8 ff ff ff ff ff ff ff 01 be f5 ff 7f 00 48 21 f8 48 c1 e6 29 48 c1 e0 06 48 01 f0 <4c> 8b 40 08 48 89 c6 41 f6 c0 01 74 04 49 8d 70 ff 4c 8b 46 08 41
Jun  9 13:02:16 Unraid kernel: RSP: 0018:ffffc9000ae07b90 EFLAGS: 00010286
Jun  9 13:02:16 Unraid kernel: RAX: fffff8ef07828900 RBX: ffffc9000ae07c98 RCX: 0000000000000000
Jun  9 13:02:16 Unraid kernel: RDX: ffff8887c3ebb620 RSI: ffffea0000000000 RDI: 3e00003bbc1e0a24
Jun  9 13:02:16 Unraid kernel: RBP: 0000145e69c24000 R08: 0000000000023eb8 R09: ffff888006a73080
Jun  9 13:02:16 Unraid kernel: R10: 8080808080808080 R11: ffff8887fe9dfb40 R12: 0000145e69cf0000
Jun  9 13:02:16 Unraid kernel: R13: ffff88810e85cbc8 R14: 00000002fa882025 R15: ffffea000bea2080
Jun  9 13:02:16 Unraid kernel: FS:  0000145e6ef82700(0000) GS:ffff8887fe9c0000(0000) knlGS:0000000000000000
Jun  9 13:02:16 Unraid kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jun  9 13:02:16 Unraid kernel: CR2: fffff8ef07828908 CR3: 0000000100298000 CR4: 00000000003406e0
Jun  9 13:02:16 Unraid kernel: Call Trace:
Jun  9 13:02:16 Unraid kernel: unmap_vmas+0x50/0x84
Jun  9 13:02:16 Unraid kernel: exit_mmap+0xb2/0x15d
Jun  9 13:02:16 Unraid kernel: ? finish_task_switch+0xbc/0x222
Jun  9 13:02:16 Unraid kernel: mmput+0x4d/0xe5
Jun  9 13:02:16 Unraid kernel: do_exit+0x3e6/0x922
Jun  9 13:02:16 Unraid kernel: do_group_exit+0x9a/0x9a
Jun  9 13:02:16 Unraid kernel: get_signal+0x514/0x546
Jun  9 13:02:16 Unraid kernel: do_signal+0x31/0x589
Jun  9 13:02:16 Unraid kernel: ? put_unused_fd+0x32/0x42
Jun  9 13:02:16 Unraid kernel: exit_to_usermode_loop+0x30/0xa2
Jun  9 13:02:16 Unraid kernel: do_syscall_64+0xdf/0xf2
Jun  9 13:02:16 Unraid kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
Jun  9 13:02:16 Unraid kernel: RIP: 0033:0x145e70f11dae
Jun  9 13:02:16 Unraid kernel: Code: Bad RIP value.
Jun  9 13:02:16 Unraid kernel: RSP: 002b:0000145e6ef81740 EFLAGS: 00000293 ORIG_RAX: 0000000000000101
Jun  9 13:02:16 Unraid kernel: RAX: fffffffffffffe00 RBX: 00000000006769f0 RCX: 0000145e70f11dae
Jun  9 13:02:16 Unraid kernel: RDX: 0000000000000000 RSI: 00000000006769fc RDI: 00000000ffffff9c
Jun  9 13:02:16 Unraid kernel: RBP: 0000145e6ef817c0 R08: 0000000000000000 R09: 00000000006c2f50
Jun  9 13:02:16 Unraid kernel: R10: 0000000000000000 R11: 0000000000000293 R12: 00000000006769c8
Jun  9 13:02:16 Unraid kernel: R13: 0000000000676910 R14: 0000000000000000 R15: 00000000006769f0
Jun  9 13:02:16 Unraid kernel: Modules linked in: veth xt_nat xt_CHECKSUM ipt_REJECT ip6table_mangle ip6table_nat nf_nat_ipv6 iptable_mangle macvlan ip6table_filter ip6_tables vhost_net tun vhost tap ipt_MASQUERADE iptable_filter iptable_nat nf_nat_ipv4 nf_nat ip_tables xfs md_mod edac_mce_amd kvm_amd ccp kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd cryptd i2c_piix4 wmi_bmof i2c_core glue_helper k10temp r8169 igb(O) realtek nvme nvme_core video ahci backlight libahci wmi button pcc_cpufreq acpi_cpufreq
Jun  9 13:02:16 Unraid kernel: CR2: fffff8ef07828908
Jun  9 13:02:16 Unraid kernel: ---[ end trace 8cdcbad0b47b0310 ]---
Jun  9 13:02:16 Unraid kernel: RIP: 0010:unmap_page_range+0x6ac/0x931
Jun  9 13:02:16 Unraid kernel: Code: 0c ff 8c 24 90 00 00 00 e9 8d 00 00 00 48 b8 ff ff ff ff ff ff ff 01 be f5 ff 7f 00 48 21 f8 48 c1 e6 29 48 c1 e0 06 48 01 f0 <4c> 8b 40 08 48 89 c6 41 f6 c0 01 74 04 49 8d 70 ff 4c 8b 46 08 41
Jun  9 13:02:16 Unraid kernel: RSP: 0018:ffffc9000ae07b90 EFLAGS: 00010286
Jun  9 13:02:16 Unraid kernel: RAX: fffff8ef07828900 RBX: ffffc9000ae07c98 RCX: 0000000000000000
Jun  9 13:02:16 Unraid kernel: RDX: ffff8887c3ebb620 RSI: ffffea0000000000 RDI: 3e00003bbc1e0a24
Jun  9 13:02:16 Unraid kernel: RBP: 0000145e69c24000 R08: 0000000000023eb8 R09: ffff888006a73080
Jun  9 13:02:16 Unraid kernel: R10: 8080808080808080 R11: ffff8887fe9dfb40 R12: 0000145e69cf0000
Jun  9 13:02:16 Unraid kernel: R13: ffff88810e85cbc8 R14: 00000002fa882025 R15: ffffea000bea2080
Jun  9 13:02:16 Unraid kernel: FS:  0000145e6ef82700(0000) GS:ffff8887fe9c0000(0000) knlGS:0000000000000000
Jun  9 13:02:16 Unraid kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jun  9 13:02:16 Unraid kernel: CR2: 0000145e70f11d84 CR3: 0000000100298000 CR4: 00000000003406e0
Jun  9 13:02:16 Unraid kernel: Fixing recursive fault but reboot is needed!

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.