array start hanging


Recommended Posts

last entries before the latest crash in syslog, unfortunately didn't get anything.

 

Nov 11 02:26:26 unraid root: Total Spundown: 0
Nov 11 02:31:26 unraid root: Total Spundown: 0
Nov 11 02:36:26 unraid root: Total Spundown: 0
Nov 11 02:41:26 unraid root: Total Spundown: 0
Nov 11 02:46:27 unraid root: Total Spundown: 0

 

when i reboot after the crash, the array doesn't finish starting. my normal process is to reboot to safe mode, start the array, stop it, and reboot normally. it'll start then

 

i was curious so I checked the idrac coredump and it shows this around the time it went down:

 

Nov 11 14:06:23 idrac-9CKDVR1 kernel: g_kbdmouse: full speed config #1 for Keyboard/Mouse
Nov 11 14:06:23 idrac-9CKDVR1 kernel: usb_composite_set_config_status_stage() SET_CONFIGURATION sending status stage
Nov 11 14:06:23 idrac-9CKDVR1 kernel: g_composite g_ci13410_fs: full speed config #1
Nov 11 14:07:29 idrac-9CKDVR1 kernel: g_kbdmouse: full speed config #1 for Keyboard/Mouse
Nov 11 14:07:29 idrac-9CKDVR1 kernel: usb_composite_set_config_status_stage() SET_CONFIGURATION sending status stage
Nov 11 14:07:29 idrac-9CKDVR1 kernel: g_composite g_ci13410_fs: full speed config #1

 

when I've been ssh'd in to the unraid box when it it's starting to go down, I noticed there was a kvm process taking 100% cpu that I couldn't kill. could the idrac be doing something messing with the vm's?

 

latest diags attached

unraid-diagnostics-20221111-0721.zip

Link to comment
17 minutes ago, mrbiiggy said:

when i reboot after the crash, the array doesn't finish starting. my normal process is to reboot to safe mode, start the array, stop it, and reboot normally. it'll start then

This is very strange.

 

17 minutes ago, mrbiiggy said:

could the idrac be doing something messing with the vm's?

Difficult for me to say but it's possible.

 

Link to comment
  • 2 weeks later...

hmm i re-seated the drive but it's in a netapp shelf so no sure why it would randomly do that. it's still showing red x.

 

the array is hanging starting when not in maintenance mode again. my workaround starting in maintenance mode then rebooting isn't working anymore. plugins: ca auto turbo write, ca auto update, ca dynamix unlimited width, ca mover tuning, community applications, nerdtools, rclone, tips and tweaks, unassigned devices, unassigned devices preclear, user scripts

 

log

text  error  warn  system  array  login  

Nov 22 09:25:14 unraid  emhttpd: shcmd (114): mount -t btrfs -o noatime,space_cache=v2 /dev/md15 /mnt/disk15
Nov 22 09:25:14 unraid kernel: BTRFS info (device md15): using free space tree
Nov 22 09:25:14 unraid kernel: BTRFS info (device md15): has skinny extents
Nov 22 09:25:27 unraid  emhttpd: shcmd (115): btrfs filesystem resize max /mnt/disk15
Nov 22 09:25:27 unraid root: Resize device id 1 (/dev/md15) from 7.28TiB to max
Nov 22 09:25:27 unraid  emhttpd: shcmd (116): mkdir -p /mnt/disk16
Nov 22 09:25:27 unraid  emhttpd: shcmd (117): mount -t btrfs -o noatime,space_cache=v2 /dev/md16 /mnt/disk16
Nov 22 09:25:27 unraid kernel: BTRFS info (device md16): using free space tree
Nov 22 09:25:27 unraid kernel: BTRFS info (device md16): has skinny extents
Nov 22 09:25:36 unraid  emhttpd: shcmd (118): btrfs filesystem resize max /mnt/disk16
Nov 22 09:25:36 unraid root: Resize device id 1 (/dev/md16) from 9.09TiB to max
Nov 22 09:25:36 unraid  emhttpd: shcmd (119): mkdir -p /mnt/disk17
Nov 22 09:25:36 unraid  emhttpd: shcmd (120): mount -t xfs -o noatime,nouuid /dev/md17 /mnt/disk17
Nov 22 09:25:36 unraid kernel: SGI XFS with ACLs, security attributes, no debug enabled
Nov 22 09:25:36 unraid kernel: XFS (md17): Mounting V5 Filesystem
Nov 22 09:25:37 unraid kernel: XFS (md17): Ending clean mount
Nov 22 09:25:37 unraid kernel: xfs filesystem being mounted at /mnt/disk17 supports timestamps until 2038 (0x7fffffff)
Nov 22 09:25:37 unraid  emhttpd: shcmd (121): xfs_growfs /mnt/disk17
Nov 22 09:25:37 unraid root: meta-data=/dev/md17              isize=512    agcount=10, agsize=268435455 blks
Nov 22 09:25:37 unraid root:          =                       sectsz=512   attr=2, projid32bit=1
Nov 22 09:25:37 unraid root:          =                       crc=1        finobt=1, sparse=1, rmapbt=0
Nov 22 09:25:37 unraid root:          =                       reflink=1    bigtime=0 inobtcount=0
Nov 22 09:25:37 unraid root: data     =                       bsize=4096   blocks=2441609203, imaxpct=5
Nov 22 09:25:37 unraid root:          =                       sunit=0      swidth=0 blks
Nov 22 09:25:37 unraid root: naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
Nov 22 09:25:37 unraid root: log      =internal log           bsize=4096   blocks=521728, version=2
Nov 22 09:25:37 unraid root:          =                       sectsz=512   sunit=0 blks, lazy-count=1
Nov 22 09:25:37 unraid root: realtime =none                   extsz=4096   blocks=0, rtextents=0
Nov 22 09:25:37 unraid  emhttpd: shcmd (122): mkdir -p /mnt/disk18
Nov 22 09:25:37 unraid  emhttpd: shcmd (123): mount -t xfs -o noatime,nouuid /dev/md18 /mnt/disk18
Nov 22 09:25:37 unraid kernel: XFS (md18): Mounting V5 Filesystem
Nov 22 09:25:37 unraid kernel: XFS (md18): Ending clean mount
Nov 22 09:25:37 unraid kernel: xfs filesystem being mounted at /mnt/disk18 supports timestamps until 2038 (0x7fffffff)
Nov 22 09:25:37 unraid  emhttpd: shcmd (124): xfs_growfs /mnt/disk18
Nov 22 09:25:37 unraid root: meta-data=/dev/md18              isize=512    agcount=11, agsize=268435455 blks
Nov 22 09:25:37 unraid root:          =                       sectsz=512   attr=2, projid32bit=1
Nov 22 09:25:37 unraid root:          =                       crc=1        finobt=1, sparse=1, rmapbt=0
Nov 22 09:25:37 unraid root:          =                       reflink=1    bigtime=0 inobtcount=0
Nov 22 09:25:37 unraid root: data     =                       bsize=4096   blocks=2929721331, imaxpct=5
Nov 22 09:25:37 unraid root:          =                       sunit=0      swidth=0 blks
Nov 22 09:25:37 unraid root: naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
Nov 22 09:25:37 unraid root: log      =internal log           bsize=4096   blocks=521728, version=2
Nov 22 09:25:37 unraid root:          =                       sectsz=512   sunit=0 blks, lazy-count=1
Nov 22 09:25:37 unraid root: realtime =none                   extsz=4096   blocks=0, rtextents=0
Nov 22 09:25:37 unraid  emhttpd: shcmd (125): mkdir -p /mnt/disk19
Nov 22 09:25:37 unraid  emhttpd: shcmd (126): mount -t xfs -o noatime,nouuid /dev/md19 /mnt/disk19
Nov 22 09:25:37 unraid kernel: XFS (md19): Mounting V5 Filesystem
Nov 22 09:25:39 unraid kernel: XFS (md19): Ending clean mount
Nov 22 09:25:39 unraid kernel: xfs filesystem being mounted at /mnt/disk19 supports timestamps until 2038 (0x7fffffff)
Nov 22 09:25:39 unraid  emhttpd: shcmd (127): xfs_growfs /mnt/disk19
Nov 22 09:25:39 unraid root: meta-data=/dev/md19              isize=512    agcount=11, agsize=268435455 blks
Nov 22 09:25:39 unraid root:          =                       sectsz=512   attr=2, projid32bit=1
Nov 22 09:25:39 unraid root:          =                       crc=1        finobt=1, sparse=1, rmapbt=0
Nov 22 09:25:39 unraid root:          =                       reflink=1    bigtime=0 inobtcount=0
Nov 22 09:25:39 unraid root: data     =                       bsize=4096   blocks=2929721331, imaxpct=5
Nov 22 09:25:39 unraid root:          =                       sunit=0      swidth=0 blks
Nov 22 09:25:39 unraid root: naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
Nov 22 09:25:39 unraid root: log      =internal log           bsize=4096   blocks=521728, version=2
Nov 22 09:25:39 unraid root:          =                       sectsz=512   sunit=0 blks, lazy-count=1
Nov 22 09:25:39 unraid root: realtime =none                   extsz=4096   blocks=0, rtextents=0
Nov 22 09:25:39 unraid  emhttpd: shcmd (128): mkdir -p /mnt/cache
Nov 22 09:25:39 unraid  emhttpd: shcmd (129): mount -t xfs -o noatime,nouuid /dev/sdz1 /mnt/cache
Nov 22 09:25:39 unraid kernel: XFS (sdz1): Mounting V5 Filesystem
Nov 22 09:25:39 unraid kernel: XFS (sdz1): Ending clean mount
Nov 22 09:25:39 unraid kernel: xfs filesystem being mounted at /mnt/cache supports timestamps until 2038 (0x7fffffff)
Nov 22 09:25:39 unraid  emhttpd: shcmd (130): sync
Nov 22 09:25:39 unraid  emhttpd: shcmd (131): mkdir /mnt/user0
Nov 22 09:25:39 unraid  emhttpd: shcmd (132): /usr/local/sbin/shfs /mnt/user0 -disks 1048574 -o default_permissions,allow_other,noatime  |& logger
Nov 22 09:25:39 unraid  emhttpd: shcmd (133): mkdir /mnt/user
Nov 22 09:25:39 unraid  emhttpd: shcmd (134): /usr/local/sbin/shfs /mnt/user -disks 1048575 -o default_permissions,allow_other,noatime -o remember=330  |& logger
Nov 22 09:25:39 unraid  emhttpd: shcmd (136): /usr/local/sbin/update_cron
Nov 22 09:25:40 unraid root: Starting CA Auto Turbo Mode
Nov 22 09:25:40 unraid  emhttpd: error: hotplug_devices, 1730: No such file or directory (2): Error: tagged device HUH721010AL42C0_2THR3ZPD_35000cca26a609a94 was (sdx) is now (sdb)
Nov 22 09:25:40 unraid  emhttpd: device /dev/sdx problem getting id
Nov 22 09:25:40 unraid  emhttpd: read SMART /dev/sdb
Nov 22 09:25:41 unraid kernel: emhttpd[6250]: segfault at 674 ip 000055d7653169d4 sp 00007fffceb44580 error 4 in emhttpd[55d765304000+21000]
Nov 22 09:25:41 unraid kernel: Code: 8e 27 01 00 48 89 45 f8 48 8d 05 72 27 01 00 48 89 45 f0 e9 79 01 00 00 8b 45 ec 89 c7 e8 89 b1 ff ff 48 89 45 d8 48 8b 45 d8 <8b> 80 74 06 00 00 85 c0 0f 94 c0 0f b6 c0 89 45 d4 48 8b 45 e0 48
Nov 22 09:25:41 unraid kernel: mdcmd (36): set md_write_method 1
Nov 22 09:25:41 unraid kernel: 
Nov 22 09:25:41 unraid unassigned.devices: Mounting 'Auto Mount' Devices...
Nov 22 09:25:41 unraid unassigned.devices: Disk with ID '35000cca26a609a94 (sdb)' is not set to auto mount.
Nov 22 09:25:41 unraid unassigned.devices: Adding partition 'sdc1'...
Nov 22 09:25:41 unraid unassigned.devices: Mounting partition 'sdc1' at mountpoint '/mnt/disks/dl'...
Nov 22 09:25:41 unraid unassigned.devices: Mount drive command: /sbin/mount -t 'xfs' -o rw,noatime,nodiratime,discard '/dev/sdc1' '/mnt/disks/dl'
Nov 22 09:25:41 unraid kernel: XFS (sdc1): Mounting V5 Filesystem
Nov 22 09:25:41 unraid kernel: XFS (sdc1): Ending clean mount
Nov 22 09:25:41 unraid kernel: xfs filesystem being mounted at /mnt/disks/dl supports timestamps until 2038 (0x7fffffff)
Nov 22 09:25:42 unraid unassigned.devices: Successfully mounted 'sdc1' on '/mnt/disks/dl'.
Nov 22 09:25:42 unraid unassigned.devices: Disk with ID 'ST8000DM004-2CX188_WCT0270C (sdp)' is not set to auto mount.
Nov 22 09:25:42 unraid  rsyslogd: [origin software="rsyslogd" swVersion="8.2102.0" x-pid="15225" x-info="https://www.rsyslog.com"] start
Nov 22 09:28:30 unraid  ntpd[1669]: kernel reports TIME_ERROR: 0x41: Clock Unsynchronized

 

Link to comment

i moved the parity to another bay but getting read errors again

 

 

Nov 25 18:01:53 unraid kernel: mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
Nov 25 18:01:53 unraid kernel: mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
Nov 25 18:01:53 unraid kernel: mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
Nov 25 18:01:53 unraid kernel: mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
Nov 25 18:01:53 unraid kernel: mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
Nov 25 18:01:53 unraid kernel: sd 3:0:13:0: [sdo] tag#9752 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=9s
Nov 25 18:01:53 unraid kernel: sd 3:0:13:0: [sdo] tag#9752 Sense Key : 0x3 [current] [descriptor] 
Nov 25 18:01:53 unraid kernel: sd 3:0:13:0: [sdo] tag#9752 ASC=0x11 ASCQ=0x0 
Nov 25 18:01:53 unraid kernel: sd 3:0:13:0: [sdo] tag#9752 CDB: opcode=0x88 88 00 00 00 00 04 ab 9d 75 10 00 00 03 70 00 00
Nov 25 18:01:53 unraid kernel: critical medium error, dev sdo, sector 20059092240 op 0x0:(READ) flags 0x0 phys_seg 110 prio class 0
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092176
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092184
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092192
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092200
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092208
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092216
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092224
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092232
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092240
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092248
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092256
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092264
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092272
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092280
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092288
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092296
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092304
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092312
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092320
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092328
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092336
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092344
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092352
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092360
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092368
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092376
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092384
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092392
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092400
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092408
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092416
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092424
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092432
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092440
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092448
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092456
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092464
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092472
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092480
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092488
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092496
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092504
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092512
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092520
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092528
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092536
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092544
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092552
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092560
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092568
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092576
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092584
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092592
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092600
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092608
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092616
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092624
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092632
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092640
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092648
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092656
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092664
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092672
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092680
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092688
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092696
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092704
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092712
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092720
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092728
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092736
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092744
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092752
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092760
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092768
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092776
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092784
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092792
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092800
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092808
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092816
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092824
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092832
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092840
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092848
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092856
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092864
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092872
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092880
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092888
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092896
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092904
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092912
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092920
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092928
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092936
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092944
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092952
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092960
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092968
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092976
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092984
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059092992
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059093000
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059093008
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059093016
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059093024
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059093032
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059093040
Nov 25 18:01:53 unraid kernel: md: disk0 read error, sector=20059093048

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.