Disk 12 - ST3000NM0033_SA_Z1Y1X1PX (sdp) (errors 341)


xtrap225

Recommended Posts

so i got this email from the server when i work up this morning.

 

Event: Unraid array errors
Subject: Warning [WORK-PC] - array has errors
Description: Array has 1 disk with read errors
Importance: warning

Disk 12 - ST3000NM0033_SA_Z1Y1X1PX (sdp) (errors 341)

however the drive is 'green' and nothing seems to be broken on the server.

 

here is the 'disk log' and attached is the diagnostics

 

Aug 17 16:25:46 WORK-PC emhttpd: ST3000NM0033_SA_Z1Y1X1PX (sdp) 512 5860533168
Aug 17 16:25:56 WORK-PC emhttpd: ST3000NM0033_SA_Z1Y1X1PX (sdp) 512 5860533168
Aug 17 16:26:01 WORK-PC emhttpd: ST3000NM0033_SA_Z1Y1X1PX (sdp) 512 5860533168
Aug 17 16:26:10 WORK-PC emhttpd: ST3000NM0033_SA_Z1Y1X1PX (sdp) 512 5860533168
Aug 17 16:26:10 WORK-PC kernel: mdcmd (13): import 12 sdp 64 2930266532 0 ST3000NM0033_SA_Z1Y1X1PX
Aug 17 16:26:10 WORK-PC kernel: md: import disk12: (sdp) ST3000NM0033_SA_Z1Y1X1PX size: 2930266532
Aug 17 16:26:22 WORK-PC emhttpd: ST3000NM0033_SA_Z1Y1X1PX (sdp) 512 5860533168
Aug 17 16:26:22 WORK-PC kernel: mdcmd (13): import 12 sdp 64 2930266532 0 ST3000NM0033_SA_Z1Y1X1PX
Aug 17 16:26:22 WORK-PC kernel: md: import disk12: (sdp) ST3000NM0033_SA_Z1Y1X1PX size: 2930266532
Aug 17 16:26:26 WORK-PC emhttpd: ST3000NM0033_SA_Z1Y1X1PX (sdp) 512 5860533168
Aug 17 16:26:26 WORK-PC kernel: mdcmd (13): import 12 sdp 64 2930266532 0 ST3000NM0033_SA_Z1Y1X1PX
Aug 17 16:26:26 WORK-PC kernel: md: import disk12: (sdp) ST3000NM0033_SA_Z1Y1X1PX size: 2930266532
Aug 17 16:26:50 WORK-PC emhttpd: shcmd (327): /usr/local/sbin/set_ncq sdp 1
Aug 17 16:26:50 WORK-PC root: set_ncq: setting sdp queue_depth to 1
Aug 17 16:26:50 WORK-PC emhttpd: shcmd (328): echo 128 > /sys/block/sdp/queue/nr_requests
Aug 17 17:01:44 WORK-PC emhttpd: shcmd (671): /usr/local/sbin/set_ncq sdp 1
Aug 17 17:01:44 WORK-PC emhttpd: shcmd (672): echo 128 > /sys/block/sdp/queue/nr_requests
Aug 17 17:08:36 WORK-PC emhttpd: shcmd (1055): /usr/local/sbin/set_ncq sdp 1
Aug 17 17:08:36 WORK-PC emhttpd: shcmd (1056): echo 128 > /sys/block/sdp/queue/nr_requests
Aug 17 23:15:50 WORK-PC emhttpd: writing GPT on disk (sdp), with partition 1 byte offset 32K, erased: 0
Aug 17 23:15:50 WORK-PC emhttpd: shcmd (1167): sgdisk -Z /dev/sdp
Aug 17 23:15:50 WORK-PC kernel: sdp: sdp1 sdp2
Aug 17 23:15:51 WORK-PC emhttpd: shcmd (1168): sgdisk -o -a 8 -n 1:32K:0 /dev/sdp
Aug 17 23:15:52 WORK-PC kernel: sdp: sdp1
Aug 25 18:02:47 WORK-PC root: #012/dev/sdp:#012 drive state is: unknown
Aug 25 18:07:47 WORK-PC root: #012/dev/sdp:#012 drive state is: unknown
Sep 5 08:50:06 WORK-PC kernel: sd 7:0:14:0: [sdp] tag#15 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08
Sep 5 08:50:06 WORK-PC kernel: sd 7:0:14:0: [sdp] tag#15 Sense Key : 0x3 [current] [descriptor]
Sep 5 08:50:06 WORK-PC kernel: sd 7:0:14:0: [sdp] tag#15 ASC=0x11 ASCQ=0x0
Sep 5 08:50:06 WORK-PC kernel: sd 7:0:14:0: [sdp] tag#15 CDB: opcode=0x88 88 00 00 00 00 00 ed 59 a9 48 00 00 04 00 00 00
Sep 5 08:50:06 WORK-PC kernel: print_req_error: critical medium error, dev sdp, sector 3982076882
Sep 5 08:50:09 WORK-PC kernel: sd 7:0:14:0: [sdp] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08
Sep 5 08:50:09 WORK-PC kernel: sd 7:0:14:0: [sdp] tag#0 Sense Key : 0x3 [current] [descriptor]
Sep 5 08:50:09 WORK-PC kernel: sd 7:0:14:0: [sdp] tag#0 ASC=0x11 ASCQ=0x0
Sep 5 08:50:09 WORK-PC kernel: sd 7:0:14:0: [sdp] tag#0 CDB: opcode=0x88 88 00 00 00 00 00 ed 59 ad 48 00 00 04 00 00 00
Sep 5 08:50:09 WORK-PC kernel: print_req_error: critical medium error, dev sdp, sector 3982077532
Sep 5 08:50:12 WORK-PC kernel: sd 7:0:14:0: [sdp] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08
Sep 5 08:50:12 WORK-PC kernel: sd 7:0:14:0: [sdp] tag#0 Sense Key : 0x3 [current] [descriptor]
Sep 5 08:50:12 WORK-PC kernel: sd 7:0:14:0: [sdp] tag#0 ASC=0x11 ASCQ=0x0
Sep 5 08:50:12 WORK-PC kernel: sd 7:0:14:0: [sdp] tag#0 CDB: opcode=0x88 88 00 00 00 00 00 ed 59 b1 48 00 00 04 00 00 00
Sep 5 08:50:12 WORK-PC kernel: print_req_error: critical medium error, dev sdp, sector 3982078413
Sep 5 08:50:21 WORK-PC kernel: sd 7:0:14:0: [sdp] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08
Sep 5 08:50:21 WORK-PC kernel: sd 7:0:14:0: [sdp] tag#0 Sense Key : 0x3 [current] [descriptor]
Sep 5 08:50:21 WORK-PC kernel: sd 7:0:14:0: [sdp] tag#0 ASC=0x11 ASCQ=0x0
Sep 5 08:50:21 WORK-PC kernel: sd 7:0:14:0: [sdp] tag#0 CDB: opcode=0x88 88 00 00 00 00 00 ed 59 b9 48 00 00 04 00 00 00
Sep 5 08:50:21 WORK-PC kernel: print_req_error: critical medium error, dev sdp, sector 3982080653
Sep 5 14:31:28 WORK-PC emhttpd: shcmd (1403): /usr/local/sbin/set_ncq sdp 1
Sep 5 14:31:28 WORK-PC emhttpd: shcmd (1404): echo 128 > /sys/block/sdp/queue/nr_requests
Sep 5 14:31:28 WORK-PC emhttpd: shcmd (1435): /usr/local/sbin/set_ncq sdp 1
Sep 5 14:31:28 WORK-PC emhttpd: shcmd (1436): echo 128 > /sys/block/sdp/queue/nr_requests
Sep 5 14:31:36 WORK-PC emhttpd: shcmd (1467): /usr/local/sbin/set_ncq sdp 1
Sep 5 14:31:36 WORK-PC emhttpd: shcmd (1468): echo 128 > /sys/block/sdp/queue/nr_requests
Sep 5 14:31:36 WORK-PC emhttpd: shcmd (1499): /usr/local/sbin/set_ncq sdp 1
Sep 5 14:31:36 WORK-PC emhttpd: shcmd (1500): echo 128 > /sys/block/sdp/queue/nr_requests
Sep 5 14:31:38 WORK-PC emhttpd: shcmd (1531): /usr/local/sbin/set_ncq sdp 1
Sep 5 14:31:38 WORK-PC emhttpd: shcmd (1532): echo 128 > /sys/block/sdp/queue/nr_requests
Sep 5 14:31:38 WORK-PC emhttpd: shcmd (1563): /usr/local/sbin/set_ncq sdp 1
Sep 5 14:31:38 WORK-PC emhttpd: shcmd (1564): echo 128 > /sys/block/sdp/queue/nr_requests

 

i did a quick smart scan from command line and it came up fine. and i did a long scan it said this.

root@WORK-PC:~# smartctl -t short -C /dev/sdp

smartctl 7.0 2018-12-30 r4883 [x86_64-linux-4.19.56-Unraid] (local build)

Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org



Short Foreground Self Test Successful

root@WORK-PC:~# smartctl -t long -C /dev/sdp

smartctl 7.0 2018-12-30 r4883 [x86_64-linux-4.19.56-Unraid] (local build)

Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org





Long (extended) foreground self test failed [medium or hardware error (serious)]

 

i am currently running the following but i don't have a result yet.


 

root@WORK-PC:~# badblocks -v /dev/sdp |tee -a  back-blocks-sdp.txt

Checking blocks 0 to 2930266583

Checking for bad blocks (read-only test): 

all scans were done with the arrray up, so i don't know if that is bad or not.

 

also some more more smart output that i saw was asked for in a similar situation when i was looking through the forums

root@WORK-PC:~# smartctl -r ioctl,2 -i -H /dev/sdp

smartctl 7.0 2018-12-30 r4883 [x86_64-linux-4.19.56-Unraid] (local build)

Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org



sdp -> /sys/class/scsi_host/host7/proc_name: "mpt2sas"

>>>> do_scsi_cmnd_io: sg_io_ver=3

 [inquiry: 12 00 00 00 24 00 ]

  scsi_status=0x0, sg_transport_status=0x0, sg_driver_status=0x0

  sg_info=0x0  sg_duration=2 milliseconds  resid=0

  Incoming data, len=36:

 00     00 00 05 02 73 01 30 02  53 45 41 47 41 54 45 20

 10     53 54 33 30 30 30 4e 4d  30 30 33 33 20 20 53 41

 20     4e 41 30 30

>>>> do_scsi_cmnd_io: sg_io_ver=3

 [inquiry: 12 01 00 00 fc 00 ]

  scsi_status=0x0, sg_transport_status=0x0, sg_driver_status=0x0

  sg_info=0x0  sg_duration=0 milliseconds  resid=235

  Incoming data, len=17:

 00     00 00 00 0d 00 80 83 86  88 89 90 b0 b1 d0 d1 d2

 10     d3

>>>> do_scsi_cmnd_io: sg_io_ver=3

 [inquiry: 12 00 00 00 24 00 ]

  scsi_status=0x0, sg_transport_status=0x0, sg_driver_status=0x0

  sg_info=0x0  sg_duration=1 milliseconds  resid=0

  Incoming data, len=36:

 00     00 00 05 02 73 01 30 02  53 45 41 47 41 54 45 20

 10     53 54 33 30 30 30 4e 4d  30 30 33 33 20 20 53 41

 20     4e 41 30 30

=== START OF INFORMATION SECTION ===

Vendor:               SEAGATE

Product:              ST3000NM0033  SA

Revision:             NA00

Compliance:           SPC-3

>>>> do_scsi_cmnd_io: sg_io_ver=3

 [read capacity(16): 9e 10 00 00 00 00 00 00 00 00 00 00 00 20 00 00 ]

  scsi_status=0x0, sg_transport_status=0x0, sg_driver_status=0x0

  sg_info=0x0  sg_duration=0 milliseconds  resid=0

  Incoming data, len=32:

 00     00 00 00 01 5d 50 a3 af  00 00 02 00 00 00 00 00

 10     00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00

User Capacity:        3,000,592,982,016 bytes [3.00 TB]

Logical block size:   512 bytes

>>>> do_scsi_cmnd_io: sg_io_ver=3

 [inquiry: 12 01 b1 00 40 00 ]

  scsi_status=0x0, sg_transport_status=0x0, sg_driver_status=0x0

  sg_info=0x0  sg_duration=0 milliseconds  resid=0

  Incoming data, len=64:

 00     00 b1 00 3c 1c 20 00 02  00 00 00 00 00 00 00 00

 10     00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00

 20     00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00

 30     00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00

Rotation Rate:        7200 rpm

Form Factor:          3.5 inches

>>>> do_scsi_cmnd_io: sg_io_ver=3

 [mode sense(6): 1a 00 1c 00 40 00 ]

  scsi_status=0x0, sg_transport_status=0x0, sg_driver_status=0x0

  sg_info=0x0  sg_duration=0 milliseconds  resid=40

  Incoming data, len=24:

 00     17 00 00 08 ff ff ff ff  00 00 02 00 9c 0a 10 06

 10     00 00 00 00 00 00 00 01

>>>> do_scsi_cmnd_io: sg_io_ver=3

 [mode sense(6): 1a 00 5c 00 40 00 ]

  scsi_status=0x0, sg_transport_status=0x0, sg_driver_status=0x0

  sg_info=0x0  sg_duration=1 milliseconds  resid=40

  Incoming data, len=24:

 00     17 00 00 08 ff ff ff ff  00 00 02 00 9c 0a b9 0f

 10     ff ff ff ff ff ff ff ff

>>>> do_scsi_cmnd_io: sg_io_ver=3

 [inquiry: 12 01 83 00 fc 00 ]

  scsi_status=0x0, sg_transport_status=0x0, sg_driver_status=0x0

  sg_info=0x0  sg_duration=0 milliseconds  resid=216

  Incoming data, len=36:

 00     00 83 00 20 01 03 00 08  50 00 0c 90 00 64 ff 5c

 10     61 93 00 08 50 00 0c 90  00 64 ff 5d 61 94 00 04

 20     00 00 00 01

Logical Unit id:      0x50000c900064ff5c

>>>> do_scsi_cmnd_io: sg_io_ver=3

 [inquiry: 12 01 80 00 fc 00 ]

  scsi_status=0x0, sg_transport_status=0x0, sg_driver_status=0x0

  sg_info=0x0  sg_duration=0 milliseconds  resid=228

  Incoming data, len=24:

 00     00 80 00 14 5a 31 59 31  58 31 50 58 20 20 20 20

 10     20 20 20 20 20 20 20 20

Serial number:        Z1Y1X1PX

Device type:          disk

Transport protocol:   SAS (SPL-3)

Local Time is:        Thu Sep  5 22:25:21 2019 EDT

>>>> do_scsi_cmnd_io: sg_io_ver=3

 [test unit ready: 00 00 00 00 00 00 ]

  scsi_status=0x0, sg_transport_status=0x0, sg_driver_status=0x0

  sg_info=0x0  sg_duration=0 milliseconds  resid=0

SMART support is:     Available - device has SMART capability.

SMART support is:     Enabled

Temperature Warning:  Enabled



=== START OF READ SMART DATA SECTION ===

>>>> do_scsi_cmnd_io: sg_io_ver=3

 [log sense: 4d 00 40 00 00 00 00 00 04 00 ]

  scsi_status=0x0, sg_transport_status=0x0, sg_driver_status=0x0

  sg_info=0x0  sg_duration=0 milliseconds  resid=0

  Incoming data, len=4:

 00     80 00 00 15

>>>> do_scsi_cmnd_io: sg_io_ver=3

 [log sense: 4d 00 40 00 00 00 00 00 1a 00 ]

  scsi_status=0x0, sg_transport_status=0x0, sg_driver_status=0x0

  sg_info=0x0  sg_duration=0 milliseconds  resid=1

  Incoming data, len=25:

 00     80 00 00 15 00 02 03 05  06 0d 0e 10 18 19 2f 30

 10     31 32 33 34 35 36 37 38  3e

scsiGetSupportedLogPages: number of unreported (standard) log pages: 1 (sub-pages: 0)

>>>> do_scsi_cmnd_io: sg_io_ver=3

 [log sense: 4d 00 6f 00 00 00 00 00 04 00 ]

  scsi_status=0x0, sg_transport_status=0x0, sg_driver_status=0x0

  sg_info=0x0  sg_duration=21 milliseconds  resid=0

  Incoming data, len=4:

 00     af 00 00 08

>>>> do_scsi_cmnd_io: sg_io_ver=3

 [log sense: 4d 00 6f 00 00 00 00 00 0c 00 ]

  scsi_status=0x0, sg_transport_status=0x0, sg_driver_status=0x0

  sg_info=0x0  sg_duration=17 milliseconds  resid=0

  Incoming data, len=12:

 00     af 00 00 08 00 00 03 04  00 00 23 00

>>>> do_scsi_cmnd_io: sg_io_ver=3

 [request sense: 03 00 00 00 12 00 ]

  scsi_status=0x0, sg_transport_status=0x0, sg_driver_status=0x0

  sg_info=0x0  sg_duration=0 milliseconds  resid=0

  Incoming data, len=18:

 00     70 00 00 00 00 00 00 0a  00 00 00 00 00 00 00 00

 10     00 00

>>>> do_scsi_cmnd_io: sg_io_ver=3

 [log sense: 4d 00 4d 00 00 00 00 00 04 00 ]

  scsi_status=0x0, sg_transport_status=0x0, sg_driver_status=0x0

  sg_info=0x0  sg_duration=0 milliseconds  resid=0

  Incoming data, len=4:

 00     8d 00 00 0c

>>>> do_scsi_cmnd_io: sg_io_ver=3

 [log sense: 4d 00 4d 00 00 00 00 00 10 00 ]

  scsi_status=0x0, sg_transport_status=0x0, sg_driver_status=0x0

  sg_info=0x0  sg_duration=1 milliseconds  resid=0

  Incoming data, len=16:

 00     8d 00 00 0c 00 00 43 02  00 23 00 01 43 02 00 ff

SMART Health Status: OK

 

work-pc-diagnostics-20190906-0349.zip

Edited by xtrap225
Link to comment

i guess its not bad blocks

 

root@WORK-PC:~# badblocks -v /dev/sdp |tee -a  bad-blocks-sdp.txt

Checking blocks 0 to 2930266583

Checking for bad blocks (read-only test): done                                                 

Pass completed, 0 bad blocks found. (0/0/0 errors)

 

Link to comment

booooo! same error.  am i running that correctly? can you recommend any other smartclt or other tests i should/could run?

 

root@WORK-PC:~# smartctl -t long -C /dev/sdp

smartctl 7.0 2018-12-30 r4883 [x86_64-linux-4.19.56-Unraid] (local build)

Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org

 

Long (extended) foreground self test failed [medium or hardware error (serious)]

Link to comment
24 minutes ago, xtrap225 said:

booooo! same error.  am i running that correctly? can you recommend any other smartclt or other tests i should/could run?

 

root@WORK-PC:~# smartctl -t long -C /dev/sdp

smartctl 7.0 2018-12-30 r4883 [x86_64-linux-4.19.56-Unraid] (local build)

Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org

 

Long (extended) foreground self test failed [medium or hardware error (serious)]

That suggests that the disk really is failing.

Link to comment

i am working on moving all the data off the drive so i can test it outside of the system with the manufacturers utility, and then i  will of course be prepared to replace it, to a certain degree.

 

of course i will likely want to do a parity swap, because all the drives in the storage shelf are 3TB and i certainly don't want to go out and buy a 3TB.

 

one more interesting thing i found... this is the only drive with write-back cache disabled on it.  i enabled write-back cache and found this morning it was disabled again.

 

strange.

 

is there a rc.d script that sets the write-back cache using hdparm at startup?

Link to comment
16 hours ago, xtrap225 said:

NetApp DS4246

If you're using that it might be a good idea to at least connect the drive to normal controller and re-run SMART tests and check attributes, as the NetApp access SATA disks as SAS and that might cause some SMART issues with tests, besides the SMART report not being accessible.

Link to comment

so i used Seagate SeaTools for Windows with the drive plugged in via USB3 Asmedia ASM1053E.

 

it passed all the tests in the program.

 

S.M.A.R.T Check, Short Drive Self Test, Short Generic, and Long Generic. 

 

my next step is to put it back in the server and run the pre-check 3 times and then rebuild it from parity. 

Link to comment

may i ask you smart folks, do you know how to re-detect that drive i put in without rebooting?

 

i have a SAS2008 card connecting my desktop computer to that netapp previously discussed.

 

Dell LSI SAS 9202-16e 6Gb/s SAS Host Bus Adapter

from lspci

03:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)

 

from dmesg

 

303.920071] mdcmd (13): import 12

[  303.920072] md: import_slot: 12 missing

 

root@WORK-PC:~# dmesg |grep -B2 -A2 SAS2008

[   13.633974] mpt2sas_cm0: Current Controller Queue Depth(1948),Max Controller Queue Depth(2040)

[   13.634737] mpt2sas_cm0: Scatter Gather Elements per IO(128)

[   13.681883] mpt2sas_cm0: LSISAS2008: FWVersion(20.00.07.00), ChipRevision(0x03), BiosVersion(07.39.02.00)

[   13.683391] mpt2sas_cm0: Protocol=(

[   13.683391] Initiator

--

[   13.871549] mpt2sas_cm1: Current Controller Queue Depth(1948),Max Controller Queue Depth(2040)

[   13.872281] mpt2sas_cm1: Scatter Gather Elements per IO(128)

[   13.919956] mpt2sas_cm1: LSISAS2008: FWVersion(20.00.07.00), ChipRevision(0x03), BiosVersion(07.39.02.00)

[   13.921470] mpt2sas_cm1: Protocol=(

[   13.921471] Initiator

Link to comment

you are correct it was there i just didn't find it due to my own stupidity.

 

is this the smart output?

 

root@WORK-PC:~# smartctl -a /dev/sdp
smartctl 7.0 2018-12-30 r4883 [x86_64-linux-4.19.56-Unraid] (local build)
Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor:               SEAGATE
Product:              ST3000NM0033  SA
Revision:             NA00
Compliance:           SPC-3
User Capacity:        3,000,592,982,016 bytes [3.00 TB]
Logical block size:   512 bytes
Rotation Rate:        7200 rpm
Form Factor:          3.5 inches
Logical Unit id:      0x50000c900064ff5c
Serial number:        Z1Y1X1PX
Device type:          disk
Transport protocol:   SAS (SPL-3)
Local Time is:        Mon Sep  9 09:47:42 2019 EDT
SMART support is:     Available - device has SMART capability.
SMART support is:     Enabled
Temperature Warning:  Enabled

=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK

Current Drive Temperature:     36 C
Drive Trip Temperature:        <not available>

Manufactured in week    of year
Specified cycle count over device lifetime:  0
Accumulated start-stop cycles:  56
Read defect list: asked for grown list but didn't get it
Vendor (Seagate Cache) information
  Blocks sent to initiator = 62
  Blocks received from initiator = 0
  Blocks read from cache and sent to initiator = 12
  Number of read and write commands whose size <= segment size = 0
  Number of read and write commands whose size > segment size = 0

Vendor (Seagate/Hitachi) factory information
  number of hours powered up = 24761.00
  number of minutes until next internal SMART test = 28

Error counter log:
           Errors Corrected by           Total   Correction     Gigabytes    Total
               ECC          rereads/    errors   algorithm      processed    uncorrected
           fast | delayed   rewrites  corrected  invocations   [10^9 bytes]  errors
read:          0        0         0         0          0          0.000           0
write:         0        0         0         0          0          0.000           0

Non-medium error count:       88


[GLTSD (Global Logging Target Save Disable) set. Enable Save with '-S on']
SMART Self-test log
Num  Test              Status                 segment  LifeTime  LBA_first_err [SK ASC ASQ]
     Description                              number   (hours)
# 1  Background short  Completed                   -   24748                 - [-   -    -]
# 2  Foreground long   Aborted (device reset ?)    -   24748                 0 [0xb 0x82 0x0]
# 3  Foreground short  Completed                   -   24748                 - [-   -    -]
# 4  Background short  Aborted (by user command)   -   24748                 0 [0xb 0x81 0x0]
# 5  Foreground long   Aborted (device reset ?)    -   24721                 0 [0xb 0x82 0x0]
# 6  Foreground long   Aborted (device reset ?)    -   24708                 0 [0xb 0x82 0x0]
# 7  Foreground long   Aborted (device reset ?)    -   24687                 0 [0xb 0x82 0x0]
# 8  Foreground short  Completed                   -   24681                 - [-   -    -]
# 9  Background short  Completed                   -   24681                 - [-   -    -]

Long (extended) Self-test duration: 22140 seconds [369.0 minutes]

Link to comment

actually how does this look w/ regards to smart output done through the netapp still but using proper LSI Fusion MPT SAS2 driver settings?

 

does this look like it has the full output?

 

root@WORK-PC:~# smartctl -d sat -a /dev/sdp
smartctl 7.0 2018-12-30 r4883 [x86_64-linux-4.19.56-Unraid] (local build)
Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:     SEAGATE ST3000NM0033
Serial Number:    Z1Y1X1PX
LU WWN Device Id: 5 000c50 0675cb0e7
Firmware Version: NS00
User Capacity:    3,000,592,982,016 bytes [3.00 TB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Mon Sep  9 13:10:00 2019 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART Status not supported: Incomplete response, ATA output registers missing
SMART overall-health self-assessment test result: PASSED
Warning: This result is based on an Attribute check.

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (  105) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        ( 369) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x50bf) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   081   063   044    Pre-fail  Always       -       156604416
  3 Spin_Up_Time            0x0003   097   096   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       56
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       5
  7 Seek_Error_Rate         0x000f   086   060   030    Pre-fail  Always       -       406887179
  9 Power_On_Hours          0x0032   072   072   000    Old_age   Always       -       24771
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       23
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   095   095   000    Old_age   Always       -       5
188 Command_Timeout         0x0032   100   098   000    Old_age   Always       -       8590065667
189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   064   053   045    Old_age   Always       -       36 (Min/Max 34/36)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       20
193 Load_Cycle_Count        0x0032   100   100   000    Old_age   Always       -       1083
194 Temperature_Celsius     0x0022   036   047   000    Old_age   Always       -       36 (0 21 0 0 0)
195 Hardware_ECC_Recovered  0x001a   034   019   000    Old_age   Always       -       156604416
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0

SMART Error Log Version: 1
ATA Error Count: 8 (device log contains only the most recent five errors)
        CR = Command Register [HEX]
        FR = Features Register [HEX]
        SC = Sector Count Register [HEX]
        SN = Sector Number Register [HEX]
        CL = Cylinder Low Register [HEX]
        CH = Cylinder High Register [HEX]
        DH = Device/Head Register [HEX]
        DC = Device Command Register [HEX]
        ER = Error register [HEX]
        ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 8 occurred at disk power-on lifetime: 24721 hours (1030 days + 1 hours)
  When the command that caused the error occurred, the device was in an unknown state.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  04 51 00 00 00 00 00  Error: ABRT

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  00 00 00 00 00 00 00 ff  15d+17:03:05.309  NOP [Abort queued commands]
  b0 d4 00 82 4f c2 00 00  15d+12:00:01.038  SMART EXECUTE OFF-LINE IMMEDIATE
  ec 00 01 00 00 00 00 00  15d+12:00:00.942  IDENTIFY DEVICE
  ec 00 01 00 00 00 00 00  15d+12:00:00.941  IDENTIFY DEVICE
  e5 00 00 00 00 00 00 00  15d+11:59:00.770  CHECK POWER MODE

Error 7 occurred at disk power-on lifetime: 24708 hours (1029 days + 12 hours)
  When the command that caused the error occurred, the device was in an unknown state.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  04 51 00 00 00 00 00  Error: ABRT

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  00 00 00 00 00 00 00 ff  15d+04:40:50.041  NOP [Abort queued commands]
  b0 d4 00 82 4f c2 00 00  14d+23:37:45.768  SMART EXECUTE OFF-LINE IMMEDIATE
  ec 00 01 00 00 00 00 00  14d+23:37:45.673  IDENTIFY DEVICE
  ec 00 01 00 00 00 00 00  14d+23:37:45.672  IDENTIFY DEVICE
  e5 00 00 00 00 00 00 00  14d+23:18:07.902  CHECK POWER MODE

Error 6 occurred at disk power-on lifetime: 24687 hours (1028 days + 15 hours)
  When the command that caused the error occurred, the device was in an unknown state.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  04 51 00 00 00 00 00  Error: ABRT

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  00 00 00 00 00 00 00 ff  14d+07:40:17.431  NOP [Abort queued commands]
  b0 d4 00 82 4f c2 00 00  14d+02:37:13.161  SMART EXECUTE OFF-LINE IMMEDIATE
  ec 00 01 00 00 00 00 00  14d+02:37:13.066  IDENTIFY DEVICE
  ec 00 01 00 00 00 00 00  14d+02:37:13.065  IDENTIFY DEVICE
  e5 00 00 00 00 00 00 00  14d+02:31:51.028  CHECK POWER MODE

Error 5 occurred at disk power-on lifetime: 24675 hours (1028 days + 3 hours)
  When the command that caused the error occurred, the device was in standby mode.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 00 ff ff ff 4f 00  13d+19:52:05.743  READ FPDMA QUEUED
  e2 00 00 00 00 00 00 00  13d+19:52:04.691  STANDBY
  b0 d6 01 e0 4f c2 00 00  13d+19:52:04.691  SMART WRITE LOG
  b0 d6 01 e0 4f c2 00 00  13d+19:52:04.690  SMART WRITE LOG
  ef 10 02 00 00 00 00 00  13d+19:52:04.690  SET FEATURES [Enable SATA feature]

Error 4 occurred at disk power-on lifetime: 24675 hours (1028 days + 3 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 00 ff ff ff 4f 00  13d+19:52:00.514  READ FPDMA QUEUED
  60 00 00 ff ff ff 4f 00  13d+19:52:00.487  READ FPDMA QUEUED
  61 00 80 ff ff ff 4f 00  13d+19:52:00.483  WRITE FPDMA QUEUED
  61 00 f0 ff ff ff 4f 00  13d+19:52:00.481  WRITE FPDMA QUEUED
  61 00 78 ff ff ff 4f 00  13d+19:52:00.480  WRITE FPDMA QUEUED

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%     24748         -
# 2  Extended captive    Interrupted (host reset)      00%     24748         -
# 3  Short captive       Completed without error       00%     24748         -
# 4  Short offline       Aborted by host               30%     24748         -
# 5  Extended captive    Interrupted (host reset)      20%     24721         -
# 6  Extended captive    Interrupted (host reset)      20%     24708         -
# 7  Extended captive    Interrupted (host reset)      20%     24687         -
# 8  Short captive       Completed without error       00%     24681         -
# 9  Short offline       Completed without error       00%     24681         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

 

Link to comment

now if i figure out how to properly modify the smartd.conf to properly detect the drives with the -d sat instead of doing DEVICESCAN which is putting in scsi maybe i can have a better unraid smart experience in general.

 

not sure if i should use /dev/sdp or the equivalent /dev/sgX  but assuming ill stick with 'sd' here is what my /etc/smartd.conf would look like.

 

#DEVICESCAN
# Monitor LSI's disk SMART through SCSI generic
/dev/sdb -d sat -a -s L/../../3/02
/dev/sdc -d sat -a -s L/../../3/03
/dev/sdd -d sat -a -s L/../../3/04
/dev/sde -d sat -a -s L/../../3/05
/dev/sdf -d sat -a -s L/../../3/06
/dev/sdg -d sat -a -s L/../../3/07
/dev/sdh -d sat -a -s L/../../3/08
/dev/sdi -d sat -a -s L/../../3/09
/dev/sdj -d sat -a -s L/../../3/10
/dev/sdk -d sat -a -s L/../../3/11
/dev/sdl -d sat -a -s L/../../3/12
/dev/sdm -d sat -a -s L/../../3/13
/dev/sdn -d sat -a -s L/../../3/14
/dev/sdo -d sat -a -s L/../../3/15
/dev/sdp -d sat -a -s L/../../3/16
/dev/sdq -d sat -a -s L/../../3/17
/dev/sdr -d sat -a -s L/../../3/18
/dev/sds -d sat -a -s L/../../3/19
/dev/sdt -d sat -a -s L/../../3/20
/dev/sdu -d sat -a -s L/../../3/21
/dev/sdv -d sat -a -s L/../../3/22
/dev/sdw -d sat -a -s L/../../3/23
/dev/sdx -d sat -a -s L/../../3/24
/dev/sdy -d sat -a -s L/../../3/25
/dev/sdz -d sat -a -s L/../../3/26

Link to comment

put that in my go file and started another smartctl long test on a different drive that i have less faith in while witing for my preclear to finish on sdp.

 

sdp and sdm (the one i am running long test on now) are the only drives that when the system is rebooted lose their write-cache setting, and i have to use hdparm -W1 /dev/sdm and sdp to turn it back on. ( i have also now added that to the go file). 

 

i am only making note of these things now so that people can benefit form this experience and i will have the commands noted in this thread.

 

root@WORK-PC:~# smartctl -d sat -t long -C /dev/sdm
smartctl 7.0 2018-12-30 r4883 [x86_64-linux-4.19.56-Unraid] (local build)
Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF OFFLINE IMMEDIATE AND SELF-TEST SECTION ===
Sending command: "Execute SMART Extended self-test routine immediately in captive mode".
Drive command "Execute SMART Extended self-test routine immediately in captive mode" successful.
Testing has begun.
Please wait 369 minutes for test to complete.
Test will complete after Mon Sep  9 23:29:45 2019

and here is a copy of my /boot/config/go file.

root@WORK-PC:~# cat /boot/config/go
#Setup smartd.conf
mv /etc/smartd.conf /etc/smartd.conf.backup
cp -p /boot/smartd.conf.bak /etc/smartd.conf
chmod 644 /etc/smartd.conf
chmod 755 /etc/rc.d/rc.smartd
/etc/rc.d/rc.smartd start

#enable write cache on sdm and sdp
hdparm -W1 /dev/sdm
hdparm -W1 /dev/sdp

#Setup drivers for hardware transcoding in Plex
modprobe i915
chown -R nobody:users /dev/dri
chmod -R 777 /dev/dri

#!/bin/bash
# Start the Management Utility
/usr/local/sbin/emhttp &

and for anyone with a netapp here is the /etc/smartd.conf i will be using.

root@WORK-PC:~# cat /etc/smartd.conf
#DEVICESCAN
# Monitor LSI's disk SMART through SCSI generic
/dev/sdb -d sat -a -s L/../../3/02
/dev/sdc -d sat -a -s L/../../3/03
/dev/sdd -d sat -a -s L/../../3/04
/dev/sde -d sat -a -s L/../../3/05
/dev/sdf -d sat -a -s L/../../3/06
/dev/sdg -d sat -a -s L/../../3/07
/dev/sdh -d sat -a -s L/../../3/08
/dev/sdi -d sat -a -s L/../../3/09
/dev/sdj -d sat -a -s L/../../3/10
/dev/sdk -d sat -a -s L/../../3/11
/dev/sdl -d sat -a -s L/../../3/12
/dev/sdm -d sat -a -s L/../../3/13
/dev/sdn -d sat -a -s L/../../3/14
/dev/sdo -d sat -a -s L/../../3/15
/dev/sdp -d sat -a -s L/../../3/16
/dev/sdq -d sat -a -s L/../../3/17
/dev/sdr -d sat -a -s L/../../3/18
/dev/sds -d sat -a -s L/../../3/19
/dev/sdt -d sat -a -s L/../../3/20
/dev/sdu -d sat -a -s L/../../3/21
/dev/sdv -d sat -a -s L/../../3/22
/dev/sdw -d sat -a -s L/../../3/23
/dev/sdx -d sat -a -s L/../../3/24
/dev/sdy -d sat -a -s L/../../3/25
/dev/sdz -d sat -a -s L/../../3/26

here is the /var/log/syslog of the smartd loading up with the new config

root@WORK-PC:~# less /var/log/syslog 

Sep  9 17:12:01 WORK-PC smartd[9481]: Device: /dev/sdu [SAT], not found in smartd database.
Sep  9 17:12:01 WORK-PC smartd[9481]: Device: /dev/sdu [SAT], not capable of SMART Health Status check
Sep  9 17:12:02 WORK-PC smartd[9481]: Device: /dev/sdu [SAT], is SMART capable. Adding to "monitor" list.
Sep  9 17:12:02 WORK-PC smartd[9481]: Device: /dev/sdv [SAT], opened
Sep  9 17:12:02 WORK-PC smartd[9481]: Device: /dev/sdv [SAT], SEAGATE ST3000NM0033, S/N:Z1Y1T5V8, WWN:5-000c50-0675c7981, FW:NS00, 3.00 TB
Sep  9 17:12:02 WORK-PC smartd[9481]: Device: /dev/sdv [SAT], not found in smartd database.
Sep  9 17:12:02 WORK-PC smartd[9481]: Device: /dev/sdv [SAT], not capable of SMART Health Status check
Sep  9 17:12:02 WORK-PC smartd[9481]: Device: /dev/sdv [SAT], is SMART capable. Adding to "monitor" list.
Sep  9 17:12:02 WORK-PC smartd[9481]: Device: /dev/sdw [SAT], opened
Sep  9 17:12:02 WORK-PC smartd[9481]: Device: /dev/sdw [SAT], SEAGATE ST3000NM0033, S/N:Z1Y1X2MN, WWN:5-000c50-0675c76b3, FW:NS00, 3.00 TB
Sep  9 17:12:02 WORK-PC smartd[9481]: Device: /dev/sdw [SAT], not found in smartd database.
Sep  9 17:12:02 WORK-PC smartd[9481]: Device: /dev/sdw [SAT], not capable of SMART Health Status check
Sep  9 17:12:02 WORK-PC smartd[9481]: Device: /dev/sdw [SAT], is SMART capable. Adding to "monitor" list.
Sep  9 17:12:02 WORK-PC smartd[9481]: Device: /dev/sdx [SAT], opened
Sep  9 17:12:02 WORK-PC smartd[9481]: Device: /dev/sdx [SAT], SEAGATE ST3000NM0033, S/N:Z1Y1X2JX, WWN:5-000c50-0675c876f, FW:NS00, 3.00 TB
Sep  9 17:12:02 WORK-PC smartd[9481]: Device: /dev/sdx [SAT], not found in smartd database.
Sep  9 17:12:02 WORK-PC smartd[9481]: Device: /dev/sdx [SAT], not capable of SMART Health Status check
Sep  9 17:12:02 WORK-PC smartd[9481]: Device: /dev/sdx [SAT], is SMART capable. Adding to "monitor" list.
Sep  9 17:12:02 WORK-PC smartd[9481]: Device: /dev/sdy [SAT], opened
Sep  9 17:12:02 WORK-PC smartd[9481]: Device: /dev/sdy [SAT], SEAGATE ST3000NM0033, S/N:Z1Y1X2NR, WWN:5-000c50-0675c859a, FW:NS00, 3.00 TB
Sep  9 17:12:02 WORK-PC smartd[9481]: Device: /dev/sdy [SAT], not found in smartd database.
Sep  9 17:12:02 WORK-PC smartd[9481]: Device: /dev/sdy [SAT], not capable of SMART Health Status check
Sep  9 17:12:02 WORK-PC smartd[9481]: Device: /dev/sdy [SAT], is SMART capable. Adding to "monitor" list.
Sep  9 17:12:02 WORK-PC smartd[9481]: Device: /dev/sdz [SAT], opened
Sep  9 17:12:02 WORK-PC smartd[9481]: Device: /dev/sdz [SAT], SEAGATE ST3000NM0033, S/N:Z1Y1T69A, WWN:5-000c50-0675c6f7d, FW:NS00, 3.00 TB
Sep  9 17:12:02 WORK-PC smartd[9481]: Device: /dev/sdz [SAT], not found in smartd database.
Sep  9 17:12:02 WORK-PC smartd[9481]: Device: /dev/sdz [SAT], not capable of SMART Health Status check
Sep  9 17:12:03 WORK-PC smartd[9481]: Device: /dev/sdz [SAT], is SMART capable. Adding to "monitor" list.
Sep  9 17:12:03 WORK-PC smartd[9481]: Monitoring 25 ATA/SATA, 0 SCSI/SAS and 0 NVMe devices
Sep  9 17:12:04 WORK-PC smartd[9481]: Device: /dev/sdm [SAT], previous self-test was interrupted by the host with a reset
Sep  9 17:12:05 WORK-PC smartd[9481]: Device: /dev/sdp [SAT], previous self-test was interrupted by the host with a reset
Sep  9 17:12:06 WORK-PC smartd[10307]: smartd has fork()ed into background mode. New PID=10307.
Sep  9 17:12:06 WORK-PC smartd[10307]: file /run/smartd.pid written containing PID 10307

 

Edited by xtrap225
Link to comment
  • 2 weeks later...

i just want to bottom line this for anyone who jumped into their setup like me, because they wanted a bunch of storage on the cheap.

 

i own a NetApp DS4246, with all 3TB sata drives in it.  using a Dell LSI SAS 9202-16e 6Gb/s SAS Host Bus Adapter to plug it into a pretty generic Lenovo desktop computer w/ 32GB of ram and an Intel i5-4570.

 

anyway, the big mistake i made for years while running UnRaid was to think that i was supposed to use SAS or SCSI settings on the SMART setup because of the NetApp and/or the HBA, that is NOT correct.  pretty obviously when you think about it, SMART is internal to the drives and the drives are SATA so set your SMART up as SATA and it will work perfectly.

 

No extra setup required.  (i just thought i would add this final note for anyone reading this thread).

Link to comment

To do this you would need to use Tools->New Config and select the option to keep current assignments.    When you return to the main screen you can make any changes you want before starting the array.    The default on starting the array will then be to build new parity based on the new disk assignments so do not do this if you have any disabled disks as you would lose the ability to rebuild their contents.

 

If you have exactly the same drives and do not have a parity2 drive you can tick the “Parity is Valid” box before starting the array as the parity1 calculation is independent of disk slot.    However parity2 calculation DOES take into account the disk slot so you cannot use that option if you have dual parity.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.