Preclear plugin


Recommended Posts

2 hours ago, Frank1940 said:

Look in the first post of this thread at the first and third questions in the Q&A section for the location of the script for downloading.  If you are wondering why the scripts are not installed with the plugin, read that first post very carefully...  (like a lawyer)

Ahh, didn't read that. Thank you!

But how can I install previews version of the plugin it self (not the scripts)? Older updates worked for my other drives I pre-cleared.

12 minutes ago, wgstarks said:

Also, if you're going to run the script from terminal via screen, any location on the flash will work as long as you know what it is. I just use a "scripts" folder at the root of the flash drive.

Yeah, I know. But I don't really like screen, I prefer tmux wish is much better in my opinion. :) 

 

Now I haven't debug this much, but currently I getting this error:

$ ./preclear_disk.sh -t /dev/sdj
./preclear_disk.sh: line 1307: strings: command not found
sfdisk: invalid option -- 'R'
Try 'sfdisk --help' for more information.
Sorry: Device /dev/sdj is busy.: 1

I the script I downloaded was this (version 1.15): https://forums.unraid.net/applications/core/interface/file/attachment.php?id=11467

And I got this link from bottom on first post:

 

Line 1307 is this (plus above comment):

#----------------------------------------------------------------------------------
# First, do some basic tests to ensure the disk  is not part of the arrray
# and not mounted, and not in use in any way.
#----------------------------------------------------------------------------------
devices=`/root/mdcmd status | strings | grep rdevName | sed 's/\([^=]*\)=\([^=]\)/\/dev\/\2/'`

For some reason there is no strings command in my installation of unraid, so I guess I need to install additonal packages somewhere else?

Link to comment
27 minutes ago, rxnelson said:

 

 


Don’t you have to put sda or something on there to tell it what drive. I’ve seen scripts with what look like switches as well -f I think.


Sent from my iPhone using Tapatalk

 

 

I've never used the bjp script but I'm sure you're correct. Probably disk and number of cycles at least.

 

@walle Sent you a PM.

Link to comment

I found the problem with the plugin (sort of)!

 

For some reason a old tmux pre-clear session got stuck:

$ ps uax | grep tmux
root      1695  0.0  0.0  28968  7364 ?        Ss   Aug31   1:07 /usr/bin/tmux new-session -d -x 140 -y 200 -s preclear_disk_WD-WCC7K6SFD5TJ
root     10801  0.0  0.0   9812  2136 pts/1    S+   02:28   0:00 grep tmux

After I killed the process (`kill 1695`), it seams that it works as expected (it has passed "Started..." and currently running pre-read)

An other way to solve it would be to restart the server.

 

Thank you for your help!

Edited by walle
Clerification
Link to comment
On 9/1/2018 at 7:30 AM, Frank1940 said:

Had a preclear failure yesterday.  Using standard script.  It stopped (apparently) at the beginning of the Post-read.  The Preclear Status condition was "unRAID's signature on the MBR is valid".  It has been 'stuck' at this state for more than eight hours and probably closer to twelve hours

 

 I have attached all of the pertinent information in the attached files.  The Preclear_Failure.txt file is the current state of the Status Window.

 

EDIT:  I did preclear a Seagate ST3000DM001_1CH166  disk (one of the infamous series) a couple of days ago without a problem.

 

I am having the same issue.  I tried stopping the preclear and resuming but now it is stuck at "starting" the post read.

 

Any next steps?

 

Attaching Diags and Preclear logs.

debo-server-diagnostics-20180907-2041.zip

DEBO-SERVER-preclear.disk-20180907-2036.zip

Link to comment
10 hours ago, B_Sinn3d said:

I am having the same issue.  I tried stopping the preclear and resuming but now it is stuck at "starting" the post read.

 

Any next steps?

 

Attaching Diags and Preclear logs.

debo-server-diagnostics-20180907-2041.zip

DEBO-SERVER-preclear.disk-20180907-2036.zip

My disk also got stuck at "unRAID's signature on the MBR is valid", but all I needed to do was to stop pre-clear and resume the operation again.

 

See my post above, if you got stuck on "Starting", try to restart the server then try again.

Edited by walle
Typo
Link to comment

While pre-clearing is no longer necessary, I greatly appreciate you continuing to support this as it's a great way to test a new drive prior to installation.

 

I love the little icon in the status bar of the WebGUI - one small request for that. If you could add the drive temp to that display, it would be perfect!

 

It would be especially handy for those of us doing preclear testing of USB drives prior to shucking them. There's so little airflow through those external cases, drive temps can easily skyrocket and being able to monitor them at a glance would be fantastic.

 

Thanks again!

Link to comment

A huge thanks to @gfjardim & co for keeping this alive and kicking. 

The ability to preclear - postclear - and stress check (new and antiquated drives alike) is, in my opinion, the most critical tool this platform provides.  This plugin (which usually works :) ) is far easier to implement than the old Joe script (which requires a patch command and manual terminal commands to use). 

Link to comment
On 9/4/2018 at 1:14 PM, Evin said:

Hi!

 

I tried to preclear two 8TB WD Reds and one disk seems to be stuck in the postread:

 


############################################################################################################################
#                                                                                                                          #
#                                         unRAID Server Preclear of disk VAG4N04L                                          #
#                                       Cycle 1 of 1, partition start on sector 64.                                        #
#                                                                                                                          #
#                                                                                                                          #
#   Step 1 of 5 - Pre-read verification:                                                   [27:49:14 @ 79 MB/s] SUCCESS    #
#   Step 2 of 5 - Zeroing the disk:                                                        [22:29:00 @ 98 MB/s] SUCCESS    #
#   Step 3 of 5 - Writing unRAID's Preclear signature:                                                          SUCCESS    #
#   Step 4 of 5 - Verifying unRAID's Preclear signature:                                                        SUCCESS    #
#   Step 5 of 5 - Post-Read in progress ...                                                                                #
#                                                                                                                          #
#                                                                                                                          #
#                                                                                                                          #
#                                                                                                                          #
#                                                                                                                          #
#                                                                                                                          #
#                                                                                                                          #
############################################################################################################################
#                              Cycle elapsed time: 50:18:32 | Total elapsed time: 50:18:35                                 #
############################################################################################################################


############################################################################################################################
#                                                                                                                          #
#                                               S.M.A.R.T. Status default                                                  #
#                                                                                                                          #
#                                                                                                                          #
#   ATTRIBUTE                    INITIAL  STATUS                                                                           #
#   5-Reallocated_Sector_Ct      0        -                                                                                #
#   9-Power_On_Hours             106      -                                                                                #
#   194-Temperature_Celsius      37       -                                                                                #
#   196-Reallocated_Event_Count  0        -                                                                                #
#   197-Current_Pending_Sector   0        -                                                                                #
#   198-Offline_Uncorrectable    0        -                                                                                #
#   199-UDMA_CRC_Error_Count     0        -                                                                                #
#                                                                                                                          #
#                                                                                                                          #
#                                                                                                                          #
#                                                                                                                          #
#                                                                                                                          #
############################################################################################################################
#   SMART overall-health self-assessment test result: PASSED                                                               #
############################################################################################################################

 

The other one failed:

 


############################################################################################################################
#                                                                                                                          #
#                                         unRAID Server Preclear of disk VAG54G5L                                          #
#                                       Cycle 1 of 1, partition start on sector 64.                                        #
#                                                                                                                          #
#                                                                                                                          #
#   Step 1 of 5 - Pre-read verification:                                                   [27:48:24 @ 79 MB/s] SUCCESS    #
#   Step 2 of 5 - Zeroing the disk:                                                        [22:29:10 @ 98 MB/s] SUCCESS    #
#   Step 3 of 5 - Writing unRAID's Preclear signature:                                                          SUCCESS    #
#   Step 4 of 5 - Verifying unRAID's Preclear signature:                                                        SUCCESS    #
#   Step 5 of 5 - Post-Read verification:                                                                          FAIL    #
#                                                                                                                          #
#                                                                                                                          #
#                                                                                                                          #
#                                                                                                                          #
#                                                                                                                          #
#                                                                                                                          #
#                                                                                                                          #
############################################################################################################################
#                              Cycle elapsed time: 50:46:02 | Total elapsed time: 50:46:04                                 #
############################################################################################################################


############################################################################################################################
#                                                                                                                          #
#                                               S.M.A.R.T. Status default                                                  #
#                                                                                                                          #
#                                                                                                                          #
#   ATTRIBUTE                    INITIAL  STATUS                                                                           #
#   5-Reallocated_Sector_Ct      0        -                                                                                #
#   9-Power_On_Hours             106      -                                                                                #
#   194-Temperature_Celsius      38       -                                                                                #
#   196-Reallocated_Event_Count  0        -                                                                                #
#   197-Current_Pending_Sector   0        -                                                                                #
#   198-Offline_Uncorrectable    0        -                                                                                #
#   199-UDMA_CRC_Error_Count     0        -                                                                                #
#                                                                                                                          #
#                                                                                                                          #
#                                                                                                                          #
#                                                                                                                          #
#                                                                                                                          #
############################################################################################################################
#   SMART overall-health self-assessment test result: PASSED                                                               #
############################################################################################################################

--> FAIL: Post-Read verification failed. Your drive is not zeroed.

 

Any ideas what I should do?

SERVER-preclear.disk-20180904-1759.zip

 

You have encountered an odd error, but apparently your disk wasn't zeroed around byte 158607605760. Most common cause for this error is bad memory, so run memtest is advised.

 

I have a theory about the "stuck in beginning of post-read" error. Will revise the code and make some changes to avoid this.

Edited by gfjardim
Link to comment

A big thanks to @gfjardim for adding the disk temp to the page footer! Great turnaround time - the kind of fantastic support we're spoiled with here!

 

A pre-clear question. This is my report from a pre-clear on an older drive:

Quote

############################################################################################################################
#                                                                                                                          #
#                                      unRAID Server Preclear of disk STF604MH0EN8XB                                       #
#                                       Cycle 1 of 1, partition start on sector 64.                                        #
#                                                                                                                          #
#                                                                                                                          #
#   Step 1 of 5 - Pre-read verification:                                                    [4:04:31 @ 68 MB/s] SUCCESS    #
#   Step 2 of 5 - Zeroing the disk:                                                         [4:17:07 @ 64 MB/s] SUCCESS    #
#   Step 3 of 5 - Writing unRAID's Preclear signature:                                                          SUCCESS    #
#   Step 4 of 5 - Verifying unRAID's Preclear signature:                                                        SUCCESS    #
#   Step 5 of 5 - Post-Read verification:                                                   [3:51:50 @ 71 MB/s] SUCCESS    #
#                                                                                                                          #
#                                                                                                                          #
#                                                                                                                          #
#                                                                                                                          #
#                                                                                                                          #
#                                                                                                                          #
#                                                                                                                          #
############################################################################################################################
#                              Cycle elapsed time: 11:47:27 | Total elapsed time: 11:47:32                                 #
############################################################################################################################


############################################################################################################################
#                                                                                                                          #
#                                               S.M.A.R.T. Status default                                                  #
#                                                                                                                          #
#                                                                                                                          #
#   ATTRIBUTE                    INITIAL  CYCLE 1  STATUS                                                                  #
#   5-Reallocated_Sector_Ct      8        8        -                                                                       #
#   9-Power_On_Hours             83360    83371    Up 11                                                                   #
#   194-Temperature_Celsius      47       46       Down 1                                                                  #
#   196-Reallocated_Event_Count  8        8        -                                                                       #
#   197-Current_Pending_Sector   0        0        -                                                                       #
#   198-Offline_Uncorrectable    0        0        -                                                                       #
#   199-UDMA_CRC_Error_Count     0        0        -                                                                       #
#                                                                                                                          #
#                                                                                                                          #
#                                                                                                                          #
#                                                                                                                          #
#                                                                                                                          #
############################################################################################################################
#   SMART overall-health self-assessment test result: PASSED                                                               #
############################################################################################################################


--> ATTENTION: Please take a look into the SMART report above for drive health issues.

--> RESULT: Preclear Finished Successfully!.

 

I pulled this drive from my main server because of the 8 reallocated sectors. It got mixed into a bunch that I just replaced and pulled due to age. I'm planning on putting these drives into my off-site backup server. Since the reallocated sector count didn't change during the pre-clear, is that a generally good sign that the drive will be OK, or should I just disassemble it to ensure I don't accidentally ever use it again?

 

Full SMART report attached

backup-smart-20180916-0708.zip

 

Also - I've got another drive pre-clearing and the status is "PAUSED". I'm not sure how it got paused or how to get it restarted. Should I just hit the red "X" to stop the preclear then restart it to pick up where it left off? How would it have been paused?

Edited by FreeMan
Link to comment
4 minutes ago, FreeMan said:

A big thanks to @gfjardim for adding the disk temp to the page footer! Great turnaround time - the kind of fantastic support we're spoiled with here!

 

A pre-clear question. This is my report from a pre-clear on an older drive:

 

I pulled this drive from my main server because of the 8 reallocated sectors. It got mixed into a bunch that I just replaced and pulled due to age. I'm planning on putting these drives into my off-site backup server. Since the reallocated sector count didn't change during the pre-clear, is that a generally good sign that the drive will be OK, or should I just disassemble it to ensure I don't accidentally ever use it again?

 

Full SMART report attached

backup-smart-20180916-0708.zip

 

There is a disk in my array that has 21 reallocated sectors for ages (more than 5 years!). Reallocated sectors can be a symptom for a failing drive or can reflect a surface defect that occurred only once. The only way to tell the difference is to put the disk to run and monitor it. Since it's a backup server, you should be safe as long as you keep monitoring it.

Link to comment
Just now, gfjardim said:

 

There is a disk in my array that has 21 reallocated sectors for ages (more than 5 years!). Reallocated sectors can be a symptom for a failing drive or can reflect a surface defect that occurred only once. The only way to tell the difference is to put the disk to run and monitor it. Since it's a backup server, you should be safe as long as you keep monitoring it.

Thank you sir! I'll do just that.

Link to comment
2 minutes ago, FreeMan said:

Weird. I stopped the "paused" pre-clear and restarted it. I also stopped/restarted the one that had said "MBR signature correct" (or something to that effect) for the last 30 min or so.

 

Almost immediately after restarting, both pre-clears are now paused. Suggestions?

2018-09-16 10_31_33-Backup_Preclear.png

 

Take a look into the plugin log and see if there's a reason it got paused (can be array operations, sync command issued, smartctl or hdparm instances that got stuck).

Link to comment

I don't see anything here that looks like it would have caused a pause, but I'm no expert. Your thoughts?

Quote

Sep 16 07:35:31 preclear_disk_5XW27ZVJ_12066: Command: /usr/local/emhttp/plugins/preclear.disk/script/preclear_disk.sh --load-file /boot/config/plugins/preclear.disk/5XW27ZVJ.resume /dev/sdf
Sep 16 07:35:31 preclear_disk_5XW27ZVJ_12066: Preclear Disk Version: 0.9.7b-beta
Sep 16 07:35:31 preclear_disk_5XW27ZVJ_12066: Restoring previous instance of preclear
Sep 16 07:35:31 preclear_disk_5XW27ZVJ_12066: S.M.A.R.T. info type: default
Sep 16 07:35:31 preclear_disk_5XW27ZVJ_12066: S.M.A.R.T. attrs type: default
Sep 16 07:35:33 preclear_disk_5XW2EWWX_13813: Command: /usr/local/emhttp/plugins/preclear.disk/script/preclear_disk.sh --load-file /boot/config/plugins/preclear.disk/5XW2EWWX.resume /dev/sdd
Sep 16 07:35:33 preclear_disk_5XW2EWWX_13813: Preclear Disk Version: 0.9.7b-beta
Sep 16 07:35:33 preclear_disk_5XW2EWWX_13813: Restoring previous instance of preclear
Sep 16 07:35:34 preclear_disk_5XW2EWWX_13813: S.M.A.R.T. info type: default
Sep 16 07:35:34 preclear_disk_5XW2EWWX_13813: S.M.A.R.T. attrs type: default
Sep 16 07:35:35 preclear_disk_5XW27ZVJ_12066: Continuing disk read from byte 697982517248
Sep 16 07:35:35 preclear_disk_5XW27ZVJ_12066: Post-Read: verifying the beggining of the disk.
Sep 16 07:35:35 preclear_disk_5XW27ZVJ_12066: Post-Read: dd if=/dev/sdf bs=512 count=4095 skip=1 conv=notrunc iflag=nocache 2>/tmp/.preclear/sdf/dd_output | cmp - /dev/zero &>/tmp/.preclear/sdf/cmp_out
Sep 16 07:35:35 preclear_disk_5XW27ZVJ_12066: Post-Read: dd pid [16068]
Sep 16 07:35:35 preclear_disk_5XW27ZVJ_12066: Post-Read: verifying the rest of the disk.
Sep 16 07:35:35 preclear_disk_5XW27ZVJ_12066: Post-Read: dd if=/dev/sdf bs=2097152 skip=697982517248 count=1302416416768 conv=notrunc iflag=nocache,count_bytes,skip_bytes 2>/tmp/.preclear/sdf/dd_output | cmp - /dev/zero &>/tmp/.preclear/sdf/cmp_out
Sep 16 07:35:35 preclear_disk_5XW27ZVJ_12066: Post-Read: dd pid [16144]
Sep 16 07:35:38 preclear_disk_5XW2EWWX_13813: Continuing disk read from byte 574619648
Sep 16 07:35:38 preclear_disk_5XW2EWWX_13813: Post-Read: verifying the beggining of the disk.
Sep 16 07:35:38 preclear_disk_5XW2EWWX_13813: Post-Read: dd if=/dev/sdd bs=512 count=4095 skip=1 conv=notrunc iflag=nocache 2>/tmp/.preclear/sdd/dd_output | cmp - /dev/zero &>/tmp/.preclear/sdd/cmp_out
Sep 16 07:35:38 preclear_disk_5XW2EWWX_13813: Post-Read: dd pid [18431]
Sep 16 07:35:38 preclear_disk_5XW2EWWX_13813: Post-Read: verifying the rest of the disk.
Sep 16 07:35:38 preclear_disk_5XW2EWWX_13813: Post-Read: dd if=/dev/sdd bs=2097152 skip=574619648 count=1999824314368 conv=notrunc iflag=nocache,count_bytes,skip_bytes 2>/tmp/.preclear/sdd/dd_output | cmp - /dev/zero &>/tmp/.preclear/sdd/cmp_out
Sep 16 07:35:38 preclear_disk_5XW2EWWX_13813: Post-Read: dd pid [18451]

There's literally nothing running on this server. I put the hardware together, booted it into unRAID, installed CA and used that to install pre-clear. That's it. No disks are assigned to the array, no dockers, no VMs, no parity, no cache, nothing. Here's the diagnostics to prove it! (Not that anyone's doubting me...)

 

Here's the log specifically for the preclear on sdf: (note - this was from a stop/start after the above preclear log was pulled)

Quote

Sep 15 17:01:50 Backup kernel: ata5: SATA max UDMA/133 abar m1024@0xfe9ffc00 port 0xfe9ffe00 irq 22
Sep 15 17:01:51 Backup kernel: ata5: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
Sep 15 17:01:51 Backup kernel: ata5.00: ATA-8: ST32000542AS, 5XW27ZVJ, CC95, max UDMA/133
Sep 15 17:01:51 Backup kernel: ata5.00: 3907029168 sectors, multi 16: LBA48 NCQ (depth 31/32)
Sep 15 17:01:51 Backup kernel: ata5.00: configured for UDMA/133
Sep 15 17:01:51 Backup kernel: sd 5:0:0:0: [sdf] 3907029168 512-byte logical blocks: (2.00 TB/1.82 TiB)
Sep 15 17:01:51 Backup kernel: sd 5:0:0:0: [sdf] Write Protect is off
Sep 15 17:01:51 Backup kernel: sd 5:0:0:0: [sdf] Mode Sense: 00 3a 00 00
Sep 15 17:01:51 Backup kernel: sd 5:0:0:0: [sdf] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Sep 15 17:01:51 Backup kernel: sdf: sdf1
Sep 15 17:01:51 Backup kernel: sd 5:0:0:0: [sdf] Attached SCSI disk
Sep 15 17:01:59 Backup emhttpd: ST32000542AS_5XW27ZVJ (sdf) 512 3907029168
Sep 15 21:58:03 Backup emhttpd: ST32000542AS_5XW27ZVJ (sdf) 512 3907029168
Sep 16 06:52:03 Backup preclear.disk: Pausing preclear of disk 'sdf'

 

and from sdd (this was from the same attempt as the log at the top):

Quote

Sep 15 17:01:50 Backup kernel: ata3: SATA max UDMA/133 abar m1024@0xfe9ffc00 port 0xfe9ffd00 irq 22
Sep 15 17:01:51 Backup kernel: ata3: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
Sep 15 17:01:51 Backup kernel: ata3.00: ATA-8: ST32000542AS, 5XW2EWWX, CC34, max UDMA/133
Sep 15 17:01:51 Backup kernel: ata3.00: 3907029168 sectors, multi 16: LBA48 NCQ (depth 31/32)
Sep 15 17:01:51 Backup kernel: ata3.00: configured for UDMA/133
Sep 15 17:01:51 Backup kernel: sd 3:0:0:0: [sdd] 3907029168 512-byte logical blocks: (2.00 TB/1.82 TiB)
Sep 15 17:01:51 Backup kernel: sd 3:0:0:0: [sdd] Write Protect is off
Sep 15 17:01:51 Backup kernel: sd 3:0:0:0: [sdd] Mode Sense: 00 3a 00 00
Sep 15 17:01:51 Backup kernel: sd 3:0:0:0: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Sep 15 17:01:51 Backup kernel: sdd: sdd1
Sep 15 17:01:51 Backup kernel: sd 3:0:0:0: [sdd] Attached SCSI disk
Sep 15 17:01:59 Backup emhttpd: ST32000542AS_5XW2EWWX (sdd) 512 3907029168
Sep 15 21:58:03 Backup emhttpd: ST32000542AS_5XW2EWWX (sdd) 512 3907029168
Sep 16 05:55:01 Backup kernel: sdd: sdd1
Sep 16 06:52:03 Backup preclear.disk: Pausing preclear of disk 'sdd'

 

backup-diagnostics-20180916-1040.zip

Link to comment
43 minutes ago, FreeMan said:

very high "Raw read error rate" and "Seek error rate". How critical does this look?

Most Seagates use a multibyte value for both those attributes, for that disk the raw_read_error_rate is only reporting the total reads, errors are still at 0, for the seek_error_rate there are some errors, but low and better than at some some point in the past, so not much to worry about for now, of course it's never a good sign, same for the reported_uncorrected.

Link to comment
1 minute ago, johnnie.black said:

Most Seagates use a multibyte value for both those attributes, for that disk the raw_read_error_rate is only reporting the total reads, errors are still at 0, for the seek_error_rate there are some errors, but low and better than at some some point in the past, so not much to worry about for now, of course it's never a good sign, same for the reported_uncorrected.

Thanks for the reassurance, j.b

 

I don't think these drives will be long for this server. As I upgrade drives in the main server, the replaced drives will be moved to the backup and the backup drives will be retired. This is just all I've got to work with at the moment, so it'll have to do.

Link to comment
When I have Preclear plugin 2018.09.17 installed on 6.6.0-rc4 the User Shares - Add Share - Share Settings Included disks and Excluded disks values are obscured. See  https://screenshots.firefox.com/MYrjvbEQdcjM5RFf/192.168.10.50 
 
There is a short thread in bug reports with diagnostics.
 
I need you to send me your web console output:

https://developer.mozilla.org/en-US/docs/Tools/Web_Console
Link to comment
7 hours ago, laterdaze said:

The problem is not Firefox dependant, does the same with Chrome or Edge.  Still want the web console output?

Nope, thought it was a Javascript error, but it's a CSS error. It displayed white text in a light background. Will be fixed in the next version.

 

Thanks for reporting.

Link to comment
22 minutes ago, bonienl said:

Easiest would be to use the css styling provided by the theme style sheets and not create a specific style in the plugin.

Yep. In fact it was a legacy code that loaded the default jQueryUI css from Dynamix styles directory. Since the plugin doesn't uses jQueryUI anymore, it was easy to fix that.

 

Thanks for helping on fix this, @bonienl!

Edited by gfjardim
Link to comment
  • Squid unpinned this topic

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.