Re: preclear_disk.sh - a new utility to burn-in and pre-clear disks for quick add


Recommended Posts

Hmm. I planned for this to be my parity drive, but not being able to see the differential bothers me a bit. If it were a data drive I'd probably be okay with it, but I think I'll just wait and not be hasty. Adding the drive cages and cabling is the time consuming part which I can still get done, I don't need the extra capacity yet so no need to rush it.

 

I think I'll let it finish, and if the differential isn't available, then I'll just add the bays and cabling, then I'll have space to preclear all the drives at once, and I'll just redo it on that one at the same time.

 

And next time I'll earmark some of my upgrade funds for a UPS. I'm pretty sure a brown out is what killed my last power supply a few months back, but I was away so not positive on that.

Link to comment

Here's how to do it manually.

 

smartctl -a (device) > log

smartctl -t long (device)

(wait for what feels like for eva!!)

 

smartctl -a (device) > newerlog

 

diff -u log newerlog

 

However I do agree that the preclear script is great in keeping the logs for everything.

 

 

I wrote my own smartctl.sh wrapper to do that for me.

I like to make them dos txt files so that I can review them with notepad or via emhttp directly.

 

To use it, I use the full disk path with the serial number.

 

/boot/local/bin/smartctl.sh -a /dev/disk/by-id/(press tab key for cut/paste)

Puts smartctl logs in /boot/logs

One day I'll make a menu utility to do this.

 

root@unRAID1:/# /boot/local/bin/smartctl.sh -a /dev/disk/by-id/ (press tab key for list and cut and paste)
scsi-1ATA_ST3000DM001-1CH166_W1F1GWZK                 scsi-1ATA_WDC_WD20EARX-00PASB0_WD-WCAZAC993717-part1
scsi-1ATA_ST3000DM001-1CH166_W1F1GWZK-part1           scsi-1ATA_WDC_WD20EARX-00PASB0_WD-WCAZAJ255820
scsi-1ATA_ST3000DM001-9YN166_W1F191JR                 scsi-1ATA_WDC_WD20EARX-00PASB0_WD-WCAZAJ255820-part1
scsi-1ATA_ST3000DM001-9YN166_W1F191JR-part1           usb-Generic-_USB3.0_CRW_-0_000000002318-0:0
scsi-1ATA_WDC_WD20EARX-00PASB0_WD-WCAZAC993717        usb-Generic-_USB3.0_CRW_-0_000000002318-0:0-part1
root@unRAID1:/# /boot/local/bin/smartctl.sh -A /dev/disk/by-id/

 

root@unRAID1:/# cat /boot/local/bin/smartctl.sh 

#!/bin/bash

[ ${DEBUG:=0} -gt 0 ] && set -x -v

for drive; do true; done

OP="${drive#/dev/disk/by-id/}.%Y%m%d-%H%M.txt"
OP="${OP#*_}"
OP=/boot/logs/`date +"${OP}"`

# echo $OP; exit

/usr/sbin/smartctl $* 2>&1 | todos >> ${OP}
ls -l ${OP}

root@unRAID1:/# ls -l /boot/logs
total 3440
-rwxrwxrwx 1 root root   6166 2014-10-27 13:22 ST3000DM001-1CH166_W1F1GWZK.20141027-1322.txt*
-rwxrwxrwx 1 root root   5725 2014-10-27 22:53 ST3000DM001-1CH166_W1F1GWZK.20141027-2253.txt*
-rwxrwxrwx 1 root root  10468 2014-10-27 13:48 ST3000DM001-9YN166_W1F191JR.20141027-1348.txt*
-rwxrwxrwx 1 root root    544 2014-10-27 13:49 ST3000DM001-9YN166_W1F191JR.20141027-1349.txt*
-rwxrwxrwx 1 root root    552 2014-10-27 13:51 ST3000DM001-9YN166_W1F191JR.20141027-1351.txt*
-rwxrwxrwx 1 root root  10651 2014-10-27 22:53 ST3000DM001-9YN166_W1F191JR.20141027-2253.txt*
-rwxrwxrwx 1 root root   4960 2014-10-27 22:54 WDC_WD20EARX-00PASB0_WD-WCAZAC993717.20141027-2254.txt*
-rwxrwxrwx 1 root root   5327 2014-10-27 13:57 WDC_WD20EARX-00PASB0_WD-WCAZAJ255820.20141027-1357.txt*
-rwxrwxrwx 1 root root   5438 2014-10-27 14:00 WDC_WD20EARX-00PASB0_WD-WCAZAJ255820.20141027-1400.txt*
-rwxrwxrwx 1 root root   4966 2014-10-27 22:54 WDC_WD20EARX-00PASB0_WD-WCAZAJ255820.20141027-2254.txt*
-rwxrwxrwx 1 root root   4967 2014-11-06 12:43 WDC_WD20EARX-00PASB0_WD-WCAZAJ255820.20141106-1243.txt*
-rwxrwxrwx 1 root root 257152 2013-07-07 17:56 syslog-20130707-175604.txt*
-rwxrwxrwx 1 root root 367557 2013-07-07 17:56 syslog-20130707-175641.txt*
-rwxrwxrwx 1 root root 478658 2013-07-07 18:00 syslog-20130707-180055.txt*
-rwxrwxrwx 1 root root 589807 2013-07-07 18:01 syslog-20130707-180159.txt*
-rwxrwxrwx 1 root root 700892 2013-07-07 18:02 syslog-20130707-180234.txt*
-rwxrwxrwx 1 root root 815649 2013-07-07 18:03 syslog-20130707-180359.txt*
-rwxrwxrwx 1 root root  84059 2013-07-07 18:04 syslog.zip*

Link to comment

This part.

 

What I suspect you won't see is the complete report that shows any changes in the SMART data during the process (not sure what might have already been saved by the pre-clear utility)

 

Don't forget to check the preclear_reports folder on the flash drive -- as I noted earlier, I'm not certain exactly when Joe's script writes the various logs.  You may already have a "preclear_start_..." report for the drive -- so you may simply be able to compare that SMART data with the final SMART data and see the differential for yourself.  I doubt you'll have the preclear_rpt_... file, which would summarize everything for you => but it's pretty simple to just note what's different.

 

Link to comment

That's only for drives less then 2.2 TB. It's ignored on larger drives, even if you do attempt to use it.

 

Thought I would add this for those who haven't precleared 6 TB drives yet, or to add to the benchmark section of the wiki.

 

WD 6TB Red (WD60EFRX)

 

Preclear Successful

... Total time 54:43:12

... Pre-Read time 13:55:07 (119 MB/s)

... Zeroing time 12:46:33 (130 MB/s)

... Post-Read time 28:00:32 (59 MB/s)

 

Preclear Successful

... Total time 54:50:58

... Pre-Read time 14:02:36 (118 MB/s)

... Zeroing time 12:46:51 (130 MB/s)

... Post-Read time 28:00:32 (59 MB/s)

 

 

Link to comment

Been awhile since I have last pre-cleard a drive. Do I still need to use -A option on unRaid 5.0.6 with 4TB drives (HGST Deskstar NAS)?

You can use it, or not...  It is silently ignored on  disks over 2.2TB as it has no meaning or purpose on disks using a GPT partition (all disks over 2.2TB use a GPT partition)

 

So... use it, or don't use it... makes absolutely no difference unless the disk is under 2.2TB.

 

Joe L.

Link to comment

Been awhile since I have last pre-cleard a drive. Do I still need to use -A option on unRaid 5.0.6 with 4TB drives (HGST Deskstar NAS)?

You can use it, or not...  It is silently ignored on  disks over 2.2TB as it has no meaning or purpose on disks using a GPT partition (all disks over 2.2TB use a GPT partition)

 

So... use it, or don't use it... makes absolutely no difference unless the disk is under 2.2TB.

 

Joe L.

 

should I use it for disks under 2.2TB? for example 500GB? and under what circumstances?

Link to comment
  • 1 month later...

Hi all, quick one. I used the preclear script a few years back to add in some 3TB drives. It was good doing it via the script as I could tell the percentage etc of the clears. I've now recently added in a new disk and simply kicked off the clear via the unraid GUI.

 

Obviously the page just sits on connecting until the clear if finished. My question is, is there a command I could run via telnet for it to bring up it's current clearing progress and stage? and also, when initiated via the unraid gui, what is the default amount of cycles it's set to? ie 1,3, 5? etc.

 

Thanks in advance!

Link to comment

Not sure if there is a way that f getting progress when adding a drive via the GUI.

 

Note that the GUI way is functionally different in operation to the pr_clear script.  All it does is write zeroes to the drive so it can be added without breaking parity.    It has no concept of passes or reading from the drive to check it is good.

Link to comment

Ah right, I see. Thanks for the clarification :)

 

It's just been 3 days now (for a 3TB drive) and wondering how much longer before I can get it back up and running is all.

That does not sound right.  I would expect something more like 9 hours.

 

It might be worth posting a syslog so we can see if it gives any indication of what is going on.

Link to comment
  • 2 months later...

Hi,

 

I just need a refresher in using the preclear script again.  The last time I ran it was 2013 and on an old rc 5.x version of Unraid so it has been awhile.

 

I have Unraid 5.0.5 and I have the 1.15 version of preclear script.  I have a drive that is an old one with 512 byte sectors. Does this process sound right?

 

Stop the array

Powerdown the server

Add drive

Leave the new drive unassigned to the array

Start the array

 

Login via telnet and run

 

./preclear_disk.sh -a -c 3 -r 65536 -w 65536 -b 200 /dev/sdx

 

Then add the disk to the array assuming everything went fine with the preclear.  I'm not using Unmenu and I'm doing everything the old fashioned way which is fine as I will upgrade everything with Unraid 6.0 release.

 

Thanks!

 

Link to comment
  • 3 weeks later...

Can you set the maximum speed when preclearing?

 

I faced an issue when I was preclearing my new 6 TB HDDs, the speed was around 150 MBps, and it cause my live array to be unresponsive.

 

I had the same issue.  Invoke the preclear_disk.sh script using "nice":

 

nice -n 19 preclear_disk.sh /dev/sdb

 

I now have another machine that I use for preclearing.

Link to comment
  • 1 month later...

This may have been answered but a search through the 83 odd pages here hasn't revealed anything...

 

Whenever I run precelar on my server, it invariably locks up the system requiring a hard-reset.

 

My most recent attempt was to preclear a 4TB drive. I opened a screen session and initiated the process. Step 1 completed and step 2 reached 50% when the entire system locked up - VMs froze, GUI was inaccessible, direct typing on the server command line was not accepted. Reset switch was the only way back.

 

This has happened me before on larger drives. I had problems with a previous 4TB drive but I thought it might be the drive itself. Since the, I've replaced the MB and PSU on the system and this is a brand new HD.

 

Normally I'd try this in another system but I don't have any other unraidable systems that support > 2Tb drives.

 

Pointers appreciated.

 

Peter

 

Link to comment

This may have been answered but a search through the 83 odd pages here hasn't revealed anything...

 

Whenever I run precelar on my server, it invariably locks up the system requiring a hard-reset.

 

My most recent attempt was to preclear a 4TB drive. I opened a screen session and initiated the process. Step 1 completed and step 2 reached 50% when the entire system locked up - VMs froze, GUI was inaccessible, direct typing on the server command line was not accepted. Reset switch was the only way back.

 

This has happened me before on larger drives. I had problems with a previous 4TB drive but I thought it might be the drive itself. Since the, I've replaced the MB and PSU on the system and this is a brand new HD.

 

Normally I'd try this in another system but I don't have any other unraidable systems that support > 2Tb drives.

 

Pointers appreciated.

 

Peter

How much memory do you have?
Link to comment

Hey!

 

I've just updated from 5.0.6 to 6.0 RC3.  Is there something I have to keep in mind if I want to use preclear under 6.0?

 

If I start preclear using telnet it will be terminated when I close the telnet window. Will it also be terminated if I use SSH and close the SSH window?

 

My server is headless, is screen still the only option to keep preclear running if I close the terminal? Can I use screen under 6.0?

 

Sorry for the -maybe- stupid questions but it seems a lot has changed since 5.0.6  :o

Link to comment

Hey!

 

I've just updated from 5.0.6 to 6.0 RC3.  Is there something I have to keep in mind if I want to use preclear under 6.0?

 

If I start preclear using telnet it will be terminated when I close the telnet window. Will it also be terminated if I use SSH and close the SSH window?

 

My server is headless, is screen still the only option to keep preclear running if I close the terminal? Can I use screen under 6.0?

 

Sorry for the -maybe- stupid questions but it seems a lot has changed since 5.0.6  :o

Use screen. You can install it with the NerdPack plugin.
Link to comment

This may have been answered but a search through the 83 odd pages here hasn't revealed anything...

 

Whenever I run precelar on my server, it invariably locks up the system requiring a hard-reset.

 

My most recent attempt was to preclear a 4TB drive. I opened a screen session and initiated the process. Step 1 completed and step 2 reached 50% when the entire system locked up - VMs froze, GUI was inaccessible, direct typing on the server command line was not accepted. Reset switch was the only way back.

 

This has happened me before on larger drives. I had problems with a previous 4TB drive but I thought it might be the drive itself. Since the, I've replaced the MB and PSU on the system and this is a brand new HD.

 

Normally I'd try this in another system but I don't have any other unraidable systems that support > 2Tb drives.

 

Pointers appreciated.

 

Peter

How much memory do you have?

 

FFS.

 

Got to 8% on the first stage and locked up the server. Can't ssh, access webGUI or type on the server console itself. VMs still running and shares accessible but that's it.

 

This is super frustrating.

 

P

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.