Preclear plugin

[captain_video]

OK, that makes sense about the rebuild with a replacement disk.  Thanks for the clarification.  However, that brings me back to my original predicament.  I can't seem to preclear a disk that was already precleared and then had data added after being inserted into the array.  I basically want to wipe it clean so I can install it as a new disk without affecting parity, as you mentioned.  I've tried it with several different disks and the preclear function either fails, stalls, or is extremely slow.  The last disk I tried to preclear was only 52% through the pre-read process before it stalled.  That took 19 hours for a 1.5TB disk.

 

Doesn't unRAID write the clear signature to the disk in an area that would be unaffected by a low level format or am I off base with my thought process?  This is why I was asking whether a low level format would zero the disk and still allow unRAID to see it as a precleared disk.  If the rest of the disk is zeroed out then wouldn't that basically be seen by unRAID as a new disk that had been precleared?  I'm just trying to figure out how to make this work without having to do a parity rebuild or do a new configuration from scratch.

 

In a nutshell, how does one install a drive in an unRAID array that been previously used in the same array but was replaced so it is no longer part of the current configuration?  I'd like to clear whatever data is on the drive so unRAID sees it as a blank, precleared drive.  Doing a preclear on the drive doesn't appear to be a workable option, at least not using the plug-in.  I also tried running the command line preclear script but the drive does not show up as a candidate for being precleared, which makes sense since it had already been precleared previously.  I don't really want to install it and do a new configuration because it would do a new parity build with duplicate data on both the old drive and the one that replaced it.

 

I'm starting to think that I may have to install all of the drives in the new enclosure, start with a new configuration, and let it rebuild parity even if the drives contain data.  Once the parity rebuild is complete I can access the individual drives and delete the data from the drives.  Parity will be updated to reflect the deleted data.  This would limit the system to a single parity rebuild and still allow me to access the data.  Does this seem like the only scenario that will work or do I have other options?

[itimpi]

I do not see your problem when using unRAID in normal mode.

 

if the disk is added to a new slot in a parity protected array then unRAID wipes its current contents (by clearing the drive by writing zeroes to it).

if the disk is replacing an existing disk then it is overwritten by the rebuild process.

 

The only time a disks existing contents are retained is if:

- it has previously been partitioned and formatted by unRAID.

- you have added the drive by doing a 'New Config' (which is not the normal way of adding an additional disk)

 

More worrying is the fact that you have pre-clears that do not finish.    That suggests there is some sort of issue at the hardware level on your system.

[trurl]

12 minutes ago, captain_video said:

If the rest of the disk is zeroed out then wouldn't that basically be seen by unRAID as a new disk that had been precleared?

unRAID doesn't look at the whole disk to see if it is clear. That would take a long time. It just looks to see if it has the clear signature, which it won't if it hasn't been cleared by unRAID or preclear. The whole point of the clear signature is so unRAID can see if a disk is clear without having to read the entire disk.

[captain_video]

18 minutes ago, itimpi said:

I do not see your problem when using unRAID in normal mode.

 

if the disk is added to a new slot in a parity protected array then unRAID wipes its current contents (by clearing the drive by writing zeroes to it).

if the disk is replacing an existing disk then it is overwritten by the rebuild process.

 

The only time a disks existing contents are retained is if:

- it has previously been partitioned and formatted by unRAID.

- you have added the drive by doing a 'New Config' (which is not the normal way of adding an additional disk)

 

More worrying is the fact that you have pre-clears that do not finish.    That suggests there is some sort of issue at the hardware level on your system.

 

All of the drives in question have been partitioned and formatted in unRAID so they will not be precleared automatically when added to a new slot.  They were all new, blank drives that had been precleared prior to insertion into the array for the first time.  I have never had an issue using the preclear plug-in or the old command line version on a new drive, only with drives that had been removed from the array and replaced.  None of the disks in question had reported any errors.

 

I agree that it is disconcerting about the drives that are not completing the preclear process.  I intend to run a complete diagnostic on them to see if they're reporting any issues.  This whole situation is confusing and quite perplexing.  I've re-purposed drives from the array before and never had any issues.

Edited April 22, 2017 by captain_video

[trurl]

2 minutes ago, captain_video said:

All of the drives in question have been partitioned and formatted in unRAID so they will not be precleared automatically when added to a new slot.

If you add a disk to a new slot unRAID will clear it if it doesn't have the clear signature, regardless of the history of the disk. It doesn't even know the history of the disk.

[captain_video]

Got it.  I was hoping I wouldn't have to let the array preclear it because it takes it out of commission while the drive is being precleared.

 

Thanks for all of the input.  I appreciate all of the comments and suggestions.

Edited April 22, 2017 by captain_video

[trurl]

4 minutes ago, captain_video said:

Got it.  I was hoping I wouldn't have to let the array preclear it because it takes it out of commission while the drive is being precleared.

 

Thanks for all of the input.  I appreciate all of the comments and suggestions.

unRAID doesn't preclear a disk when you add it, it clears it. Preclear is what you do when you clear a disk before (pre) adding it.

 

Older versions of unRAID would take the array offline when clearing a disk, but recent versions do not.

[captain_video]

Sorry about the semantics.  I'm so used to using preclear I forgot that the terminology changes once the drive is inserted into the array.  I was unaware that the array isn't taken offline with the newer versions when clearing a disk.  That simplifies things considerably and solves my problem, assuming there's nothing wrong with the drives themselves.  I've really got to start reading the release notes when I upgrade my software.

 

Update:  I installed five disks in the external bays and assigned each one to a slot.  I was presented with a warning that all data on the drives would be erased when the array was started.  I started the array and the disks are currently being cleared while still leaving the array active and unaffected.  I can access files and play videos from the server with no problems.  This is exactly what I wanted to do so it's all good.  Like I said, I hadn't read the release notes on any current versions of unRAID so I wasn't aware that it had been fixed to clear the drives and still allow the array to be used normally.  I was so used to it being unavailable during the clearing process that it never occurred to me that the authors may have changed that functionality.  Is it any wonder why I love this program?  Been using it for over nine years now and it was the best money I've ever spent on a piece of software.  It just keeps getting better. 

Edited April 23, 2017 by captain_video

[kriddles]

On 4/22/2017 at 4:38 PM, captain_video said:

Sorry about the semantics.  I'm so used to using preclear I forgot that the terminology changes once the drive is inserted into the array.  I was unaware that the array isn't taken offline with the newer versions when clearing a disk.  That simplifies things considerably and solves my problem, assuming there's nothing wrong with the drives themselves.  I've really got to start reading the release notes when I upgrade my software.

 

Update:  I installed five disks in the external bays and assigned each one to a slot.  I was presented with a warning that all data on the drives would be erased when the array was started.  I started the array and the disks are currently being cleared while still leaving the array active and unaffected.  I can access files and play videos from the server with no problems.  This is exactly what I wanted to do so it's all good.  Like I said, I hadn't read the release notes on any current versions of unRAID so I wasn't aware that it had been fixed to clear the drives and still allow the array to be used normally.  I was so used to it being unavailable during the clearing process that it never occurred to me that the authors may have changed that functionality.  Is it any wonder why I love this program?  Been using it for over nine years now and it was the best money I've ever spent on a piece of software.  It just keeps getting better. 

 

Agreed! I've been using Unraid for a very very long time and it's awesome to see all the changes that have occurred to make it better. The community has been amazing from the get go working on improvements to help out just like this pluggin which I will use for the first time this week. (been using the preclear from Joe L since he wrote it)

[StanC]

On 4/8/2017 at 8:56 AM, Switchblade said:

 

Please answer this question

 

 

Do we have an answer on what was changed on 2017.03.31?

[Squirrel623]

Hey everyone, new user here so bear with me.

 

I started the unRAID v6 trial and I recently installed the preclear plugin so I assume I have the latest version - I can check for the specific version when I get home. I have two new 8TB WD Reds in my server and I am trying to run preclear on both of them (at the same time) but I am running into a problem. Before I go into detail, the problem I am having is that at some point during preclear my server disappears from the network (at least I think it does). I can log into the server directly (not ssh or telnet, but by hooking up a monitor & keyboard) but can't run any commands.

 

Here is a list of events:

1. Start running preclear on both drives. Everything is fine, I can see the pre-read % going up.
2. Telnet into the server and run "tail -f" on the syslog, because this is the second time I am trying to preclear after a failed attempt.
3. Go to bed when pre-read % is at around 25%.
4. Wake up, preclear status page says pre-read is at ~6 hours and  ~35%. By this point it has been ~12 hours since I started the preclear. The server is no longer on the network, so this information is old. I came to the conclusion that the server is no longer on the network because I can't access any web gui pages, and when restarting my router I do not see the server try to grab an IP address. Before restarting the router the server was still in my DHCP table.
5. My telnet session that is running "tail -f" on the syslog stopped updating shortly after I went to bed. Before I went to bed, It was regularly updating. I couldn't see any abnormal output - it looked the same as it had been before I went to bed. I assume it stopped because it couldn't access the server on the network? Would a telnet session automatically close if the server stopped being accessible on the network?
6. Try to log into the server directly. Once logged in I can't seem to run any command. I can type things in the terminal, but hitting enter just takes me to a new line. Trying to ctrl-c or ctrl-z does not do anything except produce "^C" and "^Z" in the terminal that can't be erased.

I am currently at work so I can give more specifics on what the terminal and "tail -f" command output looks like when I get home. I know this might not be a preclear issue, but I figured that I would at least post to see if anyone here knows what is going on. Relevant system specs is CPU:Ryzen 5 1600, RAM: 16GB 2400MHz DDR4.

 

My next step will be to try preclearing one drive at a time because I read a few pages back that someone was having trouble trying to preclear more than one drive at a time. I will also try to monitor more log files (in addition to syslog).

[trurl]

2 minutes ago, Squirrel623 said:

Hey everyone, new user here so bear with me.

...

Relevant system specs is CPU:Ryzen 5 1600, RAM: 16GB 2400MHz DDR4.

...

Have you done a memtest? Maybe something for you in the Ryzen thread.

 

[Squirrel623]

2 hours ago, trurl said:

Have you done a memtest? Maybe something for you in the Ryzen thread.

 

 

No, I haven't tried that. Thanks for the link and suggestion, I will try memtest and take a look at that thread.

[Squirrel623]

Thanks for the help trurl, reading that thread helped a lot! The problem seems to related to my Ryzen CPU - I ended up disabling the "Global C-state Control" and I am currently in the post-read phase (~38 hours in). Sorry for cluttering this thread with a non-preclear issue.

[mhaaland]

Currently clearing a 8TB Red.  Everything was fine until the Post-Read phase which seems to have locked up and not updated the percent or time elapsed.  It should have finished the script at this point.   I did notice the reads on the drive seem to be going up from the main page.     Should I cancel the script or let it run a little longer? 

 

 

############################################################################################################################
#                                                                                                                          #
#                                       unRAID Server Preclear of disk 5236473633565059                                    #
#                                       Cycle 1 of 1, partition start on sector 64.                                        #
#                                                                                                                          #
#                                                                                                                          #
#   Step 1 of 5 - Pre-read verification:                                                  [15:44:19 @ 141 MB/s] SUCCESS    #
#   Step 2 of 5 - Zeroing the disk:                                                       [15:27:21 @ 143 MB/s] SUCCESS    #
#   Step 3 of 5 - Writing unRAID's Preclear signature:                                                          SUCCESS    #
#   Step 4 of 5 - Verifying unRAID's Preclear signature:                                                        SUCCESS    #
#   Step 5 of 5 - Post-Read in progress:                                                                     (37% Done)    #
#                                                                                                                          #
#                                                                                                                          #
#                                                                                                                          #
#                                                                                                                          #
#                                                                                                                          #
#   ** Time elapsed: 4:56:26 | Current speed: 171 MB/s | Average speed: 170 MB/s                                           #
#                                                                                                                          #
############################################################################################################################
#                              Cycle elapsed time: 36:08:07 | Total elapsed time: 36:08:07                                 #
############################################################################################################################

 

[pfp]

13 minutes ago, mhaaland said:

Currently clearing a 8TB Red.  Everything was fine until the Post-Read phase which seems to have locked up and not updated the percent or time elapsed.  It should have finished the script at this point.   I did notice the reads on the drive seem to be going up from the main page.     Should I cancel the script or let it run a little longer? 

 

 

############################################################################################################################
#                                                                                                                          #
#                                       unRAID Server Preclear of disk 5236473633565059                                    #
#                                       Cycle 1 of 1, partition start on sector 64.                                        #
#                                                                                                                          #
#                                                                                                                          #
#   Step 1 of 5 - Pre-read verification:                                                  [15:44:19 @ 141 MB/s] SUCCESS    #
#   Step 2 of 5 - Zeroing the disk:                                                       [15:27:21 @ 143 MB/s] SUCCESS    #
#   Step 3 of 5 - Writing unRAID's Preclear signature:                                                          SUCCESS    #
#   Step 4 of 5 - Verifying unRAID's Preclear signature:                                                        SUCCESS    #
#   Step 5 of 5 - Post-Read in progress:                                                                     (37% Done)    #
#                                                                                                                          #
#                                                                                                                          #
#                                                                                                                          #
#                                                                                                                          #
#                                                                                                                          #
#   ** Time elapsed: 4:56:26 | Current speed: 171 MB/s | Average speed: 170 MB/s                                           #
#                                                                                                                          #
############################################################################################################################
#                              Cycle elapsed time: 36:08:07 | Total elapsed time: 36:08:07                                 #
############################################################################################################################

 

 

I'm attempting to preclear a couple 2tb drives and am seeing the same type of behavior.  First both drives made it to the post read. On the preclear screen one drive was frozen at 26% and the other at 76%.  Neither the percent nor the time was advancing any more.  When I checked the Main > unassigned drives I could see that both drives were still reading.  The both eventually finished but even then preclear showed them frozen.  I'm not sure if it really completed or not.

 

On a second attempt at clearing the disks one disk appeared to freeze in preclear during the preread.  The other disk was looked fine and was well into the zeroing cycle.  I checked the main > unassigned drives screen and the disk that preclear showed as frozen was not reading or writing.  It's just a wild guess but i suspect when preclear freezes the current task finishes but any additional tasks never start.  

 

I'm not sure what to make of this. I stopped this preclear and am now trying it with  the joe l script instead.

 

 

[mhaaland]

1 hour ago, pfp said:

 

I'm attempting to preclear a couple 2tb drives and am seeing the same type of behavior.  First both drives made it to the post read. On the preclear screen one drive was frozen at 26% and the other at 76%.  Neither the percent nor the time was advancing any more.  When I checked the Main > unassigned drives I could see that both drives were still reading.  The both eventually finished but even then preclear showed them frozen.  I'm not sure if it really completed or not.

 

On a second attempt at clearing the disks one disk appeared to freeze in preclear during the preread.  The other disk was looked fine and was well into the zeroing cycle.  I checked the main > unassigned drives screen and the disk that preclear showed as frozen was not reading or writing.  It's just a wild guess but i suspect when preclear freezes the current task finishes but any additional tasks never start.  

 

I'm not sure what to make of this. I stopped this preclear and am now trying it with  the joe l script instead.

 

 

I ended up stopping the script.  I will also try and use the Joe L script.  It there any way to use the modified script to work with GUI or does it need to be run manually through ssh and screen?   I tried to copy the sh file to the preclear.disk folder but it doesn't show up in tools.  Any help would be appreciated.  thanks

 

 

[wgstarks]

14 minutes ago, mhaaland said:

I ended up stopping the script.  I will also try and use the Joe L script.  It there any way to use the modified script to work with GUI or does it need to be run manually through ssh and screen?   I tried to copy the sh file to the preclear.disk folder but it doesn't show up in tools.  Any help would be appreciated.  thanks

 

 

The OP describes how to use whichever script you prefer in the plugin.

[mhaaland]

19 minutes ago, wgstarks said:

The OP describes how to use whichever script you prefer in the plugin.

Q) I want to use one of the older scripts(Joe L. or bjp999) in conjunction with notifications. Is that possible?

A) Yes. I've made some adjustments on both scripts so they become compatible with unRAID notifications; Joe L. version can be found here and bjp999 can be found here.

 

When i try to install the Joe L script from that link it get this error.

 

plugin: installing: https://raw.githubusercontent.com/gfjardim/unRAID-plugins/master/source/packages/preclear_disk_15b.sh
plugin: downloading https://raw.githubusercontent.com/gfjardim/unRAID-plugins/master/source/packages/preclear_disk_15b.sh
plugin: downloading: https://raw.githubusercontent.com/gfjardim/unRAID-plugins/master/source/packages/preclear_disk_15b.sh ... done

Warning: simplexml_load_file(): /tmp/plugins/preclear_disk_15b.sh:1: parser error : Start tag expected, '<' not found in /usr/local/emhttp/plugins/dynamix.plugin.manager/scripts/plugin on line 214

Warning: simplexml_load_file(): #!/bin/bash in /usr/local/emhttp/plugins/dynamix.plugin.manager/scripts/plugin on line 214

Warning: simplexml_load_file(): ^ in /usr/local/emhttp/plugins/dynamix.plugin.manager/scripts/plugin on line 214

Warning: simplexml_load_file(): /tmp/plugins/preclear_disk_15b.sh:1: parser error : Start tag expected, '<' not found in /usr/local/emhttp/plugins/dynamix.plugin.manager/scripts/plugin on line 214

Warning: simplexml_load_file(): #!/bin/bash in /usr/local/emhttp/plugins/dynamix.plugin.manager/scripts/plugin on line 214

Warning: simplexml_load_file(): ^ in /usr/local/emhttp/plugins/dynamix.plugin.manager/scripts/plugin on line 214

Warning: simplexml_load_file(): /tmp/plugins/preclear_disk_15b.sh:1: parser error : Start tag expected, '<' not found in /usr/local/emhttp/plugins/dynamix.plugin.manager/scripts/plugin on line 214

Warning: simplexml_load_file(): #!/bin/bash in /usr/local/emhttp/plugins/dynamix.plugin.manager/scripts/plugin on line 214

Warning: simplexml_load_file(): ^ in /usr/local/emhttp/plugins/dynamix.plugin.manager/scripts/plugin on line 214
plugin: xml parse error

 

[pfp]

51 minutes ago, mhaaland said:

I ended up stopping the script.  I will also try and use the Joe L script.  It there any way to use the modified script to work with GUI or does it need to be run manually through ssh and screen?   I tried to copy the sh file to the preclear.disk folder but it doesn't show up in tools.  Any help would be appreciated.  thanks

 

 

 

35 minutes ago, wgstarks said:

The OP describes how to use whichever script you prefer in the plugin.

 

It's the first Q&A question

 

One thing to note is that you must make the change noted here to the Joe L preclear script for it to run on 6.2+

Run that very last bold command in a telnet session while in the same directory as the Joe L preclear script.

 

Edited May 5, 2017 by pfp

[pfp]

The script is the very last thing in the first post here:

 

But you do have to modify it to run on 6.2.  See my previous post.

 

 

[mhaaland]

3 hours ago, pfp said:

The script is the very last thing in the first post here:

 

But you do have to modify it to run on 6.2.  See my previous post.

 

Thanks. Got it figured out now.  I see there is an option to choose the script you want to use when you run preclear.   Somehow I missed that they run together "in conjunction".  Thanks to all.

 

[mrow]

Will this install and run under unRAID 5.x? If so can you control it from the command line? I have an old tower with a 5 in 3 trayless cage that I use for preclears/wiping drives I'm selling. I use a version of 5 because I can preclear the drives without mounting them/adding them to the array and fire up screen in a remote terminal session and monitor them that way. It'd be nice if I can replace bjp999's script and use this since this is apparently faster.

[DoeBoye]

On 5/5/2017 at 10:45 PM, mrow said:

I use a version of 5 because I can preclear the drives without mounting them/adding them to the array

Not sure if it will install under 5, but for the record, you can use 6.3.3 and the Unassigned Devices plugin. You can preclear drives that are listed under Unassigned Devices, without having to mount/add them to array.

[DoeBoye]

15 hours ago, BRiT said:

You do not need to use Unassigned Devices Plugin to be able to preclear drives using this plugin

Absolutely! Command line works just fine. But the integration with Unassigned Devices is another option if you would like to do it through the GUI and not go through Putty.