Unassigned Devices Preclear - a utility to preclear disks before adding them to the array


dlandon

Recommended Posts

Unassigned Devices Preclear (UD Preclear) is an add on plugin to UD that allows you to manage preclearing disks either from the UD page or a separate page (Tools->Disk Utilities->Disk Preclear) meant to manage preclearing unassigned disks.  This plugin is derived from the Preclear Disk Plugin authored by gfjardim.

 

Disks that are candidates for preclearing will have a preclear icon next to the disk serial number in UD, and will be listed in the 'Tools->Disk Utilities->Settings->Disk Preclear' page.  Only presenting unassigned disks that are eligible candidates for a preclear will prevent accidental data loss.  You don't want to preclear a disk with data on it.

 

For a disk to be considered a preclear candidate, it must have one of the following attributes:

  • An Unassigned disk with no partitions.
  • An Unassigned disk with a preclear signature and a partition with no file system.

There are two methods available for preclearing:

  • 'Enhanced' preclear written by gfjardim that supports pausing and resuming preclears.  This is the recommended script to use.
  • 'Docker' preclear is the preclear docker container authored by binhex using the latest version of the legacy preclear script.  The advantage to using the docker container is that preclears are isolated to the docker container.  This preclear script cannot be paused and resumed, and doesn't support notifications.  The binhex preclear docker must be installed and running before this will show as a preclear choice.

Both methods write a preclear signature that is accepted by Unraid.  When a disk is precleared by UD Preclear, and subsequently installed in the array, Unraid will accept it as a cleared disk and not perform another lengthy clear operation.  You do not have to pre-format the disk before adding it to the array.  Unraid accepts a pre-cleared disk and will not do a clear when the disk in added if it sees the preclear signature.

 

Preclears are done in the background so you don't have to leave any consoles open.  The Tmux package is included so the preclears can run on background consoles.  Please do not update the Tmux package through the NerdPack GUI.  The latest version of Tmux has a bug and messes up the status display because the terminal size is defaulted to 24x80 and can't be changed.

 

If you want to format the disk for use in UD or as a Pool device, click the red 'X' next to the preclear status line to remove the Preclear Log, then click the red 'X' next to the serial number to remove the preclear signature and all partitions.  You'll then be able to format the disk.

 

The choices for preclear operations are:

  • Clear Disk - The disk is cleared and the Unraid signature is written to the disk.  This will create an empty partition on the disk.
  • Erase Disk - Erase the entire disk by writing all zeros.  The preclear signature is removed.
  • Erase and Clear Disk - random patterns are written to the disk and then zeroes are written.  This is a good choice for a disk to be discard to ensure all traces of files are removed.
  • Verify Signature - The disk signature is checked.
  • Verify Disk - The disk signature is checked and the disk is verified to be zeroed.
  • Post-read Verify - Perform a post read verify to check the disk.  This is not available in the 'Enhanced' script.
  • Clear Signature - Remove the preclear signature from the disk.  This is not available in the 'Enhanced' script.

You can follow the status of preclears in UD, on the UD Preclear page, or by hovering your mouse on the preclear icon in the Unraid footer.

 

While preclearing disks is not necessary with the current Unraid versions, users get a sense of confidence with pre-testing their disks.  When the original preclear script was written, Unraid would take the array off line while it cleared a disk if it was not precleared.  That's why the script is called 'preclear'.  In the old days, preclearing a disk would prepare it for the array and Unraid would not take the array offline when a new disk was added.

 

UD Preclear is a direct replacement for the Preclear Disk Plugin.  You will have to remove that plugin before the UD Preclear can be installed.  UD must be installed before UD Preclear will install.  Go to 'Apps' and search for Unassigned Devices Preclear and install it from there.  UD+ is not necessary for preclearing, although you'll need it to delete partitions and format disks.

 

Because UD Preclear uses some features in Unraid starting with 6.9, it is not backwards compatible with any Unraid version older than 6.9.
 

  • Like 6
  • Thanks 2
Link to comment

Some Best Practices:

  • Use the 'Enhanced' preclear script.  Preclears will be paused if the array is shutdown or rebooted preventing hang-ups and unclean shutdowns.
  • Once a preclear is completed, just add the disk to the array.  You don't need to format the disk first.
  • It is not recommended to run multiple cycles on a disk.  One cycle will normally be enough to find a premature failure of a disk.  UD Preclear has a max limit of three cycles.  There is no reason to do more than that unless you want to kill your new disk.
  • If you used the binhex Docker Container to preclear a disk and you want to then clear the disk so you can format it, do a 'Clear Signature' operation on the binhex docker and then clear the disk in UD.  UD doesn't remove a binhex docker generated signature.
Link to comment
  • Squid pinned this topic
On 2/27/2022 at 9:55 AM, dlandon said:

 

If you want to format the disk for use in UD, click the red 'X' next to the preclear status line to remove the Preclear Log, then click the red 'X' next to the serial number to remove the preclear signature and all partitions.  You'll then be able to format the disk.

Suggestion, add a popup confirmation stating something to the effect of "Formatting the disk will remove the clear status, and Unraid will have to go through the lengthy clearing process AGAIN if you want to add this disk to an empty data slot of a parity protected array. New Unraid data disks must NOT be formatted before adding them to the array. Are you SURE you want to do this?"

Link to comment
1 hour ago, JonathanM said:

Suggestion, add a popup confirmation stating something to the effect of "Formatting the disk will remove the clear status, and Unraid will have to go through the lengthy clearing process AGAIN if you want to add this disk to an empty data slot of a parity protected array. New Unraid data disks must NOT be formatted before adding them to the array. Are you SURE you want to do this?"

We can't always assume that a disk being cleared is a precleared disk.  I've added a shortened version of this in the clear disk dialog just to give a heads up if the disk is precleared.

Picture1.png.a8cc5dac7c261166273d8470c739596c.png

 

This would be the first step required before formating a precleared disk and would remove the signature at this point.

 

Edit: This message only shows if the preclear plugin is installed.

  • Like 1
Link to comment
29 minutes ago, RiDDiX said:

Anyway it works flawlessly tested it already with a 16GB Toshiba MAG

Thanks for the feedback.  I assume you used the 'Enhanced' script?  That script is great for notifications on the progress of a preclear.

 

I had to do a lot of rewrite to the background task being used before that was called 'rc.diskinfo'.  I had several instances where a preclear would hang.

 

There was a lot of stuff duplicated by the Preclear Disk plugin that was already being handled by UD.  That's why it is an add on to UD.  It uses some UD functionality.

 

Most of the work other than that was Gui stuff, and I felt it very important to only present disks that were actual candidates for a preclear, rather than all unassigned disks.

 

I think in general you'll find this version a lot smoother.

Link to comment
3 hours ago, dlandon said:

Thanks for the feedback.  I assume you used the 'Enhanced' script?  That script is great for notifications on the progress of a preclear.

 

I had to do a lot of rewrite to the background task being used before that was called 'rc.diskinfo'.  I had several instances where a preclear would hang.

 

There was a lot of stuff duplicated by the Preclear Disk plugin that was already being handled by UD.  That's why it is an add on to UD.  It uses some UD functionality.

 

Most of the work other than that was Gui stuff, and I felt it very important to only present disks that were actual candidates for a preclear, rather than all unassigned disks.

 

I think in general you'll find this version a lot smoother.

 

I always try software as it is. I used the "Enhanced" script because its the only option. xD It worked out of the box without any problems. Last time I used the "old" preclear plugin it took approximately 20 hours +/-. Maybe some other stuff was running but with this plugin here it took 16 hours +/- some minutes.

So far cant tell anything badly behaving or something similar. It just works thank you.

Link to comment

I have a stack of 24 ssd that were part of a linux array that I'm pre-clearing.

I figure this would be a good time to switch from the preclear plugin to UD w/UD prelear to test out.

 

Uninstalled precleared, installed UD and put in SSD and it show "There are no disks that can be precleared."

So normal preclear plugin normally would work fine as it doesn't care what was on the disk before.

 

UD Preclear wont touch it because it sees it was part of a raid and has existing filesystem as you noted above.

So installed UD+, and nuke partition on the disk and now I see I can preclear it.

 

So had to install UD, UD Preclear, UD+ to be able to preclear a drive.

 

Can we make it so the user doesn't need to install UD+ to nuke the drive partition in order to preclear a drive?

 

 

 

 

Link to comment
7 hours ago, zoggy said:

Can we make it so the user doesn't need to install UD+ to nuke the drive partition in order to preclear a drive?

UD is not just about preclearing disks.  The reason for the destructive mode is that most of the time a user would not be removing partitions, clearing disks, or formatting disks and this is a protection to prevent accidental data loss.  The way it currently stands is a good user experience, and It is not appropriate to enable clearing a disk when the destructive mode is disabled just so a preclear user doesn't have to install UD+.

Link to comment

I was noting that the change of going from preclear to UD Preclear means that I had to do an extra step now to get UD+ to remove the partition to clear, which is a change.

Once a user is familiar with the process and the initial steps of getting UD, UD+, UD Preclear installed then it is not a huge deal.

 

 

Link to comment
5 hours ago, JonathanM said:

Why not just blkdiscard them? preclearing ssd's seems like unnecessary wear.

 

The came from an outside source and I want to validate them. They are enterprise ssd with pretty high tbw, so doing a full drive write is trivial for them.

Link to comment

To note small change in output formatting, original gfjardim plugin had the time/speed/result right aligned:

#                                                                                                                          #
#   Step 1 of 5 - Pre-read verification:                                                   [0:32:00 @ 417 MB/s] SUCCESS    #
#   Step 2 of 5 - Zeroing the disk:                                                        [0:45:02 @ 296 MB/s] SUCCESS    #
#   Step 3 of 5 - Writing unRAID's Preclear signature:                                                          SUCCESS    #
#   Step 4 of 5 - Verifying unRAID's Preclear signature:                                                        SUCCESS    #
#   Step 5 of 5 - Post-Read verification:                                                  [0:25:24 @ 525 MB/s] SUCCESS    #
#                                                                                                                          #
#                                                                                                                          #
############################################################################################################################
#                               Cycle elapsed time: 1:42:29 | Total elapsed time: 1:42:29                                  #
############################################################################################################################

 

With new 'enhanced' version it is not:

#   Step 1 of 5 - Pre-read verification: [0:30:36 @ 436 MB/s] SUCCESS                              #
#   Step 2 of 5 - Zeroing the disk: [0:44:05 @ 302 MB/s] SUCCESS                                   #
#   Step 3 of 5 - Writing Unraid's Preclear signature: SUCCESS                                     #
#   Step 4 of 5 - Verifying Unraid's Preclear signature: SUCCESS                                   #
#   Step 5 of 5 - Post-Read verification: [0:25:27 @ 524 MB/s] SUCCESS                             #
#                                                                                                  #
####################################################################################################
#       Cycle elapsed time: 1:40:11 | Total elapsed time: 1:40:12                                  #
####################################################################################################

 

 

So far everything looks to be running the same, similar speed and no issues so far.

Edited by zoggy
Link to comment
8 minutes ago, zoggy said:

I was noting that the change of going from preclear to UD Preclear means that I had to do an extra step now to get UD+ to remove the partition to clear, which is a change.

Once a user is familiar with the process and the initial steps of getting UD, UD+, UD Preclear installed then it is not a huge deal.

 

 

The UD Preclear won't allow preclearing just any unassigned disk.  This was done for safety and yes you have to remove the partitions before a disk can be precleared.

  • Like 1
Link to comment
On 2/27/2022 at 3:55 PM, dlandon said:

Preclears are done in the background so you don't have to leave any consoles open.  The Tmux package is included so the preclears can run on background consoles.  Please do not update the Tmux package through the NerdPack GUI.  The latest version of Tmux has a bug and messes up the status display because the terminal size is defaulted to 24x80 and can't be changed.

 

Hi.

 

I have Tmux installed through the NerdPack as I use it often to run (time consuming) scripts or commands from the terminal. Should I actually uninstall the NerdPack Tmux package before installing this UD Preclear plugin?

 

Thanks.

Link to comment

I was running the preclear operation on one of my drives and I noticed this warning going off every second in the preclear log. I was wondering if this was normal.

 

Quote

Mar 05 04:50:50 preclear_disk_5XW1ZSTY_32575: /usr/local/emhttp/plugins/unassigned.devices.preclear/scripts/preclear_disk.sh: line 976: [: : integer expression expected

 

Edited by EraserRain
Link to comment

I initiated a pre-clear before the fixed version on March 5 which is still running. Using the old version progress preview looks fine/normal but log file view shows the error every second already mentioned in this thread.

 

Mar 06 15:10:15 preclear_disk_71G0A46WFQDH_16775: /usr/local/emhttp/plugins/unassigned.devices.preclear/scripts/preclear_disk.sh: line 1510: [: : integer expression expected
 

If/when the pre-clear completes using the old version can I trust the disk if it completes successfully?

 

Pre-clear progress preview:

 

########################################################################################

Unraid Server Preclear of disk 71G0A46WFQDH

# # Cycle 1 of 1, partition start on sector 64. # # # # Step 1 of 5 - Pre-read verification: [22:50:17 @ 218 MB/s] SUCCESS

# # Step 2 of 5 - Zeroing the disk: [22:55:56 @ 218 MB/s] SUCCESS # # Step 3 of 5 - Writing Unraid's Preclear signature: SUCCESS

# # Step 4 of 5 - Verifying Unraid's Preclear signature: SUCCESS

# # Step 5 of 5 - Post-Read in progress: (33% Done)

# # # # ** Time elapsed: 6:03:55 | Current speed: 267 MB/s | Average speed: 276 MB/s # # #

########################################################################################

Link to comment
11 minutes ago, BigFam said:

I initiated a pre-clear before the fixed version on March 5 which is still running. Using the old version progress preview looks fine/normal but log file view shows the error every second already mentioned in this thread.

The error message is related to notifications and does not affect the disk preclear.

 

Note: You can stop the preclear at any time, update the plugin, and then start a preclear and select resume.  It will pick back up where it left off.

  • Thanks 1
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.