Preclear plugin


3042 posts in this topic Last Reply

Recommended Posts

 
It takes sometimes 3-4 minutes, sometimes up to 20min or more for it to eventually stop. If I issue the stop once and wait, it will eventually stop. But if I try to kill the container or the process, it can get stuck for much longer.
 
I don't notice anything strange with cpu utilization. During preclear, core 0 seems to be pegged at 100% the whole time and the others are pretty much idle (I have 16 virtual cores).
 
I'm only preclearing 1 drive at a time and have plenty of ram (64GB), with only 10% utilization.
 
Actually, now that I tried it a bunch more times, I noticed that with just the preclear, on unraid dashboard, the core 0 utilization bounces between 80% and 100%, but if I try to stop a container, it will remain at 100% constantly. Perhaps it is a cpu bottleneck where both processes hammer core 0 at the same time.
Reading between the lines of this

https://lime-technology.com/forums/topic/33415-stopping-docker-img-causes-array-wide-spin-up/?tab=comments#comment-656791

A sync is issued when docker stops which would explain what you're seeing
Link to post
  • Replies 3k
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Popular Posts

Hi guys, this is a simple plugin that allows users to clear their disks before add them to the array.   The main characteristics of this plugin are: Modularity: can be used standalon

I think Thanks should be directed at all those members that have been dealing with the mess the last few months. @dlandon and @Squid have been working behind the scenes to try and clean up, and o

Dontlookoverthere over on the unRAID subreddit figured out how to get the plugin running again with a simple edit to the plg:   https://www.reddit.com/r/unRAID/comments/7wjpus/preclear_worki

Posted Images

OK.  I just ran a test on my Test Bed server.  I started a preclear on an (old) 3TB drive.  This CPU has a dual core with four threads with a passmark of ~5500.  (that would give a passmark of ~2750 to each core.)  All four  threads vary from ~14% to a high of 100%.  Average CPU usage is in the 35-to-60% range.  

 

I have one Docker installed on this server--- Krusader.  I started-and-stopped that Docker.  (It is normally stopped because when it is started, it keeps the cache drive always spun-up.)  for start operation, the average CPU usage went up in the 90% range with dramatic increases in the thread percentages.  Stopping it, the average CPU usage was in 75% range.  From my previous experience, the actual times were about the same-- four to eight second range. 

Link to post

Hi!

 

After upgrading from unRAID 6.5.0 to 6.5.2 today I get the following errors every 4-5 seconds in my log:

 

May 16 11:32:01 unRAID root: error: /plugins/preclear.disk/Preclear.php: wrong csrf_token
May 16 11:32:05 unRAID root: error: /plugins/preclear.disk/Preclear.php: wrong csrf_token
May 16 11:32:07 unRAID root: error: /plugins/preclear.disk/Preclear.php: wrong csrf_token
May 16 11:32:13 unRAID root: error: /plugins/preclear.disk/Preclear.php: wrong csrf_token
May 16 11:32:16 unRAID root: error: /plugins/preclear.disk/Preclear.php: wrong csrf_token
May 16 11:32:19 unRAID root: error: /plugins/preclear.disk/Preclear.php: wrong csrf_token
May 16 11:32:25 unRAID root: error: /plugins/preclear.disk/Preclear.php: wrong csrf_token
May 16 11:32:27 unRAID root: error: /plugins/preclear.disk/Preclear.php: wrong csrf_token
May 16 11:32:31 unRAID root: error: /plugins/preclear.disk/Preclear.php: wrong csrf_token
May 16 11:32:37 unRAID root: error: /plugins/preclear.disk/Preclear.php: wrong csrf_token
May 16 11:32:38 unRAID root: error: /plugins/preclear.disk/Preclear.php: wrong csrf_token
May 16 11:32:43 unRAID root: error: /plugins/preclear.disk/Preclear.php: wrong csrf_token
May 16 11:32:49 unRAID root: error: /plugins/preclear.disk/Preclear.php: wrong csrf_token
May 16 11:32:49 unRAID root: error: /plugins/preclear.disk/Preclear.php: wrong csrf_token
May 16 11:32:55 unRAID root: error: /plugins/preclear.disk/Preclear.php: wrong csrf_token
May 16 11:33:00 unRAID root: error: /plugins/preclear.disk/Preclear.php: wrong csrf_token
May 16 11:33:01 unRAID root: error: /plugins/preclear.disk/Preclear.php: wrong csrf_token
May 16 11:33:07 unRAID root: error: /plugins/preclear.disk/Preclear.php: wrong csrf_token
May 16 11:33:11 unRAID root: error: /plugins/preclear.disk/Preclear.php: wrong csrf_token
May 16 11:33:13 unRAID root: error: /plugins/preclear.disk/Preclear.php: wrong csrf_token

What may be the cause of this? Does the plugin need an update to work with 6.5.2?

Link to post
  • 2 weeks later...

Hey,

 

I'm running a preclear on an unassigned 3TB drive, it's done pre read, zeroing and I checked it when I woke up this morning and it just says: "unRAID's signature on the MBR is valid" . The preview is saying that post read has started but does not show a percentage.  Is it stuck or will it continue eventually?

 

UPDATE:  I've just stopped and resumed the pre clear operation and it now just stays on "starting". The preview still shows started post read but I don't see any progress.

 

What am I to do?

Edited by rctneil
Link to post

I'm having a lot of problems with the preclear plugin and I can't determine whether it's a bug in the script, a problem with the drives or a problem with the SATA cables (which I have swapped out).

 

These drives completed pre-read and zeroing just fine but they fail on post-read around 68% every single time... when done as part of a regular clear operation.  They WILL complete an independent "verify all the disk" pass just fine.  The preclear signature is written fine to the disks.  Attempting to resume the post-read always results in extremely slow 1MB/s rate at the beginning that slowly ramps up before failing again eventually.  This is happening on two WD Red 8 TBs ordered direct from the manufacturer.  I'm hesitant to put them in the array if they're really erroring out but I can't tell whether this is the preclear bug from earlier and I should just try a different version of the script or what...

 

Log excerpt:

 

May 28 12:09:51 preclear_disk_R6GZ0XBY_16111: Command: /usr/local/emhttp/plugins/preclear.disk/script/preclear_disk.sh --load-file /boot/config/plugins/preclear.disk/R6GZ0XBY.resume /dev/sdf
May 28 12:09:51 preclear_disk_R6GZ0XBY_16111: Preclear Disk Version: 0.9.6-beta
May 28 12:09:51 preclear_disk_R6GZ0XBY_16111: Restoring previous instance of preclear
May 28 12:09:51 preclear_disk_R6GZ0XBY_16111: S.M.A.R.T. info type: default
May 28 12:09:51 preclear_disk_R6GZ0XBY_16111: S.M.A.R.T. attrs type: default
May 28 12:09:54 preclear_disk_R6GZ0XBY_16111: Continuing disk read from byte 5396131479552
May 28 12:09:54 preclear_disk_R6GZ0XBY_16111: Post-Read: verifying the beggining of the disk.
May 28 12:09:54 preclear_disk_R6GZ0XBY_16111: Post-Read: dd if=/dev/sdf bs=512 count=4095 skip=1 conv=notrunc iflag=direct  2>/tmp/.preclear/sdf/dd_output | cmp - /dev/zero &>/tmp/.preclear/sdf/cmp_out
May 28 12:09:54 preclear_disk_R6GZ0XBY_16111: Post-Read: dd pid [18967]
May 28 12:09:54 preclear_disk_R6GZ0XBY_16111: Post-Read: verifying the rest of the disk.
May 28 12:09:54 preclear_disk_R6GZ0XBY_16111: Post-Read: dd if=/dev/sdf bs=2097152 skip=5396131479552 count=2605431742464 conv=notrunc iflag=nocache,sync,count_bytes,skip_bytes 2>/tmp/.preclear/sdf/dd_output | cmp - /dev/zero &>/tmp/.preclear/sdf/cmp_out
May 28 12:09:54 preclear_disk_R6GZ0XBY_16111: Post-Read: dd pid [18983]
May 28 12:10:00 preclear_disk_R6GZ0XBY_16111: Post-Read: dd - read 0 of 8001563222016.
May 28 12:10:00 preclear_disk_R6GZ0XBY_16111: Post-Read: dd command failed, exit code [141].
May 28 12:10:00 preclear_disk_R6GZ0XBY_16111: Post-Read: dd output -> 
May 28 12:10:09 preclear_disk_R6GZ0XBY_20407: Command: /usr/local/emhttp/plugins/preclear.disk/script/preclear_disk.sh --load-file /boot/config/plugins/preclear.disk/R6GZ0XBY.resume /dev/sdf
May 28 12:10:09 preclear_disk_R6GZ0XBY_20407: Preclear Disk Version: 0.9.6-beta
May 28 12:10:09 preclear_disk_R6GZ0XBY_20407: Restoring previous instance of preclear
May 28 12:10:09 preclear_disk_R6GZ0XBY_20407: S.M.A.R.T. info type: default
May 28 12:10:10 preclear_disk_R6GZ0XBY_20407: S.M.A.R.T. attrs type: default
May 28 12:10:12 preclear_disk_R6GZ0XBY_20407: Post-Read: verifying the beggining of the disk.
May 28 12:10:12 preclear_disk_R6GZ0XBY_20407: Post-Read: dd if=/dev/sdf bs=512 count=4095 skip=1 conv=notrunc iflag=direct  2>/tmp/.preclear/sdf/dd_output | cmp - /dev/zero &>/tmp/.preclear/sdf/cmp_out
May 28 12:10:12 preclear_disk_R6GZ0XBY_20407: Post-Read: dd pid [23439]
May 28 12:10:12 preclear_disk_R6GZ0XBY_20407: Post-Read: verifying the rest of the disk.
May 28 12:10:12 preclear_disk_R6GZ0XBY_20407: Post-Read: dd if=/dev/sdf bs=2097152 skip=2097152 count=8001561124864 conv=notrunc iflag=nocache,sync,count_bytes,skip_bytes 2>/tmp/.preclear/sdf/dd_output | cmp - /dev/zero &>/tmp/.preclear/sdf/cmp_out
May 28 12:10:12 preclear_disk_R6GZ0XBY_20407: Post-Read: dd pid [23476]
May 28 12:13:03 preclear_disk_R5GKH9UV_12442: Post-Read: dd - read 5392956391424 of 8001563222016.
May 28 12:13:03 preclear_disk_R5GKH9UV_12442: Post-Read: dd command failed, exit code [141].
May 28 12:13:03 preclear_disk_R5GKH9UV_12442: Post-Read: dd output -> 526+0 records in

Link to post

This probably isn't the correct thread to reply to but it is related.  Earlier some made mention of the preclear utility to clear the disk.  Is there anything in unRaid or a user created plugin to do a DOD wipe similar to the Darik's Boot and Nuke (DBAN) boot disk.

Link to post
Just now, rxnelson said:

Is there anything in unRaid or a user created plugin to do a DOD wipe similar to the Darik's Boot and Nuke (DBAN) boot disk.

No. But you could look up the appropriate command line switches and use badblocks to accomplish the same thing.

Link to post
On 5/28/2018 at 3:40 AM, rctneil said:

Hey,

 

I'm running a preclear on an unassigned 3TB drive, it's done pre read, zeroing and I checked it when I woke up this morning and it just says: "unRAID's signature on the MBR is valid" . The preview is saying that post read has started but does not show a percentage.  Is it stuck or will it continue eventually?

 

UPDATE:  I've just stopped and resumed the pre clear operation and it now just stays on "starting". The preview still shows started post read but I don't see any progress.

 

What am I to do?

You might just want to settle on the drive manufacturers test programs to test it and then add it to your array and let unRAID clear it.  Unlike in versions from 5 and before (maybe even early versions of 6) - the array is NOT offline (with current versions of unRAID) when you add a drive that has to be cleared to the array.  I still use the command line script version of preclear to test my drives and you could use that as well as another option if you want.

Edited by BobPhoenix
Link to post
  • 2 weeks later...

Hello,

 

This is my first time using unRAID and I'd call myself moderately (or semi-moderately) computer-savvy. I've been following the instructions from Space Invader's videos and started preclearing my 2 8TB Seagate Ironwolf drives yesterday. Everything was going fine, until the post-read, when after about an hour each into the post-read they both failed, with this message from both: "Post-Read verification failed - Aborted". Here's the log from one of them:

 

############################################################################################################################
#                                                                                                                          #
#                                         unRAID Server Preclear of disk ZA1B4JKF                                          #
#                                       Cycle 1 of 1, partition start on sector 64.                                        #
#                                                                                                                          #
#                                                                                                                          #
#   Step 1 of 5 - Pre-read verification:                                                  [11:37:02 @ 191 MB/s] SUCCESS    #
#   Step 2 of 5 - Zeroing the disk:                                                       [11:31:22 @ 192 MB/s] SUCCESS    #
#   Step 3 of 5 - Writing unRAID's Preclear signature:                                                          SUCCESS    #
#   Step 4 of 5 - Verifying unRAID's Preclear signature:                                                        SUCCESS    #
#   Step 5 of 5 - Post-Read verification:                                                                          FAIL    #
#                                                                                                                          #
#                                                                                                                          #
#                                                                                                                          #
#                                                                                                                          #
#                                                                                                                          #
#                                                                                                                          #
#                                                                                                                          #
############################################################################################################################
#                              Cycle elapsed time: 24:26:52 | Total elapsed time: 24:26:53                                 #
############################################################################################################################


############################################################################################################################
#                                                                                                                          #
#                                               S.M.A.R.T. Status default                                                  #
#                                                                                                                          #
#                                                                                                                          #
#   ATTRIBUTE                    INITIAL  STATUS                                                                           #
#   5-Reallocated_Sector_Ct      0        -                                                                                #
#   9-Power_On_Hours             6        -                                                                                #
#   184-End-to-End_Error         0        -                                                                                #
#   187-Reported_Uncorrect       0        -                                                                                #
#   190-Airflow_Temperature_Cel  38       -                                                                                #
#   197-Current_Pending_Sector   0        -                                                                                #
#   198-Offline_Uncorrectable    0        -                                                                                #
#   199-UDMA_CRC_Error_Count     0        -                                                                                #
#                                                                                                                          #
#                                                                                                                          #
#                                                                                                                          #
#                                                                                                                          #
############################################################################################################################
#   SMART overall-health self-assessment test result: PASSED                                                               #
############################################################################################################################

--> FAIL: Post-Read verification failed. Your drive is not zeroed.

 

I think I've read in here that this may just be an error with the plugin? If so, am I good to go on these drives? Or what should I do instead? I'm running version 6.5.2 and the plugin is from 2018.05.03. Thank you for your help.

Link to post
1 hour ago, herlschin said:

Is this for me? So this could be a RAM issue?

Yes. Sorry was on my mobile before.

 

I had bad RAM cause this myself. I think others did too. But there was also an issue with the plugin, don't know if it was resolved.Anyway memtest can't hurt.

Link to post
13 minutes ago, trurl said:

Yes. Sorry was on my mobile before.

 

I had bad RAM cause this myself. I think others did too. But there was also an issue with the plugin, don't know if it was resolved.Anyway memtest can't hurt.

 

Thanks. I've been running it and it's already thrown up thousands of errors. So you're probably right.

Link to post
32 minutes ago, herlschin said:

 

Thanks. I've been running it and it's already thrown up thousands of errors. So you're probably right.

 

Any memory errors at all are unacceptable. Try running test with only one module at a time, try running test with module in different slots, etc to isolate the problem.

Link to post
1 hour ago, trurl said:

 

Any memory errors at all are unacceptable. Try running test with only one module at a time, try running test with module in different slots, etc to isolate the problem.

So, what does it mean if I run the test once and get a lot of errors, but run it again (exactly the same way, both sticks, nothing has changed) and get none? It just went through all 4 passes with no errors at all.

Link to post
37 minutes ago, herlschin said:

So, what does it mean if I run the test once and get a lot of errors, but run it again (exactly the same way, both sticks, nothing has changed) and get none? It just went through all 4 passes with no errors at all.

 

It means it is going to be harder to troubleshoot, but troubleshoot you must.All of your servers data and instructions goes through memory. If memory is unreliable everything is.

Link to post
40 minutes ago, herlschin said:

So, what does it mean if I run the test once and get a lot of errors, but run it again (exactly the same way, both sticks, nothing has changed) and get none? It just went through all 4 passes with no errors at all.

It means you have the worst issue possible, a system you can't trust. Keep in mind ALL data that you read and write to the server flows through that RAM, so until you can have confidence in it again, consider all files possibly corrupt.

Link to post
1 hour ago, rcmpayne said:

FYI: Preclearance plugin and 6.5.3 issues

 

 

Are others having this issue?  I just picked up 5x 8TB drives and want to preclear.  I’ll avoid updating unRAID until this is resolved.

 

craigr

Link to post
27 minutes ago, craigr said:

Are others having this issue?  I just picked up 5x 8TB drives and want to preclear.  I’ll avoid updating unRAID until this is resolved.

 

craigr

I’m not. You could always just run the script through and not use the plugin.

Link to post
6 hours ago, wgstarks said:

I’m not. You could always just run the script through and not use the plugin.

It seems that the script is more aggressive on testing the drive and takes a lot longer. Is that my imagination?  

 

craigr 

Link to post
8 hours ago, jonathanm said:

It means you have the worst issue possible, a system you can't trust. Keep in mind ALL data that you read and write to the server flows through that RAM, so until you can have confidence in it again, consider all files possibly corrupt.

 

Good thing this is a brand new build and I don't have any files on there yet!

 

8 hours ago, trurl said:

 

It means it is going to be harder to troubleshoot, but troubleshoot you must.All of your servers data and instructions goes through memory. If memory is unreliable everything is.

 

Yeah, ran it again overnight and there are a few thousand errors again. I'm just going to send this RAM back and get a new set. I'll run memtest right away on the new set.

Link to post
6 hours ago, craigr said:

It seems that the script is more aggressive on testing the drive and takes a lot longer. Is that my imagination?  

 

craigr 

I know it does take longer. Of course preclearing is completely optional these days.

Link to post
I know it does take longer. Of course preclearing is completely optional these days.
Optional but very much highly recommended

Sent from my SM-G935F using Tapatalk

Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.