Preclear plugin


3040 posts in this topic Last Reply

Recommended Posts

Having problems with preclear on 6.6.7.  It starts up fine, but a few hours later I come back and its stopped.  Screenshot is of the preclear screens/log and the system log.  I can't tell what is causing it to stop.  Could it just be the disk failing?  Seems I'd have a error or something that would tell me...

 

Thanks in advance!

hunternas-diagnostics-20190312-0656.zip

Edited by jeffreywhunter
Link to post
  • Replies 3k
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Popular Posts

Hi guys, this is a simple plugin that allows users to clear their disks before add them to the array.   The main characteristics of this plugin are: Modularity: can be used standalon

I think Thanks should be directed at all those members that have been dealing with the mess the last few months. @dlandon and @Squid have been working behind the scenes to try and clean up, and o

Dontlookoverthere over on the unRAID subreddit figured out how to get the plugin running again with a simple edit to the plg:   https://www.reddit.com/r/unRAID/comments/7wjpus/preclear_worki

Posted Images

47 minutes ago, jeffreywhunter said:

Could it just be the disk failing?  Seems I'd have a error or something that would tell me...

Device Model:     ST5000DM000-1FK178
Serial Number:    W4J0J1HS
ID# ATTRIBUTE_NAME          FLAGS    VALUE WORST THRESH FAIL RAW_VALUE
187 Reported_Uncorrect      -O--CK   074   074   000    -    26
197 Current_Pending_Sector  -O--C-   100   100   000    -    112
198 Offline_Uncorrectable   ----C-   100   100   000    -    112

Should show you this in Dashboard SMART warnings and in Attributes on the page for that Unassigned Device.

Link to post
52 minutes ago, jeffreywhunter said:

Having problems with preclear on 6.6.7.  It starts up fine, but a few hours later I come back and its stopped.  Screenshot is of the preclear screens/log and the system log.  I can't tell what is causing it to stop.  Could it just be the disk failing?  Seems I'd have a error or something that would tell me...

 

Thanks in advance!

hunternas-diagnostics-20190312-0656.zip

I need to see the preclear plugin log (see main OP.)

Link to post
  • 2 weeks later...
Mar 26 07:38:32 preclear_disk_ZA21WDL6_9840: Post-Read: cmp command failed - disk not zeroed
Mar 26 07:38:33 preclear_disk_ZA21WDL6_9840: Post-Read: dd - read 9931354800128 of 10000831348736.
Mar 26 07:38:33 preclear_disk_ZA21WDL6_9840: Post-Read: dd command failed, exit code [141].

 

Is this still the same bug from like a year ago where the plugin wasn't zeroing the entire drive? I'm on 6.7-rc5 with latest version of the plugin.

Link to post
24 minutes ago, tronyx said:

Mar 26 07:38:32 preclear_disk_ZA21WDL6_9840: Post-Read: cmp command failed - disk not zeroed
Mar 26 07:38:33 preclear_disk_ZA21WDL6_9840: Post-Read: dd - read 9931354800128 of 10000831348736.
Mar 26 07:38:33 preclear_disk_ZA21WDL6_9840: Post-Read: dd command failed, exit code [141].

 

Is this still the same bug from like a year ago where the plugin wasn't zeroing the entire drive? I'm on 6.7-rc5 with latest version of the plugin.

Another possible cause of post-read fail is bad RAM. Happened to me and others.

Link to post
2 minutes ago, trurl said:

Another possible cause of post-read fail is bad RAM. Happened to me and others.

I haven't seen any other signs of bad RAM and 3 other drives of equal or larger capacity have cleared without issues, but I'll look into it.

Link to post
  • 3 weeks later...

hey there!

 

started yesterday to preclear a new disk and after awhile i recognized that there are continuously writings to the flash drive. it is flipping between 256,0 B/s and 4,6 KB/s (sometimes up to 4,9 KB/s). toggling the display shows an already impressive number of 416870 writes (and 1269 reads).

 

so i'm pretty confident, that these writes do come from the preclear plug-in.

maybe it's writing some kind of status on and on to the flash drive?

 

anyway, it's kinda of a killer feature ;-)

 

edit: after the preclear was successfully completed, these writings stopped and everything is back to normal; but the counter shows an impressive 733000 writes to the usb-stick.

Edited by s.Oliver
Link to post

nobody has seen this strange issue?

 

i'm running on 6.7RC7 and did another test with the next drive to clear; exactly when i started the pre-clear job, these writes started immediately again. stopped the pre-clear job and so the writes to the flash drive stopped.

 

so right now, i simply can't pre-clear anymore, without risking the health status of my flash drive.

 

any help please?

Edited by s.Oliver
Link to post
On 4/12/2019 at 4:41 AM, s.Oliver said:

hey there!

 

started yesterday to preclear a new disk and after awhile i recognized that there are continuously writings to the flash drive. it is flipping between 256,0 B/s and 4,6 KB/s (sometimes up to 4,9 KB/s). toggling the display shows an already impressive number of 416870 writes (and 1269 reads).

 

so i'm pretty confident, that these writes do come from the preclear plug-in.

maybe it's writing some kind of status on and on to the flash drive?

 

anyway, it's kinda of a killer feature ;-)

 

edit: after the preclear was successfully completed, these writings stopped and everything is back to normal; but the counter shows an impressive 733000 writes to the usb-stick.

Yes, the Preclear plugin writes a resume file to the USB stick about three times a minute, and that file has about 400 bytes. If you have a 40 hour Preclear session, you will have a total of 2.74MB written to your flash drive at the end of it. If your flash drive has 32GB and each cell can be written 10.000 times, you can easily write Terabytes to it until it fails.

Link to post
23 hours ago, gfjardim said:

Yes, the Preclear plugin writes a resume file to the USB stick about three times a minute, and that file has about 400 bytes. If you have a 40 hour Preclear session, you will have a total of 2.74MB written to your flash drive at the end of it. If your flash drive has 32GB and each cell can be written 10.000 times, you can easily write Terabytes to it until it fails.

thanks alot gfjardim for clearing things up a bit.

respectfully a few notes/ideas below:

 

my observation about the amount/count of writes is different than what you wrote. with every GUI update on the Main tab of unRAID i see a flip between the 256 Bytes and 4,x KBytes shown. the update is around every 5 secs so in average that would be around 5 KBytes written every 10 secs. = roughly 70 MB per 40 hour pre-clear. now, it might be that the GUI doesn't show real values...

anyway, my concern isn't the rough amount of data written, but the continuously writing (the stick gets written to for lets say 40 hrs. without break) . from my understanding the USB sticks aren't made for that exact use case (usually you write a bunch of files, then they are maybe read again and stay for some time and after a while you delete a few, write a few and so on). i wouldn't count on that everybody does have absolute good quality sticks. and these sticks can get quite hot while in continuously use – might not help with long lasting usage.

 

unRAID's concept is (and please correct me here) to write only absolutely necessary config changes back to the USB stick. all other stuff is in RAM. now i imagine, that you like to have this resume file for some reason. but, because of these really long pre-clear sessions couldn't it be optimized to be on a lesser often schedule? i imagine, that if for some reason the operation is stopped, it probably wouldn't be too bad to have a resume file state of half (or even a full) hour back then on the stick.

 

so my idea would be: use temp storage (ram) location for these high frequency updates and do only copy/or write (newly) once in awhile to a (power loss safe) location on the USB stick. so it would help to dramatically reduce writes to the stick and still have the resume file safely on stick (albeit maybe – and only in case of bigger problems – not in it's latest up to date state). but would it hurt to loose maybe 30-60 min. of documented pre-clear state? and would this be the high priority case which needs to be covered? my unRAID runs pretty stable for long time periods.

 

i only like to help to make it even better (and more safe).

thanks again for your great work on this really appreciated plug-in/tool for unRAID.

Edited by s.Oliver
Link to post

Interesting question.   I was writing to the flash from my parity.tuning plugin every time a parity check was paused or resumed and I was asked if I could reduce that frequency (which is of the order of hours apart) !

 

I think this is only likely to be of much concern for those who have largish dives where a preclear can take many hours (or even days if doing multiple passes).  I would guess something like 30 minutes would be a good compromise between keeping the number of writes down and not having to repeat too much of the preclear on a resume. 

Link to post
3 hours ago, gfjardim said:

@s.Oliver, I couldn't decide it for myself, so I'll open this for debate.

 

Guys, which is the desirable time lapse to write the resume info to the flash drive?

thx. for being open to ideas 🙂

 

i would vote for hourly writes. my pre-clears with newer drives take 32-40 hrs. and i don't trace the progress too often – so i wouldn't miss a time span of 30 mins. anyway.

 

Link to post
22 hours ago, s.Oliver said:

thx. for being open to ideas 🙂

 

i would vote for hourly writes. my pre-clears with newer drives take 32-40 hrs. and i don't trace the progress too often – so i wouldn't miss a time span of 30 mins. anyway.

 

 

Well, my flash drive block is 4k sized. So 15 minutes between writes will result in 160 writes over 40 hour span, which result in 640KB written to the flash drive. I think this is reasonable, isn't it?

Link to post
9 hours ago, gfjardim said:

 

Well, my flash drive block is 4k sized. So 15 minutes between writes will result in 160 writes over 40 hour span, which result in 640KB written to the flash drive. I think this is reasonable, isn't it?

it definitely is and i'm already pre-clearing another big drive (with the new version) – looking pretty good here.

a big, big thank you!

Link to post

I just tried the new version, it's much slower, like 150MB/s to 30MB/s. So I reinstalled the plugin, now the plugin icons are gone completely, can someone confirm the new version is working, Or did I miss something? 

Link to post
1 minute ago, river said:

I just tried the new version, it's much slower, like 150MB/s to 30MB/s. So I reinstalled the plugin, now the plugin icons are gone completely, can someone confirm the new version is working, Or did I miss something? 

Don't know about the icons, but the speed seems more like a hardware issue. Bad connection, etc.

Link to post
50 minutes ago, trurl said:
Don't know about the icons, but the speed seems more like a hardware issue. Bad connection, etc.


Thank you for the fast reply. I’m in the middle of a preclear process, the speed dropped dramatically after upgrading to the new version.

Edited by river
Link to post
1 hour ago, river said:

I just tried the new version, it's much slower, like 150MB/s to 30MB/s. So I reinstalled the plugin, now the plugin icons are gone completely, can someone confirm the new version is working, Or did I miss something? 

Can you please send a screenshot of the missing icons?

Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.