[Support] binhex - Preclear


binhex

Recommended Posts

Thank you @binhex for this docker, and also @Frank1940 for the tutorial.

Had a drive fail which I RMA'd back to Seagate so was opportunity to try this docker. Had an interesting final result.

 

From the top of the final report:

== invoked as: /usr/local/bin/preclear_binhex.sh -f -c 2 -M 4 /dev/sdd

== ST10000VN0004-1ZD101 (removed)

== Disk /dev/sdd has been successfully precleared

== with a starting sector of 64

== Ran 2 cycles

==

== Using :Read block size = 1000448 Bytes

== Last Cycle's Pre Read Time : 17:34:23 (158 MB/s)

== Last Cycle's Zeroing time : 0:00:31 (322607 MB/s)

== Last Cycle's Post Read Time : 17:56:33 (154 MB/s)

== Last Cycle's Total Time : 17:58:10

==

== Total Elapsed Time 68:43:17

==

== Disk Start Temperature: 32C

==

== Current Disk Temperature: 33C,

==

============================================================================ 

 

I received 4 email notifications within one minute:

1. Disk /dev/sdd has successfully finished a preclear cycle

2. Zeroing Disk /dev/sdd Started. Disk Temperature: 33C,

3. Zeroing Disk /dev/sdd in progress: 99% complete. ( of 10,000,831,348,736 bytes Wrote ) Disk Temperature: 33C, Next report at 50% Calculated Write Speed: 526359 MB/s Elapsed Time of current cycle: 0:00:19 Total Elapsed time: 50:45:28

4. Zeroing Disk /dev/sdd Done. Zeroing Elapsed Time: 0:00:31 Total Elapsed Time: 50:45:40 Disk Temperature: 33C, Calculated Write Speed: 322607 MB/s

 

Checking with preclear_binhex.sh -t /dev/sdd i got this:

 

Model Family:     Seagate IronWolf
Device Model:     ST10000VN0004-1ZD101
Serial Number:    removed
LU WWN Device Id: 5 000c50 0a51f22c3
Firmware Version: SC61
User Capacity:    10,000,831,348,736 bytes [10.0 TB]
Disk /dev/sdd: 9.1 TiB, 10000831348736 bytes, 19532873728 sectors
Disk model: ST10000VN0004-1Z
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0x00000000

Device     Boot Start        End    Sectors Size Id Type
/dev/sdd1          64 4294967358 4294967295   2T  0 Empty
########################################################################
failed test 6
========================================================================1.19
==
== Disk /dev/sdd is NOT precleared
== 64 4294967295 19532873664
============================================================================

 

Got any thoughts other than trying again?

Link to comment

Strange email status message during preclear-

 


Preclear: Post Read in Progress on sdo 25% complete (inf MB/sec) cycle 2 of 3


Post Read in progress on /dev/sdo: 25% complete.  
(  2,000,495,820,800  of  8,001,563,222,016  bytes read )at inf MB/sec
Disk Temperature: 33C,  
Using Block size of  1,000,448  Bytes
Next report at 50% 
Calculated Read Speed: 196 MB/s 
Elapsed Time of current cycle: 2:49:28
Total Elapsed time: 56:02:26


 

“inf MB/sec”???

Maybe a bug in the reporting? Calculated Read Speed is shown at 196 MB/s so I’m not sure where the infinite is coming from?

Link to comment
On 12/7/2019 at 3:34 PM, wirenut said:

failed test 6
========================================================================1.19
==
== Disk /dev/sdd is NOT precleared
== 64 4294967295 19532873664
============================================================================

 

Got any thoughts other than trying again?

hmm not sure why it failed, especially after claiming a successful preclear!, looking at the script test 6 is to do with a comparison of the partition size on the disk against what it should be set to, snippet from the script:-

 

  if [ $partition_size -ne $sl ]
  then
    cleared_ok="n"
    echo "failed test 6"
  fi
}

do you have unassigned devices plugin installed by any chance?, i wonder if this could be in some way messing with the drive ,btw drive looks healthy to me so i dont have any concern with it being added to the array.

Link to comment
On 12/7/2019 at 3:34 PM, wirenut said:

== Disk /dev/sdd is NOT precleared
== 64 4294967295 19532873664

so looking at the test at the bottom you see this, those numbers are in actual fact:-

 

64 = start sector

4294967295 = sector end of the partition to test against

19532873664= partition size on disk (in sectors) (calculated by full size of disk - number of sectors)

 

so the last number looks to be incorrectly calculated, as you can see from the test result that the start and end look correct:-

On 12/7/2019 at 3:34 PM, wirenut said:

/dev/sdd1          64 4294967358 4294967295   2T  0 Empty

so unless im reading this wrong it looks like a big fat bug!, i will take a look into it but for now i would conclude the disk looks correctly precleared and you can add it to the array, please let me know what unraid thinks of the disk and whether it accepts it as already precleared or not.

Link to comment
1 hour ago, binhex said:

 

do you have unassigned devices plugin installed by any chance?,

I do.

I originally installed it to copy data from old discs awhile back. Occasionally this and with the preclear plugin. Thats all.

Quote

please let me know what unraid thinks of the disk and whether it accepts it as already precleared or not.

I will

 

Thank you.

 

Link to comment
On 12/8/2019 at 8:05 AM, wgstarks said:

Strange email status message during preclear-

 

 


Preclear: Post Read in Progress on sdo 25% complete (inf MB/sec) cycle 2 of 3


Post Read in progress on /dev/sdo: 25% complete.  
(  2,000,495,820,800  of  8,001,563,222,016  bytes read )at inf MB/sec
Disk Temperature: 33C,  
Using Block size of  1,000,448  Bytes
Next report at 50% 
Calculated Read Speed: 196 MB/s 
Elapsed Time of current cycle: 2:49:28
Total Elapsed time: 56:02:26

 


 

“inf MB/sec”???

Maybe a bug in the reporting? Calculated Read Speed is shown at 196 MB/s so I’m not sure where the infinite is coming from?

This seems to only happen with the 25% notification on the post read. Ran 3 cycles and got the same message on each one.

Link to comment
4 hours ago, wgstarks said:

This seems to only happen with the 25% notification on the post read. Ran 3 cycles and got the same message on each one.

strange, im not seeing this in my reports, i precleared a disk and all the reports show the correct speed, or at least not an infinite speed :-), so not sure yet whats causing this, it is of course a cosmetic issue by the look of it.

 

are you sure you saw the same issue on ALL cycles?, i suspect cycle 1 would be ok.

Edited by binhex
Link to comment
27 minutes ago, binhex said:

, it is of course a cosmetic issue by the look of it.

Yes, I agree that it’s likely just cosmetic. Just thought I’d let you know in case you wanted to take a look. It repeated at the same point on the post read for all 3 cycles. Strange that you’re not seeing it.

Link to comment
Just now, wgstarks said:

Strange that you’re not seeing it.

yeah VERY, it reported a speed of just over 200 MB/s for me in the subject line and in the body of the email for all percentages, hmm going to be a bit tricky to fix, what were your parameters for executing the script?.

Link to comment

I run the script on new WD 12TB drive using USB caddy with following statement:

 

preclear_binhex.sh /dev/sdi

 

The process completed, however Unassigned Devices is not allowing me to format the drive. Previously, before running preclear script, UD showed format button. Right now there is only mount button that does nothing. UD is not showing any size of the drive. It is also set to destructive mode. I did not see any errors during preclear process, my questions are:

 

1. Should I run the script again with -A flag?

2 Format drive using console/terminal command?

3. If I swap array drive with new 12TB drive, will unraid recognize it and allow formatting?

 

Thanks

Link to comment
49 minutes ago, talkto_menow said:

I run the script on new WD 12TB drive using USB caddy with following statement:

 

preclear_binhex.sh /dev/sdi

 

The process completed, however Unassigned Devices is not allowing me to format the drive. Previously, before running preclear script, UD showed format button. Right now there is only mount button that does nothing. UD is not showing any size of the drive. It is also set to destructive mode. I did not see any errors during preclear process, my questions are:

 

1. Should I run the script again with -A flag?

2 Format drive using console/terminal command?

3. If I swap array drive with new 12TB drive, will unraid recognize it and allow formatting?

 

Thanks

1. that shouldnt be necessary, no.

2. so what are you trying to achieve here?, is this disk to be added to the array or is this disk to be used as a unassigned drive?

3. assuming you want to add the new drive to the array then yes unraid should recognise the new drive and if its a replacement for an existing drive then it will start rebuilding the contents of the old drive onto the new drive.

 

basically you shouldn't be attempting to use unassigned devices after preclearing a drive UNLESS the drive is going to be used ONLY as an unassigned drive, formatting it or doing anything else with the drive using UD will most probably invalidate the preclear signature and you would need to preclear again (or let unraid do it).

Link to comment
25 minutes ago, binhex said:

1. that shouldnt be necessary, no.

2. so what are you trying to achieve here?, is this disk to be added to the array or is this disk to be used as a unassigned drive?

3. assuming you want to add the new drive to the array then yes unraid should recognise the new drive and if its a replacement for an existing drive then it will start rebuilding the contents of the old drive onto the new drive.

 

basically you shouldn't be attempting to use unassigned devices after preclearing a drive UNLESS the drive is going to be used ONLY as an unassigned drive, formatting it or doing anything else with the drive using UD will most probably invalidate the preclear signature and you would need to preclear again (or let unraid do it).

I bought two 12TB drives. One will replace parity drive and second will be added to the array. UD is not allowing me even touch this drive, so I was worried that something went wrong during preclear process. Some people reported before that they had to use flag -A on their larger drives, but I think this issue was fixed already. Thanks

Link to comment
9 minutes ago, talkto_menow said:

One will replace parity drive and second will be added to the array.

ok in that case do NOT use UD to interact with either of these newly precleared drives.

 

10 minutes ago, talkto_menow said:

Some people reported before that they had to use flag -A on their larger drives, but I think this issue was fixed already. Thanks

it was corrected some time ago yep.

Link to comment
5 hours ago, binhex said:

yeah VERY, it reported a speed of just over 200 MB/s for me in the subject line and in the body of the email for all percentages, hmm going to be a bit tricky to fix, what were your parameters for executing the script?.

It gets weirder...I had a look at my emails again, and two of the drives reported 200MB/s, and the other one "inf MB/sec", only on the 25% post read email. Unfortunately I've lost track of which drive was plugged into where...but I had two drives plugged into a USB hub, and the other one directly into the back of the computer, if that matters. I'm guessing it shouldn't. Thought I'd mention it.

Link to comment

So first time testing this container on my new 8TB drive. "Disk /dev/sdc has been successfully precleared" according to the report. I stopped the array and put the disk in the next available slot and started the array. To my suprise unraid now starts clearing the disk which I thought was already cleared.. No errors shows up in the reports as far as I can see.

 

I'm on 6.8 stable btw. Any thoughts?

Link to comment
On 12/15/2019 at 10:22 PM, strike said:

So first time testing this container on my new 8TB drive. "Disk /dev/sdc has been successfully precleared" according to the report. I stopped the array and put the disk in the next available slot and started the array. To my suprise unraid now starts clearing the disk which I thought was already cleared.. No errors shows up in the reports as far as I can see.

 

I'm on 6.8 stable btw. Any thoughts?

im not ignoring this, honest :-), its noted and i will have to stare at the code some more and see if i can spot any bugs, i know its far from ideal but this is not a show stopper right now, just means an unnecessary long wait for it to be cleared again.

Link to comment
8 hours ago, binhex said:

im not ignoring this, honest :-), its noted and i will have to stare at the code some more and see if i can spot any bugs, i know its far from ideal but this is not a show stopper right now, just means an unnecessary long wait for it to be cleared again.

Good to know you're on top of it. In the meantime have a nice weekend and enjoy your Christmas holiday! :)

Link to comment

FWIW, I have the same scenario as strike.  Using the binhex preclear docker, I successfully precleared a 4TB and 8TB drive.  I stopped the array, assigned the new drives and started the array.  Both drives are "clearing" and should be complete in ~ 24 hours.  I would have thought unRAID would have known the drives were already precleared.  Anyway it's just going to take a bit longer for the second "clearing" cycle to complete, no big deal for now.  Merry Christmas to all!

 

edit:  Another oddity on the second clearing topic.  A few days earlier, I precleared 2 new 10TB drives using the binhex preclear docker, preclear worked fine.  The 2 10TB drives were then assigned as parity replacements (1 10 TB at a time for a dual parity configuration).  I didn't have any issue with unRAID clearing these drives for a second time, meaning unRAID did not clear these 2 10TB drives a second time when I added them as new parity drives and rebuilt/re-synced array parity.  I'm also on 6.8 stable.

Edited by moose
Link to comment
11 hours ago, moose said:

The 2 10TB drives were then assigned as parity replacements (1 10 TB at a time for a dual parity configuration).  I didn't have any issue with unRAID clearing these drives for a second time, meaning unRAID did not clear these 2 10TB drives a second time when I added them as new parity drives and rebuilt/re-synced array parity.

That's not odd it's perfectly normal. The only time a clear disk is needed is to add a completely new data slot in an array with valid parity. ANY replacement operation gets rebuilt with data from the rest of the disks, whether that is a parity or data slot. Any NEW parity is rebuilt (or built, semantics) from the rest of the array, so parity disks NEVER get cleared, regardless.

 

Since the primary function of preclear is now testing disks instead of preparing them for addition to the array, I see little point in focusing major attention on keeping the valid signature portion of preclear functional. My reasoning is...

 

If you trust the disk is good, add it to the array and let unraid do what it needs to do. The array is up and functional the entire time, the only delay is the availability of the newly installed space.

 

If you don't trust the disk, running preclear is a long process anyway, if the disk passes and you wish to then add it, waiting the extra hours for the space to be useable isn't really a big deal in the scheme of things.

 

The only case is for people who wish to keep a disk tested and ready to add capacity, and are too impatient to wait at the moment of addition. I submit that this case is only a super tiny fraction of people.

 

It's nice if it works as designed, but hardly a high priority to fix, since the primary purpose is now testing, not drive insertion prep.

 

Also, I suspect at least some of the preclear signature detection failure by unraid is caused by some action people do with the disks after the preclear is done. ANY writes to the disk will invalidate the signature.

  • Like 1
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.