[Support] binhex - Preclear

[FreeMan]

3 hours ago, binhex said:

what is the uptime for the container right now?

 

Currently 15 hours. It was 12 when you asked. That would explain a lot...

 

Not sure why it restarted - Server's been up and there wasn't an update.

[binhex]

 
Currently 15 hours. It was 12 when you asked. That would explain a lot...
 
Not sure why it restarted - Server's been up and there wasn't an update.

Ok that's kinda good to confirm it was the container that restarted, but I'm clueless as to why that happened, even if the container stopped it wouldn't restart, and if the script crashed out it also wouldn't cause a restart, so unknown at this time, just check your server uptime incase your server restarted without you knowing.

Sent from my CLT-L09 using Tapatalk

[FreeMan]

hrm... this isn't a good sign. Container uptime 9 hours. Server uptime 2 days, 15 hours 43 minutes.

 

The container shut down at 02:47 this morning. Here's the last line in the log from the start up on 9/21, followed by the shutdown early am on 9/22 and the startup 1 hour later on 9/22.

 

Never mind...

Quote

Appdata Backup

When a Backup is running (either manually or a scheduled backup), your docker applications will be stopped and then restarted at the conclusion of the backup

Because:

 

Disables CA Backup. Makes calendar entry to re-enable it when preclear is finished.

Makes additional note to re-read Note in FAQ Answer #7.  It may do well to highlight that note - it took both of us 2 days to realize what was causing the issue, so it's likely others will miss it as well.

 

 

 

Actually - a better option exists in the CA Backup config:

 

Leave backups enabled, just don't allow it to stop binhex-preclear. I don't know if there would be any open files in the docker that may not get backed up, but considering that there aren't really any options to set (defaults work just fine for me!) and even if you do customize ports or something else, you'll probably have multiple backups made before and after running an individual preclear, this seems reasonable.

Edited September 22, 2019 by FreeMan

[binhex]

hrm... this isn't a good sign. Container uptime 9 hours. Server uptime 2 days, 15 hours 43 minutes.  

The container shut down at 02:47 this morning. Here's the last line in the log from the start up on 9/21, followed by the shutdown early am on 9/22 and the startup 1 hour later on 9/22.

 

Never mind...

 

Appdata Backup When a Backup is running (either manually or a scheduled backup), your docker applications will be stopped and then restarted at the conclusion of the backup

 

Because:

 

Disables CA Backup. Makes calendar entry to re-enable it when preclear is finished.

Makes additional note to re-read Note in FAQ Answer #7.  It may do well to highlight that note - it took both of us 2 days to realize what was causing the issue, so it's likely others will miss it as well.

 

 

 

Actually - a better option exists in the CA Backup config:

 

Leave backups enabled, just don't allow it to stop binhex-preclear. I don't know if there would be any open files in the docker that may not get backed up, but considering that there aren't really any options to set (defaults work just fine for me!) and even if you do customize ports or something else, you'll probably have multiple backups made before and after running an individual preclear, this seems reasonable.

 

 

Lol ok I had ruled that out as I assumed you had read the FAQ and just to confirm there is nothing to backup for this container, apart from final preclear reports which are written to /config

 

Sent from my CLT-L09 using Tapatalk

 

 

 

 

 

[wgstarks]

48 minutes ago, binhex said:

Actually - a better option exists in the CA Backup config:

Where is this? I don’t see anything in CA Backup/Restore that resembles your screenshots at all.

[Squid]

Where is this? I don’t see anything in CA Backup/Restore that resembles your screenshots at all.

Advanced settings

Sent from my NSA monitored device

[wgstarks]

52 minutes ago, Squid said:

Advanced settings

Sent from my NSA monitored device
 

Thanks. The purple text is really hard to see in the dark template and I was trying to find it in the top right corner. Totally missed it the first dozen times. 🙂

[Squid]

Thanks. The purple text is really hard to see in the dark template and I was trying to find it in the top right corner. Totally missed it the first dozen times. 

Submit a report on the thread I will forget all about that

Sent from my NSA monitored device

[FreeMan]

7 hours ago, wgstarks said:

Where is this? I don’t see anything in CA Backup/Restore that resembles your screenshots at all.

Well, there was a big arrow pointing at the text, but I suppose when it was totally out of context, it's hard to figure out where that was actually pointing. Especially since the list of dockers doesn't appear until after you click the "advanced" link.

[FreeMan]

On 9/22/2019 at 12:47 PM, binhex said:

Lol ok I had ruled that out as I assumed you had read the FAQ and just to confirm there is nothing to backup for this container, apart from final preclear reports which are written to /config

Just because I had read it once did not mean that this little detail stuck with me several days later when the drive arrived and it was actually time to kick off the preclear.

 

 

OK. The not-so-comedy of errors continues...

 

It was nearly done - about 30% into the post-read. Honestly, this is the fastest I've ever had a disk preclear (in my memory). It was around 29 hours for an 8TB drive and it was doing the post read at about 200MB/s. I was trying to capture the whole text output to paste here to brag. Instead of hitting ctrl-shift-c to copy the selected text, it seems I hit ctrl-c and aborted the script.

 

I did preclear_binhex.sh -t /dev/sdc to confirm that it was, in fact, precleared. At the end of the output, I get this:

Quote

Partition 1 does not start on physical sector boundary.
########################################################################
========================================================================1.18
==
== DISK /dev/sdc IS PRECLEARED with a GPT Protective MBR
==
============================================================================

With the "Partition 1..." message highlighted in red.

 

Obviously, the disk is cleared. Should I have any concern about the "Partition 1..." message? The original invocation was preclear_binhex.sh -f /dev/sdc exactly as specified in the instructions. I didn't set any other parameters or options.

[binhex]

10 hours ago, FreeMan said:

Just because I had read it once did not mean that this little detail stuck with me several days later when the drive arrived and it was actually time to kick off the preclear.

its all cool, i wasnt having a go, honest, i appreciate the feedback, as you suggested i have put that 'note' regards ca backup at the top of the FAQ now, so hopefully it will be a bit more obvious.

10 hours ago, FreeMan said:

Partition 1 does not start on physical sector boundary.

ok so i believe this is due to the 4k alignment for disks that are larger than 2.2TB, from what i have read this can safely be ignored, i didn't see this in my testing as i was using a 500GB drive and thus the alignment would be 512KB, i will add this to the FAQ.

 

i guess the question is, should you re-run the preclear again?, as its runs quite quickly on your system and you didn't see the final report where it confirms smart attributes havent changed during the preclear to confirm the disk is 'good' i would personally re-run it again if it was a disk i wanted to add to the array, alternatively you COULD just take a look at smartctl values to confirm it looks ok, the option is yours.

[FreeMan]

Slightly more disturbing is the output of    preclear_binhex.sh -f /dev/sdc

 

Quote

 Pre-Clear unRAID Disk /dev/sdc
################################################################## 1.18
Model Family:     Seagate Enterprise Capacity 3.5 HDD
Device Model:     ST8000NM0055-1RM112
Serial Number:    ZA1FS9VW
LU WWN Device Id: 5 000c50 0b6d764f6
Firmware Version: SN05
User Capacity:    8,001,563,222,016 bytes [8.00 TB]
Disk /dev/sdc: 7.3 TiB, 8001563222016 bytes, 15628053168 sectors
Disk model: ST8000NM0055-1RM
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0x00000000

 

Device     Boot Start               End         Sectors Size  Id Type
/dev/sdc1                1 4294967295 4294967295  2T  0 Empty

 

Partition 1 does not start on physical sector boundary.
########################################################################
invoked as  /usr/local/bin/preclear_binhex.sh -f /dev/sdc
########################################################################
(MBR 4k-aligned set. Partition will start on sector 64 for disks <= 2.2TB and sector 1 for disks > 2.2TB)
Are you absolutely sure you want to clear this drive?
(Answer Yes to continue. Capital 'Y', lower case 'es'):

Why does it have one partition of 2TB?

 

I've got 2.4TB of free space left in my data array, so I'm restarting the preclear again. This will be finished before I manage to fill that up.

[binhex]

54 minutes ago, FreeMan said:

Why does it have one partition of 2TB?

i think this is due to a limitation with fdisk, it cannot create (or list) partitions larger than 2TB, taken from an article:-

Quote

Frankly speaking, you cannot create a Linux partition larger than 2 TB using the fdisk command. The fdisk won’t create partitions larger than 2 TB.

i think i will include 'parted' in the image and MAYBE (depends on how much work), switch to using this as it supports GPT partitions, or at least allow people to use parted to confirm.

Edited September 24, 2019 by binhex

[binhex]

1 hour ago, FreeMan said:

Why does it have one partition of 2TB?

if you havent already restarted a preclear then you could try issuing the following command, which should show the correct partition size (supports gpt):-

sfdisk -l /dev/sdX

edit - actually looks like sfdisk may have the same limitation as fdisk - the correct way is to use parted, so im going to include this in the next build.

Edited September 24, 2019 by binhex

[FreeMan]

3 minutes ago, binhex said:

if you havent already restarted a preclear

 

too late...

 

[binhex]

 
too late...
 

Ok no worries

Sent from my CLT-L09 using Tapatalk

[FreeMan]

Well, the docker updated on me last night in the middle of a preclear - when last I looked, it was 98% done with the preread.

 

On the bright side, I got to run

sfdisk -l /dev/sdX

as you suggested, and this is what it gave me:

[root@b377c9e81bea /]# sfdisk -l /dev/sdc
Disk /dev/sdc: 7.3 TiB, 8001563222016 bytes, 15628053168 sectors
Disk model: ST8000NM0055-1RM
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0x00000000

Device     Boot Start        End    Sectors Size Id Type
/dev/sdc1           1 4294967295 4294967295   2T  0 Empty

Partition 1 does not start on physical sector boundary.

It still shows a 2TB partition.

 

Starting the preclear for the... 3rd(?) time...

Edited September 26, 2019 by FreeMan

[binhex]

8 hours ago, FreeMan said:

Well, the docker updated on me last night in the middle of a preclear

hmm of course, ok i will add that an exclusion for auto updates needs to be done for this docker, im sure you have now done that right?.

 

8 hours ago, FreeMan said:

On the bright side, I got to run

sfdisk -l /dev/sdX

yeah i guess you didnt see my edit :-), ok well lets just see what we get at the end of the process, i must say i am a little concerned that its showing as 2TB partition, but according to the forum posts i have read the preclear script should auto detect that the drive is over 2TB and thus use GPT partition type.

 

I have historically always run preclear script with the -A flag, but this is more out of habit than anything, and again according to the forum posts and changelog the -A flag should actually be ignored (drive is over 2TB).

 

so lets see what we get, if it does turn out that its created a 2TB partition then i guess one or more of these is true:-

 

1. the preclear script has a bug, where drives over 2TB are incorrectly getting detected as 2TB drives and thus a dos style partition is created - this MIGHT be worked around by using the -A flag which creates a partitions on sector 64 (without -a or -A it starts on sector 1)

or

2. there is a change in the tool used to create the partition, and this is causing an issue.

 

sadly i currently do not have a drive larger than 500GB to test with, otherwise i would run a preclear myself and see what happens :-(.

 

for ref see the following in the changelog:-

#                 1.13 has the fixes and should work properly for GPT partitions and proper recognition of
#                 default in the absence of "-A" or "-a" option.
# Version 1.14  - Added text describing how -A and -a options are not used or needed on disks > 2.2TB.

 

[wgstarks]

3 hours ago, binhex said:

sadly i currently do not have a drive larger than 500GB to test with, otherwise i would run a preclear myself and see what happens :-(.

I’ll have a new 8TB drive Saturday that I’ll be preclearing. I plan on running 3 passes. Any other flags you want me to try?

[binhex]

28 minutes ago, wgstarks said:

Any other flags you want me to try?

thanks for the offer, so depending on how @FreeMan gets on, if his disk preclears fine and the 2TB issue is purely fdisk limitation then you can run as per the FAQ, obviously in your case with the additional passes flag. 

 

if however freeman has an issue, where the drive preclears but is only showing as a 2TB drive (when in fact its a 8TB drive) then you could try the following, perhaps a single pass to see how it goes first, no point doing 3 passes if its not working as expected:-

preclear_binhex.sh -A -f /dev/sdX # where X is your drive letter

 

if you can also run the following command at the end of the preclear that would be handy too, this should show the size of the partition and partition table, it should be obviously the size of your disk and Partition Table of 'gpt'

parted -l

 

 

 

Edited September 26, 2019 by binhex

[wgstarks]

Can't get this to work.

Sorry: /dev/sdo does not exist as a block device
Clearing will NOT be performed
[root@bef0414753a7 /]# 

/dev/sdo is seen as an unassigned disk.

 

I tried formatting and mounted/unmounted. Same error every time. The drive is installed in the server, not external. Any ideas?

[binhex]

Can't get this to work.

Sorry: /dev/sdo does not exist as a block deviceClearing will NOT be performed[root@bef0414753a7 /]# 

/dev/sdo is seen as an unassigned disk.

 
I tried formatting and mounted/unmounted. Same error every time. The drive is installed in the server, not external. Any ideas?

Try restarting the container

Sent from my CLT-L09 using Tapatalk

[wgstarks]

3 minutes ago, binhex said:

Try restarting the container

Sent from my CLT-L09 using Tapatalk
 

worked thanks

[wgstarks]

Trying to figure out the mail options. Would it be something like this?

preclear_binhex.sh -A -f -M 4 [email protected] /dev/sdX

And not sure what I need to do for this?

Quote

The -m, -M options requires that a valid working mail command is installed.

 

[FreeMan]

40 hours 37 minutes 3 seconds later...

========================================================================1.18
== invoked as: /usr/local/bin/preclear_binhex.sh -f /dev/sdc
== ST8000NM0055-1RM112   ZA1FS9VW
== Disk /dev/sdc has been successfully precleared
== with a starting sector of 1 
== Ran 1 cycle
==
== Using :Read block size = 1000448 Bytes
== Last Cycle's Pre Read Time  : 14:00:01 (158 MB/s)
== Last Cycle's Zeroing time   : 12:18:43 (180 MB/s)
== Last Cycle's Post Read Time : 14:17:20 (155 MB/s)
== Last Cycle's Total Time     : 40:37:03
==
== Total Elapsed Time 40:37:03
==
== Disk Start Temperature: 31C
==
== Current Disk Temperature: 32C, 
==
============================================================================
** Changed attributes in files: /tmp/smart_start_sdc  /tmp/smart_finish_sdc
                ATTRIBUTE   NEW_VAL OLD_VAL FAILURE_THRESHOLD STATUS      RAW_VALUE
      Raw_Read_Error_Rate =    83      82           44        ok          217697608
          Seek_Error_Rate =    77      76           45        ok          55541351
         Spin_Retry_Count =   100     100           97        near_thresh 0
         End-to-End_Error =   100     100           99        near_thresh 0
  Airflow_Temperature_Cel =    68      69           40        ok          32
       G-Sense_Error_Rate =    99     100            0        ok          2432
      Temperature_Celsius =    32      31            0        ok          32
   Hardware_ECC_Recovered =    83      82            0        ok          217697608
 No SMART attributes are FAILING_NOW

 0 sectors were pending re-allocation before the start of the preclear.
 0 sectors were pending re-allocation after pre-read in cycle 1 of 1.
 0 sectors were pending re-allocation after zero of disk in cycle 1 of 1.
 0 sectors are pending re-allocation at the end of the preclear,
    the number of sectors pending re-allocation did not change.
 0 sectors had been re-allocated before the start of the preclear.
 0 sectors are re-allocated at the end of the preclear,
    the number of sectors re-allocated did not change. 
============================================================================

Also:

parted -l

Model: ATA ST8000NM0055-1RM (scsi)
Disk /dev/sdc: 8002GB
Sector size (logical/physical): 512B/4096B
Partition Table: msdos
Disk Flags: 

Number  Start  End  Size  Type  File system  Flags

So it does recognize that it's an 8TB drive. The partition table of "msdos" is a little disconcerting, the others are showing gpt. The others also show partitions under the table listing at the bottom and a file system of "xfs". I would presume at this point that these differences are due to the fact that the disk hasn't been formatted yet.

 

Any thoughts, comments or concerns, or should I just shove it in the array now with a variety of lessons learned?