Preclear plugin

[icedragonslair]

ty so much

[s.Oliver]

hey gfjardim, hi guys!

 

guess i got a problem here, somehow, somewhere...

 

installed plug-in & latest script v1.15, everything looks good (file sizes, etc.). the PreClear page does show all unassigned drives.

but when i start a preclear for a disk and accept all the default values (Clear, 1 Cycle, Read/Write size: Default, Skip Pr-read: NO) it says "Starting..." and never changing; even after a hour.

 

this is for an 6 TB WD Red Pro drive, attached to the onboard SAS-controller. i can, however, format the drive without problem. on another new drive i observed quite a bunch of errors while formatting it, so this time i wanted to test all drives beforehand with preclear.

 

when clicking the blue "observe/eye"-icon, i see this attached outputted – and please have a look at the last line:

 

Pre-Clear unRAID Disk /dev/sdd

################################################################## 1.15

smartctl may not be able to run on /dev/sdd with the -d ata option.

however this should not affect the clearing of a disk.

smartctl exit status = 2

smartctl 6.2 2013-07-26 r3841 [x86_64-linux-4.1.7-unRAID] (local build)

Copyright © 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

Read Device Identity failed: Invalid argument

A mandatory SMART command failed: exiting. To continue, add one or more '-T permissive' options.$

Do you wish to continue?

(Answer Yes to continue. Capital 'Y', lower case 'es'): Yes

Disk /dev/sdd: 6001.2 GB, 6001175126016 bytes

255 heads, 63 sectors/track, 729601 cylinders, total 11721045168 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 4096 bytes

I/O size (minimum/optimal): 4096 bytes / 4096 bytes

########################################################################

invoked as  /boot/config/plugins/preclear.disk/preclear_disk.sh -c 1 -W /dev/sdd

########################################################################

(MBR 4k-aligned set. Partition will start on sector 64 for disks <= 2.2TB and sector 1 for disks > 2.2TB)

Are you absolutely sure you want to clear this drive?

(Answer Yes to continue. Capital 'Y', lower case 'es'):

 

it seems to need a last capital "Y" injected.

any ideas, what's wrong with my installation?

 

thx. Blacky

[RobJ]

Pre-Clear unRAID Disk /dev/sdd

################################################################## 1.15

smartctl may not be able to run on /dev/sdd with the -d ata option.

however this should not affect the clearing of a disk.

smartctl exit status = 2

smartctl 6.2 2013-07-26 r3841 [x86_64-linux-4.1.7-unRAID] (local build)

Copyright © 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

Read Device Identity failed: Invalid argument

A mandatory SMART command failed: exiting. To continue, add one or more '-T permissive' options.$

Do you wish to continue?

 

The drive isn't being recognized correctly, on that SAS card.  Try connecting it to a motherboard port, and redo the Preclear.

[s.Oliver]

thx. RobJ, but all other indications do tell something else. unRAID and the plug-in do read out all informations correctly. especially when i simply assign the drive to a slot, then unRAID does all smart-stuff correctly and does format/use the drive without problems.

 

[RobJ]

thx. RobJ, but all other indications do tell something else. unRAID and the plug-in do read out all informations correctly. especially when i simply assign the drive to a slot, then unRAID does all smart-stuff correctly and does format/use the drive without problems.

 

All I could go by is the info you provided, the Preclear output.  According to it, the old Preclear script did not recognize the drive correctly.  I still think there's a better chance for that old script if the drive is on a standard non-SAS SATA port.

[s.Oliver]

hmm.... are there any better solutions out there? i would prefer the GUI vvariant of it, so i followed (and read) lots of articels about that combination plug-in & script. maybe i ask, what you do for pre-clearing?

[ivez]

I want to pre-clear two new 4tb disks, my question is if I add these to my existing tower and try to pre-clear using this script will my array still be accessible for media?

[wgstarks]

I want to pre-clear two new 4tb disks, my question is if I add these to my existing tower and try to pre-clear using this script will my array still be accessible for media?

Yes. You can still read/write to the array while the script is running.

[trurl]

I want to pre-clear two new 4tb disks, my question is if I add these to my existing tower and try to pre-clear using this script will my array still be accessible for media?

Yes it will. That is the main purpose of preclear.

 

If you don't preclear a disk to be added, unRAID will have to clear it so parity remains valid, and your server is not accessible while unRAID clears a disk.

 

The preclear script was developed so you could clear a disk before you give it to unRAID, so it will not have to clear it. Preclear is also often used to test new disks.

[s.Oliver]

well, well... updated the firmware of the onboard sas-controller and now the pre-clear is running; 1st stage, pre-read 3% ~170 MB/s.

 

one question to check correct operation: my drives do make some "crunching" sound; like when the read/write-heads are moving quite much, but always in the same relative movement so emitting the same rhythm/scrubbing-noise.

 

is that normal? one would think, that it would be a linear, continuing (from begin to end) reading/movement.

[JonathanM]

my drives do make some "crunching" sound; like when the read/write-heads are moving quite much, but always in the same relative movement so emitting the same rhythm/scrubbing-noise. is that normal?

Drive models seem to have unique sound signatures, as long as like drives all sound alike it's probably ok. What I don't like to hear is one drive making lots more noise than others of the exact same model.

[s.Oliver]

my drives do make some "crunching" sound; like when the read/write-heads are moving quite much, but always in the same relative movement so emitting the same rhythm/scrubbing-noise. is that normal?

Drive models seem to have unique sound signatures, as long as like drives all sound alike it's probably ok. What I don't like to hear is one drive making lots more noise than others of the exact same model.

 

ya, these days it's called noise acoustic design.

 

update: it took 12 hrs for stage 1 to complete. funny thing is, i started pre-clearing 4 drives nearly simultaneously (just with a few minutes difference), but the last in the row was the first to finish step 1 by quite a difference. same models, same onboard-controller, etc.

[gfjardim]

... The device limit ONLY APPLIES TO STARTING THE ARRAY, not booting unraid. You can have as many drives hooked up as you want, as long as you don't need to actually start the array. Preclear does NOT need the array to be started, and works fine with any number of drives installed. You don't even need to apply for a trial key for preclear use.

 

This is true of course ... but that effectively shuts down the array for the duration of your pre-clears => which, with modern drives (and 2-3 passes) could easily be several days.      Not, in most cases, a very desirable way to go.  One of the key benefits of pre-clearing is you don't have an unavailable array during a clear operation.

 

You don't need to use another system; just unplug all soon-to-clear disks, start the array, plug them in and then start the preclear. NOTE: if you stop the array while preclearing, you won't be able to start it again.

 

Hey, I just loaded up your plugin on a backup machine to preclear some drives using a trial key.  When I go to the PreClear plugin page, It says this at the top and I am unable to use the plugin:

 

Warning: parse_ini_file(/boot/config/disk.cfg): failed to open stream: No such file or directory in /usr/local/emhttp/plugins/preclear.disk/Preclear.php on line 84 Warning: Invalid argument supplied for foreach() in /usr/local/emhttp/plugins/preclear.disk/Preclear.php on line 84

 

Edit - disregard, I hit new config in the tools section and I am good now.

 

This is a bug, and will be fixed. Thanks for reporting this out.

 

 

Is there a fix for this small problem

I am using Chrome as my browser

I have a difficult time getting the Plugin to run after it starts

It takes 10 or 20 or more clicks of the Start button to start the plugin and just as many to the X button to stop the plugin.

 

 

I see this same behavior as well, regardless of which browser I use.  However, it might be related to the array disks being spun down.  If the majority of the disks in my system are spun down, it's sometimes impossible to get the preclear plugin popup to appear.  If the disks are spun up, then it's usually only one or two clicks to get the popup.

 

I continue to experience this failure to open and shutdown problem sometimes it opens after 20 plus clicks and sometimes it refuses to open.

All my other plugins and buttons work after one click on my system.

Can someone please describe how to correct what is happening.

The disk to PreClear is unassigned and should be already spun up

 

I did some testing again with this bug and here is what I found.

The bug occurs consistently with Google Chrome, Firefox and Safari.

The bug does NOT occur and the plugin starts every time with Internet Explorer.

Hopefully the plugin author can figure out the reason.

 

This is difficult one to debug; I'll add some debug capabilities to the plugin in the next version, then we can fix this, ok?

 

 

[gfjardim]

Guys, specially those who have Areca cards, can you please PM me the output of this command?

 

php -r "$(curl -skL https://gist.githubusercontent.com/gfjardim/c81fba928425e7a1dfac/raw/)"

 

Thanks!

[interwebtech]

Guys, specially those who have Areca cards, can you please PM me the output of this command?

 

php -r "$(curl -skL https://gist.githubusercontent.com/gfjardim/c81fba928425e7a1dfac/raw/)"

 

Thanks!

 

login as: root
root@tower's password:
Linux 4.1.7-unRAID.
root@Tower:~# php -r "$(curl -skL https://gist.githubusercontent.com/gfjardim/c81fba928425e7a1dfac/raw/)"

Disks:
+ /dev/sdl => /dev/disk/by-id/scsi-SATA_ST6000DM001-1XY_Z4D23K9N
+ /dev/sdm => /dev/disk/by-id/scsi-SATA_WDC_WD20EZRX-00_WD-WMC4M2038014
+ /dev/sdi => /dev/disk/by-id/scsi-SATA_WDC_WD60EFRX-68_WD-WX51D6402971
+ /dev/sdj => /dev/disk/by-id/scsi-SATA_WDC_WD20EZRX-00_WD-WMC1T0154634
+ /dev/sdh => /dev/disk/by-id/scsi-SATA_ST8000AS0002-1N_Z8406M0L
+ /dev/sdk => /dev/disk/by-id/scsi-SATA_WDC_WD20EZRX-00_WD-WCC300673196
+ /dev/sdn => /dev/disk/by-id/scsi-SATA_Samsung_SSD_840S1DBNEAD812507A
+ /dev/sdg => /dev/disk/by-id/scsi-MD04ACA500_24R6K00RFS9A
+ /dev/sdf => /dev/disk/by-id/scsi-ST2000DM001-1CH1_Z1F5DKBF
+ /dev/sde => /dev/disk/by-id/scsi-ST4000DM000-1F21_Z3024WMZ
+ /dev/sdc => /dev/disk/by-id/scsi-ST4000DM000-1F21_Z3024WY8
+ /dev/sdd => /dev/disk/by-id/scsi-ST6000DX000-1H21_Z4D04L2A
+ /dev/sdb => /dev/disk/by-id/scsi-ARC-1231-RAID00_0000002273521567
+ /dev/sdm => /dev/disk/by-id/ata-WDC_WD20EZRX-00D8PB0_WD-WMC4M2038014
+ /dev/sdk => /dev/disk/by-id/ata-WDC_WD20EZRX-00DC0B0_WD-WCC300673196
+ /dev/sdj => /dev/disk/by-id/ata-WDC_WD20EZRX-00DC0B0_WD-WMC1T0154634
+ /dev/sdi => /dev/disk/by-id/ata-WDC_WD60EFRX-68MYMN1_WD-WX51D6402971
+ /dev/sdl => /dev/disk/by-id/ata-ST6000DM001-1XY17Z_Z4D23K9N
+ /dev/sdh => /dev/disk/by-id/ata-ST8000AS0002-1NA17Z_Z8406M0L
+ /dev/sdn => /dev/disk/by-id/ata-Samsung_SSD_840_EVO_250GB_S1DBNEAD812507A
+ /dev/sda => /dev/disk/by-id/usb-SanDisk_U3_Cruzer_Micro_1031811B45D2F146-0:0
unRAID Serials:
+ ARC-1231-RAID00_0000002273521567
+ MD04ACA500_24R6K00RFS9A
+ ST6000DX000-1H21_Z4D04L2A
+ WDC_WD60EFRX-68MYMN1_WD-WX51D6402971
+ ST6000DM001-1XY17Z_Z4D23K9N
+ WDC_WD20EZRX-00DC0B0_WD-WMC1T0154634
+ WDC_WD20EZRX-00D8PB0_WD-WMC4M2038014
+ WDC_WD20EZRX-00DC0B0_WD-WCC300673196
+ ST8000AS0002-1NA17Z_Z8406M0L
+ ST4000DM000-1F21_Z3024WY8
+ ST4000DM000-1F21_Z3024WMZ
+ ST2000DM001-1CH1_Z1F5DKBF
+ Samsung_SSD_840_EVO_250GB_S1DBNEAD812507A
+ /dev/disk/by-id/usb-SanDisk_U3_Cruzer_Micro_1031811B45D2F146-0:0-part1
+ /dev/disk/by-id/usb-SanDisk_U3_Cruzer_Micro_1031811B45D2F146-0:0
unRAID Disks:
+ /dev/disk/by-id/scsi-ARC-1231-RAID00_0000002273521567
+ /dev/disk/by-id/scsi-MD04ACA500_24R6K00RFS9A
+ /dev/disk/by-id/scsi-ST6000DX000-1H21_Z4D04L2A
+ /dev/disk/by-id/ata-WDC_WD60EFRX-68MYMN1_WD-WX51D6402971
+ /dev/disk/by-id/ata-ST6000DM001-1XY17Z_Z4D23K9N
+ /dev/disk/by-id/ata-WDC_WD20EZRX-00DC0B0_WD-WMC1T0154634
+ /dev/disk/by-id/ata-WDC_WD20EZRX-00D8PB0_WD-WMC4M2038014
+ /dev/disk/by-id/ata-WDC_WD20EZRX-00DC0B0_WD-WCC300673196
+ /dev/disk/by-id/ata-ST8000AS0002-1NA17Z_Z8406M0L
+ /dev/disk/by-id/scsi-ST4000DM000-1F21_Z3024WY8
+ /dev/disk/by-id/scsi-ST4000DM000-1F21_Z3024WMZ
+ /dev/disk/by-id/scsi-ST2000DM001-1CH1_Z1F5DKBF
+ /dev/disk/by-id/ata-Samsung_SSD_840_EVO_250GB_S1DBNEAD812507A
+ /dev/disk/by-id/usb-SanDisk_U3_Cruzer_Micro_1031811B45D2F146-0:0
Unassigned Disks:
+ Array
(
)
root@Tower:~# 

[gfjardim]

Guys, specially those who have Areca cards, can you please PM me the output of this command?

 

php -r "$(curl -skL https://gist.githubusercontent.com/gfjardim/c81fba928425e7a1dfac/raw/)"

 

Thanks!

 

login as: root
root@tower's password:
Linux 4.1.7-unRAID.
root@Tower:~# php -r "$(curl -skL https://gist.githubusercontent.com/gfjardim/c81fba928425e7a1dfac/raw/)"

Disks:
+ /dev/sdl => /dev/disk/by-id/scsi-SATA_ST6000DM001-1XY_Z4D23K9N
+ /dev/sdm => /dev/disk/by-id/scsi-SATA_WDC_WD20EZRX-00_WD-WMC4M2038014
+ /dev/sdi => /dev/disk/by-id/scsi-SATA_WDC_WD60EFRX-68_WD-WX51D6402971
+ /dev/sdj => /dev/disk/by-id/scsi-SATA_WDC_WD20EZRX-00_WD-WMC1T0154634
+ /dev/sdh => /dev/disk/by-id/scsi-SATA_ST8000AS0002-1N_Z8406M0L
+ /dev/sdk => /dev/disk/by-id/scsi-SATA_WDC_WD20EZRX-00_WD-WCC300673196
+ /dev/sdn => /dev/disk/by-id/scsi-SATA_Samsung_SSD_840S1DBNEAD812507A
+ /dev/sdg => /dev/disk/by-id/scsi-MD04ACA500_24R6K00RFS9A
+ /dev/sdf => /dev/disk/by-id/scsi-ST2000DM001-1CH1_Z1F5DKBF
+ /dev/sde => /dev/disk/by-id/scsi-ST4000DM000-1F21_Z3024WMZ
+ /dev/sdc => /dev/disk/by-id/scsi-ST4000DM000-1F21_Z3024WY8
+ /dev/sdd => /dev/disk/by-id/scsi-ST6000DX000-1H21_Z4D04L2A
+ /dev/sdb => /dev/disk/by-id/scsi-ARC-1231-RAID00_0000002273521567
+ /dev/sdm => /dev/disk/by-id/ata-WDC_WD20EZRX-00D8PB0_WD-WMC4M2038014
+ /dev/sdk => /dev/disk/by-id/ata-WDC_WD20EZRX-00DC0B0_WD-WCC300673196
+ /dev/sdj => /dev/disk/by-id/ata-WDC_WD20EZRX-00DC0B0_WD-WMC1T0154634
+ /dev/sdi => /dev/disk/by-id/ata-WDC_WD60EFRX-68MYMN1_WD-WX51D6402971
+ /dev/sdl => /dev/disk/by-id/ata-ST6000DM001-1XY17Z_Z4D23K9N
+ /dev/sdh => /dev/disk/by-id/ata-ST8000AS0002-1NA17Z_Z8406M0L
+ /dev/sdn => /dev/disk/by-id/ata-Samsung_SSD_840_EVO_250GB_S1DBNEAD812507A
+ /dev/sda => /dev/disk/by-id/usb-SanDisk_U3_Cruzer_Micro_1031811B45D2F146-0:0
unRAID Serials:
+ ARC-1231-RAID00_0000002273521567
+ MD04ACA500_24R6K00RFS9A
+ ST6000DX000-1H21_Z4D04L2A
+ WDC_WD60EFRX-68MYMN1_WD-WX51D6402971
+ ST6000DM001-1XY17Z_Z4D23K9N
+ WDC_WD20EZRX-00DC0B0_WD-WMC1T0154634
+ WDC_WD20EZRX-00D8PB0_WD-WMC4M2038014
+ WDC_WD20EZRX-00DC0B0_WD-WCC300673196
+ ST8000AS0002-1NA17Z_Z8406M0L
+ ST4000DM000-1F21_Z3024WY8
+ ST4000DM000-1F21_Z3024WMZ
+ ST2000DM001-1CH1_Z1F5DKBF
+ Samsung_SSD_840_EVO_250GB_S1DBNEAD812507A
+ /dev/disk/by-id/usb-SanDisk_U3_Cruzer_Micro_1031811B45D2F146-0:0-part1
+ /dev/disk/by-id/usb-SanDisk_U3_Cruzer_Micro_1031811B45D2F146-0:0
unRAID Disks:
+ /dev/disk/by-id/scsi-ARC-1231-RAID00_0000002273521567
+ /dev/disk/by-id/scsi-MD04ACA500_24R6K00RFS9A
+ /dev/disk/by-id/scsi-ST6000DX000-1H21_Z4D04L2A
+ /dev/disk/by-id/ata-WDC_WD60EFRX-68MYMN1_WD-WX51D6402971
+ /dev/disk/by-id/ata-ST6000DM001-1XY17Z_Z4D23K9N
+ /dev/disk/by-id/ata-WDC_WD20EZRX-00DC0B0_WD-WMC1T0154634
+ /dev/disk/by-id/ata-WDC_WD20EZRX-00D8PB0_WD-WMC4M2038014
+ /dev/disk/by-id/ata-WDC_WD20EZRX-00DC0B0_WD-WCC300673196
+ /dev/disk/by-id/ata-ST8000AS0002-1NA17Z_Z8406M0L
+ /dev/disk/by-id/scsi-ST4000DM000-1F21_Z3024WY8
+ /dev/disk/by-id/scsi-ST4000DM000-1F21_Z3024WMZ
+ /dev/disk/by-id/scsi-ST2000DM001-1CH1_Z1F5DKBF
+ /dev/disk/by-id/ata-Samsung_SSD_840_EVO_250GB_S1DBNEAD812507A
+ /dev/disk/by-id/usb-SanDisk_U3_Cruzer_Micro_1031811B45D2F146-0:0
Unassigned Disks:
+ Array
(
)
root@Tower:~# 

 

interwebtech, you don't have any unassigned devices right now, correct?

[interwebtech]

That is correct.

[gfjardim]

That is correct.

Ok, thank you very much!

[JustinAiken]

Don't have an Areca card, but in case my output helps anyhow, here it is: https://gist.github.com/JustinAiken/0cd637e1c217a823c37c

 

Thanks for the plug, it's made the last couple of drives I've precleared -so- much easier than remember the commands/flags for both screen and the preclear script itself. 

 

 

[gfjardim]

Don't have an Areca card, but in case my output helps anyhow, here it is: https://gist.github.com/JustinAiken/0cd637e1c217a823c37c

 

Thanks for the plug, it's made the last couple of drives I've precleared -so- much easier than remember the commands/flags for both screen and the preclear script itself.

 

What I wish with this test is to improve array disks detection, to fix that /boot/config/disk.cfg not found bug.

 

It's very helpful, thanks!

[mikedpitt420]

I am helping a friend build a new system. There was initially no disk.cfg as there was no array ever, which made the preclear plugin throw an error about there not being one. After creating the array, starting, then stopping again, I was able to get to start a preclear. He is on unRAID 6.1.3, no controller (all drives plugged directly into mobo) and started preclearing 5 x 4TB WD Purples. Status reported fine on all drives through the preclear plugin, until step 10, where status stopped showing in the UI, and the plugin information just hung at "Retrieving Information" with the UI throwing all kinds of odd errors and becomes largely unresponsive and almost unusable (icons missing, other odd behavior):

 

Warning: readlink(): No such file or directory in /usr/local/emhttp/plugins/dynamix/include/local_prepend.php on line 18

Warning: readlink(): No such file or directory in /usr/local/emhttp/plugins/dynamix/include/local_prepend.php on line 18 43%

core 0 / 1 Warning: readlink(): No such file or directory in /usr/local/emhttp/plugins/dynamix/include/local_prepend.php on line 18 3714 MHz

eth0 Warning: readlink(): No such file or directory in /usr/local/emhttp/plugins/dynamix/include/local_prepend.php on line 18 1000Mb/s - full duplex

 

*I also checked the tmux session just to make sure that 43% at the end of one of the lines wasn't garbled information from a tmux screen, and it's not.

 

Dropping into the only active tmux session showed only one screen, of one drive at ~ 25% done with post read (still moving). I will report back as to whether the GUI un-hangs after it completes, or a reboot was required.

[gfjardim]

I am helping a friend build a new system. There was initially no disk.cfg as there was no array ever, which made the preclear plugin throw an error about there not being one. After creating the array, starting, then stopping again, I was able to get to start a preclear. He is on unRAID 6.1.3, no controller (all drives plugged directly into mobo) and started preclearing 5 x 4TB WD Purples. Status reported fine on all drives through the preclear plugin, until step 10, where status stopped showing in the UI, and the plugin information just hung at "Retrieving Information" with the UI throwing all kinds of odd errors and becomes largely unresponsive and almost unusable (icons missing, other odd behavior):

 

Warning: readlink(): No such file or directory in /usr/local/emhttp/plugins/dynamix/include/local_prepend.php on line 18

Warning: readlink(): No such file or directory in /usr/local/emhttp/plugins/dynamix/include/local_prepend.php on line 18 43%

core 0 / 1 Warning: readlink(): No such file or directory in /usr/local/emhttp/plugins/dynamix/include/local_prepend.php on line 18 3714 MHz

eth0 Warning: readlink(): No such file or directory in /usr/local/emhttp/plugins/dynamix/include/local_prepend.php on line 18 1000Mb/s - full duplex

 

*I also checked the tmux session just to make sure that 43% at the end of one of the lines wasn't garbled information from a tmux screen, and it's not.

 

Dropping into the only active tmux session showed only one screen, of one drive at ~ 25% done with post read (still moving). I will report back as to whether the GUI un-hangs after it completes, or a reboot was required.

 

Can you please post the result of this command: ls -lah /etc/localtime*

[mikedpitt420]

After the tmux session that I could see finished, the GUI was still exhibiting the same issues. He had to perform a reboot, and adding the drives to the array took much longer than expected for precleared drives, so at least one or two of the drives if not all of them never seemed to finish. Is the output of this command still relevant after a reboot? The GUI is now fine, responsive, etc.

[gfjardim]

After the tmux session that I could see finished, the GUI was still exhibiting the same issues. He had to perform a reboot, and adding the drives to the array took much longer than expected for precleared drives, so at least one or two of the drives if not all of them never seemed to finish. Is the output of this command still relevant after a reboot? The GUI is now fine, responsive, etc.

 

I think it's a bug in the core of unRAID. If you didn't setup a time zone in the UI, it's still relevant.

[mikedpitt420]

He did not. Wow, that was probably the last thing that would have dawned on me. Thanks very much.