Preclear plugin

kyis · February 23, 2017

54 minutes ago, wgstarks said:

Thanks. I may just uninstall this until these issues are addressed then. I've got an 8TB drive in hot standby so probably won't need to preclear anything soon. Hopefully I'll get the issue with current versions of unRAID resolved by then and this will all be moot anyway.

~~Preclear plugin archive on github : https://github.com/gfjardim/unRAID-plugins/tree/master/archive~~

Edited February 23, 2017 by kyis

wgstarks · February 23, 2017

@kyis

thanks for the link. I'll grab the previous version from there if I don't still have it in CA Backup.

kyis · February 23, 2017

@wgstarks

History of plg files (way better ^^) following what trurl said : https://github.com/gfjardim/unRAID-plugins/commits/master/plugins/preclear.disk.plg

For the hell of it what does this command return on your system ?:

lsblk -nbP -o name,type,label,size,mountpoint,fstype,serial 2>/dev/null

@ mentions DO work! That's great, no more endless quoting. ^^

Edited February 23, 2017 by kyis

wgstarks · February 23, 2017

7 hours ago, kyis said:
@wgstarks

History of plg files (way better ^^) following what trurl said : https://github.com/gfjardim/unRAID-plugins/commits/master/plugins/preclear.disk.plg

For the hell of it what does this command return on your system ?:
lsblk -nbP -o name,type,label,size,mountpoint,fstype,serial 2>/dev/null
@ mentions DO work! That's great, no more endless quoting. ^^

Looks like nothing?

root@Brunnhilde:~# lsblk -nbP -o name,type,label,size,mountpoint,fstype,serial 2>/dev/null                        
root@Brunnhilde:~#

chepnut · February 24, 2017

Ok, I am still having problems with trying to preclear a 8tb disk, I took a old tower and put a new install of unraid on a usb stick, stuck the 8tb drive in the case, installed unassigned devices and the preclear plugin. It started up, and was moving along for a little over 5 hours, and now it looks like its frozen again. It hasn't progressed any farther than this in about 2 hours

############################################################################################################################
#                                                                                                                          #
#                                         unRAID Server Preclear of disk Z840W7T2                                          #
#                                       Cycle 1 of 1, partition start on sector 64.                                        #
#                                                                                                                          #
#                                                                                                                          #
#   Step 1 of 5 - Pre-read in progress:                                                                      (44% Done)    #
#                                                                                                                          #
#                                                                                                                          #
#                                                                                                                          #
#                                                                                                                          #
#                                                                                                                          #
#                                                                                                                          #
#                                                                                                                          #
#                                                                                                                          #
#                                                                                                                          #
#   ** Time elapsed: 5:16:48 | Current speed: 187 MB/s | Average speed: 187 MB/s                                           #
#                                                                                                                          #
############################################################################################################################
#                               Cycle elapsed time: 5:16:51 | Total elapsed time: 5:16:52                                  #
############################################################################################################################


############################################################################################################################
#                                                                                                                          #
#                                               S.M.A.R.T. Status default                                                  #
#                                                                                                                          #
#                                                                                                                          #
#   ATTRIBUTE                    INITIAL  STATUS                                                                           #
#   5-Reallocated_Sector_Ct      0        -                                                                                #
#   9-Power_On_Hours             108      -                                                                                #
#   183-Runtime_Bad_Block        0        -                                                                                #
#   184-End-to-End_Error         0        -                                                                                #
#   187-Reported_Uncorrect       0        -                                                                                #
#   190-Airflow_Temperature_Cel  25       -                                                                                #
#   197-Current_Pending_Sector   0        -                                                                                #
#   198-Offline_Uncorrectable    0        -                                                                                #
#   199-UDMA_CRC_Error_Count     0        -                                                                                #
#                                                                                                                          #
#                                                                                                                          #
#                                                                                                                          #
############################################################################################################################
#   SMART overall-health self-assessment test result: PASSED                                                               #
############################################################################################################################

Any clues, this is starting to drive me nuts because I am running out of space on my setup and need to add drives fairly soon

root@Tower:~# ps -ef | grep preclear_disk.s
root 17197 17185 0 09:19 pts/0 00:02:53 /bin/bash /usr/local/emhttp/plugins/preclear.disk/script/preclear_disk.sh --cycles 1 --no-prompt /dev/sdf
root 17201 17197 0 09:19 pts/0 00:00:00 /bin/bash /usr/local/emhttp/plugins/preclear.disk/script/preclear_disk.sh --cycles 1 --no-prompt /dev/sdf
root 21224 17197 99 14:36 pts/0 01:36:58 /bin/bash /usr/local/emhttp/plugins/preclear.disk/script/preclear_disk.sh --cycles 1 --no-prompt /dev/sdf
root 21225 21224 0 14:36 pts/0 00:00:00 [preclear_disk.s] <defunct>
root 31341 23833 0 16:13 pts/1 00:00:00 grep preclear_disk.s

And it looks like preclear_disk.s is a zombie again. I don't know if the process is actually running in the background and it just stopped updating the progress screen. And I am not sure what to even look for to see if it is or not

Edited February 24, 2017 by chepnut

kyis · February 24, 2017

@wgstarks ok, typical output should be :

NAME="sdf" TYPE="disk" LABEL="" SIZE="4000787030016" MOUNTPOINT="" FSTYPE="" SERIAL="WD-WCC4E1YTHK4R"
NAME="sdf1" TYPE="part" LABEL="" SIZE="4000786980352" MOUNTPOINT="" FSTYPE="xfs" SERIAL=""
NAME="sdd" TYPE="disk" LABEL="" SIZE="250059350016" MOUNTPOINT="" FSTYPE="" SERIAL="S2R6NX0HB36987T"
NAME="sdd1" TYPE="part" LABEL="" SIZE="250059317248" MOUNTPOINT="" FSTYPE="btrfs" SERIAL=""
NAME="md6" TYPE="md" LABEL="" SIZE="6001175072768" MOUNTPOINT="/mnt/disk6" FSTYPE="" SERIAL=""
NAME="sdb" TYPE="disk" LABEL="" SIZE="6001175126016" MOUNTPOINT="" FSTYPE="" SERIAL="WD-WX11D76D3L8C"
NAME="sdb1" TYPE="part" LABEL="" SIZE="6001175076352" MOUNTPOINT="" FSTYPE="xfs" SERIAL=""
NAME="md4" TYPE="md" LABEL="" SIZE="4000786976768" MOUNTPOINT="/mnt/disk4" FSTYPE="" SERIAL=""
NAME="sdk" TYPE="disk" LABEL="" SIZE="6001175126016" MOUNTPOINT="" FSTYPE="" SERIAL="WD-WX21DC5J246D"
...

And lines where type is "disk" are used by php to populate the array your error says is null.

Try running the command again but remove the "2>/dev/null" maybe it'll give you some useful error.

wgstarks · February 24, 2017

@kyis

Here's the results-

root@Brunnhilde:~# lsblk -nbP -o name,type,label,size,mountpoint,fstype,serial
lsblk: unknown column: serial
root@Brunnhilde:~# lsblk -nbP -o name,type,label,size,mountpoint,fstype
NAME="md1" TYPE="md" LABEL="" SIZE="4000786976768" MOUNTPOINT="/mnt/disk1" FSTYPE=""
NAME="md2" TYPE="md" LABEL="" SIZE="4000786976768" MOUNTPOINT="/mnt/disk2" FSTYPE=""
NAME="md3" TYPE="md" LABEL="" SIZE="4000786976768" MOUNTPOINT="/mnt/disk3" FSTYPE=""
NAME="md4" TYPE="md" LABEL="" SIZE="4000786976768" MOUNTPOINT="/mnt/disk4" FSTYPE=""
NAME="md5" TYPE="md" LABEL="" SIZE="4000786976768" MOUNTPOINT="/mnt/disk5" FSTYPE=""
NAME="md6" TYPE="md" LABEL="" SIZE="4000786976768" MOUNTPOINT="/mnt/disk6" FSTYPE=""
NAME="sda" TYPE="disk" LABEL="" SIZE="320072933376" MOUNTPOINT="" FSTYPE=""
NAME="sda1" TYPE="part" LABEL="" SIZE="320071532544" MOUNTPOINT="/mnt/disks/ST332062_0A" FSTYPE="ext4"
NAME="sdb" TYPE="disk" LABEL="" SIZE="15512174592" MOUNTPOINT="" FSTYPE=""
NAME="sdb1" TYPE="part" LABEL="UNRAID" SIZE="15512173568" MOUNTPOINT="/boot" FSTYPE="vfat"
NAME="sdc" TYPE="disk" LABEL="" SIZE="8001563222016" MOUNTPOINT="" FSTYPE=""
NAME="sdc1" TYPE="part" LABEL="" SIZE="8001563172352" MOUNTPOINT="" FSTYPE=""
NAME="sdd" TYPE="disk" LABEL="" SIZE="1000204886016" MOUNTPOINT="" FSTYPE=""
NAME="sdd1" TYPE="part" LABEL="" SIZE="1000203091968" MOUNTPOINT="/mnt/cache" FSTYPE="xfs"
NAME="sde" TYPE="disk" LABEL="" SIZE="8001563222016" MOUNTPOINT="" FSTYPE=""
NAME="sde1" TYPE="part" LABEL="" SIZE="2199023255040" MOUNTPOINT="" FSTYPE=""
NAME="sdf" TYPE="disk" LABEL="" SIZE="4000787030016" MOUNTPOINT="" FSTYPE=""
NAME="sdf1" TYPE="part" LABEL="" SIZE="4000786980352" MOUNTPOINT="" FSTYPE="xfs"
NAME="sdg" TYPE="disk" LABEL="" SIZE="4000787030016" MOUNTPOINT="" FSTYPE=""
NAME="sdg1" TYPE="part" LABEL="" SIZE="4000786980352" MOUNTPOINT="" FSTYPE="xfs"
NAME="sdh" TYPE="disk" LABEL="" SIZE="4000787030016" MOUNTPOINT="" FSTYPE=""
NAME="sdh1" TYPE="part" LABEL="" SIZE="4000786980352" MOUNTPOINT="" FSTYPE="xfs"
NAME="sdi" TYPE="disk" LABEL="" SIZE="4000787030016" MOUNTPOINT="" FSTYPE=""
NAME="sdi1" TYPE="part" LABEL="" SIZE="4000786980352" MOUNTPOINT="" FSTYPE="xfs"
NAME="sdj" TYPE="disk" LABEL="" SIZE="4000787030016" MOUNTPOINT="" FSTYPE=""
NAME="sdj1" TYPE="part" LABEL="" SIZE="4000786980352" MOUNTPOINT="" FSTYPE="xfs"
NAME="sdk" TYPE="disk" LABEL="" SIZE="4000787030016" MOUNTPOINT="" FSTYPE=""
NAME="sdk1" TYPE="part" LABEL="" SIZE="4000786980352" MOUNTPOINT="" FSTYPE="xfs"
NAME="loop0" TYPE="loop" LABEL="" SIZE="21474836480" MOUNTPOINT="" FSTYPE="btrfs"
root@Brunnhilde:~#

serial showed "unknown column" so I dropped it.

kyis · February 24, 2017

@wgstarks ok here is my version (under unRAID 6.3.2) and truncated columns list which includes SERIAL, I think this column is a requirement you may be stuck until you can update unRAID if it isn't supported by your version. If it has just been renamed maybe...

root@BlackHole:~# lsblk -V
lsblk from util-linux 2.28.2
root@BlackHole:~# lsblk -h
...
Available columns (for --output):
        NAME  device name
       KNAME  internal kernel device name
     MAJ:MIN  major:minor device number
      FSTYPE  filesystem type
  MOUNTPOINT  where the device is mounted
       LABEL  filesystem LABEL
        UUID  filesystem UUID
    PARTTYPE  partition type UUID
   PARTLABEL  partition LABEL
    PARTUUID  partition UUID
   PARTFLAGS  partition flags
          RA  read-ahead of the device
          RO  read-only device
          RM  removable device
     HOTPLUG  removable or hotplug device (usb, pcmcia, ...)
       MODEL  device identifier
      SERIAL  disk serial number
        SIZE  size of the device
       STATE  state of the device
       OWNER  user name
       GROUP  group name
        MODE  device node permissions
   ALIGNMENT  alignment offset
      MIN-IO  minimum I/O size
      OPT-IO  optimal I/O size
     PHY-SEC  physical sector size
     LOG-SEC  logical sector size
        ROTA  rotational device
       SCHED  I/O scheduler name
     RQ-SIZE  request queue size
        TYPE  device type
    DISC-ALN  discard alignment offset
   DISC-GRAN  discard granularity
    DISC-MAX  discard max bytes
   DISC-ZERO  discard zeroes data
       WSAME  write same max bytes
         WWN  unique storage identifier
        RAND  adds randomness
      PKNAME  internal parent kernel device name
        HCTL  Host:Channel:Target:Lun for SCSI
        TRAN  device transport type
  SUBSYSTEMS  de-duplicated chain of subsystems
         REV  device revision
      VENDOR  device vendor
...

wgstarks · February 24, 2017

@kyis

Looks like it's not in the list.


root@Brunnhilde:~# lsblk -h

 

Usage:

 lsblk [options] [<device> ...]

 

Options:

 -a, --all            print all devices

 -b, --bytes          print SIZE in bytes rather than in human readable format

 -d, --nodeps         don't print slaves or holders

 -D, --discard        print discard capabilities

 -e, --exclude <list> exclude devices by major number (default: RAM disks)

 -f, --fs             output info about filesystems

 -h, --help           usage information (this)

 -i, --ascii          use ascii characters only

 -m, --perms          output info about permissions

 -l, --list           use list format ouput

 -n, --noheadings     don't print headings

 -o, --output <list>  output columns

 -P, --pairs          use key="value" output format

 -r, --raw            use raw output format

 -t, --topology       output info about topology

 

Available columns:

       NAME  device name

      KNAME  internal kernel device name

    MAJ:MIN  major:minor device number

     FSTYPE  filesystem type

 MOUNTPOINT  where the device is mounted

      LABEL  filesystem LABEL

       UUID  filesystem UUID

         RO  read-only device

         RM  removable device

      MODEL  device identifier

       SIZE  size of the device

      STATE  state of the device

      OWNER  user name

      GROUP  group name

       MODE  device node permissions

  ALIGNMENT  alignment offset

     MIN-IO  minimum I/O size

     OPT-IO  optimal I/O size

    PHY-SEC  physical sector size

    LOG-SEC  logical sector size

       ROTA  rotational device

      SCHED  I/O scheduler name

    RQ-SIZE  request queue size

       TYPE  device type

   DISC-ALN  discard alignment offset

  DISC-GRAN  discard granularity

   DISC-MAX  discard max bytes

  DISC-ZERO  discard zeroes data

 

For more information see lsblk(8).

root@Brunnhilde:~#

wgstarks · February 24, 2017

Previous post was clipped. Here's the rest-

I've reverted back to 2017.02.05 which was the latest working version I had in backup. This is working good for me for now, I'll just have to remember not to update (unless you have some other suggestion?). Hopefully gfgardim will correct the min version level so that CA won't keep prompting me to update (if I'm understanding that process correctly).

kyis · February 25, 2017

@wgstarks I would wait to be able to update unRAID as a whole. Updating dependencies here and there is asking for trouble. ^^

Squid · February 25, 2017

15 hours ago, wgstarks said:

Previous post was clipped. Here's the rest-

I've reverted back to 2017.02.05 which was the latest working version I had in backup. This is working good for me for now, I'll just have to remember not to update (unless you have some other suggestion?). Hopefully gfgardim will correct the min version level so that CA won't keep prompting me to update (if I'm understanding that process correctly).

The notifications that you're seeing are from one of two places (or both)

Dynamix telling you that there's an update available -> can be disabled completely under notifications settings

FCP complaining that there's an update available -> you can ignore this warning on a per-app basis.

For an author, there are two places to set minimum and maximum unRaid versions:

- Within the plugin itself. This will prevent the plugin from installing / updating on incompatible versions

- Within the xml for CA. This will prevent CA from displaying the app itself in any of its lists (under normal circumstances).

The plugin manager (dynamix) and FCP both just do the check to see if an update is actually available. Neither one of them actually check if the plugin is compatible or not. (As its not really feasible for them to do that short of actually installing the update.

Here's what I would do in your situation:

- Disable the plugin notification within Settings - Notifications. Now dynamix will never check for (and notify you) of any updates available

- Make sure that CA AutoUpdate is installed

- Set it up accordingly to how you want it done. You don't need to have ANY plugins set to auto update if you don't want it to.

- Set the Frequency to check for updates to be something other than disabled (daily is good) - Now AutoUpdate will check for updates, but will only install what you want (if any). Note that AutoUpdate never sends out a notification that an update is available -> only if it installs one (selectable)

- Make sure that Fix Common Problems is installed

- When it sends out a notification for Preclear telling you that an update is available for it, Click on the Ignore button.

- Also click ignore when FCP states that Plugin Update check is not enabled

Now, you will get notifications for any/all plugins having updates available with the exception of Preclear.

Edited February 25, 2017 by Squid

wgstarks · February 25, 2017

3 minutes ago, Squid said:

The notifications that you're seeing are from one of two places (or both)

Dynamix telling you that there's an update available -> can be disabled completely under notifications settings

FCP complaining that there's an update available -> you can ignore this warning on a per-app basis.

For an author, there are two places to set minimum and maximum unRaid versions:

- Within the plugin itself. This will prevent the plugin from installing / updating on incompatible versions

- Within the xml for CA. This will prevent CA from displaying the app itself in any of its lists (under normal circumstances).

The plugin manager (dynamix) and FCP both just do the check to see if an update is actually available. Neither one of them actually check if the plugin is compatible or not. (As its not really feasible for them to do that short of actually installing the update.

Here's what I would do in your situation:

- Disable the plugin notification within Settings - Notifications. Now dynamix will never check for (and notify you) of any updates available

- Make sure that CA AutoUpdate is installed

- Set it up accordingly to how you want it done. You don't need to have ANY plugins set to auto update if you don't want it to.

- Set the Frequency to check for updates to be something other than disabled (daily is good) - Now AutoUpdate will check for updates, but will only install what you want (if any). Note that AutoUpdate never sends out a notification that an update is available -> only if it installs one (selectable)

- Make sure that Fix Common Problems is installed

- When it sends out a notification for Preclear telling you that an update is available for it, Click on the Ignore button.

Now, you will get notifications for any/all plugins having updates available with the exception of Preclear.

Thanks Squid. I may disable checking for updates. Wish it wasn't global though. Looks like CA Auto Update isn't compatible with 6.1.9 though.

Hopefully at some point soon I'll be able to schedule a tech support session with LT and figure out exactly what's causing the issue with current versions of unRAID and what hardware I'll have to replace. LT support is rather expensive though so I'm having to wait for a time when I can be sure of no interruptions. Don't want to be paying $90/hr while I'm trying to unclog the toilet.

Squid · February 25, 2017

1 minute ago, wgstarks said:

Thanks Squid. I may disable checking for updates. Wish it wasn't global though. Looks like CA Auto Update isn't compatible with 6.1.9 though.

Oops... Yeah, I always forget that

But, in the case of any user wishing to only be notified about certain plugins having an update and not others the procedure works (on 6.3+)

Tiger770 · February 27, 2017

Got a new 6TB drive this weekend and ran into the "disk is busy" bug with Joe L,'s script. Before I could find (buried deep in this 50 page post) the link to fix that issue, I tried this plugin and it's working great.

I had originally shied away from this plugin, as based on the wiki page it is slated as being broken. I might suggest an update to the wiki is in order to remove the (false?) information that this plugin is broken.

https://lime-technology.com/wiki/index.php/Add_One_or_More_New_Data_Drives

RE:

======================================================================

===============PLUGIN BELOW IS BROKEN AS OF 8/28/2016=================

======================================================================

RobJ · February 28, 2017

10 hours ago, Tiger770 said:

Got a new 6TB drive this weekend and ran into the "disk is busy" bug with Joe L,'s script. Before I could find (buried deep in this 50 page post) the link to fix that issue, I tried this plugin and it's working great.

I had originally shied away from this plugin, as based on the wiki page it is slated as being broken. I might suggest an update to the wiki is in order to remove the (false?) information that this plugin is broken.

https://lime-technology.com/wiki/index.php/Add_One_or_More_New_Data_Drives

RE:

======================================================================

===============PLUGIN BELOW IS BROKEN AS OF 8/28/2016=================

======================================================================

I've updated that wiki page partially, but it needs more work. Thanks for reporting that.

Dissones4U · March 5, 2017

I've read in the forums that running preclear from screen is recommended due to potential loss of connection which may abort the process. I couldn't figure out nerd tools so I tested this premise by closing the browser after starting preclear. I came back to the gui about 20 minutes later and it was still running as expected. I will leave my remote computer on tonight just in case I'm missing something but it seems to me that I can start preclear then turn off my remote computer and the script will continue to run. Can anyone confirm that this is the case or is nerd tools working in the background somehow? I did install it and start screen, I just couldn't figure out how to initialize a terminal from the gui.

itimpi · March 5, 2017

That advice applied when you started preclear from a telnet/ssh session - not when started via the plugin.

RobJ · March 5, 2017

@gfjardim, this (no need for screen any more) sounds like a good candidate for a Q & A, in your first post.

DZMM · March 7, 2017

I have a new m.2 drive I'm trying to clear nvme0n1 but it doesn't like that the drive name is more than 3 characters - how do I preclear?

Thanks

JorgeB · March 7, 2017

41 minutes ago, DZMM said:

I have a new m.2 drive I'm trying to clear nvme0n1 but it doesn't like that the drive name is more than 3 characters - how do I preclear?

Thanks

You should avoid preclearing solid state devices, I you just want to wipe it use blkdiscard instead:

blkdiscard /dev/nvme0n1

trurl · March 7, 2017

6 minutes ago, johnnie.black said:
You should avoid preclearing solid state devices, I you just want to wipe it use blkdiscard instead:
blkdiscard /dev/nvme0n1

This should be in the FAQ and in the first post of this thread.

thespooler · March 7, 2017

I'm trying to do a preclear in a VirtualBox VM on Windows. I have a trial unRAID, with no config or key, unRAID has started. I have installed the Preclear plugin. The HD is passed through and recognized with VirtualBox's RAW mode. Preclear is running, but there is no SMART info available. Am I just wasting my time with this? Or will Preclear still be able to tell me if the drive is okay through actual drive errors it might encounter? I can always check the SMART stats once it's hooked up natively or through Windows. The drive is a NewEgg HGST refurb (supposedly EOL from a data center) and it passed all the SMART tests with HGST's WinDFT.

DZMM · March 7, 2017

2 hours ago, trurl said:

This should be in the FAQ and in the first post of this thread.

Agree, as I'd already pre-cleared 3 SSDs in the past. Thankfully I found this info in time

thespooler · March 8, 2017

9 hours ago, thespooler said:

I'm trying to do a preclear in a VirtualBox VM on Windows. I have a trial unRAID, with no config or key, unRAID has started. I have installed the Preclear plugin. The HD is passed through and recognized with VirtualBox's RAW mode. Preclear is running, but there is no SMART info available. Am I just wasting my time with this? Or will Preclear still be able to tell me if the drive is okay through actual drive errors it might encounter? I can always check the SMART stats once it's hooked up natively or through Windows. The drive is a NewEgg HGST refurb (supposedly EOL from a data center) and it passed all the SMART tests with HGST's WinDFT.

Hmm, well it doesn't appear to have worked. Preclear is hung at 98% on the first of three cycles' preread stage using the built in preclear script. How should I recover from that? The "/tmp/preclear_stat_sd?" file has a time stamp of 3 hours ago (the preclear itself reports 6 hours have passed, but I started with a reboot and my system uptime is 10 hours. Top shows preclear with 99% usage though the uptime is only 187:33. Is that the missing 3 hours?

Perhaps the preread did complete and the next phase is where it hung.

Pre-Read (1 of 3): 98% @ 126 MB/s (6:30:25)

Preclear plugin

Recommended Posts

Link to comment

Top Posters In This Topic

Popular Days

Top Posters In This Topic

Popular Days

Popular Posts

gfjardim

Squid

CHBMB

Posted Images

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

======================================================================

===============PLUGIN BELOW IS BROKEN AS OF 8/28/2016=================

======================================================================

Link to comment

======================================================================

===============PLUGIN BELOW IS BROKEN AS OF 8/28/2016=================

======================================================================

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Join the conversation