Preclear plugin

[dlandon]

13 minutes ago, gfjardim said:

 

It's not easy to cache information, update it when needed and keep everything in a consistent state. Let's see if this time it work as expected.

 

Thanks for your feedback.

I can imagine.  The diskinfo daemon is slick, but I can see where you'd get into a race condition easily.

[ekim]

All working now. Just started my preclear. 

 

Thanks

[maratin536]

Model information now showing for me, however when I try start preclear it hangs in 'starting'

 

If I stop this 

 

Trying to run preclear_disk.sh -A /dev/sdd 

and i get the following:
Sorry: Device /dev/sdd is busy.: 1

All 3 of my disks say busy, even though they are unattached.

 

Thanks

[dlandon]

41 minutes ago, maratin536 said:

Model information now showing for me, however when I try start preclear it hangs in 'starting'

 

If I stop this 

 

Trying to run preclear_disk.sh -A /dev/sdd 

and i get the following:
Sorry: Device /dev/sdd is busy.: 1

All 3 of my disks say busy, even though they are unattached.

 

Thanks

Download and post the preclear log and diagnostics.  No one can help you with the limited information you've provided.

[interwebtech]

2 array disks listed in Preclear (were also listed in UD but now fixed with the latest version), 4TB data disk & 8TB parity. I have updated plugins as well as reinstalling. Logs attached.

tower-diagnostics-20170707-0806.zip

TOWER-preclear.disk-20170707-0803.zip

Edited July 7, 2017 by interwebtech

[gfjardim]

On 07/07/2017 at 0:09 PM, interwebtech said:

2 array disks listed in Preclear (were also listed in UD but now fixed with the latest version), 4TB data disk & 8TB parity. I have updated plugins as well as reinstalling. Logs attached.

tower-diagnostics-20170707-0806.zip

TOWER-preclear.disk-20170707-0803.zip

 

Could not found the bug, but change some of the detection routine. If you could please update your plugin and see if it helps, I'll be grateful.

[interwebtech]

14 minutes ago, gfjardim said:

 

Could not found the bug, but change some of the detection routine. If you could please update your plugin and see if it helps, I'll be grateful.

No disks listed for preclear now with plugin updated to 7/8. Thanks!

[gfjardim]

7 minutes ago, interwebtech said:

No disks listed for preclear now with plugin updated to 7/8. Thanks!

 

You're welcome!

[dlandon]

Diskinfo is not detecting mounted status correctly:

BACKUPSERVER-preclear.disk-20170708-1516.zip

[gfjardim]

8 hours ago, dlandon said:

Diskinfo is not detecting mounted status correctly:

BACKUPSERVER-preclear.disk-20170708-1516.zip

 

I don't know why, but diskinfo was stopped when you grabbed that log. Try running it ( /etc/rc.d/rc.diskinfo --daemon ) and see if it's really a bug.

[dlandon]

7 hours ago, gfjardim said:

 

I don't know why, but diskinfo was stopped when you grabbed that log. Try running it ( /etc/rc.d/rc.diskinfo --daemon ) and see if it's really a bug.

When I mount a particular disk the rc.diskinfo daemon stops once in a while.  It's hard for me to reproduce.  I can see the daemon has stopped by ls /var/run/.  This disk does have an unclean shutdown flag set on it.  The log in UD says the mount command failed, but it still mounts.  I'm trying to understand that.  I'm not sure it has anything to do with the rc.diskinfo daemon stopping.

 

I also uninstalled and then re-installed preclear once and the rc.diskinfo daemon did not start.  Again this is not consistently happening.

 

Is there a way to capture a log or something that will show why rc.diskinfo has stopped?

[gfjardim]

4 minutes ago, dlandon said:

When I mount a particular disk the rc.diskinfo daemon stops once in a while.  It's hard for me to reproduce.  I can see the daemon has stopped by ls /var/run/.  This disk does have an unclean shutdown flag set on it.  The log in UD says the mount command failed, but it still mounts.  I'm trying to understand that.  I'm not sure it has anything to do with the rc.diskinfo daemon stopping.

 

I also uninstalled and then re-installed preclear once and the rc.diskinfo daemon did not start.  Again this is not consistently happening.

 

Is there a way to capture a log or something that will show why rc.diskinfo has stopped?

 

1) the detection of a successful mount is done by the command exit code; if it's zero then is successful, if not, it show as an error. Maybe mount is trowing a non-zero exit code when the dirty flag is set even if the mount routine itself is successful.

 

2) the diskinfo.log file is the file you want, but I saw no PHP errors there that could lead to a daemon stop. Maybe the system is killing the daemon in a, out-of-memory event.

[Frank1940]

IF you think it is an out-of-memory issue and since you seem to have a lot of memory installed in your server, you might try this:

 

1-- Install the Tips and Tweaks  plugin.

 

2-- On the 'Tweats' tab, reduce the "Disk Cache 'vm.dirty_background_ratio' (%):"  to    2      and change the " Disk Cache 'vm.dirty_ratio' (%): " to double that value (4). 

 

This value is the percentage of RAM set aside for this function (Delayed writes to disk) and has been the same forever.  While it may be a good value for systems with a few MBs of RAM, it is probably way too much for today's configurations!

Edited July 9, 2017 by Frank1940
"Disk Cache 'vm.dirty_ratio' (%):" was not initially inserted.

[dlandon]

25 minutes ago, gfjardim said:

 

1) the detection of a successful mount is done by the command exit code; if it's zero then is successful, if not, it show as an error. Maybe mount is trowing a non-zero exit code when the dirty flag is set even if the mount routine itself is successful.

 

2) the diskinfo.log file is the file you want, but I saw no PHP errors there that could lead to a daemon stop. Maybe the system is killing the daemon in a, out-of-memory event.

I think it's on my side.  UD mounted the drive with the unclean shutdown, and then decided to try mounting it again after it was already mounted.  Hold on this issue for the moment.  I believe it is on my end.

[dlandon]

54 minutes ago, gfjardim said:

 

1) the detection of a successful mount is done by the command exit code; if it's zero then is successful, if not, it show as an error. Maybe mount is trowing a non-zero exit code when the dirty flag is set even if the mount routine itself is successful.

 

2) the diskinfo.log file is the file you want, but I saw no PHP errors there that could lead to a daemon stop. Maybe the system is killing the daemon in a, out-of-memory event.

I think I've sorted it out.  I'm sure it was caused from my mounting an already mounted disk.

 

The result I am getting from the unclean shutdown disk mount is:

Jul 08 15:02:51 Mount failed with error: The disk contains an unclean file system (0, 0).
Metadata kept in Windows cache, refused to mount.
Falling back to read-only mount because the NTFS partition is in an
unsafe state. Please resume and shutdown Windows fully (no hibernation
or fast restarting.)

I always assumed in UD that this was a mount failure, but in fact the mount is successful and the disk is mounted read only.  I then tried to mount the device read only, but it was already mounted.  It seems that the ntfs mount has changed and a disk in this situation is now mounted read only, when in the past it would fail to mount with this condition and I then mounted it read only myself.

 

I'll post an update to UD that will fix this situation.

[dabl]

I wanted to report a successful preclear with the below.

 

I did not see any unexpected behavior. The preclear was started before the latest updates.

 

Thank you to gfjardim and dlandon for these plugins.

 

======================

 

Seagate Archive 8 TB ST8000AS0002
Preclear Disks           2017.07.05c
Unassigned Devices 2017.07.04a
unRAID                     6.3.3

[gfjardim]

3 minutes ago, dabl said:

I wanted to report a successful preclear with the below.

 

I did not see any unexpected behavior. The preclear was started before the latest updates.

 

Thank you to gfjardim and dlandon for these plugins.

 

======================

 

Seagate Archive 8 TB ST8000AS0002
Preclear Disks           2017.07.05c
Unassigned Devices 2017.07.04a
unRAID                     6.3.3

 

Thanks for reporting.

[RevAtB]

G,

BUG Report - 

For the past few updates, the preclear update script does not provide a {DONE} button in the gui/web interface.

This is the only one that I use that gets all the way to...

 

-----------------------------------------------------------
preclear.disk has been installed.
This plugin requires Dynamix webGui to operate
Copyright 2015-2017, gfjardim
Version: 2017.07.10
-----------------------------------------------------------

plugin: updated

 

missing                               >>>>>>>>>>>>>>>>>>>>>>>> {DONE}

 

and it just sits there...is it finished? really? 

So to exit the update screen you have to find the lil red (x) to close the output screen.

 

should be a quick and easy fix.

 

Have a great day, and thank you for your efforts.

 

Rev

[tabish121]

I've had a preclear running on a new 8TB WD Read drive and I checked a bit ago and things seemed to have gotten stuck, the preclear_disk process is pegged at 100% CPU

 

I grabbed logs and attached here. 

TOWER-preclear.disk-20170714-1455.zip

[interwebtech]

Just finished Preclear of a Seagate 8TB Archive drive and got the popup to share stats anonymously but it keeps failing. IKs there something I need to set at my end?

[Yousty]

I'm having the same problem as tabish121. I was preclearing an 8TB Seagate and it went perfectly through the pre-read and zeroing and then at 95% of the post-read it just stopped and the timer is frozen at 15:09:01 and now unraid shows the disk as spun down. I also noticed that 1 core of my 8 on the cpu is maxing out at 100%. Here is the preclear log:

Jul 12 18:03:09 preclear_disk_Z840ZW8W_1465: Command: /usr/local/emhttp/plugins/preclear.disk/script/preclear_disk.sh --notify 2 --frequency 3 --cycles 1 --no-prompt /dev/sdh
Jul 12 18:03:09 preclear_disk_Z840ZW8W_1465: Preclear Disk Version: 0.8.9-beta
Jul 12 18:03:09 preclear_disk_Z840ZW8W_1465: S.M.A.R.T. info type: default
Jul 12 18:03:11 preclear_disk_Z840ZW8W_1465: Pre-Read: dd if=/dev/sdh of=/dev/null bs=2097152 conv=notrunc,noerror iflag=direct
Jul 12 18:03:11 preclear_disk_Z840ZW8W_1465: Pre-Read: dd pid [2776]
Jul 13 08:59:20 preclear_disk_Z840ZW8W_1465: Pre-Read: dd - read 8001565319168 of 8001563222016.
Jul 13 08:59:20 preclear_disk_Z840ZW8W_1465: Pre-Read: 0
Jul 13 08:59:21 preclear_disk_Z840ZW8W_1465: Zeroing: ionice -c 3 dd if=/dev/zero of=/dev/sdh bs=2097152 seek=1 conv=noerror,notrunc oflag=direct
Jul 13 08:59:21 preclear_disk_Z840ZW8W_1465: Zeroing: dd pid [4556]
Jul 13 13:24:00 preclear_disk_Z840ZW8W_1465: hdparm exec_time: 1s
Jul 13 13:27:45 preclear_disk_Z840ZW8W_1465: hdparm exec_time: 1s
Jul 13 13:28:07 preclear_disk_Z840ZW8W_1465: hdparm exec_time: 1s
Jul 13 13:30:17 preclear_disk_Z840ZW8W_1465: smartctl exec_time: 1s
Jul 13 13:46:07 preclear_disk_Z840ZW8W_1465: hdparm exec_time: 1s
Jul 13 16:00:50 preclear_disk_Z840ZW8W_1465: smartctl exec_time: 1s
Jul 13 22:24:21 preclear_disk_Z840ZW8W_1465: hdparm exec_time: 1s
Jul 13 23:50:11 preclear_disk_Z840ZW8W_1465: Zeroing: dd - wrote 8001563222016 of 8001563222016.
Jul 13 23:50:12 preclear_disk_Z840ZW8W_1465: Zeroing: 1
Jul 13 23:50:17 preclear_disk_Z840ZW8W_1465: Post-Read: dd if=/dev/sdh bs=512 count=4096 skip=1 conv=notrunc,noerror iflag=direct
Jul 13 23:50:17 preclear_disk_Z840ZW8W_1465: Post-Read: dd pid [8746]
Jul 13 23:50:17 preclear_disk_Z840ZW8W_1465: Post-Read: dd if=/dev/sdh bs=2097152 skip=1 conv=notrunc,noerror iflag=direct 2>/tmp/.preclear/sdh/dd_output | cmp - /dev/zero &>/tmp/.preclear/sdh/cmp_out 
Jul 13 23:50:17 preclear_disk_Z840ZW8W_1465: Post-Read: dd pid [8811]

 

Edited July 14, 2017 by Yousty

[dabl]

Maybe an obvious/not particularly useful observation but it seems there is something about the WD Reds that give the current preclear routine(s)/and or plugin fits.

 

I had the same kind of failures with one of those with an earlier version of the plugin (almost makes it to the end then stalls).

 

The last disk I tried, a Seagate Archive precleared successfully.

 

I wonder if running the one of the preclear routines standalone/independent of the plugin would yield different results.

 

Obviously it is time consuming and frustrating to test this kind of failure.

[gfjardim]

Guys, I'm out of home and unable to support you guys for another week or two.

Enviado de meu SM-G930F usando Tapatalk

[nAffie]

Great plugin! I've used it a lot now I moved my 7 year old 12bay tower to a 24 bay case and finally had some free slots available to preclear all my old drives that didn't fit in my old case anymore.

 

So I've noticed some little things.... 

 

All the e-mail updates specify the device id (sdX), except the e-mail that is sent when a step fails. See example below:

Event: Preclear
Subject: FAIL! Zeroing the disk failed.
Description: FAIL! Zeroing the disk failed.
Importance: alert

Zeroing the disk failed - Aborted

 

Another example:

Event: Preclear
Subject: FAIL! Post-Read verification failed.
Description: FAIL! Post-Read verification failed.
Importance: alert

Post-Read verification failed - Aborted

 

 

 

I would like to see which drive fails, especially when preclearing multiple drives at once.

 

Another issue I had is with the drive sizes.

The plugin displays a 1,5TB drive as a 2TB drive. See the attached
screenshot of my main page and the preclear page.

 

 

[opentoe]

Having trouble doing a pre-clear. I started one about an hour ago and the screen still looks like this? Any ideas? I selected the JOE.L drop down and left all the other options default.