[Plugin] Parity Check Tuning

[itimpi]

7 hours ago, gellux said:

hi, is this still available? CA finds nothing after various searches. nothing for "parity", and only returns the Mover Tuning when searching for "tuning". thanks.

As was mentioned you need to be on a 6.7 release for this to show up.    I think at one point it was incorrectly showing up in CA for earlier Unraid releases.  I added a check so that if you managed to install on an older Unraid release then users who navigate to the Settings->Scheduler tab will now find that although the fields for the plugin are shown the user cannot switch on the options with an error message displayed indicating they are not running a release of Unraid that allows this plugin to work as intended.

[gellux]

2 hours ago, itimpi said:

As was mentioned you need to be on a 6.7 release for this to show up.    I think at one point it was incorrectly showing up in CA for earlier Unraid releases.  I added a check so that if you managed to install on an older Unraid release then users who navigate to the Settings->Scheduler tab will now find that although the fields for the plugin are shown the user cannot switch on the options with an error message displayed indicating they are not running a release of Unraid that allows this plugin to work as intended.

oh i see now. i'm on 6.6.7 so as i skim read the requirements as i only built my system a couple weeks ago i assumed i was on the most up-to-date version! oh well, never mind!

[itimpi]

17 minutes ago, gellux said:

oh i see now. i'm on 6.6.7 so as i skim read the requirements as i only built my system a couple weeks ago i assumed i was on the most up-to-date version! oh well, never mind!

6.6.7 is the latest Stable release.    6.7.0 is still in rc stage (currently rc7) although it is expected to go Stable soon.   If you want to try 6.7.0 rc7 then it is available via the Bug Reports->Prereleases section of the forum.   The plugin needs a new feature that is coming as part of the 6.7.0 release for its operation.

[gellux]

1 hour ago, itimpi said:

6.6.7 is the latest Stable release.    6.7.0 is still in rc stage (currently rc7) although it is expected to go Stable soon.   If you want to try 6.7.0 rc7 then it is available via the Bug Reports->Prereleases section of the forum.   The plugin needs a new feature that is coming as part of the 6.7.0 release for its operation.

thanks for the info. i'll look into how to update unraid when i've had a bit more experience with it!

[itimpi]

I have released an update that implements the collection of history across a parity check that is run in increments.   The standard system parity check log is then updated to give the correct values for the duration and average speed (up to now the values displayed have only been for the last increment).    The plugin is also adding the total elapsed time and the number of increments to the history information.

 

At the moment these additional items are not being displayed by the standard built-in Unraid option to display parity history.   I have submitted a pull request on GitHub to see if displaying these extra fields can be implemented as a built-in feature of Unraid going forward.    If not I will release an update to the plugin that adds a History button on the settings page.

[wgstarks]

Will they be displayed anywhere with this update?

[itimpi]

13 minutes ago, wgstarks said:

Will they be displayed anywhere with this update?

The additional fields are not displayed with this update but the corrected duration and speed are displayed using the current history capability on the Main page. These additional fields ae being stored though for future use.   If you want to examine the fields you can do it manually by looking in the /boot/config/parity-checks.log file as it is only a text file  (once you have run a parity check to get them populated by the plugin).

 

If my requested change to the built-in method of displaying history from the Main page is accepted these additional fields will be displayed from the History option on the Main page at the next Unraid release.  I am hoping that it will be accepted as this would be the tidiest implementation and the suggested change does not cause problems if the Parity Check Tuning plugin is not installed.   As I said if the change is rejected then I will issue an update to the plugin to add a 'History' button on the Settings page.   I guess I could add it there anyway, but I would prefer not to as that just adds something else that has to be maintained and kept up-to-date.  It also seems sensible to only have one way of displaying history - not two different ones that display slightly different information.   I will wait on feedback from others to see what they think.   

 

I wanted to get this update out to start at least gathering this additional history information.   After all parity checks are not something that one runs frequently and tend to take a long time to complete   

Edited April 15, 2019 by itimpi

[wgstarks]

3 minutes ago, itimpi said:

guess I could add it there anyway, but I would prefer not to as that just adds something else that has to be maintained and kept up-to-date. 

And more nested features that are hard to find. Not the best solution.

[itimpi]

Looks like my suggested change for enhancing the parity history displayed when clicking on the History button on the Main tab is going to be rejected so I will have to add my own History Hutton on the Settings page.    I hope users will not get confused by the fact that the results shown by the Main page history button will show one set of Values (with some of them wrong) while my history button will show different (correct) ones for what is the same parity check.

[JonathanM]

19 minutes ago, itimpi said:

Looks like my suggested change is going to be rejected

Any reason given? Is there a chance of an amended request being accepted?

 

Without more info, this rejection seems extremely short sighted.

[itimpi]

The main reason is that if you do not have this plugin installed then you have some new columns (Elapsed Time and Increment Count) that would always Display an “Unknown” value as the fields would not be present in the history file.  That is a valid reason although I was hoping it might not be seen as that important as they really are Unknown values in standard Unraid.    I can work around my suggested change not being accepted by replacing the standard script for displaying history with my own customised version.   That means that a future Unraid update could make an incompatible change, but since the script has not changed much in recent history maybe that is low risk.   From an end-user perspective the two approaches are the same but I see having my own customised script as less elegant from a developer perspective.

 

seperately I have had some feedback suggesting that adding a History button to the Settings page might be a good idea even if it merely displays the same history as available via the History button on the Main page.   I can see some validity to that approach as you might well want to be able to easily examine history while playing around with settings relating to parity checking.

 

i will push out an update tomorrow implementing both those changes.   If later my proposed change to the standard script for displaying history gets accepted into Unraid I can always back out installing my own customised history script without affecting end-users.   

Edited April 16, 2019 by itimpi

[JonathanM]

Makes sense. Replacing the stock script is really no issue if version control is well implemented. Just make sure your script will only overwrite the stock script if there are no changes. Worst thing that will happen if they update the stock script is that your changes won't be made until you change your code to check for the new version.

 

The history on your settings page will always be your code.

[itimpi]

8 hours ago, jonathanm said:

Makes sense. Replacing the stock script is really no issue if version control is well implemented. Just make sure your script will only overwrite the stock script if there are no changes. Worst thing that will happen if they update the stock script is that your changes won't be made until you change your code to check for the new version.

 

The history on your settings page will always be your code.

Easier said than done to check that the stock version does not change.    Maybe I should checksum it before replacing it which would guard against such a change.   However I was assuming that if it did get changed I would quickly get feedback that my plugin needed updating   It is not as though Unraid would stop functioning during the short period it took me to roll out a fix if one is required.

 

I am hoping that eventually most (if not all) of the plugin functionality will get subsumed into standard Unraid at some future date so the plugin can then either be simplified or deprecated.   I made certain that all the copyright notices in the code gave explicit permission for Limetech to use it as they saw fit to potentially help with this happening

 

I have released an update to the plugin that does the replacement of the stock script.   Tell me what you think of the results - is it an improvement?   Also any feedback on the change?  As an example I have displayed "Unknown" when there is no explicit elapsed time information.  Would 'Unavailable' be better (or something else entirely)?   Note that one known issue outstanding is the summary text displayed alongside the History button on the Main page - this still only displays the information of the last increment (although if you press the button you get the correct history displayed).  Trying to correct for this is a much bigger change than I was happy with trying to make at this time and definitely a riskier change.   It may well become irrelevant anyway if the core Unraid system gets cleverer about tracking information across pause/resume.   There is also now a History button available on the Settings page that displays the same information - is this seen as a good idea or completely superfluous?

[JonathanM]

15 minutes ago, itimpi said:

Easier said than done to check that the stock version does not change.

Maybe

diff -s reference.script current.script

where reference.script is the current known version saved with your plugin?

[itimpi]

1 minute ago, jonathanm said:

Maybe

diff -s reference.script current.script

where reference.script is the current known version saved with your plugin?

That would not work with the way I currently handle it   What I do at the moment when the plugin is installed (either on first install or after a reboot when a fresh copy of Unraid is loaded into RAM) is rename the standard one *by adding .orig to the name) and then set up a symbolic link with the old name pointing to my customised version.  If you uninstall the plugin I reverse that procedure.   This avoids the need for me to currently save anything about the built-in version.   I need to think whether storing a checksum or a copy of the file is easier to handle (assuming I bother with a version check at all).

[bonienl]

9 minutes ago, itimpi said:

What I do at the moment when the plugin is installed (either on first install or after a reboot when a fresh copy of Unraid is loaded into RAM) is rename the standard one *by adding .orig to the name) and then set up a symbolic link with the old name pointing to my customised version

The proper way to replace a stock page with a customized page is to give the customized page the same name as the stock page and store this under your plugin folder.

When the GUI is loaded, it will automatically overwrite the stock page with the customized page.

[itimpi]

6 minutes ago, bonienl said:

The proper way to replace a stock page with a customized page is to give the customized page the same name as the stock page and store this under your plugin folder.

When the GUI is loaded, it will automatically overwrite the stock page with the customized page. 

I had not seen that mentioned previously - I must add it to my developer notes.  Which plugin folder - the one in RAM or the one on the USB stick?

 

Does this apply when it is not a .page file but is instead (as in this case) a script (from the dynamix include folder) that is being replaced?

Edited April 16, 2019 by itimpi

[bonienl]

2 minutes ago, itimpi said:

Does this apply when it is not a .page file but is instead (as in this case) a script (from the dynamix include folder) that is being replaced?

This only applies to page files, but the customized page can refer to a customized version of a script in the include folder of your plugin.

[itimpi]

30 minutes ago, bonienl said:

 This only applies to page files, but the customized page can refer to a customized version of a script in the include folder of your plugin.

Fair enough.  However in this case I want an existing built-in page (not customised) to also use the revised version of the script so my way may still be appropriate?  I did check that the existing script is only called from one place in the current GUI, and since the script simply populates a pop-up it will not affect normal operation.

Edited April 16, 2019 by itimpi

[Squid]

8 hours ago, bonienl said:

The proper way to replace a stock page with a customized page is to give the customized page the same name as the stock page and store this under your plugin folder.

When the GUI is loaded, it will automatically overwrite the stock page with the customized page.

Is this by design, or by accident? In what order are the .page files processed? Is /.../plugins/dynamix/*.page processed first and then everything else or is it done alphabetically by the folder name?

[bonienl]

1 hour ago, Squid said:

Is this by design, or by accident? In what order are the .page files processed? Is /.../plugins/dynamix/*.page processed first and then everything else or is it done alphabetically by the folder name?

By design ... of course

First the webGui pages are loaded and then the additional plugin pages. This allows any plugin to either add or replace functionality.

 

[cpshoemake]

If I select "No" to "Let scheduled mover run during a parity check / rebuild" in the Mover Tuning options, the mover will not run if there is a paused parity check in progress. Is this expected behavior?

[Squid]

58 minutes ago, cpshoemake said:

If I select "No" to "Let scheduled mover run during a parity check / rebuild" in the Mover Tuning options, the mover will not run if there is a paused parity check in progress. Is this expected behavior?

Mover tuning predates the ability to pause parity checks, and what you're seeing is probably how it works.  I'll look at it

[vw-kombi]

I Just completed my first one of these new multi step parity checks.  I was doing them weekly before.  

I have never had any errors in the parity checks in the past (50+ OF THEM).

The first one of these with the plugin states 'completed with 1127 sync errors corrected'......

The parity read check history for this states 0 errors however.

 

Any ideas ?

[itimpi]

On 7/11/2019 at 12:00 AM, vw-kombi said:

I Just completed my first one of these new multi step parity checks.  I was doing them weekly before.  

I have never had any errors in the parity checks in the past (50+ OF THEM).

The first one of these with the plugin states 'completed with 1127 sync errors corrected'......

The parity read check history for this states 0 errors however.

 

Any ideas ?

My suspicion is that the corrected count displayed is the true value and that there is a bug in the plugin with updating the history correctly with the results when errors are corrected during the run.

 

Can you look to see if you have the file 'config/plugins/parity.check.tuning/parity.check.tuning.progress.save' on your flash drive.   Sending me a copy of this file should allow me to see what happened at each phase of the last check and confirm what the true count should be.  Ideally I would also like a copy of the 'config/parity-checks.log' file off the flash drive so I can run a test against the actual history data from your system.  If you do not want to post these files publicly you san send them via forum PM.

 

In the meantime I will look at the plugin code to see if I can spot the problem.

Edited July 12, 2019 by itimpi