Preclear plugin


Recommended Posts

What I expect to collect:

 

1) number of disks that are precleared (dying to know  ;D);

2) any silent bugs that gets reported on the logs;

3) average size of disks that gets precleared, their model and the average speed and time we should expect to that model;

4) success rate;

5) rate of disks with SMART problems;

 

As far as I am concerned, #2 is the only one that I would consider is not an invasion of my privacy.  While I realize that the other information is nice to have, it is not essential to working of this product.  While LimeTech 'knows' how many time its software is download (which I assume you also know) and how many licenses it has issued, it has no real knowledge of how its product is being used, actual numbers of disks and with what plugins, Dockers, or VM's.  (There are a fair number of us with Pro licenses that aren't using anywhere near the numbers of disks to require it!)

 

I realize that I have exposed some of this data in my 'signature' line of this post.  However, the collection of that data would be very difficult and expensive process and even more difficult to tie directly to me as an individual.  (The upload of the data you propose to Google will have my IP address as a part of that data which would serve Google's purposes for THEIR data collection!  Read their terms of use...)

 

A custom made solution for data collection doesn't worth the effort, so I guess IP recording is inevitable (but every site/service track us nowadays even this forum). I agree you have all the right to choose if your data is uploaded or not, so I'm making this optional.

 

I must admit that if it is properly anomonized i do  cannot see how any of the suggested items invades priority.  However I guess if the decision as to whether such stats can be uploaded is an end-user choice everyone can make their own decision.

 

Is something like this enough?

 

UDxIdB7.png

Link to comment

Absolutely!  I assume that this popup choice will be presented upon the completion of each preclear process.

 

I was thinking about presenting it on first open of Preclear/Unassigned Devices, and let users change it on the Help Section of Preclear Plugin page.

 

Why do you think it should be done after each preclear session?

Link to comment

Absolutely!  I assume that this popup choice will be presented upon the completion of each preclear process.

not sure why the user needs prompting each time.    However it would probably be a good idea to display whether the statistics gathering is active whenever the user calls up the dialog to preclear a disk.    That should act as enough of a reminder to know whether you want to change the setting or not.    You could also make the displayed information a link to the place to change the setting thus making it simply accessible, or make it a toggle on that dialog that is preset to the current setting with the option to change it for this run.
Link to comment

Absolutely!  I assume that this popup choice will be presented upon the completion of each preclear process.

 

I was thinking about presenting it on first open of Preclear/Unassigned Devices, and let users change it on the Help Section of Preclear Plugin page.

 

Why do you think it should be done after each preclear session?

 

Why not?  That way if something did not seems right or there was a problem (or even a recently introduced size or disk model), the user could decide that in this case, he might want to upload the data.  If you are going to provide an check-box type option, you might provide an "Ask me each Time" as a third choice to hard 'Yes' or 'No' choice.  You are providing this software free to the user base and you do (in my opinion) have the right to be insistent in requesting an upload of the results. 

Link to comment

Absolutely!  I assume that this popup choice will be presented upon the completion of each preclear process.

 

I was thinking about presenting it on first open of Preclear/Unassigned Devices, and let users change it on the Help Section of Preclear Plugin page.

 

Why do you think it should be done after each preclear session?

 

Why not?  That way if something did not seems right or there was a problem (or even a recently introduced size or disk model), the user could decide that in this case, he might want to upload the data.  If you are going to provide an check-box type option, you might provide an "Ask me each Time" as a third choice to hard 'Yes' or 'No' choice.  You are providing this software free to the user base and you do (in my opinion) have the right to be insistent in requesting an upload of the results.

Absolutely. I accept as a user of your software that you ask certain things in return to help you. I just ask that you be open and transparent when you make those requests.

 

I would be perfectly fine with a mandatory nag at the end of each preclear that asks "Will you help me improve this software by providing me with the statistical results and / or errors of this preclear?"

 

What I will not stand for is the silent transmission of my information with no explicit permission.

Link to comment

Absolutely!  I assume that this popup choice will be presented upon the completion of each preclear process.

 

I was thinking about presenting it on first open of Preclear/Unassigned Devices, and let users change it on the Help Section of Preclear Plugin page.

 

Why do you think it should be done after each preclear session?

 

Why not?  That way if something did not seems right or there was a problem (or even a recently introduced size or disk model), the user could decide that in this case, he might want to upload the data.  If you are going to provide an check-box type option, you might provide an "Ask me each Time" as a third choice to hard 'Yes' or 'No' choice.  You are providing this software free to the user base and you do (in my opinion) have the right to be insistent in requesting an upload of the results.

Absolutely. I accept as a user of your software that you ask certain things in return to help you. I just ask that you be open and transparent when you make those requests.

 

I would be perfectly fine with a mandatory nag at the end of each preclear that asks "Will you help me improve this software by providing me with the statistical results and / or errors of this preclear?"

 

What I will not stand for is the silent transmission of my information with no explicit permission.

 

Will this work?

 

LWPGOEP.png

 

How did I implemented it?

 

1) The script will write a ".stats" file on /boot/config/plugins/preclear.disk directory after it's done;

 

2) The webui will present that popup for each ".stats" file it found on /boot/config/plugins/preclear.disk directory;

 

3) ".stats" file will be deleted after sent, of after "Cancel" is clicked.

Link to comment

Will this work?

 

LWPGOEP.png

I'd like to see google mentioned, since they are the entity that is actually receiving the data on your behalf. That way if I'm watching the bits flow through my router, I'll know with reasonable certainty that a connection from unraid to google is a result of the dialog I just clicked.

 

The "anonymous" wording may be the end result to YOU, but it certainly isn't anonymous to google.

Link to comment

Will this work?

 

LWPGOEP.png

I'd like to see google mentioned, since they are the entity that is actually receiving the data on your behalf. That way if I'm watching the bits flow through my router, I'll know with reasonable certainty that a connection from unraid to google is a result of the dialog I just clicked.

 

The "anonymous" wording may be the end result to YOU, but it certainly isn't anonymous to google.

 

Any way to anonymize the IP address?

Link to comment

I'd like to see google mentioned, since they are the entity that is actually receiving the data on your behalf. That way if I'm watching the bits flow through my router, I'll know with reasonable certainty that a connection from unraid to google is a result of the dialog I just clicked.

 

The "anonymous" wording may be the end result to YOU, but it certainly isn't anonymous to google.

 

Any way to anonymize the IP address?

Not that I'm aware of if you continue to use google. One reason google is "free" is because they collect every bit of data they can on every user and computer they can. Cross referencing all the little tidbits allows them to sell user profiles as a "product" to advertisers. They can't do that if they can't correlate info somehow.

 

As long as you disclose that google is the one receiving and holding the form data, I see no issue with it. The data really is anonymous to you, and anyone else accessing the forms from outside. It's only google's internal workings that do the correlation and data mining.

Link to comment

I'd like to see google mentioned, since they are the entity that is actually receiving the data on your behalf. That way if I'm watching the bits flow through my router, I'll know with reasonable certainty that a connection from unraid to google is a result of the dialog I just clicked.

 

The "anonymous" wording may be the end result to YOU, but it certainly isn't anonymous to google.

 

Any way to anonymize the IP address?

Not that I'm aware of if you continue to use google. One reason google is "free" is because they collect every bit of data they can on every user and computer they can. Cross referencing all the little tidbits allows them to sell user profiles as a "product" to advertisers. They can't do that if they can't correlate info somehow.

 

As long as you disclose that google is the one receiving and holding the form data, I see no issue with it. The data really is anonymous to you, and anyone else accessing the forms from outside. It's only google's internal workings that do the correlation and data mining.

 

Maybe TOR?

Link to comment

I'd like to see google mentioned, since they are the entity that is actually receiving the data on your behalf. That way if I'm watching the bits flow through my router, I'll know with reasonable certainty that a connection from unraid to google is a result of the dialog I just clicked.

 

The "anonymous" wording may be the end result to YOU, but it certainly isn't anonymous to google.

 

Any way to anonymize the IP address?

Not that I'm aware of if you continue to use google. One reason google is "free" is because they collect every bit of data they can on every user and computer they can. Cross referencing all the little tidbits allows them to sell user profiles as a "product" to advertisers. They can't do that if they can't correlate info somehow.

 

As long as you disclose that google is the one receiving and holding the form data, I see no issue with it. The data really is anonymous to you, and anyone else accessing the forms from outside. It's only google's internal workings that do the correlation and data mining.

 

Maybe TOR?

 

I'm able to make use of TOR to anonymize the IP address. Is that ok for you, guys?

Link to comment

It's why I cringe every time I see someone specify to just use 8.8.8.8 or 8.8.4.4 for DNS.

 

OFT!!!  I have done this because, unfortunately, it is one pair of DNS servers that are available almost everywhere.  Do you know of another set of DNS servers which is universally available with all of the baggage?

No clue how honest these people are, but at least they claim to do no evil with your data. https://freedns.zone/en/
Link to comment
I'm able to make use of TOR to anonymize the IP address. Is that ok for you, guys?
Could you change the google form type to log the IP address, so that we can see that what is logged is not our own IP?

 

On that same note, are you planning to allow us to peruse the data that you collect on the google form? If you say no, I understand, and don't necessarily object, but it would be nice if you would add a link to the form data on the submission confirmation after you send data.

Link to comment

I'm able to make use of TOR to anonymize the IP address. Is that ok for you, guys?
Could you change the google form type to log the IP address, so that we can see that what is logged is not our own IP?

 

On that same note, are you planning to allow us to peruse the data that you collect on the google form? If you say no, I understand, and don't necessarily object, but it would be nice if you would add a link to the form data on the submission confirmation after you send data.

 

I don't think it's necessary, the plugin does the comparison by itself:

 

  case 'send_statistics':
    $file = $_POST["file"];
    $stat = parse_ini_file($file,true) ?: [];

    shell_exec("/etc/rc.d/rc.tor start 2>&1");

    $myip = trim(shell_exec("curl -s http://whatismyip.akamai.com 2>/dev/null"));
    $torip = trim(shell_exec("curl -s --socks5-hostname 127.0.0.1:9050 http://whatismyip.akamai.com 2>/dev/null"));

    if ($myip == $torip)
    {
      exit();
    }

    shell_exec("echo sending preclear report:|logger -t'preclear_disk'");
    shell_exec("echo my ip = $myip|logger -t'preclear_disk'");
    shell_exec("echo tor ip = $torip|logger -t'preclear_disk'");
    $cmd = "curl -s --socks5-hostname 127.0.0.1:9050 \"https://docs.google.com/forms/d/e/<id>/formResponse\" -d ifq";
    foreach ($stat as $key => $value)
    {
      $cmd .= " -d \"{$value['entry']}=".str_replace("^n", "\n", $value['value'])."\"";
    }
    shell_exec("$cmd 2>&1");
    shell_exec("/etc/rc.d/rc.tor stop 2>&1");
    @unlink($file);
    break;

 

I'll put the form URL and the respective spreadsheet in the main OP, so anyone can use it.

Link to comment

When I preclear, its always "Starting..."

This is the contents of the notification window.

/usr/local/emhttp/plugins/preclear.disk/script/preclear_disk.sh  --notify 7 --frequency 1 --cycles 3 --no-prompt /dev/sdf

 

Tried with ie edge, ie 11, firefox and chrome. None works.

 

Any ideas?

 

Please update the script and try it again.

Link to comment

Guys, the statistics report system has been put in place. The great think about scripts is that they allow anyone to see what happens in the background. I'll comment some technical details to make this as transparent as possible.

 

When finishes or fails in some of the stages the preclear script will write a ".stats" file under /boot/config/plugins/preclear.disk/ directory. This file is a multi-level INI file and each of it's sections contains a information that can get exported to Google Forms. Each section has an entry variable which corresponds to the desired field in "Google Forms". It also has a title variable which corresponds the webui title and a value variable which contains the actual data. The relevant function in script can be found here.

 

When Unassigned Devices or Preclear Disk plugins gets loaded by the web interface, the getOpenStatistics() javascript function will probe for any existing ".stats" files on the plugin config directory (here). If a report is found, it will trigger an alert asking the user to Send or Cancel that report.

 

If you hit Send, the getOpenStatistics() function will send a POST request with "action=send_statistics" form-data to "Preclear.php". That request will be processed by this portion of the code; the script will start the TOR service, compare both local external IP and the TOR remote IP (and exit if they match), create the curl request command and run it, stop the TOR service and remove the ".stats" file at the end.

 

If you hit Cancel, the getOpenStatistics() function will send a POST request with "action= remove_statistics" form-data to "Preclear.php", and the script will remove the ".stats" file.

 

I hope this will remove any doubts about this functionality, and I do expect for your statistics!  ;)

Link to comment

I was just composing this when you posted your announcement about the new version.

 

Got this email from the daily cron after the auto-update to the new preclear version. I wasn't even using the plugin at the time and in fact, haven't used it in months.

 

I assume your plugin is to blame due to the mention of "tor" and it is the only thing that changed overnight.

 

error: Ignoring tor because of bad file mode - must be 0644 or 0444.

error: skipping "/var/log/apcupsd.events" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.

error: skipping "/var/log/docker.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.

error: skipping "/var/log/syslog" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.

error: skipping "/var/log/vsftpd.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.

error: skipping "/var/log/wtmp" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.

error: skipping "/var/log/btmp" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.

Link to comment

Hey guys,

I had to remove and re-install the plugin for some unknown reason it wasnt detecting the update being available.

 

anyways, now im seeing the below in the "/var/log/preclear.disk.log"

 

Tue Dec 20 23:32:37 AEDT 2016: PHP Warning: mkdir(): Invalid path in /usr/local/emhttp/plugins/preclear.disk/Preclear.php on line 30
Tue Dec 20 23:32:47 AEDT 2016: PHP Warning: mkdir(): Invalid path in /usr/local/emhttp/plugins/preclear.disk/Preclear.php on line 30
Tue Dec 20 23:32:57 AEDT 2016: PHP Warning: mkdir(): Invalid path in /usr/local/emhttp/plugins/preclear.disk/Preclear.php on line 30
Tue Dec 20 23:33:07 AEDT 2016: PHP Warning: mkdir(): Invalid path in /usr/local/emhttp/plugins/preclear.disk/Preclear.php on line 30
Tue Dec 20 23:33:17 AEDT 2016: PHP Warning: mkdir(): Invalid path in /usr/local/emhttp/plugins/preclear.disk/Preclear.php on line 30
Tue Dec 20 23:33:27 AEDT 2016: PHP Warning: mkdir(): Invalid path in /usr/local/emhttp/plugins/preclear.disk/Preclear.php on line 30
Tue Dec 20 23:33:37 AEDT 2016: PHP Warning: mkdir(): Invalid path in /usr/local/emhttp/plugins/preclear.disk/Preclear.php on line 30
Tue Dec 20 23:33:47 AEDT 2016: PHP Warning: mkdir(): Invalid path in /usr/local/emhttp/plugins/preclear.disk/Preclear.php on line 30
Tue Dec 20 23:33:57 AEDT 2016: PHP Warning: mkdir(): Invalid path in /usr/local/emhttp/plugins/preclear.disk/Preclear.php on line 30
Tue Dec 20 23:34:08 AEDT 2016: PHP Warning: mkdir(): Invalid path in /usr/local/emhttp/plugins/preclear.disk/Preclear.php on line 30
Tue Dec 20 23:34:18 AEDT 2016: PHP Warning: mkdir(): Invalid path in /usr/local/emhttp/plugins/preclear.disk/Preclear.php on line 30
Tue Dec 20 23:34:28 AEDT 2016: PHP Warning: mkdir(): Invalid path in /usr/local/emhttp/plugins/preclear.disk/Preclear.php on line 30
Tue Dec 20 23:34:38 AEDT 2016: PHP Warning: mkdir(): Invalid path in /usr/local/emhttp/plugins/preclear.disk/Preclear.php on line 30
Tue Dec 20 23:34:48 AEDT 2016: PHP Warning: mkdir(): Invalid path in /usr/local/emhttp/plugins/preclear.disk/Preclear.php on line 30
Tue Dec 20 23:34:58 AEDT 2016: PHP Warning: mkdir(): Invalid path in /usr/local/emhttp/plugins/preclear.disk/Preclear.php on line 30
Tue Dec 20 23:35:08 AEDT 2016: PHP Warning: mkdir(): Invalid path in /usr/local/emhttp/plugins/preclear.disk/Preclear.php on line 30
Tue Dec 20 23:35:18 AEDT 2016: PHP Warning: mkdir(): Invalid path in /usr/local/emhttp/plugins/preclear.disk/Preclear.php on line 30
Tue Dec 20 23:35:28 AEDT 2016: PHP Warning: mkdir(): Invalid path in /usr/local/emhttp/plugins/preclear.disk/Preclear.php on line 30
Tue Dec 20 23:35:38 AEDT 2016: PHP Warning: mkdir(): Invalid path in /usr/local/emhttp/plugins/preclear.disk/Preclear.php on line 30
Tue Dec 20 23:35:48 AEDT 2016: PHP Warning: mkdir(): Invalid path in /usr/local/emhttp/plugins/preclear.disk/Preclear.php on line 30
Tue Dec 20 23:35:58 AEDT 2016: PHP Warning: mkdir(): Invalid path in /usr/local/emhttp/plugins/preclear.disk/Preclear.php on line 30
Tue Dec 20 23:36:08 AEDT 2016: PHP Warning: mkdir(): Invalid path in /usr/local/emhttp/plugins/preclear.disk/Preclear.php on line 30
Tue Dec 20 23:36:18 AEDT 2016: PHP Warning: mkdir(): Invalid path in /usr/local/emhttp/plugins/preclear.disk/Preclear.php on line 30
Tue Dec 20 23:36:28 AEDT 2016: PHP Warning: mkdir(): Invalid path in /usr/local/emhttp/plugins/preclear.disk/Preclear.php on line 30
Tue Dec 20 23:36:38 AEDT 2016: PHP Warning: mkdir(): Invalid path in /usr/local/emhttp/plugins/preclear.disk/Preclear.php on line 30

 

Thoughts?

Link to comment
  • Squid featured, unfeatured and pinned this topic
  • Squid unpinned this topic

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.