Jump to content
gfjardim

Preclear plugin

2429 posts in this topic Last Reply

Recommended Posts

Will this work?

 

LWPGOEP.png

I'd like to see google mentioned, since they are the entity that is actually receiving the data on your behalf. That way if I'm watching the bits flow through my router, I'll know with reasonable certainty that a connection from unraid to google is a result of the dialog I just clicked.

 

The "anonymous" wording may be the end result to YOU, but it certainly isn't anonymous to google.

Share this post


Link to post

Will this work?

 

LWPGOEP.png

I'd like to see google mentioned, since they are the entity that is actually receiving the data on your behalf. That way if I'm watching the bits flow through my router, I'll know with reasonable certainty that a connection from unraid to google is a result of the dialog I just clicked.

 

The "anonymous" wording may be the end result to YOU, but it certainly isn't anonymous to google.

 

Any way to anonymize the IP address?

Share this post


Link to post

I'd like to see google mentioned, since they are the entity that is actually receiving the data on your behalf. That way if I'm watching the bits flow through my router, I'll know with reasonable certainty that a connection from unraid to google is a result of the dialog I just clicked.

 

The "anonymous" wording may be the end result to YOU, but it certainly isn't anonymous to google.

 

Any way to anonymize the IP address?

Not that I'm aware of if you continue to use google. One reason google is "free" is because they collect every bit of data they can on every user and computer they can. Cross referencing all the little tidbits allows them to sell user profiles as a "product" to advertisers. They can't do that if they can't correlate info somehow.

 

As long as you disclose that google is the one receiving and holding the form data, I see no issue with it. The data really is anonymous to you, and anyone else accessing the forms from outside. It's only google's internal workings that do the correlation and data mining.

Share this post


Link to post

I'd like to see google mentioned, since they are the entity that is actually receiving the data on your behalf. That way if I'm watching the bits flow through my router, I'll know with reasonable certainty that a connection from unraid to google is a result of the dialog I just clicked.

 

The "anonymous" wording may be the end result to YOU, but it certainly isn't anonymous to google.

 

Any way to anonymize the IP address?

Not that I'm aware of if you continue to use google. One reason google is "free" is because they collect every bit of data they can on every user and computer they can. Cross referencing all the little tidbits allows them to sell user profiles as a "product" to advertisers. They can't do that if they can't correlate info somehow.

 

As long as you disclose that google is the one receiving and holding the form data, I see no issue with it. The data really is anonymous to you, and anyone else accessing the forms from outside. It's only google's internal workings that do the correlation and data mining.

 

Maybe TOR?

Share this post


Link to post

Not that I'm aware of if you continue to use google. One reason google is "free" is because they collect every bit of data they can on every user and computer they can. Cross referencing all the little tidbits allows them to sell user profiles as a "product" to advertisers. They can't do that if they can't correlate info somehow.

 

As long as you disclose that google is the one receiving and holding the form data, I see no issue with it. The data really is anonymous to you, and anyone else accessing the forms from outside. It's only google's internal workings that do the correlation and data mining.

 

Not pertinent to PreClear, but absolutely relevant to Google.

 

This is exactly why I refuse to use Google DNS services (and why my Docker DNS request to allow for overriding Google DNS settings). There is absolutely no valid reason to give Google the insight into all the network traffic, applications and usage patterns one uses. It's why I cringe every time I see someone specify to just use 8.8.8.8 or 8.8.4.4 for DNS.

Share this post


Link to post

It's why I cringe every time I see someone specify to just use 8.8.8.8 or 8.8.4.4 for DNS.

 

OFT!!!  I have done this because, unfortunately, it is one pair of DNS servers that are available almost everywhere.  Do you know of another set of DNS servers which is universally available with all of the baggage?

Share this post


Link to post

I'd like to see google mentioned, since they are the entity that is actually receiving the data on your behalf. That way if I'm watching the bits flow through my router, I'll know with reasonable certainty that a connection from unraid to google is a result of the dialog I just clicked.

 

The "anonymous" wording may be the end result to YOU, but it certainly isn't anonymous to google.

 

Any way to anonymize the IP address?

Not that I'm aware of if you continue to use google. One reason google is "free" is because they collect every bit of data they can on every user and computer they can. Cross referencing all the little tidbits allows them to sell user profiles as a "product" to advertisers. They can't do that if they can't correlate info somehow.

 

As long as you disclose that google is the one receiving and holding the form data, I see no issue with it. The data really is anonymous to you, and anyone else accessing the forms from outside. It's only google's internal workings that do the correlation and data mining.

 

Maybe TOR?

 

I'm able to make use of TOR to anonymize the IP address. Is that ok for you, guys?

Share this post


Link to post

It's why I cringe every time I see someone specify to just use 8.8.8.8 or 8.8.4.4 for DNS.

 

OFT!!!  I have done this because, unfortunately, it is one pair of DNS servers that are available almost everywhere.  Do you know of another set of DNS servers which is universally available with all of the baggage?

No clue how honest these people are, but at least they claim to do no evil with your data. https://freedns.zone/en/

Share this post


Link to post
I'm able to make use of TOR to anonymize the IP address. Is that ok for you, guys?
Could you change the google form type to log the IP address, so that we can see that what is logged is not our own IP?

 

On that same note, are you planning to allow us to peruse the data that you collect on the google form? If you say no, I understand, and don't necessarily object, but it would be nice if you would add a link to the form data on the submission confirmation after you send data.

Share this post


Link to post

I'm able to make use of TOR to anonymize the IP address. Is that ok for you, guys?
Could you change the google form type to log the IP address, so that we can see that what is logged is not our own IP?

 

On that same note, are you planning to allow us to peruse the data that you collect on the google form? If you say no, I understand, and don't necessarily object, but it would be nice if you would add a link to the form data on the submission confirmation after you send data.

 

I don't think it's necessary, the plugin does the comparison by itself:

 

  case 'send_statistics':
    $file = $_POST["file"];
    $stat = parse_ini_file($file,true) ?: [];

    shell_exec("/etc/rc.d/rc.tor start 2>&1");

    $myip = trim(shell_exec("curl -s http://whatismyip.akamai.com 2>/dev/null"));
    $torip = trim(shell_exec("curl -s --socks5-hostname 127.0.0.1:9050 http://whatismyip.akamai.com 2>/dev/null"));

    if ($myip == $torip)
    {
      exit();
    }

    shell_exec("echo sending preclear report:|logger -t'preclear_disk'");
    shell_exec("echo my ip = $myip|logger -t'preclear_disk'");
    shell_exec("echo tor ip = $torip|logger -t'preclear_disk'");
    $cmd = "curl -s --socks5-hostname 127.0.0.1:9050 \"https://docs.google.com/forms/d/e/<id>/formResponse\" -d ifq";
    foreach ($stat as $key => $value)
    {
      $cmd .= " -d \"{$value['entry']}=".str_replace("^n", "\n", $value['value'])."\"";
    }
    shell_exec("$cmd 2>&1");
    shell_exec("/etc/rc.d/rc.tor stop 2>&1");
    @unlink($file);
    break;

 

I'll put the form URL and the respective spreadsheet in the main OP, so anyone can use it.

Share this post


Link to post

When I preclear, its always "Starting..."

This is the contents of the notification window.

/usr/local/emhttp/plugins/preclear.disk/script/preclear_disk.sh  --notify 7 --frequency 1 --cycles 3 --no-prompt /dev/sdf

 

Tried with ie edge, ie 11, firefox and chrome. None works.

 

Any ideas?

Share this post


Link to post

When I preclear, its always "Starting..."

This is the contents of the notification window.

/usr/local/emhttp/plugins/preclear.disk/script/preclear_disk.sh  --notify 7 --frequency 1 --cycles 3 --no-prompt /dev/sdf

 

Tried with ie edge, ie 11, firefox and chrome. None works.

 

Any ideas?

 

Please update the script and try it again.

Share this post


Link to post

Guys, the statistics report system has been put in place. The great think about scripts is that they allow anyone to see what happens in the background. I'll comment some technical details to make this as transparent as possible.

 

When finishes or fails in some of the stages the preclear script will write a ".stats" file under /boot/config/plugins/preclear.disk/ directory. This file is a multi-level INI file and each of it's sections contains a information that can get exported to Google Forms. Each section has an entry variable which corresponds to the desired field in "Google Forms". It also has a title variable which corresponds the webui title and a value variable which contains the actual data. The relevant function in script can be found here.

 

When Unassigned Devices or Preclear Disk plugins gets loaded by the web interface, the getOpenStatistics() javascript function will probe for any existing ".stats" files on the plugin config directory (here). If a report is found, it will trigger an alert asking the user to Send or Cancel that report.

 

If you hit Send, the getOpenStatistics() function will send a POST request with "action=send_statistics" form-data to "Preclear.php". That request will be processed by this portion of the code; the script will start the TOR service, compare both local external IP and the TOR remote IP (and exit if they match), create the curl request command and run it, stop the TOR service and remove the ".stats" file at the end.

 

If you hit Cancel, the getOpenStatistics() function will send a POST request with "action= remove_statistics" form-data to "Preclear.php", and the script will remove the ".stats" file.

 

I hope this will remove any doubts about this functionality, and I do expect for your statistics!  ;)

Share this post


Link to post

I was just composing this when you posted your announcement about the new version.

 

Got this email from the daily cron after the auto-update to the new preclear version. I wasn't even using the plugin at the time and in fact, haven't used it in months.

 

I assume your plugin is to blame due to the mention of "tor" and it is the only thing that changed overnight.

 

error: Ignoring tor because of bad file mode - must be 0644 or 0444.

error: skipping "/var/log/apcupsd.events" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.

error: skipping "/var/log/docker.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.

error: skipping "/var/log/syslog" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.

error: skipping "/var/log/vsftpd.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.

error: skipping "/var/log/wtmp" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.

error: skipping "/var/log/btmp" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.

Share this post


Link to post

I have removed the plugin and rebooted. We'll see if it happens again tonight.

 

This is tor f#ckin up the logrotate directory. I'll change the build script ant update the package. Thanks for reporting.

Share this post


Link to post

Hey guys,

I had to remove and re-install the plugin for some unknown reason it wasnt detecting the update being available.

 

anyways, now im seeing the below in the "/var/log/preclear.disk.log"

 

Tue Dec 20 23:32:37 AEDT 2016: PHP Warning: mkdir(): Invalid path in /usr/local/emhttp/plugins/preclear.disk/Preclear.php on line 30
Tue Dec 20 23:32:47 AEDT 2016: PHP Warning: mkdir(): Invalid path in /usr/local/emhttp/plugins/preclear.disk/Preclear.php on line 30
Tue Dec 20 23:32:57 AEDT 2016: PHP Warning: mkdir(): Invalid path in /usr/local/emhttp/plugins/preclear.disk/Preclear.php on line 30
Tue Dec 20 23:33:07 AEDT 2016: PHP Warning: mkdir(): Invalid path in /usr/local/emhttp/plugins/preclear.disk/Preclear.php on line 30
Tue Dec 20 23:33:17 AEDT 2016: PHP Warning: mkdir(): Invalid path in /usr/local/emhttp/plugins/preclear.disk/Preclear.php on line 30
Tue Dec 20 23:33:27 AEDT 2016: PHP Warning: mkdir(): Invalid path in /usr/local/emhttp/plugins/preclear.disk/Preclear.php on line 30
Tue Dec 20 23:33:37 AEDT 2016: PHP Warning: mkdir(): Invalid path in /usr/local/emhttp/plugins/preclear.disk/Preclear.php on line 30
Tue Dec 20 23:33:47 AEDT 2016: PHP Warning: mkdir(): Invalid path in /usr/local/emhttp/plugins/preclear.disk/Preclear.php on line 30
Tue Dec 20 23:33:57 AEDT 2016: PHP Warning: mkdir(): Invalid path in /usr/local/emhttp/plugins/preclear.disk/Preclear.php on line 30
Tue Dec 20 23:34:08 AEDT 2016: PHP Warning: mkdir(): Invalid path in /usr/local/emhttp/plugins/preclear.disk/Preclear.php on line 30
Tue Dec 20 23:34:18 AEDT 2016: PHP Warning: mkdir(): Invalid path in /usr/local/emhttp/plugins/preclear.disk/Preclear.php on line 30
Tue Dec 20 23:34:28 AEDT 2016: PHP Warning: mkdir(): Invalid path in /usr/local/emhttp/plugins/preclear.disk/Preclear.php on line 30
Tue Dec 20 23:34:38 AEDT 2016: PHP Warning: mkdir(): Invalid path in /usr/local/emhttp/plugins/preclear.disk/Preclear.php on line 30
Tue Dec 20 23:34:48 AEDT 2016: PHP Warning: mkdir(): Invalid path in /usr/local/emhttp/plugins/preclear.disk/Preclear.php on line 30
Tue Dec 20 23:34:58 AEDT 2016: PHP Warning: mkdir(): Invalid path in /usr/local/emhttp/plugins/preclear.disk/Preclear.php on line 30
Tue Dec 20 23:35:08 AEDT 2016: PHP Warning: mkdir(): Invalid path in /usr/local/emhttp/plugins/preclear.disk/Preclear.php on line 30
Tue Dec 20 23:35:18 AEDT 2016: PHP Warning: mkdir(): Invalid path in /usr/local/emhttp/plugins/preclear.disk/Preclear.php on line 30
Tue Dec 20 23:35:28 AEDT 2016: PHP Warning: mkdir(): Invalid path in /usr/local/emhttp/plugins/preclear.disk/Preclear.php on line 30
Tue Dec 20 23:35:38 AEDT 2016: PHP Warning: mkdir(): Invalid path in /usr/local/emhttp/plugins/preclear.disk/Preclear.php on line 30
Tue Dec 20 23:35:48 AEDT 2016: PHP Warning: mkdir(): Invalid path in /usr/local/emhttp/plugins/preclear.disk/Preclear.php on line 30
Tue Dec 20 23:35:58 AEDT 2016: PHP Warning: mkdir(): Invalid path in /usr/local/emhttp/plugins/preclear.disk/Preclear.php on line 30
Tue Dec 20 23:36:08 AEDT 2016: PHP Warning: mkdir(): Invalid path in /usr/local/emhttp/plugins/preclear.disk/Preclear.php on line 30
Tue Dec 20 23:36:18 AEDT 2016: PHP Warning: mkdir(): Invalid path in /usr/local/emhttp/plugins/preclear.disk/Preclear.php on line 30
Tue Dec 20 23:36:28 AEDT 2016: PHP Warning: mkdir(): Invalid path in /usr/local/emhttp/plugins/preclear.disk/Preclear.php on line 30
Tue Dec 20 23:36:38 AEDT 2016: PHP Warning: mkdir(): Invalid path in /usr/local/emhttp/plugins/preclear.disk/Preclear.php on line 30

 

Thoughts?

Share this post


Link to post

A bug report of sorts to raise - I just popped a new disk in (fresh out of a virgin anti static bag) and went to use the plugin.

 

I have previously used it to pre clear another disk, which (perhaps unrelated) was the same make/model/capacity.

 

The disk shows as pre clear having already completed, so there is no option to start it. Similarly in the main unraid dashboard it shows in Unassigned Devices as "Preclear in progress... Preclear Finished Successfully!"

 

I suspect this is because I have a permanently empty bay which I use for preclearing, then when its complete I move the drive to its 'correct' location. This means (again, speculative) that the new drive has come up with the same sdX as the previous successful pre clear, and the plugin thinks its already been dealt with.

 

If I run the old-style preclear_disk.sh script with -l, it does indeed identify this new one as being in need of a pre clear, and this serves as my stopgap method using the old screen approach.

 

Im not sure if a reboot would fix this, and its not a show stopper by any means, but I thought it couldn't hurt to raise it. Thank you for your ongoing efforts, gfjardim :)

Share this post


Link to post

A bug report of sorts to raise - I just popped a new disk in (fresh out of a virgin anti static bag) and went to use the plugin.

 

I have previously used it to pre clear another disk, which (perhaps unrelated) was the same make/model/capacity.

 

The disk shows as pre clear having already completed, so there is no option to start it. Similarly in the main unraid dashboard it shows in Unassigned Devices as "Preclear in progress... Preclear Finished Successfully!"

 

I suspect this is because I have a permanently empty bay which I use for preclearing, then when its complete I move the drive to its 'correct' location. This means (again, speculative) that the new drive has come up with the same sdX as the previous successful pre clear, and the plugin thinks its already been dealt with.

 

If I run the old-style preclear_disk.sh script with -l, it does indeed identify this new one as being in need of a pre clear, and this serves as my stopgap method using the old screen approach.

 

Im not sure if a reboot would fix this, and its not a show stopper by any means, but I thought it couldn't hurt to raise it. Thank you for your ongoing efforts, gfjardim :)

 

This is because you removed the old drive and the Linux UDEV subsystem reused it's device name. You can easily remove the old instance by clicking the X icon next to the new drive name.

 

Share this post


Link to post

I suspect this is because I have a permanently empty bay which I use for preclearing, then when its complete I move the drive to its 'correct' location.

 

Do you mean that you physically move the drive without shutting down the server?

 

Share this post


Link to post

 

Got this email from the daily cron after the auto-update to the new preclear version. I wasn't even using the plugin at the time and in fact, haven't used it in months.

 

I assume your plugin is to blame due to the mention of "tor" and it is the only thing that changed overnight.

 

error: Ignoring tor because of bad file mode - must be 0644 or 0444.

error: skipping "/var/log/apcupsd.events" because parent directory has insecure permissions (It's world writable or writable by group [...]

 

Can data collecting be an opt-in feature PLEASE?  both of my home servers and my work server autoupdated the plugin last night.  The plugin was not in use on any machine.  If this cannot be an opt-in feature I'll sadly have to go back to a v5 build with the JoeL script just for clearing disks at work as we do 3 preclears on all new disks... and 3 preclears on all decommissioned disks and data cannot flow out of the company.

 

While receiving this email was mildly irksome on the 2 home machines... the use of TOR, which is STRICTLY prohibited on any work network, set off a bunch of red flags at work as my business revolves around government/bank/broker/customer confidential information... and I was called in EARLY to investigate who was attempting to breach NDAs to leak restricted information and have been in meetings with the parent company and our customers over what happened for the last 3 hours.   

 

I have been very happy with your contributions/work... this update was less than pleasing.

Share this post


Link to post

I suspect this is because I have a permanently empty bay which I use for preclearing, then when its complete I move the drive to its 'correct' location.

 

Do you mean that you physically move the drive without shutting down the server?

 

I stop the array, pull the drive, relocate the drive, restart the array.

 

This is because you removed the old drive and the Linux UDEV subsystem reused it's device name. You can easily remove the old instance by clicking the X icon next to the new drive name.

 

So when I hover over the red X, the tooltip says "Stop Preclear" (obviously there isn't one ongoing) and clicking it refreshes the page, but doesn't actually change anything that I can see - I've still no option to start a preclear. Pretty sure by now that Im missing something obvious, but not sure what it is!

Share this post


Link to post

I suspect this is because I have a permanently empty bay which I use for preclearing, then when its complete I move the drive to its 'correct' location.

 

Do you mean that you physically move the drive without shutting down the server?

 

I stop the array, pull the drive, relocate the drive, restart the array.

 

This is because you removed the old drive and the Linux UDEV subsystem reused it's device name. You can easily remove the old instance by clicking the X icon next to the new drive name.

 

So when I hover over the red X, the tooltip says "Stop Preclear" (obviously there isn't one ongoing) and clicking it refreshes the page, but doesn't actually change anything that I can see - I've still no option to start a preclear. Pretty sure by now that Im missing something obvious, but not sure what it is!

 

Is it mounted by Unassigned Devices?

Share this post


Link to post

While receiving this email was mildly irksome on the 2 home machines... the use of TOR, which is STRICTLY prohibited on any work network, set off a bunch of red flags at work as my business revolves around government/bank/broker/customer confidential information... and I was called in EARLY to investigate who was attempting to breach NDAs to leak restricted information and have been in meetings with the parent company and our customers over what happened for the last 3 hours.   

How on earth did an unsecure machine get outbound internet access in a secure facility? Was the firewall breached somehow? Unraid should never have been allowed on the network in the first place, it's pretty much a security breach waiting to happen.

Share this post


Link to post

I suspect this is because I have a permanently empty bay which I use for preclearing, then when its complete I move the drive to its 'correct' location.

 

Do you mean that you physically move the drive without shutting down the server?

 

I stop the array, pull the drive, relocate the drive, restart the array.

 

This is because you removed the old drive and the Linux UDEV subsystem reused it's device name. You can easily remove the old instance by clicking the X icon next to the new drive name.

 

So when I hover over the red X, the tooltip says "Stop Preclear" (obviously there isn't one ongoing) and clicking it refreshes the page, but doesn't actually change anything that I can see - I've still no option to start a preclear. Pretty sure by now that Im missing something obvious, but not sure what it is!

 

Is it mounted by Unassigned Devices?

 

This is it in unassigned devices:

 

erU1iGk.png

 

The tooltip over its blue icon shows "New Device" but it looks as though the only available action is to spin it down. Thank you for replying to my questions, gfjardim - it is very much appreciated.

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.