gfjardim

In the next update a Erase option will be available. It will use random generated data to fill the disk, making drive wipe more secure. I'm pretty sure you will make a good use of it.

Just updated the plugin. You will be asked to install "Statistics" after update.

You brought to my attention this problem with corporate rules, and I think it's very positive for the plugin development. I'll remove the report functionality from the plugin altogether, and make a separate plugin for that. Upon installation, users will be prompted if they want to install the new plugin or not. This will avoid the download of TOR inside corporate facilities and avoid this kind of situation. My deepest apologies for the mess it created.

The firewall is only open at certain days/times and only when certain conditions are met. The firewall controls are separate from the rest of the network. The firewall slams shut if certain kinds of traffic are detected (TOR being one of those) Unraid is not work's primary backup. It is an onsite backup to the backups. We replaced our Ubuntu based backup years ago as Unraid was more stable for this function. We are required to have this onsite backup and it must use a different OS/filesystem from the primary data. CA Auto Update is scheduled to run at a time when other machines have access to reach out for updates while the firewall is open. As such, no user has access to touch files. A script from another machine runs via SSH which stops samba, unmounts each disk, etc The firewall will not open if samba is detected on the network. Before the start of the day, samba and the array is restarted ... I do not have access to manage the firewall or the Samba policies/protocols -> that belongs to a separate independent entity. … Back to the original question. Can the preclear data collection be Opt-In… or at least an option to Opt-Out? It is Opt-In. The Tor service is only activated while uploading the stats. What do you need to satisfy your company's policy?

Do you mean that you physically move the drive without shutting down the server? I stop the array, pull the drive, relocate the drive, restart the array. So when I hover over the red X, the tooltip says "Stop Preclear" (obviously there isn't one ongoing) and clicking it refreshes the page, but doesn't actually change anything that I can see - I've still no option to start a preclear. Pretty sure by now that Im missing something obvious, but not sure what it is! Is it mounted by Unassigned Devices?

This is because you removed the old drive and the Linux UDEV subsystem reused it's device name. You can easily remove the old instance by clicking the X icon next to the new drive name.

This is tor f#ckin up the logrotate directory. I'll change the build script ant update the package. Thanks for reporting.

Guys, the statistics report system has been put in place. The great think about scripts is that they allow anyone to see what happens in the background. I'll comment some technical details to make this as transparent as possible. When finishes or fails in some of the stages the preclear script will write a ".stats" file under /boot/config/plugins/preclear.disk/ directory. This file is a multi-level INI file and each of it's sections contains a information that can get exported to Google Forms. Each section has an entry variable which corresponds to the desired field in "Google Forms". It also has a title variable which corresponds the webui title and a value variable which contains the actual data. The relevant function in script can be found here. When Unassigned Devices or Preclear Disk plugins gets loaded by the web interface, the getOpenStatistics() javascript function will probe for any existing ".stats" files on the plugin config directory (here). If a report is found, it will trigger an alert asking the user to Send or Cancel that report. If you hit Send, the getOpenStatistics() function will send a POST request with "action=send_statistics" form-data to "Preclear.php". That request will be processed by this portion of the code; the script will start the TOR service, compare both local external IP and the TOR remote IP (and exit if they match), create the curl request command and run it, stop the TOR service and remove the ".stats" file at the end. If you hit Cancel, the getOpenStatistics() function will send a POST request with "action= remove_statistics" form-data to "Preclear.php", and the script will remove the ".stats" file. I hope this will remove any doubts about this functionality, and I do expect for your statistics!

Tried with ie edge, ie 11, firefox and chrome. None works. Any ideas? Please update the script and try it again.

Could you change the google form type to log the IP address, so that we can see that what is logged is not our own IP? On that same note, are you planning to allow us to peruse the data that you collect on the google form? If you say no, I understand, and don't necessarily object, but it would be nice if you would add a link to the form data on the submission confirmation after you send data. I don't think it's necessary, the plugin does the comparison by itself: case 'send_statistics': $file = $_POST["file"]; $stat = parse_ini_file($file,true) ?: []; shell_exec("/etc/rc.d/rc.tor start 2>&1"); $myip = trim(shell_exec("curl -s http://whatismyip.akamai.com 2>/dev/null")); $torip = trim(shell_exec("curl -s --socks5-hostname 127.0.0.1:9050 http://whatismyip.akamai.com 2>/dev/null")); if ($myip == $torip) { exit(); } shell_exec("echo sending preclear report:|logger -t'preclear_disk'"); shell_exec("echo my ip = $myip|logger -t'preclear_disk'"); shell_exec("echo tor ip = $torip|logger -t'preclear_disk'"); $cmd = "curl -s --socks5-hostname 127.0.0.1:9050 \"https://docs.google.com/forms/d/e/<id>/formResponse\" -d ifq"; foreach ($stat as $key => $value) { $cmd .= " -d \"{$value['entry']}=".str_replace("^n", "\n", $value['value'])."\""; } shell_exec("$cmd 2>&1"); shell_exec("/etc/rc.d/rc.tor stop 2>&1"); @unlink($file); break; I'll put the form URL and the respective spreadsheet in the main OP, so anyone can use it.

Any way to anonymize the IP address? Not that I'm aware of if you continue to use google. One reason google is "free" is because they collect every bit of data they can on every user and computer they can. Cross referencing all the little tidbits allows them to sell user profiles as a "product" to advertisers. They can't do that if they can't correlate info somehow. As long as you disclose that google is the one receiving and holding the form data, I see no issue with it. The data really is anonymous to you, and anyone else accessing the forms from outside. It's only google's internal workings that do the correlation and data mining. Maybe TOR? I'm able to make use of TOR to anonymize the IP address. Is that ok for you, guys?

Any way to anonymize the IP address? Not that I'm aware of if you continue to use google. One reason google is "free" is because they collect every bit of data they can on every user and computer they can. Cross referencing all the little tidbits allows them to sell user profiles as a "product" to advertisers. They can't do that if they can't correlate info somehow. As long as you disclose that google is the one receiving and holding the form data, I see no issue with it. The data really is anonymous to you, and anyone else accessing the forms from outside. It's only google's internal workings that do the correlation and data mining. Maybe TOR?

I'd like to see google mentioned, since they are the entity that is actually receiving the data on your behalf. That way if I'm watching the bits flow through my router, I'll know with reasonable certainty that a connection from unraid to google is a result of the dialog I just clicked. The "anonymous" wording may be the end result to YOU, but it certainly isn't anonymous to google. Any way to anonymize the IP address?

I was thinking about presenting it on first open of Preclear/Unassigned Devices, and let users change it on the Help Section of Preclear Plugin page. Why do you think it should be done after each preclear session? Why not? That way if something did not seems right or there was a problem (or even a recently introduced size or disk model), the user could decide that in this case, he might want to upload the data. If you are going to provide an check-box type option, you might provide an "Ask me each Time" as a third choice to hard 'Yes' or 'No' choice. You are providing this software free to the user base and you do (in my opinion) have the right to be insistent in requesting an upload of the results. Absolutely. I accept as a user of your software that you ask certain things in return to help you. I just ask that you be open and transparent when you make those requests. I would be perfectly fine with a mandatory nag at the end of each preclear that asks "Will you help me improve this software by providing me with the statistical results and / or errors of this preclear?" What I will not stand for is the silent transmission of my information with no explicit permission. Will this work? How did I implemented it? 1) The script will write a ".stats" file on /boot/config/plugins/preclear.disk directory after it's done; 2) The webui will present that popup for each ".stats" file it found on /boot/config/plugins/preclear.disk directory; 3) ".stats" file will be deleted after sent, of after "Cancel" is clicked.

I was thinking about presenting it on first open of Preclear/Unassigned Devices, and let users change it on the Help Section of Preclear Plugin page. Why do you think it should be done after each preclear session?

As far as I am concerned, #2 is the only one that I would consider is not an invasion of my privacy. While I realize that the other information is nice to have, it is not essential to working of this product. While LimeTech 'knows' how many time its software is download (which I assume you also know) and how many licenses it has issued, it has no real knowledge of how its product is being used, actual numbers of disks and with what plugins, Dockers, or VM's. (There are a fair number of us with Pro licenses that aren't using anywhere near the numbers of disks to require it!) I realize that I have exposed some of this data in my 'signature' line of this post. However, the collection of that data would be very difficult and expensive process and even more difficult to tie directly to me as an individual. (The upload of the data you propose to Google will have my IP address as a part of that data which would serve Google's purposes for THEIR data collection! Read their terms of use...) A custom made solution for data collection doesn't worth the effort, so I guess IP recording is inevitable (but every site/service track us nowadays even this forum). I agree you have all the right to choose if your data is uploaded or not, so I'm making this optional. Is something like this enough?

What I expect to collect: 1) number of disks that are precleared (dying to know ); 2) any silent bugs that gets reported on the logs; 3) average size of disks that gets precleared, their model and the average speed and time we should expect to that model; 4) success rate; 5) rate of disks with SMART problems;

Where is this data going? If it is to be transmitted back to you, will there be an opt-out setting? To Google Forms. Since all data is anonymous (no serial exposed, for example), is there a real need for an opt-out setting? To be brutally honest, I am very suspicious of any type of application that phones-home. Almost everyone of them promises that all Data is be anonymized but it has often been found later that was not the complete case. I would prefer to have an opt-out option but at least in this case, I have the option to doing the preclear by using the old-old shell script. (I admit that I am paranoid and that I have disable both Flash and Java on all of my household computers. I refuse to link e-mail servers together and/or any of the social media sites together. I avoid clicking on 98% of the links that I am sent by 'Friends' until I have verified them from a second source. But then I have never been hit by a virus or malware to date...) Since you fell strong about it can add a toggle to disable the report. These are the statistics I expect to collect: https://docs.google.com/spreadsheets/d/1zERJQlFWGgMGKnKgGUNKnguGUdCQ2g0XT9Q5yMJio6o/edit?usp=sharing

Where is this data going? If it is to be transmitted back to you, will there be an opt-out setting? To Google Forms. Since all data is anonymous (no serial exposed, for example), is there a real need for an opt-out setting?

Guys, I'm considering add some sort of statistics report inside the plugin script. This will allow me to track potential bugs and performance data. Initially I'm thinking in gathering model, size, smart device type, total elapsed time, log and if it was successful or not. Any concerns and/or suggestions?

Awesome video, gridrunner, added to OP if you don't mind.

Just did that. Please note that if you use CloudPrint you need to readopt your printers by accessing the URL presented in the container log. If possible, please wait until CA updates the template or use the updated directly from my repository: https://github.com/gfjardim/docker-containers/tree/templates. For unsupported Brother printers like yours, you'll need do add the driver to the container config directory, following these steps: 1) Go tohttp://support.brother.com/g/b/countrytop.aspx?c=us〈=en ; 2) Search for your model number using the search input; 3) In the Downloads menu, you need to select Linux in Select OS Family, and then select Linux(deb) in Select OS Version. Then hit Search; 4) In then next page, you need to download the CUPSwrapper printer driver (deb package). 5) After Accepting the EULA and downloading the file (in this case, hll2300dcupswrapper-3.2.0-1.i386.deb), you have to create the "brother" folder inside your container config directory (example: /mnt/cache/appdata/CUPS/brother) and put the .deb file in there (ex: /mnt/cache/appdata/CUPS/brother/hll2300dcupswrapper-3.2.0-1.i386.deb). 6) Restart your container.

My script formats in 4k alignment by default; didn't implemented "start on sector 63" because I thought it wasn't relevant anymore. If you need, you can use Joe L. script and convert it. Enviado de meu SM-G930F usando Tapatalk

Not a con, since the second read was done using RAM cache and didn't stress the disk. If I'm not mistaken, -A switch is used to change the SMART device type, and it's not implemented in my script. I don't remember what -f is used for. Enviado de meu SM-G930F usando Tapatalk

Saving some time is always welcome! How is it possible? Is this script also available for commandline use? I don't like to preclear on my main server and have a spare machine not connected to the network doing it. Joe L. script read the disk in chunks, sum them and check for a zero result. In fact, it read the disk twice, once for speed calculation and other for comparison, relying on cache to keep the first read on RAM to speed up the process. My script read all the disk at once, and compare it against the pseudo block device /dev/zero. Using direct read, it also skip the RAM cache entirely so it has a smaller memory footprint. It's not ready for command line because it doesn't have help, but if you want, you can always dig up the code for available options.