Re: preclear_disk.sh - a new utility to burn-in and pre-clear disks for quick add


Recommended Posts

2 hours ago, sureguy said:

Looks like unRAID 6.4.1 (not sure about 6.4.0) no longer has strings as a command, but does have strings-BSD which appears to output the same thing.  There is also a call to /root/mdcmd which now resides in /usr/local/sbin/mdcmd  - The following command will resolve both these issues:

 


sed -i -e  "s|strings|strings-BSD|" -e "s|\/root\/mdcmd|\/usr\/local\/sbin\/mdcmd|" preclear_disk.sh

 

Looks like that got rid of one error. Still get lots of vt-100 errors though using the web terminal. My guess is that this is a screen issue though. I notice many of my keyboard keys don't work correctly in screen. For instance the delete key generates a space instead.

Link to comment
46 minutes ago, wgstarks said:

Looks like that got rid of one error. Still get lots of vt-100 errors though using the web terminal. My guess is that this is a screen issue though. I notice many of my keyboard keys don't work correctly in screen. For instance the delete key generates a space instead.

see here about vt100 - http://lime-technology.com/wiki/index.php/Configuration_Tutorial#Preclearing_With_Screen

Link to comment

Rather than "fiddle" with pre-clear issues, I simply no longer use it.   Note that the utility was written for two purposes:  (1)  to eliminate the long downtime in an array when you added a disk -- which was not useable until it had cleared the new disk  (this was arguably Joe's primary motivation for writing it); and (2) to test new disks to be sure there weren't any "infant mortality" issues.

 

#1 is completely obsolete => v6 clears new disks without any impact on array functionality.

#2 is easily done with any of the manufacturer's diagnostic utilities.   I use WD's Data Lifeguard, but SeaTools, HGST's WinDFT,  Toshiba's Storage Diagnostic, or any of the various 3rd party disk testing utilities (HDTune, Crystal Disk, etc.) can do this ... and these have the advantage that you can test your disk on another system without impacting the server.

 

When I get a new disk, I attach it to a spare SATA port on my desktop; then run Data Lifeguard and do a Short Test, Long Test, full Erase, and then repeat the short and long tests.   ANY errors and I return the disk -- otherwise I add it to whatever system I bought it for (whether UnRAID, an HTPC, or one of our desktops). 

 

Link to comment
5 hours ago, garycase said:

Rather than "fiddle" with pre-clear issues, I simply no longer use it.   Note that the utility was written for two purposes:  (1)  to eliminate the long downtime in an array when you added a disk -- which was not useable until it had cleared the new disk  (this was arguably Joe's primary motivation for writing it); and (2) to test new disks to be sure there weren't any "infant mortality" issues.

 

#1 is completely obsolete => v6 clears new disks without any impact on array functionality.

#2 is easily done with any of the manufacturer's diagnostic utilities.   I use WD's Data Lifeguard, but SeaTools, HGST's WinDFT,  Toshiba's Storage Diagnostic, or any of the various 3rd party disk testing utilities (HDTune, Crystal Disk, etc.) can do this ... and these have the advantage that you can test your disk on another system without impacting the server.

 

When I get a new disk, I attach it to a spare SATA port on my desktop; then run Data Lifeguard and do a Short Test, Long Test, full Erase, and then repeat the short and long tests.   ANY errors and I return the disk -- otherwise I add it to whatever system I bought it for (whether UnRAID, an HTPC, or one of our desktops). 

 

 

To point #2 this assumes the user has a second system that is compatible with the software, and they want to run that system for an extended period of time (presumably without restarting it, etc).  There is some advantage to doing a preclear on an unRAID system, especially if the intention is to add the drive to the array if it passes testing.

  • Upvote 1
Link to comment
21 hours ago, sureguy said:

Looks like unRAID 6.4.1 (not sure about 6.4.0) no longer has strings as a command, but does have strings-BSD which appears to output the same thing.  There is also a call to /root/mdcmd which now resides in /usr/local/sbin/mdcmd  - The following command will resolve both these issues:

 


sed -i -e  "s|strings|strings-BSD|" -e "s|\/root\/mdcmd|\/usr\/local\/sbin\/mdcmd|" preclear_disk.sh

 

 

Just for confirmation:

https://lime-technology.com/forums/topic/68985-unraid-os-version-641-stable-release-available/?page=5&tab=comments#comment-632515

 

Edited by Frank1940
Link to comment
10 hours ago, Frank1940 said:

 

@garycase, could describe exactly what your physical setup is to do this.  Did you install a hot swap bay or what?

 

EDIT:  One more quick question, will WD Data Lifeguard work with any manufacturer's drive?

 

I have an eSATA port on my desktop connected to one of these docks:  https://www.newegg.com/Product/Product.aspx?Item=9SIA1DS28Y2007&cm_re=esata_dock-_-9SIA1DS28Y2007-_-Product

 

You don't really need the docking unit -- just an eSATA cable and power cable would work fine.   You can also run it with a USB dock, although it takes a long time with USB v2 (v3 docks work just fine).

 

And yes, Data Lifeguard works fine with every drive I've tried it with -- WD, Seagate, HGST, Maxtor, Toshiba, Fujitsu, Samsung, etc.

 

 

Link to comment
6 hours ago, sureguy said:

 

To point #2 this assumes the user has a second system that is compatible with the software, and they want to run that system for an extended period of time (presumably without restarting it, etc).  There is some advantage to doing a preclear on an unRAID system, especially if the intention is to add the drive to the array if it passes testing.

 

Compatibility with testing software isn't an issue -- any Windows system has a plethora of options (as I listed above); and there are several testing utilities for Macs as well.

 

It's true the system used to test a drive does indeed have to be on for the duration of the test -- whether it's your UnRAID server or another system.

 

As for the advantage of doing the test on your server -- that's a very minor advantage.   It takes perhaps 20 seconds to pop a drive in my eSATA dock and start the testing.   You do, of course, then have to install the drive in whatever system it's destined for after the test.   If it's a drive destined for one of my UnRAID servers I suppose you can argue that's 20 seconds wasted -- but I think most would agree that's not much of a factor :D.     

 

Link to comment
6 hours ago, garycase said:

 

Compatibility with testing software isn't an issue -- any Windows system has a plethora of options (as I listed above); and there are several testing utilities for Macs as well.

 

It's true the system used to test a drive does indeed have to be on for the duration of the test -- whether it's your UnRAID server or another system.

 

As for the advantage of doing the test on your server -- that's a very minor advantage.   It takes perhaps 20 seconds to pop a drive in my eSATA dock and start the testing.   You do, of course, then have to install the drive in whatever system it's destined for after the test.   If it's a drive destined for one of my UnRAID servers I suppose you can argue that's 20 seconds wasted -- but I think most would agree that's not much of a factor :D.     

 

 

You can't simply take your use case scenario, and then apply it to everyone else.  Some people use unRAID simply as a media server and don't necessarily have a PC that can be on for extended periods of time for this work (a road warrior with only a laptop for work for instance).  Some people may only have a work laptop, and consume media with hardware other than a PC.  Most people won't have an e-sata dock like you have - e-sata was never a very popular technology.  Your proposed solution requires a financial investment on the part of some of the (probably less technically savvy) members of the community for something that once required none.

 

If the preclear script can be made to work in unRAID 6.4.1 I really don't see the harm.

Edited by sureguy
Link to comment

 

12 minutes ago, sureguy said:

 

You can't simply take your use case scenario, and then apply it to everyone else.  Some people use unRAID simply as a media server and don't necessarily have a PC that can be on for extended periods of time for this work (a road warrior with only a laptop for work for instance).  Some people may only have a work laptop, and consume media with hardware other than a PC.  Most people won't have an e-sata dock like you have.  Your proposed solution requires a financial investment on the part of some of the (probably less technically savvy) members of the community for something that once required none.

 

If the preclear script can be made to work in unRAID, I really don't see the harm.

 

If a desktop Windows computer is available, one could use a hot swap bay like this one rather than an ESATA solution:

 

     https://www.amazon.com/dp/B00126U0VA/_encoding=UTF8?coliid=I2HTCJRBPFQG14&colid=11QMLXLGKSS72&psc=1

 

While there is a cost involved, it is rather nominal.  

Edited by Frank1940
Link to comment

Many different methods to clear and/or certify a disk. 
I use unraid itself on a refurbished laptop with badblocks.
I do a multipass pattern test. 
At the end of it I use the preclear script to add the signature. 
It's an old Dell E series laptop which has a eSATA port along with an ORICO Tool-Free USB 3.0 & eSATA to 2.5" & 3.5 " SATA External Hard Disk Drive Lay-Flat Docking Station HDD.
It's a pretty compact temporary solution to getting the job done and away from the server. 
Granted it is an extra cost. A refurbished Dell can be scored pretty cheaply on eBay. E6510, E4310, E4300, etc, etc.
These models have an eSATA port but do not support PMP. They are useful for diagnostics and/or clearing/certifying a disk.
I'm sure a more modern refurbished laptop with USB 3.0 would suffice as well.

Link to comment
23 hours ago, garycase said:

 

 

When I get a new disk, I attach it to a spare SATA port on my desktop; then run Data Lifeguard and do a Short Test, Long Test, full Erase, and then repeat the short and long tests.   ANY errors and I return the disk -- otherwise I add it to whatever system I bought it for (whether UnRAID, an HTPC, or one of our desktops). 

 

Could I pass a new drive through to a VM and test?  I don't have a 2nd system anymore.  I guess it should work

Link to comment

A couple of additional comments ....

 

r.e. "... Most people won't have an e-sata dock like you have - e-sata was never a very popular technology ..."  => Actually, all you need is a SATA connection and a power cable.   You could simply run a SATA cable from any available SATA port out the back of the PC and leave it available for connecting a spare drive when needed -- or you could connect it to an eSATA dock if you wanted to make it more convenient.    Ditto a power cable (which you wouldn't need with a dock, since it would provide power).    Or you could use a USB3 docking unit and avoid a SATA connection altogether.

 

r.e. "... one could use a hot swap bay like this one ..."  => An excellent choice, and with the tray-less unit Frank suggested it's  just as easy as plugging a drive in an external bay.

 

r.e. "... Could I pass a new drive through to a VM and test? "  ==>  Certainly.   Just set up a Windows VM, pass a SATA port through to it; and you can do the testing via your VM.    Of course since you'll already have the drive installed in your UnRAID server in this case, you could simply use pre-clear -- depends on whether you want the added flexibility of the multiple tests you can choose with the various other testing utilities.

 

r.e. "... f the preclear script can be made to work in unRAID 6.4.1 I really don't see the harm. " ==>  I certainly didn't intend to imply otherwise.    There's no reason not to use pre-clear;  I was simply noting that it's not really needed, since there are plenty of other testing utilities to test a new disk; and the original advantage when it was written of not having your server down while a new disk was being cleared is no longer a factor.

 

 

 

 

Link to comment
20 minutes ago, garycase said:

r.e. "... Could I pass a new drive through to a VM and test? "  ==>  Certainly.   Just set up a Windows VM, pass a SATA port through to it; and you can do the testing via your VM.    Of course since you'll already have the drive installed in your UnRAID server in this case, you could simply use pre-clear -- depends on whether you want the added flexibility of the multiple tests you can choose with the various other testing utilities.

 

 

 

 

 

 

I'm on 6.4.1 so the script doesn't work anymore.

 

Do you mean pass through a sata port or add the drive unassigned? If the former, I didn't know you could do that

Link to comment
  • 1 month later...

IT pro here so probably used to a little more follow through...


Is there a good reason why nobody can be bothered to update the plug-in and maintain it for the current release? How many people have wasted 30-45mins digging through these forum posts in order to find out that two lines of the script are bad?

 

Like how hard is it to do an environment check, identify what's missing, offer to fix it or inform the user? Or just patch the file and update the link.

I bet it's easier that writing up in detail why it doesn't work. Just fix it and post it, no? This is why github.

Link to comment
23 minutes ago, geekazoid said:

IT pro here so probably used to a little more follow through...


Is there a good reason why nobody can be bothered to update the plug-in and maintain it for the current release? How many people have wasted 30-45mins digging through these forum posts in order to find out that two lines of the script are bad?

 

Like how hard is it to do an environment check, identify what's missing, offer to fix it or inform the user? Or just patch the file and update the link.

I bet it's easier that writing up in detail why it doesn't work. Just fix it and post it, no? This is why github.

 

Most of the plugins and dockers are provided by users like yourself (possibly even IT pros^_^), completely without compensation except possibly in the form of donations from other users. And the vast majority of support provided on the forum is also provided by users like yourself (see above). We are mostly all volunteers here. If you would like to volunteer to support the plugin I'm sure other users would be grateful.

 

Limetech is a very small company and concentrate on core functionality, but they have a lot of enthusiastic users who volunteer their time and effort to produce extensions, apps, help and advice to the rest of the user community.

Link to comment

I don't know what Limetech has to do with my comment. I'm well aware why they aren't maintaining a community script. Not their scope.


Most of the web runs on volunteered code from users. I'm just saying, if its in github already, do a pull request and fix it, maintain the link in a wiki not a forum. The comments in this thread with the figuring out bits are the good stuff, but shouldn't be of interest to EVERY SINGLE NEW USER. Someone, anyone, just needs to push that back into the code and update it. Then move on and let the next geek fix what he finds broken. That's how it works, usually.

I'll definitely volunteer but that doesn't take away from the point that this stuff isn't working and leaving it broken for years is... well it feels like 2002 not 2018.

 

 

Link to comment
9 minutes ago, geekazoid said:

Someone, anyone, just needs to push that back into the code and update it.

Sounds like you are the right man for the job.

 

I'm still not sure who your original rant was aimed at, but the person or persons who take over a plugin are generally expected to support it, when that doesn't happen for whatever reason, it can fall by the wayside where no one wants to take over the responsibility of maintaining it, probably for similar reasons of why it was abandoned in the first place.

 

If you are willing to support it, there are MANY people who would be grateful to you. If you make a change and it erases someones data, they will be very mad at you. It's a lot of responsibility to take on with NO tangible reward.

 

This particular script has a very long and colorful past, with many different hands touching the code along the way. It is no longer as valuable as it once was, there are many other tools available outside of unraid that accomplish largely the same thing. Bottom line, it's not really needed, so the priority to keep it maintained is very low.

Link to comment

I wasn't aiming at anyone, I was frankly wondering why this was being done on a "no good deed goes unpunished" basis. Seems like whoever picks up the ball gets pummelled with feature requests. Meanwhile there are snippets of code fixes that aren't merged. Github has a whole solution for this. If someone abandons a fork, just start a new one. They can always come back and merge your pulls into their fork and add something new of their own. (edit: looks like someone has done this, per a reddit post I just saw)

Regarding ownership, is there a license? If there is none, its public domain. It doesn't matter who is mad about it, that's how it works. Copyright is not retroactive. Anyone can modify it with nothing more than a line of credit in the code, and that is but a courtesy of convention with things like bash scripts.

Anyway I just wanted to see if anyone cared or if this was a dead thread.


In this particular case, I prefer your comment about other tools. Sounds more pragmatic.

Edited by geekazoid
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.