Joe L.

Actually, the preclear script logs its reports on the flash drive in /boot/preclear_reports You might look there. If the report is not there, then it finished the clearing step, but not the post-read phase to see if it was successfully zeroed. Joe L.

Ha! I think you need glasses if that's all the change you see But I totally get the need for a better description than the change-log. I don't have the time right now to go into details and I don't remember everything I did but this is what I wrote earlier: I have added adaptive depth level, to prevent cache_dirs from thrashing disks when they are otherwise occupied and cache is evicted. I found the cache was often evicted with the number of files I had when system become occupied with other things. I added the ability to adjust depth automatically based on whether scans are judged to cause disk access or not. It judges that a disk has been accessed during scan if scan takes a long time or if any recent disk access was made (and no recent disk access was made before scanning). The purpose being to avoid the situations where cache_dirs will continuously search through my files keeping disks busy all the time. Before it was also rather difficult to tell if cache_dirs was 'trashing' my disks, now its quite clear from the log if logging is enabled (though the log is rather large at the moment). If disks are kept spinning for some consecutive scans, the depth is decreased, and future rescan is scheduled at higher depth. If the file '/var/log/cache_dirs_lost_cache.log' exists then it will write a log that is easily imported into spreadsheet (excel) so its easier to check whether it trashes disks with current settings. I also added the kill I mentioned and some other quite minor bug-fixes. If you need more let me know, and I might supply more detail over christmas. If you think it looks good and useful I might do a clean up run on the script. I havn't felt like spending more time on the script if nobody but me used it. Best Alex no, not moved on... Just have precious free time to be as heavily involved as I was a few years ago. (when I was not working.) My servers are both built with out-dated hardware. I cannot contribute in the same way I did in the past. (One is an original server sold by Limetech, with IDE based drives, the second newer, but incapable to handle virtualization) I do follow the threads... and respond occasionally... Joe L.

I would try running the short smart test before doing anything that would power cycle the disk. type smartctl -t short /dev/sdi then wait for the time it indicates and get a new smart report smartctl -A /dev/sdi followed by the same steps for the long test, waiting several hours or more as indicated when invoked before getting a subsequent smartctl -A report. (Don't forget to disable any spin-down timers, as spinning down the disk will terminate the long test.) smartctl -t long /dev/sdi waiting hours as needed, then smartctl -A /dev/sdi It might have currently stopped responding to read requests, but might start again if power cycled. The actual issue could be with the disk controller OR the disk itself. (That is not a good behavior, as if you cease being able to read a disk it is a very bad thing in any network-storage-device)

Actually, it said "0 bytes copied" so it could not read the disk when it was trying to. Might be fine in operation, but I expect you might want to keep an eye on it. Can you get a smart report on the drive right now? (does it respond at all to read requests?) What do you see when you run this command that attempts to read the disk's first 195 sectors: (it will print, at most, 30 lines of text) dd if=/dev/sdi count=195 | od -c -A d | sed 30q

based on the smart report, it is highly likely to be bad cabling to the drive, or cabling picking up noise from adjacent cables. (if you neatly tie-wrapped all the drive cables, you've caused the problem. Do NOT run the cables all parallel to each other or to power cables.) It might also possibly indicate a power supply at its limits, with the power supplied to the drive being noisy causing the checksum errors in communicating with the drive that are showing in the SMART report:

Then it indicates your disk is dropping off-line in some way and can no longer be accessed. (either a bad disk, or a power supply that cannot supply proper power to the disk, or a disk controller that is stopping to respond, or a loose cable or connector, or loose drive tray, or back-plane. ) Sorry to say, difficiult to isolate which it might be) Thanks Joe. I think that I am just going to RMA the drive even thought it passes all of the Seagate SeaTools tests. I am using a Norco 4224 case and I have tried preclearing the drive in multiple slots to eliminate the possibility of a bad cable, or PCI-E card with no luck. I was able to preclear an old 2 TB drive just fine so I am suspecting the drive. Better to RMA the drive now, then to have it stop responding when attempting to load it with your data (or fail when using it to recover another failed disk). Joe L.

Then it indicates your disk is dropping off-line in some way and can no longer be accessed. (either a bad disk, or a power supply that cannot supply proper power to the disk, or a disk controller that is stopping to respond, or a loose cable or connector, or loose drive tray, or back-plane. ) Sorry to say, difficiult to isolate which it might be)

exactly. Thee signature is partly based on the drive size. When seen as its full size on your main system, it would have the wrong preclear-signature, and the wrong partition-type.

PRetty sure it says the same thing regardless if it is pre-cleared, or not.... but skips the lengthy clearing step it would normally perform if once it starts on the process of adding the drive to the array. I did not see in your syslog any evidence of you actually adding the disk to the array. Did you press the button to actually add the disk, or just stop because it says it will clear the disk? Joe L. I stopped it because it just clears it again. I tried removing the Dynamix plugins and preclearing and then adding the drive, but have the same problem of it wanting to clear the drive again. No matter what I seem to do, it keeps saying wanting to clear the drive and never lets me get to the format drive. Are you sure you are using the latest version of preclear? Are you using unRAID v6? There was an issue with the preclear signature on v6 which has been fixed in the latest version of preclear I am 100% sure I am using the latest version. 1.15. If I do a preclear_disk.sh -v it outputs 1.15. I am on unRaid 5.0.6 SO I upgraded to unRaid 6 Beta 12 and tried preclear and then tried adding and getting the same exact problem. I am almost at capacity, what can I do? It is pretty obvious... Does the disk still test as "precleared" when you invoke: preclear_disk.sh -t /dev/sdX (where X = your drive letter) If that test says it is pre-cleared, it is... and you can add the disk to your array and let unRAID do as it wants, regardless of what the screen seems to say. If I'm right, it will skip the step of writing zeros to the entire drive and add it to your array. If I'm wrong, it will write zeros to the drive and then add it to your array. Either way, it ends up added to your array. Joe L.

PRetty sure it says the same thing regardless if it is pre-cleared, or not.... but skips the lengthy clearing step it would normally perform if once it starts on the process of adding the drive to the array. I did not see in your syslog any evidence of you actually adding the disk to the array. Did you press the button to actually add the disk, or just stop because it says it will clear the disk? Joe L.

It looks great. Joe L.

PM me a link. Your changes sound interesting and useful. I'll review and incorporate the improvements. Joe L.

Well... maybe... but maybe not. The preclear report said: == Disk /dev/sdi has NOT been successfully precleared == Postread detected un-expected non-zero bytes on disk== Basically, it wrote all zeros to the disk, but when it went to read them back to verify the write was successful, there were some bytes that were not zero. That is a very bad thing, since you cannot rely on the disk to store your data accurately. Nor can you rebuild any other failed disk accurately if one were to fail. I would try a parity check at this point. It might have an error or two as it expected all zeros... Joe L.

My first gut instinct response is that it is a memory related problem. I know you said you tested it, but not with the disks also drawing power. What make/model power supply are you using? Many have reported issues with over 6 or 7 drives on a multi-rail power supply... perhaps it is the interaction of both drawing from the power supply. If a single-rail power supply is being used, I would try a different version of unRAID (with a different version of "bash" as it seems to be the one with the segfault) from your syslog: segfault at 2 ip 0807e3a0 sp bfd912a0 error 4 in bash[8048000+be000]

Good that your memory tested OK. It is always the first suspect. What version of unRAID? I suppose it is possible for newer versions to have an issue with malloc/free in (apparently) either the shell or "dd". Nobody else has reported the same issues. In any case, you can use the -r -b and -w options to the preclear script to alter the memory "dd" uses when pe-clearing.

only two possibilities I can think of.. 1. you've run out of memory. 2. you've got some defective memory. Are you running a lot of add-ons that might be using all/most of your low-memory? Have you looked at your syslog for other hints of memory related issues? Odds are better that it is #2 with an assertion error. I'd perform a memory check for several cycles, preferably overnight. Joe L.

Just what I thought, thanks for confirming. What about the spin retry count? should I read the NEW and OLD value or the RAW? kind of worries me. RAW. It has never had to re-try. (the value is zero) Therefore the normalized value is at its starting value of 100. Apparently, if it fails to spin up a few times, the smart firmware will consider that drive to have failed.

I have precleared several disks 3-4Tb with v 1.13, is it a big problem? Did that version not completely test disks lager than 2.2Tb? The testing was fine, the written pre-clear signature was, at times, not recognized by unRAID as being present.

You did not specify which version of unRAID you are running, but... according to the preclear_disk thread: The current version is 1.15. If you have an older version, please download the newest one. Older versions prior to 1.14 did not have the ability to properly handle larger disks. (larger than 2.2TB) Versions prior to 1.15 did not work properly on 64 Bit unRAID. The current version is 1.15. If you have an older version, please download the newest one. Older versions prior to 1.14 did not have the ability to properly handle larger disks. (larger than 2.2TB) Versions prior to 1.15 did not work properly on 64 Bit unRAID. Assuming you are not on 64 bit unRAID, and not clearing a disk greater than 2.2TB, your disk did not contain all zeros after being written with all zeros. (that is not a good thing) In any case, you might want to download the current version as you are two versions behind. Joe L.

Interesting observation. (and I never gave much thought about the +1 minute, but of course you are correct, it is the next minute, which might only be seconds away.) You are also correct in that it does not look for the top level folder once started. It assumes they are in place and did not consider the issue when initially mounting disks. Yes, the problem is when first starting cache_dirs from the config/go script. Do something like this instead in the "go" script... (assuming cache_dirs resides at /boot/cache_dirs, otherwise, fix the path to where you put it. you'll also need to add whatever " -i sharename" options you like to the line that creates the svcs_restarted script.): mkdir -p /usr/local/emhttp/plugins/cache_dirs/event/ echo "/boot/cache_dirs -q" >/usr/local/emhttp/plugins/cache_dirs/event/stopping_svcs echo "/boot/cache_dirs -w" >/usr/local/emhttp/plugins/cache_dirs/event/svcs_restarted chmod +x /usr/local/emhttp/plugins/cache_dirs/event/stopping_svcs chmod +x /usr/local/emhttp/plugins/cache_dirs/event/svcs_restarted

you'll be fine, as the intermediate post-read serves as the pre-read for the subsequent cycle. Let it continue to completion.

You are trying to thoroughly test the drive, so cutting back on the testing seems a little counter-productive, but yes the preread is the least useful part of the testing, so skipping it shouldn't affect the result and would save time. Actually, it is only when "reading" that un-readable sectors can be identified. If you skip the pre-read, you'll only identify marginal sectors on the post-read, and not have any subsequent "write" to attempt to re-allocate those marginal sectors. When requesting multiple cycles, the post-read of the first/prior cycle is used as the pre-read of the next, so some time is saved and you still get the benefit of the full test. Joe L.

Keep an eye on the high-fly-writes. The normalized value has dropped quite a bit and is getting closer to its affiliated failure threshold. It has not failed SMART, but it will be something for you to watch on that drive.

They both have their place. If, for example, you've made a copy of an mp3 file, and edited it in some way (added tags, or changed length, or whatever) and put the copy in a different directory, but with the exact same name, it will be found by your script, but not mine, since its checksum and size will be different. My script will find exactly identical files on different disks or in different directories regardless if they are named the same name or different names. users of your script should NOT just delete duplicates until they verify which of the files is the one they wish to keep. They should open each in turn to verify AND check their length. More than once I've copied a movie file from one disk to another only to run out of space and have the target file truncated and end up much shorter than the original. It would be horrible if I deleted the original file and kept a similarly named, but corrupted file that was in a different directory found by your script. Joe L.

A script I wrote a while back to find duplicate files is described in and attached to this post: http://lime-technology.com/forum/index.php?topic=7018.msg68073;topicseen#msg68073 It is a series of commands that create and use intermediate files on /mnt/disk1 You can run one line at a time and examine the intermediate files if you wish. It differs from the script in this post in that it does NOT rely on the file names to be identical. It will find duplicate files even if you've renamed them. It narrows down the files it needs to examine by eliminating files that cannot be duplicates by using this logic: 1. It first gets a list of all the files on your disks, and their lengths, and eliminates any file whose length is not the same as another file. (A unique length indicates it must be a unique file) 2. Then, it generates the md5 checksum for the first 4Meg of every file where their length is not unique. (A unique MD5 checksum in the first 4 Meg would indicate a unique file, no need to verify by reading the remainder) 3. Then, it generates the md5 checksum on the entire file where the md5 checksum on the first 4Meg is not unique. (If the md5 on the first 4 Meg was not unique we need to read the remainder) 4. Lastly, it lists the files, grouping them where they have identical md5 checksums, (and identical contents), even though they may be in different folders, or even have different names. The resulting list will look like this (each group is set of identical files, regardless of their names): /mnt/disk4/Mp3/KC and the Sunshine Band/KC_The_Sunshine_Band_-_Shake_Your_Booty.mp3 /mnt/disk4/Mp3/KC and the Sunshine Band/Shake Your Booty-Earth Wind and Fire.mp3 /mnt/disk3/data/mg35/mg-kernel/mg-kernel/include/linux/i2c-id.h /mnt/disk3/data/mg35/mg35tools/firmware/uClinux-2.4/include/linux/i2c-id.h /mnt/disk1/Pictures/2009 - July 4th/IMG_1374.JPG /mnt/disk1/Pictures/PictureFrame/DCIM/2009 - July 4th/IMG_1374.JPG /mnt/disk3/data/mg35/mg-kernel/mg-kernel/fs/nls/nls_cp865.c /mnt/disk3/data/mg35/mg35tools/firmware/uClinux-2.4/fs/nls/nls_cp865.c /mnt/disk1/Movies/SD_VIDEO/PRG002/MOV0AB.avi /mnt/disk1/Pictures/Movies/20080315_135748.MPG /mnt/disk1/Pictures/PRG002/MOV0AB.MOD /mnt/disk1/Movies/SD_VIDEO/PRG001/MOV024.avi /mnt/disk3/data/USADanceShowcase-Sept2006/MOV024.MOD /mnt/disk3/data/shared/JVC-VIDEO-CAMERA/SD_VIDEO/PRG001/MOV024.MOD Once you get the list it will be up to you to decide which files in each set can be deleted. (Make sure you leave at least one of them, otherwise for certain, you will not have another copy elsewhere.) Have fun. Joe L.