Joe L.

Incorrect, if no option is specified, then the unRAID configuration preference setting is used. Look under Settings->Disk Settings->Default-partition-format to set your default preference. Thanks for the details Joe. But just for clarification, isn't it true that the default setting for v5 is 4K aligned? ... in which case I'd think you don't need the -A switch (with v5) UNLESS you've changed that setting. Correct? I do not know the default...

I thought that the latest version of the pre-clear script would, when run on a system with UnRAID v5, automatically always use sector 64 => but perhaps that's not the case. Incorrect, if no option is specified, then the unRAID configuration preference setting is used. Look under Settings->Disk Settings->Default-partition-format to set your default preference. One more thing... Unless your disk is a WD "EARS" drive with the firmware that performs poorly when partitioned to start on sector 63, then odds are you would not notice any difference in performance at all regardless of where the partition starts. (and even that drive would work just fine with the partition starting on sector 63 unless you were really anal about performance) You likely could have left things as they were.

Correct anything with a current normalized value within 25 of its affiliated failure threshold is "near threshold" For some attributes, the manufacturer sets the failure threshold within a count or two from the factory initial value. same above.

Where's the report showing the changes? The high-fly-writes went from 100 to 75 during the pre-clear. Although nowhere near the failure threshold, keep an eye on it in the next few months. Thanx Joe .... I will add it to the array now to replace another failing hd. What is a high-fly? should I be worried? I will keep an eye on it the next few months. Sorry here is the report forgot there where 3 files. From the SMART Wiki: High Fly Writes HDD producers implement a Fly Height Monitor that attempts to provide additional protections for write operations by detecting when a recording head is flying outside its normal operating range. If an unsafe fly height condition is encountered, the write process is stopped, and the information is rewritten or reallocated to a safe region of the hard drive. This attribute indicates the count of these errors detected over the lifetime of the drive. This feature is implemented in most modern Seagate drives[1] and some of Western Digital’s drives, beginning with the WD Enterprise WDE18300 and WDE9180 Ultra2 SCSI hard drives, and will be included on all future WD Enterprise products.[20]

Where's the report showing the changes? The high-fly-writes went from 100 to 75 during the pre-clear. Although nowhere near the failure threshold, keep an eye on it in the next few months.

I did some research. The number after the "+" is the number of partial blocks read or written. Since the pre-clear script periodically sends a query to the process actually writing a disk that is using "dd", and has no way to synchronize the request, it is possible to get a response while in the middle of writing a "block" of data. For that reason you'll see the +n increment over time. It is just a measure of how many times you managed to ask its percentage complete when it was in the middle of writing a block of data. Joe L.

They are just the statistics reported by the "dd" command. I never cared what they meant (was more interested in the number of blocks read). Probably indicates some partial blocks of bytes. (nothing to worry about) You would have to look up the "dd" command to learn more. Joe L.

Smething happened prior to the preclear to mark those sectors as having a checksum at the end of the sector that did not match the contents of the sector. As already mentioned, they were re-written in place, and the contents then matched the expected checksum for those 7 sectors. A subsequent preclear cycle will tell you more. With any luck you'll not see any additional un-readable sectors (all checksums at ends of sectors will match the contents of their affiliated sector) Joe L.

Ah-ha! ========================================================================1.13 == invoked as: ./preclear_disk.sh -A /dev/sdb == ST4000DM000-1F2168 W3009TM3 == Disk /dev/sdb has been successfully precleared == with a starting sector of 1 == Ran 1 cycle == == Using :Read block size = 8225280 Bytes == Last Cycle's Pre Read Time : 9:32:25 (116 MB/s) == Last Cycle's Zeroing time : 8:22:56 (132 MB/s) == Last Cycle's Post Read Time : 19:03:40 (58 MB/s) == Last Cycle's Total Time : 37:00:07 == == Total Elapsed Time 37:00:07 == == Disk Start Temperature: 29C == == Current Disk Temperature: 36C, == ============================================================================ ** Changed attributes in files: /tmp/smart_start_sdb /tmp/smart_finish_sdb ATTRIBUTE NEW_VAL OLD_VAL FAILURE_THRESHOLD STATUS RAW_VALUE Raw_Read_Error_Rate = 118 100 6 ok 182808288 Spin_Retry_Count = 100 100 97 near_thresh 0 End-to-End_Error = 100 100 99 near_thresh 0 Airflow_Temperature_Cel = 64 71 45 near_thresh 36 Temperature_Celsius = 36 29 0 ok 36 No SMART attributes are FAILING_NOW 0 sectors were pending re-allocation before the start of the preclear. 0 sectors were pending re-allocation after pre-read in cycle 1 of 1. 0 sectors were pending re-allocation after zero of disk in cycle 1 of 1. 0 sectors are pending re-allocation at the end of the preclear, the number of sectors pending re-allocation did not change. 0 sectors had been re-allocated before the start of the preclear. 0 sectors are re-allocated at the end of the preclear, the number of sectors re-allocated did not change. ============================================================================ Looks great!!! Joe L.

If it completed it would have logged the results in the /boot/preclear_reports directory.

All looks good to me. Hope you enjoy the initial setup/config of your server.. Joe L.

Not entirely true. You are using CPU cycles, and the processes are all sharing the same CPU on a time-slice basis, and the I/O requests are still being queued to the disk buffer cache, and it has to be read to return the disk buffer even if not being read from a physical disk.)... Good..

It would depend on if your explorer is indexing the files, creating thumbnail images, or not. In any case, using file-explorer to simply list the files is exactly the same as a "find" command in cache-dirs. It will have a minimal effect on a parity check (barely noticeable, if at all)

I agree. RMA it. You DO win the "drive with the most reallocated sectors" I've ever seen reported contest (if it is any consolation). Reallocated_Sector_Ct = 72 100 10 ok 35928 Current_Pending_Sector = 91 100 0 ok 1600

I am in the middle of making the necessary code changes, but am trying to work out how to determine if a parity check/sync/reconstruction is in progress. The only way I know is to query /proc/mdcmd, but since on a properly tuned system, the "find loop" occurs every 10 seconds, or less, I need to code cache_dirs to only look for a "parity sync/calc/reconstruction" on a less frequent basis than every few seconds... probably every 5 minutes or so would do it, otherwise, it might impact the md driver if the status command was invoked too frequently. Then, cache_dirs would suspend itself within the first 5 minutes of a parity sync/calc. Ideas? Feedback? Joe L.

Does this comment also apply to my note r.e. suspending Cache_Dirs during parity checks? It sure seems that would be a VERY useful/nice feature. Seems like a simple "If parity check in progress, don't start the next Find" check in Cache_Dirs would basically suspend it => at least after the current Find completed. No reason to shut down Cache_Dirs ... it would simply keep checking at the current intervals, but as long as the parity check was in progress, wouldn't initiate any more Finds ... thus not interfering with the check. ... and of course once the parity check was over, the next time Cache_Dirs checked all the disks would still be spinning, so it'd be very quickly up-to-date. It does this today (puts itself to sleep) when the "mover" runs, so adding the logic for a parity check/disk rebuild is fairly easy. I thought it might be. Does that mean you're going to add it and update the Cache_Dirs download? Not today, I've got a very busy afternoon/evening.

Does this comment also apply to my note r.e. suspending Cache_Dirs during parity checks? It sure seems that would be a VERY useful/nice feature. Seems like a simple "If parity check in progress, don't start the next Find" check in Cache_Dirs would basically suspend it => at least after the current Find completed. No reason to shut down Cache_Dirs ... it would simply keep checking at the current intervals, but as long as the parity check was in progress, wouldn't initiate any more Finds ... thus not interfering with the check. ... and of course once the parity check was over, the next time Cache_Dirs checked all the disks would still be spinning, so it'd be very quickly up-to-date. It does this today (puts itself to sleep) when the "mover" runs, so adding the logic for a parity check/disk rebuild is fairly easy.

Hi JoeL., what do you think about an output to the log file, if cachedirs takes longer than a certain amout of time repeatedly (let's say more than twice - this filters the first time reading and any rereading when the array was accessed and dcahes flushed) ? I think that would help a lot to identify the case, when it is not working properly and "overloaded" - instead of trying to observe that by checking spindown behaviour. If it is not taking too much cpu cycles, such an output could include the number of files read by find - so user get's an info with what number of files that case happens - what do you think? Doable? some interesting ideas... I'll give them some thought. More diagnostic output is certainly better when trying to understand what is happening. Might eliminate a post or two saying "why is my array spinning up" I'm running cache_dirs.

no clue what this means though between last 2 lines there was a mover action going on just fyi It indicates any given "find" command is taking over 1000 seconds. (that is pretty bad ) It must be going to the physical disks to get data not in disk buffer cache in memory. That is over 16 minutes, so if you've set your spin-down timer shorter, it might have to spin up the disks every time. I'd drop all the extra options other than the "-e" options to start. On my system, it looks more like this, with the "find" operating in RAM and returning in a fraction of a second. cache_dirs -F -e data Executed find in 0.294915 seconds, weighted avg=0.294915 seconds, now sleeping 5 seconds Executed find in 0.295294 seconds, weighted avg=0.295168 seconds, now sleeping 4 seconds Executed find in 0.295314 seconds, weighted avg=0.295241 seconds, now sleeping 4 seconds Executed find in 0.335031 seconds, weighted avg=0.311157 seconds, now sleeping 3 seconds Executed find in 0.296573 seconds, weighted avg=0.306296 seconds, now sleeping 4 seconds Executed find in 0.297404 seconds, weighted avg=0.303755 seconds, now sleeping 5 seconds Executed find in 0.296930 seconds, weighted avg=0.302049 seconds, now sleeping 6 seconds Executed find in 0.296885 seconds, weighted avg=0.300901 seconds, now sleeping 7 seconds Executed find in 0.370029 seconds, weighted avg=0.314727 seconds, now sleeping 6 seconds

no, that is expected, the most recently used disk buffers re-use the least recently used ones, and no. Other than to look at the output of free -l there are no commands. Nothing can pin the directory entries in memory. Joe L.

It is more than just those two files people are interested in... (I use XBMC, and it also has .nfo, .tbn) To keep them in memory requires reading them and sending the contents to /dev/null You can add a lines in the main loop find "$i"/"$share_dir" -type f -name "*.jpg" -exec cat {} > /dev/null \; find "$i"/"$share_dir" -type f -name "*.xml" -exec cat {} > /dev/null \; Oh cool. Why send the contents to /dev/null \;? The contents must be read to have them in cache. (it is not enough to just have the file names in cache) You must do something with the blocks read, so /dev/null is best. Now, this will take a lot more RAM if you have a lot of large .jpg files. Each file occupies (potentially) 100k or more. It you have 1000 thumbnails, then 100k * 1000 = 1Gig of additional RAM needed for cache.

It is more than just those two files people are interested in... (I use XBMC, and it also has .nfo, .tbn) To keep them in memory requires reading them and sending the contents to /dev/null You can add a lines in the main loop find "$i"/"$share_dir" -type f -name "*.jpg" -exec cat {} > /dev/null \; find "$i"/"$share_dir" -type f -name "*.xml" -exec cat {} > /dev/null \;

I do not support cache_dirs insalled via SimpleFeatures. It is not the program I have attached to the cache_dirs thread. It is a copy modified for the plugin. (yes, it is confusing, since they did not change the version number) Try running it in the foreground with verbose mode enabled to see where it is spending its time. Personally, I would NEVER use the -m or -M options or ask it to go 10 directories deep.

I've never tried it on another Linux distro but I suspect it will not work. It may work, HOWEVER when you list drives eligible for clearing it will not be able to read any of the normal unRAID config files, and therefore will may list all the drives. It is especially important to know the model/serial number of the drive you wish cleared, and its affiliated /dev/sdX device name, since you'll feel pretty darned dumb/stupid/careless if you answer "Yes" to zeroing a disk holding your ubuntu installation.

cache_dirs -d NN -w Where NN = the maxdepth used by the "find" command. I would normally say you could try cache_dirs -d 5 -w as most directory trees for media are less than 5 levels deep. /mnt/disk5/Movies/moviename/files = only 4 levels