Manticore Posted October 18, 2014 Share Posted October 18, 2014 Good to know...thanks guys. So I set my WD Red 6tb to run 2 preclears. Is there a way to get it to safely quit after the first successful preclear? I want to add my ssd to the array and then run the second pre-clear Quote Link to comment
garycase Posted October 18, 2014 Share Posted October 18, 2014 Good to know...thanks guys. So I set my WD Red 6tb to run 2 preclears. Is there a way to get it to safely quit after the first successful preclear? I want to add my ssd to the array and then run the second pre-clear You can abort the pre-clear with Ctrl-C It's always "safe" to do ... it just may (depending on where in the cycle you stopped it) not have a valid pre-clear signature on the drive, so you'll either need to do it again or just let UnRAID clear the disk when you add it. As for adding your SSD to the array ... you are, I presume, aware that you'll not likely see any major performance gains with this. Reason is simple: (a) For writes, the speed is still going to be limited by the speed of the parity drive; and (b) for reads, you're likely already limited by the speed of your network. Reads will, of course, start much quicker, due to the near-instantaneous "seek" times; but the network will quickly be the bottleneck. Quote Link to comment
Manticore Posted October 18, 2014 Share Posted October 18, 2014 As for adding your SSD to the array ... you are, I presume, aware that you'll not likely see any major performance gains with this. Reason is simple: (a) For writes, the speed is still going to be limited by the speed of the parity drive; and (b) for reads, you're likely already limited by the speed of your network. Reads will, of course, start much quicker, due to the near-instantaneous "seek" times; but the network will quickly be the bottleneck Are you saying the speeds are relation to the ssd being added to the array like another "storage" drive or that it applies to a cache drive also? Sorry, I was not clear with what I said. I want to add the ssd as a cache only drive, probably just for dockers/vm's etc. I have never had a cache drive before and never had any issues with speed so I'll stick with writing to parity directly. Quote Link to comment
garycase Posted October 18, 2014 Share Posted October 18, 2014 As for adding your SSD to the array ... you are, I presume, aware that you'll not likely see any major performance gains with this. Reason is simple: (a) For writes, the speed is still going to be limited by the speed of the parity drive; and (b) for reads, you're likely already limited by the speed of your network. Reads will, of course, start much quicker, due to the near-instantaneous "seek" times; but the network will quickly be the bottleneck Are you saying the speeds are relation to the ssd being added to the array like another "storage" drive or that it applies to a cache drive also? Sorry, I was not clear with what I said. I want to add the ssd as a cache only drive, probably just for dockers/vm's etc. I have never had a cache drive before and never had any issues with speed so I'll stick with writing to parity directly. I misunderstood your intent -- when you said you wanted to "add my SSD to the array" I assumed you meant add it to the array. A cache drive is not part of the array ... it's "outside" of the storage array ... so my comments don't apply. It will in fact provide some nice performance gains as a cache drive. No impact on reads; but a nice impact on writes, since they'll now be at the full speed of the cache drive, without being limited by parity. And for any local usage within the UnRAID box (i.e. Dockers, VM's, etc.) you'll get the full benefit of the SSD. Quote Link to comment
WeeboTech Posted October 25, 2014 Share Posted October 25, 2014 The extra day matters when you are in an emergency situation and have to preclear a drive for a rebuild. In my case, I usually do a 5-7 pass badblocks test before even doing a preclear. Then I stage the drive in the emergency repair zone. That can take almost a week. I'm sure many people will appreciate saving a day! Thanks bjp999! Would you care to share how you do this step by step for a new user ? Explain this also if you will. "Then I stage the drive in the emergency repair zone" As far as staging to an emergency zone. Well that's me just being a bit silly. I prepare the drive then sit it on top of the server.(or near the server). I'm cross posting the other text from another post about 4TB preclear times. I've done my own form of preclear on 2 4T drives. This consists of a smart long test (Which I do on every drive right away). I suppose this is equivalent to the pre-read. A 4 Pass pattern using badblocks 0xaa, 0x55, 0xff and finally 0x00. Then another final smart long test I suppose this is equivalent to the post-read. I did these both at the same time via screen on unRAID 5.0.6 under ESXI 5 on an HP Microserver. Times are below. Model Pre-read Badblocks Post-Read Total Seagate ST4000DX001 - 4TB 5400 RPM Hybrid 9:03 67:34 9:03 85 Hitachi HDN724040ALE640 - 4TB 7200 RPM 9:37 65:56 9:37 83 I've included the logs in case anyone is curious about using smartctl long test and the badblocks pattern test for burn in/clear Seagate ST4000DX001 - 4TB 5400 RPM Hybrid root@unRAID2:/boot/logs# date;badblocks -sv -w -c 512 -t0xaa -t0x55 -t0xff -t0x00 -o /boot/logs/ST4000DX001-1CE168_Z3014JCE.201410221000.badblocks /dev/sdd Wed Oct 22 10:15:47 EDT 2014 Checking for bad blocks in read-write mode From block 0 to 3907018583 Testing with pattern 0xaa: done Reading and comparing: done Testing with pattern 0x55: done Reading and comparing: done Testing with pattern 0xff: done Reading and comparing: done Testing with pattern 0x00: done Reading and comparing: 59:11:20 elapsed 86.75% done, 65:59:48 elapsed 86.75% done, 65:59:50 elapsed done Pass completed, 0 bad blocks found. root@unRAID2:/boot/logs# date Sat Oct 25 05:50:13 EDT 2014 Hitachi HDN724040ALE640 - 4TB 7200 RPM root@unRAID2:/boot/logs# date;badblocks -sv -w -c 512 -o /boot/logs/HGST_HDN724040ALE640_PK2334PCGTEKJB.201410221000.badblocks -t0xaa -t0x55 -t0xff -t0x00 /dev/sdc Wed Oct 22 10:13:56 EDT 2014 Checking for bad blocks in read-write mode From block 0 to 3907018583 Testing with pattern 0xaa: done Reading and comparing: done Testing with pattern 0x55: done Reading and comparing: done Testing with pattern 0xff: done Reading and comparing: done Testing with pattern 0x00: done Reading and comparing: 94% done, 59:13:05 elapsed done Pass completed, 0 bad blocks found. root@unRAID2:/boot/logs# date Sat Oct 25 04:09:51 EDT 2014 root@unRAID2:/boot/logs# smartctl -t long /dev/sdc smartctl 6.2 2013-07-26 r3841 [i686-linux-3.9.11p-unRAID] (local build) Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org === START OF OFFLINE IMMEDIATE AND SELF-TEST SECTION === Sending command: "Execute SMART Extended self-test routine immediately in off-line mode". Drive command "Execute SMART Extended self-test routine immediately in off-line mode" successful. Testing has begun. Please wait 577 minutes for test to complete. Test will complete after Sat Oct 25 13:51:50 2014 At the end of this all,. I'll use JoeL's preclear to put the signature on the drive. I prefer to use smartctl long test for the pre/post reads as it logs it into the drive itself. I prefer to use badblocks as I can do multiple passes with patterns. I'm not fond of a blind dd. I like the ability to use patterns walking one way or another then clear with the -t0x00 However, as JoeL mentions, badblocks alone cannot determine the drive's health. You need to look at the syslog for retries and the smart logs for changes. Quote Link to comment
JonathanM Posted October 25, 2014 Share Posted October 25, 2014 As far as staging to an emergency zone. Well that's me just being a bit silly. I prepare the drive then sit it on top of the server.(or near the server). That's fine as long as you are the only living thing larger than a couple pounds with access to the room. Those of us with pets / kids / etc can't just leave drives laying around. Quote Link to comment
ClunkClunk Posted October 25, 2014 Share Posted October 25, 2014 That's fine as long as you are the only living thing larger than a couple pounds with access to the room. Those of us with pets / kids / etc can't just leave drives laying around. Hah! My server lives in the closet in my 2 year old's room. It's locked so he can't get in. Quote Link to comment
garycase Posted October 25, 2014 Share Posted October 25, 2014 ... It's locked so he can't get in. ... Perhaps not while he's 2 -- but just wait a couple years Quote Link to comment
MartinQ Posted October 25, 2014 Share Posted October 25, 2014 Sorry about that. The skip_pct was something that I was working on to resume a preclear that was partially started. I accidentally posted the version I was testing with. EVERYONE, please download the version below. The original post has also been updated to download this corrected version. Here is the link to the download file Fast Preclear Can you put this link in the first post? It could get tricky to find in the future. Thanks! Quote Link to comment
garycase Posted October 25, 2014 Share Posted October 25, 2014 ... Can you put this link in the first post? It could get tricky to find in the future. Good idea. Not sure if Brian's in town or not, so I added the link to the first post in this thread. Quote Link to comment
c3 Posted October 25, 2014 Share Posted October 25, 2014 Sorry about that. The skip_pct was something that I was working on to resume a preclear that was partially started. I accidentally posted the version I was testing with. EVERYONE, please download the version below. The original post has also been updated to download this corrected version. Here is the link to the download file Fast Preclear Can you put this link in the first post? It could get tricky to find in the future. Thanks! It was already done. No need to duplicate. Quote Link to comment
grahamr Posted November 18, 2014 Share Posted November 18, 2014 Just as a point of reference - Using an old Dell laptop with WD 2G green drives, i had the following results - thats a pretty substantial time savings overall -for me on a 2tb drive its the difference between starting a preclear late at night, and being up and running the next evening vs the day after - so thanks! ========================================================================1.15 == invoked as: ./preclear_disk.sh -A /dev/sdc == WDCWD20EZRX-00D8PB0 WD-WMC4M3155598 == Disk /dev/sdc has been successfully precleared == with a starting sector of 64 == Ran 1 cycle == == Using :Read block size = 8388608 Bytes == Last Cycle's Pre Read Time : 5:33:15 (100 MB/s) == Last Cycle's Zeroing time : 5:11:48 (106 MB/s) == Last Cycle's Post Read Time : 11:57:17 (46 MB/s) == Last Cycle's Total Time : 22:43:21 == == Total Elapsed Time 22:43:21 ========================================================================1.15b == invoked as: ./preclear_bjp.sh -f -A /dev/sdc == WDCWD20EZRX-00D8PB0 WD-WMC4M3155598 == Disk /dev/sdc has been successfully precleared == with a starting sector of 64 == Ran 1 cycle == == Using :Read block size = 8388608 Bytes == Last Cycle's Pre Read Time : 5:33:08 (100 MB/s) == Last Cycle's Zeroing time : 5:04:06 (109 MB/s) == Last Cycle's Post Read Time : 6:20:01 (87 MB/s) == Last Cycle's Total Time : 16:58:15 == == Total Elapsed Time 16:58:15 Quote Link to comment
STxFarmer Posted December 16, 2014 Share Posted December 16, 2014 I use a HP N36L Microserver with 1gb memory for preclearing using 32bit unraid. Here are my results while preclearing 4 - 6tb WD Red drives at the same time. These results were of the first pass preclear using Joe's standard preclear and the second pass preclear using the faster preclear. ========================================================================1.15 == invoked as: ./preclear_disk.sh -r 65536 -w 65536 -b 2000 -A /dev/sdd == WDCWD60EFRX-68MYMN1 WD-WX11D9435LPF == Disk /dev/sdd has been successfully precleared == with a starting sector of 1 == Ran 1 cycle == == Using :Read block size = 65536 Bytes == Last Cycle's Pre Read Time : 17:16:54 (96 MB/s) == Last Cycle's Zeroing time : 13:55:03 (119 MB/s) == Last Cycle's Post Read Time : 40:40:12 (40 MB/s) == Last Cycle's Total Time : 71:53:11 == == Total Elapsed Time 71:53:12 == == Disk Start Temperature: 35C == == Current Disk Temperature: 36C, == ============================================================================ ========================================================================1.15b == invoked as: ./preclear_bjp.sh -f -r 65536 -w 65536 -b 2000 -A /dev/sdd == WDCWD60EFRX-68MYMN1 WD-WX11D9435LPF == Disk /dev/sdd has been successfully precleared == with a starting sector of 1 == Ran 1 cycle == == Using :Read block size = 65536 Bytes == Last Cycle's Pre Read Time : 17:19:37 (96 MB/s) == Last Cycle's Zeroing time : 13:55:42 (119 MB/s) == Last Cycle's Post Read Time : 22:31:47 (73 MB/s) == Last Cycle's Total Time : 53:48:07 == == Total Elapsed Time 53:48:07 == == Disk Start Temperature: 32C == == Current Disk Temperature: 36C, == ============================================================================ As you can see it cut the Post Read time by 18 hours or almost 50%. Quote Link to comment
FreeMan Posted December 23, 2014 Share Posted December 23, 2014 I got an error on line 1182: read_speed=`/boot/readvz if=$1 bs=$units count=$bcount skip=$skip memcnt=50 2>>/tmp/postread_errors$disk_basename | awk '{ print $8,$9 }'` because it's hard coded to /boot Yes, I read the readme.txt indicating they both needed to be in /boot, but I didn't really think you meant it... Could you change the "`/boot/readvz" to "`./readvz" to allow the preclear_bjp.sh and readvz to be in the same directory, whatever it may be? I'd give it a shot, but it already took 32 hours to get through the first 9 steps, and I don't want to go through that again just to find that it won't work. I've moved readvz to /boot for now... Quote Link to comment
WeeboTech Posted December 23, 2014 Share Posted December 23, 2014 If this were put at the top of the shell [ ${DEBUG:=0} -gt 0 ] && set -x -v B=${0##*/} # basename of program D=${0%%/$B} # dirname of program E=${B#*.} # Extension P=${B%.*} # strip off after last . character echo "D=${D} B=${B} P=${P} E=${E}" echo "readz=${D}/readz" Then you can determine how the shell was started and use ${D} to get the root directory of the shell's execution. Example: root@unRAID:/boot/bin# /boot/bin/rootofshelltest.sh D=/boot/bin B=rootofshelltest.sh P=rootofshelltest E=sh readz=/boot/bin/readz root@unRAID:/boot/bin# ./rootofshelltest.sh D=. B=rootofshelltest.sh P=rootofshelltest E=sh readz=./readz The requirement would then be to have readz installed in the same directory as the calling program. i.e. preclear. Having the requirement that it exists in the root of /boot is a bit strict. Quote Link to comment
FreeMan Posted December 26, 2014 Share Posted December 26, 2014 Great tip, Weebo! And the results of my preclear: Preclear Successful ... Total time 81:00:40 ... Pre-Read time 14:59:36 (74 MB/s) ... Zeroing time 17:25:10 (63 MB/s) ... Post-Read time 15:32:50 (71 MB/s) That was a 2-pass on a 4TB HGST NAS drive. That's ~48 hours for the 2nd pass, but only about 33 hours for the first pass. Lots of varying use on the server over the last 3 days, so I guess that could be enough to account for the difference. Also, it's plugged into a slow card - I'll rearrange all my SATA connections when I rearrange the drives here in the next few days. Since that's the 2nd & 3rd run (the first failed just before the pre-read), and there have been 0 pending sectors the whole time, I think I'll call this one good! Quote Link to comment
Fireball3 Posted December 31, 2014 Share Posted December 31, 2014 I'm preclearing 2 drives and just checked the progress. At the moment the post-read of one drive is running and it says: ./preclear_bjp.sh: line 1186: post_read_err: command not found Obviously it's running pretty fast as the screen updates approximately every .3s It seems there is something wrong here. Probably related to the elderly hardware? Motherboard: CPU Typ DualCore Intel Core 2 Duo E4300, 1800 MHz (9 x 200) Motherboard Name MSI MS-7293VP (FSC OEM) (2 PCI, 1 PCI-E x1, 1 PCI-E x16, 2 DDR2 DIMM, Audio, LAN, IEEE-1394) Motherboard Chipset VIA PT890 RAM 2 GB The run is over soon. I will post some news along with the preclear report next year ^^. ...6 minutes to go... Quote Link to comment
SSD Posted January 1, 2015 Author Share Posted January 1, 2015 Your drive, which should have been all zeroes after the writing phase, is not. Line 1186 says: post_read_err = "Y" it should say: post_read_err="Y" (Note there should not be spaces before and after the "=". Did I every say that I had shell scripts because the syntax is so incredibly unforgiving!) I will post an update, but in the mean time it would be an easy edit for you to make. This is not a good thing. It seems to imply that your system is not able to reliably write to the disk. Could be a disk issue (very unlikely, a memory issue, or a controller issue). A post read mismatch is very rare. Quote Link to comment
Fireball3 Posted January 1, 2015 Share Posted January 1, 2015 Happy New Year! Here is the screen after the preclear is finished. Unfortunately I switched consoles and now I'm not able to scroll upwards. Also attached the preclear logs. Did I every say that I had shell scripts because the syntax is so incredibly unforgiving! Oh, how well I know that feeling...! This is not a good thing. It seems to imply that your system is not able to reliably write to the disk. Could be a disk issue (very unlikely, a memory issue, or a controller issue). A post read mismatch is very rare. I can't remember if I ever precleared drives with this PC. Maybe the onboard controller has issues with big drives? But wouldn't there be write errors also (first phase of the script)? preclear_reports.zip Quote Link to comment
SSD Posted January 1, 2015 Author Share Posted January 1, 2015 The screen shot looks like it is having problems running readvz. This is the heart of the post-read speed improvemnt. Based on this I take back what I said about there being a write issue. Problem is probably just that your post read verify didn't run. Ensure you have the correct version (32 bit for version 5, 64 bit for version 6) of readvz in the boot of the flash drive (/boot). Quote Link to comment
Fireball3 Posted January 1, 2015 Share Posted January 1, 2015 You're probably right with the write error issue. After thinking further before I went to bed, I remember having cleared a drive but when I plugged it into my PC, windows still reported 2 partitions on that drive. I was in a hurry and did not investigate further but now it makes sense. There are probably no writes at all on the drives when clearing with this rig. I attached the syslog - perhaps you can find something? I don't know where the issue with readvz could be. I downloaded the zip in the OP 2 days ago. Edit: Probably found the issue. It's unraid 6b6 and I used the 32bit readvz. Will this solve the write issue also? Is it still necessary to use the LC_CTYPE? There is no info on this in the readme. syslog.txt Quote Link to comment
SSD Posted January 1, 2015 Author Share Posted January 1, 2015 You need to use the 64 bit version of readvz with unRAID version 6. I do think the disk read and disk zeroing steps completed. But the final step is the post read / verify. And although it was reading, it was not verifying. I would fix the line 1186 issue, put the 64-bit version of readvz in /boot, and re-preclear the disk (use -W option). You can confirm that the version runs by entering the command /boot/readvz using putty or on the unRAID console. It should return some usage instructions if properly installed. Quote Link to comment
Fireball3 Posted January 2, 2015 Share Posted January 2, 2015 OK, I updated the the 6b12 and copied the 64bit readvz into the root. Also removed the blanks in the script, as you recommended. /boot/readvz returned no error just something about missing arguments... I started a preclear with -f -W on both drives...patience. Quote Link to comment
Fireball3 Posted January 3, 2015 Share Posted January 3, 2015 All worked well so far. Both drives precleared successfully. Thanks for your support. I also found that the internal SATA ports of that board can't provide the full SATA transfer rate when 2 drives run simultaneusly. I will try a spare controller in the PCIe port or junk it. Quote Link to comment
superloopy Posted January 3, 2015 Share Posted January 3, 2015 I've just precleared a WD 6TB using the script and it's giving me an 'HPA?' entry against the drive when viewed via unmenu. A bit out of the loop here but is this anything to get concerned about? Drive has been allocated as parity for the array and has been precleared 3 cycles and parity checked against existing data drives which are still at 4TB. Am i likely to see any problems as i migrate drives to 6TB if there actually IS an hpa present or is this a known issue? Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.