November 9, 200817 yr I searched and saw this as a feature request but couldn't find if this was yet possible. Is there a way to clear and format a drive prior to putting it into the array so that it is ready to go. It would be nice if unraid cleared drives outside the array without stopping but.. It would even be nice if there were some way to do this from my windows machine prior to putting the drive in the unraid. Anyone have thoughts? Thanks.
November 9, 200817 yr It is not yet possible... I do have this next on my personal to-to list to put into the unmenu.awk Fron an e-mail conversation with Tom, he described to me what would be needed: The bytes in the MBR describing the starting sector and number of sectors for the first partition must be set exactly as if the disk was partitioned and formatted with a reiserfs (using the entire disk as a single partition) by unRAID. These bytes in the MBR vary depending on your specific disk geometry. (more accurately... geometry and disk size) This is the hard part of doing all this. Once this has been done, the byte in the MBR defining the partition type on partition 1 must be zeroed. (It is somewhere in the middle of the 16 bytes describing the first partition) Then, the MBR from byte address 0 up to the 16 bytes describing the first partition must be cleared. The bytes in the MBR describing partition2 and onward to the end of the disk must all be zeroed. (This is where it can take hours for large drives) last, the bytes at decimal 510 and 511 in the MBR must be st to 0x55 and 0xAA for a valid MBR to be detected. At this point, everything is zeroed except for the two bytes at the end of the MBR and the 16 bytes in the MBR describing the first partition. If you do all this correctly, it is my understanding that unRAID will not invoke the clearing process, but simply format it. It would be very difficult to do this on a windows machine. (not impossible, but very difficult.) Obviously, it is critical to do this to the CORRECT drive. This is NOT something you can undo if you accidentally invoked it on the wrong drive, and it would be much more likely to use it on the wrong drive on a windows machine, as it has no concept of the drives in the unRAID array, and those not assigned in the array. I'm going to add code for this to the section in unmenu.awk dealing with un-assigned drives (those not assigned on the unRAID devices page) At least it has a concept of assigned vs. unassigned drives and can limit the zeroing to those not assigned. Joe L.
November 12, 200817 yr It is not yet possible... I do have this next on my personal to-to list to put into the unmenu.awk Joe L. This is now possible... although because of the time it takes to clear a drive I did not put it into unmenu.awk, but left it as a stand-alone shell script. My script tries to protect you from yourself, but you can still clear a drive you did not intend, if careless. It will not clear a drive that is in the unRAID array or mounted anywhere. You must be able to figure out the correct /dev/hd? or /dev/sd? device name to invoke it. It must be run from a telnet session, and you must leave the telnet session open while it performs the clearing of the disk... (Or you can do it from the system console, and stay logged on while it clears the disk) While clearing, every 10 seconds, it gives you the status of its progress. Depending on the disk, and the speed of your array, clearing a disk can take many many hours. You invoke it by typing: preclear_disk.sh /dev/xxx Where xxx = hda, hdb, hdc, sda, sdb, sdc, etc. Any volunteers to give it a try? Joe L.
November 12, 200817 yr sure.... where is it? Oops... guess the attachment did not get attached... Remember, this will completely erase anything you have on the disk being cleared. As I said earlier, it will not clear a disk assigned to the unRAID array, it will not clear a disk that is mounted. It will clear an NTFS disk you moved to the unRAID array to copy data from, if it is just sitting there idle and not currently mounted. It does NOT care if there is a file-system... If not assigned to the unRAID array, and not currently mounted, it will be cleared. If you have two or more un-assigned disks in your computer and choose the wrong one to clear, goodbye data... You have been warned... When invoked it does show you the current partitioning and asks you respond with "Yes" to continue... Anything other than Capital "Y" and lower case "es" will result in it not proceeding with the clearing of the drive. It will not be changed in any way. If you are NOT sure you have the correct drive, respond with a "no" You have been warned.... I am not responsible for you using it, but it works in my tests here........ It is NOT supported by Tom at Lime-Technology in any way other than his telling me in an e-mail what I needed to do to pre-clear a drive. My script tries to protect you from yourself, but you can still clear a drive you did not intend, if careless. It will not clear a drive that is in the unRAID array or mounted anywhere. You must be able to figure out the /dev/hd? or /dev/sd? device name to invoke it. It must be run from a telnet session, and you must leave the telnet session open while it performs the clearing of the disk... (Or you can do it from the system console, and stay logged on while it clears the disk) While clearing, every 10 seconds, it gives you the status of its progress. Depending on the disk, and the speed of your array, clearing a disk can take many many hours. You invoke it by typing: preclear_disk.sh /dev/xxx Where xxx = hda, hdb, hdc, sda, sdb, sdc, etc. Joe L. Edit: 11/21/2008 added a bit more code to clear the MBR code area. The attachment has been updated. Edit: 11/22/2008 deleted the attachment as it is not partitioning exactly like unRAID, and therefore not detected as pre-cleared. New version to follow. See posts in this thread for details.. sfdisk reported different number of sectors than blockdev for a given disk... helps if you use the SAME numbers. Edit: zip file is attached to this post: http://lime-technology.com/forum/index.php?topic=2696.msg22662#msg22662 Edit: newest version is here: http://lime-technology.com/forum/index.php?topic=2817.msg23246#msg23246
November 13, 200817 yr That's a pretty long disclaimer (Insert sarcasm here)...So can I clear any drive I choose Thank you Joe the next time I add a drive I'll give it a try.
November 13, 200817 yr Just tried it out on my cache drive since I don't transfer a lot of files. Works great. I even tried to clear a disk (it had nothing on it) in the array and it wouldn't let me. Once again thanks Joe
November 13, 200817 yr damn, you guys are quick.... I just got a chance to download it... and it's time to hit the hay....
November 13, 200817 yr The clear went fine and then I added the drive and rebooted. I was left with 2 options either Start the array or Restore. I checked restore since the drive was already cleared. The array came online and a parity check has been started. It also said the disk was unformatted so I clicked format about 30 sec later it appeared to be finished and the parity check was still in progress. Should I have picked start instead? If left the impression that is would start the array and then clear and format the drive.
November 13, 200817 yr You should have pressed start. Restore will drop parity protection, re-establish the array based on the current device settings, and rebuild parity. It should almost never be pressed, unless you are creating a new array, or removing a disk from your array (no way around this one). Restore is a very bad button name. Chances are very high your parity rebuild will complete and you'll be fine. But start would have been the wise choice. If you were going to press restore, you would not have had to clear the drive!
November 13, 200817 yr The clear went fine and then I added the drive and rebooted. I was left with 2 options either Start the array or Restore. I checked restore since the drive was already cleared. The array came online and a parity check has been started. It also said the disk was unformatted so I clicked format about 30 sec later it appeared to be finished and the parity check was still in progress. Should I have picked start instead? If left the impression that is would start the array and then clear and format the drive. You are fortunate in that all your drives were working when you pressed the button labeled "restore" After clearing the drive, all you needed to do is stop the array, navigate to the "Devices" page, assign the cleared drive to a slot on your array, and then return to the "Main" page, check the checkbox under "Start" and press "Start." A "reboot" was not needed at all. Because the drive was pre-cleared, it would have skipped the lengthy clearing step and started the array. It would then allow you to format the new drive. As bjp999 said, the button labeled "restore" is very poorly labeled... It does not restore anything. It does rename the existing super.dat to super.old. This has the effect of forcing unRAID to immediately throw away any knowledge it had previously of your array's parity. It forces unRAID to build a new super.dat based on the currently assigned and working drives, and then to completely rebuild the parity drive. If you had pressed the "restore" button when you had a failed drive in the array you would have thrown away any chance of rebuilding the failed drive's data onto a new drive. Instead of protecting your data with parity, you would have thrown away the parity and lost your data that was on the failed drive. Always use the "Start" button when adding a drive (or replacing a failed drive). It maintains parity protection during the process of adding a drive. using the "Restore" button when adding a drive does not maintain parity protection and in fact, you lose parity protection until a full parity calc is completed. For the most part, unless you are removing a drive from the array, and do not intend to replace it shortly thereafter, you will NEVER use the "Restore" button. (there are exceptions to this, when the restore button is the only way to proceed, and you are not deleting a disk from the array, but they are not common, and in all cases, the remaining disks are all thought to be healthy ) Never use the "Restore" button if you have had a drive failure, unless you want to throw away all data on the failed drive. Joe L.
November 13, 200817 yr After clearing the drive, all you needed to do is stop the array, navigate to the "Devices" page, assign the cleared drive to a slot on your array, and then return to the "Main" page, check the checkbox under "Start" and press "Start." A "reboot" was not needed at all. Because the drive was pre-cleared, it would have skipped the lengthy clearing step and started the array. It would then allow you to format the new drive. After I stopped the array and assigned the drive to the array and went back to the main page I did try start but it only showed my original 14 disks. I tried refresh and even opened a new browser window and it didn't show up so I rebooted which led me to pick one option. Thanks again Joe
November 13, 200817 yr After clearing the drive, all you needed to do is stop the array, navigate to the "Devices" page, assign the cleared drive to a slot on your array, and then return to the "Main" page, check the checkbox under "Start" and press "Start." A "reboot" was not needed at all. Because the drive was pre-cleared, it would have skipped the lengthy clearing step and started the array. It would then allow you to format the new drive. After I stopped the array and assigned the drive to the array and went back to the main page I did try start but it only showed my original 14 disks. I tried refresh and even opened a new browser window and it didn't show up so I rebooted which led me to pick one option. Thanks again Joe Perhaps the reboot is needed to get emhttp to see the new drive. We will learn more as others try this. Thanks for the clairification. Regardless, use the "Start" button..., not the one labeled "restore" Joe L.
November 19, 200817 yr I just used the preclear disk script to clear a 1TB drive in my array. At the same time, I found an error in the script. I've since replaced the attachment. The error would have not cleared the cylinder/head/sector of the correct disk in your array, but instead it would have tried to do it on /dev/hdl (The disk I was using in my tests) The attempt would have failed if /dev/hdl was already a disk in your array, as sfdisk would not have proceeded if the disk was in use. I've since fixed the hard-coded device name in the attachment in the prior post. I'm going to add a bit of logic to verify the first 512 bytes were cleared correctly and then post a new, slightly improved version. Joe L.
November 21, 200817 yr Author I am certain these are two stupid questions but... 1. How do I install this shell script? Is there more to it than copying it on to my flash drive? 2. Is there a command to see what drive assignments are (ie. sda, sdb, sdc...)? Thanks.
November 21, 200817 yr I am certain these are two stupid questions but... 1. How do I install this shell script? Is there more to it than copying it on to my flash drive? Unzip it at the top folder on the flash drive.... It is as easy as that. (download the newest version, I've made improvements) When you log into the server via telnet, you must "cd" (change directoy) to that same folder. The command is: cd /boot then to invoke it preclear_disk.sh /dev/aaa where aaa is sda, sdb, hda, hdb, etc on your array. Remember, you cannot pre-clear a disk after it is in use, so it must be after your physically install it, but before you add it to the array. 2. Is there a command to see what drive assignments are (ie. sda, sdb, sdc...)? Easiest is ls -l /dev/disk/by-id Joe L.
November 21, 200817 yr I added a bit more code to the preclear_disk.sh script clear the MBR code area. The attachment in the prior post has been updated. Please download the newest version... (Odds are earlier versions would not have completely cleared all the bytes needed, and the disk would have not been properly recognized as a pre-cleared disk by unraid.) Joe L.
November 21, 200817 yr Author Thank you Joe L!!!! As always your help is unbelievably fast and much appreciated.
November 21, 200817 yr Thank you Joe L!!!! As always your help is unbelievably fast and much appreciated. I added some code to verify the pre-clear worked as expected. I just started a clear of my new 1TB disk. I'll let you know in about 4 or 5 hours how it went. If it is successful, I'll replace the version in the attachment above with it. (test cycles were Sooooooo much faster when I was testing with an 8Gig drive ) Joe L.
November 21, 200817 yr Thank you Joe L!!!! As always your help is unbelievably fast and much appreciated. I added some code to verify the pre-clear worked as expected. I just started a clear of my new 1TB disk. I'll let you know in about 4 or 5 hours how it went. If it is successful, I'll replace the version in the attachment above with it. (test cycles were Sooooooo much faster when I was testing with an 8Gig drive ) Joe L. The test run completed, and everything looks good. Unfortunately, when I went to add the disk to the array it still started the process of clearing it. I don't have the signature exactly right. Looking at its partitioning vs. mine, it has a different size. My partition started at sector 63 and was 1953520002 sectors long. unRAID started at sector 63 and shows the partition as 1953525105 sectors long. Because it calculated it differently, it did not consider the disk pre-cleared. Now... to figure out where I did the math wrong... First.. get the geometry: root@Tower:/boot# sfdisk -g /dev/hdl /dev/hdl: 121601 cylinders, 255 heads, 63 sectors/track my math says 121601 * 255 * 63 = 1953520065 sectors If we start at sector 63, and have a partition length of 1953520002 we should use up all the space. unRAID seems to say the length of the partition is 1953525105. That seems bigger than my physical drive... Either I am doing something wrong, or... I don't understand how this all works... The difference is 5040 sectors. The plot thickins... root@Tower:/boot# blockdev --getsz /dev/hdl 1953525168 If I subtract 63 from that number, I get the same value as unRAID for the length... But I wonder... which is correct... I'll get a third opinion... root@Tower:/boot# smartctl -a -d ata /dev/hdl smartctl version 5.38 [i486-slackware-linux-gnu] Copyright © 2002-8 Bruce Allen Home page is http://smartmontools.sourceforge.net/ === START OF INFORMATION SECTION === Device Model: WDC WD10EACS-00D6B1 Serial Number: WD-WCAU44206983 Firmware Version: 01.01A01 User Capacity: 1,000,204,886,016 bytes Device is: Not in smartctl database [for details use: -P showall] ATA Version is: 8 ATA Standard is: Exact ATA specification draft version not indicated Local Time is: Fri Nov 21 14:56:19 2008 EST SMART support is: Available - device has SMART capability. SMART support is: Enabled looks like 1,000,204,886,016 bytes when I divide by 512, I get 1953525168.... Hummmm... looks like sfdisk is lying to me... I think I trust the smart report... Time to figure out how to ask sfdisk to give me the correct geometry. (you would not think it was this difficult, would you?) I'll investigate a tiny bit more. In the interim, I have a really neat tool to exercise a new hard disk before adding it to your array. ;D Joe L.
November 21, 200817 yr It is getting interesting... root@Tower:/boot# fdisk -l /dev/hdl Disk /dev/hdl: 1000.2 GB, 1000204886016 bytes 1 heads, 63 sectors/track, 31008336 cylinders Units = cylinders of 63 * 512 = 32256 bytes Device Boot Start End Blocks Id System /dev/hdl1 2 31008336 976762552+ 83 Linux At least the size agrees with smartctl. It uses different "notation" for start and end (It uses cylinders, not sectors). Joe L.
November 21, 200817 yr My array is half way through "clearing" the new drive... when it is done, I'll reset everything and remove the new drive from the array. It will be tomorrow before I can post the tested replacement script to preclear. (My logic was correct... the information returned by sfdisk is just plain wrong, erroring on the side of reporting less space than actually available physically. ) Oh well... if you grabbed an earlier copy of this utility, it will do no harm, it will exercise your new disk, but when it finishes with the clearing process it will not have the same partition size as expected, and therefore will not be recognized as "pre-cleared" Hopefully, tomorrow I'll post an updated version that does set the partition size the same as unRAID and will be recognized as pre-cleared. (the updated version is coded, but I need to wait till my array finishes clearing the new drive before I can run another test cycle) Joe L. Edit: I deleted the attached "zip" file of the preclear_disk.sh utility in the prior post in this thread. I'll replace it tomorrow, after I do a test with the corrected version. Edit: zip file is attached to this post: http://lime-technology.com/forum/index.php?topic=2696.msg22662#msg22662 Joe L.
November 21, 200817 yr I just tried my "corrected version" or preclear_disk.sh... with the "correct" number of sectors for the partition... Guess what??? sfdisk complains the partition is too big... ******** Partitioning disk ******************************* Checking that no-one is using this disk right now ... OK Disk /dev/hdl: 121601 cylinders, 255 heads, 63 sectors/track Old situation: Warning: The partition table looks like it was made for C/H/S=*/1/0 (instead of 121601/255/63). For this listing I'll assume that geometry. Units = sectors of 512 bytes, counting from 0 Device Boot Start End #sectors Id System /dev/hdl1 63 1953525167 1953525105 83 Linux /dev/hdl2 0 - 0 0 Empty /dev/hdl3 0 - 0 0 Empty /dev/hdl4 0 - 0 0 Empty Warning: given size (1953525105) exceeds max allowable size (1953520002) sfdisk: bad input I'll try a bigger hammer. (to force the size to fit) Joe L.
November 21, 200817 yr I'll try a bigger hammer. (to force the size to fit) Joe L. The bigger hammer worked (the --force option) sfdisk complains the partition extends beyond the end of the disk, but we know better. It is clearing once more.... It will be 4 or 5 more hours before it is done... but I did confirm the partitioning is the same as unRAID would have made it. Hopefully, in about 4 or 5 hours, it will be recognized as pre-cleared. Joe L.
November 22, 200817 yr Finally... the new disk was cleared... but not without slight mishap. It turns out that you can get into a deadlock situation if you are not careful. You will want to shut down "unmenu, or at least get off of its "Main" page. It uses "hdparm" to get information about the disks. If you are clearing and partitioning a disk at the same time both programs might try to read the same disk at the same time, and both will wait (forever) for the other to finish. Since both wait (forever) this will take a really long time..... Today I had a similar situation and I ended up using "control-Z" to suspend the foreground task and let me regain control... Last night I ended up rebooting my server to get control back. I'll do another experiment or two before I publish the preclear_disk.sh script. The good news is that my pre-cleared disk was brought into the array, skipping the lengthy clearing step (Really Nice) When I started the array it then said it was un-formatted (true) and gave me the option to format it. At that point, my array was on-line, so my movie collection was available throughout the process, all except when I stopped the array to add the disk on the "devices" page and re-start it (after checking the "I'm sure button under the "Start" button.) One or two more test runs and I'll post the updated version. (I'm still wondering why sfdisk shows a smaller number of sectors than fdisk, hdparm, or smartctl, but it really does not matter at this point.) Joe L.
Archived
This topic is now archived and is closed to further replies.