September 12, 200916 yr I have four drives in my array: 2x 1.5TB (one parity), 1x 300GB, and an old 160GB that I really only stuck in there for the exercise. I have just passed 1.2TB of data, so unRAID's high-water algorithm has just started to put data on the 300GB drive. I've bought a third 1.5TB drive, and I would like to use it to replace the 160GB while it's still empty I've pre-cleared the new drive, and I'm currently running a parity check on the array. Once the parity check is complete, I plan on shutting down the array, removing the 160 and putting the new 1.5 in its slot, and restarting the array. What I'd like to do is minimize the amount of time I'm without parity protection. Since the old drive is empty, and the new one is pre-cleared, does the existing parity remain valid, and if so, how do I tell unRAID to trust the parity and just bring the array online?
September 13, 200916 yr I have four drives in my array: 2x 1.5TB (one parity), 1x 300GB, and an old 160GB that I really only stuck in there for the exercise. I have just passed 1.2TB of data, so unRAID's high-water algorithm has just started to put data on the 300GB drive. I've bought a third 1.5TB drive, and I would like to use it to replace the 160GB while it's still empty I've pre-cleared the new drive, and I'm currently running a parity check on the array. Once the parity check is complete, I plan on shutting down the array, removing the 160 and putting the new 1.5 in its slot, and restarting the array. What I'd like to do is minimize the amount of time I'm without parity protection. Since the old drive is empty, and the new one is pre-cleared, does the existing parity remain valid, and if so, how do I tell unRAID to trust the parity and just bring the array online? The existing drive is formatted... the formatting causes it to not be "empty" There is a way to remove it and keep parity while you go through the process. It is described here. In effect you need to write zeros to it to "clear" it.. then you can remove it once it is all zeros and ask the array to "trust" the parity... since the zeros will not change parity. Then, you can add the pre-cleared drive as usual. the procedure is described here http://lime-technology.com/forum/index.php?topic=2591.msg20919#msg20919 Just remember, you must issue the mdcmd set invalidslot 99 after using "dd" to fill the drive to be removed with zeros and then after pressing the "restore" button, but BEFORE starting the array by pressing the "Start" button. Joe L.
September 13, 200916 yr But ONLY if the 160G drive was precleared before adding it to the array!! I don't think that matters if the 160 Gig drive was originally precleared, as if it is currently in the array it already is part of parity (with its formatted file-system, and whatever other garbage in the un-allocated blocks if not cleared in one way or another) AND we will be clearing it as part of the steps below. It is critical that the 1.5T drive you are swapping in be pre-cleared.... The steps are 1. un-mount the 160 Gig drive 2. Write zeros to the entire 160 Giig drive using "dd" (this will set parity as to handle all zeros in the replacement drive) 3. un-assign the 160 Gig drive 4. Assign the PRE-CLEARED 1.5TB drive to the slot the 160 was in previously. 5. Press "Restore" to save a new disk configuration BUT DO NOT START THE ARRAY... Do NOT press "Start" until step 7. 6. Log in and type: cd mdcmd set invalidslot 99 Make sure it responds the command was accepted OK 7. Press "Start" to start the array. 8. Press "Format" to format the new 1.5TB drive.
September 13, 200916 yr Author Thanks for that. I knew I'd seen a magic command mentioned somewhere, but damned if I could find it. You're right about it being a slow old process though - I fired off the dd command over sixteen hours ago, and it's still running. That's less than 10GB per hour!
September 14, 200916 yr Thanks for that. I knew I'd seen a magic command mentioned somewhere, but damned if I could find it. You're right about it being a slow old process though - I fired off the dd command over sixteen hours ago, and it's still running. That's less than 10GB per hour! That's close to what is expected actually... Don't forget, for every block the "dd" command is writing, it is first reading the block, and the equivalent parity block, then doing the xor math to calculate the new parity based on old and new data and old parity, then writing both the parity block and the data block, after waiting for the disk platters to spin a full revolution for the disk heads to get back to the same sector. You are at the mercy of basic physics...For each block on the disk, two reads, two writes, and each disk must spin at least once between the read and the write of the same sector. Even with 7200 RPM drives, it is slow... and I can guarantee the older drives are NOT moving at 7200 RPM... but more likely half that speed. Did you specify the block size on the "dd" command? Like this: dd if=/dev/zero bs=2048k of=/dev/md? Without it, on my older drive, using the default 512 byte block size I found writing at about half the speed as with setting it to 2048k. On the other hand, if you look at the illustration on the link I gave earlier, their "dd" speed was at about 400 kB/s.. It took over 10 hours to clear a 20 Gig drive. Just remember, your parity protection is still in place during this... you are trading this time for that otherwise needed to go without parity protection. Joe L.
September 14, 200916 yr Author Did you specify the block size on the "dd" command? Just remember, your parity protection is still in place during this... you are trading this time for that otherwise needed to go without parity protection. Joe L. No, I wasn't aware of the blocksize parameter - I simply copied the command verbatim from the example you cited. Right now, 18 hours in, it's at 73k writes. If the default blocksize is indeed 512k then it's done around 36GB, or a little less than 25%. I realize that this is the price one pays for continuous parity protection, which is exactly what I wanted, but at the same time I'd still like to get this new drive in there this week. Perhaps I'll kill the dd process and fire it off again with a larger blocksize; even if it has to do the first 25% again, at double the rate it should still complete about a day earlier!
September 14, 200916 yr Did you specify the block size on the "dd" command? Just remember, your parity protection is still in place during this... you are trading this time for that otherwise needed to go without parity protection. Joe L. No, I wasn't aware of the blocksize parameter - I simply copied the command verbatim from the example you cited. Right now, 18 hours in, it's at 73k writes. If the default blocksize is indeed 512k then it's done around 36GB, or a little less than 25%. I realize that this is the price one pays for continuous parity protection, which is exactly what I wanted, but at the same time I'd still like to get this new drive in there this week. Perhaps I'll kill the dd process and fire it off again with a larger blocksize; even if it has to do the first 25% again, at double the rate it should still complete about a day earlier! Where are you seeing 73k writes? Are you looking at the unRAID management console? You can get exactly the amount written, without stopping the "dd" command. I'm assuming it is running in a telnet session, or on the system console. Whe you can do is open a second telnet session, or, if on the system console,press "ALT-F2" to switch to the second console screen, log in again and then 1. determine the process ID of the "dd" command by typing: pidof dd the output is the process ID of the "dd" command. (hopefully, you only have one running) then, you can ask the "dd" process to tell you how far it has gotten... (it will not stop it, but just tell you the status) To do that, you send the "dd" process a special signal. kill -USR1 nnnnn where nnnnn = the process ID of the "dd" command returned by the pidof command. If the pidof command returned the number 4053, the kill command would then be kill -USR1 4053 Type this on the second screen you logged in on. On the screen with the running "dd" command, it will display its progress so far. If you switched system consoles, use ALT-F1 to get back to the first "virtual" console. (You have 6, ALT-F1 through ALT-F6) then, let's assume the "dd" has gotten 100 Meg done... we can stop it (by typing Control-C on the screen running it, and type dd if=/dev/zero bs=1000000 seek=99 of=/dev/md? It will seek past the first 99 Meg, and then start writing from there using the 1meg block siize. ( I purposely overlapped the zeroing to make sure I did not miss any ) Adjust the numbers as needed of course, but you probably get the idea. Joe L.
September 14, 200916 yr Author Where are you seeing 73k writes? Are you looking at the unRAID management console? Yes, I had reset the statistics prior to firing off the dd command. I sent the kill command per your suggestion, and it confirmed it had written almost 29GB: 56203537+0 records in 56203537+0 records out 28776210944 bytes (29 GB) copied, 70179.6 s, 410 kB/s (It also repeated that information when I terminated the command, but your advice was a good safety-check.) Trying to decide what blocksize to specify, I did some googling and wound up confusing myself still further! From what I'm reading, the seek or skip parameters are in terms of blocks, and thus are directly related to the bs parameter. Not wanting to risk missing any part of the drive, I simply went with dd if=/dev/zero bs=4k of=/dev/md2 to start again from the beginning. I don't think I'll even bother stopping it again to specify a start offset even if I find a definitive answer, since it's now humming along at something like 20x the original speed: 1490386+0 records in 1490386+0 records out 6104621056 bytes (6.1 GB) copied, 734.133 s, 8.3 MB/s That's 4% in about 12 minutes; should be all done in about five hours at that rate. Thanks for the tip!
September 14, 200916 yr Where are you seeing 73k writes? Are you looking at the unRAID management console? Yes, I had reset the statistics prior to firing off the dd command. I sent the kill command per your suggestion, and it confirmed it had written almost 29GB: 56203537+0 records in 56203537+0 records out 28776210944 bytes (29 GB) copied, 70179.6 s, 410 kB/s (It also repeated that information when I terminated the command, but your advice was a good safety-check.) Trying to decide what blocksize to specify, I did some googling and wound up confusing myself still further! From what I'm reading, the seek or skip parameters are in terms of blocks, and thus are directly related to the bs parameter. Not wanting to risk missing any part of the drive, I simply went with dd if=/dev/zero bs=4k of=/dev/md2 to start again from the beginning. I don't think I'll even bother stopping it again to specify a start offset even if I find a definitive answer, since it's now humming along at something like 20x the original speed: 1490386+0 records in 1490386+0 records out 6104621056 bytes (6.1 GB) copied, 734.133 s, 8.3 MB/s That's 4% in about 12 minutes; should be all done in about five hours at that rate. Thanks for the tip! You are right... my example should have said "seek=99" to seek 99 blocksize of 1M. Oh well. I'll edit my previous post for the next person reading what to do... Glad you are going much faster. Joe L.
September 14, 200916 yr As an FYI, I used 2048k as the block size when I wrote the preclear_disk.sh script to zero the disks. Your use of 4k is smaller than what I would have tried, but as long as the performance is good, go with it. It really does not matter what you used if performance is better.... and it is now writing at 8.3MB/s instead of 410 kB/s. As you said, about 20 times faster.
September 14, 200916 yr Author I decided to try a 4k blocksize after my googling turned up one guy's testing that seemed to show no improvement over 4k. I figured I'd try it, and could always stop and restart the command with a larger blocksize if the performance was still poor, but since it took off like a rocket I just left well enough alone. The clear completed overnight in a little under five hours: dd: writing `/dev/md2': No space left on device 39072719+0 records in 39072718+0 records out 160041852928 bytes (160 GB) copied, 17731.8 s, 9.0 MB/s Per your instructions, this morning I stopped the array, switched out the old 160 for the new (pre-cleared) 1.5 in slot 2, issued the mdcmd command, and restarted the array. It's now checking parity at around 70MB/s, with an ETA of a little over five hours. I'm going to hold off formatting the new drive until the parity check is complete. One final question: If something was to happen to the new drive at any point while it's still unformatted, could I simply swap the old 160 back into that slot by repeating the same process? (i.e., would unRAID allow the same kludge if an unformatted drive failed while the array was started, vs if it was removed while it was stopped?)
September 14, 200916 yr I decided to try a 4k blocksize after my googling turned up one guy's testing that seemed to show no improvement over 4k. I figured I'd try it, and could always stop and restart the command with a larger blocksize if the performance was still poor, but since it took off like a rocket I just left well enough alone. I would have done the same... I'm going to hold off formatting the new drive until the parity check is complete. It is your choice... To unRAID, bits are bits... you can format now, or later... won't make a difference at all. You are protected by parity now, regardless of what happens. Waiting until the parity check finishes just delays the availability of the extra space. Since you said you needed it by the end of the week, looks like you are ahead of schedule. One final question: If something was to happen to the new drive at any point while it's still unformatted, could I simply swap the old 160 back into that slot by repeating the same process? (i.e., would unRAID allow the same kludge if an unformatted drive failed while the array was started, vs if it was removed while it was stopped?) We would need to know more... if the array is stopped, then any changes made to any data drive would not be reflected in parity, and it would result in errors then next time it is checked. (The errors would be corrected by the "check" process, but you would not be able to rely on parity to reconstruct other drives) As long as your current parity check is showing no errors, and the odds are it will not, you should be fine. If a different drive were to fail right now.. we just treat it exactly as normal... (replacing it as soon as possible) It does not matter that the new drive does not yet have a file-system on it. It is still part of parity (all zeros, but part of the protected array, as expected in the config/super.dat file) Unless you have a really good reason, don't swap back in the old/smaller drive and try to reverse the steps... at least not without a lot of thought. Joe L.
Archived
This topic is now archived and is closed to further replies.