January 23, 201016 yr I have a 2TB drive that was used in a linux system as a data drive that is formatted in the ResierFS. It has data that I would prefer to not lose. Can I add this to an existing array with the data?
January 23, 201016 yr It depends on how it was partitioned. If it has a single partition, and that partition starts on the first full cylinder (usually sector 63), and that single partition ends on the last possible sector on the disk, and it is a valid reiserfs, then unRAID will probably recognize it and use it, and not clear it as a drive it does not recognize. You can test its partitioning with a script I attached to this post: http://lime-technology.com/forum/index.php?topic=5072.msg47122#msg47122 I had written the script to help a user whose partitioning had been clobbered by a HPA (Host-Protected-Area) that made his disk look to only be 33Meg in size. It can fix the partitioning to be what unRAID expects, but if your file-system was not where unRAID expected it, re-writing the partition table would do more damage than not. In his case, the HPA was added by his Gigabyte BIOS going wild during a power hit. As long as you do not use the "-p" option the script will not write to the disk being given as an argument and not modify it in any way. It will let you know if the disk is partitioned in a way unRAID would be able to be use. If it is partitioned as needed, then it is simply a matter of assigning the disk to a slot in the array and pressing the "Restore" button to cause unRAID to save a new disk configuration. ("restore" does not restore data, instead it sets a new initial disk configuration in the array. It immediately invalidates parity and when you subsequently start the array by pressing "Start" a full parity calculation will begin.) You will not be protected from a disk failure until the parity calc is complete. If the script says the disk is not partitioned as needed, post the output here. Just remember, do not use the "-p" option. Joe L.
January 24, 201016 yr Author I ran the script and received the following response from the script. It looks close but says not partitioned properly. root@Tower:/mnt/user/Music# ./unraid_partition_disk.sh /dev/sdb ######################################################################## Device Model: ST32000542AS Serial Number: 6XW00XD4 Firmware Version: CC32 User Capacity: 2,000,398,934,016 bytes Disk /dev/sdb: 2000.3 GB, 2000398934016 bytes 255 heads, 63 sectors/track, 243201 cylinders, total 3907029168 sectors Units = sectors of 1 * 512 = 512 bytes Disk identifier: 0x000ab883 Device Boot Start End Blocks Id System /dev/sdb1 63 3907024064 1953512001 83 Linux ######################################################################## ============================================================================ == Disk /dev/sdb is NOT partitioned for unRAID properly. == expected start = 63, actual start = 63 == expected size = 3907029105, actual size = 3907024002 ============================================================================ root@Tower:/mnt/user/Music#
January 24, 201016 yr I ran the script and received the following response from the script. It looks close but says not partitioned properly. root@Tower:/mnt/user/Music# ./unraid_partition_disk.sh /dev/sdb ######################################################################## Device Model: ST32000542AS Serial Number: 6XW00XD4 Firmware Version: CC32 User Capacity: 2,000,398,934,016 bytes Disk /dev/sdb: 2000.3 GB, 2000398934016 bytes 255 heads, 63 sectors/track, 243201 cylinders, total 3907029168 sectors Units = sectors of 1 * 512 = 512 bytes Disk identifier: 0x000ab883 Device Boot Start End Blocks Id System /dev/sdb1 63 3907024064 1953512001 83 Linux ######################################################################## ============================================================================ == Disk /dev/sdb is NOT partitioned for unRAID properly. == expected start = 63, actual start = 63 == expected size = 3907029105, actual size = 3907024002 ============================================================================ root@Tower:/mnt/user/Music# If you are a tiny bit adventurous, you can fix the partitioning, moving the end of the partition to use up the un-allocated space at the end of your drive (by using the "-p" option to my script) and then run resize_reiserfs /dev/sdb1 which will resize the reiser file system to be able to use all the available space in the partition. The manual page for resize_reiserfs is here: http://linux.die.net/man/8/resize_reiserfs It mentions the starting sector must not change in the newly sized partition (and yours would not, as it is already correct starting at sector 63. The script would grow the existing partition to use the 5103 unallocated sectors at the end of the disk.) It mentions using cfdisk to resize the partition, but my shell script will do it much easier (using the -p option), and it will be exactly as unRAID needs it. There are risks... but they are minimal. You need to decide if you care to go forward. At least you know, the disk, as is, will not be recognized as an unRAID disk and will be cleared if you attempt to assign it to the array. Re-partitioned, it will be recognized. (I say re-partitioned, but it is just making the existing partition table larger to extend the end of the partition to the end of the disk. It does not change your file system nor touch it) Actually, I honestly don't think you need to use the resize_reiserfs program to resize the file-system in the larger partition at all if you do not want to. If you don't use it, you just will leave the last 5103 sectors unused. The 5103 sectors are probably unused because the program that did the partitioning made a partition to contain whole cylinders. The 5103 would represent a partial cylinder and represents about 2 meg of space. However... current drives don't use cylinder/sector/head geometry internally. The just use a linear set of sectors. The whole concept of a constant number of sectors per cylinder has probably been replaced by a variable geometry for many many years. Joe L.
January 24, 201016 yr Author I ran the script with the -p option and after the modification was done to the drive, the UNRAID server accepted the drive and the data was in tact. Thanks.
January 24, 201016 yr I ran the script with the -p option and after the modification was done to the drive, the UNRAID server accepted the drive and the data was in tact. Thanks. Did you need to press the button labeled "restore" for it to accept it... Since hardly anybody has done what you've done can you take a few minutes to describe the process of adding it to your array? Obviously... Stop the array Assign the drive to a slot on the "devices" page And then....back on the main page ? It sounds like you did not resize the reiserFS. Or did you? When you re-started the array, did it calculate parity again based on the added drive? Joe L.
January 24, 201016 yr Author I ran the script with the -p option and after the modification was done to the drive, the UNRAID server accepted the drive and the data was in tact. Thanks. Did you need to press the button labeled "restore" for it to accept it... Since hardly anybody has done what you've done can you take a few minutes to describe the process of adding it to your array? Obviously... Stop the array Assign the drive to a slot on the "devices" page And then....back on the main page ? It sounds like you did not resize the reiserFS. Or did you? When you re-started the array, did it calculate parity again based on the added drive? Joe L. This is what I did. 1. I ran your script unraid_partition_disk.sh without the -p option. The script reported that the drive was the incorrect size. 2. I then ran your script with the -p option. The script queried for me to resize and I said "Yes". This extended the size to what UNraid expected. 3. After that I ran the script without the -p option and the script reported that unraid would accept. 4. I then assigned the drive to an empty slot on the devices page. 5. I then pressed the restore button for the array to accept the drive as a new configuration. 6. I then started the array, unraid accepted the configuration and started the parity sync. 6. The parity-sync is now in progress. My array is unprotected until the completion of the parity sync. I hope that helps. I never ended up using a resize_reiserfs script. The only script I used was the unraid_partition_disk.sh that you provided.
January 24, 201016 yr Thanks... That was exactly what I requested. (and written very clearly too) Glad I was able to help. It is only because your existing partition was already starting on sector 63 that it was so easy. I doubt you'll ever miss the couple of meg that was not part of your old file-system at the end of the disk. You might not have even known it was un-allocated in your old system. Might I ask what you used when you originally partitioned that disk? did you use mkreiserfs? (Just wondering what left part of the disk un-allocated) As I said earlier, I wrote the shell script to examine/fix the MBR and partition table when one user's BIOS added an HPA making their disk look like it was only 33Meg in size. We removed the HPA but then were faced with fixing the partition table. Rather than trying to remotely describe/walk somebody through using fdisk, sfdisk, or cfdisk, I wrote the the unraid_partition_disk.sh script. It seemed far easier for him, and for me . Glad you were able to find a second use for it. Joe L.
January 25, 201016 yr Author The drive was originally formatted in a linux computer running Ubuntu 9.10. I am fairly sure I used the mkreiserfs command. I might have used a graphical tool, I don' remember specifically. I normally use command line, so most likely it was mkreiserfs. I do not use any options when formatting the disk so the command was something like mkreiserfs /dev/sda1 I have a couple more drives that I plan on migrating over eventually. I will post any issues that I have so others can benefit. If I would have planned better, I would have put a few more details in the procedure I used (including screenshots), but I didn't think about it when I was performing the operation so the procedure in my last post was from memory (but still accurate.) Thanks again for the script. I probably would have ended up copying everything over manually, so you definitely saved me some time. BTW, the unraid server is the new RB-1200 model. I have been using unraid just for a short amount of time, but I am hooked. TZ
August 17, 201213 yr Hey Joe L. I want to start a new unraid server with my two NTFS formated HDDs. One is 3TB, the other 2TB. Lets say both are little less than half full. Would it work if I follow the following procedure? 1. Boot my Desktop Computer with Ubuntu. 2. For each drive: 2.1 Take the empty half of it and format it exactly as you said unraid expects it. 2.2 Copy the contents from the NTFS part to the new Reiser part. 2.3 Extend the partition to the full size of the disk. 2.4 Run your script to verify the partition table is correct. 3. Buy a new 3TB drive and put it into the server, mark as parity. 4. Put the two old drive with the existing data on it in the server and rebuild parity I realy don't want to copy 4 TB+ from HDD to HDD if I can avoid it. I wouldn't even know where to put those files... Thank you!
August 17, 201213 yr Hey Joe L. I want to start a new unraid server with my two NTFS formated HDDs. One is 3TB, the other 2TB. Lets say both are little less than half full. Would it work if I follow the following procedure? 1. Boot my Desktop Computer with Ubuntu. 2. For each drive: 2.1 Take the empty half of it and format it exactly as you said unraid expects it. 2.2 Copy the contents from the NTFS part to the new Reiser part. 2.3 Extend the partition to the full size of the disk. 2.4 Run your script to verify the partition table is correct. 3. Buy a new 3TB drive and put it into the server, mark as parity. 4. Put the two old drive with the existing data on it in the server and rebuild parity I realy don't want to copy 4 TB+ from HDD to HDD if I can avoid it. I wouldn't even know where to put those files... Thank you! no disrespect intended, but if you have to ask this question, the odds of your success is almost non-existant. If I were you, I would (on your window's system) copy ALL the contents of the two NTFS drives onto one of them. It would be easiest if the disk you empty is the 3TB drive. If not, you are SOL and need an additional 3TB drive. Then, once you have an empty disk to use, start your unRAID array with it as the data drive. It is strongly suggested you pre-clear all the disks before adding them to the unRAID server. It is not mandatory, but one out about 5 drives has issues when brand new. (do you feel lucky?) Have you gotten a SMART report on your existing NTFS drives? Windows just mangles your data and keeps you in the dark about the disk health... Are there any sectors pending re-allocation? or already re-allocated? Purchase a new 3TB drive, pre-clear it, and add it as your parity drive. Then, migrate the data from your current 3TB NTFS drive to the unRAID array. Do this over the LAN, using something like teracopy that can perform a verification checksum to ensure the transfer was as expected. Lastly, once you know the array is on-line and problem free,(wait a week or so), and parity protected, use the pre-clear script to clear the 2TB drive prior to adding it as the second data drive. I'd bet your original proposal is near impossible. unRAID expects only 1 partition, and it must be EXACTLY as it would have defined it. Odds are VERY high your existing NTFS partitions already use the disk space needed, and the "free" space is not really free, but part of its file-system. Basically, you have no choice. You WILL be copying 4TB of data... even if your scheme were possible, it would be copying 4TB from one section of a disk to another sector on the same disk, and that, with all the head-seeking involved, would be VERY slow. Joe L.
August 17, 201213 yr thank you very much for your answer, very helpfull in deed! I understand that copying the files is unavoidable. I in deed do feel better about starting fresh too... But it's gonna be hard to find enaugh space to keep my files while i start the server... I will see :-)
August 17, 201213 yr This is what I did. 1. I ran your script unraid_partition_disk.sh without the -p option. The script reported that the drive was the incorrect size. 2. I then ran your script with the -p option. The script queried for me to resize and I said "Yes". This extended the size to what UNraid expected. 3. After that I ran the script without the -p option and the script reported that unraid would accept. 4. I then assigned the drive to an empty slot on the devices page. 5. I then pressed the restore button for the array to accept the drive as a new configuration. 6. I then started the array, unraid accepted the configuration and started the parity sync. 6. The parity-sync is now in progress. My array is unprotected until the completion of the parity sync. I hope that helps. I never ended up using a resize_reiserfs script. The only script I used was the unraid_partition_disk.sh that you provided. Should put this in the Wiki/help pages! Great "how to".
August 19, 201213 yr no disrespect intended, but if you have to ask this question, the odds of your success is almost non-existant. If I were you, I would (on your window's system) copy ALL the contents of the two NTFS drives onto one of them. It would be easiest if the disk you empty is the 3TB drive. If not, you are SOL and need an additional 3TB drive. Then, once you have an empty disk to use, start your unRAID array with it as the data drive. It is strongly suggested you pre-clear all the disks before adding them to the unRAID server. It is not mandatory, but one out about 5 drives has issues when brand new. (do you feel lucky?) Have you gotten a SMART report on your existing NTFS drives? Windows just mangles your data and keeps you in the dark about the disk health... Are there any sectors pending re-allocation? or already re-allocated? Purchase a new 3TB drive, pre-clear it, and add it as your parity drive. Then, migrate the data from your current 3TB NTFS drive to the unRAID array. Do this over the LAN, using something like teracopy that can perform a verification checksum to ensure the transfer was as expected. Lastly, once you know the array is on-line and problem free,(wait a week or so), and parity protected, use the pre-clear script to clear the 2TB drive prior to adding it as the second data drive. I'd bet your original proposal is near impossible. unRAID expects only 1 partition, and it must be EXACTLY as it would have defined it. Odds are VERY high your existing NTFS partitions already use the disk space needed, and the "free" space is not really free, but part of its file-system. Basically, you have no choice. You WILL be copying 4TB of data... even if your scheme were possible, it would be copying 4TB from one section of a disk to another sector on the same disk, and that, with all the head-seeking involved, would be VERY slow. Joe L. Just a few more questions: If I have to purchase another 3TB drive for parity. Can't I just start the array with the new drive as data drive without any parity. Move the contents from the full 2TB drive on it, add the 2TB drive to the array, move the data from the old 3TB drive to the array. Than I can add the old 3TB drive as parity and start sync? I would take the risk of loosing my data in the time I am copying and syncing the parity but would it work? Do this over the LAN, using something like teracopy that can perform a verification checksum to ensure the transfer was as expected. Do I have to move the files via LAN to the server or can i plug them into USB with an external SATA controller? When copying from the USB to internal HDDs does unRaid perform a checksum verification or is that the main reason I should use LAN and teracopy? And: generally speaking: Which of the two 3TB drives should I use for data, the old or the new one? Logically speaking I would say the new one for data and the old one for parity but I thought I ask you guys first :-) Again: thanks in advance!
August 19, 201213 yr Just a few more questions: .... And: generally speaking: Which of the two 3TB drives should I use for data, the old or the new one? Logically speaking I would say the new one for data and the old one for parity but I thought I ask you guys first :-) Again: thanks in advance! If it were I, I would use the old 3TB drive for Data and the new drive for Parity. My reasoning is that the new drive is more likely to fail in the short term than the old drive. (The failure mode is know as 'Infant Mortality'.) If the Parity Drive fails, all of your data is still safe and intact. The data drives safely contain all of your data. Rebuilding parity is a relatively safe operation. Reconstructing data from a data disk rebuild is a bit more chancy-- more things can go wrong, go wrong, go wrong...
August 21, 201213 yr Thank you Frank, I always thought that a new drive is more safe than an older one, I never had problems with a new drive... Are there any reasoable numbers on this 'Infant Mortality'? Any thoughts for my other questions? If I have to purchase another 3TB drive for parity. Can't I just start the array with the new drive as data drive without any parity. Move the contents from the full 2TB drive on it, add the 2TB drive to the array, move the data from the old 3TB drive to the array. Than I can add the old 3TB drive as parity and start sync? I would take the risk of loosing my data in the time I am copying and syncing the parity but would it work? Do this over the LAN, using something like teracopy that can perform a verification checksum to ensure the transfer was as expected. Do I have to move the files via LAN to the server or can i plug them into USB with an external SATA controller? When copying from the USB to internal HDDs does unRaid perform a checksum verification or is that the main reason I should use LAN and teracopy? And: generally speaking: Which of the two 3TB drives should I use for data, the old or the new one? Logically speaking I would say the new one for data and the old one for parity but I thought I ask you guys first :-) Again: thanks in advance!
August 21, 201213 yr Just google "infant mortality electronics". This phenomenon was first discovered in WWII and I can remember that when I had my first all electronic test sets built back in the 1970's, most of them would have one or more components fail in the first 100 hours or so after we applied power to them. Most of them NEVER had a component fail after that point. Back when PC's were introduced, many vendors advertised that they 'burned-in' their systems for 48 to 72 hours before shipping. While reliability has improved over the years, infant morality is still there. That is one of the reasons why 'preclearing' is recommended for all drives before they are added to unRAID systems. Some folks say that you should do three preclear cycles which puts more than 100 hours on a drive. A 'reasonable number'... Not quite sure what you mean. It is like saying if one million people leave work for home, only one will die before he/she gets there. It doesn't mean much UNLESS you are the One... To be brutally honest, manufacturers never give any truly meaningful reliability numbers to the general public on their products. First, they are expensive to generate, take a long period of time to determine, and they are what they are. That is why they give warranties instead. From that we are to infer that their products are reliable. They build the expected warranty costs into their product price. And they know that from prior experience on similar products previously manufactured. The difference between a 90 day warranty and lifetime warranty is not in the inherent reliability of the product but its cost.
August 21, 201213 yr Just google "infant mortality electronics". This phenomenon was first discovered in WWII and I can remember that when I had my first all electronic test sets built back in the 1970's, most of them would have one or more components fail in the first 100 hours or so after we applied power to them. Most of them NEVER had a component fail after that point. Back when PC's were introduced, many vendors advertised that they 'burned-in' their systems for 48 to 72 hours before shipping. While reliability has improved over the years, infant morality is still there. That is one of the reasons why 'preclearing' is recommended for all drives before they are added to unRAID systems. Some folks say that you should do three preclear cycles which puts more than 100 hours on a drive. A 'reasonable number'... Not quite sure what you mean. It is like saying if one million people leave work for home, only one will die before he/she gets there. It doesn't mean much UNLESS you are the One... To be brutally honest, manufacturers never give any truly meaningful reliability numbers to the general public on their products. First, they are expensive to generate, take a long period of time to determine, and they are what they are. That is why they give warranties instead. From that we are to infer that their products are reliable. They build the expected warranty costs into their product price. And they know that from prior experience on similar products previously manufactured. The difference between a 90 day warranty and lifetime warranty is not in the inherent reliability of the product but its cost. From the results of unRAID users pre-clearing new drives I'd guess somewhere between 10 and 20% of the drives are reported as failing when brand new. I suppose it depends on the batch of drives you get, and your luck (and how roughly they were treated in shipping) I'd never trust a brand new drive more than one in operation for several years. google "bathtub curve" for more details. Joe L.
August 21, 201213 yr Okay. Thank you for this Conclusion! So I guess I will take the 2 drives I have for data and the new one for parity.... Can I just start the array without a parity drive, copy all contents to it and then put in the new drive, clear it and start building parity on it? If I have to purchase another 3TB drive for parity. Can't I just start the array with the new drive as data drive without any parity. Move the contents from the full 2TB drive on it, add the 2TB drive to the array, move the data from the old 3TB drive to the array. Than I can add the old 3TB drive as parity and start sync? I would take the risk of loosing my data in the time I am copying and syncing the parity but would it work? Can I proceed like I described here or do I have to copy everything via LAN? Do this over the LAN, using something like teracopy that can perform a verification checksum to ensure the transfer was as expected. Do I have to move the files via LAN to the server or can i plug them into USB with an external SATA controller? When copying from the USB to internal HDDs does unRaid perform a checksum verification or is that the main reason I should use LAN and teracopy? Thnak you very much!
August 21, 201213 yr Preclear that new drive if you haven't already done so!. I did a quick scan of this thread and I didn't see that you had. Finding a marginal drive in preclear that will fail soon is much better scenio than having a drive fail shortly after adding it to an array. When I converted to unRAID. I copied then data over with the parity drive running. I did the transfers overnight and it only took about three nights to move about 2TB of data over using Teracopy. I start at bedtime with a selection of files and find it finished in the morning. I kept the files on the NTFS disks that they were stored on for a couple of weeks until I was sure that the server was really stable. Every computer and USB drive in the house was FULL.
August 24, 201213 yr I heared of another program called rsync that can copy with checksum verification too. Has anyone done this before and maybe give me tips/hints on how to use rsync? It would be faster to copy the data via usb than via 100 Mbit LAN... Thank you
November 26, 201213 yr I heared of another program called rsync that can copy with checksum verification too. Has anyone done this before and maybe give me tips/hints on how to use rsync? It would be faster to copy the data via usb than via 100 Mbit LAN... Thank you Has anyone tried to copy files with rsync via USB? I would like to stick the full HDD into a SATA-USB adapter to copy the files to the unRAID array because it is much faster than LAN :-)
November 27, 201213 yr USB 2.0 is 480Mbps Gig-Ethernet is 1000Mbps The LAN should be twice as fast. The thing is: As I said I have only 100Mbps Ethernet. So I am searching for another opportunity to Transfer the files...
Archived
This topic is now archived and is closed to further replies.