October 31, 201015 yr Good morning, I used the wrong disk with the prebuild command Not a bis issue because that it´s my smallest disk and I have a backup available but maybe a good opportunity to play a little bit around with my server in order to learn something. As the new disk is still being precleared I was able to assign the old 250gb disk and the only option in the main menu was to format the disk. Now my questions: how do I rebuild the data from parity? This is the status - the wrongly preclear disk is of course without any data - just 1%: Status Disk Mounted Device Model/Serial Temp Reads Writes Errors Size Used %Used Free OK parity /dev/sda 00S_WD-WCAVY2530562 30°C 7079 8818 OK /dev/md1 /mnt/disk1 /dev/sdc 00S_WD-WCAVY2657059 29°C 125360 5692 1.50T 896.10G 60% 604.16G OK /dev/md2 /mnt/disk2 /dev/sde 40T_WD-WCANY1940426 25°C 4004535 478935 250.05G 33.64M 1% 250.02G OK /dev/md3 /mnt/disk3 /dev/sdb 00S_WD-WCAVY2465093 30°C 5129 54 1.50T 1.26T 84% 242.75G Total: 3.25T 2.15T 66% 1.10T Thank you very much.
October 31, 201015 yr Good morning, I used the wrong disk with the prebuild command Not a bis issue because that it´s my smallest disk and I have a backup available but maybe a good opportunity to play a little bit around with my server in order to learn something. As the new disk is still being precleared I was able to assign the old 250gb disk and the only option in the main menu was to format the disk. Now my questions: how do I rebuild the data from parity? This is the status - the wrongly preclear disk is of course without any data - just 1%: Status Disk Mounted Device Model/Serial Temp Reads Writes Errors Size Used %Used Free OK parity /dev/sda 00S_WD-WCAVY2530562 30°C 7079 8818 OK /dev/md1 /mnt/disk1 /dev/sdc 00S_WD-WCAVY2657059 29°C 125360 5692 1.50T 896.10G 60% 604.16G OK /dev/md2 /mnt/disk2 /dev/sde 40T_WD-WCANY1940426 25°C 4004535 478935 250.05G 33.64M 1% 250.02G OK /dev/md3 /mnt/disk3 /dev/sdb 00S_WD-WCAVY2465093 30°C 5129 54 1.50T 1.26T 84% 242.75G Total: 3.25T 2.15T 66% 1.10T Thank you very much. I'm not sure what "prebuild" command you are referring to. If you used preclear_disk.sh on the wrong disk then it must not have been assigned to the array as the preclear_disk.sh script is written to not allow it to be used on a drive assigned to the array. Please explain EXACTLY what you did, in detail. It appears as if you assigned the 250Gig drive to your array and formatted it. Parity is NOT a backup. It does not allow you to recover from files you overwrote. It allows you to recover from a failed drive. If you want to see how to recover a failed drive you can write a file or two to disk2 and then stop the array, power down, and un-plug it. Then power up. You'll see the disk indicator red, but still be able to BOTH read and write to the drive you unplugged. Copy a new file to the disk you unplugged. (Confirming you can write to the drive) Then stop the array once more. Go go the Devices page, un-assign disk2. Then go back to the main page and start the array once more. You can still read and write the un-assigned drive. (This step of un-assigning it will allow the drive to be used as its own replacement) An un-assigned drive is treated exactly like a failed drive. Copy another file to the un-assigned disk. (The un-assigned, unplugged disk now has two new files) Now, Stop the array once more and power down. Plug the drive cable back in. Power up. The drive should still show as "red" Stop the array and use the "Devices" page to re-assign disk2. Go back to the main page and start the array by pressing "Start" The re-constructed contents of the drive will be written to the physical disk along with all the changes you made while it was unplugged. If you cleared the wrong drive and want to recover the files that were on it... sorry, a precleared disk is completely zeroed. There's nothing there to recover any more. Joe L.
October 31, 201015 yr Author This is the command I used: ./preclear_disk.sh /dev/sde but I should have used ./preclear_disk.sh /dev/sdd instead. sde was a disk assigned to the array when I used the above command. That´s the reason why I was getting a "Missing Disk" after the preclear run. I assigned it back to the array, was asked to format and now it´s back in the array. My understanding was that it´s possible to rebuild data from the parity; my fault. Thanks a lot.
October 31, 201015 yr This is the command I used: ./preclear_disk.sh /dev/sde but I should have used ./preclear_disk.sh /dev/sdd instead. There are four different tests in the pre-clear script that would prevent it from being used on a disk assigned to the array. One checks to see if the drive is part of your assigned array. Another checks to see if it is one of the devices in your config/disk.cfg file A third checks to see if it is mounted A fourth checks to see if it is "busy" Any one of the above tests would have prevented the disk from being cleared. Therefore, I'm a bit confused. was the array running when you did the clearing? Was it on a different PC? sde was a disk assigned to the array when I used the above command. That´s the reason why I was getting a "Missing Disk" after the preclear run. I'm not so sure. I suspect something else caused the "Missing Disk" as that indicates a hardware issue... (loose cable most likely) I assigned it back to the array, was asked to format and now it´s back in the array.to unRAID the format button is present whenever a disk cannot be mounted as a reiserfs file system. The formatting is part of the parity calculations. Right now, if the disk were to fail you could re-construct a formatted disk. My understanding was that it´s possible to rebuild data from the parity; my fault. Thanks a lot. Parity is not a backup. Parity is a bit of math to allow the array to deduce what bits existed on a missing drive. I'll try to explain... Lets say I set up my new UnRaid server with 5 disks for data and 1 for parity. For arguments sake: the very first file located on the first disk is a text file joe.txt the very first file located on the second disk is a picture of my cat there are no files on the third disk... it is currently empty the very first file located on the fourth disk is a MS-Word document the very first file located on the fifth disk is a PDF document Now... I also have a parity disk and I need to store data on it so if I lose any single drive I can still rebuild its data. So I start by reading the first bit of each file on each disk. the very first bit in the text file joe.txt on the first disk is a "1" the very first bit in the picture of my cat on the second disk is a "0" for the third disk the spot on the disk where the first file would be is a "0" the very first bit in the MS-Word document on the fourth disk is a "1" the very first bit in the PDF document on the fifth disk is a "1" Now... Tom has said he stores EVEN parity... so far we have an ODD number of ones in this bit position across all the drives. Since we store EVEN parity we need to store a "1" on the parity drive for this bit position. (that will give us a total of four "1s", an EVEN number, in this bit position on the drives. ) Remember, we always set the parity bit so we have an even number of '1's for a bit position. If the second bit position on ALL the data drives in my array had a total of 2 of the bits set to a "1" the parity bit would be set to '0' as we already had an EVEN number of '1's in that bit position. Now... lets say one of the drives fails and the unRaid software detects the failed drive... since we have a parity drive we can figure out what would have been on the failed drive if we could have read it... To do this we need to read the first bit from all the drives still available. we read the very first bit on the first disk as a "1" (joe.txt) we read the very first bit on the second disk is a "0" (picture of cat) we read the very first bit off the empty third disk is a "0" we CANNOT READ the fourth disk as it has failed - this is the bit we need to recreate we read the very first bit on the fifth disk is a "1" (PDF document) we read the very first bit in parity drive and it is a "1" Now... we know that when we originally calculated and stored the bit on the parity drive we had an EVEN number of "1s" in that bit position on the drives. So... with that knowledge we can figure out what the missing bit would have been if we could have read it. Since in all the bits we can still read (data bits in the working drives and the bit in the parity drive) we have an odd number of "1s", the missing bit we cannot read because of the failed drive must have also been a "1" to make it an even number of "1s." The bit on the failed drive can't be a "0" since we would then only have three bits set to "1" in that bit position, (an odd number) and since know we originally calculated and stored EVEN parity... the missing bit had to be a "1", for a total of 4 bits set to "1" (an EVEN number) So... the parity algorithm lets us recreate any single missing bit by using the data on the parity drive in combination with all the other data drives that can still be read. Obviously it will need to do this process individually for each bit on the drive but for this example, we just used the first bit of the first byte on the disk. Perform the same math a few trillion times and you can re-construct the entire set of bits that existed in the failed (or missing, or un-assigned) drive. Also, in reality, the first byte on each disk in the protected array is part of the housekeeping bytes of the file system and not the files themselves, the files follow the initial file-system structures, but is does not matter since parity does not work at the file level... to it, it is just a bunch of bits. Everything, including the formatting and file system structures is just bits. It is why you can add a pre-cleared drive to an array. Adding an disk with all zeros does not change parity, since it does not change the EVEN number of bits across a given bit position on the array of disks. Joe L.
October 31, 201015 yr Author Joe, thank you for the explanation. The array was running what I started the (wrong) preclear. Anyway, I included both - the new and the formatted - disk into the array and have massive sync errors now. Model / Serial No. Temperature Size Free Reads Writes Errors parity WDC_WD15EARS-00S_WD-WCAVY2530562 34°C 1,465,138,552 - 343,735 135,108 0 disk1 WDC_WD15EARS-00S_WD-WCAVY2657059 34°C 1,465,138,552 589,992,960 587,460 123,202 0 disk2 WDC_WD2500JS-40T_WD-WCANY1940426 30°C 244,198,552 244,158,124 229,001 30 0 disk3 WDC_WD15EARS-00S_WD-WCAVY2465093 34°C 1,465,138,552 237,057,632 233,180 109 0 disk4 WDC_WD10EAVS-00D_WD-WCAU40038426 31°C 976,762,552 976,699,896 229,708 18 0 Started Stop will take the array off-line. Parity-Check in progress. Cancel will stop the Parity-Check. Total size: 1,465,138,552 KB Current position: 90,645,956 (6.1%) Estimated speed: 8,603 KB/sec Estimated finish: 2661.8 minutes Sync errors: 22357418 What´s your advise? I started the "parity check without error correcction" from unMenu.
October 31, 201015 yr Joe, thank you for the explanation. The array was running what I started the (wrong) preclear. Anyway, I included both - the new and the formatted - disk into the array and have massive sync errors now. Model / Serial No. Temperature Size Free Reads Writes Errors parity WDC_WD15EARS-00S_WD-WCAVY2530562 34°C 1,465,138,552 - 343,735 135,108 0 disk1 WDC_WD15EARS-00S_WD-WCAVY2657059 34°C 1,465,138,552 589,992,960 587,460 123,202 0 disk2 WDC_WD2500JS-40T_WD-WCANY1940426 30°C 244,198,552 244,158,124 229,001 30 0 disk3 WDC_WD15EARS-00S_WD-WCAVY2465093 34°C 1,465,138,552 237,057,632 233,180 109 0 disk4 WDC_WD10EAVS-00D_WD-WCAU40038426 31°C 976,762,552 976,699,896 229,708 18 0 Started Stop will take the array off-line. Parity-Check in progress. Cancel will stop the Parity-Check. Total size: 1,465,138,552 KB Current position: 90,645,956 (6.1%) Estimated speed: 8,603 KB/sec Estimated finish: 2661.8 minutes Sync errors: 22357418 What´s your advise? I started the "parity check without error correcction" from unMenu. It shows you do not have parity protection. I have no idea how you got into the situation you are, but right now your parity disk is not correct. If you think your data disks are correct, then perform a normal "Ceck" of parity (which will find those same errors and correct them) If you think parity is correct, un-assign the drive you think is wrong, start the array without it assigned, stop the array once more, re-assign it, and let the array re-construct it. Joe L.
November 1, 201015 yr Author Good morning John, there are massive sync errors after the run finished. 10 movies are lost. So what do you think - is the parity correct? Is that the process I need to follow to get the movies back?? stop the array un-assign the drive which is wrong (this would be disk2 which I precleared and formatted and re-integrated into the array) start the array without drive assigned stop the array once more re-assign disk2 let the array re-construct disk2 Array Status STARTED, 5 disks in array. Parity is Valid:. Last parity check < 1 day ago . Parity updated 60635189 times to address sync errors. Array Disk Status Status Disk Mounted Device Model/Serial Temp Reads Writes Errors Size Used %Used Free OK parity /dev/sda 00S_WD-WCAVY2530562 34°C 4242227 389931 OK /dev/md1 /mnt/disk1 /dev/sdc 00S_WD-WCAVY2657059 33°C 4805898 368782 1.50T 896.12G 60% 604.14G OK /dev/md2 /mnt/disk2 /dev/sde 40T_WD-WCANY1940426 * 624181 35 250.05G 33.75M 1% 250.02G OK /dev/md3 /mnt/disk3 /dev/sdb 00S_WD-WCAVY2465093 33°C 3913160 156 1.50T 1.26T 84% 242.75G OK /dev/md4 /mnt/disk4 /dev/sdd 00D_WD-WCAU40038426 27°C 2573239 42 1.00T 33.63M 1% 1.00T Total: 4.25T 2.15T 50% 2.10T
November 1, 201015 yr Good morning John, there are massive sync errors after the run finished. 10 movies are lost. So what do you think - is the parity correct? Is that the process I need to follow to get the movies back?? stop the array un-assign the drive which is wrong (this would be disk2 which I precleared and formatted and re-integrated into the array) start the array without drive assigned stop the array once more re-assign disk2 let the array re-construct disk2 Array Status STARTED, 5 disks in array. Parity is Valid:. Last parity check < 1 day ago . Parity updated 60635189 times to address sync errors. Array Disk Status Status Disk Mounted Device Model/Serial Temp Reads Writes Errors Size Used %Used Free OK parity /dev/sda 00S_WD-WCAVY2530562 34°C 4242227 389931 OK /dev/md1 /mnt/disk1 /dev/sdc 00S_WD-WCAVY2657059 33°C 4805898 368782 1.50T 896.12G 60% 604.14G OK /dev/md2 /mnt/disk2 /dev/sde 40T_WD-WCANY1940426 * 624181 35 250.05G 33.75M 1% 250.02G OK /dev/md3 /mnt/disk3 /dev/sdb 00S_WD-WCAVY2465093 33°C 3913160 156 1.50T 1.26T 84% 242.75G OK /dev/md4 /mnt/disk4 /dev/sdd 00D_WD-WCAU40038426 27°C 2573239 42 1.00T 33.63M 1% 1.00T Total: 4.25T 2.15T 50% 2.10T If you un-assign the drive, the contents that could be restored would still be visible from the server as it emulates the un-assigned drive. If you do not see the movies they will not be re-constructed onto a replacement drive. (Basically, you erased them when you zeroed the drive) Joe L.
November 1, 201015 yr Author I have 4 hours of re-construction time behind me but none of the files are back. I´m asked to press the format button....is that normal behaviour? Here is the current main menu status: Disk status Model / Serial No. Temperature Size Free Reads Writes Errors parity WDC_WD15EARS-00S_WD-WCAVY2530562 35°C 1,465,138,552 - 719,452 61,185 0 disk1 WDC_WD15EARS-00S_WD-WCAVY2657059 34°C 1,465,138,552 589,979,832 957,906 53,297 0 disk2 WDC_WD2500JS-40T_WD-WCANY1940426 30°C 244,198,552 Unformatted 66 697,144 0 disk3 WDC_WD15EARS-00S_WD-WCAVY2465093 35°C 1,465,138,552 237,057,648 679,561 79 0 disk4 WDC_WD10EAVS-00D_WD-WCAU40038426 31°C 976,762,552 976,699,836 674,304 50 0 Command area Started Stop will take the array off-line. Unformatted disk(s) present Format will create a file system in all Unformatted disks, discarding all data currently on those disks. I'm sure I want to do this Parity is valid. Check will start a Parity-Check. (Last checked on 11/1/2010 8:06:52 PM, finding 60635189 errors.)
November 1, 201015 yr I have 4 hours of re-construction time behind me but none of the files are back. I´m asked to press the format button....is that normal behaviour? Here is the current main menu status: Disk status Model / Serial No. Temperature Size Free Reads Writes Errors parity WDC_WD15EARS-00S_WD-WCAVY2530562 35°C 1,465,138,552 - 719,452 61,185 0 disk1 WDC_WD15EARS-00S_WD-WCAVY2657059 34°C 1,465,138,552 589,979,832 957,906 53,297 0 disk2 WDC_WD2500JS-40T_WD-WCANY1940426 30°C 244,198,552 Unformatted 66 697,144 0 disk3 WDC_WD15EARS-00S_WD-WCAVY2465093 35°C 1,465,138,552 237,057,648 679,561 79 0 disk4 WDC_WD10EAVS-00D_WD-WCAU40038426 31°C 976,762,552 976,699,836 674,304 50 0 Command area Started Stop will take the array off-line. Unformatted disk(s) present Format will create a file system in all Unformatted disks, discarding all data currently on those disks. I'm sure I want to do this Parity is valid. Check will start a Parity-Check. (Last checked on 11/1/2010 8:06:52 PM, finding 60635189 errors.) A disk is considered unformatted in unRAID if it cannot be mounted as a reiserfs file system. That could be because of any number of issues. If you press the format button ALL data on that disk will be erased. Do not do it if you wish to re-construct data onto a replacement. Instead determine why the "mount" failed. Sometimes it is just a matter of doing a file-system-check to repair a damaged file system. Attach a syslog to your next post. It will provide the clues. Joe L.
November 1, 201015 yr Author When I press the file system check button in unMenu it shows: Checking /dev/md2 (/dev/sdf) Sorry, no file system detected on /dev/md2 Before I erase everything by formatting the disk I would appreciate if you can look into the log file:
Archived
This topic is now archived and is closed to further replies.