I am a bit unclear on the parity on the unRAID server.  Let's say I have 2 disks, the parity is generated from disk 1 and written to the parity disk.  Let's say I added disk 2 (3 disks now), so now the parity is generated from both disks and written to the parity disk.  If I lose disk 1, does the data recovery really need disk 2?  Let's say I lost both data disks, disk 2's data is toast, but I should still be able to recover data from disk 1 shouldn't I?  The data that were on the original disk 1 had parity generated and written on the parity disk without any knowledge of the data on disk 2.  Actually, once disk 1 is recovered, I should be able to recover disk 2 using data from disk 1 and the parity shouldn't I?  This of course have the limitation that since the addition of disk 2, no new data have been written to or removed from disk 1.

Here is a good explanation of how unRAID parity works:  http://lime-technology.com/forum/index.php?topic=938.msg6404#msg6404.

 

Ok, that explains some parts of it.  But question still remains.  In my example, disk 1's first bit is a 0, so on the parity disk that position is marked as 0.  Now after I added disk 2, and I wrote something to it.  If it ends up being written to the last bit, then the original parity for disk 1 is till untouched correct?  So if at this point both data disks fail, would I be able to recover disk 1 from the parity disk?  My guess is that bit 1 on disk 1 would be recovered, but the last bit on my disk 1 would not and might cause issues.  Since that bit could be a part of anything.  Partition info etc etc.   I guess I get it now.

 

 

Now a new question.  The parity protects against a drive failure.  How does it protect against bad sectors?

1) Parity as I understand, is created "over" the filesystem. (so doesn't calculate for every single bit of a drive, but in fact every single bit of INFORMATION of a drive)

 

2) The link Rob gave you does reply your first question. You NEED the other drives (and the parity) to restore a crashed disk.

 

3) To your new question. I'd say bad sectors are the responsibility of the filesystem (and the lowlevelformat).

 

 

1) Parity as I understand, is created "over" the filesystem. (so doesn't calculate for every single bit of a drive, but in fact every single bit of INFORMATION of a drive)

 

2) The link Rob gave you does reply your first question. You NEED the other drives (and the parity) to restore a crashed disk.

 

3) To your new question. I'd say bad sectors are the responsibility of the filesystem (and the lowlevelformat).

 

 

Actually, parity is calculated for every single byte that is physically on the drive, regardless of if it is part of a file system or not.

For those drives that are smaller than the parity drive zeros are substituted for a given bit position when doing the calcs for the larger drives.

 

Here is a description of how parity works that I wrote some time ago

http://www.avsforum.com/avs-vb/showpost.php?p=6278295&postcount=157

 

Joe L.

 

I know how parity works.

 

But I think in our case, where different sized disks are allowed, "every bit on surface" is not only unnecessary, it could potentially be bad for the system.

 

You see if I remove disk 1 (a 320GB WD for example) and put in disk 1b in its place (a 500GB Seagate), then filesystem format details are different. Restoring every single bit of disk 1 surface would probably mess things up (and in the best of cases recreate a pseudo-320GB disk).

 

Now if (and I think this is the case) parity is calculated on the actual data (again as they are put physically on the disk but excluding information that has to do with the actual DISK as a medium), then unRAID probably just formats the drive (where in "normal" RAID5 there is no need for format as what you described will happen so the disk will be "formated" by the rebuild itself), then from this parity restores the actual data.

 

This is how I'd build it. After all unRAID is a "higher-level" filesystem (than an actual file-system). This is why tomorrow the developer could come out and say that the system can use (for example) ext3 as the underlying fs.

 

Of course we are only talking theory (except if you helped building this thing - in that case sorry, please let us know) and maybe if the developer finds the time, could clear this out some time (and I wouldn't be NLS if I didn't mention this )...

 

 

I know how parity works.

 

But I think in our case, where different sized disks are allowed, "every bit on surface" is not only unnecessary, it could potentially be bad for the system.

 

You see if I remove disk 1 (a 320GB WD for example) and put in disk 1b in its place (a 500GB Seagate), then filesystem format details are different. Restoring every single bit of disk 1 surface would probably mess things up (and in the best of cases recreate a pseudo-320GB disk).

 

Now if (and I think this is the case) parity is calculated on the actual data (again as they are put physically on the disk but excluding information that has to do with the actual DISK as a medium), then unRAID probably just formats the drive (where in "normal" RAID5 there is no need for format as what you described will happen so the disk will be "formated" by the rebuild itself), then from this parity restores the actual data.

 

This is how I'd build it. After all unRAID is a "higher-level" filesystem (than an actual file-system). This is why tomorrow the developer could come out and say that the system can use (for example) ext3 as the underlying fs.

 

Of course we are only talking theory (except if you helped building this thing - in that case sorry, please let us know) and maybe if the developer finds the time, could clear this out some time (and I wouldn't be NLS if I didn't mention this )...

 

 

Sorry, no, I am just a customer, not affiliated with Tom or unRaid other than participating in the forum.  I am a software developer with much Unix experience so I can do things at the command line easier than most.

 

It sounds like you have a very good grasp of how disks are structured. That is good.  In the case of replacing a smaller disk with a larger replacement (a very likely scenario) the written description that I remember said that first the original disks contents would be written to the new disk (this would create the psuedo-smaller-size disk, exactly as you described, using the original partition size and formatting) and THEN the file-system on that disk would be expanded to use the full disk.

 

It is described on this page http://www.lime-technology.com/wordpress/?page_id=47

Here is an excerpt:

Replace a single disk with a bigger one

 

This is the case where you are replacing a single small disk with a bigger one:

 

  1. Stop the array.

  2. Power down the unit.

  3. Replace smaller disk with new bigger disk.

  4. Power up the unit.

  5. Start the array.

 

When you start the array, the system will reconstruct the contents of the original smaller disk onto the new disk. Upon completion, the disk’s file system will be expanded to reflect the new size. You can only expand one disk at a time.

 

I think the choice of only using reiserfs vs ext3, or other file systems was to make the management tool easier.  As you said, the unRaid driver is a "higher level" and cares only about blocks of data on the disks and could care less about the actual file system used.

The current management web-page does care about the specific file system.  It only scans for specific types to allow you to add them to the array.  It considers anything that does not have a reiserfs on it to be "unformatted"  If it were made smarter, then any file system could be supported.

 

Lastly,

This is not the first time Tom has been inactive on the forum.... Check out the thread on AVS for history.  He just moved to a different state.  I expect he got a new job, is setting up in a new house, and dealing with whatever else in his life that keeps him busy.  UnRaid was not his full-time salary in the past, it probably is not enough to support him now either.  He'll post here again when he finds time.

 

Joe L.

It all make sense now.  The parity gets changed without having to touch drives that aren't being written to.  and non-existent parts of the HD are just assumed 0.  Thanks for the wealth of info guys, it helped out a lot.  Not that I have any use for these information, but it satisfied my curiousity. 

ok thanks

 

as for Tom I tottaly understand - still since there is cost involved, people WILL have certain "demands" (excuse the word but...)