Can I remove 2 empty drives and keep parity?


Kode

Recommended Posts

I read somewhere on here if you make an exact copy of a drive you can remove both and parity will still be valid, but it wasn't really explained how to do that.

 

I've just moved the contents of 2 x 1.5TB drives to a 4TB drive, so I now have 2 empty 1.5TB drives that I want to remove, which made me think of that procedure, so I was wondering if I could format both to xfs (they are both currently reiserfs) so that they are both definitely clear and then remove both and tell unraid that the parity is still valid?  Or is there more to it than that?

Link to comment
14 minutes ago, Kode said:

I was wondering if I could format both to xfs (they are both currently reiserfs) so that they are both definitely clear and then remove both and tell unraid that the parity is still valid?

No. Formatting does NOT clear the drive, it just writes an empty table of contents, the actual bits that make up the files are still on the drive.

 

Given the totality of your situation and experience level, I would recommend just removing the drives and setting a new config without them, then rebuilding parity. Do the rest of your drives have clean smart reports?

Link to comment

Yeah, the rest are all good, I was only trying to avoid doing another parity rebuild because I've already done 3 this month (the monthly scheduled one on the 1st, and another 2 when I removed 2 x 1TB drives - not at the same time, or I'd have done it in 1 go, which is what I should have done) + the next scheduled one is in 6 days

 

*edit*

Quote

No. Formatting does NOT clear the drive, it just writes an empty table of contents, the actual bits that make up the files are still on the drive.

 

There are no files though, the drives are empty

Edited by Kode
Link to comment

Parity build != parity check. After you do a parity build, you should do a parity check to make sure it built correctly. You did a check at the 1st of the month, but haven't done one since you rebuilt parity for the 2 drive removals. So, remove your 2 drives, together this time, build parity, and let the monthly parity check on May 1 be your test that all your drive removals and parity rebuilds were handled successfully.

Link to comment
1 hour ago, Kode said:

There are no files though, the drives are empty

 

 

Most of the bits of the files that were there before the format are still there, and those bits are all still part of parity. As mentioned, formatting doesn't clear anything, it just makes a new empty top folder. You shouldn't try to get fancy with parity without understanding it.

Link to comment

I don't mean there are no files because it's been formatted, I mean there are no files because the drives are empty...

 

Regardless though, I've done as suggested, removed the drives and a "Parity-Sync/Data-Rebuild" is in progress.

Edited by Kode
Link to comment

Also @trurl, I understand perfectly how parity works, if you don't understand how 2 identical drives removed from an array would mean parity is still valid then maybe it's you who doesn't understand parity.  The issue at stake was whether the drives being empty *were* in fact identical, that's what I wasn't sure of, I wondered if formatting would help, it wouldn't as @jonathanm explained, so I played it safe and am doing a rebuild.

 

*edit* I was being a bit facetious there as I'm sure trurl does understand how it works, but for anyone reading this who doesn't understand how removing 2 identical drives would mean parity was still correct and wants to know, I'll try to explain

 

Imagine you have 4 drives, you want to remove 2.  As long as they are *absolutely* identical they cancel each other out in terms of parity, in the example below 3 and 4 are the identical drives

 

1 2 3 4 P

1 1 1 1 0

0 1 0 0 1

1 0 1 1 1

0 0 0 0 0

1 1 0 0 0

 

So if you remove 3 and 4 you are left with

 

1 2 P

1 1 0

0 1 1

1 0 1

0 0 0

1 1 0

 

As you can see parity is exactly the same. The reason for this is the value for parity is always whatever is needed to make the sum of the drives even. The sum of 2 identical drives is *always* even (0+0 = 0 | 1 + 1 = 0), so removing them can't have any effect on the parity value, if parity is 1, then 1 + 0 is still 1 and if it's 0, then 0 + 0 is still 0.

 

How you can be *absolutely* sure they are identical is another question, as if they aren't 100% identical you will have invalid parity, so it's very dangerous

Edited by Kode
Link to comment

I do understand that 2 identical disks would cancel each other out in parity, it is getting them identical that is the problem. You would basically have to clone one disk onto the other with dd or something, while keeping both still in the array so parity would be updated as the contents of one of the disks got changed.

 

Understanding parity isn't the only piece of this puzzle. You also have to understand the word "format", which many people use all the time but with only a vague idea of what it does. "Format" means "write an empty filesystem to this disk". That is what it has always meant in every operating system you have ever used. If you format a disk in the array, unRAID treats this just as it does every other write, by updating parity. All that actually gets written to the disk is an empty top level folder. All of the other bits that were previously on the disk are unchanged. So formatting doesn't take nearly as long as actually clearing a disk, but all of those old bits are still there and are part of the calculated parity.

 

It is important to remember that at the bit level, everything except a read is actually a write. Deleting a file just writes to the filesystem a little bit of information saying the file is no longer part of the filesystem and so the space can be reused. Until it is reused, the other bits of that deleted file remain unchanged.

 

Another technique often mentioned here for removing a disk while maintaining parity is to clear the disk using dd to write zeros to the entire disk while the disk is still in the array. Since these writes update parity and in the end all those zeros don't affect the calculated parity, the disk can then be removed and a New Config with valid parity can be done. There is even a script for this method.

 

Link to comment

I think the misunderstanding began with

 

22 hours ago, Kode said:

I've just moved the contents of 2 x 1.5TB drives to a 4TB drive, so I now have 2 empty 1.5TB drives

 

and continued with

 

5 hours ago, Kode said:

I don't mean there are no files because it's been formatted, I mean there are no files because the drives are empty...

 

and the subsequent discussion about formatting is just a complication. A move from one device to another is simply a copy followed by a delete, but delete does not remove the content of the files and replace it with zeros, so the assumption that the disks were "empty" is a false one.

Link to comment
21 minutes ago, John_M said:

I think the misunderstanding began with

 

 

and continued with

 

 

and the subsequent discussion about formatting is just a complication. A move from one device to another is simply a copy followed by a delete, but delete does not remove the content of the files and replace it with zeros, so the assumption that the disks were "empty" is a false one.

 

So the misunderstanding is about what move (or delete) does. At the bit level, it is all just writes and bits and every bit has some value. There isn't really any such thing as an "empty" disk, unless by empty we mean all bits zero, which is the only "empty" that matters for parity purposes.

Link to comment

Yes, I was wrong thinking an "empty" disk was actually empty.

 

The real takeaway, and answer to my original question, is if you want to remove the disks and keep parity valid you need to zero the drives manually while they are part of the array.

 

*edit* might be a nice feature to have, if a drive is empty have a "Remove this drive and shrink array" button, which zeros the drive and unassigns it from the array.

Edited by Kode
Link to comment
1 hour ago, Kode said:

*edit* might be a nice feature to have, if a drive is empty have a "Remove this drive and shrink array" button, which zeros the drive and unassigns it from the array.

 

I think that has been requested but don't know if there are any near term plans to implement. I too would like to see such a feature, since I often see people running a lot of small disks and there are several advantages to running fewer larger disks, including increased performance due to greater disk density, and more reliability due to fewer points of failure.

 

I think zeroing a disk in the array would take longer than just rebuilding parity, but it would have the advantage of keeping parity valid throughout the process and parity would even be valid if the process were interrupted for some reason.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.