For fun: how can parity be improved

[Joe L.]

I should also be clear that when i was tlaking about replacing a drive i used confusing terminology... I am talking about replacing a working drive with another working drive (aka my array is full take this 250GB one and replace it with a 1TB one).

That has always been possible.  But the new, 1TB drive cannot have existing data on it.  (or rather, any contents would be lost as the process writes the new image of the disk being replaced) I've upgraded drives a number of times... most of us have.

 

In that situation, it is treated exactly as if the one you were upsizing had failed.  The remaining drives+parity are used to re-construct the contents of the older,smaller drive onto the newer, larger replacement.

 

Joe L.

[bubbaQ]

So let's assume you had this feature:  "Pull disk from array with data intact" sounds like what you want.

 

To do this, you have to "subtract" the data on that disk from parity.  

 

Then what happens if, part way through that process, you have an error or a drive dies or a power blip?  What happens is that you can't reconstruct from parity.  So basically, during this process, you have no parity protection.  That is no different from just unassigning the drive, and letting parity recalc.  There is no way to do this and keep parity, unless you had a second parity disk to hold a "new" parity, and then swapped the parity disks when the operation was done (and you'd have to handle writes to other disks that happened during the process too)

 

Writing zeros will wipe the data on the drive, but during the whole process, parity stays valid.  That is a MUCH better process from a data integrity point of view.

[NAS]

Subtracting a disk would not take 30 hours which is what the better way takes

[Joe L.]

Subtracting a disk would not take 30 hours which is what the better way takes

Actually, it probably would take 12 hours or so...  at least for a large drive. 

[bubbaQ]

In that situation, it is treated exactly as if the one you were upsizing had failed.

 

There is a safer procedure that could be done.  If you did the swap to a larger drive, and let it rebuild, you are at risk of a second drive failure during rebuild.  (Theoretically, since the original didn't die and you are doing a size upgrade, you theoretically sould put the old drive back, force the array to trust parity, and then rebuild the drive that actually died.... but I digress).

 

1. add new drive.  Clear and Format it.  Let it come online.

2. copy data from old drive to new drive.

 

Now when you remove the old drive, you lose parity protection.... and it has to recalc.  So instead, a new function could:

 

3. zero out old drive (retaining parity during the process)

4. remove drive from the array after it is zeroed out --- retaining valid parity.

 

Parity remains valid during the entire process.  It takes longer, but you have more protection.... and aint' that what it's all about?

 

PS.  This also lets you swap to a SMALLER drive (assuming the first one was not 100% full and its contents would fit o a smaller drive) ... something that currently you can't do.

[NAS]

Subtracting a disk would not take 30 hours which is what the better way takes

Actually, it probably would take 12 hours or so...  at least for a large drive. 

 

Im not disagreeing with your but when my array was smaller my parity calcs were almost exactly 6 times faster. Even if it took 12 hours thats one night a not an entire day and into the next day of no parity backup.

 

bubbaQ procedure sounds very sensible, minimising parity outages.

[bubbaQ]

Subtracting a disk would not take 30 hours which is what the better way takes

 

Whether it is 6 hours or 60, the point is that you will have no parity protection during the subtraction process.  You can only maintain parity if you clear the drive to 0 as you subtract it.

 

There is no way to subtract a drive, keep the data, and maintain parity.... you can only do 2 of the three (unless you introduce an extra parity drive, and that would be quite complex).

 

Clearing an active drive should be easy to implement.  Removing a cleared drive from the array while retaining parity should also be fairly simple (it is just the opposite of adding a new, cleared drive). 

 

This has the added benefit of having securely wiped the drive, so you can sell it on eBay and be assured that your data was safely overwritten.

[praeses]

Only idea I have is an older one, which I still think has merit.  For sometime in the distant future of unRAID: staggered parity.

Theoretically it could have potential to help in situations with a large fast parity drive and many smaller attached on slower shared buses (pci, port multipliers, etc).  It could introduce a lot of user issues however.  But hey, it does say "For fun" in the subject of this thread.

 

http://lime-technology.com/forum/index.php?topic=3049.0

[Joe L.]

Subtracting a disk would not take 30 hours which is what the better way takes

 

Whether it is 6 hours or 60, the point is that you will have no parity protection during the subtraction process.  You can only maintain parity if you clear the drive to 0 as you subtract it.

 

There is no way to subtract a drive, keep the data, and maintain parity.... you can only do 2 of the three (unless you introduce an extra parity drive, and that would be quite complex).

Actually, there is a way I can think of of maintaining parity, keeping the data, and subtracting "two" drives.

 

It would involve having an identical data drive, of the same size in the array to use as a "mirror" pair.

 

Let's say you have five data disks in your array, but one is unused... completely unused.  These are in addition to a sixth parity drive.

 

Now, to delete "two" of the drives from the array, ... lets say your have /dev/sde and /dev/sdf  and /dev/sdf is assigned to your array, but unused.  Let's say you assigned them to disk4 and disk5 affiliated with /dev/md4 and /dev/md5 in the protected array.

 

With the array running, you can copy all of /dev/sde to /dev/sdf. 

dd if=/dev/md4 of=/dev/md5 bs=32k

 

When this is finished, you can then delete both of them, at the same time from the array without affecting parity at all.

 

Joe L.

[bubbaQ]

There are several other solutions if you introduce another drive!  Yours would work, but they would have to be identical in number of sectors.

 

Also, IIRC, there can be hiccups with dd raw writing to a mounted disk.... when writing to a raw device, I always dd to an unmounted device to be safe.

[NAS]

Subtracting a disk would not take 30 hours which is what the better way takes

 

Whether it is 6 hours or 60, the point is that you will have no parity protection during the subtraction process.  You can only maintain parity if you clear the drive to 0 as you subtract it.

 

There is no way to subtract a drive, keep the data, and maintain parity.... you can only do 2 of the three (unless you introduce an extra parity drive, and that would be quite complex).

 

Clearing an active drive should be easy to implement.  Removing a cleared drive from the array while retaining parity should also be fairly simple (it is just the opposite of adding a new, cleared drive). 

 

This has the added benefit of having securely wiped the drive, so you can sell it on eBay and be assured that your data was safely overwritten.

 

If an option at the beginning of this was added to move drive data to other drives, we would have a relatively simple upgrade a drive procedure.

 

 

I also like the staggered drive idea. You could extend this idea if you had either a parity drive that was double the size of any signle drive. This now that makes sense since since 1TB drives are probably the best price point and 2TB drives are available.

 

So in this scenario drives 1- 10 could be in the first half of the parity and drives 11-20 could be on the second. This would complicate limit the choice of data and parity drives but the logic itself is simple enough.

 

Joes idea is also very clever but obviously a bit ore complicated.

 

There are lots of good ideas here and thats was after all the point of the thread.

[bubbaQ]

RAID6 would be much more desirable than staggered parity, since you would be using the same amount of parity space, but get protection from 2 drive failures.

 

As I said, if you introduce a third disk, there are additional options.

 

With RAID6 and 2 parity disks, you can subtract a drive from one parity disk at a time, so only one parity disk at a time is invalid during the process.

 

It would, obviously, take twice as long.

 

During the process, your protection would be degraded, as you would only have 1 valid parity drive, rather than 2, so you are vulnerable to the 2-drive failure scenario until the disk had been subtracted from both parity drives

 

There is also something you need to consider, since time seems to be important to you.  Subtracting is slow, because you have to read from the parity disk, and then write back to the exact same sector, necessitating a full rotation of the drive.  Speed will likely be about the same as writing to a parity-protected data disk.  Caching on the drive obviously helps, but you ought to calculate the track and cylinder size and compare that to the cache size.  Optimally, you's want the disk cache to be greater than one cylinder of data.

[NAS]

RAID 6 is definitely and excellent solution that would be an improvement but for PCI bottle-necked machines i suspect staggering might be faster (perhaps not its hard to guess with so many factors).

 

Its not all about speed. Speed is good obviously but theres a trade of between how many days a year you dont have parity (if i change 10 disks a year with add/replace/remove) currently im looking at 14 days of no parity protection minimum. That may be unavoidable but it should not be be ignored as a considerable period where something can go wrong. Thi is especially true since a building block of uNRAID design and features is the abilty to use pretty much any disk.

 

You trade of speed with elegance, simplicity and features. Already in this thread there have been some excellent suggestions on how to automate some of the most common parity tasks.

 

Keep the ideas coming

 

[bubbaQ]

Speed is good obviously but theres a trade of between how many days a year you dont have parity (if i change 10 disks a year with add/replace/remove) currently im looking at 14 days of no parity protection minimum.

 

Changing out 10 disks a year?  It is generally a bad idea to design software around rare outlier data points.

 

So would you prefer to have RAID6, and then be reduced to single-drive failure tolerance for a cumulative 28 days a year?

[NAS]

The wholepoint of unRAID is disk flexibility so i suggest rare is the wrong word.

 

It is not unreasonable to assume someone will start with 3 disks and move to 20 within 2 years. Thats 10 disks a year.

[Joe L.]

The wholepoint of unRAID is disk flexibility so i suggest rare is the wrong word.

 

It is not unreasonable to assume someone will start with 3 disks and move to 20 within 2 years. Thats 10 disks a year.

No, not unreasonable, but I'd say not common either.  With 1 to 2 TB disks you are talking about adding between 10TB and 20TB a year.  I'd guess most HD movies are 20Gig or so...  That would be between 500 and 1000 movies if I did my math right.  If storing more highly compressed versions, it would be many many thousands.  Very few people have collections that large.

 

It is unreasonable to expect that calculating parity on 3 disks will take the same time as on 20. 

For maximum flexibility, with that many disks, I'd say two servers, with half the disks in each is the answer if you want shorter parity check/ reconstruct times.

 

See, unRAID is flexible, you get to choose... more disks in a single array, or fewer in multiple arrays.  At one point, Tom mentioned the possibility of having multiple arrays in a single server, each with their own parity drive.  That might be the best solution for flexibility (if he ever implements it)

 

Joe L.

[NAS]

There has to be a demand as thats the only reasonable reason the disk limit was increased.

 

S to say rare or uncommon is simply a guess. It might very well be true but were guessing none the less and it could quite as easily be common or often.

 

You are absolutely right that 3 disks parity is faster than 20, and new server will not be PCI bus crippled so the future is bright but there a large legacy of pro users that will fit this bill.

 

But thats besides the point. Calculating parity on 20 disks will be slow that is a given... its how we can speed up or otherwise improve the common parity tasks such as add/remove/replace/check a disk that where the real interest is at.

 

I had thought about doing some calcs on how much outage you could expect based on an escalation of 3 to 20 disks, with a couple of replacements a year but I realised it was pointless because it really is. Parity calculated the current all or nothing official way is always going to be slow so focusing on that isnt going to go anywhere.

[bubbaQ]

I said:

 

Changing out 10 disks a year?  It is generally a bad idea to design software around rare outlier data points.

 

You said:

 

It is not unreasonable to assume someone will start with 3 disks and move to 20 within 2 years. Thats 10 disks a year.

 

You keep change the target... do you want to talk about ADDING drives or about SWAPPING an existing drive for another larger one in its place?  Those are two very different concepts.

 

I can understand a large growth spurt the first year as you rip your existing library.  But anyone who's data grows by 20 TB every year, needs something more than unRAID.

[NAS]

...  But anyone who's data grows by 20 TB every year, needs something more than unRAID.

 

I disagree i think this is exactly unRAID territory

[WeeboTech]

...  But anyone who's data grows by 20 TB every year, needs something more than unRAID.

 

I disagree i think this is exactly unRAID territory

 

I'm going to agree with this one.

 

The one time that people need something more then unRAID is if you need contiguous space more then 20TB.

In a case within my environment, we drop loads of data in a filesystem, then use a system of links to deliver files to customer areais for FTP retrieval. In that case I do not think unRAID could fit the bill.

 

For the most part where you need gobs and gobs of storage, unRAID usually fits the bill very nicely.

 

[NAS]

Let me throw this into the mix.

 

I reckon a fair comment would be that the majority of unRAID media storage users have large amounts of data that doesnt change.

 

This opens up the concept of "closed off disks". So if you had 10*1TB "closed off "data disks full and further set of "open" data disks with two parity sets you could in theory only need to recalculate parity on the open disks on subsequent disk insertions. Every now and again you could close off a disk and insert it into the static party array.

 

The hurdles are logistical and "manner of usage" problems rather than technical ones. It introduces the limitation that a "closed off" disk cant be changed but that perhaps might not be as big a deal as it sounds.

 

The downsides are it needs 2 parity disks. The upside is that your essentially locking of data and with no need to write to that parity drive it could even be completely unmounted. It removes the risk of files being deleted by mistake etc

 

Thoughts

[WeeboTech]

Let me throw this into the mix.

 

I reckon a fair comment would be that the majority of unRAID media storage users have large amounts of data that doesnt change.

 

This opens up the concept of "closed off disks". So if you had 10*1TB "closed off "data disks full and further set of "open" data disks with two parity sets you could in theory only need to recalculate parity on the open disks on subsequent disk insertions. Every now and again you could close off a disk and insert it into the static party array.

 

The hurdles are logistical and "manner of usage" problems rather than technical ones. It introduces the limitation that a "closed off" disk cant be changed but that perhaps might not be as big a deal as it sounds.

 

The downsides are it needs 2 parity disks. The upside is that your essentially locking of data and with no need to write to that parity drive it could even be completely unmounted. It removes the risk of files being deleted by mistake etc

 

Thoughts

 

I think this is unnecessarily complex logic when in practicality, multiple separate arrays (or even unraid machines) would suffice.

 

It seems that you're really leaning towards limiting how many data drives per parity drive to handle a time factor for calculating parity.

 

 

[NAS]

Im definitely leaning towards anything that keeps the array protected longer.

 

Its not unfeasible for new unRAID users to be looking at 40TB arrays now and the current mechanism no matter how fast your kit is will take a long long time to do any task that requires a disk recalculation. This is excluding the users who are on last gen much slower hardware.

 

Yes you could build a machine with a load of disks on day one, calculate parity and forget about it but thats not what unRAID is all about. IRL parity calculation results in many days/weeks of unprotected data over a couple of years.

[WeeboTech]

Yes you could build a machine with a load of disks on day one, calculate parity and forget about it but thats not what unRAID is all about. IRL parity calculation results in many days/weeks of unprotected data over a couple of years.

 

Sometimes I think hairs are being split.

Before unRAID or with desktops we have many days/weeks of unprotected data.

If a person does not have confidence in the drives to handle a few days of being unprotected then there is a bigger issue.

 

What is the goal?

 

Using drives until ultimate failure?

Multiple Arrays?

Handling more then one simultaneous drive failure?

Speed up Parity?

[NAS]

...

What is the goal?

 

Using drives until ultimate failure?

Multiple Arrays?

Handling more then one simultaneous drive failure?

Speed up Parity?

 

Any of the above... its only about fun and playing with ideas that could (not should) make unRAID better 

 

But whats the ultimate goal... maintain the ability to not lose data as close to 100% of the time as possible.