Protect offline disks

[NAS]

For fun.

 

I am coming to the conclusion that I can never have to many disks. Disks are so cheap now but to keep them online almost triples the per GB cost and lets face its annoying constantly buying server bits.

 

Most of my collection doesnt need to be online. If i wanted to watch an ep from season 1 of Buffy its not really a hardship to go plug a disk in.

 

So what if you could have an unRAID server with some disks online all the time but also have the ability to plug in disks without reboot (so via USB this now) on demand. And what if these disks could be protected even whilst offline (assuming they only ever are plugged into unRAID and never elsewhere say).

 

Sounds impossible but it probably isnt. Rebuilding a broken disk would be a PITA as you would need to do something fancy where you plugin every other offline disk.

 

But it isnt impossible but could it be done, should it be done, would it ever be done?

 

I know i would love to just buy disks and forget about PSU sizes for that nth online disk and if the MB i need is available etc

 

Comments

[kapperz]

I am coming to the conclusion that I can never have to many disks. Disks are so cheap now but to keep them online almost triples the per GB cost and lets face its annoying constantly buying server bits.

How does having disks in the server triple the cost per GB?

 

I wouldn't think it possible to have the USB drives protected by the parity drive.

 

Why make it any bit harder than it has to be? Just put a bunch of cheap drives in your server, and forget about it.

 

If you case has USB ports, simply plug an external drive in and mount it (with a script even). Then your Buffy episodes will be available.

 

I know i would love to just buy disks and forget about PSU sizes for that nth online disk and if the MB i need is available etc

Maybe the coffee hasn't kicked in yet. Is your PSU not powerful enough for the number of drives you have/will buy?

[neilt0]

Take a look at http://www.openegg.org/FlexRAID.curi

 

I looked at FlexRAID before unRAID, but I couldn't be bothered with updating parity manually.

[Rajahal]

Just thinking out loud here...

 

I'm imagining a server with 10 hot swap bays (two 5-in-3's, for example).  Now say you have twenty hard drives to play with.  Would it be possible to essentially have two separate servers running off this single mobo/cpu/ram/etc (so two flash drives each with a 10 drive server configuration)?  I believe that would mean you would have to swap out all 10 hdds, which would be a pain.  However, it may be appropriate if you have a large archive that would would like to keep offline most of the time.  Plus, swapping out 10 drives and a flash drive would take 10 minutes at most.  The disks would all have to be meticulously labeled, of course, since any mix-up could cause big problems.

 

A USB external solution (maybe use one of those USB toaster deals to cut costs) would definitely be better.  However, I thought parity could not be built across USB devices?

[NAS]

I am coming to the conclusion that I can never have to many disks. Disks are so cheap now but to keep them online almost triples the per GB cost and lets face its annoying constantly buying server bits.

How does having disks in the server triple the cost per GB?...

 

offline costs = disk costs

online costs = disk costs + chassis + MB + RAM + CPU + Flash + unraid lic + PSU + cables + power

 

if actually add it up it really mounts up

 

ok so you still need a server but once its full ... offline disks = zero extra cost whereas that one extra online disk = buying a whole new server.

 

unRAID is all about protected online disks but i simply dont need every bit of my collection online all the time if i am honest.

 

Just for the record I wouldn't move away from unRAID to another solution. I tried them all already and unRAID ticks almost every box.

 

...

I wouldn't think it possible to have the USB drives protected by the parity drive....

 

why not?

 

 

 

 

[kizer]

Why not just shelf a few drives and buy a Sata to USB adapter and plug them in when ever you need them. Heck use the S.N.A.P. program and have it automount when plugged in.

 

S.N.A.P.

http://lime-technology.com/forum/index.php?topic=5904.0

 

Adapter

http://www.amazon.com/USB-SATA-5-25-Cable-Adapter/dp/B000YJBL78

[NAS]

Theres nothing really unique about that. That is just bung in a USB drive. You can do that on any OS.

 

The key is protection.

[queeg]

Why not just shelf a few drives and buy a Sata to USB adapter and plug them in when ever you need them. Heck use the S.N.A.P. program and have it automount when plugged in.

 

S.N.A.P.

http://lime-technology.com/forum/index.php?topic=5904.0

 

Adapter

http://www.amazon.com/USB-SATA-5-25-Cable-Adapter/dp/B000YJBL78

 

I'm using a dual hdd dock with SNAP for this.  It has both usb and esata connections.  I attached the fan with some hot glue to the edge of the dock and have an external cheap 12 volt wall wart power supply for the fan and it's running very slow speed.  It keeps the drives cool even when room temps were in the low 90's this summer (no air conditioning in the NW).

[kizer]

I'm very aware there is no protection, however in the current state of things USB is about as hot swapable as we can get from an off the shelf solution without a fancy motherboard or getting exotic with Linux configurations.

 

Is it even remotely possible to include detached items into the protected array? As it sits unless I'm nuts when the parity check is performed it takes into account everything that is currently connected. I'm guessing what you are asking is it possible to account for removed items, which would be really nice.

[queeg]

I wanted to be able to pull out or add drives into the array (hot) without having to rebuild parity for the whole array.  That turned into this feature request.

I doubt it will ever be implemented.  But I think it would get used if it did.

[kapperz]

I am coming to the conclusion that I can never have to many disks. Disks are so cheap now but to keep them online almost triples the per GB cost and lets face its annoying constantly buying server bits.

How does having disks in the server triple the cost per GB?...

 

offline costs = disk costs

online costs = disk costs + chassis + MB + RAM + CPU + Flash + unraid lic + PSU + cables + power

 

if actually add it up it really mounts up

 

So your current server is completely full? Can't you increase one of the HDDs to a larger one? Use the drive you took out as an off-site backup. I planned my server for a total of 12 drives. If I filled it with 2TB drives that would be 22TB (11 - parity)*2). I can't image me needing that kind of space, but I'm set for a long time.

[NAS]

...

Is it even remotely possible to include detached items into the protected array? As it sits unless I'm nuts when the parity check is performed it takes into account everything that is currently connected. I'm guessing what you are asking is it possible to account for removed items, which would be really nice.

 

Well current protection is just a XOR. If a we had a parity disk onliine al the time one way would be to just XOR each disk as they are added. This make the asumption that all disks are never plugged in anywhere else.

 

Thats one way i wonder if there is another.

 

This could even be a completely new unRAID product since the main machine would only need a tiny PSU and one disk + HDD caddy(s)

So your current server is completely full? Can't you increase one of the HDDs to a larger one? Use the drive you took out as an off-site backup. I planned my server for a total of 12 drives. If I filled it with 2TB drives that would be 22TB (11 - parity)*2). I can't image me needing that kind of space, but I'm set for a long time.

 

Whilst all help is appreciated remember this is a for fun thread in the lounge not a call for help

 

 

who know knows that the real fun part

[BRiT]

The assumption that a disk may never be written to is not completely correct. The disk is written to when mounting and unmounting, unless explicitly mounted as read only.

[NAS]

Surely that would make no difference since the assumption is actually "This make the assumption that all disks are never plugged in anywhere else."?

[BRiT]

Not quite. Even if the disk is plugged into an unRAID system, the disk must explicitly be mounted as read-only. Failure to do so will cause writes to the disks to occur which will invalidate your parity.

 

Unless you will never ever ever read the data written on those drives. Are you really asking for a parity protected array of WORN drives? Write Once Read Never.

[NAS]

How so? The parity disk in this scenario would be online as well so all unRAID would have to do is deal with it.

 

Also keep in mind this is only one solution. Perhaps 1 parity per 3 data disks is a solution. maybe par2/3 helps us. USB3 might see 5 in 3 USB based hot swap trays come into existance. Maybe for every data disk you need a parity disk and use two USB HDD caddies.

 

Lots of things to ponder

[bubbaQ]

Protection  means you can recover from a lost drive.  You want to remove the drive, but have it protected.

 

So in essence, you don't need to put it back (except for speed, or if a different disk fails and recovery ability is lost), since if it is lost, you can recover it.

 

The problem with this paradigm, is if you want to remove X amount of data, you need X amount of space to store the information needed to reconstruct it.

 

Now if you want to require that the removed drives have no data changed on them outside of unRAID, you could do this:

 

- All you keep "live" is a parity drive.

- a drive can be added, and if new, it is cleared, and added to parity.

- any drives in the system in addition to parity can be written to and parity recalculated.

- any drive (other than parity) can be removed.

 

In the event of a failure, have to either 1) put ALL the drives in the system as the same time, or 2) add a new drive equal to or larger than the lost drive, and then put each of the other data drives in one at a time, have it rebuild the lost disk..... VERY slow.

 

To check/recalculate parity, you have to either 1) put ALL the drives in the system as the same time, or 2) put each one in one at a time, have it recalc parity based on that disk, and then go to the next disk... VERY slow and you are unprotected a LONG time.

[kizer]

Not quite. Even if the disk is plugged into an unRAID system, the disk must explicitly be mounted as read-only. Failure to do so will cause writes to the disks to occur which will invalidate your parity.

 

Unless you will never ever ever read the data written on those drives. Are you really asking for a parity protected array of WORN drives? Write Once Read Never.

 

I wouldn't mind a WORN drive externally since it would be an offline copy anyways, however I'm sure some would argue that the data kept offline is simply because it was offline because its either not accessed often or not as important as what is online in the first place. Or heck maybe its so important that its not kept online because if/when it was damaged it could be devastating as a loss.

 

I'd completely vote for a WORN setup, but then what would happen if you have say some OLD photos and then grandma presents you with some old photos of her parents and you want to store. Well in a WORN system you couldn't add without changing something in the array.

 

Would this be a setup where a second parity drive would be needed?

[lionelhutz]

I think bubbaQ hits the problem directly, in that a rebuild or parity check would require you to either install all the drives or cycle through them.

 

What about just using a eSATA to 5 SATA port multiplier module in a simple 4 or 5 disk case and then set it up for say RAID5? You could have it automatically mounted and accessable when plugged in. Should add maybe $40 per drive.

 

You could use a 2 bay dock and mirror the drives but that'll cost as much as building a server.

 

Peter

 

[queeg]

You could use a 2 bay dock and mirror the drives but that'll cost as much as building a server.

Peter

 

 

My math works differently.    But I'm not using my dock to create an alternative protected storage.  I'm using it to get non-array drives mounted quickly when I want to. 

[NAS]

What about just using a eSATA to 5 SATA port multiplier module in a simple 4 or 5 disk case and then set it up for say RAID5? You could have it automatically mounted and accessable when plugged in. Should add maybe $40 per drive.

 

 

You could but lets put another spin on this. I must have perhaps  30 mixed 500GB and 1TB drives now (and prob twice that smaller drives) as i have replaced drives with bigger ones. I am sure other people are in the same boat. Lets say I replaced them with 2TB drives that would be about $1000. OK so i could do that but in a few years time I will be spending another $1000 to do the same. So that's money to replace things that arent broke and whilst not cutting edge or especially large they still equate to 1000's of CD worth of data and that medium is still going strong.

 

Perhaps this is a flight of fancy but that is by definition what "for fun" threads are for.

 

Also perhaps VERY SLOW might be ok for some people.

 

You could do this another way and instead of using parity earmark a portion of each drives data (using some uknown method to clasify it). This data would be copied to the always online disk and when another transient disk is added this data is added to it. Remember we are still talking here about essentially closed off disk.. disks that data doesn't change at all or very very rarely.

 

I havent thought the number though properly but this could be a much simpler route where only a few drives would need plugged in to recover a drive.

 

 

 

 

 

[queeg]

So you are saying you want be able to pull a drive from the array and have the array treat it like it's just spun down?

[NAS]

Not really.

 

All I am saying is wouldn't it be nice to have offline disks protected.

 

How that could be done is the topic at hand. The world of possibilities is bigger than arrays and or parity

[queeg]

Not really.

 

All I am saying is wouldn't it be nice to have offline disks protected.

 

How that could be done is the topic at hand. The world of possibilities is bigger than arrays and or parity

 

Define protected in this sense.  Do you mean rebuildable onto another hard drive?  I believe the domain of possibilities is pretty limited.  Or are you referring to physical storage?

 

Since the offline data can't be modified is it possible to create layers of data onto a single parition by xor'ing (or something) in such a way as to run it in reverse would pull off the data again.

Example:

Take a new formatted partition and encode the data onto it from disk1 and then disk2 on top of that and so on.  Is it possible to reverse the encoding twice and recover the data for disk1?

 

EDIT:

parity(1) = parity(0) xor disk1 xor 1

parity(2) = parity(1) xor disk2 xor 2

 

I wonder if it's possible to recreate disk2 from the (parity(2) xor 2) and recreate parity(1) from (disk2 xor 2) which can then be used to recreate disk1.

 

The answer is no.  Parity depends on having a known value when parity is calculated.

So, offline disks can't be encoded in layers.

 

[NAS]

Trying to get the wording right because we keep coming back to traditional unRAID parity solutions which might not be the only way to do this:

 

"Protected Disk: A disk whos contents can be restored using any mechanism"