I would love to "fold" a drive in and out of the parity protected array without destroying the data for the sheer backup purpose alone.

 

This would let me put a drive into the array with files and have it protected and accessible.

Then rsync other files to backup, then remove it from the array without destroying the data.

 

As far as moving data in a drive loss, rsync and user shares could do a nice job of moving files while letting usershare logic decide where files could go. I don't think I would use that part of this request.

 

I do prefer to organize my data and directories where I remember them. No matter how vast the collection.

There is always slocate when I need to find a file 

I do prefer to organize my data and directories where I remember them. No matter how vast the collection.

There is always slocate when I need to find a file 

 

Purely out of curiosity, why do you care where the files end up?  The same reason as prostuff1 and/or Joe L.?

I would love to "fold" a drive in and out of the parity protected array without destroying the data for the sheer backup purpose alone.

The only way I can think of to "move" a drive in and out of the parity protected array without losing parity protection for any length of time, and without zeroing all the data would be if "it" existed in the array as a parity protected "mirrored pair of drives".  Then the "mirrored pair" could be removed, or added in a single operation, since as an identical pair they would have no effect on the overall parity calculations.

 

I know it is inefficient, and uses up a second drive (identically sized), but it could be done logically.  The two drives would need to be treated separately as far as the parity calcs go, but have the contents automatically "mirrored" by the unRAID driver when writing.  An advantage might be that the pair might be treated as raid1 when not part of the parity protected array, for extra safety.

 

Obviously, unRAID does not do this today....  Adding or removing a drive is a lengthy process.

 

Joe L.

I would love to "fold" a drive in and out of the parity protected array without destroying the data for the sheer backup purpose alone.

 

This would let me put a drive into the array with files and have it protected and accessible.

Then rsync other files to backup, then remove it from the array without destroying the data.

 

I understand what you want, but I fail to see its usefulness.

 

To fold a drive out, means rewriting parity --- slow.

To fold a drive with data in, means rewriting parity --- slow.

Both of these processes are very dangerous -- if a hiccup happens, parity is toast.... half of it is with the drive and half is without.... so you can not "trust my parity" and get it back... you have to to a recalc unless some sophisticated progress tracking was implemented (and I wouldn't' trust it).

 

Why does this removable drive need to be in the array if you are just going to rsync it?  Just add a drive to the server outside the array, mount it, rsync to it, and remove it.  That would be MUCH faster and much less risky.

 

Or do a rsync to a drive already in the array, then copy it to a removable drive outside the array, then delete from the array drive to free up space if needed.

To fold a "mirrored pair of drives" out, means parity is completely unchanged.  --- fast,

    since no parity re-calc need be performed, only the config/super.dat need be updated to remove the identical pair of drives.

To fold a "mirrored pair of drives" in, means parity is completely unchanged.  --- fast,

    since no parity re-calc need be performed, only the config/super.dat need be updated to add the identical pair of drives.

 

Both of these processes are very safe-- if a hiccup happens while writing to the super.dat file, press "restore" to re-create a new super.dat file. (Parity will be correct, so you can use the "trust" process)

 

However, I agree with BubbaQ.  You can rsync from/to a drive outside of the array far more easily, and do it today.

I would love to "fold" a drive in and out of the parity protected array without destroying the data for the sheer backup purpose alone.

 

This would let me put a drive into the array with files and have it protected and accessible.

Then rsync other files to backup, then remove it from the array without destroying the data.

 

I understand what you want, but I fail to see its usefulness.

 

To fold a drive out, means rewriting parity --- slow.

To fold a drive with data in, means rewriting parity --- slow.

Both of these processes are very dangerous -- if a hiccup happens, parity is toast.... half of it is with the drive and half is without.... so you can not "trust my parity" and get it back... you have to to a recalc unless some sophisticated progress tracking was implemented (and I wouldn't' trust it).

 

Why does this removable drive need to be in the array if you are just going to rsync it?  Just add a drive to the server outside the array, mount it, rsync to it, and remove it.  That would be MUCH faster and much less risky.

 

Or do a rsync to a drive already in the array, then copy it to a removable drive outside the array, then delete from the array drive to free up space if needed.

 

While the drive is in the array, I want to use it and have it protected.

When I choose to take the drive off site, I want to disconnect it from the array and remove it from parity.

I don't necessarily want to rsync it onto the array drives.

it could be in the array for a week or a month acquiring rsync backups from another host.

Adding or removing a drive from the array is not that lengthy for me.

If the drive is smaller then others, then parity only needs to be calculated for that drives length the rest would be assumed to be 0.

 

In any case, it's not a matter for lengthy discussion.

Just the removal of a drive from the array keeping parity intact without physically zeroing out the drive would work.

I suppose some fancy mode that would just read the data drive & parity drive and zero out the specific bits from the data drive would be pretty fast. (as fast as two drives can read/write).

Just the removal of a drive from the array keeping parity intact without physically zeroing out the drive would work.

 

It would go slow.  Keeping parity intact would proceed at the rate of parity drive read/writes.... which good systems have reported in the range of 15-23 MB/sec, or about an hour and a half per 100GB.

 

During that time, parity might be valid, but I would consider it precarious because during the process, parity is part with the subject disk, and part without.  If you had a data drive failure doing that process, it could be handled, but it would be a mess.  If you have a power outage or other shutdown event, parity is invalid, you have no protection, and you have to recalc parity when you come back up.

 

I don't see where this gets you anything better than rsync to a drive already in the array, then copy or rsync that to a removable drive that is not in the array.  I do that now with Encase images and indexes.... stuff I am currently working on is on the desktop machine, and periodically synced to unRAID, and when a case is over, what is on unRAID is offloaded to another drive outside the array, and sent to offsite storage.  Then the files on unRAID are deleted.  Plus by using a USB removable drive, it can all be done w/o starting or stopping the array.

One point however, I *do* like the idea of ZEROing out the data and removing a drive from the array.  I've done it using the previously mentioned instructions, and agree that would be a good feature to have built in.  This is especially useful when you are 1) limited in slots (HW or unRAID license) and are replacing drives with large ones, 2) proactively replacing old drives due to age.  Plus it is a secure erase, so I don't have to remount the drive to wipe it before it is retasked.

 

Of course, utility functions like this can easily be added with the web interface due in unRAID 5.0.