I suppose this is a feature request unless someone points me in the proper direction to where it is already discussed or implemented, at which time I'll prepare the foot...  Did some quick searching and didn't see this mentioned.

 

Lots of discussion it seems about two parity drives to add tolerance for a two drive failure.

 

In setting up a trial unraid server for the first time, the thought crossed my mind that I wouldn't really need or care for dual parity drives for much of my storage needs, the vast majority going to house my considerable DVD collection.  I can always re-rip the disc if I lose a data drive somehow.

 

But there are some files, like family photos or documents, that would be nice to have an additional degree of protection so that I don't feel the need to duplicate on another machine in my house and can only worry then about off-site backup.

 

Since these files are typically much smaller than a big movie/media collection, I wouldn't mind the small loss of disk space by simply making a duplicate copy within the server.  I think my photos and home videos total about 100GB currently, documents a small sliver of that, and I'd rather lose 100GB of storage on an existing drive than dedicate another 2TB or larger (especially in the future) drive to extra parity. 

 

Using the existing include/exclude options I think I can figure out how to create two separate shares that are on mutually exclusive data drives, and then I guess every time I update and add information I'll have to do it for both shares (unless this has been discussed and there is a script to do this?).  But wouldn't it be nice if in addition to the include/exclude etc. share options there is a simple "duplicate" option that forces a duplication of all data on that share whereby the two copies must reside on two separate data disks... this would be transparent then to the user, with but a single share and seeing only one copy of the data, and the duplication being done in the background?  A raid 0 sort of thing I suppose, but in the unRaid way where you could then selectively decide which shares to duplicate.

 

Retarded?  Already done?

Not retarded and actually a nice idea... It does not fill the same need as the double parity though; if the two drives fail that you keep the original and copy-share on you are still out of data.. But ofcourse it does reduce the likelyhood of this happening...

 

Nice thing is that you can make this work without Tom actually building something.. And if some smart guy takes a couple of hours it could even be made into a nice plugin:

 

- Create an extra share that will hold your share-copy (make sure it is not using the same drives as the original one)

- Create  a copy command running once a day (put in in Cron):

 

cp -r -u /mnt/user/MyDocs /mnt/user/Mydocs2

 

The -r will copy including subdirs and the -u makes sure you only copy stuff changed or not existing in the MyDocs2 location.

 

Crontab line should read (please check, I am rusty here):

 

0 2 * * * cp -r -u /mnt/user/MyDocs /mnt/user/Mydocs2

 

This would copy the Mydocs dir to its shadow every night at 02:00. You can do it more often if you want.

 

You would make sure that the Mydocs2 share is not exported (just to be sure you do not mess with it by accident).

 

If you are really paranoid you could even create a Mydocs3 that you copy every week to even another disk...

 

(setting up crontab is explained in the fora: http://lime-technology.com/forum/index.php?topic=2116.msg15705#msg15705 )

I think you'd want to use rsync to synchronize the files and not copy.

That would be even quicker, not sure about syntax though ( I was actually kind of proud i remembered the cp options ;-)

 

Also had a bit more thought about this... Something like this would also significantly reduce the chance of data-loss since this would also guard against accidental deletions or accidental file corruption.. This got me thinking towards hidden shares called:

 

<Sharename>_Yesterday

<Sharename>_LastMonday

<Sharename>_LastMonthStart

 

With appropriate running rsync actions on a daily, weekly and monthly basis..

 

 

Your replication could be tied into unraid using http://www.greyhole.net/ presuming the samba module would build (I don't see why it wouldn't).

 

There would be a bit of integration work and making nice on the web interface but it would be do-able. I think it would be more likely to come from the community than from limetech though.

 

What would be nice to come from limetech, expanding on your hidden share idea, would be 'proper' filesystem level snapshots that could be taken once per day (or on a configurble schedule).

 

I don't think reiserfs support snapshots out the box so it would either mean fiddling that code, moving to another filesystem or 'doing something else'. Possibly even just getting clever with hardlinks. Unlikely to happen in anything but the longest term I would imagine though

 

Shame as it would be nice and I suspect, for a lot of people, wouldn't be very resource intensive given the relatively small amount of churn likely on an unraid box.

rsync can be used to create the dated snapshots or replicate areas using the technique discussed in

http://blog.interlinked.org/tutorials/rsync_time_machine.html

 

I run a similar script to "backup" files on unRAID diskA to a dated snapshot of the files on unRAID diskB

I have now implemented the following on my unraid system:

 

SITUATION:

I allready had a "Users" share that holds all my personal documents, as stated by the previous poster these are files you do not want to use but they actually use very little diskspace, that makes it not a problem at all to store duplicate copies on a different disk that are synchronised in a specific schedule. This enhances file security on unraid twofold:

 

Why ?

1) Disk failure will have an extra layer of security for these important files (ofcourse mainly thru parity but should two drives fail then there still is a copy on the other disk)

 

2) SUF (Stupid User Failure) You will always have a copy of your most important files readily accessible should you delete something by accident.

 

How ?

1) Create an extra share on your unraid system, make sure data is on a different disk then where the data is on that you want to extra-protect (I have my important data in "USERS" and I have created an extra share called "USERS_YESTERDAY");

 

2) Mark the share "Not exported" (that will make sure you do not work on the copy share by accident)

 

3) Edit your GO script and add the following line at the bottom (ofcourse change share names according to your specific setup):

 

echo "rsync -avz /mnt/user/Users /mnt/user/Users_Yesterday" | at 23:00

 

This will make sure the two shares will be mirrored every evening at 23:00.

 

 

Stuff to consider:

 

- The "yesterday" share will grow only since files are not deleted if they are removed from the basic share, I did this on purpose, I rarely delete files from this directory and not having deleted files removed from the Yesterday share will be good for me, an accidental deleted file will still be there even months after I accidentily deleted it. If you do not want this then adding --delete-- to the command in the go script will also remove files in the Yesterday share that are no longer present in the basic share (I do not recomment however, a catastrophic user failure deleting all files in the basic dir will result in an empty Yesterday share after 23:00).

 

- In case of a two drive failure with your extra protected data on a failing drive you will still have these files on the Yesterday share, unless ofcourse your Yesterday share is located on the second drive that fails... You can go as far as You want with this and even add a second yesterday share..

 

Possible improvements:

 

With a little more tinkering you could also have a "USERS_LASTWEEK" and "USERS_LASTMONTH" share. For me that will be a bit over the top but it is as easy as the above with the exception of the need to use CRON instead of the "AT" command.

 

Something I have no idea of doing but I do think would be nice is adding this to the interface as a plugin.. You could have a standard "Yesterday" share created for every share that got an extra checkmark (kind of windows home server like).

The script you posted will run ONCE, at 23:00 each time you reboot the server.  It will not run daily, it does not run a second time on the next day at 23:00. (unless you reboot the server daily, and the "at' script is scheduled when the "go" script line is invoked again )

 

You'll need to create a cron entry for that same task, and not use "at"  It is not that hard, so you are close to your solution.

 

Joe L.

Something like this will work  (entered all on a single line in the "go" script, or on the command line)

(crontab -l | grep -v Users_Yesterday; echo "23 00 * * * rsync -avz /mnt/user/Users /mnt/user/Users_Yesterday >/dev/null 2>&1") | crontab -

 

The initial "crontab -l | grep -v Users_Yesterday"

will output the existing crontab entries without the line containing "Users_Yesterday".  The echo then appends the desired command and it is all submitted back to a second "crontab -" that installs the new command.

 

Once run, you'll see your rsync command if you type

crontab -l

 

It will then run every night at 23:00.

 

Joe L.

Seriously.. This is exactly why I like products like this, I am improving and learning in the process, thanks !!

Cool... thanks guys.  Still learning linux in general, so it would have taken me quite a while to stumble onto the proper way to create this solution on my own.

 

Curious about one comment...

It does not fill the same need as the double parity though; if the two drives fail that you keep the original and copy-share on you are still out of data.. But ofcourse it does reduce the likelyhood of this happening...

Ah.  This is because unRaid can't currently protect against a two drive failure, of any combination?  So if you lose two data drives, you still have readable files on all the other data drives but there is no way to rebuild from parity, right?  I guess if the two drives you lost were one of the data and the parity, you'd be good, so this is a little better protected than just one copy.... increases the odds as Helmonder said.

 

Didn't think about that.  I suppose the quick solution is to just add an additional level of share, i.e. "original", "copy", and "copy without deletions" or some sort so that there are three physical discs (or however many you want depending on paranoia level) with copies.  A bit of a waste of space but again since these aren't typically big files I would be OK with that.  The more elegant solution would be for any designated share to be duplicated by unRaid to have a dedicated space in parity that is essentially the third copy, separated from the other parity data.  In that case, anything in this "protected" space would be recoverable only from parity as it would essentially be a duplicate copy as well, not true parity data of the "x" number of other copies on the data drives.  I'm sure that would take quite a bit of fiddling with the core of unRaid, and I assume it isn't something likely to be on the roadmap (ever, I'm sure ), so I'll be good with the pointers already provided here to work out my own best solution for my needs.

 

Thanks again.

And thank you !  I actually learned something fiddling with your idea and am using it actively right now... !