mirroring important files on multiple drives

[charsi]

Hi

 

I am currently using WHS with Drivepool. It has an extremely useful feature that allows marking a few folders to be mirrored on multiple drives. I use this to maintain multiple copies of my family pics and videos folders on 2x data drives. This gives me raid 1 level protection for the really important data while leaving parity to handle the non crucial stuff (movies, etc).

 

It will be great to have Unraid provide this additional level of security too.

 

EDIT: Found a number of threads asking for the same feature..

Topic: The ability to mirror data shares/folders on UnRAID

Topic: Mirror user share?

Topic: Feature Request - Data/Disk Mirroring

Topic: mirror a folder

Topic: Create a new User-Share feature, copy across multiple drives

Topic: Multi-disk copies of "important" data

[gundamguy]

Hi

 

I am currently using WHS with Drivepool. It has an extremely useful feature that allows marking a few folders to be mirrored on multiple drives. I use this to maintain multiple copies of my family pics and videos folders on 2x data drives. This gives me raid 1 level protection for the really important data while leaving parity to handle the non crucial stuff (movies, etc).

 

It will be great to have Unraid provide this additional level of security too.

 

Relevant thread- The ability to mirror data shares/folders on UnRAID

 

You can totally accomplish this in unRAID. If the same file exists on multiple disks, it only shows up once in the user shares.

 

i.e. 
/mnt/disk1/Movies/Movie.mkv 
/mnt/disk2/Movies/Movie.mkv

Will show up only once in the useshare
/mnt/user/Movies/Movie.mkv

 

Which if I understand correctly this accomplishes what you want.

 

(Also I'm not sure I agree with this backup philosophy since your 3rd backup should ideally be located in a different machine, or the cloud or someplace that won't be lost to fire, theft, or flooding... IMO.)

[hugenbdd]

 

You can totally accomplish this in unRAID. If the same file exists on multiple disks, it only shows up once in the user shares.

 

i.e. 
/mnt/disk1/Movies/Movie.mkv 
/mnt/disk2/Movies/Movie.mkv

Will show up only once in the useshare
/mnt/user/Movies/Movie.mkv

 

 

Any idea which disk unRAID chooses to read from first?

[itimpi]

 

You can totally accomplish this in unRAID. If the same file exists on multiple disks, it only shows up once in the user shares.

 

i.e. 
/mnt/disk1/Movies/Movie.mkv 
/mnt/disk2/Movies/Movie.mkv

Will show up only once in the useshare
/mnt/user/Movies/Movie.mkv

 

 

Any idea which disk unRAID chooses to read from first?

I believe that unRAID takes the lower number disk in the case of a name collision.

 

However I am doubt that this feature would satisfy you requirements because:

• Making a copy would be a manual process
  
• UnRAID will noit automatically keep the copies aligned.  Therefore if you updated one copy it would be up to you to make sure that the other copy was updated.
  

[charsi]

 

However I am doubt that this feature would satisfy you requirements because:

• Making a copy would be a manual process
  
• UnRAID will noit automatically keep the copies aligned.  Therefore if you updated one copy it would be up to you to make sure that the other copy was updated.
  

 

Yep, having this as a built in feature with Unaid automatically duplicating the data in the folders would be nice.

[itimpi]

 

However I am doubt that this feature would satisfy you requirements because:

• Making a copy would be a manual process
  
• UnRAID will noit automatically keep the copies aligned.  Therefore if you updated one copy it would be up to you to make sure that the other copy was updated.
  

 

Yep, having this as a built in feature with Unaid automatically duplicating the data in the folders would be nice.

I do not see there being enough demand for such a feature to ever justify implementing it.

[gundamguy]

I do not see there being enough demand for such a feature to ever justify implementing it.

 

I agree.

 

Also no offense to the OP, but I think you are approaching this the wrong way.

 

1) Dual Parity coming Soon^TM will give you the same level of redundancy.

2) If the data is actually critical / irreplaceable, then I suggest an offsite backup (to reduce the risk of Acts of God destroying your system and all of the data), backup disks (mount using unassigned devices and copy onto an external, or any number of more advanced ways to pull this off), or some other way of backing up your system where the critical data isn't stored on the same system as the other copy. There is a lot of ways to accomplish this, and I think nearly all of them would be better then having a second set of files in the same unRAID array.

 

 

[kizer]

Easy.

 

/disk1/Movies/

/disk2/Movies/

/disk3/Backup/Movies/

 

From my experience you don't want copies of the same files in the same shares because logs start to complain. Maybe its different with newer versions, but in the past I saw it a few times.

 

I have my Home Videos/Photos backed up on two drives as a just in case, but I as well keep a copy offline aka on a Portable drive on a shelf. My thought process is I can easy backup my photos to two locations in my workflow and once a month or so I copy to the portable.

[charsi]

I do not see there being enough demand for such a feature to ever justify implementing it.

 

To me this was the only protection I had for my pictures for a number of years before I bothered with parity (via flexraid). I still consider this as my first line of defence if something goes wrong. I also have most of the pictures uploaded online but I can't do it very regularly as the internet speeds aren't great where I am located.

 

Completely understand that there needs to be enough demand for developers to spend any time on this.

 

 

Also no offense to the OP, but I think you are approaching this the wrong way.

 

1) Dual Parity coming Soon^TM will give you the same level of redundancy.

 

True but that would require throwing another 3TB disk in the array to protect a couple of folders which are only ~150GB. With my current solution I can have drivepool duplicate the folders on every drive in the array (5 at the moment), giving me protection against 4x drive failures. If I really wanted protection against multiple drive failures I would like to do be able to chose which data actually warrants that level of protection and then do it as efficiently as possible.

 

 

EDIT: Found a number of threads asking for the same feature..

Topic: The ability to mirror data shares/folders on UnRAID

Topic: Mirror user share?

Topic: Feature Request - Data/Disk Mirroring

Topic: mirror a folder

Topic: Create a new User-Share feature, copy across multiple drives

Topic: Multi-disk copies of "important" data

[WeeboTech]

You could probably do this via USB and utilize rsync to synchronize the desired directories.

Doing this to two drives on the same array is do-able, but not recommended. (unless you want to keep dated versions of these directories as backups).

 

In the event of a catastrophe, you want to grab the smallest device with the most critical files.

 

If you really wanted to go with a RAID 1 solution, an areca controller could mirror a specified drive at the hardware level.

Then there is the software BTRFS raid1 solution which sort of mimics the drive pool feature.

It's my understanding with BTRFS raid1 that no matter how many drives are in a pool, there is guaranteed at least 2 copies of the file.

 

Still with ~150gb of files/directories to backup, an external 2.5 Laptop USB drive seems like a simple solution.

 

If using rsync at the linux level is too much, there's a cool windows program called puresync that can synchronize certain directories when a specific USB drive is plugged in.

[NAS]

I am with the OP that this idea actually fits quite nicely with the core idea that uNRAID as a bunch of disks that also may have parity i.e. the unique unRAID model. I dont see dual parity changing this core idea.

 

In fact i believe this is the 3rd time it has been suggested and discussed.

 

cron + rsync gives you a reasonable match to the request although there is nothing special about that, its merely a good backup.

 

To be seamless this needs to be lower level (obviously).

 

Let me throw in another idea to compliment it. If we set a sane limit on the folder size you could in theory also have change control

 

e.g. https://github.com/PressLabs/gitfs

 

that is a whole new level of useful

[billington.mark]

Yes Dual parity is the same level of redundancy, but not everyone will want to sacrifice a second disk for parity if they only want to keep a single share or folder duplicated.

I think the best way to achieve this would be to create a new share called "backup" or something similar and having a sync script set on a schedule to update the folder as someone has stated earlier in the thread.

[charsi]

I think it just needs to be something Unraid does at the same time when the parity is calculated or data is copied over from the cache disks.

 

The use case for this seems to be quite common - NAS full of movies, tv shows, etc taking 90%+ of the used storage space and one extremely crucial 'Pictures' folder. If Unraid had this I think a large portion of the userbase will be making use of it.

 

I also like NAS's idea of adding version control (or snapshots) which will also protect users against accidental deletions etc.

[WeeboTech]

Samba has version control modules, although I do not know much about it.

 

Perhaps this could be used in line with NAS's accelerator drive functionality.

 

The accelerator suggestion was that files created by certain rules are placed in a specific location.

Tom suggested at some point, files created that match a rule could have a script or program run against them.

 

Expanding this further, those rules and/or programs/scripts could do double duty of storing on an accelerator drive and/or rsyncing a copy to a backup area.

 

So perhaps we need to lobby for the user share file system to support 'active configurable rules/expressions' that will run commands when matched.