June 22, 201115 yr So I've got a topic of discussion that I would like to start over here on the unRAID forums as it's the only technical board that I frequent and it has a very knowledgeable user base that might be able to help and/or point me in a direction. So from the topic I am very interested in discussing and understanding how I can create a solution to a problem that I'm developing... Yes, that is correct, I'm creating my own problem, I'm such an ass Here is what I am attempting to do Goal: Create 3 offsite unRAID and/or Linux RAID's that will all mirror from my unRAID server at my house. The directory in question will be an ever expanding directory, whichi is currently 15TB in size. Notes: 1. All 3 of the yet to be built and deployed boxes will have the initial 15TB pushed onto them locally before they are sent to their offsite locations. 2. The new files will range from 20-30GB when added (we are talking about raw video masters and that is why I want to have remote offsite backup solutions. Irreplaceable files here) 3. My location has an unlimited 22/5Mbps connection 4. I want to find a better solution that Rsync. 5. I'm far from a linux guru, but I am continuing to learn and understand the OS The solution that I dreamed about on my drive from Portsmouth, VA to Raleigh, NC after a concert @ 3am would be something similar to Rsync, but make use of p2p so that way all are sharing the bandwidth distribution. So with that said - What strategy would you take to accomplish this? Yes, I could setup Rsync's, but it's just not bandwidth sensitive talking about multi clients with large files that will need to be mirrored. I'm very interested to see what you guys think as I begin marching down this path...
June 22, 201115 yr Author If this was my iPhone I would just assume 'there's an app for that'. I also don't think Crashplan is an available option as again I need to figure out some sort of p2p to built in...
June 22, 201115 yr I was just about to suggest Crashplan, as it will backup to multiple destinations, has deduplication, etc. I've never heard of anything that will do what you want. At some point these other nodes need to verify that the data they're backing up to each other is correct... How often are you going to be adding 20-30GB? Edit: And what effective upload rate do you get?
June 22, 201115 yr Author I was just about to suggest Crashplan, as it will backup to multiple destinations, has deduplication, etc. I've never heard of anything that will do what you want. At some point these other nodes need to verify that the data they're backing up to each other is correct... How often are you going to be adding 20-30GB? Edit: And what effective upload rate do you get? I should add that I would like the remote servers to have usable files, in that these servers will be at friends and I want them to be able to read these files. I do not believe that is something that Crashplan supports. My upload here is 5Mbps. I can't say how often I will be adding these large files, but lets say that as of today and if I had these servers deployed and needed to mirror the content, we are talking about roughly 500GB with maybe another 100-200GB pending in the next month.
June 22, 201115 yr Author I just had my mind blown.... http://www.cs.cmu.edu/~dga/papers/dsync-usenix2008/ A linux program called dsync seems to be very close to what I'm looking for to solve my problem. Anyone ever heard of this?
July 19, 201114 yr So, TeamHood, did you ever find a solution to this? I'm working on RSync over OpenVPN (or Hamachi) but haven't got it all worked out yet. Would love some help/advise. (I too need usable files on each end) Russell
July 24, 201114 yr So I've got a topic of discussion that I would like to start over here on the unRAID forums as it's the only technical board that I frequent and it has a very knowledgeable user base that might be able to help and/or point me in a direction. So from the topic I am very interested in discussing and understanding how I can create a solution to a problem that I'm developing... Yes, that is correct, I'm creating my own problem, I'm such an ass Here is what I am attempting to do Goal: Create 3 offsite unRAID and/or Linux RAID's that will all mirror from my unRAID server at my house. The directory in question will be an ever expanding directory, whichi is currently 15TB in size. Notes: 1. All 3 of the yet to be built and deployed boxes will have the initial 15TB pushed onto them locally before they are sent to their offsite locations. 2. The new files will range from 20-30GB when added (we are talking about raw video masters and that is why I want to have remote offsite backup solutions. Irreplaceable files here) 3. My location has an unlimited 22/5Mbps connection 4. I want to find a better solution that Rsync. 5. I'm far from a linux guru, but I am continuing to learn and understand the OS The solution that I dreamed about on my drive from Portsmouth, VA to Raleigh, NC after a concert @ 3am would be something similar to Rsync, but make use of p2p so that way all are sharing the bandwidth distribution. So with that said - What strategy would you take to accomplish this? Yes, I could setup Rsync's, but it's just not bandwidth sensitive talking about multi clients with large files that will need to be mirrored. I'm very interested to see what you guys think as I begin marching down this path... Make sure of course when you build your unRAID server you build 2 of them. This way you can at least have a complete duplicate locally. Unless your home/office is prone to fires and disasters?
Archived
This topic is now archived and is closed to further replies.