CrashPlan

[boof]

I hadn't considered the rc scripts... Since I've rebooted multiple times I imagine they're long gone by now. I'll reinstall to get a copy of them. I'd be inclined to give them a space in the directory structure I created and link them across to where CrashPlan expects to find them. Soft links strike me as a more elegant solution than moving files around. Is there a reason not to do it that way?

 

No downside I can think of, other than having to remember to create the symlinks at boot time.

 

 

[Rustbucket]

I hadn't considered the rc scripts... Since I've rebooted multiple times I imagine they're long gone by now. I'll reinstall to get a copy of them. I'd be inclined to give them a space in the directory structure I created and link them across to where CrashPlan expects to find them. Soft links strike me as a more elegant solution than moving files around. Is there a reason not to do it that way?

 

No downside I can think of, other than having to remember to create the symlinks at boot time.

 

 

That doesn't seem like a large price to pay. My go script already isn't stock, since I'm using it to launch SAB at boot. Cool!

 

I appreciate all the time you took on documentation boof - your wiki page made everything much easier!

[boof]

As much as I'd like to take credit, I don't think I did the wiki page. So whilst most of the content is probably based on my original write up someone else took the initiative and put it on the wiki - no doubt making it bit more coherent along the way

 

Regardless, glad you had no issues getting it running.

[owen1978]

My heads spinning just read upto page 22....

 

From what I gather people install CP server & Client on UNRAID

and then backup data from devices to the Crashplan server then server

backs up to the cloud.

 

I just want the CP Client so UNRAID just backups to the cloud?

So would i need to do the whole install still or just continue the guide

from the client install steps?

[boof]

Hopefully this will reduce rather than add to your confusion - there is no dinstinction with crashplan between client and server.

 

They're the same piece of software.

 

That single piece of software will behave as a backup client if you ask it to backup data up to crashplan themselves from the local machine.

 

Or it will act as a crashplan server if you point another crashplan client at it.

 

So yes, you need to follow the install guide as you end up with 'crashplan' on your machine. There are no seperate installation steps based on what you want to use it for.

 

The guide may be poorly worded wrt client and server terminology - it was done a while ago

 

So install crashplan as per guide and configure unraid to backup some of your data to crashplans server. That's probably the simplest usage scenario you could ever do with crashplan so hopefully it should be straightforward for you! Ignore any sense of using your own crashplan install as a backup server.

 

If you hit a problem post up when you get stuck and we'll have a look.

 

[hellbringer]

Well the guide could use some polishing, maybe some pictures but with some knowledge of packages & the GO script its possible to make it work for everyone.

When I have some spare time I will try to make it a bit clearer, maybe add some pictures/screenshots/notes/etc.

 

I just found out about CrashPlan and I was thinking: if only this would work with my unRAID server.

And it DOES! I've got to say: I love you boof 

(and Crashplan for they're great software/service)

[CrashnBrn]

Well the guide could use some polishing, maybe some pictures but with some knowledge of packages & the GO script its possible to make it work for everyone.

When I have some spare time I will try to make it a bit clearer, maybe add some pictures/screenshots/notes/etc.

 

I just found out about CrashPlan and I was thinking: if only this would work with my unRAID server.

And it DOES! I've got to say: I love you boof 

(and Crashplan for they're great software/service)

 

I would love to see a nice polished guide! I think that would get me to go ahead and install crashplan instead of having it on my todo (for over a month)!

[flipz]

I have a quick question about getting this running. I am new to UnRAID and haven't even finished my setup yet. I'm doing all of my reading and testing (thanks to virtual box) before I actually take the plunge.

 

I'm reading on the wiki here and one thing has me confused: http://lime-technology.com/wiki/index.php?title=CrashPlan#Satisfying_the_dependencies

 

It says

If you have unRAID version 4.5.5 or above running, just put those three packages above into the "extra" folder on the flash drive directory ("/boot/extra"). If your version is older than 4.5.5, put those three packages under "/boot/packages" and add these lines into your "go" script:

And is then followed up by 7 lines that you put in to your go script, all of which point to /boot/packages.

 

Sorry for the noob question but do I change all 7 lines to point to /boot/extra if that's where I have it, or does unraid do something to move stuff to /boot/packages when it's in ./extra? Hopefully someone can lay it out for me.

 

Thanks!

 

Edit:

Well I decided to just ignore that part for now and throw it right in to /boot/packages as I have plenty of other things running from there and I wanted to try this out. Everything is going smoothly and I have it installed but I'm stuck at the SSH part.

 

I have installed all of the required packages but I still can't ssh in to unraid. I just get connection refused.

root@Tower:~# installpkg /boot/packages/openssh-5.1p1-i486-1.tgz
Verifying package openssh-5.1p1-i486-1.tgz.
Installing package openssh-5.1p1-i486-1.tgz:
PACKAGE DESCRIPTION:
# openssh (Secure Shell daemon and clients)
#
# ssh (Secure Shell) is a program for logging into a remote machine and
# for executing commands on a remote machine.  It is intended to replace
# rlogin and rsh, and provide secure encrypted communications between
# two untrusted hosts over an insecure network.  sshd (SSH Daemon) is
# the daemon program for ssh.  OpenSSH is based on the last free version
# of Tatu Ylonen's SSH, further enhanced and cleaned up by Aaron
# Campbell, Bob Beck, Markus Friedl, Niels Provos, Theo de Raadt, and
# Dug Song.  It has a homepage at http://www.openssh.com/
#
Executing install script for openssh-5.1p1-i486-1.tgz.
Package openssh-5.1p1-i486-1.tgz installed.

 

 

Edit 2:

 

Well I have my actual server up and running now instead of playing around in virtual box. I installed the 3 required components (ssh/ssl/cpio) via unmenu instead and continued on with the guide and everything is working great. Thanks so much all!

[ScoHo]

Well the guide could use some polishing, maybe some pictures but with some knowledge of packages & the GO script its possible to make it work for everyone.

When I have some spare time I will try to make it a bit clearer, maybe add some pictures/screenshots/notes/etc.

 

I just found out about CrashPlan and I was thinking: if only this would work with my unRAID server.

And it DOES! I've got to say: I love you boof 

(and Crashplan for they're great software/service)

 

I updated the guide a bit on page 23 of this thread back in March: http://lime-technology.com/forum/index.php?topic=4008.msg108627#msg108627

 

No pictures though!

[boof]

I would suspect that the guides in this thread (both the originals and the ones that have been enhanced as above) would be a better resource than the wiki.

 

Interesting to see how many people go to the wiki for the guide though (and then come on the thread to ask questions )

 

Difficult to wade through this (long) thread I guess for all the info.

[tr0910]

I want to syncronize one unraid to another unraid box across town moving about 30 gb/week.  Is this a recommended use of crashplan?

 

Should I just install following the wiki?

 

Any special router configurations I need to do for security?

 

I didn't think I would be able to use Crashplan to backup one unraid to another unraid across town, but new internet upload caps have increased here in Canada, so now I can get 500 gb/month.  When I was limited to 30gb/month I wouldn't have been able to use it.

[aiden]

I want to syncronize one unraid to another unraid box across town moving about 30 gb/week.  Is this a recommended use of crashplan?

 

Crashplan is a backup application, and as such it will encrypt your data on the other machine so that you cannot read it without running a restore.

 

If you're just wanting to do a straight data copy and use the data in both locations, a scheduled rsync job is a much better idea.

 

Any special router configurations I need to do for security?

 

You'll want to set up a VPN.  There are several guides in the forums showing how to do so.

[Talsit]

Howdy folks.  Thanks to Boof and Pro for the fantastic guide!

 

I've been trying to get this to run all day and I'm not making progress.

 

I downloaded CP 3.0.3 for Linux and installed in /boot/packages/CrashPlan-install

 

I stepped through the procedure, used Putty to establish my tunnel, edited ui.properties and... nothing

 

The client software starts but doesn't connect.

 

I'm using 4.7 with Unmenu.  CP shows up in Unmenu as running.

 

It looks like I'm having an issue getting my Windows machine to talk to my Unraid box through Putty.

 

I can't find any useful logs on my CP install either.

 

Any suggestions?

[prostuff1]

Howdy folks.  Thanks to Boof and Pro for the fantastic guide!

 

I've been trying to get this to run all day and I'm not making progress.

 

I downloaded CP 3.0.3 for Linux and installed in /boot/packages/CrashPlan-install

 

I stepped through the procedure, used Putty to establish my tunnel, edited ui.properties and... nothing

 

The client software starts but doesn't connect.

 

I'm using 4.7 with Unmenu.  CP shows up in Unmenu as running.

 

It looks like I'm having an issue getting my Windows machine to talk to my Unraid box through Putty.

 

I can't find any useful logs on my CP install either.

 

Any suggestions?

Is ssh installed on the unRAID machine?  you do have to "set up" ssh the first time, before you can connect with crashplan... if I remember correctly.

[Talsit]

Pro,

 

I do have SSH installed on Ubraid box. I setup tunneling in Putty (L 4200 tower:4243) login to the SSH via Putty, start CP and CP attempts to connect to the server. It eventually times out.

 

If I comment out the servicePort=4200 in ui.properties CP immediately connects to CP.

[HAVOC]

Hey has anyone been seeing increased writes to flash drive? I installed to my cache drive .crashplan is the directory. Just curious because I keep getting this error

 

Sep 21 00:50:44 Tower sshd[24839]: error: Could not get shadow information for root (Errors)

Sep 21 00:50:44 Tower sshd[24839]: Accepted password for root from 192.168.1.13 port 1839 ssh2

Sep 21 00:50:44 Tower sshd[24843]: lastlog_filetype: Couldn't stat /var/log/lastlog: No such file or directory

Sep 21 00:50:44 Tower sshd[24843]: lastlog_openseek: /var/log/lastlog is not a file or directory!

Sep 21 00:50:44 Tower sshd[24843]: lastlog_filetype: Couldn't stat /var/log/lastlog: No such file or directory

Sep 21 00:50:44 Tower sshd[24843]: lastlog_openseek: /var/log/lastlog is not a file or directory!

 

[prostuff1]

Hey has anyone been seeing increased writes to flash drive? I installed to my cache drive .crashplan is the directory. Just curious because I keep getting this error

 

Sep 21 00:50:44 Tower sshd[24839]: error: Could not get shadow information for root (Errors)

Sep 21 00:50:44 Tower sshd[24839]: Accepted password for root from 192.168.1.13 port 1839 ssh2

Sep 21 00:50:44 Tower sshd[24843]: lastlog_filetype: Couldn't stat /var/log/lastlog: No such file or directory

Sep 21 00:50:44 Tower sshd[24843]: lastlog_openseek: /var/log/lastlog is not a file or directory!

Sep 21 00:50:44 Tower sshd[24843]: lastlog_filetype: Couldn't stat /var/log/lastlog: No such file or directory

Sep 21 00:50:44 Tower sshd[24843]: lastlog_openseek: /var/log/lastlog is not a file or directory!

 

That is just you logging in via ssh, no worries

[sacretagent]

you can get rid of these messages by adding this to the end of your go file

 

#create sshd lastlog
touch /var/log/lastlog

 

no more issues with these annoying errors

[HAVOC]

Thank you guys for your prompt reply. Crash plan is the next great thing since siced bread haha.

[dheg]

Hi guys,

 

I have a memory leak issue I can't get to sort out.

 

Summing up, the installation is done in an SDD drive outside of the array mounted on system startup with SNAP. I symlinked the /usr/local/crashplan and /var/lib/crasplan directories to the SSD relevant directories.

 

I'm backing up my Win 7 desktop to unRaid and everything seems to work fine. The backing directory is /mnt/users/crashplan.bak and is actually located on disk3 of my array. I can see the size of this directory grow as the back up goes, so my assumption is that it 's not writing to RAM.

 

During back up I'm monitoring RAM usage with free -m -s 5 and I can see the cached section grow until I run out of memory and CrashPlan then crashes. For what is worth, I have 4GB and standard use is around 500 MB before starting back up. Then cached memory grows, free memory shrinks and finally system crashes.

 

BTW, if I run echo 3 > /proc/sys/vm/drop_caches, the cached memory drops up to around 300MB, but then starts growing again, so the story goes.

 

Any ideas?

 

UPDATE:

It's been running all night without crashing but memory cached remains stable at 3,370MB and buffers at 172, leaving just 120MB of free memory. Is this normal? My back up is at 50% and running. I didn't runt this time the memory flush and I'll report if either it crashes before finishes or how the memory usage behave after completion

 

[dheg]

Backup finished ok with cache memory usage above 3300MB. I googled it and looks like the cache memory is actually a functionality of linux... so you all know now I'm a complete newbie 

 

I don't know why the crashplan engine crashed after first installation, but it looks like it's running fine now. Not that is important, but any idea?

 

BTW, 3h after finishing back up cache memory hadn't drop a bit so I run a cache memory flush with echo 3 > /proc/sys/vm/drop_caches to check what would happen. Surprisingly (for me), it didn't grew back... I have some questions though:

 

1. is this safe to run this command?

2. why while crashplan was  backing up cache memory grew above 3GB but it doesn't grow now that crashplan is iddle? (engine is still started)

3. why it didn't reduce the cache footprint when it finished?

4. and most important, should I add a cron job to periodically clear the cache memory?

 

thanks guys,

daniel

[boof]

I've not 100% thought through your problem - so I'm just throwing this in off the cuff but I think the response is sensible.

 

Backup finished ok with cache memory usage above 3300MB. I googled it and looks like the cache memory is actually a functionality of linux... so you all know now I'm a complete newbie 

 

The kernel will use memory as a filesystem cache where it can, so having all your memory used is normal. What is more important, as you've found, is to see how much of the memory is actually in use by the kernel and processes and how much is in the cache.

 

I don't know why the crashplan engine crashed after first installation, but it looks like it's running fine now. Not that is important, but any idea?

 

I can't answer that from the info you've given. Even though you've already looked into it this is almost exclusively caused by something writing to RAM somewhere. If you've ruled out your actual backup target (and make sure your /mnt/users/crashplan.bak is *really* physically ending up on a disk) then check also any logfiles, crashplan cache files etc too.

 

BTW, 3h after finishing back up cache memory hadn't drop a bit so I run a cache memory flush with echo 3 > /proc/sys/vm/drop_caches to check what would happen. Surprisingly (for me), it didn't grew back... I have some questions though:

 

1. is this safe to run this command?

2. why while crashplan was  backing up cache memory grew above 3GB but it doesn't grow now that crashplan is iddle? (engine is still started)

3. why it didn't reduce the cache footprint when it finished?

4. and most important, should I add a cron job to periodically clear the cache memory?

 

1 - Yes it's safe, though you may see a performance hit in the short term and more disk i/o than expected in the mid term. You will definitely notice if you're running the cache_dirs script.

 

2 - At a guess simply because there isn't much disk i/o when it's not backing up? So nothing further to cache. It will keep what its already cached / recycle the cache with new data as necessary. There is also a cap on how much cache it can use as it won't (/shouldn't) displace active memory being used by processes. i.e your cache usage can't keep growing it has to stop sometime.

 

3 - If it doesn't need the memory for anything else why would it? There would be no efficiencies in caching the data if it purged it from memory as soon as it was finished caching it. As above it might not be the same data in the cache though you'll find - relative to the memory needed by your running processes - the amount of cache memory should stay as high as possible. This is normal and good.

 

4 - Only if you want to forcibly reduce disk performance. I can't think of a reason you'd want to do this unless you find the memory scheduling done by default by the kernel isn't working in your particular usage. I'd find that scenario, for anything you could do with unraid on it's scale, unlikely.

 

The kernel will sort it all out. Perfectly normal to have all your memory in use including cache. Linux will drop stuff out the cache if it needs the memory for real processes.

 

Couple of examples this end of running machines :

 

             total       used       free     shared    buffers     cached
Mem:         16058       9763       6294          0        442       8544
-/+ buffers/cache:        776      15281
Swap:         1983          0       1983

 

16 gigs of memory but only 776 actually in use by anything running on the machine. Only 8 gigs cached so actually ~6 gigs memory completely free. That can probably tell you alot about the usage profile of that server!

 

            total       used       free     shared    buffers     cached
Mem:        516238       6488     509750          0       1343        524
-/+ buffers/cache:       4620     511617
Swap:        49151          0      49151

 

512G with 4.5 gigs actually used by processes. Only 2G cached and the rest free. Again you can tell something about the workflow by those numbers - hint on that box there is very little activity on a local filesystem so not much to actually cache..

 

And from one of my unraid servers :

 

             total       used       free     shared    buffers     cached
Mem:          4049       3934        115          0        176       2575
-/+ buffers/cache:       1182       2867
Swap:            0          0          0

 

4 gigs with just over 1 gig used by applications. The rest in buffers. Perfectly normal as this is a fileserver so lots of things to cache. And hopefully for you reassuringly similar to your usage..?

 

[nick5429]

So I was emailing a bit with Crashplan support trying to get a few things clarified about the private data key options, and the CSR mentioned that in certain scenarios, incorrectly entering your private data key will cause your backup data to be deleted.  

 

I asked him to clarify and elaborate. Response below.

 

If you perform the Adoption procedure, the CrashPlan software will ask for the encryption key for the archive that you are adopting. If you mis-enter the key, then the archive will be cleared.

 

Just thought you all might find this ... interesting.

[dheg]

I've not 100% thought through your problem - so I'm just throwing this in off the cuff but I think the response is sensible.

 

Backup finished ok with cache memory usage above 3300MB. I googled it and looks like the cache memory is actually a functionality of linux... so you all know now I'm a complete newbie 

 

The kernel will use memory as a filesystem cache where it can, so having all your memory used is normal. What is more important, as you've found, is to see how much of the memory is actually in use by the kernel and processes and how much is in the cache.

 

I don't know why the crashplan engine crashed after first installation, but it looks like it's running fine now. Not that is important, but any idea?

 

I can't answer that from the info you've given. Even though you've already looked into it this is almost exclusively caused by something writing to RAM somewhere. If you've ruled out your actual backup target (and make sure your /mnt/users/crashplan.bak is *really* physically ending up on a disk) then check also any logfiles, crashplan cache files etc too.

 

BTW, 3h after finishing back up cache memory hadn't drop a bit so I run a cache memory flush with echo 3 > /proc/sys/vm/drop_caches to check what would happen. Surprisingly (for me), it didn't grew back... I have some questions though:

 

1. is this safe to run this command?

2. why while crashplan was  backing up cache memory grew above 3GB but it doesn't grow now that crashplan is iddle? (engine is still started)

3. why it didn't reduce the cache footprint when it finished?

4. and most important, should I add a cron job to periodically clear the cache memory?

 

1 - Yes it's safe, though you may see a performance hit in the short term and more disk i/o than expected in the mid term. You will definitely notice if you're running the cache_dirs script.

 

2 - At a guess simply because there isn't much disk i/o when it's not backing up? So nothing further to cache. It will keep what its already cached / recycle the cache with new data as necessary. There is also a cap on how much cache it can use as it won't (/shouldn't) displace active memory being used by processes. i.e your cache usage can't keep growing it has to stop sometime.

 

3 - If it doesn't need the memory for anything else why would it? There would be no efficiencies in caching the data if it purged it from memory as soon as it was finished caching it. As above it might not be the same data in the cache though you'll find - relative to the memory needed by your running processes - the amount of cache memory should stay as high as possible. This is normal and good.

 

4 - Only if you want to forcibly reduce disk performance. I can't think of a reason you'd want to do this unless you find the memory scheduling done by default by the kernel isn't working in your particular usage. I'd find that scenario, for anything you could do with unraid on it's scale, unlikely.

 

The kernel will sort it all out. Perfectly normal to have all your memory in use including cache. Linux will drop stuff out the cache if it needs the memory for real processes.

 

Couple of examples this end of running machines :

 

             total       used       free     shared    buffers     cached
Mem:         16058       9763       6294          0        442       8544
-/+ buffers/cache:        776      15281
Swap:         1983          0       1983

 

16 gigs of memory but only 776 actually in use by anything running on the machine. Only 8 gigs cached so actually ~6 gigs memory completely free. That can probably tell you alot about the usage profile of that server!

 

            total       used       free     shared    buffers     cached
Mem:        516238       6488     509750          0       1343        524
-/+ buffers/cache:       4620     511617
Swap:        49151          0      49151

 

512G with 4.5 gigs actually used by processes. Only 2G cached and the rest free. Again you can tell something about the workflow by those numbers - hint on that box there is very little activity on a local filesystem so not much to actually cache..

 

And from one of my unraid servers :

 

             total       used       free     shared    buffers     cached
Mem:          4049       3934        115          0        176       2575
-/+ buffers/cache:       1182       2867
Swap:            0          0          0

 

4 gigs with just over 1 gig used by applications. The rest in buffers. Perfectly normal as this is a fileserver so lots of things to cache. And hopefully for you reassuringly similar to your usage..?

 

 

thanks a lot boof, after some reading I think I understand better the cached memory issue.

 

It's been around 1 month and crashplan is working fine so far, although I ran into other issues (for those interested see this http://lime-technology.com/forum/index.php?topic=16534.0)

 

 

[dheg]

So I was emailing a bit with Crashplan support trying to get a few things clarified about the private data key options, and the CSR mentioned that in certain scenarios, incorrectly entering your private data key will cause your backup data to be deleted.  

 

I asked him to clarify and elaborate. Response below.

 

If you perform the Adoption procedure, the CrashPlan software will ask for the encryption key for the archive that you are adopting. If you mis-enter the key, then the archive will be cleared.

 

Just thought you all might find this ... interesting.

 

this actually happened to me when I started using crashplan (about 1.5 years ago). I though I had lost about 200GB... imagine the pain  :'(

 

However, i immediately stop the backup and got in contact with customer service. A really nice chap called Michael  helped me through and got me back the lost files. This was the minute in which I converted my trial into a premium account. I can only talk wonders of its customer service.

 

PS: and I'm not affiliated with them in any way