CrashPlan

[kizer]

I sorta feel like an idiot. I followed the howto to the letter I belive and when I tried to copy files to my unRAID server it shows a progress bar, but when I look on my unRAID there is nothing there. Did I miss a permissions step or something? I even told it to store files /mnt/disk4/MISC then tried /mnt/usr/MISC

 

More or less I installed it twice just to see if I could figure out where I was making the mistake. I never tried to reboot to make sure the settings stuck, but I tried to at least make a successful copy and restore, but nothing was ever copied.

 

Yes I was able to login via my windows machine to the Java of my Unraid, but I wasn't sure if I was supposed to set something up on that end or it was all accomplished via the command line during install.

[boof]

I think you'll have to explain in a bit more detail what you've done and where you're getting stuck. I can't quite figure it out from the above I'm afraid

[kizer]

I followed your install process to the letter. I think the only thing I did different was change the location of where my files where to be stored. /mnt/disk4/MISC

 

I changed the port forwarding like you suggested and even logged into my unRAID console and established an account and password that was the same I was using on my windows machine.

 

Resaved my configuration in my windows client like you said to do. When I fire up Crashplan on my windows machine I see tower as a machine I am able to save to. When I launch a backup it appears to be backing up, when I actually check my unRAID machine there are no files, but I'm not getting any idication of permissions or errors so I'm kinda lost.

[boof]

Can you list step by step exactly what you've done? Ignore the guide, we need to know what *you* have done.

 

I followed your install process to the letter.

 

and even logged into my unRAID console and established an account and password that was the same I was using on my windows machine.

 

That wasn't in the process so you haven't followed it to the letter

 

What is this and what are you using for?

 

Resaved my configuration in my windows client like you said to do. When I fire up Crashplan on my windows machine I see tower as a machine I am able to save to. When I launch a backup it appears to be backing up, when I actually check my unRAID machine there are no files, but I'm not getting any idication of permissions or errors so I'm kinda lost.

 

Where are you looking for files? and where are you looking to see if you are receiving errors?

 

What makes you think a backup is going through ok? Once the backup is complete are you able to browse data for a restore?

 

What are you ultimately trying to do? backup your windows machine to unraid?

 

As a first guess I'd presume your backups are *not* going to /mnt/disk4/MISC and instead, possibly, onto the unraid ram disk in a default location.

 

Does /mnt/disk4/MISC exist? Have you paid attention to case sensitivity when giving crashplan that path?

 

How big is the backup set? bigger than the physical memory in your unraid machine?

 

If you've followed the guide you should have logs in /usr/local/crashplan/log to look at.

 

[kizer]

Thanks boof, I'll give all those questions an addressing soon in my head and on the screen and maybe the setting up an account on the unRAID was the thing that broke it, but I'm not 100% sure.

 

As well I hope you don't think I'm coming across as ungreatful for your howto or your help. I really am, but I just wanted you to know it wasn't my intent. 

[boof]

No - of course not. It *does* work so we just need to find out what's tripping things up on your system and we can get it fixed!

 

It's such a pain to install headless it could be anywhere along the line really.

 

I was really hoping the recent 3.0 update would bring some enhancements to help but sadly....

[LVLAaron]

Boof, I've done as you described back on page 2 and things are going fairly swimmingly.

 

 

I have a few questions though, and please excuse me if they've been covered.

* If I reboot unraid, backups don't seem to be running unless I connect to the server with the client.

* When I connect the client after a reboot, I have to login again.

 

Are these normal?

[boof]

Boof, I've done as you described back on page 2 and things are going fairly swimmingly.

 

 

I have a few questions though, and please excuse me if they've been covered.

* If I reboot unraid, backups don't seem to be running unless I connect to the server with the client.

* When I connect the client after a reboot, I have to login again.

 

Are these normal?

 

No, definitely not. I suspect they're both caused by the same problem. Not sure what that problem is though.

 

- You can configure crashplan to always require the use of a password when connecting via the client. I'm presuming you haven't done this? This wasn't the default fresh installation option in 2.x it's possible this has changed in 3

 

- When you log in after a reboot do you have to configure any of your backup sets etc again?

 

If you're not expecting crashplan to require you to login each time you run the client, then the only thing that springs to mind that would block this is if the /var/lib/crashplan/.identity file either does not exist or does not contain your data. Implying it hasn't been saved / restored across a reboot.

 

I'm not 100% on how crashplan generate your machines ID it could be possible that it will always be identical which means as soon as you login in again (thus creating a new .identity file with your details and forcing crashplan to generate an id) it will be the same as the previous id before the reboot. Crashplan can then get your backup configs from their end meaning you would have no need to reconfnigure and your backups would then start running.

 

This sounds most plausible.

 

In summary :

 

- Check the contents of /var/lib/crashplan/.identity *before* rebooting. Take a note of this, or better yet copy the file to a non volatile location - like an array disk.

 

- reboot

 

- Check the contents of /var/lib/crashplan/.identity once your server boots and crashplanengine is running *but before you login via the client*.

 

- If the contents differ to the file you looked at pre reboot, this is our problem. You need to make sure the .identity file survives a reboot and it isn't at present.

 

For reference the .identity file should contain, amongst other things :

 

guid=your machines unique id

username=your crashplan username

 

If, after a reboot, you don't have the username section filled in correctly and / or the guid differs from the one contained in the .identity file pre reboot we have a problem.

 

The section of the install guide that covers this is :

 

- tar up the crashplan folder including the .identify file thats been causing us greif!

 

Code:

tar -cvf /boot/packages/crashplan.tar /usr/local/crashplan /var/lib/crashplan

 

We're backing up two locations :

 

- /usr/local/crashplan

- /var/lib/crashplan

 

It's important to get both as the .identity file lives in the latter.

[LVLAaron]

I guess I didn't setup my backup when installing before I restarted it the first time. I had an .identity file in the tarball but it wasn't correct. I connected everything, checked the file, stopped crashplan, recreated the tarball, rebooted and: (insert 10 minutes for a reboot... I need a faster flash drive )

 

---- everything works! Hooray. I was starting to wonder what the benefit was if I had to do all of this fiddling after a reboot.

 

Thanks for your help!

[boof]

I guess I didn't setup my backup when installing before I restarted it the first time. I had an .identity file in the tarball but it wasn't correct. I connected everything, checked the file, stopped crashplan, recreated the tarball, rebooted and: (insert 10 minutes for a reboot... I need a faster flash drive )

 

---- everything works! Hooray. I was starting to wonder what the benefit was if I had to do all of this fiddling after a reboot.

 

Thanks for your help!

 

Good work!

 

If you're feeling particularly brave you could symlink (in your go script) /usr/crashplan and /var/lib/crashplan to somewhere on your array. And install / copy crashplan in there.

 

You wouldn't then need to worry about untarring or recreating your tarball after an update.

 

I've done this since writing the guide and I much prefer it. Also keeps the crashplan binaries and logfiles out of the ram disk which saves a reasonable chunk of memory for more important things.

[LVLAaron]

That's a pretty good idea. I've got 2gb of memory right now and that's plenty for sabnzbd and crashplan... My next project is to get unraid running on a full slackware install...

[aiden]

I guess I didn't setup my backup when installing before I restarted it the first time. I had an .identity file in the tarball but it wasn't correct. I connected everything, checked the file, stopped crashplan, recreated the tarball, rebooted and: (insert 10 minutes for a reboot... I need a faster flash drive )

 

---- everything works! Hooray. I was starting to wonder what the benefit was if I had to do all of this fiddling after a reboot.

 

Thanks for your help!

 

Awesome!  Glad to hear you got this going!

[aiden]

If you're feeling particularly brave you could symlink (in your go script) /usr/crashplan and /var/lib/crashplan to somewhere on your array. And install / copy crashplan in there.

 

You wouldn't then need to worry about untarring or recreating your tarball after an update.

 

I've done this since writing the guide and I much prefer it. Also keeps the crashplan binaries and logfiles out of the ram disk which saves a reasonable chunk of memory for more important things.

 

Boof, if it isn't too much bother, is there a guide you could put together for this, kind of like an "Advanced Configuration" or something?  I don't mind retarring after an update, but as you say, it is much more preferable to not have to mess with it at all.

[boof]

Boof, if it isn't too much bother, is there a guide you could put together for this, kind of like an "Advanced Configuration" or something?  I don't mind retarring after an update, but as you say, it is much more preferable to not have to mess with it at all.

 

It's quite simple in principle, though there are lots of ways to go about getting your current configuration over to the new one.

 

Presuming you're running as a result of this guide you could :

 

- stop crashplan

 

/usr/local/crashplan/bin/CrashPlanEngine stop

 

- re-tar as per original instructions if you think you need to. May be a good idea just to have a recent backup.

 

- Remove your two existing crashplan directories (and hence your current install) - be sure you're comfortable you can get back from this.

 

rm -rf /usr/local/crashplan
rm -rf /var/lib/crashplan

 

- Create two new directories somewhere on your array. I used my cache drive. Worth remembering that running crashplan from the array directly will likely keep the disk it's on permanently spun up. So you probably want to either put it on the cache disk or force it directly onto a /mnt/user/diskX share to avoid user shares splitting the data and at least have only one disk spun up.

 

mkdir /mnt/cache/.permanent/crashplan
mkdir /mnt/cache/.permanent/crashplan-var

 

- Above I've put it onto the cache drive in a hidden directory (prefixed with the .) this makes sure the mover script ignores anything in there. As an aside it's where most of my installed apps live (SABnzbd, sickbeard, couchpotato etc)

 

- Create symlinks in the root filesystem to point to your new directories accordingly. Be sure these are correct. The directories will have to be created already as above.

 

ln -s /mnt/cache/.permanent/crashplan /usr/local/crashplan
ln -s /mnt/cache/.permanent/crashplan-var /var/lib/crashplan

 

- You can now untar your tarball to install crashplan again as per the original instructions. This will unpack your tarball into /usr/local/crashplan and /var/lib/crashplan. But the symlinks will redirect everything onto your disk drive. At this point crashplan now lives on disk, you don't need to worry about tar'ing or untar'ing again.

 

- Start up crashplan and check it works as per usual.

 

/usr/local/crashplan/bin/CrashPlanEngine start

 

- All being well crashplan should now be working ok - this time from your disk. Easy enough to check, nip into your /mnt/cache/.permenant/crashplan directory (or wherever you chose to locate it) and look in the log folder / generally check timestamps of the files.

 

- You now need to make this persist across a reboot. Unlike the original guide you *do not need* to untar anything now. The only thing living on unraids root ram disk are the symlinks. So we just need to create them before we try to start crashplan. Edit /boot/config/go accordingly and add

 

ln -s /mnt/cache/.permanent/crashplan /usr/local/crashplan
ln -s /mnt/cache/.permanent/crashplan-var /var/lib/crashplan
/usr/local/crashplan/bin/CrashPlanEngine start

 

You're done.

 

When crashplan auto updates it will overwrite on disk.

 

The only drawbacks to this are :

 

- will keep a disk spun up

 

- you're no longer backing up your crashplan install as routine.

 

You can combat the latter by continuing to tar things up periodically. All initial commands / methods you've used to do this will still work (including the unmenu user script gui) as due to the symlinks everything appears to be in the same place.

 

Or you could use crashplan to..backup your crashplan install. Bit of a chicken and egg when it comes to a restore though.

 

The only really important thing is your .identity file. And it's easy enough to regenerate. For disaster recovery you can install crashplan from scratch as previously and then tell code42 your 'new' machine is actually your old one. Which will fix your identity file and give you all your backups ready to restore. More info on their website.

 

I believe there's an install time option to relocate crashplan at that point. Which you could use to install into /mnt/cache/.permanent directly. I'm not sure if this will also relocate the .identity file though so you may still need at least one symlink. If anyones tried this perhaps they could update?

 

I hope this all makes sense? Let me know if not.

 

[aiden]

- will keep a disk spun up

 

Yikes... didn't think about this at all.  That's a reason not to put it inside the array, but on a separate disk, perhaps a spare flash disk?

 

I hope this all makes sense? Let me know if not.

 

Yes it does.  Thank you very much.

 

[boof]

- will keep a disk spun up

 

Yikes... didn't think about this at all.  That's a reason not to put it inside the array, but on a separate disk, perhaps a spare flash disk?

 

 

You could do - but crashplan is quite log file happy. You'd have to be wary of the amount of writes the logs would flush down to your flash drive. It might not last long.

 

It's really a perfect candidate for a cache disk. But if you don't have one...I can see that having another disk having to be spun up all the time wouldn't be ideal.

 

Yes it does.  Thank you very much.

 

You're very welcome.

 

[aiden]

You could do - but crashplan is quite log file happy. You'd have to be wary of the amount of writes the logs would flush down to your flash drive. It might not last long.

 

  To help clarify, using the "default" installation for unRAID puts everything into memory, so the unRAID Flash drive is spared this type of consistent transacting, correct?  Moving the installation out of the boot and onto a physical disk transfers this logging out of memory and onto the disk, thus the concern over using a flash storage. 

 

I don't employ a cache drive, primarily because I don't write enough to my array in one sitting to require the extra throughput, and I don't want to waste a SATA port.  But, this is a compelling reason to look into adding one.  Thanks again.

[boof]

You could do - but crashplan is quite log file happy. You'd have to be wary of the amount of writes the logs would flush down to your flash drive. It might not last long.

 

  To help clarify, using the "default" installation for unRAID puts everything into memory, so the unRAID Flash drive is spared this type of consistent transacting, correct?  Moving the installation out of the boot and onto a physical disk transfers this logging out of memory and onto the disk, thus the concern over using a flash storage. 

 

Yep 100% correct.

 

 

[LVLAaron]

Crashplan is stuck for me. It's at 72 percent on a file about 4.5gb into my initial 22gb backup.

 

How can I figure out what the cause is?

[aiden]

For me the only way to monitor what's going on is to run the client with the spoofed config file.  That should show you what's going on within the GUI.  The good news is that if you need a reboot, you don't have to start over.  FWIW, my initial upload was 71.5GB, and it ran without a hiccup.

[LVLAaron]

spoofed config file

 

? You mean the ssh tunnel deal? I'm using the GUI on a my windows workstation to see what's going on, that's how I know it was hung up. I removed the directory that file was in (10mg RAW image) and it started on other things. So we'll see.

[aiden]

? You mean the ssh tunnel deal?

 

Yes

[boof]

Crashplan is stuck for me. It's at 72 percent on a file about 4.5gb into my initial 22gb backup.

 

How can I figure out what the cause is?

 

Sounds like you've moved passed this / sidestepped it for now.

 

Crashplan should tell you the file it's stuck on in the GUI. And the logs may / may not give you more detail.

 

Things for info :

 

- How big is the file? I believe crashplan does block level encryption on the data it backs up. So the size of the file shouldn't matter. But it's plausible if you backup a large file there might be a pause whilst it sorts out encryption / dedupe etc. I think I'm plucking at straws a bit here, I've never looked to get metrics to see if file size affects backup speed (other than in the obvious way!)

 

- Could you read the file outwith crashplan? i.e you don't have a failing disk / bad blocks / corrupted filesystem that's just not allowing crashplan to read the file at the OS level?

 

- Where are you backing up to / from? Is wherever the backup is going to definitely up and available? The transfer hasn't just stalled because the target isn't responding either because it's down or because you're seeing a network glitch locally?

 

The only failure I've seen with crashplan is when I 'seeded' a backup from my home machine to a crashplan target on my work office machine. I copied ~250 gigs of data between the two using a fairly ropey USB disk I had kicking about.

 

Once completed crashplan on my home system then verified the contents of the backup my office system and found a few blocks failed crc checks. It then re backed up the affected data automatically.

 

So on that evidence it handled a failure gracefully. I've never had cause to see how it would handle a local i/o error though.

 

 

edit : sorry I see you mentioned the size of file. THat shouldn't be big enough to affect backup rates (beyond wire speed) one way or the other presuming you're not running on a 286!

[LVLAaron]

Well, as it turns out I'm having upstream trouble with my cable modem. I'm supposed to be 10mbit down /1mbit up.

 

Right now my upload is so bad that things like speedtest.net just timeout and quit.  I'm sure that's why CrashPlan is just sitting there.

 

rant: I love how when you call your provider (Insight in my case) they try and upsell you on internet and phone service after they tell you that they are sorry your stuff isn't working and that they can't have someone out to look at it for 2 days.  >

[aiden]

rant: I love how when you call your provider (Insight in my case) they try and upsell you on internet and phone service after they tell you that they are sorry your stuff isn't working and that they can't have someone out to look at it for 2 days.  >

 

Well, it's your fault for being so cheap and not paying a premium for a "better" connection.  If you would only pay more, they could afford to give you better service.