Limits of Crashplan 142000 files 907GB

[tr0910]

I have had Crashplan running for over a year just fine, but in the last 3 months, it has been making my server just groan.  And its not backing up any new files.  It just sits trying to backup, but no new files get sent to crashplan central and it has about 1000 new files to backup.  I have 4 gb ram in the v5.05 server and it is installed with the process:

 

https://lime-technology.com/wiki/index.php/CrashPlan

 

The memory available to crashplan has been increased via:

 

http://support.code42.com/CrashPlan/Latest/Troubleshooting/Adjusting_CrashPlan_Settings_For_Memory_Usage_With_Large_Backups

 

but here is top reporting crashplan taking 144% of the CPU.  It's not always that large, but its around 100 always.

 

Tasks: 139 total,  1 running, 138 sleeping,  0 stopped,  0 zombie

Cpu(s):  0.0%us,  0.8%sy, 35.0%ni, 64.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st

Mem:  4136248k total,  3076572k used,  1059676k free,  265212k buffers

Swap:        0k total,        0k used,        0k free,  2158016k cached

 

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND

12328 root      39  19 1200m  73m 9.9m S  144  1.8  0:05.04 java

12283 root      20  0  2472 1004  756 R    1  0.0  0:00.03 top

  10 root      20  0    0    0    0 S    0  0.0  0:35.33 rcu_sched

 

I am soon ready to move it to v6 and Docker, but want to get it working with 100% before I cut over to v6. 

[gfjardim]

I have had Crashplan running for over a year just fine, but in the last 3 months, it has been making my server just groan.  And its not backing up any new files.  It just sits trying to backup, but no new files get sent to crashplan central and it has about 1000 new files to backup.  I have 4 gb ram in the v5.05 server and it is installed with the process:

 

https://lime-technology.com/wiki/index.php/CrashPlan

 

The memory available to crashplan has been increased via:

 

http://support.code42.com/CrashPlan/Latest/Troubleshooting/Adjusting_CrashPlan_Settings_For_Memory_Usage_With_Large_Backups

 

but here is top reporting crashplan taking 144% of the CPU.  It's not always that large, but its around 100 always.

 

Tasks: 139 total,  1 running, 138 sleeping,  0 stopped,  0 zombie

Cpu(s):  0.0%us,  0.8%sy, 35.0%ni, 64.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st

Mem:  4136248k total,  3076572k used,  1059676k free,  265212k buffers

Swap:        0k total,        0k used,        0k free,  2158016k cached

 

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND

12328 root      39  19 1200m  73m 9.9m S  144  1.8  0:05.04 java

12283 root      20  0  2472 1004  756 R    1  0.0  0:00.03 top

  10 root      20  0    0    0    0 S    0  0.0  0:35.33 rcu_sched

 

I am soon ready to move it to v6 and Docker, but want to get it working with 100% before I cut over to v6.

 

CrashPlan uses a lot of memory and processing for compression and deduplication.

[tr0910]

CrashPlan uses a lot of memory and processing for compression and deduplication.

 

I had reduced the backup size from 1,341gb to 907gb and from 221,000 files to 142,000 files.  That didn't help.

 

I just now changed the options for compression to none, and deduplication to minimal.  We'll see if that helps.  I have your docker running on 6beta14b on a test server.  I will have 16gb RAM once I change over to v6. 

 

I wonder if your docker version will have more scalablity than the v5 method?  Do you know of any users of your Docker with very large backups?

 

 

[Leifgg]

I am using the Docker app and have 4.1 TB data (76000 files). 1.6 TB remaining to backup (around 1100 files). Data de-duplication = Automatic, Compression = On and Encryption enabled. Works fine so far.

[boof]

I only have 85,000 files (across 4TB) in my largest crashplan backup set so not quite on the same scale.

 

One question though - what does the full process / option list of your running java look like?

 

The crashplan link you sent to makes me wonder if that's just a point in time fix and wouldn't take effect next time you restarted.

 

What does the running java process say is the memory limit and what does your $CRASHPLAN/bin/run.conf contain? I've always upped the memory limit by adjusting the values in run.conf.

 

[tr0910]

I only have 85,000 files (across 4TB) in my largest crashplan backup set so not quite on the same scale.

 

One question though - what does the full process / option list of your running java look like?

 

The crashplan link you sent to makes me wonder if that's just a point in time fix and wouldn't take effect next time you restarted.

 

What does the running java process say is the memory limit and what does your $CRASHPLAN/bin/run.conf contain? I've always upped the memory limit by adjusting the values in run.conf.

 

Do either of these give you what you want??

 

root@Server1:~# ps l 24542
F   UID   PID  PPID PRI  NI    VSZ   RSS WCHAN  STAT TTY        TIME COMMAND
0     0 24542     1  39  19 1279948 124444 futex_ SNl ?         3:10 /usr/local/
root@Server1:~#

root@Server1:~# ps -F 24542
UID        PID  PPID  C    SZ   RSS PSR STIME TTY      STAT   TIME CMD
root     24542     1  1 319987 124476 0 08:26 ?        SNl    3:11 /usr/local/cr
root@Server1:~#

 

Is this the right file (see attached run.conf) from /boot/packages/crashplan-install/scripts  Wrong file noted, removed.

[boof]

Do either of these give you what you want??

 

No - they're truncated. Try..

 

 ps -ww -fp 24542

 

To get wider output.

 

Is this the right file (run.conf) from /boot/packages/crashplan-install/scripts  It doesn't look like the -Xmx512m is right.  I think I changed that to 1024m.  I didn't do it from within Crashplan like they show.  I edited a file, but I am not sure I edited the right one.

 

I don't know - it depends how you've installed crashplan.

 

From the look of the beginning of the process list you've pasted above you probably want to look at /usr/local/crashplan/bin/run.conf to see what it's set to.

 

I'd thoroughly recommend upgrading and running the docker container. Makes life much easier.

[tr0910]

 

 ps -ww -fp 24542

 

From the look of the beginning of the process list you've pasted above you probably want to look at /usr/local/crashplan/bin/run.conf to see what it's set to.

 

I'd thoroughly recommend upgrading and running the docker container. Makes life much easier.

 

Here is the wider output and I've attached the correct file you referenced.  I did get -Xmx512m increased to 1024m.  Maybe I need to go higher?

 

root@Server1:~# ps -ww -fp 24542

UID        PID  PPID  C STIME TTY          TIME CMD

root    24542    1  1 08:26 ?        00:03:32 /usr/local/crashplan/jre/bin/java -Dfile.encoding=UTF-8 -Dapp=CrashPlanService -DappBaseName=CrashPlan -Xms20m -Xmx1024m -Djava.net.preferIPv4Stack=true -Dsun.net.inetaddr.ttl=300 -Dnetworkaddress.cache.ttl=300 -Dsun.net.inetaddr.negative.ttl=0 -Dnetworkaddress.cache.negative.ttl=0 -Dc42.native.md5.enabled=false -classpath /usr/local/crashplan/lib/com.backup42.desktop.jar:/usr/local/crashplan/lang com.backup42.service.CPService

root@Server1:~#

run.conf

[boof]

Looks like it will be worth a try.

 

You can also check the logs - it usually flags memory issues in there.

[tr0910]

Looks like it will be worth a try.

 

You can also check the logs - it usually flags memory issues in there.

 

Ugh, looks like the service log has lots of errors like:

 

[04.13.15 20:21:50.386 ERROR  QPub-BackupMgr      backup42.service.backup.BackupController] OutOfMemoryError occurred...RESTARTING!

 

Zipped log file was too large to attach. I'll just up the memory size in run.conf and reboot the server unless you want to try something else.

[boof]

Yup that's the same symptom I've seen before. Upping the memory reservation fixed it.

 

From the desktop client things look they're running but nothing ever backs up. What's actually happening is the backend server is restarting constantly and so quickly that the client never notices to disconnect.

 

Upgrade to 6 and the docker image