Hi,

 

Since I've been using unRAID, I've been using a script to backup files via rsync.  Everything was fine for many years, but as the number of files grew (close to 100,000 file/folder entities), rsync would more often than not crash before completing the backup.

 

So I did some digging and found that it was basically normal for rsync to crash when handling this many files, and that it shouldn't be used for such large transfers.

 

So I modified my script to break the transfers into chunks.  Instead of one rysnc command that copied everything, I broke it up into five commands that would copy a group of files and exclude the rest.  In this was, I was evenly able to divide transfers up into about 20,000 per rsync command line in the script.

 

This did seem to do better, but rsync still seems to crash (SIGTERM) before the 5th transfer is fully complete.  And if I keep re-running the script, it will keep hanging in the same place repeatedly.

 

So just on a whim, I copied each rsync command out of the script, pasted into the command line, and ran it directly.  Well, it would run right through, and quit a bit faster too.  So I did some tests and copy/pasting in the 5 command lines and running each from the command prompt, is still about 25% faster (including the time it takes me to copy and paste the commands) than running the same 5 commands through a script, and line by line it never errors out, yet through a script it does.

 

So my question is, why would running the rsync commands through a script appear to slow them down, and apparently cause the last one to hang? 

 

I guess I understand why it failed when I ran one big rsync for all 100000 entities.  I guess rysnc commits to memory 100000 temp files to serve as place holders before any transfer starts.  Then it writes  temp files to the destination.  So before any actual data is transferred, you are loading up your RAM with temp files, loading up the CPU, and busying up file I/O.  So that all made sense to me. 

 

But I thought breaking into five commands, the script should run each rsync command, resolve it, free up memory/CPU, then run the next instance., same as if done through command line.

I've rsynced whole drives of over 250,000 files.

The real issue is how much spare memory exists for rsync internals, kernel buffers dentry and inode tables.

 

Before each rsync you can try and drop the cache.  For me this helped when I had allot of disks.

http://www.linuxinsight.com/proc_sys_vm_drop_caches.html

 

Also, the more disks you have, the more md/unraid internal buffers are held in low memory.

 

On my 4 drive serve with 4GB of ram, I was able to rsync over 250,000 files per drive without crashing.

 

 

As far as speed, perhaps the script is being niced via cron. i.e. lowered in priorty.

Also I found a trick to speed up the rsync IF (notice the big if) you are rsyncing to an rsync server (not over ssh).

 

Add this to the top of the rsyncd.conf above all the directory definitions.

 

socket options  = SO_SNDBUF=524288,SO_RCVBUF=524288

 

 

It lets more data be buffered in the kernel with tcp. I was getting up to 95MB/s on some large transfers (depending on the downstream server)

I've rsynced whole drives of over 250,000 files.

The real issue is how much spare memory exists for rsync internals, kernel buffers dentry and inode tables.

 

Before each rsync you can try and drop the cache.  For me this helped when I had allot of disks.

http://www.linuxinsight.com/proc_sys_vm_drop_caches.html

 

Also, the more disks you have, the more md/unraid internal buffers are held in low memory.

 

On my 4 drive serve with 4GB of ram, I was able to rsync over 250,000 files per drive without crashing.

 

 

As far as speed, perhaps the script is being niced via cron. i.e. lowered in priorty.

Also I found a trick to speed up the rsync IF (notice the big if) you are rsyncing to an rsync server (not over ssh).

 

Add this to the top of the rsyncd.conf above all the directory definitions.

 

socket options  = SO_SNDBUF=524288,SO_RCVBUF=524288

 

 

It lets more data be buffered in the kernel with tcp. I was getting up to 95MB/s on some large transfers (depending on the downstream server)

 

I have 20 drives in the server, but all these files are coming off a single drive (which is setup as its own share too).  I guess I did start adding new drives around the same time I hit 100,000 files and started having problems.  Added four news ones in the past six months, so maybe that made unRAID use just enough extra memory.  I originally had 4GB of RAM (2 x 2GB), but about two years ago, my md5 checksums on transferred files all started not matching (bad transfer/corrupted files).  I traced the problem back to a corrupted stick of RAM (or bad RAM socket, since it tested good when put in another PC, but would test bad in either socket of my unRAID MB) and removed it and kept running with 2GB.

 

I am syncing to an rsync server, so I may try your socket options trick.

 

I was originally thinking it was the receiving server.  They are pre-made boxes with a proprietary processor and limited amount of onboard RAM.  They are running some version of Linux with a read only operating system.  So basically you can't do any hardware or software tweaks on the receiver.

 

But after running from command line and everything working flawlessly, and then running the same commands via script and it choking, it turned my suspicions to the unRAID box where the script is being executed.

But after running from command line and everything working flawlessly, and then running the same commands via script and it choking, it turned my suspicions to the unRAID box where the script is being executed.

 

 

There could be priority variances via cron.

Look at dropping the cache before each rsync and then at the end of the script. If your rsync runs with the mover, that would cause issues too.

I would also suggest (if possible) that you access the disk share directly rather then the user share.