rsync errors & High Water not working?

[tucansam]

Guys,

 

I'm posting this here because the signal to noise ratio on other linux forums I've see is so very,very low.  Maybe someone here can help.

 

I finally got rsync up and running and spent all day watching about 2TB of data transfer.  I got one rsync error message that terminated the transfer (it was super late at night and I googled it, didn't understand it, started rsync again and went to bed)

 

This morning I now have this:

 

rsync: writefd_unbuffered failed to write 4 bytes [sender]: Connection reset by

peer (104)

rsync: write failed on "files/junk/virtual machines/T7400/T7400.vmdk" (in backup

): No space left on device (28)

rsync error: error in file IO (code 11) at receiver.c(298) [receiver=3.0.4]

rsync: connection unexpectedly closed (130702 bytes received so far) [sender]

rsync error: error in rsync protocol data stream (code 12) at io.c(632) [sender=

3.0.4]

 

That last error (io.c(632) [sender=3.0.4]) is the error I received last night.  No idea what it means.

 

The no space left on device problem....

 

--

root@ffs2:/mnt/user/backup# df -k

Filesystem          1K-blocks      Used Available Use% Mounted on

/dev/sda1              7800304    164496  7635808  3% /boot

/dev/md1            976732736 976732736        0 100% /mnt/disk1

/dev/md2            488371640    32848 488338792  1% /mnt/disk2

/dev/md3            488371640    32840 488338800  1% /mnt/disk3

/dev/md4            312561648    32840 312528808  1% /mnt/disk4

/dev/md5            156286096    32840 156253256  1% /mnt/disk5

shfs                2422323760 976864104 1445459656  41% /mnt/user

root@ffs2:/mnt/user/backup#

--

 

I have the share setting to High Water.  Any ideas?

 

Thanks.

 

[Joe L.]

Guys,

 

I'm posting this here because the signal to noise ratio on other linux forums I've see is so very,very low.  Maybe someone here can help.

 

I finally got rsync up and running and spent all day watching about 2TB of data transfer.  I got one rsync error message that terminated the transfer (it was super late at night and I googled it, didn't understand it, started rsync again and went to bed)

 

This morning I now have this:

 

rsync: writefd_unbuffered failed to write 4 bytes [sender]: Connection reset by

peer (104)

rsync: write failed on "files/junk/virtual machines/T7400/T7400.vmdk" (in backup

): No space left on device (28)

rsync error: error in file IO (code 11) at receiver.c(298) [receiver=3.0.4]

rsync: connection unexpectedly closed (130702 bytes received so far) [sender]

rsync error: error in rsync protocol data stream (code 12) at io.c(632) [sender=

3.0.4]

 

That last error (io.c(632) [sender=3.0.4]) is the error I received last night.  No idea what it means.

 

The no space left on device problem....

 

--

root@ffs2:/mnt/user/backup# df -k

Filesystem          1K-blocks      Used Available Use% Mounted on

/dev/sda1              7800304    164496  7635808  3% /boot

/dev/md1            976732736 976732736        0 100% /mnt/disk1

/dev/md2            488371640    32848 488338792  1% /mnt/disk2

/dev/md3            488371640    32840 488338800  1% /mnt/disk3

/dev/md4            312561648    32840 312528808  1% /mnt/disk4

/dev/md5            156286096    32840 156253256  1% /mnt/disk5

shfs                2422323760 976864104 1445459656  41% /mnt/user

root@ffs2:/mnt/user/backup#

--

 

I have the share setting to High Water.  Any ideas?

 

Thanks.

yes,

 

you might not have "split level" and "min-free" set as you need.  So far, all your files are on disk1, and it is out of space.

[tucansam]

Well, at this point I'm still not sure what to set my split levels at. 

 

Also, now I am getting an error with rsync telling me "connection unexpectedly closed"

 

I had transferred over about 1TB, then deleted some unnecessary files (freed 200GB on that disk), and tried to rsync the same data to the same share on the backup server, it sends three tiny files (none of which have changed...) and then hangs for 30+ mins before returning the error.  Other rsync commands issues to copy other shares to the backup server execute fine. 

[Joe L.]

Well, at this point I'm still not sure what to set my split levels at. 

 

Also, now I am getting an error with rsync telling me "connection unexpectedly closed"

 

I had transferred over about 1TB, then deleted some unnecessary files (freed 200GB on that disk), and tried to rsync the same data to the same share on the backup server, it sends three tiny files (none of which have changed...) and then hangs for 30+ mins before returning the error.  Other rsync commands issues to copy other shares to the backup server execute fine.

To help us to understand your needs, you need to tell us the full path to the files being copied that are failing AND the share specific settings for "split level" and "min-free"

 

Once below the split level ALL files will go to the same disk as the parent folder.    See here:

http://lime-technology.com/wiki/index.php/Un-Official_UnRAID_Manual#User_shares_2

and here:

http://lime-technology.com/wiki/index.php/Un-Official_UnRAID_Manual#Split_level

 

[tucansam]

Currently I have a "backup" share only, this is on the backup unraid server. 

 

Its set at High-Water, 80GB free, split level 1 (although I've tried everything from zero to four).

 

The backup server has a 2TB parity disk, 1TB data disk, two 500GB data disks, a 320GB data disk, and a 160GB data disk.  Unraid sees 2.48TB free total on the array when its empty.

 

I am trying to move about 2.2TB of data and would be fine with it being spread across all disks however unraid decides to do it; this is a backup server after all.  My largest file size is on the order of 70GB (virtual machine image).  Although I am copying the largest share to be backed up first, assuming it will fill the 1TB and 500TB disks first, and the rest of the files (several other users' "My Documents" redirects) will fill in the gaps.

 

I'm wondering if, since I have a single share called "backup," I shouldn't just turn split levels off altogether?

 

I just rebooted both systems and rsync is running.... for the moment... Will post back shortly if there are issues.

 

[tucansam]

OK.  Here's an excerp from rsync's log:

 

2013/12/01 10:57:53 [5767] cd+++++++++ user/iso files/

2013/12/01 10:58:59 [5767] <f+++++++++ user/iso files/openlinux-141-x86.iso

2013/12/01 11:00:05 [5767] <f+++++++++ user/iso files/openlinux-141-x86_64.iso

2013/12/01 11:00:40 [5767] <f+++++++++ user/iso files/zorin-os-7.1-lite.iso

 

The first two files were 1.18 and 1.27GB respectively.  The last (zorin) is 824.18M and shows completed, however rsync is now "hung" on the console and the backup server is no longer showing writes to any disks.  In a half hour or so I will get the error message from rsync.

 

If I run rsync on other user shares, regardless of file number, size, or depth, its running fine.

 

[tucansam]

Well dammit, it looks like the fantastic Realtek ethernet interface is puking with huge files, sometimes.  The file transfer stutters for a second and "eth0: link up" is being written to the syslog once in a great while.  It seems to fix itself quickly and rsync isn't complaining on my current transfer, and my ssh terminal sessions stay open, so whatever is happening is happening quick.

 

This is an Asus C60 board and I'm running 5.0-rc5.... Might be time to upgrade or get another NIC and see if any of the problems go away.

[tucansam]

Well this:

 

http://grinux.wordpress.com/2012/12/18/rsync-out-of-memory/

 

and this

 

http://lowendtalk.com/discussion/6734/rsync-out-of-memory-try-this

 

lead me to believe that I may be running out of memory -- the share I am trying to copy (my own "My Documents" redirect) contains almost a million files and tens of thousands of directories.  So I am running the command now with "--no-i-r" and it is currently at "building files list..." (just completed at "767564 files to consider")

 

I'll see what this does, but correct me if I'm wrong, don't I want incremental recursion? 

 

This system has 8GB RAM but according to my reading its a process-specific memory cap (for lack of the proper term as I am totally unfamiliar with this) vs a system RAM issue.

 

 

EDIT

 

The complete lack of network traffic to my backup server, and no new writes to any disk, leads me to believe that this hasn't worked, and that in 28 minutes I will get an rsync error after it gives up.

[Joe L.]

A split level of 99 might work for you...

 

The original error had this as the file path:

backup/files/junk/virtual machines/T7400/T7400.vmdk

  1        2      3            4                  5

 

Even with a split level of 4, the only disk possible to use would be one with a "virtual machines" directory already existing.

 

(I think leaving split level blank is the same as setting it to a very high number)

[lionelhutz]

I'm wondering if the rsync makes all the directories first and then it populates them. Level 4 should work with that structure if the directory and file both get written together. All directories being written first will screw it up since the files will all get locked to a single disk.

 

If this is happening you'd have to figure out how to write the directories and files at the same time unless you are OK with multiple directories repeated on each drive even though some may be empty.

[tucansam]

Well, in the following week I've added an Intel gig NIC to my backup server and have not seen the interface go down/up as I had with the Realtek.

 

rsync still doesn't work on my share with a zillion files, but it does still work on the other shares.  If they grow too large, I fear it will stop working.

 

I tried mounting the backup server using nfs, couldn't get past permission problems.

 

I have now mounted the backup server using cifs, and using 'cp -R -u -v /mnt/user/share /mnt/backupserver' I am at least getting files to copy over.

 

This feels kludgey at best.  I was hoping to automate the backup but for now I'll be doing it by hand.

 

I did a lot of reading about rsync and it seems that is has a lot of trouble with moving enormous amounts of files.

 

If anyone has a solution similar to rsync, but with the ability to handle huge file sets, I'd love to try it.

 

I ran through several boxes of old hard disks this week and dumped the entire filesystem contents of numerous old Sun/linux systems onto my unraid box, to slowly sort through the old data when I have time.  I copied entire partitions, so essentially "/" from each system.

 

Each machine had 25-50k files, so I've increased my personal user share's number of folders and files by a large number.  I realize this is messy/bad practice, but I use my unraid server as a big place to dump data temporarily so I can sort through it later.  So just by the nature of how I work and organize (or don't), I am going to have a million+ files on my user share all the time.

 

rsync is no longer an option, it seems.

 

Suggestions welcome.