August 10, 201510 yr Hey guys, I can't figure this one out. I have two unraid servers connected by a DD-WRT OpenVPN Server. They have the rsync dameon setup. When I run the rsync, it drops the files into a .sync/Archive folder, my understanding because I am not using the -inplace option. I keep seeing the same files copy over to this folder. I have yet to successfully run a single rsync to completion. My longest attempt before an I/O error or connection time out was about a 2 week run. The sync will get cut, and then I will rsync the movies to the correct folder locally and then delete the extra data in .sync/Archive. What's interesting, is I am pulling over nearly 200 movie files before something kills my sync. I will then do a rsync local from .sync/Archive to the correct folder and it will move like 2 new files over. The local rsync works as it should, proving I have duplicate data in .sync/Archive The idea is, the two folders should be closer and closer to getting into sync, even if I have to run it again. however I just keep getting duplicate data. My raid array maxed out on me eventually. This is how I am running the sync. I tried with and with out the --checksum option hoping it would identify files correctly and not download unnecessary files. Same results, massive duplication in the .sync/Archive folder rsync -auzP --stats --inplace --ignore-existing --checksum --exclude-from /boot/rsync-exclude --log-file=/mnt/user/jffs/rsync.log rsync://192.168.4.5/mnt/user/Movies/ /mnt/user/Movies Note I had to create an exclude for the DSstore and Appledouble files. Data was originally uploaded by a Mac to the remote unraid server. The data duplication was even worse until I excluded these. I'm at like 40 attempts, and tried with the no exec option to get the results I wanted, no go....
August 10, 201510 yr Author Here is an example of my latest rsync locally after my VPN dropped again. I was looking for potential new files, apparently 100% duplicate data This case no files were transferred between the .sync/Archive and correct movie folder. root@Tower:/mnt/user/Movies/Movie1/.sync/Archive# pwd /mnt/user/Movies/Movie1/.sync/Archive root@Tower:/mnt/user/Movies/Movie1/.sync/Archive# du -sm * | sort -n | tail -10 2036 Minority Report 3089 Meet Joe Black (1998) 720p 3109 Middle of Nowhere (1080p HD) 3294 To Be Takei (1080p HD).mp4 3591 Memento (1080p HD) 3917 Mean Girls (1080p HD) 4116 Message In a Bottle (1080p HD) 7777 Men In Black Movies 8145 Prince of Persia 2011 1080p.mkv 13702 Mission Impossible Movies root@Tower:/mnt/user/Movies/Movie1/.sync/Archive# rsync -aun --ignore-existing --stats ./ /mnt/user/Movies/Movie1 Number of files: 357 Number of files transferred: 0 Total file size: 60677399772 bytes Total transferred file size: 0 bytes Literal data: 0 bytes Matched data: 0 bytes File list size: 13844 File list generation time: 0.013 seconds File list transfer time: 0.000 seconds Total bytes sent: 15192 Total bytes received: 1347
August 15, 201510 yr Author Yeah same results still, duplicate data is being sent to the .sync/Archive folder of the destination directory. I've resorted to coping files one at a time
Archived
This topic is now archived and is closed to further replies.