randall526 Posted August 10, 2015 Share Posted August 10, 2015 Hey guys, I can't figure this one out. I have two unraid servers connected by a DD-WRT OpenVPN Server. They have the rsync dameon setup. When I run the rsync, it drops the files into a .sync/Archive folder, my understanding because I am not using the -inplace option. I keep seeing the same files copy over to this folder. I have yet to successfully run a single rsync to completion. My longest attempt before an I/O error or connection time out was about a 2 week run. The sync will get cut, and then I will rsync the movies to the correct folder locally and then delete the extra data in .sync/Archive. What's interesting, is I am pulling over nearly 200 movie files before something kills my sync. I will then do a rsync local from .sync/Archive to the correct folder and it will move like 2 new files over. The local rsync works as it should, proving I have duplicate data in .sync/Archive The idea is, the two folders should be closer and closer to getting into sync, even if I have to run it again. however I just keep getting duplicate data. My raid array maxed out on me eventually. This is how I am running the sync. I tried with and with out the --checksum option hoping it would identify files correctly and not download unnecessary files. Same results, massive duplication in the .sync/Archive folder rsync -auzP --stats --inplace --ignore-existing --checksum --exclude-from /boot/rsync-exclude --log-file=/mnt/user/jffs/rsync.log rsync://192.168.4.5/mnt/user/Movies/ /mnt/user/Movies Note I had to create an exclude for the DSstore and Appledouble files. Data was originally uploaded by a Mac to the remote unraid server. The data duplication was even worse until I excluded these. I'm at like 40 attempts, and tried with the no exec option to get the results I wanted, no go.... Quote Link to comment
randall526 Posted August 10, 2015 Author Share Posted August 10, 2015 Here is an example of my latest rsync locally after my VPN dropped again. I was looking for potential new files, apparently 100% duplicate data This case no files were transferred between the .sync/Archive and correct movie folder. root@Tower:/mnt/user/Movies/Movie1/.sync/Archive# pwd /mnt/user/Movies/Movie1/.sync/Archive root@Tower:/mnt/user/Movies/Movie1/.sync/Archive# du -sm * | sort -n | tail -10 2036 Minority Report 3089 Meet Joe Black (1998) 720p 3109 Middle of Nowhere (1080p HD) 3294 To Be Takei (1080p HD).mp4 3591 Memento (1080p HD) 3917 Mean Girls (1080p HD) 4116 Message In a Bottle (1080p HD) 7777 Men In Black Movies 8145 Prince of Persia 2011 1080p.mkv 13702 Mission Impossible Movies root@Tower:/mnt/user/Movies/Movie1/.sync/Archive# rsync -aun --ignore-existing --stats ./ /mnt/user/Movies/Movie1 Number of files: 357 Number of files transferred: 0 Total file size: 60677399772 bytes Total transferred file size: 0 bytes Literal data: 0 bytes Matched data: 0 bytes File list size: 13844 File list generation time: 0.013 seconds File list transfer time: 0.000 seconds Total bytes sent: 15192 Total bytes received: 1347 Quote Link to comment
randall526 Posted August 15, 2015 Author Share Posted August 15, 2015 Yeah same results still, duplicate data is being sent to the .sync/Archive folder of the destination directory. I've resorted to coping files one at a time Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.