April 28, 20251 yr is there a Rsync option when copying files to make sure the file is copied correctly? the situation i found is ... if rsyncing script and if i copying a file from 1 server to another and when it hasnt reached that folder yet.. and the file is only transfered say 10% of the file.. then rsync sees it copys the 10% of the file but copys the time and file stamp and pretends the file is completed... so when you re run rsync and the file is completed... rsync skips over that file thinking its done even though its a false flag that file is only 10% done with a stamp of complete... my question is is there a option that verifies all the file bits are actually sent over like a true file integrity check? i currently use ``` rsync -avzue ssh ``` and if say rsync isnt a good file backup solution what is a better file server backup does everyone recommend as i rsync 80tb+ as backups but who knows now if some files never got replicated correctly.. and i looking for a solution if i make a file and it backup and i made a new change a few times like 3 times and backups but i want that first change is there a way to get backup copy of the 3 changes or i need 3 serveers 1 for daily weekly monthy so it has like different backups? Edited April 28, 20251 yr by comet424
April 28, 20251 yr Community Expert kinda... rsync copies the changes of a file... --this is why I prefer to use sftp and a different protcall for theses to have thess features and to guranee that file. Why Your Current Rsync Fails: rsync -avzue ssh -a preserves timestamps, permissions, etc. BUT: rsync trusts timestamps and file size to detect if files are "the same" (unless you tell it otherwise). If a file was incomplete but the time/size matched, rsync will skip it later (as you saw). You need to tell rsync: Don't trust just timestamps/sizes. Force checksum comparison (--checksum). And avoid overwriting a file that is still being written. rsync -avzue ssh --inplace --partial --append-verify --checksum Meaning: --checksum: forces a checksum comparison rather than relying only on timestamp and size. --partial: keep partially transferred files if transfer is interrupted (instead of deleting them). --append-verify: if a file exists and is shorter, append to it, then verify. --inplace: write directly to the destination file (optional, improves speed for huge files). Full safer rsync command suggestion rsync -avzue ssh --partial --append-verify --checksum SOURCE/ DESTINATION/ or if copying remotely: rsync -avzue ssh --partial --append-verify --checksum user@remote:/path/to/files/ /local/destination/ Bonus Tip: Safer Transfer Pattern (Staging location) Another real-world trick for backups: rsync to a temp file first, then rename it when fully done. rsync already partially handles this when using --partial-dir or --temp-dir. rsync -avzue ssh --temp-dir=/tmp/rsync-temp SOURCE/ DESTINATION/ rsync is good but its not the end all be all.. Quote i looking for a solution if i make a file and it backup and i made a new change a few times like 3 times and backups but i want that first change -That’s called versioned backups or snapshots. *rsync alone does NOT version files (it just overwrites). some Options: Use rsnapshot (a wrapper around rsync for versioned backups). Use borgbackup, restic, or duplicacy (modern backup tools that do deduplication and snapshots). ZFS filesystem snapshots (if you can). If sticking with rsync + basic Linux: You can do "cheap" versioning by rsyncing to dated folders: rsync -avzue ssh --partial --append-verify --checksum /source/ /backups/$(date +%Y-%m-%d)/ after review the above this is a full rsync command for you that should hit your needs... rsync -avzue ssh --partial --append-verify --checksum --progress user@server:/source/ /destination/ ✅ Integrity checked ✅ Partial transfers safe ✅ Can survive reruns without skipping incomplete files
April 28, 20251 yr Author ok ill have to re look into it all what you mentioned... you use the sftp to send.. can you do that in a user script?? as i set them in a user script. and i also use the --delete to delete and files to change.. but i guess thats no good for the reduncy backups is there one better then the other? so i can learn another backup solution.. ive played with the syncthing but i never liked it.. so i just stuck with the rsync but found the issues of the file problem problem why i found some videos messed up on the backup server... so running it like this for now rsync -avzue ssh --partial --append-verify --checksum --progress user@server:/source/ /destination/ will that always check the file. i did try rsync -avzue ssh --checksum --progress user@server:/source/ /destination/ --delete it did go slow but i dont know if it repairs that file that is only 10% copied yet still retians its fully done.. when you do rsync -avzue ssh --partial --append-verify --checksum --progress user@server:/source/ /destination/ and it finishes it when you re run it again will it go slow or will it go fast the next time like without the checksum? like does it keep a record that it fully copied the first time wth the checksum? so that the 2nd time it will just zip through.. as i also backup files to my sisters unraid and uses a 3mbp/s connection. so like i found issues i mentioned over the network and then i also update over slow internet as offsite... so i just trying to find the best of both worlds.. is 1 better over then the other? like i know the snapshots but thats really only on truenas and i dont use it cuz i wanted to be able to add hard drives as i go not need all the hard drives same size at once... ill look into borgbackup and restic... is one better then the other?
April 28, 20251 yr Author so running rsync -avzue ssh --checksum --progress user@server:/source/ /destination/ --delete even when the file already copied did nothing i guess i gotta blow the backups and then re run it rsync -avzue ssh --partial --append-verify --checksum --progress user@server:/source/ /destination/
April 28, 20251 yr Community Expert 2 hours ago, comet424 said: ok ill have to re look into it all what you mentioned... you use the sftp to send.. can you do that in a user script?? as i set them in a user script. and i also use the --delete to delete and files to change.. but i guess thats no good for the reduncy backups is there one better then the other? so i can learn another backup solution.. ive played with the syncthing but i never liked it.. so i just stuck with the rsync but found the issues of the file problem problem why i found some videos messed up on the backup server... so running it like this for now rsync -avzue ssh --partial --append-verify --checksum --progress user@server:/source/ /destination/ will that always check the file. i did try rsync -avzue ssh --checksum --progress user@server:/source/ /destination/ --delete it did go slow but i dont know if it repairs that file that is only 10% copied yet still retians its fully done.. when you do rsync -avzue ssh --partial --append-verify --checksum --progress user@server:/source/ /destination/ and it finishes it when you re run it again will it go slow or will it go fast the next time like without the checksum? like does it keep a record that it fully copied the first time wth the checksum? so that the 2nd time it will just zip through.. as i also backup files to my sisters unraid and uses a 3mbp/s connection. so like i found issues i mentioned over the network and then i also update over slow internet as offsite... so i just trying to find the best of both worlds.. is 1 better over then the other? like i know the snapshots but thats really only on truenas and i dont use it cuz i wanted to be able to add hard drives as i go not need all the hard drives same size at once... ill look into borgbackup and restic... is one better then the other? I use a combination of tools, scritps and settings... I'm a fan of sftp. I recenlty made a docker where I can share and use file zilla to access my unraid and move data between pc. Yes rsync can connect to a sftp remote to pull data and vise vera to send data... syncthing was a pain and endedup forceing the home folder for defualt synthing to send data. I don't agree withsome of syncthing tor principle and file sahring it does. you need atleast 3-4 machines that need the same data before syncthing is viable... seafile is another example... then theres the rclone plugin and doceker as example instaead of using a user script. nerdtool / unget / 3rd party unraid slack ware applciaiton installs for some other tools to use installed things like the rwrapper above... My systems use zfs and I can even leveragy zfs send snapshot and copies between disks... It all comes down to how you want to interact with it, and what you want it to do... 1 hour ago, comet424 said: so running rsync -avzue ssh --checksum --progress user@server:/source/ /destination/ --delete even when the file already copied did nothing i guess i gotta blow the backups and then re run it rsync -avzue ssh --partial --append-verify --checksum --progress user@server:/source/ /destination/ you need to specific your soucre and destination... not enoth info to assist... I don't know if the disk are all local, or you reomte coneted to a share. whats runnign the comand... what your trying to do with xyz data... This all dictates how one would go about moving and touching the data between systems....
April 28, 20251 yr Community Expert review: and script: https://codeberg.org/mgutt/rsync-incremental-backup/src/branch/main/incbackup.sh
April 28, 20251 yr 8 hours ago, comet424 said: rsync skips over that file thinking its done even though its a false flag that file is only 10% done with a stamp of complete Rsyncs default is to compare timestamp AND size. So this does not really make sense. And as you don't use --partial, but a partial file exists, it sounds like a problem with your rsync target: > --partial keep partially transferred files Note: If you use rsync over ssh, the rsync binary on the target server runs in --server mode and does all the actions after a disconnect as for example deleting a partial transfered file.
April 30, 20251 yr Author @mgutt so for what i ment .. is when it rsync'd it transfered the timestamp and size of the file before it fully copied the file.. i wasnt using --partial so when it rsync'd like the options i showed above i normally use.. it was in the middle of a transfer.. but rsync had set that the file was complete at the start... this was an issue where the drive on the 2nd comp lost connection and rsync though it was completely copied when it wasnt... so when rsync transfered the filename timestamp and file size.. it only copied 10% of the file.. and when i re ran the rsync rsync -avzue ssh it skips that file thinking its completed yet it wasnt completed... as it only copied 10% or 20% of the file as this is what i normally use rsync -avzue ssh --exclude-from='/boot/config/plugins/user.scripts/scripts/exclude1.txt' -s --stats --numeric-ids --progress 'source' root@$serverip:'destination' --delete >> /mnt/user/Temp/logs/cronlogs/certain log file.log @bmartino1 so i been trying rsync -avzue ssh --partial --append-verify --checksum --exclude-from='/boot/config/plugins/user.scripts/scripts/exclude1.txt' -s --stats --numeric-ids --progress 'source folder' [email protected]:'destination folder' --delete >> /mnt/user/Temp/logs/cronlogs/certain log file.log oh my god so damn slow.. so i tested just 1 folder.. of 222 files of 150gig for the folder.. the first time it took more then several hour.. it sat at incremental list for 40 min or so and when it finally completed.. i re ran to see if it go faster.. no its stuck on incremental list... so this no good files are completed under rsync i using.. as if i do like /mnt/user/Videos and say you have 40TB in there i guess it incrementals the list makes it unaccessable i noticed but it takes a day and still nothing.. and thats even with 10Gig networking.... so that is unusable.. using Krusdar was faster as when it does copyy i only get 26mb/s not like krusader at times 240mb/s so i guess the prital append is too slow so from the link yo sent about the incremental rsync thats only good for small files not for big files so good for documents,pictures but not for like video backups.. so then what is the best way to do it then if i wanna backup documents, project files, videos ,etc like i have 3 Unraid Boxes on the local network 10Gig Networking between them 2 Unraid Boxes off site using a 3mbp/s connection as its slow since rsync -avzue ssh --partial --append-verify --checksum takes days and days it looks like when u wanna start over.. and even when its complete and u just run it again for updated files... its hours... as of writting this the 2nd round of a completed folder is still going on stuck on sending incremental file list.. and its been like 20 min writting this and it was already like 20 min prior.. so this not best solution when you wanna make sure like 70 TB is backup of family files documents etc even on 10gig networking its super slow.. so the rsync -avzue ssh is good if no network failure or drive failure so it but the rsync -avzue ssh --partial --append-verify --checksum is so slow its going to take a month and if it fails network connection or what not and you restart it it start back from scratch... i did look at restic but that didnt work you need a password file and u cant even run it to create one and for whatever reason my internet was down and i couldnt install syncthing so i havent tried it... as how i do it is everynight at like 12am it runs the rsync all 4 backup servers.. and if any files need adding it rysncs and then shuts down so sometimes 5 min and done.. but now the partial i guess its checking the file... the partial didnt work if the file was already corrupted i found out so like that one file 10% when i did the --partial --apend etc it rsynced butr the file was still corrupted... so i blew out my backup server.. and started fresh with the --partal --eppend-verify --checksum and that works but takes several hours to do 222 files at 150gb in the folder been like an hour to make this reply and the --partal --eppend-verify --checksum is still stuck on sending file incremental on the folder it completed transfering so its way too slow to do vertification it transfered a true copy... what is the best for 3 10 gig servers. on the local network... and you mentioned about sftp that would work as you can use zmodem or what not to upload to to the internet the other 2 remote servers... but i like the idea of this snapshots so you can rollback like a walmart commerical lol.. so for now i just krusader copying. i guess that is safe that just uses copy but i want it all unattended at like 12am every night or once a week etc
April 30, 20251 yr Author oh ya and here is the results of using rsync -avzue ssh --partial --append-verify --checksum --exclude first time no files located in the location Number of files: 222 (reg: 201, dir: 21) Number of created files: 222 (reg: 201, dir: 21) Number of deleted files: 0 Number of regular files transferred: 201 Total file size: 153,367,530,462 bytes Total transferred file size: 153,367,530,462 bytes Literal data: 153,367,530,462 bytes Matched data: 0 bytes File list size: 0 File list generation time: 0.001 seconds File list transfer time: 0.000 seconds Total bytes sent: 153,261,702,565 Total bytes received: 3,999 sent 153,261,702,565 bytes received 3,999 bytes 32,857,049.32 bytes/sec total size is 153,367,530,462 speedup is 1.00 now the 2nd time i ran it so you know would just do a check.. it was slow as mollases several hours it took to just do a check of files sending incremental file list Number of files: 222 (reg: 201, dir: 21) Number of created files: 0 Number of deleted files: 0 Number of regular files transferred: 0 Total file size: 153,367,530,462 bytes Total transferred file size: 0 bytes Literal data: 0 bytes Matched data: 0 bytes File list size: 0 File list generation time: 0.001 seconds File list transfer time: 0.000 seconds Total bytes sent: 14,723 Total bytes received: 37 sent 14,723 bytes received 37 bytes 2.14 bytes/sec total size is 153,367,530,462 speedup is 10,390,754.10 Edited April 30, 20251 yr by comet424
April 30, 20251 yr Community Expert large files take time. extra checks add procession and performance similar to a database being constantly written too... this is where you would use other protocols and commands to copy and sync/move data between multiple different machines... I would setup a sftp transfer and let ftp handle the partial, collision and data access.. this can be done with a few dockers... the linuxserver filezila docker and a ftp docker (sftp fail 2ban is my go too) (With FTP I can at least guarantee a saturated lan and bandwidth for the file copy....) rsync can speed up based on other options files and checks. this is partial due to checksum and creating and reading md5 hash to guarantee the file. This seems more like a error in use case.
April 30, 20251 yr Author what is the largest file you wanna use the rsync --parital etc before you loose performance? from the above results... the smallest file was 274meg and largest was 4Gb out of the 221 files it reported.. pok ill install filezilla linuserver and ill install sftp fail to 2ban ill look them up and then you do it by scripts? to upload like upload resume on its own and --delete option or do you run rsync to --delete if you wanna delete a file later and that script rsync i guess that be ok when you wanna use documents and project files as they be smaller.. and can you do the same with sftp like backup rsycn script does.. always learning something new just trying to do the best performance and best backup solutions.. as who wants to learn 20 different backup ways lol.. and then will it all work with user scripts? as right now how i do it is like 1 script and i have like rsync sections 1 documents, 1 family files, 1 music, 1 videos, and the largest videos in the videos is like 30gb so that partial will never end i bet lol.. and then i have a 2 and 3 user script that does the same pointing to 2 other servers.. and then i pull up dated data from my sisters 1 unraid server to backup to my main server.. and then it backs up that data to the other 3 servers.. so thats where i like the best performance.. cuz all it be doing is backing up lol.. but let me install the filezillas and the sftp and get it configured and you can help me the next steps... do you need to see the user scripts i use
April 30, 20251 yr Community Expert no script you soucle automate... first get sftp woring and use filezilla to conect to it. if a ftp clinet was at unraids host (whcih is can vai dockers...) (But this requires pre known location and data That I dont' have nor 100% willing to share How I do it for my frinds in a ring network.) As we share our plex media across each other... However, I can give you examples and how you could do xyz... Example: Use docker exec with sftp to pull a file from a remote server docker exec -it <container_name> sftp user@remote-host inside the sftp docker vi sshd commands, one could use the client to get /remote/path/file.txt /local/path/in/container/ Then inside the interactive SFTP shell: a path has to be made within each docker as this adds a layer of file locations... Since you can call dockers different names... and example of a script using user script plugin to docker execute a command on a cron... docker exec <container_name> sh -c 'sftp user@remote-host:/remote/path/file.txt /local/path/in/container/' so ill give you a generic work flow example.. ( chose FileZilla as its client has pause, stop start, ask on overwrite and other FTP command i can use as that protocol is what your looking for some of the tasks... Is it the end all be all or only way NO.. It how you want to interact with it... SO Assumptions... Full Workflow Example Assume: Remote server: 192.168.1.100 Remote file: /data/report.csv SFTP user: sftpuser Destination path inside container: /tmp FileZilla container name: filezilla Step 1 – Transfer file from remote SFTP server to container: docker exec filezilla sh -c 'sftp [email protected]:/data/report.csv /tmp/' Step 2 – Copy file from container to host: docker cp filezilla:/tmp/report.csv /home/youruser/downloads/ both docers being mapted to the places ont eh unraid host where you want the files as the docker would see /tmp or /home/youuser.... This is an example for 1 file you could call the folder or have all the files you want marked via something else to lad individually... etc etc... Go ask ai... What I'm saying is you could use a stand in PC to connect to a unraid share and initiate a filezilla transfer and get better speed and guarantee file copy with error and logs and ability to pause stop start the transfer... Especially for large files how you chose to interact and automate that is on you go ask ai... as its clear you don't understand the rsync command and not sure what your trying to achieve... read the man pages https://linux.die.net/man/1/rsync as you can use the rsync command to connect to the sftp protocol -which you appear to be misinterpreting... Large transfers I don't automate... I Monitor... As it can TAKE HOURS! and sometimes copying to a flash drive is faster and cheper to bring to one machine to the other... Chat example ftp / sftp / rsync as, how the disk are talking to each other and how you connect to the remote machine can equally be a issue... Thus I recommend SFTP and its protcall since we are dealing with files... Like are you samba connected to the 3 machine? webdav? file over http? ... To much is unknown to help you correctly automate... Are they needing to be complete mirrors? ich777 made a plugin for that... AS I'd first set you up to do a transfer once, then look into automation... But you have to go thought he steps. Understand why you want or do x over y in the instances... as large files need a onetime copy copy and then rsync to push changes... You can even make your own docker with your own entrypoint to auto connect and transfer xyz folder... Comes down to what you want it to do and how you want to interact with it... otherwise if that not working for you use a different method... docker luckybakup there many examples in the CA and on the forum to meet your needs...
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.