Using rsync for backups screws up owner/date attributes

[CaptainTivo]

Hello again.

 

I am attempting to use rsync in a cron file to backup files.

I have one share that includes all disks except disk10 name Videos

and another share that includes only disk10 named Backup1.

 

I used this command:

 

rsync -avh --delete --progress  /mnt/user/Videos/Home\ video/dashcam /mnt/user/Backup1/backups/Tower/

 

to hopefully copy all dirs and files in dashcam/ to the backup share.

 

I was monitoring this progress and noticed that the directories and files in the target backup were not getting the source owner group permissions and modify times.

 

Here is a truncated listing.  The original is on the left and the backup on the right:

 

original share													Backup share
total 4							       total 4
drwxrwxrwx 1 nobody users   48 Jan  5  2014 folder1/	       drwxrwxrwx 1 nobody users   57 Jan  5  2014 folder1/
drwxrwxrwx 1 nobody users  120 Jul 15  2014 folder10/	       drwxrwxrwx 1 nobody users   57 Jul 15  2014 folder10/
drwxrwxrwx 1 nobody users   96 Apr  3  2017 folder100/	       drwxrwxrwx 1 nobody users   41 Apr  3  2017 folder100/
drwxrwxrwx 1 nobody users   96 Apr  8  2017 folder101/	       drwxrwxrwx 1 nobody users   41 Oct 16 21:31 folder101/
drwxrwxrwx 1 nobody users   96 Apr 16  2017 folder102/	       drwxrwxrwx 1 nobody users   41 Oct 16 21:31 folder102/
drwxrwxrwx 1 nobody users   96 Apr 24  2017 folder103/	       drwxrwxrwx 1 nobody users   41 Oct 16 21:35 folder103/
drwxrwxrwx 1 nobody users   96 May  7  2017 folder105/	       drwxrwxrwx 1 nobody users   41 Oct 16 21:35 folder105/
drwxrwxrwx 1 nobody users   96 May 16  2017 folder106/	       drwxrwxrwx 1 nobody users   41 Oct 16 21:46 folder106/
drwxrwxrwx 1 nobody users   96 May 20  2017 folder107/	       drwx------ 1 root   root    10 Oct 16 21:31 folder107/
drwxrwxrwx 1 nobody users   96 May 31  2017 folder108/	       drwx------ 1 root   root    10 Oct 16 21:31 folder108/
drwxrwxrwx 1 nobody users   96 Jun  7  2017 folder109/	       drwx------ 1 root   root    10 Oct 16 21:31 folder109/
drwxrwxrwx 1 nobody users   96 Sep  3  2014 folder11/	       drwx------ 1 root   root    10 Oct 16 21:31 folder11/
drwxrwxrwx 1 nobody users   96 Jun 21  2017 folder110/	       drwx------ 1 root   root    10 Oct 16 21:31 folder110/
drwxrwxrwx 1 nobody users   96 Jul  4  2017 folder111/	       drwx------ 1 root   root    10 Oct 16 21:31 folder111/

rest of the listing ellided

So the first three directories (folder1, folder10, folder100) and all directories and files within them had the correct owner/group and mtimes 

the next four (folder101, folder102, folder103, folder105, folder106) had the correct owner/group but the current date instead of the correct mtime

and all the rest have root/root owner/group and the current date.

I have used rsync -a before when copying from disk to disk share e.g. rsync -av /mnt/disk4/ /mnt/disk10/ and have had no problems.

 

Is there something about copying from share to share that causes it to fail or am I doing something wrong?  Obviously, backups need to have the correct dates and attributes.

 

[JorgeB]

12 minutes ago, CaptainTivo said:

Is there something about copying from share to share that causes it to fail or am I doing something wrong?

No, at least never saw it before, I rsync across servers using user shares and never had any issues, very strange, but no idea what's causing it, sorry.

[Delarius]

One thing to be aware of is that I believe rsync does the permissions last - so sometimes you'll see the incorrect permissions. Generally if you wait for the rsync to finish it should be fine but it's possible it caught your listing the directory and the permission part failed.

No reason why you can't try this disk to disk rather than through the shares - to be honest I rarely even acknowledge the shares and do almost all my operations on the disks directly.

[CaptainTivo]

15 hours ago, Delarius said:

One thing to be aware of is that I believe rsync does the permissions last - so sometimes you'll see the incorrect permissions. Generally if you wait for the rsync to finish it should be fine but it's possible it caught your listing the directory and the permission part failed.

No reason why you can't try this disk to disk rather than through the shares - to be honest I rarely even acknowledge the shares and do almost all my operations on the disks directly.

 Ahh.  This might be it.  The rsync I was doing was over 3 TB !  I was using the --progress flag so that I could monitor its progress.  When I did an ls on the target, I got the wrong permission/date info.  I was surprised at the order of the copies, too.  First it did all the top level directories then filled in each of the dir/files below that.  If it goes back later and sets the perm/date then that would explain what I saw.  I will try a smaller chunk at first and see if that fixes the problem.

Thanks.

 

As for doing it disk-to-disk, I don't really know where these files are since they are stored on a share that spans 9 disks.  In fact, that is a very good question:  is there anyway to find which disk a particular file in on?

e.g. /mnt/user/Videos/Movies/DVD/Iron\ Man  how do I find the disk that contains this directory?

 

[Delarius]

Well, this could end up being somewhat complicated, but I typically would use find.

For example - let say I want to find a directory like Iron Man and we don't know which disk it might be on:

find /mnt/disk*/Videos/ -type d -name "Iron Man"

 

-type d means only list directories (you can use f for files.)

you can also use -iname instead of -name - just makes the name case insensitive which is often easier. Sometimes you'll run into an error as the shell expands the arguments in the path - sometimes it's necessary to enclose the /mnt/disk*/Videos part in single quotes '

The fun thing about find is it can find things, and then do something to them - for example if you wanted to move that Iron Man folder to disk2/Videos/Movies:

find /mnt/disk*/Videos/ -type d -name "Iron Man" -exec mv {} /mnt/disk2/Videos/Movies/ \;

The {} represents the results of the find and you need to be sure it knows that's the end of the exec command - so needs to end with \; - obviously best to test before just firing off commands with exec setup. One final option - to use with great care is the -delete option. If you do setup a 'find' command and craft it to show files you want deleted (only files not directories) then you can just tag on -delete and it'll just delete those files.

 

 

[CaptainTivo]

Of course, I used to use find all the time back in the Unix days.  That works perfectly.

BTW, I also found that you can use rsync under Cygwin on Windoze machines and backup to the server from there.

Thanks again.