Robocopy does not sync correctly

[ptr727]

I am copying files from my W2K16 server to Unraid using robocopy over SMB.

I noticed a problem where it appears that the files and folders are not being correctly synced.

 

I do:

C:\Users\Administrator\Desktop>robocopy.exe D:\Backup \\Unraid\backup /mir /r:1 /w:1 /fft /mt

-------------------------------------------------------------------------------
   ROBOCOPY     ::     Robust File Copy for Windows
-------------------------------------------------------------------------------

  Started : Wednesday, May 8, 2019 8:58:45 PM
   Source : D:\Backup\
     Dest : \\Unraid\backup\

    Files : *.*

  Options : *.* /FFT /S /E /DCOPY:DA /COPY:DAT /PURGE /MIR /MT:8 /R:1 /W:1

------------------------------------------------------------------------------

        *EXTRA Dir        -1    \\Unraid\backup\Profiles\SZ170R6V2\piete\AppData\Local\Packages\Microsoft.Windows.Cortana_cw5n1h2txyewy\L8EVMY~1\

I can run robocopy over and over, and that same directory will never delete.

While "rd \\Unraid\backup\Profiles\SZ170R6V2\piete\AppData\Local\Packages\Microsoft.Windows.Cortana_cw5n1h2txyewy\L8EVMY~1\" does delete the folder.

 

See the attached image, the directory on Unraid is there, but there are also directories that are not syncing, the "LocalState.." directory, note the double dots on the end of the folder name.

 

When I rerun robocopy, I will get the hundreds of files being reported as older on unraid, when those files have never been changed.

I am using the fat file time /FFT option to work around the lack of nanosecond granularity on EXT4/XFS vs. NTFS.

Same as the directory that does not delete, I can run robocopy clean, run it again, and it will copy a bunch of files again.

 

It is as if the file timestamps are even less granular then /FFT, or the file timestamps get changed by unraid after copy, invalidating the previous operation.

When I looked at two files where the timestamps were wrong, I can see that the unraid timestamps is not the same as that of the W2K16 server.

Is this an unraid / SMB timestamp problem?

See image.

 

I am concerned that my data copy is not accurate.

Any known problems with robocopy or differences between windows handling of names and unraid handling of names of SMB?

Edited May 9, 2019 by ptr727

[Squid]

I used to use robocopy, never had any problems like you're describing, but instead found that robocopy would consistently trash the permissions on the files making access to them difficult over a network. I switched instead to using rsync running on the server instead and never had an issue

Sent via telekinesis

[ptr727]

I don't think this is a permission problem, this is a file timestamp problem.

 

The files that will always be "older", are files that have a created date, but not a modified or accessed, interpreted as "1/1/1970 12:00:00 AM" UTC.

I wrote a .NET Core app to set the destination file timestamps to the source timestamps, does not work.

To confirm I wrote a native C++ app to do the same, same behavior.

 

robocopy NTFS to NTFS ok.

robocopy NTFS to SMB W2K16 NTFS ok.

robocopy NTFS to SMB Unraid fail.

 

I did find various bug reports of CIFS not correctly preserving times, and that in Linux there is no created time, just changed (attributes), modified (contents), and accessed times.

 

I am not certain, but it also looked as if the timestamps I set were changed during the night, i.e. I set them last night, this AM they were back to 1980 dates.

I will have to look more.

 

Try it out using the attached 7z file, extract to NTFS.

robocopy.exe "C:\Users\piete\Downloads\ttk" "\\UNRAID\Transfer\test" /mir /fft

 

The expected behavior is robocopy once, but actual behavior is older and copy over and over.

 

ttk.7z

[ptr727]

Ok, I discovered a few interesting things.

 

Unraid SMB fails to set correct timestamps if the NTFS are not linux friendly, i.e. not between 1970 and 2038.

I had several files with NTFS timestamps that were either the min value 1601 or the max value 30828.

Explorer does not display a timestamp if older than 1970.

 

I discovered some folders that ended in a ., and Unraid SMB discards the . 

I had a folder called "LocalState" and another folder "LocalState..", when copying the "LocalState.." folder is copied it into the "LocalState" folder.

 

I wrote a little app (C# .NET Core) that iterates through all the files I want to copy to Unraid, and it fixes the timestamps, and warns me of files or folders that end with a dot.

 

Once all my files were sanitized, robocopy completed successfully.

[Glassed Silver]

21 hours ago, ptr727 said:

[...]

I discovered some folders that ended in a ., and Unraid SMB discards the . 

I had a folder called "LocalState" and another folder "LocalState..", when copying the "LocalState.." folder is copied it into the "LocalState" folder. 

[...]

Hot damn, in what world is a conversion of a filename from one legal name to another legal name ever a good thing?

 

That's asking for trouble and nastly little hard to troubleshoot bugs.

 

I don't believe I have folders like that I would want to move over SMB, but certainly good to know.