6.3.0 - SMB, OS X and Carbon Copy Cloner - Modifying Metadata?


22 posts in this topic Last Reply

Recommended Posts

Chaps,

 

What has changed in 6.3.0 that would cause this?

 

I noticed today that CCC was copying data from my Photo Library to the Array - the problem being the data on this disk hasn't changed in months.

 

If I look at the CCC archives - all the files it has copied have a modified data of approximately when CCC accessed that directory. In some cases part, some cases all the files within that folder would have the modified time and date of whenever CCC accessed it.

 

Result being every time I ran CCC it would copy over more and more data that hadn't changed in years.

 

To confirm it wasn't my mac or CCC, I backed up to my pair of offline disks - works as expected.

 

I've reverted back to 6.2.4 and ran it 4 times - bar copying the files on the first which had already had their metadata changed by 6.3.0, it did not copy anything else.

 

Nothing in the logs that I can see - but as far as I am concerned 6.3.0 is seriously broken. Ended up with about 4000/40GB of files changed to today's date. MD5 of both original and changed date files are the same.

 

Link to post

Chaps,

 

What has changed in 6.3.0 that would cause this?

 

I noticed today that CCC was copying data from my Photo Library to the Array - the problem being the data on this disk hasn't changed in months.

 

If I look at the CCC archives - all the files it has copied have a modified data of approximately when CCC accessed that directory. In some cases part, some cases all the files within that folder would have the modified time and date of whenever CCC accessed it.

 

Result being every time I ran CCC it would copy over more and more data that hadn't changed in years.

 

To confirm it wasn't my mac or CCC, I backed up to my pair of offline disks - works as expected.

 

I've reverted back to 6.2.4 and ran it 4 times - bar copying the files on the first which had already had their metadata changed by 6.3.0, it did not copy anything else.

 

Nothing in the logs that I can see - but as far as I am concerned 6.3.0 is seriously broken. Ended up with about 4000/40GB of files changed to today's date. MD5 of both original and changed date files are the same.

What's it doing exactly?  Only changing the timestamps or is it actually transferring the data?

Would be useful to see diagnostics.zip.

Link to post

Something is changing the files time/date to the run time on the server.

 

CCC sees that the server file != master copy, moves the server file to the archive folder and copies the master copy over again.

 

So now I have the correct time/date files along with an identical md5 file in the archive with today's time and date as created and modified.

 

Also I noticed that in some cases the same file has been modified twice and now I have three copies on the server, one correct time/date, then two time stamped at approximately the backups run time.

E.g.

 

UK time/date format...

Original 12:34 1-2-2007

Run 1: 17:54 7-2-2017

Run 2: 18:32 7-2-2017

With all due respect, there is absolutely no chance I'll be running 6.3.0 again. Wasted 3 hours last night trying to work out what was going on.

 

Obviously doesn't help you, but I cannot risk this happening again. Lucky I noticed it otherwise this could have gone on for weeks and really buggered my backups up.

 

Edit: Could it be a conflict with http://lime-technology.com/forum/index.php?topic=44989.0

 

Although I hadn't run that script in weeks as data hadn't changed (and the files being modified were typically 5+ years old, 2007 dates IIRC.)

 

Edit 2: If I look at CCC backups where the file has been moved to the archives because the master has changed, the file retains its original timestamps, both created and modified, because all CCC is doing is moving it.

 

I.e. If the old file was created/modified 13:57 9-9-2012, then the file in the CCC archives will keep that date.

 

As I say, going back to 6.2.4 allowed all my backups to complete correctly.

 

 

Link to post

Something is changing the files time/date to the run time on the server.

 

CCC sees that the server file != master copy, moves the server file to the archive folder and copies the master copy over again.

 

So now I have the correct time/date files along with an identical md5 file in the archive with today's time and date as created and modified.

 

Also I noticed that in some cases the same file has been modified twice and now I have three copies on the server, one correct time/date, then two time stamped at approximately the backups run time.

E.g.

 

UK time/date format...

Original 12:34 1-2-2007

Run 1: 17:54 7-2-2017

Run 2: 18:32 7-2-2017

With all due respect, there is absolutely no chance I'll be running 6.3.0 again. Wasted 3 hours last night trying to work out what was going on.

 

Obviously doesn't help you, but I cannot risk this happening again. Lucky I noticed it otherwise this could have gone on for weeks and really buggered my backups up.

 

Edit: Could it be a conflict with http://lime-technology.com/forum/index.php?topic=44989.0

 

Although I hadn't run that script in weeks as data hadn't changed (and the files being modified were typically 5+ years old, 2007 dates IIRC.)

 

Edit 2: If I look at CCC backups where the file has been moved to the archives because the master has changed, the file retains its original timestamps, both created and modified, because all CCC is doing is moving it.

 

I.e. If the old file was created/modified 13:57 9-9-2012, then the file in the CCC archives will keep that date.

 

As I say, going back to 6.2.4 allowed all my backups to complete correctly.

For the second time: Would be useful to see diagnostics.zip.

Link to post

Sent to tomm@lime... - couldn't do it when I posted this morning.

 

Those diags are from running 6.2.4.  After upgrade to 6.3 (always use latest, as of this post it's 6.3.1), boot in 'safe mode' - this will prevent plugin installation.  Not sure how a plugin could be causing this issue but honestly not sure what the issue is.  Nothing we changed in 6.3 would cause timestamps to somehow change.  Could be something changed in an  upgraded component (like Samba) which is causing a different behavior.  Just saying: might be  hard to track this down.

Link to post

Sent to tomm@lime... - couldn't do it when I posted this morning.

 

Those diags are from running 6.2.4.  After upgrade to 6.3 (always use latest, as of this post it's 6.3.1), boot in 'safe mode' - this will prevent plugin installation.  Not sure how a plugin could be causing this issue but honestly not sure what the issue is.  Nothing we changed in 6.3 would cause timestamps to somehow change.  Could be something changed in an  upgraded component (like Samba) which is causing a different behavior.  Just saying: might be  hard to track this down.

 

True however IF 6.3.0 is the cause, then I really REALLY don't want to put my data at risk again.

 

Catch 22 situation I appreciate but there is nothing I can do to mitigate this. Although I do have a crashplan backup of this data so perhaps I could grab back that data...

Link to post

Upgraded today with plugins disabled - ANother set of data has been changed.

 

6.3.X is 100% broken for me.

 

Diagnostics on their way to you soon.

 

I got that, thanks.  Did you verify you see the same behavior in 6.3.1 vs. 6.3.0?  The reason I ask is I see that all your data disks use xfs and there were a number of xfs "fixes" in linux kernel 4.9.8 which is what was used in 6.3.1.

 

Note that unRaid 6.2.4 is on the linux 4.4.x kernel and unRaid 6.3.x is on linux 4.9 kernel.  If we can figure out why CCC thinks files are changing we would have a better chance of bisecting kernel to find the issue.

Link to post

Upgraded today with plugins disabled - ANother set of data has been changed.

 

6.3.X is 100% broken for me.

 

Diagnostics on their way to you soon.

 

I got that, thanks.  Did you verify you see the same behavior in 6.3.1 vs. 6.3.0?  The reason I ask is I see that all your data disks use xfs and there were a number of xfs "fixes" in linux kernel 4.9.8 which is what was used in 6.3.1.

 

Note that unRaid 6.2.4 is on the linux 4.4.x kernel and unRaid 6.3.x is on linux 4.9 kernel.  If we can figure out why CCC thinks files are changing we would have a better chance of bisecting kernel to find the issue.

 

The changed file dates are visible in finder and via the 'ls' command.

 

CCC scans all the files, if the file on the server != what is on the source, it moves the file on the server to the CCC Archive folder. This file is not modified, the creation and modification date stays the same, etc.

 

With respect to my photo drive - RAW (CR2) files are never modified and other file types very rarely modified.

 

When I upgraded to 6.3.0 last week, and 6.3.1 today, for some reason, random files are getting their date and time modified to the time at which CCC passes through that folder (seemingly that is what happens, if I run it multiple times 5 minutes apart the timestamps are then roughly 5 mins apart...)  Hence why I noticed that it was suddenly backing up thousands of files totalling 40+GB when nothing has changed.

 

So now in the _CCC Archive folder, I have loads of photos (for example ones back from 2007-2009!) that now have todays time and date.

 

It, as far as I can tell is completely random, although the majority of the files (any file type, CR2 are the majority but that is only because I shoot in RAW) are dated 2007 or earlier, .tif files with any date, with a smattering of other file types/dates.

 

I have no idea how we are going to go about solving this because every time I run it I add more date changes to the list.

 

I don't know if the dates are being modified when CCC scans the folder, or when the file is moved to the archive (i.e. the timestamps are getting 'corrupted' by the move).

 

If you think that the xfs changes have something to do with it - I would be prepared to risk my data once more using 6.3.1 with 6.2 kernel, if that is possible?

Link to post

Yes I can see how this is frustrating.

 

 

The changed file dates are visible in finder and via the 'ls' command.

 

CCC scans all the files, if the file on the server != what is on the source, it moves the file on the server to the CCC Archive folder. This file is not modified, the creation and modification date stays the same, etc.

So I'm clear: the CCC program is running on a client OS X machine (your Mac) and comparing files there against copies of those files on the server, correct?

During a scan is it referencing a "user" share on the server or a "disk" share?

It appears you are using SMB and not AFP, correct?

 

With respect to my photo drive - RAW (CR2) files are never modified and other file types very rarely modified.

 

When I upgraded to 6.3.0 last week, and 6.3.1 today, for some reason, random files are getting their date and time modified to the time at which CCC passes through that folder (seemingly that is what happens, if I run it multiple times 5 minutes apart the timestamps are then roughly 5 mins apart...)  Hence why I noticed that it was suddenly backing up thousands of files totalling 40+GB when nothing has changed.

 

So now in the _CCC Archive folder, I have loads of photos (for example ones back from 2007-2009!) that now have todays time and date.

Can you give me an example?  I think what you're saying is this:

 

on server:

  /share/animals/cat.cr2  is dated 1/1/2007

 

on client:

  source/animals/cat.cr2  is dated 1/1/2007

 

after CCC runs:

  /share/animals/_CCC/cat.cr2 is dated with today's date

  /share/animals/cat.cr2 is dated 1/1/2007  but this file has been re-transferred from the client

 

 

Link to post

Correct, CCC runs on my Mac via SMB. AFP is disabled.

 

CCC has its own user e.g. 'ccc' and mounts the disks itself (as a semi-protection for crypto-lockers).

 

It directly goes to a disk shares (disk1 and 2 in this case) as the scan performance is generally a lot quicker when I set it up two years or so ago rather than going via my 'Backups' user share (which is on disk1 and 2).

 

And finally, yes you are pretty much correct, except the _CCC is in the root folder along with the date and time the backup started.

 

I'll email you a screenshot of the tree.

 

And to confirm, on 6.2.4:

 

on server:

  /share/animals/cat.cr2  is dated 1/1/2007

 

on client:

  source/animals/cat.cr2  is dated 12/02/2017 (I.e. today)

 

after CCC runs:

  /share/animals/_CCC/cat.cr2 is dated 1/1/2007

  /share/animals/cat.cr2 is dated 12/02/2017 (I.e. today)

Link to post

Got your screenshot ... I’m still a little confused what’s happening but it occurs to me what the issue might be.  Does it ever re-backup the same file more than once?

 

One of the other things we added in 6.3 was the so-called “vfs_fruit” optimizations to speed operations with OS X via SMB.  Part of enabling that is configuring shares to have “ea support” (extended attributes).  The file systems used by unRaid: btrfs, xfs, reiserfs, all support extended attributes.  Telling Samba that shares support extended attributes (and thus clients can use them) should be “harmless”.

 

But it could be that CCC wants to store some kind of metadata in an extended attribute  but in previous unRaid OS releases this would have failed, and so CCC fell back to some other default behavior.  But now, CCC sees that the shares do support extended attributes, but it gets confused because the extended attribute it thought should’ve been created earlier isn’t there…..  make sense?

 

At the bottom, this page does talk about extended attributes – maybe check your CCC settings to see how this is configured?

https://bombich.com/kb/ccc4/advanced-settings

 

Link to post

Got your screenshot ... I’m still a little confused what’s happening but it occurs to me what the issue might be.  Does it ever re-backup the same file more than once?

 

One of the other things we added in 6.3 was the so-called “vfs_fruit” optimizations to speed operations with OS X via SMB.  Part of enabling that is configuring shares to have “ea support” (extended attributes).  The file systems used by unRaid: btrfs, xfs, reiserfs, all support extended attributes.  Telling Samba that shares support extended attributes (and thus clients can use them) should be “harmless”.

 

But it could be that CCC wants to store some kind of metadata in an extended attribute  but in previous unRaid OS releases this would have failed, and so CCC fell back to some other default behavior.  But now, CCC sees that the shares do support extended attributes, but it gets confused because the extended attribute it thought should’ve been created earlier isn’t there…..  make sense?

 

At the bottom, this page does talk about extended attributes – maybe check your CCC settings to see how this is configured?

https://bombich.com/kb/ccc4/advanced-settings

 

 

I think it has backed up the same file more than once - can't check now unfortunately as I've sorted the backups out now from Crashplan!

 

The extended attributes check box is not ticked (and has never been).

 

Can I disable vfs_fruits?

 

Link to post
  • 1 month later...
  • 2 weeks later...

In the 6.4 series there will be some significant changes in the way shfs handles timestamps.  This is to address this issue:

And address other backup-related issues.  Hopefully these changes will correct the problem your backup program is exposing as well.

Link to post
2 hours ago, Zonediver said:

I saw an other behavior since i use 6.3.x - some files (from 2010-2013) got a new date: 22.02.2022 O.o

 

With respect, this post adds nothing unless you can provide more details.  What were the circumstances under which this happened?  Were you using a backup program or just transferring files?  Did it result after file was moved by mover?  Can you reproduce?  This might not even be an unRaid issue depending on answers to this.

Link to post
1 hour ago, limetech said:

 

With respect, this post adds nothing unless you can provide more details.  What were the circumstances under which this happened?  Were you using a backup program or just transferring files?  Did it result after file was moved by mover?  Can you reproduce?  This might not even be an unRaid issue depending on answers to this.

 

It might be possible that this has nothing to do with unraid, but i saw this on various files after upgrading to 6.3 and no, this doesnt happen during a file transfer, it happens to files they are already on my unraidsystem. So i thought, i mention this - maybe it helps - but you clarify, that this has nothing to do with unraid. Then it must be an other issue...

The files which are affected lieing on unraid since 2010-2013 - as i mentioned.

I use my unraid only with plex, so when i copy files to unraid, they will never change again - like in a big vault.

But its already strange that files, which lieing on unraid since years, changing their date just so - with no reason... O.o

Edited by Zonediver
Link to post
16 hours ago, limetech said:

In the 6.4 series there will be some significant changes in the way shfs handles timestamps.  This is to address this issue:

And address other backup-related issues.  Hopefully these changes will correct the problem your backup program is exposing as well.

 

 

Excellent, thanks.

 

Might create a test backup on a small duplicated subset to test with on this occasion!

Link to post
  • 1 month later...

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.