I had posted this in another thread, but the original issue was resolved so I wanted to give this it's own home.

 

I used Carbon Copy Cloner on my Mini to copy my media over to unRAID prior to assigning parity and actually using the drive.  CCC is pretty just a pretty frontend that uses rsync on the backend.  I've never had any issues with it in the past so I didn't think it would cause me issues now.

 

I setup one single user share, Media, and synced my movies and tv directories into it so it looks like this:

 

Media

>movies

>>movie directory

>>>movie file(s)

>tv

>>tv directory

>>>Season(s)

>>>>>tv file(s)

 

After my initial clone completed I wanted to run it again to pick up any new files that were put in prior to starting the sync.  These should have ONLY been TV shows.  Right when I started the sync and it started comparing files, it picked up on a Movie that was already there (on disk 1) and proceeded to start putting it in again (on disk 2).  I stopped the sync and started to do some manual compares of things.

 

After this I shut down CCC and started it again and then tried the process on a single folder that I knew had a new episode on it.  That worked perfectly and unRAID created a new folder on disc 2 and placed the episode in there.  Great.

 

So I decided to run it on the entire TV directory which should have added about 6 or 7 new shows.  Well...  unRAID instead is moving files around from one drive to the other and I have no idea why.  Example:

 

Criminal Minds Season 2 - all episodes existed on disk 1.  There should have been no need to add any files. Now there is one episode that got placed on disk 2 and the one on disk 1 removed.

 

This is completely baffling behavior considering CCC is just a frontend for rsync and I know others are using that here.  What in the world is going on here?  Syncing up should only be adding/removing changed files, but that is absolutely not what's happening.  Maybe it's a timestamp thing?  Size thing?  Permissions thing since I'm sending from the Mini to UnRAID and not the other way around?  I have no freaking clue.

 

After that sync finished copying files it shouldn't have, I set about using rsync from the command line and the same damn thing is happening.  As soon as I run a command it's trying to copy files that are already there.  I'm at a loss here guys.  None of this is making any sense.

 

I'm randomly checking timestamps and everything looks identical.  So why can I perform a simple sync from my Mac to unRAID without sending the same files over and over?  I don't get it...

 

It's a good thing I haven't setup the parity drive.  All the back and forth I'm doing would be brutal.

 

I am wanting to get this array 100% up so I can enlarge my current drive array (which means destroying all the data) and use this as the main in the interim.  All this behavior is making me very nervous.  How the heck can I tell if everything is synced up 100% if every time I sync it's adding files it shouldn't?

 

HELP?!

Clarification questions of what you're doing.

1. you're using 'user shares' for all of this, correct? You're not copying to 'disk 1' or 'disk 2'?

2. If you set any of the user share options for 'include', 'exclude', or 'split level', then what did you set for those options for the TV and Movie shares?

3. What did you use for the CCC settings? I haven't used CCC in a while, but as I recall, there's a way to config it so that it clones insertions and deletions based upon the source. (so if you delete on the source, it deletes on the target.)

CCC is especially good at EXACT duplicates of Mac disks and files...right down to time stamps and bit flags. I'm pondering how a file 'moved' from Disk 1 to Disk 2. unRAID doesn't do that...CCC had to have done the deletion from Disk 1 and the write of the new copy on Disk 2.  I believe CCC would do that for something as simple as a date change in the file info...so something you did on your Mac with that file could have triggered a flag that caused a CCC update on unRAID.

 

Clarification questions of what you're doing.

1. you're using 'user shares' for all of this, correct? You're not copying to 'disk 1' or 'disk 2'?

2. If you set any of the user share options for 'include', 'exclude', or 'split level', then what did you set for those options for the TV and Movie shares?

3. What did you use for the CCC settings? I haven't used CCC in a while, but as I recall, there's a way to config it so that it clones insertions and deletions based upon the source. (so if you delete on the source, it deletes on the target.)

CCC is especially good at EXACT duplicates of Mac disks and files...right down to time stamps and bit flags. I'm pondering how a file 'moved' from Disk 1 to Disk 2. unRAID doesn't do that...CCC had to have done the deletion from Disk 1 and the write of the new copy on Disk 2.  I believe CCC would do that for something as simple as a date change in the file info...so something you did on your Mac with that file could have triggered a flag that caused a CCC update on unRAID.

 

Thanks for the reply Dale.  I might have this figured out, but before I celebrate how about I make sure I actually set up the share(s) properly.  Cause if I didn't then I have to start this 3 day long copy process again.

 

At first I thought I'd setup 2 shares, movies and tv, and then sync those 2 folders from my Mini and it's array.  But than I thought about it and figured why not just create one share for "Media", split level blank, nothing included/excluded, and then just copy in the movies and tv directories from my Mini and let unRAID handle all everything from there.  This seemed to work well except the odd issue we discussed in the other thread, but I'm pretty sure I know what happened there.

 

So that's the setup.  Pretty crackin simple.

 

1. I was copying to the Media share and NOT the individual disk(s).

2. Explained above.

3. CCC settings were pretty vanila.  No filters or exclusions.  Initially trying with "Delete anything that doesn't exist on the source".

 

As for the weird movement of files you mentioned, I'm pretty sure that happened since disk 1 was already at the half full mark and disk 2 was not.  So whatever unRAID was adding again, it was doing it to disk 2.  Which just looked damned odd, but probably completely normal.

 

So, just now I did a little test.  I wanted to do this from rsync, but all the changes I made to the inetd.conf (to run rsync as a daemon) and the created rsyncd.conf file I created in /etc/ on unRAID were all gone after a reboot.  I don't know how to keep that from happening right now so I just dug into CCC a bit more.

 

Being just a frontend to rsync I figured I just needed play with more settings and tried the checksum setting.  Bad idea since these files are HUGE it takes forever to check just one file.  So then I just set it so it "don't modify files newer on destination" and THAT seemed to do the trick.  Nothing got copied again.  Of course this would also mean that if for some reason I did something on unRAID first it would be ignored, but that's OK at this point.

 

So, that issue is solved, thank goodness, but the mystery of why rsync kept thinking files and/or folders were modified when they weren't is beyond me.  A little googling and i see this is actually a common issue in some cases with no clear cut reason as to why though.  unRAID might be modifying the folder timestamp or something like that. 

 

I dunno.  In all the time I've used CCC or rsync for that matter, I've never seen this happen.  But those times were always between 2 Macs or from a Mac to an external drive or between 2 drives on one Mac.  I've never tried to do this to a target that was a different OS and/or file system.  Heck, for all I know it could have something to do with using AFP shares.

 

If you can shed any more light on this that would be great, but it appears as though I have it working mostly.  Still love to know exactly why it was happening though or how best to do this in the future. 

Something just dawned on me...  Could it be that once I accessed a folder via AFP and thus OS X generates it's hidden files that this is what's causing rsync to think things with the folder itself have changed and thus just starts copying the entire thing again?

 

Just a thought...

From your summary, I think you're fine. And your unRAID array is fine.

Everything you say makes perfect sense in terms of unRAID operations.

And AFP versus SMB won't matter...don't sweat it. The upside of AFP is that TimeMachine works great (and there is some evidence that iTunes library works better over AFP). If you're on Mavericks, SMB is the default for about everything else, and also works great. The downside to AFP is that it can leave '.dot' file clutter around. Most of us just ignore that stuff.

 

MOST users also don't pick apart their array to see which physical disks hold the files from a copy to a user share... Your earlier comment about 'a second copy command' moved a file from disk to disk doesn't compute...that's GOT to be a CCC issue...I just don't know enough to point to where/how it does it. unRAID does NOT move files from physical disk to physical disk.

 

(should you ever NEED to move a file from disk1 to disk2, it can certainly be done using disk shares from either the console and MidnightCommander, or your Mac Finder, but unRAID's file management will NEVER move files on its own.)

 

 

Thanks again. I'm not on Mavericks just yet. I was waiting till that got a few more point releases before I update my systems. And yeah, I pretty much ignore all the hidden files.

 

Being as my first time using unRAID I wanted to see what it was doing and won't be poking around at the disk level too much. I still can't explain completely what I was seeing except to say it wasn't so much moving the files as it was adding them again and then deleting the other one. This makes it appear as though it was moving the file, but in reality it wasn't.

 

 

 

 

Sent from my iPad using Tapatalk

I wanted to do this from rsync, but all the changes I made to the inetd.conf (to run rsync as a daemon) and the created rsyncd.conf file I created in /etc/ on unRAID were all gone after a reboot.

Just a quick note about this so you don't keep chasing your tail in the future. When unraid boots, it unpacks the entire file system into RAM, and runs from there. The only locations that are "real" are located in /boot and /mnt/disk(X), everything else is gone and created anew when you boot. So... if you make changes to /etc config files and want the changes to be persistent, you will need to write a little script to make those same changes at boot time, and store it in /boot and call it from the go file.

I wanted to do this from rsync, but all the changes I made to the inetd.conf (to run rsync as a daemon) and the created rsyncd.conf file I created in /etc/ on unRAID were all gone after a reboot.

Just a quick note about this so you don't keep chasing your tail in the future. When unraid boots, it unpacks the entire file system into RAM, and runs from there. The only locations that are "real" are located in /boot and /mnt/disk(X), everything else is gone and created anew when you boot. So... if you make changes to /etc config files and want the changes to be persistent, you will need to write a little script to make those same changes at boot time, and store it in /boot and call it from the go file.

 

Thanks Jonathan.  The funny thing is, I knew this was the case, but when I was hunting for instructions setting up rsync as a daemon they didn't mention that.  Once I saw the changes were gone I knew what happened.  I've since found instructions on setting this up so it can be put in at boot time, but haven't actually tried it yet.