How does UnRaid handle duplicates?

[Squirreljester]

I'm in the process of moving most of my stuff to my new unraid server. I just did a comparison of what I've copied so far and I'm around 100 files shy on the unraid server. Because of the cache drive I can't just copy the files to the unraid server again because it will copy everything again.

Can someone suggest what I can do to make sure everything is copied over? I am using robocopy so it should have copied everything. I'm doing multiple folders so I have multiple of the lines below, hence the -wait.

My powershell is: start-process "robocopy.exe" -argumentlist "`"source`" `"destination`" /s /Z" -wait

 

[JonathanM]

6 minutes ago, Squirreljester said:

Because of the cache drive I can't just copy the files to the unraid server again because it will copy everything again.

That doesn't make any sense to me. Can you explain the reasoning behind it?

 

It might help if you provide an example of the "destination".

[Squirreljester]

1 minute ago, JonathanM said:

That doesn't make any sense to me. Can you explain the reasoning behind it?

 

It might help if you provide an example of the "destination".

Say I copy a bunch of movies to my unraid share called "Movies". Those files hit my cache drive first, then at 2am they are moved to my array and my cache drive is empty again.

My thinking is that if I try to copy those same files to the "Movies" share again they will first hit the now empty cache drive, and since it's empty I'm thinking they will all copy to the cache drive because it's empty. At 2am when they are moved to the array it might overwrite or skip the duplicates, I'm not sure how it would handle those.

 

Am I thinking about this wrong?

[Squirreljester]

I re-ran my script and it started copying everything normally without giving me any indication that it was skipping files that already exist. I used Windows file explorer to copy the contents of the folder to the share, it came up with an option to skip files that already exist, that copy completed, and now the folder's number of files match.

 

Is this normal behavior or do I have something set up incorrectly on my unraid server?

[trurl]

13 hours ago, Squirreljester said:

Am I thinking about this wrong?

yes you are thinking about it wrong.

 

Cache is part of user shares. The top folders on cache (or other pools) are combined with the top folders on the array to produce user shares named for those top folders.

 

Anything working with user shares will see the combined folders across all drives as being in the same place, so it will see those files already exist.

[itimpi]

30 minutes ago, Squirreljester said:

My thinking is that if I try to copy those same files to the "Movies" share again they will first hit the now empty cache drive, and since it's empty I'm thinking they will all copy to the cache drive because it's empty. At 2am when they are moved to the array it might overwrite or skip the duplicates, I'm not sure how it would handle those.

 

Am I thinking about this wrong?

Yes

 

Files normally never exist in more than one place - they will either be on the cache or on the array.   Therefore their will be no duplicates unless you actively go 'under' the covers' to create them by-passing the User Share system.   Since a User Share is a consolidated view of the files on the array AND the cache. when you copy to a User Share then you have no visibility of which drive it is going to and only new files go to the cache.

[Squirreljester]

Then why didn't my script complain that files already existed when I re-ran it? When I re-ran it it just started copying the files like the files didn't already exist.

[itimpi]

2 minutes ago, Squirreljester said:

Then why didn't my script complain that files already existed when I re-ran it? When I re-ran it it just started copying the files like the files didn't already exist.

 

Without knowing more about the script difficult to say.  If files already exist then you get the same overwrite behaviour regardless of whether they are on cache or array as tke script will not know where they are.

 

[trurl]

I haven't used robocopy in a really long time. Are you sure it would prompt you if it found identical files? Or only if it found files with the same path that weren't identical? And is that behavior controlled by options?

 

You said it ended up with the correct number of files.

[Squirreljester]

I'll have to test it and see if it would prompt to overwrite, but I don't see anything in the options for skipping existing files, and I'm not doing and force options, so that leads me to think that it's default behavior is to either skip or prompt for existing files.

 

25 minutes ago, trurl said:

You said it ended up with the correct number of files.

It ended up with the correct number of files after I did another copy using file explorer in Windows, and chose the option to skip existing files. When I re-ran the script I used to copy the files in the first place, it just started copying away like there weren't already existing files.

[itimpi]

7 minutes ago, Squirreljester said:

When I re-ran the script I used to copy the files in the first place, it just started copying away like there weren't already existing files.

That would be consistent with it silently over-writing existing files.

 

Perhaps you need to do a test locally within the windows system to see how it is handling copying when files already exist. 

[Squirreljester]

On 8/18/2021 at 9:51 AM, itimpi said:

That would be consistent with it silently over-writing existing files.

 

Perhaps you need to do a test locally within the windows system to see how it is handling copying when files already exist. 

I'm using robocopy's /E /XC /XN /XO switches now to skip files that already exist. It's been quite helpful copying my library to my unraid array.