correct commads to copy to XFS

[loady]

i am in the process of moving my array disks from RFS to XFS, having precleared a new 3tb drive i followed a post by weebo using rsync, the code i used was..

 

cd /mnt/disk3

rsync -avPX . /mnt/disk5

 

disk 5 being my newly precleared drive formtted to xfs. it is copying over via a screen session but i noticed on the windows explorer at the shares i have a 'disk5' share..why has this happened ? i didnt actively create it.

 

Not entirely sure i am going about performing this move correctly, i want to move the disks over to xfs and i have plenty of time so i want everything to be verified then have duplicates and empty folder deleted, not sure how to implement commands for my usage. from the second picture i intend to move,,

 

DISK 3 TO NEW DISK 5

DISK 1 TO DISK 3

DISK 2 TO DISK 1

DISK 1 TO DISK 4

 

 

 

[itimpi]

i am in the process of moving my array disks from RFS to XFS, having precleared a new 3tb drive i followed a post by weebo using rsync, the code i used was..

 

cd /mnt/disk3

rsync -avPX . /mnt/disk5

 

disk 5 being my newly precleared drive formtted to xfs. it is copying over via a screen session but i noticed on the windows explorer at the shares i have a 'disk5' share..why has this happened ? i didnt actively create it.

unRAID will create a share for each physical device by default.  if you want to disable this then click on the disk in question in the Main tab and you will then be able to adjust the SMB Security settings which control this.

[loady]

Ok. ..just that this was not the case for the current disk's. ...so have I given the correct command to move these files ?

[itimpi]

Ok. ..just that this was not the case for the current disk's

Not sure what you mean by this?  Both V5 and v6 both work like this.

[NAS]

Always worth considering using diskmv

 

http://lime-technology.com/forum/index.php?topic=36201.15

[lionelhutz]

Check your disks for a top level directory called "disk5". If you created this directory by mistake then it gets shared automatically.

[itimpi]

Check your disks for a top level directory called "disk5". If you created this directory by mistake then it gets shared automatically.

Easiest way to identify this is:

• Go to the Shares tab in the GUI.  If disk 5 shows up there then you have a top level directory called disk5
  
• Click on the folder icon to the right of the disk5 entry - that will show you what is in that share and what disk(s) it is on.
  

[loady]

Check your disks for a top level directory called "disk5". If you created this directory by mistake then it gets shared automatically.

Easiest way to identify this is:

• Go to the Shares tab in the GUI.  If disk 5 shows up there then you have a top level directory called disk5
  
• Click on the folder icon to the right of the disk5 entry - that will show you what is in that share and what disk(s) it is on.
  

 

so this is what i see...its only in windows it shows as a share..i never wrote 'disk5'..as did i not disk 1,2,3,4..unraid set that.

 

[loady]

Ok. ..just that this was not the case for the current disk's

Not sure what you mean by this?  Both V5 and v6 both work like this.

 

i didnt name the disks 1,2,3,4,5 it was done by unraid at formatting ?....the disk 5 only shows as a share in windows explorer..disk 1,2,3,4 do not and i dont  believe i done anything different from when i first set up the other drives..maybe that command i typed to start copying files over has something to do with it..im not confident in what i have done thus far...as for using diskmv that sounds a bit 'at your own risky'

 

really all i want to do is copy data from one disk to another with verification and duplicates and empty folders removed.

[trurl]

Your screenshots indicate that you don't have a user share named disk5, so you must be sharing the actual disk5. Click on disk5 on the main page and you can set SMB Security for that disk if you want to change that.

 

I normally have all my disks set to Export: Yes (hidden) so they don't show up when browsing, and Security: Private with just my account having access. That way I can get to them if I really want to, but nobody else can, and I don't even see them unless I actually specify the path to get to them.

[loady]

thanks...i must have set that up to be like that..dont remember doing but it would be something i would have done.

 

copy process seems to be finished now leaving this message... sent 2,741,641,156,099 bytes  received 4,719,513 bytes  23,173,505.73 bytes/sec

total size is 2,744,040,158,095  speedup is 1.00

 

does it seem ok ?...given the command i used to start the copy over does it mean that it has been verified or do i need to fire off another command ? i would like to delete any empty folders..i assume it deletes duplicate files as opposed to duplicate directories ?

seems to have taken an awful long time to copy, longer than it did to preclear i would say ?

[loady]

If someone could possibly clarify this post from weebotech...

 

I would use rsync to copy/move files from one disk to another via the command line.

 

rsync will compare modification time and size, then skip the files that already exist.

 

example usage would be

cd /mnt/disk5

rsync -avPX . /mnt/disk6

you can use the -n flag to do a dry run and see what would be copied (I almost always do that first)

 

rsync -n -avPX . /mnt/disk#

see what spits out

then do it without the -n

 

AFTER all is said and done you can do another rsync to compare files by checksum rather them modification time and size

 

do this with

 

rsync -n -rcvPX . /mnt/disk#

This will show you what might get copied because the files did not compare via checksum.

If you want to do the actual copy again for files that do not compare take off the dry run flag (-n)

 

My final command is usually to remove what compares with --remove-source-file.

 

rsync --remove-source-file -rcvPX . /mnt/disk#

 

This has the effect of comparing the tree with checksums and removing files that match.

After that I do a final review down the tree with find to make sure there are no files left over

 

find . -type f -ls

 

and a final cleanup with

 

find . -depth -type d -empty -ls -delete

which removes empty directories

 

Tips:

When doing this. Always be careful of where you are.

Always use the -n flag first to make sure you get the expected results.

 

I started with this

cd /mnt/disk3
rsync -avPX . /mnt/disk5

 

i have just done this...

rsync -n -rcvPX . /mnt/disk5
This will show you what might get copied because the files did not compare via checksum.
If you want to do the actual copy again for files that do not compare take off the dry run flag (-n)

and it is listing loads of small files, was i right to perform both operations or should i have done one or the other, so i may need to do the last command without the -n prefix because all those files it listed have not copied over for some reason ??

[gundamguy]

If someone could possibly clarify this post from weebotech...

 

[Removed to reduce quote size]

 

I started with this

cd /mnt/disk3
rsync -avPX . /mnt/disk5

 

i have just done this...

rsync -n -rcvPX . /mnt/disk5
This will show you what might get copied because the files did not compare via checksum.
If you want to do the actual copy again for files that do not compare take off the dry run flag (-n)

and it is listing loads of small files, was i right to perform both operations or should i have done one or the other, so i may need to do the last command without the -n prefix because all those files it listed have not copied over for some reason ??

Well not sure but the major difference between your first rsync and second is that you are in test mode, and you are comparing checksums instead of comparing size and modification dates.  If the trial run is showing a ton of files to me it means either those files didn't copy before for some reason, or the checksums don't match match for some reason. Either way this isn't a good thing.  Are you sure nothing is writing or modifying files on disk3?

 

One thought is that your second run isn't in archive mode -a but I'm not sure that's what would have it returning a long list of differences.

[loady]

only the original copy over which is finished given this output ?.....sent 2,741,641,156,099 bytes  received 4,719,513 bytes  23,173,505.73 bytes/sec

total size is 2,744,040,158,095  speedup is 1.00

 

dockers are disabled and nothing is connected to the server other than what i m doing.

 

how difficult can this be.

 

EDIT: its been nearly two hours now just listing files not actually copying ??..its started listing large .mkv files as well now which were on that disk but i physically checked that they were present on the new disk are they are..so whats this actually doing now ??

[gundamguy]

only the original copy over which is finished given this output ?.....sent 2,741,641,156,099 bytes  received 4,719,513 bytes  23,173,505.73 bytes/sec

total size is 2,744,040,158,095  speedup is 1.00

 

dockers are disabled and nothing is connected to the server other than what i m doing.

 

how difficult can this be.

 

EDIT: its been nearly two hours now just listing files not actually copying ??..its started listing large .mkv files as well now which were on that disk but i physically checked that they were present on the new disk are they are..so whats this actually doing now ??

 

If you have to generate checksums for a lot of files and some very large files it might take a long time to generate the file list. As to why it might be counting a file that that you've checked are in both places. Two possibilities, normally rsync checks size and timestamp to tell if a file has changed, however if you are using the -c attribute rsync compares the source file's checksum to the destination files checksum and decides what needs to be copied based on if the checksums match. The short answser here is that the checksums dont' appear to match, but I'm not sure why that would be the case.

[loady]

well i wen ahead to remove the files from the disk3 that are now supposed to be copied to disk5 by typing within a screen session

 

cd /mnt/disk3

rsync -remove-source-file -rcvPX . /mnt/disk3

 

if i am correct this command should remove anyfiles from disk3 that are now copied to disk5 (new disk)..the session just seems to have stalled after 6 hours...disk3 in the main tab on gui is showing as 2.74GB used and 265GB free and disk 5 is stating as 2.74GB but 254GB free...clearly there is less data on disk5 or does the XFS file system uses different cluster sizes or something else to make it report less??..in any case how am i sure that everything is copied over..im at a bit of a loss since i am not to versed is command line stuff

[itimpi]

well i wen ahead to remove the files from the disk3 that are now supposed to be copied to disk5 by typing within a screen session

 

cd /mnt/disk3

rsync -remove-source-file -rcvPX . /mnt/disk3

 

if i am correct this command should remove anyfiles from disk3 that are now copied to disk5 (new disk)..the session just seems to have stalled after 6 hours...disk3 in the main tab on gui is showing as 2.74GB used and 265GB free and disk 5 is stating as 2.74GB but 254GB free...clearly there is less data on disk5 or does the XFS file system uses different cluster sizes or something else to make it report less??..in any case how am i sure that everything is copied over..im at a bit of a loss since i am not to versed is command line stuff

No that would not work.  You are trying to copy the current directory (disk3) to disk3!    The last parameter should be disk5 (i.e. the target for the files). Not sure what rsync does if source and target are the same.

[loady]

so i tried again this time stipulating disk5, here is a sample of what its doing....

 

skipping non-regular file "downloads/complete/Big.Hero.6.2014.1080p.BluRay.x264-SPARKS[rarbg]/Subs/Big.Hero.6.2014.1080p.BluRay.x264-SPARKS.sub"

 

seems to be suggesting that its not doing anything...i thought the command was to remove/delete the files from disk3 that have successfully copied over to disk5 ?? im more concerned about losing my family pictures...i have been physically able to access them from a windows share and i hunted down the folders to thier locations on the disks, disk 5 contains the same folders on disk 3 thats i am about to reformat now...safe to say i can proceed ??

[JonathanM]

safe to say i can proceed ??

Not necessarily. Given all the issues you've been having with trying to do this with the command line, I think you would be better off turning on the disk shares and operating from a client you are more familiar with. That way you could use gui duplicate file utilities to make sure that disk3 and disk5 truly are identical. Just be VERY sure you do all operations with the disk shares and not the user shares. Mixing and matching disk and user shares in copy and move operations will lose data.

[loady]

I see... but that means I need to leave a pc on performing the task...I don't have the option of doing so.

[jphipps]

If you run rsync with the "-n" option, it will do a dry run and tell you what it will do.  I use that to verify 2 disks are the same, ex:

 

rsync -avn /mnt/diskX /mnt/diskY

 

Using the "-a" will include checking  permissions, so if you have any files with different permissions, they will show up as changed.

[loady]

So that could be 100's of thousands of lines of files to trawl through if indeed I could scroll back and see..which I can't.

[jphipps]

If the disks are the same, there will only be a few lines of output.  If they vary because of timestamp or permissions, you can change the parms to not include them.

 

The "-a" option is really "-rlptgoD"  The ptgo relate to the permissions, timestamp, group, owner.  You can also use a "-c" which compares by checksum.  It is slower,  check the actual file contents for a comparison.

[loady]

thanks..lots of advice here but i am struggling to tailor it to my needs..is there like a wiki page on rsync ??

[trurl]

thanks..lots of advice here but i am struggling to tailor it to my needs..is there like a wiki page on rsync ??

rsync is a Linux command not specific to unRAID. Lots of internet pages about rsync.