I took a HDD from my main computer and put it in my unraid box so it would copy faster than over the network. I then researched and found rsync should be perfect for what I want to do, which is copy over every file (excluding a few I'll note later).

I ssh into unraid and use this command

rsync -azPhv --stats /mnt/disks/Tetra/ /mnt/user/backups/Tetra --exclude '.Trashes' --exclude '.Spotlight-V100' --exclude '.fseventsd' --exclude '.DS_Store'

I left it overnight and it seemed to finish fine, but messed up by not copying the text that it said after finishing. Anyway, I want to verify to double check everything got copied over so I ssh in and run this

rsync -azPhvn --stats /mnt/disks/Tetra/ /mnt/user/backups/Tetra --exclude '.Trashes' --exclude '.Spotlight-V100' --exclude '.fseventsd' --exclude '.DS_Store'

(same thing but with the -n flag to make it be a dry run)

Attached is a pic of what happens when I run that. I do not understand how any files could have vanished and I am new to using rsync. If y'all could help me figure this out I would really appreciate it.

 

 

Can't say much with everything being redacted, are those user files or some kind of random system/temp files? What is on this disk? What filesystem is it using?

Edited July 28, 20223 yr by Kilrah

Thanks for your reply. There is a large variety of files on it, it is about 3.9TB consisting of all kinds of text files, pdfs, images, audio files, video files. The filesystem is HFS+.

I redacted the filenames because they are sensitive information and I thought it was unimportant to know what the specific files were.

The redacted files that rsync says have vanished consist of various jpg, png, mp4, json files, with one exception being a system font collection file inside an old time machine backup.

There is no way I can think of that any of these files could have been deleted from the src between the original rsync copy and the dry-run check. (I believe that is usually the reason rsync complains about files having vanished.)

Edited July 28, 20223 yr by ra_bliss
added word

It seems some of those files might have very long names? Could be an issue, as well as the character set maybe? rsync sees the files when listing but then fails to access them...

 

Also one thing that isn't blurred seems to be from a Time Machine backup, I might be mistaken but I believe those use some of the features of Apple filesystems so are basically broken when copied to another non-Apple disk...

Yes I am seeing some warnings about filenames being too long

root@Tower:~# rsync -azPhvn --stats /mnt/disks/Tetra/ /mnt/user/backups/Tetra --exclude '.Trashes' --exclude '.Spotlight-V100' --exclude '.fseventsd' --exclude '.DS_Store'
sending incremental file list
file has vanished: "/mnt/disks/Tetra/Back Ups/copy tfrom Terabyte time machine backup folder 2017-4-13/Backups.backupdb/Hackintosh/2017-04-13-120916/Mountain Lion/System/Library/User Template/th.lproj/Library/FontCollections/กำหนดความกว้าง.collection"
rsync: [generator] recv_generator: failed to stat "/mnt/user/backups/Tetra/Excel Library.library/images/L0D1E3GWRTEL8.info/35_\#342\#200\------REDACTED 269 CHARS------thumbnail.png": File name too long (36)
rsync: [generator] recv_generator: failed to stat "/mnt/user/backups/Tetra/Excel Library.library/images/L0D1E3M1DG6R9.info/------REDACTED 255 CHARS------thumbnail.png": File name too long (36)

 

This excerpt also includes the file from the time machine you referenced. I guess I just don't understand it would fail to copy it over, but that seems like a possibility that it has something to do with the file systems. (from HFS+ to xfs-encrypted).

 

Another thing that I noticed is that after the two "rsync: [generator] recv_generator: failed to stat ..." line, all of the file has vanished lines show the path/filename and all of the filenames contain the "?" char. Could that char being used in a filename be tripping rsync up?

 

Is there any other ways you or anyone could recommend to make this process of duplicating one internal disk (hfs+) to another (xfs-encrypted)? Is there maybe an app or docker that could simplify the process without me needing to dive into the details of how rsync works? Apps like duplicati, duplicity, syncthing, do you think they would be able to accomplish the same thing if I just "rm -rf" the current xfs copy and redo it with one of those apps?

 

By the way, thanks so much for your help :-)

Did you try opening these files on the source disk to check if they're actually accessible/readable yet? Or tried to change the paths/names of some of them to see if they then disappear from the "vanished" list?

So I ls one folder and compared it to the corresponding folder in the rsync error output. It seems that when i'm in the source drive and go to these vanished files, ls is saying it cannot access these files and that there is no such directory. But if it can't access them and there are no such files, why is it printing the "/bin/ls: cannot access" lines?

 

I have checked every line and these two lists list the same files.

 1 /bin/ls: cannot access '1087_The 4th dimension ??_HenWad_ffo0xw.mp4': No suc:h file or directory
  2 /bin/ls: cannot access '1097_Some more of my doodles in my first doodlebook but fast paced in a train ??_elfinnfinite_dsxjov.mp4': No such file or directory
  3 /bin/ls: cannot access '1137_Hammock, blanket, and some mushrooms. It’s a gorgeous day to die ??_ItsAGorgeouDayToDie_edr1os.jpg': No such file or directory
  4 /bin/ls: cannot access '1147_??_Sacra_profanis_dppzwu.png': No such file or directory
  5 /bin/ls: cannot access '1165_Leave a like or a comment! It helps me more than you know, love you all, stay strong ❤??_elfinnfinite_fqh3ac.jpg': No such file or directory
  6 /bin/ls: cannot access '1168_currently tripping and I just saw this, good vibes everyone ??_the-heba_de9onw.jpg': No such file or directory
  7 /bin/ls: cannot access '1232_Tripping with my boyfriend in our new house ??_rbigler22_d4dlj3.jpg': No such file or directory
  8 /bin/ls: cannot access '1927_Found this on Facebook. It just spoke volumes ??_Trip_And_tell_hmu22m.jpg': No such file or directory
  9 /bin/ls: cannot access '2096_Trip spot for the evening. 1.4grams cubensis and Handel’s Messiah with the San Francisco Symphony Orchestra. Feeling glorious reddit ❤️❤️???? ?? ?? ☮️_california_king_eau66k.jpg': No     such file or directory
 10 /bin/ls: cannot access '800_my grinder under my rainbow lights ??_DELETED_ejmqit.mp4': No such file or directory
 11 /bin/ls: cannot access '814_Another one of these vids ?? And yes I know about the runs, it was from the way I twaxed it ??_chiliwilly001_izi1mw.mp4': No such file or directory
 12 /bin/ls: cannot access '832_?? the cosmic sensation._DELETED_i9nxe7.jpg': No such file or directory
 13 /bin/ls: cannot access '860_?? @luke_schroeder_art_BennyHG_jabfxj_1.jpg': No such file or directory
 14 /bin/ls: cannot access '860_?? @luke_schroeder_art_BennyHG_jabfxj_2.jpg': No such file or directory
 15 /bin/ls: cannot access '860_?? @luke_schroeder_art_BennyHG_jabfxj_3.jpg': No such file or directory
 16 /bin/ls: cannot access '860_?? @luke_schroeder_art_BennyHG_jabfxj_4.jpg': No such file or directory
 17 /bin/ls: cannot access '860_?? @luke_schroeder_art_BennyHG_jabfxj_5.jpg': No such file or directory
 18 /bin/ls: cannot access '860_?? @luke_schroeder_art_BennyHG_jabfxj_6.jpg': No such file or directory
 19 /bin/ls: cannot access '860_?? @luke_schroeder_art_BennyHG_jabfxj_7.jpg': No such file or directory
 20 /bin/ls: cannot access '876_??_Nerihd522_m0ihn7.jpg': No such file or directory
 21 /bin/ls: cannot access '928_Came up with an impromptu artwork during last night’s acid comedown. Thought I’d leave this here for those currently on a trip to enjoy. I’d appreciate any kind of feedback! Good vi    bes and happy tripping! ??_josephleh92_fz2ax5.jpg': No such file or directory
 22 /bin/ls: cannot access '957_Found this and had to share. Enjoy ??_jo-mc00_l3c3lh.mp4': No such file or directory
 23 /bin/ls: cannot access '979_Latest doodle, hope you guys like it! Leave a comment or a like it helps me more than you know :) ❤️????_elfinnfinite_ey6j7h.jpg': No such file or directory
 24 -------------------------------- below is rsync output selection of a folder, above is 23 files that I get when I ls on the source disk dir .
 25 file has vanished: "/mnt/disks/Tetra/Trippy Pix Reddit Archive/Currentlytripping/1087_The 4th dimension ??_HenWad_ffo0xw.mp4"
 26 file has vanished: "/mnt/disks/Tetra/Trippy Pix Reddit Archive/Currentlytripping/1097_Some more of my doodles in my first doodlebook but fast paced in a train ??_elfinnfinite_dsxjov.mp4"
 27 file has vanished: "/mnt/disks/Tetra/Trippy Pix Reddit Archive/Currentlytripping/1137_Hammock, blanket, and some mushrooms. It’s a gorgeous day to die ??_ItsAGorgeouDayToDie_edr1os.jpg"
 28 file has vanished: "/mnt/disks/Tetra/Trippy Pix Reddit Archive/Currentlytripping/1147_??_Sacra_profanis_dppzwu.png"
 29 file has vanished: "/mnt/disks/Tetra/Trippy Pix Reddit Archive/Currentlytripping/1165_Leave a like or a comment! It helps me more than you know, love you all, stay strong ❤??_elfinnfinite_fqh3ac.jpg"
 30 file has vanished: "/mnt/disks/Tetra/Trippy Pix Reddit Archive/Currentlytripping/1168_currently tripping and I just saw this, good vibes everyone ??_the-heba_de9onw.jpg"
 31 file has vanished: "/mnt/disks/Tetra/Trippy Pix Reddit Archive/Currentlytripping/1232_Tripping with my boyfriend in our new house ??_rbigler22_d4dlj3.jpg"
 32 file has vanished: "/mnt/disks/Tetra/Trippy Pix Reddit Archive/Currentlytripping/1927_Found this on Facebook. It just spoke volumes ??_Trip_And_tell_hmu22m.jpg"
 33 file has vanished: "/mnt/disks/Tetra/Trippy Pix Reddit Archive/Currentlytripping/2096_Trip spot for the evening. 1.4grams cubensis and Handel’s Messiah with the San Francisco Symphony Orchestra. Feeling glorio    us reddit ❤️❤️???? ?? ?? ☮️_california_king_eau66k.jpg"
 34 file has vanished: "/mnt/disks/Tetra/Trippy Pix Reddit Archive/Currentlytripping/800_my grinder under my rainbow lights ??_DELETED_ejmqit.mp4"
 35 file has vanished: "/mnt/disks/Tetra/Trippy Pix Reddit Archive/Currentlytripping/814_Another one of these vids ?? And yes I know about the runs, it was from the way I twaxed it ??_chiliwilly001_izi1mw.mp4"
 36 file has vanished: "/mnt/disks/Tetra/Trippy Pix Reddit Archive/Currentlytripping/832_?? the cosmic sensation._DELETED_i9nxe7.jpg"
 37 file has vanished: "/mnt/disks/Tetra/Trippy Pix Reddit Archive/Currentlytripping/860_?? @luke_schroeder_art_BennyHG_jabfxj_1.jpg"
 38 file has vanished: "/mnt/disks/Tetra/Trippy Pix Reddit Archive/Currentlytripping/860_?? @luke_schroeder_art_BennyHG_jabfxj_2.jpg"
 39 file has vanished: "/mnt/disks/Tetra/Trippy Pix Reddit Archive/Currentlytripping/860_?? @luke_schroeder_art_BennyHG_jabfxj_3.jpg"
 40 file has vanished: "/mnt/disks/Tetra/Trippy Pix Reddit Archive/Currentlytripping/860_?? @luke_schroeder_art_BennyHG_jabfxj_4.jpg"
 41 file has vanished: "/mnt/disks/Tetra/Trippy Pix Reddit Archive/Currentlytripping/860_?? @luke_schroeder_art_BennyHG_jabfxj_5.jpg"
 42 file has vanished: "/mnt/disks/Tetra/Trippy Pix Reddit Archive/Currentlytripping/860_?? @luke_schroeder_art_BennyHG_jabfxj_6.jpg"
 43 file has vanished: "/mnt/disks/Tetra/Trippy Pix Reddit Archive/Currentlytripping/860_?? @luke_schroeder_art_BennyHG_jabfxj_7.jpg"
 44 file has vanished: "/mnt/disks/Tetra/Trippy Pix Reddit Archive/Currentlytripping/876_??_Nerihd522_m0ihn7.jpg"
 45 file has vanished: "/mnt/disks/Tetra/Trippy Pix Reddit Archive/Currentlytripping/928_Came up with an impromptu artwork during last night’s acid comedown. Thought I’d leave this here for those currently on a tr    ip to enjoy. I’d appreciate any kind of feedback! Good vibes and happy tripping! ??_josephleh92_fz2ax5.jpg"
 46 file has vanished: "/mnt/disks/Tetra/Trippy Pix Reddit Archive/Currentlytripping/957_Found this and had to share. Enjoy ??_jo-mc00_l3c3lh.mp4"
 47 file has vanished: "/mnt/disks/Tetra/Trippy Pix Reddit Archive/Currentlytripping/979_Latest doodle, hope you guys like it! Leave a comment or a like it helps me more than you know :) ❤️????_elfinnfinite_ey6j7h.    jpg"

 

Also what do you think about whether the "?" char in each of the filenames matters? Maybe I could batch rename the files on the source disk and remove all the question marks and see what that does. What do you think?

 

Thanks

6 hours ago, ra_bliss said:

But if it can't access them and there are no such files, why is it printing the "/bin/ls: cannot access" lines?

 

On 7/29/2022 at 12:12 AM, Kilrah said:

rsync sees the files when listing but then fails to access them...

 

I would try changing the name of say one of the files and see what can get it working.