Executed a bunch of rsync's to reorganize data on disks, somehow lost all my transferred data?

[nick5429]

The unBalance plugin was not giving me sane behavior, so I decided I could just rsync manually on the commandline to accomplish the same thing: moving all the data in a ~7TB share onto a newly-installed, empty 8TB disk -- the new disk is disk6

 

I ran the following to generate the list of commands I intended to execute:

root@nickserver:/mnt/disk6# for i in /mnt/disk*
> do
> echo rsync -av --remove-source-files --info=progress2 -X "$i/DLs" "/mnt/disk6/"
> done
rsync -av --remove-source-files --info=progress2 -X /mnt/disk1/DLs /mnt/disk6/
rsync -av --remove-source-files --info=progress2 -X /mnt/disk11/DLs /mnt/disk6/
rsync -av --remove-source-files --info=progress2 -X /mnt/disk12/DLs /mnt/disk6/
rsync -av --remove-source-files --info=progress2 -X /mnt/disk13/DLs /mnt/disk6/
rsync -av --remove-source-files --info=progress2 -X /mnt/disk14/DLs /mnt/disk6/
rsync -av --remove-source-files --info=progress2 -X /mnt/disk15/DLs /mnt/disk6/
rsync -av --remove-source-files --info=progress2 -X /mnt/disk16/DLs /mnt/disk6/
rsync -av --remove-source-files --info=progress2 -X /mnt/disk17/DLs /mnt/disk6/
rsync -av --remove-source-files --info=progress2 -X /mnt/disk18/DLs /mnt/disk6/
rsync -av --remove-source-files --info=progress2 -X /mnt/disk2/DLs /mnt/disk6/
rsync -av --remove-source-files --info=progress2 -X /mnt/disk3/DLs /mnt/disk6/
rsync -av --remove-source-files --info=progress2 -X /mnt/disk4/DLs /mnt/disk6/
rsync -av --remove-source-files --info=progress2 -X /mnt/disk5/DLs /mnt/disk6/
rsync -av --remove-source-files --info=progress2 -X /mnt/disk6/DLs /mnt/disk6/     # unnecessary, but I assessed as harmless
rsync -av --remove-source-files --info=progress2 -X /mnt/disk9/DLs /mnt/disk6/
rsync -av --remove-source-files --info=progress2 -X /mnt/disks/DLs /mnt/disk6/     # unnecessary, but I assessed as harmless

 

Not all of these were fully relevant, not all had data on them -- but I briefly evaluated that 'it should be fine anyway'.  I copied and pasted all those rsync commands back into my terminal, and walked away for a couple days.  When I returned, disk6 had essentially zero data:

root@nickserver:/mnt/disk6# du -hs /mnt/disk6
292K    /mnt/disk6

 

My log/alerts showed various other disks decreasing below the warning thresholds as their data was offloaded. Disk6 showed filling up; filling up; filling up to 97% warning threshold -- then 5 minutes later, dropping down to 'normal utilization level' (which appears to effectively be "empty"):

10-05-2022 20:53	Unraid Disk 6 message	Notice [NICKSERVER] - Disk 6 returned to normal utilization level	H7280A520SUN8.0T_001649PAR4LV_VLKAR4LV_35000cca260bc9574 (sdd)	normal	
10-05-2022 20:47	Unraid Disk 6 disk utilization	Alert [NICKSERVER] - Disk 6 is low on space (97%)	H7280A520SUN8.0T_001649PAR4LV_VLKAR4LV_35000cca260bc9574 (sdd)	alert	
10-05-2022 20:32	Unraid Disk 6 disk utilization	Alert [NICKSERVER] - Disk 6 is low on space (96%)	H7280A520SUN8.0T_001649PAR4LV_VLKAR4LV_35000cca260bc9574 (sdd)	alert	
10-05-2022 19:29	Unraid Disk 6 disk utilization	Warning [NICKSERVER] - Disk 6 is high on usage (91%)	H7280A520SUN8.0T_001649PAR4LV_VLKAR4LV_35000cca260bc9574 (sdd)	warning	
10-05-2022 18:05	Unraid Disk 4 message	Notice [NICKSERVER] - Disk 4 returned to normal utilization level	TOSHIBA_DT01ACA300_X3G716ZKS (sdx)	normal	
10-05-2022 16:28	Unraid Disk 2 message	Notice [NICKSERVER] - Disk 2 returned to normal utilization level	TOSHIBA_MG03ACA300_54G1KI4TF (sds)	normal	
10-05-2022 13:38	Unraid Disk 17 message	Notice [NICKSERVER] - Disk 17 returned to normal utilization level	HUS726060AL5210_NAGZ0EHY_35000cca242369310 (sdk)	normal

 

Did I overlook something obvious/stupid in my rsync commands? Was that ` rsync -av --remove-source-files --info=progress2 -X /mnt/disk6/usenet /mnt/disk6/ ` command destructive after all? It shouldn't be, given the description of the --remove-source-files flag (which only deletes successfully transferred source data; and this command should have moved nothing)

 

 

In related news --  do we have a preferred xfs un-delete tool; ideally which will attempt to preserve filenames as much as possible?  I have another empty disk large enough to hold any recovered data....

Edited May 12, 2022 by nick5429

[JorgeB]

15 minutes ago, nick5429 said:

rsync -av --remove-source-files --info=progress2 -X /mnt/disk6/DLs /mnt/disk6/     # unnecessary, but I assessed as harmless

 

This one isn't harmless, without a backslash after DLs you're transferring from /mnt/disk6/DLs to /mnt/disk6/DLs, so same source and dest, it would be harmless without --remove-source-files, not with it since nothing will be transferred but everything deleted, assuming you did that in the order posted all data on that share from all disks except the one from disk9 would be deleted.

[nick5429]

8 minutes ago, JorgeB said:

without a backslash after DLs you're transferring from /mnt/disk6/DLs to /mnt/disk6/DLs

This was precisely the intent

 

rsync manpage shows:

Quote

--remove-source-files This tells rsync to remove from the sending side the files (meaning non-directories) that are a part of the transfer and have been successfully duplicated on the receiving side.

 

Nothing got transferred, so nothing should be deleted

 

 

I'd have sworn that in the past when I was *TRYING* to copy-verify-and-delete data which was duplicated on two different disks (e.g., at times when the mover flaked out and identical data existed on both the cache and the regular array), it completely ignored (and explicitly did not delete) source files which also existed at the destination. But the trial example I just constructed shows different behaviour:

root@nickserver:/mnt# ls /mnt/disk4/death
myfile1  myfile2  myfile3
root@nickserver:/mnt# ls /mnt/disk5/death
myfile1  myfile2  myfile3  myfile4
root@nickserver:/mnt# rsync -av --remove-source-files --info=progress2 -X /mnt/disk4/death^Cmnt/disk5/
root@nickserver:/mnt# rsync -av --remove-source-files --info=progress2 -X /mnt/disk5/death /mnt/disk4/
sending incremental file list
death/
death/myfile1
              0 100%    0.00kB/s    0:00:00 (xfr#1, to-chk=0/5)
death/myfile2
              0 100%    0.00kB/s    0:00:00 (xfr#2, to-chk=2/5)
death/myfile3
              0 100%    0.00kB/s    0:00:00 (xfr#3, to-chk=1/5)
death/myfile4
              0 100%    0.00kB/s    0:00:00 (xfr#4, to-chk=0/5)

sent 309 bytes  received 128 bytes  874.00 bytes/sec
total size is 0  speedup is 0.00
root@nickserver:/mnt# ls /mnt/disk4/death
myfile1  myfile2  myfile3  myfile4
root@nickserver:/mnt# ls /mnt/disk5/death
root@nickserver:/mnt#

 

 

[nick5429]

It appears the difference is the '-a' flag.  Without '-a', the command I thought was safe appears to be safe:

 

root@nickserver:/mnt/disk4# rsync -v --info=progress2  --remove-source-files /mnt/disk4/death /mnt/disk4/
skipping directory death
              0 100%    0.00kB/s    0:00:00 (xfr#0, to-chk=0/0)

sent 17 bytes  received 12 bytes  58.00 bytes/sec
total size is 0  speedup is 0.00
root@nickserver:/mnt/disk4# ls -lha /mnt/disk4/death
total 0
drwxrwxrwx 2 root   root   66 May 12 11:14 ./
drwxrwxrwx 9 nobody users 142 May 12 11:13 ../
-rw-rw-rw- 1 root   root    0 May 12 11:14 myfile1
-rw-rw-rw- 1 root   root    0 May 12 11:14 myfile2
-rw-rw-rw- 1 root   root    0 May 12 11:14 myfile3
-rw-rw-rw- 1 root   root    0 May 12 11:14 myfile4
root@nickserver:/mnt/disk4# rsync -av --info=progress2  --remove-source-files /mnt/disk4/death /mnt/disk4/
sending incremental file list
              0 100%    0.00kB/s    0:00:00 (xfr#0, to-chk=0/5)

sent 144 bytes  received 49 bytes  386.00 bytes/sec
total size is 0  speedup is 0.00
root@nickserver:/mnt/disk4# ls -lha /mnt/disk4/death
total 0
drwxrwxrwx 2 root   root    6 May 12 11:24 ./
drwxrwxrwx 9 nobody users 142 May 12 11:13 ../

 

[JorgeB]

8 minutes ago, nick5429 said:

Nothing got transferred, so nothing should be deleted

Technically yes, but nothing was transferred because the files already existed on destination, and in that case they are still deleted from source, same as if you do it using a different source and dest, e.g.:

 

rsync -av /source /dest

 

All files will be transferred, if you then run:

 

rsync -av --remove-source-files /source /dest

 

nothing will be transferred but all source files will be deleted.