trouble rsync-ing cache to external hdd

[weilii]

Hello,
 

First time poster here, and I would first just like to say thanks to this wonderful community, you have all gotten me, a complete neophyte who just wanted a NAS, to the point where I've set up an awesome box and learned a ton about servers, networking, and automation. Thanks to all you guy's and your wonderful resources, plugins, and docker containers. Anyways...
 

I am having an issue trying to back my cache drive up to an external hdd (ntfs). When  I run and rsync (-vah --delete) to sync my cache drive (/mnt/cache/) to my external hdd (mnt/disks/bkup1/cache/  mounted using unassigned devices plugin) the console reports having moved ~320gb of data to the external drive. du /mnt/cache/ has confirmed this to be the case, displaying 300 some GB as the size of the cache folder on the backup drive. The problem is, my cache drive is only 128gb, and only had about 80gb of data present on it. I am worried if I format my cache disk I will not be able to get all my data back onto, as it seems to have somehow expanded. This is an issue for me as I am attempting to change the file system of the cache drive.

Does anyone know why this is happening?
Will this folder shrink back down if I try and copy it back to the drive? (seems like no)
How can I reformat this drive and get all this data back onto it?

I am afraid to try anything, as it would require deleting the cache drive to attempt.

Any help would be appreciated.

[JonathanM]

1 hour ago, weilii said:

 

I am having an issue trying to back my cache drive up to an external hdd (ntfs).

Do you not have enough free space on your array? If you have less than 80GB free, I'd advise your first course of action would be to add a new array drive or upgrade a current drive.

 

If you do have enough space, there are wiki articles and other resources describing how to accomplish exactly what you are talking about using space on the array.

[weilii]

Thanks for the idea Jonathanm.
There is enough free space on the array, which would let me temporarily hold the cache files there to reformat the drive. However that still leaves me with the question of if my off site backups (on the external drive) will be usable should I need them. I would still like to figure out a way to get them onto an external in a form that could be easily restored with a single command without having to manually rebuild the cache from folders stored on an array drive. I thought R sync would do this, and seems to work fine when used on array drives, but the cache blows up like a balloon.

[JorgeB]

If you're have vdisks (which are sparse when created) and/or snapshots they will become much larger on an ntfs drive, original size can be maintained for snapshots if the destination drive uses btrfs, and xfs or btrfs for sparse files.

[weilii]

Quote

If you're have vdisks (which are sparse when created) and/or snapshots they will become much larger on an ntfs drive

I think this may be the issue! Am I correct in assuming that my docker image falls into this category? That is the only image that should be on the cache, I think.

[JorgeB]

I was referring more to a VM vdisk, but IIRC the docker image is a sparse file, and if it's larger than normal it could be.

[JorgeB]

Also no point in backing up the docker image, it should be recreated.

[weilii]

The docker image is only backed up as a byproduct of being stored in /mnt/cache/ which is being rsynced, primarily to get my appdata.

After doing a big of digging using the keyword "sparse file" (which i had never heard of before) it turns out that plex (which I am running) also uses these kinds of files. I found a similar issue on stack exchange where a comment said that rysnc needs to use the option --sparse to back up these files effectively. I am going to add this to my rsync script and re-run it and see if i get better results. Thanks johnnie.black for the lead on "sparse files" I feel like its gotten me further towards a solution.

Edited August 14, 2019 by weilii