how to deal with rsync creating all empty folders first and causing disks to run out of space?

[Optimbic]

I understand that rsync will first create empty folders and then start moving the files that live in those directories. Since empty folders don't take up space, this can cause an issue where the copy will eventually fail since the disk will run out of space.

 

Is there a way to get around this other than manually estimating the weight of files and copying in batches that will fit in the largest drive?

[JorgeB]

Set split level to all temporarily.

[Optimbic]

28 minutes ago, JorgeB said:

Set split level to all temporarily.

 

is "all" the "automatically split any directory as required"?

 

isn't that potentially going to make a mess of putting files everywhere?

and isn't rsync still going to put all the folders on the first drive since it doesn't know that the drive will get full once files start transferring?

 

sorry If I'm being dense, still new to this. thanks for the help

 

EDIT: Wait I think I got it, it will create the empty folders on the first drive but since they are not constrained by split level it will move them somewhere else once space starts filling up with the actual files? is that how it works?

Edited January 20, 2023 by Optimbic

[JorgeB]

59 minutes ago, Optimbic said:

is "all" the "automatically split any directory as required"?

Yes.

 

59 minutes ago, Optimbic said:

Wait I think I got it, it will create the empty folders on the first drive but since they are not constrained by split level it will move them somewhere else once space starts filling up with the actual files? is that how it works?

Correct, it will leave empty folders on disk1, that's not a big deal and if you want there are easy ways to get rid of them once the transfer is complete.

[Optimbic]

1 hour ago, JorgeB said:

Yes.

 

Correct, it will leave empty folders on disk1, that's not a big deal and if you want there are easy ways to get rid of them once the transfer is complete.

Cool, thank you very much! Is it actually rsync that will create new folders in another drive? or something in the unraid OS ? I mean to say, how does rsync know that "this file should go in this folder that I already created on disk 1, but disk 1 is full, and the split level rules allow me to put this folder on another disk, so I'll create the same empty folder on disk 2 and copy the file there" ? Just curious how it works internally

 

Also, in theory if I set up my shares and split level correctly so it can always move stuff to another drive, I could do it with out setting split level to "all", right? So I don't have to go hunting files spread out across drives.

 

For example make two shares  :

 

/media/movies/movie_name_date/   - split level 2

/media/series/series_name/season_number/ - split level 2 - allocation "most free"

 

is rsync agnostic of allocation rules? will it attempt to cram all series into a single disk and give up if it fills up? if it is then I guess I can set split level 3 and do some manual re-arrangement after the fact. Or copy the series into batches that I know will fit into a drive.

Edited January 20, 2023 by Optimbic

[itimpi]

1 minute ago, Optimbic said:

Cool, thank you very much! Is it actually rsync that will create new folders in another drive? or something in the unraid OS ?

It is the Unraid OS.   Rsync will not know which drive a file is on if you are writing to a User Share.

2 minutes ago, Optimbic said:

n theory if I set up my shares and split level correctly so it can always move stuff to another drive

Note that Unraid never moves files between array drives.   The Split level setting is applied to new files at the point the file is first created.

[Optimbic]

9 minutes ago, itimpi said:

It is the Unraid OS.   Rsync will not know which drive a file is on if you are writing to a User Share.

Note that Unraid never moves files between array drives.   The Split level setting is applied to new files at the point the file is first created.

 

Thanks, that clears it up!

 

I have to import 25TB from external hard drives, so they are all going to be new files, so I guess it should work with the share/splitlevel/allocation setup I described above?

 

ED: Ah I see, it's because I said "always move stuff to another drive", yeah that was just badly worded, it's for a first time import.

Edited January 20, 2023 by Optimbic

[apandey]

I solved this by writing a custom rsync script. It basically runs rsync in dry run mode to itemize the actions it would take, then ignore directory creation greedily and instead do tgst lazily when copying files. It also handles permissions as per unraid shares

 

You can see here if interested: https://github.com/ashishpandey/scaffolding/blob/master/roles/godaam/files/scripts/mirror.sh

 

This way, you can keep the split level that you want and things fill up properly

[Optimbic]

1 hour ago, apandey said:

I solved this by writing a custom rsync script. It basically runs rsync in dry run mode to itemize the actions it would take, then ignore directory creation greedily and instead do tgst lazily when copying files. It also handles permissions as per unraid shares

 

You can see here if interested: https://github.com/ashishpandey/scaffolding/blob/master/roles/godaam/files/scripts/mirror.sh

 

This way, you can keep the split level that you want and things fill up properly

 

Nice, thank you very much!

[tgdatamover]

On 1/20/2023 at 10:40 AM, apandey said:

I solved this by writing a custom rsync script. It basically runs rsync in dry run mode to itemize the actions it would take, then ignore directory creation greedily and instead do tgst lazily when copying files. It also handles permissions as per unraid shares

 

You can see here if interested: https://github.com/ashishpandey/scaffolding/blob/master/roles/godaam/files/scripts/mirror.sh

 

This way, you can keep the split level that you want and things fill up properly

This is also exactly what I needed thanks!