Using Rsync to copy files to Unraid share - rsync stops - no space left on device

[agonza]

Need some help.

Equipment: Unraid 6.10.3 with media share config: Fill Up, No cache, minimum free 35GB, split top level only, include disks 2, 3, 4, 6 - all disks are 9.1TB

Doing a rsync copy command I got a 71% warning, then several 9x% warnings (which are fine those are default settings). Rysnc twice has terminated No space left on device (28) with a few other lines of text which i assume are part of the same error message.  nated because of not enough space. disks 3, 4, and 6 are empty. So what could be going on that these files do not continue copying to the next disk in the share?

Rsync error message- first few words are part of the actual file it stopped on.

996 1080p Bluray Remux DTS-HD AVC 5 1-decatora27.mkv": No space left on device (28)
rsync error: error in file IO (code 11) at receiver.c(378) [receiver=3.2.3]
rsync: [sender] write error: Broken pipe (32)

 

I am copying from a SMB share;  //diskstation1/plex_media  to the share.

Command is   rsync -avh /mnt/remotes/DISKSTATION1_Plex_Media/Movies/ /mnt/disk2/movies/720p_1080p

 

I THINK I JUST ANSWERED MY OWN QUESTION.  I will test this theory.  Above i choose disk2 instead of user!!!

I was following spaceinvaderone tutorial and I don't recall now if he used the disk or the user for the destination.

 

Nope that wasn't it.  It crashed just the same when using /mnt/user/movies/720p_1080p as the destination.

 

This is the share config
 

 

 

 

 

 

Edited September 2, 2022 by agonza
add details

[itimpi]

I suspect your problem could be your Split Level setting.  If there is any. contention over which drive to pick for a file then Split Level always wins.   That can mean Unraid will try to copy to a disk even though it is below the Minimum Free Space level.    Also, if a file already exists it is always updated in-situ on the same drive so any remnant from s failed earlier attempt might also cause problems.

 

You can also get the same symptoms if any file is larger than the Minimum Free Space setting as Unraid does not know how large a file will become at the point at which the drive for the file is selected.   It can therefore select a drive and later find it does not fit which causes it to error out.

 

Giving the full path to the file that failed and details on its size might help with deciding if this is likely to be your issue.

[JorgeB]

Note that rsync creates all the folders before the transfer, so they will all be in the first available disk, and depending on the split level all data can go there.

[agonza]

Looking at the other disks.  The folders have not been created.  Should they be created immediately or only if the drive fills up and then it would begin writing to the next one?

 

The path to the files 

I mounted a network share to my Synology.

Used Spaceinvaderone tutorial on file transferring in unraid using krusader, rync, etc.

/mnt/remotes/DISKSTATION1_Plex_Media/Movies

 

This unraid disk share is 

/mnt/user/movies/720p_1080p

 

The command used

rysnc -avh /mnt/remotes/DISKSTATION1_Plex_Media/Movies /mnt/user/movies/720p_1080p

Edited September 4, 2022 by agonza
add details

[itimpi]

You have only included part of the path information.  I was looking for the full path (including filename) of a file that gave an error.

 

As was mentioned rsync creates all its folders at the start.   With your Split level settings only the ‘Movies/720p_1080p’ folder can exist on multiple drives.   Any folders below that will have their content constrained to the drive where that folder is created (I.e. the first drive).

[Kilrah]

5 hours ago, agonza said:

Looking at the other disks.  The folders have not been created.  Should they be created immediately or only if the drive fills up and then it would begin writing to the next one?

 

When a folder exists on no disk and an app creates it, it goes to the disk "in order" if it is not below min free space, otherwise it goes to the next etc.

Once the folder exists and something is written to it, as long as there's enough space it goes to where the folder already exists, if not the folder gets created on the next drive as long as the split settings allow for it.

 

Your problem is that rsync creates all empty directories at the start, since at that moment the first drive has enough space they all get made there and files then get put there. Now the disk gets full but your split settings disallow unraid from creating new folders on other disks at that "level" of the folder tree so things continue to get put on the first disk and it fills up.

So rethink your reason for your choice of split level and either change it or manually move stuff to satisfy it.  

Edited September 4, 2022 by Kilrah

[agonza]

Hello everyone.  Sorry I went away.  You guys were providing me great insight but I went on a few weeks vacation on the 5th and now getting back to this.

Thank you for your patience.  This is all new to me.

 

On 9/4/2022 at 2:59 AM, Kilrah said:

 

When a folder exists on no disk and an app creates it, it goes to the disk "in order" if it is not below min free space, otherwise it goes to the next etc.

Once the folder exists and something is written to it, as long as there's enough space it goes to where the folder already exists, if not the folder gets created on the next drive as long as the split settings allow for it.

 

Your problem is that rsync creates all empty directories at the start, since at that moment the first drive has enough space they all get made there and files then get put there. Now the disk gets full but your split settings disallow unraid from creating new folders on other disks at that "level" of the folder tree so things continue to get put on the first disk and it fills up.

So rethink your reason for your choice of split level and either change it or manually move stuff to satisfy it.  

 

This split level is confusing especially since the guide talks old terminology split level 0, 1, 2,3, etc but the newer versions using different phrasing that is throwing me off.   Unraid should update its documentation and explain this newer wording comparing it to the older.  

I would appreciate it if someone can clear this up for me. 

User level is the top level and Unraid says it is Level 1;  therefore my movies share '/mnt/user/movies' is the top level.  If I have folders under movies share for 720p, kids_movies, 4k movies.  These are at level 2 according to the manual.

So what setting am i using with this newer terminology? 'Automatically split only the top xxx directory level'.  This is part of the confusion.

Apparently this 'top xxx directory level' isn't equal to the older split level 1, 2, 3, etc.

The guide says to use a split level = 2. To allow these folders to copy to each disk and allow each single movie title folder to stay on the a single disk.

 

I want to use FILL UP to keep disk usauge down.

 

[itimpi]

12 hours ago, agonza said:

So what setting am i using with this newer terminology? 'Automatically split only the top xxx directory level'.

The top level is the folder that is equivalent to the share name and is level 1.

[JorgeB]

See here for some examples:

https://forums.lime-technology.com/topic/59589-solved-v635-advice-please-on-shares-settings/?do=findComment&comment=584558

 

[itimpi]

@agonza  The section here in the online documentation accessible via the ‘Manual’ link at the bottom of the GUI or the DOCS link at the top of each forum page seems reasonably clear to me?   Is there a suggestion for improvement?

[agonza]

The verbage isn't the same so I get confused.  I can see how users find it confusing.  I followed this section in your link when setting up my shares and folders.  The first example the author did not recommend.  I followed the 2nd using the folder structure where Movies is the user share, and sub-folders for SD, HD, Kids, etc.  I also setup a separate TV user share.  I had not begun transferring anything there so I don't know if that one would also fail. 

 

My choices are; allocation: Fillup, Inc disks: 4, Min free: 40GB. The example said to use a split level of 2 for the Media user share.  So I selected Split level: "Automatically split only the top two directory levels". 

Is this the same as split level of 2? 

I assume this is what is meant but no where on the online documentation does it make a specific reference to this.   I followed this example and the transfers did not continue on to the next disk and filled up only 1 disk.

The 6.11.1 split options have 5 "automatic split N levels as required" choices and a manual "do not automatically split".  The online docs only talk about split levels 1, 2, 3, 4.  So to me there is a disconnect in the doc verbage vs what is actual in 6.11.1. 

[itimpi]

3 hours ago, agonza said:

Is this the same as split level of 2?

Yes

 

I have made this clearer in the manual by referencing the GUI text for the split levels in the example.

[agonza]

Great!   Thank you.  Perhaps this will reduce some user errors and help requests.  I shall follow the example and see how it goes. 

I noticed that rsync copies all folders at the beginning.  I used the below command to begin.   

                                 rsync -avh "/mnt/remotes/DISKSTATION1_Plex_Media/TV_Shows" "/mnt/user/TV_Shows" 

Pardon my unfamilarity with all this.  Is this rysnc transfer of files not fully automated?  When it reaches capacity on the first drive will rysnc and unraid continue on to the 2nd and 3rd?  As stated I have FILL-UP, 10GB space reserved, a split of 2, and 3 drives.  Looking at my 60% completed process running it will be awhile before i know myself.  

[JorgeB]

3 minutes ago, agonza said:

I noticed that rsync copies all folders at the beginning.

 

On 9/2/2022 at 8:20 AM, JorgeB said:

Note that rsync creates all the folders before the transfer, so they will all be in the first available disk, and depending on the split level all data can go there.

 

[agonza]

I still need some assistance and clarification.  Thank you for everything so far. 

When a user share is created via Shares - Will the share be created across all the disks that named in "included disks"?

When I created the share it only went into the first disk?

I am using allocation - Fill-up and a split level of 2. 

[Kilrah]

It will make folders on other drives as needed. With fill-up that won't happen until the first drive is full (reached the "min free space" setting).

[agonza]

thank you Kilrah for that information.  I kept expecting when I created the share w the multiple drives that it would create the share name/folder across each of the drives in the included disks.  So this doesn't happen except when the first disk in FILL UP mode is full.  I shall begin my transfer process now w the settings I have made changes to in the share