Rsync Data Transfer and Harddrive fill up

[blackbox]

Hey Everyone,

 

I currently have the latest version of Unraid running on a used Supermicro server I recent purchased from one of the tech resellers. Everything is good and has been easy going for my first time using Unraid or even having a server. I have 4 14TB drives running in the array with 1 as parity.

 

I have reached the point where my first HD is about full. Up to this point I have been transferring my files using rsync over my home network directly

sudo rsync -vr --ignore-existing -e ssh /media/linux/largestorage/media/ [email protected]:/mnt/disk1/media/ --progress

 

I have the settings for the media share set to use disk1 and disk2. I also have the disk set to an 80G buffer. (largest file I would transfer)

 

My question is what happens when my first disk gets full? will the system auto switch to disk2? If so does it create a media folder on that disk? or will something else happen?

 

Thanks for any advice.

 

[trurl]

Possibly rsync has created all the empty folders on one disk before filling those folders with files. You may need to move some of the empty folders to another disk manually so they can be filled. 

[JorgeB]

If split level is 0 it will move to the next disk once it hits the minimum share space set, if not it's like trurl posted, since rsync creates all the folders at the beginning of the transfer.

[blackbox]

3 hours ago, JorgeB said:

If split level is 0 it will move to the next disk once it hits the minimum share space set, if not it's like trurl posted, since rsync creates all the folders at the beginning of the transfer.

I have it set to split any directory as required

 

3 hours ago, trurl said:

Possibly rsync has created all the empty folders on one disk before filling those folders with files. You may need to move some of the empty folders to another disk manually so they can be filled. 

I just checked disk2 and its blank.  SO I should just make another media folder with movies images etc same file structure as disk one?

 

Also, I have two ssd's setup as Cache on this server that I never use. Would I be better off enabling Cache for this media folder and sending the files to the cache drive and allowing Mover *I think thats the name of the program* to transfer them to the array?

 

 

[JorgeB]

7 minutes ago, blackbox said:

I just checked disk2 and its blank.  SO I should just make another media folder with movies images etc same file structure as disk one?

No need if you're splitting any directory as required, it will move to the next disk when it hits the share's minimum free space.

[blackbox]

Is Split level 0 Considered "Manual: do not auto split directories" ?

[trurl]

1 hour ago, blackbox said:

Would I be better off enabling Cache for this media folder and sending the files to the cache drive and allowing Mover *I think thats the name of the program* to transfer them to the array?

If you intend to transfer more at one time than cache can hold then don't cache. Mover is intended for idle time, default is daily in the middle of the night.

 

It is impossible to move from cache to array as fast as you can write to cache.

 

58 minutes ago, blackbox said:

Is Split level 0 Considered "Manual: do not auto split directories" ?

Split Any is the setting you want. But as I already mentioned, probably rsync has created all folders in advance on that first disk, since empty folders don't take enough space to cause Unraid to choose another disk. rsync will continue by putting the files in the already created folders on that same disk unless you move the folders to another disk.

 

I had this problem when first loading my backup server. I noticed what was happening, and that it was filling the folders in alphabetical order, so I just started at the end of the alphabet and started moving empty folders to other disks so rsync would fill them there.

 

rsync, krusader, lots of other applications do things this way because they don't really know about other disks, they just see each user share as a single mount point and start by creating the folders before filling them with files.

[blackbox]

8 hours ago, trurl said:

If you intend to transfer more at one time than cache can hold then don't cache. Mover is intended for idle time, default is daily in the middle of the night.

 

It is impossible to move from cache to array as fast as you can write to cache.

 

Split Any is the setting you want. But as I already mentioned, probably rsync has created all folders in advance on that first disk, since empty folders don't take enough space to cause Unraid to choose another disk. rsync will continue by putting the files in the already created folders on that same disk unless you move the folders to another disk.

 

I had this problem when first loading my backup server. I noticed what was happening, and that it was filling the folders in alphabetical order, so I just started at the end of the alphabet and started moving empty folders to other disks so rsync would fill them there.

 

rsync, krusader, lots of other applications do things this way because they don't really know about other disks, they just see each user share as a single mount point and start by creating the folders before filling them with files.

If I turned on cache would mover treat these files differently and know to start using a different disk? Would I still need to create a file tree (on cache and or disk2) that matches the current disk or does mover have a mapping function?

I am asking just to know what would happen for a better understanding

[blackbox]

10 hours ago, JorgeB said:

No need if you're splitting any directory as required, it will move to the next disk when it hits the share's minimum free space.

So when using rsync I use the --ignore-existing. I would always want to transfer to disk1 even when full for rsync to see all the current files so I don't get duplicates? Will be interesting to use that when you have multiple filled drives.

Since I have splitting any directory as required it should just continue on the new disk where it left off.

[trurl]

11 hours ago, blackbox said:

If I turned on cache would mover treat these files differently and know to start using a different disk?

When writing new files to a user share, it will choose another disk if the settings for that user share allow it to choose another disk (Include/Exclude/Split Level), and if Minimum Free and/or Allocation Method says it should choose another disk.

 

Mover uses those same rules when moving cache-yes shares from cache to array.

 

On 5/25/2021 at 10:21 AM, blackbox said:

rsync -vr --ignore-existing -e ssh /media/linux/largestorage/media/ [email protected]:/mnt/disk1/media/ --progress

Sorry, just noticed you have rsync going directly to disk1 instead of to your media user share. So, it can only write to disk1. Nothing about user share settings apply in this case, including Minimum Free or whether to use cache. All files will go to disk1 because that is what you told it to do.

[blackbox]

12 hours ago, trurl said:

Sorry, just noticed you have rsync going directly to disk1 instead of to your media user share. So, it can only write to disk1. Nothing about user share settings apply in this case, including Minimum Free or whether to use cache. All files will go to disk1 because that is what you told it to do.

ooooohhhhhhh!!!!!!! Now its all staring to make sense for me. 

What would that look like if I was directing rsync to a share?  [email protected]:/mnt/user0/media/ --progress.   I was also looking at the cache drive but I only have appdata domains system and temp. I would assume I use the temp folder?

 

By the way. I really appreciate all the help everyone. Thank you.

[JorgeB]

On 5/25/2021 at 3:21 PM, blackbox said:

Up to this point I have been transferring my files using rsync over my home network directly

sudo rsync -vr --ignore-existing -e ssh /media/linux/largestorage/media/ [email protected]:/mnt/disk1/media/ --progress

 

I have the settings for the media share set to use disk1 and disk2. I also have the disk set to an 80G buffer. (largest file I would transfer)

 

 

I saw that, but assumed that that's what you used before, not now, anyway to transfer to a share you can use:

 

rsync --options /media/linux/largestorage/media/ [email protected]:/mnt/user/media/

 

 

[trurl]

8 hours ago, blackbox said:

What would that look like if I was directing rsync to a share?  [email protected]:/mnt/user0/media/ --progress.   I was also looking at the cache drive but I only have appdata domains system and temp. I would assume I use the temp folder?

/mnt/user0 is the user shares excluding any files on pools (cache). Better to use /mnt/user instead unless you are intentionally excluding pools, which wouldn't normally be necessary since the Use cache settings for the user share will handle files on pools as configured.

 

And temp is a separate user share named temp, either you created it in the webUI or you created it by specifying a path at the top level of a pool (cache) or array disk.

 

The user shares are just the aggregated top level folders on pool and array disks. Each user share has top level folders on pool or array disks named for the share. Unraid creates these top level folders as needed according to the settings for the share.

 

Conversely, if you create a top level folder on a pool or array disk by specifying a path for a top level folder, that top level folder is automatically part of a user share named for that folder. Any user shares created in this way will have default settings until you set them.

[blackbox]

On 5/27/2021 at 9:51 AM, trurl said:

/

 

On 5/27/2021 at 3:05 AM, JorgeB said:

/

Thank you for all the good info guys. Last night I had finally reached 100% drive capacity on my first drive. I followed the information you guys gave me. I have been using the 'user' and taking advantage of the cache and mover (let it do all the work when everyone's asleep). The drive filled up and mover switched to disk 2, made the folders and started transferring everything over like designed. I was nervous because things just never work like they are supposed to for me and you guys rock. Thanks