Split level top two: What happens if Disk is full?

[mgutt]

I have multiple movies and they are located in dirs with their starting letter. Example:

Movies/A/Alien/Alien.mkv

 

I like to have all "A" movies on the same Disk so I should choose the split level "Automatically split only the top directory level".

 

Lets say all movies with "A", "B" and "C" fit on Disk 1. What happens if I add more "A" movies and they do not fit on Disk 1 anymore? Does it move all existing "A" movies to a different Disk that has enough space left or does it ignore the split level rule in this situation and create an additional "A" dir on a different Disk?

 

 

[trurl]

Unraid will not move existing files except in accordance with the Use cache setting (between cache and array).

 

And Split Level has precedence over Allocation Method or Minimum Free.

 

Also, Unraid has no way of knowing how large a file will become when it chooses a disk to write it to.

 

So, it will choose the disk that Split Level says it should choose. 

 

Long story short, Split Level CAN make you run out of space on a disk. 

[trurl]

On the other hand, the only consequence of not keeping files together is a possible short delay while another disk spins up.

 

So I just let it split them. 

[mgutt]

This means if I add a new A movie and the Disk becomes full I receive a full disk error. Ok.

 

What are my options in such a situation, am I able to move the B and C movies manually to an other Disk?

[itimpi]

4 hours ago, mgutt said:

This means if I add a new A movie and the Disk becomes full I receive a full disk error. Ok.

 

What are my options in such a situation, am I able to move the B and C movies manually to an other Disk?

One option would be to manually create the A folder on another drive, and Unraid would then apply the allocation method in deciding which disk with an A folder to use so in this case would start using the second one as the first one is full.   I personally use a similar naming scheme but use a Split Level of 0 so I always need to manually create the letter level to control what disks each letter is allowed to use, but as a result I always know exactly which disks any particular letter can occur on.
 

You can also manually move the B, C folders to another drive.     There a wide variety of tools for doing this depending on what your comfort level is.    When moving files make sure you move disk to disk and NOT disk to User share as the latter can potentially lead to data loss.

[mgutt]

7 hours ago, itimpi said:

There a wide variety of tools for doing this depending on what your comfort level is.

Any suggestions? I found:

1. Filebrowser
2. Krusader
3. unBALANCE

unBALANCE looks promising, but I'm not sure if this would fit my target.

[itimpi]

1 hour ago, mgutt said:

Any suggestions? I found:

1. Filebrowser
2. Krusader
3. unBALANCE

unBALANCE looks promising, but I'm not sure if this would fit my target.

You are happy with the CLI then you can use the mc (Midnight Commander) command that is built into Unraid.   You san also do it over the network from a PC/Mac.

 

[mgutt]

11 hours ago, itimpi said:

You san also do it over the network from a PC/Mac.

 

What do you mean by that? Starting the CLI or something else?

[itimpi]

6 hours ago, mgutt said:

What do you mean by that? Starting the CLI or something else?

If you have disk shares enabled (under Settings >> Global Share Settings) then you can see each drive across the network just like any other share.    You can then use whatever tool you like on the PC/Mac to move the files (e.g. Windows Explorer/Finder).   This tends to be slower than doing it locally on the Unraid server but many people are more comfortable doing it that way.

 

One important rule to follow however you decide to do it is to do the copy/move diskX to diskY, and not diskX to User Share.    Mixing disk shares and User shares in the same copy/move command can lead to data loss.

 

[mgutt]

2 hours ago, itimpi said:

If you have disk shares enabled (under Settings >> Global Share Settings) then you can see each drive across the network just like any other share.

 

Ah ok. Nice to know. For creating the letter-dirs this is absolutely sufficient. Thanks.

[trurl]

When finished, I recommend disabling disk shares, or set it to Auto. That way nobody will accidentally

6 hours ago, itimpi said:

 Mixing disk shares and User shares in the same copy/move command can lead to data loss.

 

[mgutt]

Finally I created the folders manually by CLI.

 

Steps I did (maybe helpful for someone else)

 

1.) I set the share to "Manual: do not automatically split directories" (correct?)

 

2.) Then opened the terminal through Unraid Webclient:

 

3.) Created the Share Folder "Movies" and the subdirectory "KL":

 

4.) Opened the Unraid Main Tab and viewed the content of Disk3. Everything looked good:

 

 

5.) Installed the CA user scripts plugin. Added a new script with the following content (direct execution by terminal would be possible, too) to fetch the files from the source NAS through FTP (this script is helpful to backup websites as well):

wget --background --quiet --recursive --level 0 --no-host-directories --cut-dirs=2 --no-verbose --timestamping --backups=0 --user=<ftp_username> --password=<ftp_password> "ftp://10.1.2.3/video/Filme/GH/" --directory-prefix=/mnt/user/Movies

6.) Added this script multiple times for multiple copy processes on different disks and different folder names (in my case the letters). Then I clicked on "Run in Background" and now I need to wait...

 

In the main tab the "Writes" column rises on the corresponding disks, so everything looks good:

 

7.) Finally I opened the terminal again and typed in "top" to check if there are still active wget process:

 

So it works now, but I hoped it would copy faster. The maximum is 970 Mbps altough 10G network is used and a parity disk is not set up:

10G source:

 

 The load looks good as well:

Source:

 

EDIT: Started a fourth wget process (now copying to four disks at the same time) and now I reach 1.2 Gbps:

 

This means around 30 MB/s per disk. Much lower than the disks maximum write speed?!

Edited April 21, 2020 by mgutt

[JorgeB]

1 hour ago, mgutt said:

This means around 30 MB/s per disk. Much lower than the disks maximum write speed?!

Not a good idea to write to multiple array disks at the same time since parity needs to updated for all and it will bottleneck.

[mgutt]

46 minutes ago, johnnie.black said:

Not a good idea to write to multiple array disks at the same time since parity needs to updated for all and it will bottleneck.

As it's offtopic here I created a new topic: