Best way to move one large share to a new share

[qal1h]

I started my unraid server by doing the following:

• Added 6tb x2 to the array
• Create a share called movies and primary storage as array and fill up method
• Transferred about 1000 movies to those drives - both drives almost full

 

Since then i've added a third drive, 12tb.

 

I want to move my movies share to a data share. Like described by TrashGuides.

 

Question is how is it best to move the movies share into data across these 3 drives?

 

Do I:

• Create a share called data (primary storage array, included disks: disk1, disk2, disk3) - doing this has created a data folder inside disk1 only
• Create subfolder inside the share media/movies
• Create, manually, folders data/media/movies inside disk2, disk3
• move disk1/movies to disk1/data/media/movies
• move disk2/movies to disk1/data/media/movies
• Create empty folder inside disk3/data/media/movies

Just concerned with breaking something, I know you shouldnt move files between shares and disks - but this is a share already with movies on distinct disks I want to move to a new share which should be on the same distinct disks.

[whipdancer]

What I had: 
Shares - Movies, TV, Music

What I did:
Create "data" share.

Using the terminal - create a folder `/mnt/usr/data/media`. Then, I moved the Movies folder from `/mnt/usr/` to `/mnt/usr/data/media/`.  Repeat for TV and Music.

[qal1h]

21 minutes ago, whipdancer said:

What I had: 
Shares - Movies, TV, Music

What I did:
Create "data" share.

Using the terminal - create a folder `/mnt/usr/data/media`. Then, I moved the Movies folder from `/mnt/usr/` to `/mnt/usr/data/media/`.  Repeat for TV and Music.

 

Presumably Unraid "takes care" of the which disks they go to by way of that shares' settings? For example if you had set the share up to go to disk 4 and disk6 with fill up method, that is what the terminal, lets say rsync, transfer would do?

[whipdancer]

In the terminal, the only way to see a share is to navigate to `/mnt/usr`.  If you 'ls' you will see whatever shares are defined. As far as terminal commands are concerned (AFAICT), a share is folder under `/mnt/usr` - `/mnt/usr/data`, '/mnt/usr/TV', etc.

The rules of where the data is stored are defined at the share level in Unraid. If I copy/move a file to '/mnt/usr/MyShare' - I don't know what drive it will specifically be on unless that share is mapped to a single drive. Movies, TV, Music shares were set to use most of my drives (I believe 8 out of 10 of my data drives). The data share is configured identically.

[JonathanM]

@whipdancerThe path is /mnt/user, not /mnt/usr

[qal1h]

53 minutes ago, whipdancer said:

In the terminal, the only way to see a share is to navigate to `/mnt/usr`.  If you 'ls' you will see whatever shares are defined. As far as terminal commands are concerned (AFAICT), a share is folder under `/mnt/usr` - `/mnt/usr/data`, '/mnt/usr/TV', etc.

The rules of where the data is stored are defined at the share level in Unraid. If I copy/move a file to '/mnt/usr/MyShare' - I don't know what drive it will specifically be on unless that share is mapped to a single drive. Movies, TV, Music shares were set to use most of my drives (I believe 8 out of 10 of my data drives). The data share is configured identically.

Great! Right now however, there is no data folder inside the share on disk 2.

So this is the situation right now:

• (I have a share called movies on disk 1 and disk 2 which I want to move).
• I have created a share called data
• The data share currently has media/movies folder, but the share folders (data/media/movies) currently resides on disk 1 and not on disk 2

Do I create a data/media/movies folder manually in disk 2 and move the movies from movies/ to the said manually created folder?

Edited July 22 by qal1h

[whipdancer]

You don't worry about the drives. You only worry about the shares. The shares are under `/mnt/user/........<here>.......`  As long as you can see `/mnt/user/<the_name_of_the_share_i_am_interested_in>` then that share exists.  The only way you know which drive a folder will end up on is if that is the only drive assigned to that share. If you assign more than 1 drive to a share, a given folder path will not show up on a given drive until the allotment algorithm used indicates it is time to write to the next drive the share has access to.

FYI, there are details I leave out because I don't know everything that actually happens. I do know that I use the "high water allocation" algorithm/method/process/whatever for most of my shares - and eventually the "share" ends up on all the drives allocated to that share.

 

2 hours ago, JonathanM said:

@whipdancerThe path is /mnt/user, not /mnt/usr

That's what happens when I copy-paste everything after mistyping it the first time.

 

Edited July 22 by whipdancer

[qal1h]

6 hours ago, whipdancer said:

You don't worry about the drives. You only worry about the shares. The shares are under `/mnt/user/........<here>.......`  As long as you can see `/mnt/user/<the_name_of_the_share_i_am_interested_in>` then that share exists.  The only way you know which drive a folder will end up on is if that is the only drive assigned to that share. If you assign more than 1 drive to a share, a given folder path will not show up on a given drive until the allotment algorithm used indicates it is time to write to the next drive the share has access to.

FYI, there are details I leave out because I don't know everything that actually happens. I do know that I use the "high water allocation" algorithm/method/process/whatever for most of my shares - and eventually the "share" ends up on all the drives allocated to that share.

 

That's what happens when I copy-paste everything after mistyping it the first time.

 

 

My only concern is, if I move between shares, it may end up transferring files between disks. And I'd rather not have unneccessary writes. I'd rather keep each disk's files on its own disk - which might not happen if I am only concerned with moves between shares?

[JonathanM]

Yes, you can move / rename folder names and files from folder to folder on the same disk.

 

Shares are just the root folders on the array disks or pools. As long as you keep folder names case consistent across all drives, you can manipulate the folders and files inside those root folders and the user share system will happily merge them all into the share.

 

The only very large warning here is do not involve /mnt/user paths with disk or pool paths in the same copy or move operation. If you do, you risk data loss. If you wrap your head around the way user shares work, you can mix paths, but you have to be very careful, it's easy to make mistakes. User shares show a "second" copy of a file, which really only exists once, so if you copy a file from a disk or pool path or share onto itself in a user share or vice versa you end up with a zero byte file.

 

So, either work inside the /mnt/user structure, or avoid it completely, and you should be fine.

[qal1h]

1 hour ago, JonathanM said:

Yes, you can move / rename folder names and files from folder to folder on the same disk.

 

Shares are just the root folders on the array disks or pools. As long as you keep folder names case consistent across all drives, you can manipulate the folders and files inside those root folders and the user share system will happily merge them all into the share.

 

The only very large warning here is do not involve /mnt/user paths with disk or pool paths in the same copy or move operation. If you do, you risk data loss. If you wrap your head around the way user shares work, you can mix paths, but you have to be very careful, it's easy to make mistakes. User shares show a "second" copy of a file, which really only exists once, so if you copy a file from a disk or pool path or share onto itself in a user share or vice versa you end up with a zero byte file.

 

So, either work inside the /mnt/user structure, or avoid it completely, and you should be fine.

 

I think I understand, thanks.

 

I've just transferred from disk to disk (movies(disk1) -> data/media/movies(diskt1)) and about a third of folders have been split between disk 1 and disk 2. Could it be because I had the data share to high water and automatic split?

 

I am going to manually transfer all files and sub folder files into disk 1 now. But just wanted to know if thats why they went in two differnet disks.

[JonathanM]

If the source path was /mnt/disk1/movies and the destination path was /mnt/disk1/data/media/movies, then nothing would have been written to disk2. How are you doing the moving? mc at the console would be pretty much instantaneous, as it's simply a path rename.

[qal1h]

36 minutes ago, JonathanM said:

If the source path was /mnt/disk1/movies and the destination path was /mnt/disk1/data/media/movies, then nothing would have been written to disk2. How are you doing the moving? mc at the console would be pretty much instantaneous, as it's simply a path rename.

Using Krusader - the transfer was instant - but smaller files from some folders were transferred to disk 2 with the same folder name. I think because the data share was set to high water.

 

So now I've got single folders in single disks. No more split across disks. I'm not confident that doing the bulk transfer of same disk files to same disk and different folder, wont have adverse affects again.

 

All I want to do is just put them in a sub folder. Do I remove the share completely and do folder organisation and then set a share to be those folders? Or do I set the share to not do high water and no to split directories.

 

I'm not sure if moving files between same-disk folders still applies share rules to the transfer - so for example high water will split things even though i'm doing disk to disk transfer in Krusader.

[JonathanM]

split level, drive assignments, all that is completely ignored when working directly with the disks.

 

5 minutes ago, qal1h said:

Using Krusader - the transfer was instant - but smaller files from some folders were transferred to disk 2 with the same folder name.

That's not possible if the destination was /mnt/disk1

 

From your description I think perhaps Krusader is not set up to work the way you think it is. What are your Krusader path mappings?

[qal1h]

45 minutes ago, JonathanM said:

split level, drive assignments, all that is completely ignored when working directly with the disks.

 

That's not possible if the destination was /mnt/disk1

 

From your description I think perhaps Krusader is not set up to work the way you think it is. What are your Krusader path mappings?

 

 

This is the Krusader container setup

 

And source -> destination

 

So i'm not sure what happened but the following is what I did:

- Opened 4_ALL > disk1 > movies - on the left panel

- Opened 4_ALL > disk 1 > data > media > movies - on the right panel

- Selected all folders and said "move to other panel" - it instantly transferred it as expected.

 

Edited July 22 by qal1h

[whipdancer]

Why are you trying to put everything on 1 disk if you assigned multiple disks to the share?

[qal1h]

7 minutes ago, whipdancer said:

Why are you trying to put everything on 1 disk if you assigned multiple disks to the share?

I have about 5tb on disk 1 - it was fill up method when share was just movies.

Disk2 has another 4tb again movies share.

 

I want to re-oraganise and keep all files on the disk they already are on.

 

 

[qal1h]

1 hour ago, qal1h said:

I have about 5tb on disk 1 - it was fill up method when share was just movies.

Disk2 has another 4tb again movies share.

 

I want to re-oraganise and keep all files on the disk they already are on.

 

 

So I - as a precaution - changed the share from high water to fill up, automatic split to manual: do not split, unselected all disks (so all disks) and then did the transfer and this time the expected behaviour occurred! All files are still on their respective disks. Now I have changed the share to "most free" and expect moving forward for files to go to the new 12tb drive.

[whipdancer]

On 7/22/2024 at 1:18 PM, qal1h said:

So I - as a precaution - changed the share from high water to fill up, automatic split to manual: do not split, unselected all disks (so all disks) and then did the transfer and this time the expected behaviour occurred! All files are still on their respective disks. Now I have changed the share to "most free" and expect moving forward for files to go to the new 12tb drive.

Sounds like you found a satisfactory solution.