user share questions

[CaptainTivo]

Hello,

 

I am planning to do a massive pruning of old/low capacity disks (see here for details: https://forums.unraid.net/topic/95257-advice-needed-on-massive-array-shrink  and I find I don't fully understand how user shares work.

My plan is to copy data which should remain on the server from a disk ("copy from" disk)  to be removed to a disk which is to remain ("copy to" disk) and then remove the "copy from" disk and rebuild parity.

Since these are fairly large disks (2,3 and 4 TB), these copies will take some time.  My questions are about how the shares will behave during this copy time.

 

1) I am aware of some of the pitfalls of copying between disks (e.g. /mnt/disk1/Videos) and user shares (/mnt/user/Videos) so all copies will be using rsync from /mnt/diskx to /mnt/disky.  The problem here is that I will have to manually determine that there is enough free space on the target disk to hold all of the data on the source disk assuming I simply run "rsync -avh /mnt/disk1 /mnt/disk2".  Is there a better way to do this?

 

1) assume the copy will take a very long time and that there is some free space on the "copy from" disk. During the copy period, I would not want new files to be allocated on the "copy from" disk because they might be missed and not copied to the "copy to" disk and therefore disappear from the server.  If the "copy from" disk is included in a share,  any new files written to a user share could be allocated onto the "copy from" disk and we need to avoid that.  If I exclude (or do not include) the "copy from" disk from the share, will that prevent new files being allocated on it?  If not, is there another solution to this problem?

 

2) for a period during the copy operation, any given file may exist on BOTH the "copy from" and the "copy to" disks.  Which copy will be used when the file is read from the user share? My mental "model" of user shares is that they only operate during allocation but I don't know how a file is chosen when the file is read from a share.  In other words, if a file exists on two disks in the the same place relative to the directory tree, which one is chosen to be read?

 

3) I am planning to exclude all shares that have these file from being cached but selecting "Use cache (for new files/directories): No" in "Share settings".  Is this sufficient to prevent new files from being left on the cache during the copy?  Is this even necessary?

 

This is all I can think of.  Are there other pitfalls I should be aware of?

 

Thanks for any help.

 

[Energen]

You probably could have continued this in your other thread.. but anyways, so having recently gone through something similar -- not for replacing disks but having to move data to format one disk at a time, here's some suggestions.

 

First, the Unbalance plugin makes things a little easier.  You can move everything from one drive to another with the visual click of a couple buttons.

 

Second, you should probably do what you can to "shut down" the server while the moving is in progress... and that will solve all your other questions.  Don't do any major operations while you're moving the data... don't add new files, don't remove files, etc, to the extent possible.  The files you're copying just shouldn't be accessed while you're doing this.. don't add to the situation. 

 

This is like a surgical operation and while you're on the table you're not doing anything else.. if you want to make massive changes to your server with files and folders and disks, you shouldn't also be trying to use it as normal at the same time.

 

You can exclude a disk from every share to prevent new data being written to it while copying off of it, which wouldn't be a problem if you're not actively using the server.

[trurl]

6 hours ago, CaptainTivo said:

I am planning to exclude all shares that have these file from being cached but selecting "Use cache (for new files/directories): No" in "Share settings".  Is this sufficient to prevent new files from being left on the cache during the copy?  Is this even necessary?

When working directly with the disks, cache is never involved (unless cache is one of the disks you are working directly with, of course).

[trurl]

6 hours ago, CaptainTivo said:

if a file exists on two disks in the the same place relative to the directory tree, which one is chosen to be read?

In disk order starting with cache

[trurl]

6 hours ago, CaptainTivo said:

If I exclude (or do not include) the "copy from" disk from the share, will that prevent new files being allocated on it?

yes

 

But having answered all that I agree with @Energen generally.

[JonathanM]

If you are ok with the files on the source drive temporarily disappearing completely from user shares until they land on their destination, you can globally exclude the source disk from user shares. That will eliminate the source drive as a destination for any user share writes. That setting was introduced pretty much for exactly what you are doing, and makes it safe to copy from the globally excluded disk to a user share. It's the only way to safely mix disk shares and user shares. 

 

Settings, Global Share Settings.

[CaptainTivo]

14 hours ago, jonathanm said:

If you are ok with the files on the source drive temporarily disappearing completely from user shares until they land on their destination, you can globally exclude the source disk from user shares. That will eliminate the source drive as a destination for any user share writes. That setting was introduced pretty much for exactly what you are doing, and makes it safe to copy from the globally excluded disk to a user share. It's the only way to safely mix disk shares and user shares. 

 

Settings, Global Share Settings.

Thanks, I did not know about that setting. I had always been told to copy ONLY from user share to user share or from disk share to disk share or data would be destroyed.  So, with this setting, I could copy from a particular disk (e.g. /mnt/disk1) to a user share (e.g. /mnt/user/Videos/Movies/DVD) and it will not clobber the file?

 

You say that when you globally exclude a disk from a user share that files on the disk will "disappear" from a user share during the period that it is being copied?  Why?  I mean, it still exists on the source before it is copied and then it exits on both the source and the destination after it has been copied.

 

What I really want is to have no new files be allocated on the source disk because, depending on when during the copy that happened, rsync might not actually copy it to the destination.  This setting appears to do that. Is this correct?

 

Also, I have been reading that some people use the plugin "UnBalance" for this purpose but I do not understand how it can be used for this purpose.  I need to do more reading, I guess.

 

 

[trurl]

5 minutes ago, CaptainTivo said:

So, with this setting, I could copy from a particular disk (e.g. /mnt/disk1) to a user share (e.g. /mnt/user/Videos/Movies/DVD) and it will not clobber the file?

 

You say that when you globally exclude a disk from a user share that files on the disk will "disappear" from a user share during the period that it is being copied?  Why?

Because the disk is no longer part of user shares. That is the whole point of the setting. All array and cache disks are combined into the user shares, except for disks explicitly excluded in Global Share Settings. So any files on the excluded disk are not combined into the user shares.

 

The problem with mixing disks and user shares when moving or copying files is simply because you could specify an operation where the source file and the destination file were actually the same file, but linux wouldn't know it because it just sees the user shares as another mount point. So linux tries to write what it is trying to read, with the result that the file is truncated when it is opened for writing, and then there is nothing to read. This can't happen with a disk excluded from user shares.

[FreeMan]

You could use the unBalance plug in to do the moves for you. It will read from a disk and write to the same share on one or more other disks as necessary - you decide which destination disk(s) you want it to consider.

 

I've used it numerous times to clear a disk prior to removal from my system and it works a treat. You don't even need a terminal session going, and, IIRC, you don't need to leave the browser window open either (I'm not 100% certain on that - it's been a bit since I last used it, and I leave the browser open anyway).

[Energen]

3 hours ago, CaptainTivo said:

Also, I have been reading that some people use the plugin "UnBalance" for this purpose but I do not understand how it can be used for this purpose.  I need to do more reading, I guess.

It's basically a GUI for rsync.  You don't understand how because you have not installed it and looked at it.

 

Select your source, select your destination, and run.  The end.