"Include disks" and "Exclude disks" -- Global vs. per share

[huladaddy]

What's the difference between the "Include disks" and "Exclude disks" in the Shares/Share Settings and the "Include disks" and "Exclude disks" in the Settings/Global Share Settings. When do I use one or the other? What do each of them do specifically?

[garycase]

The global settings are just what they say => you can define which disks are available for use in your shares or are excluded.  [You shouldn't use both -- use either the Include or Exclude, depending on which is easiest for what you want to do]

 

The individual share settings define which disks -- of those that are available -- will be used for your individual shares.

 

 

[huladaddy]

I am wanting a better understanding. For instance, I have read that when copying from a single disk to a share that includes that disk, you must exclude that disk globally, and not by just using the individual share settings. There seems to be a difference there.

 

Also, it seems that when I exclude a disk, the files that are contained on that disk are still available through the share. Does the exclusion only apply to writes, but not reads?

 

So I'd like to know if the above is correct, and if there is anything else I should know?

[gundamguy]

I am wanting a better understanding. For instance, I have read that when copying from a single disk to a share that includes that disk, you must exclude that disk globally, and not by just using the individual share settings. There seems to be a difference there.

 

Also, it seems that when I exclude a disk, the files that are contained on that disk are still available through the share. Does the exclusion only apply to writes, but not reads?

 

So I'd like to know if the above is correct, and if there is anything else I should know?

 

The first thing you should know is that doing this puts you at a huge risk for Data Loss. Hence the reason why you want to exclude the disk it's currently on to prevent the copying from causing the glitch... (it's not a glitch exactly as much as a known issue)

 

But my question is... is there a better way to accomplish what you are trying to accomplish?

 

What is the end game you are trying to achieve that has you making this move?

[huladaddy]

Ah. I'm not actually wanting to do what I mentioned in the line above that you have bolded. I just cited that as an example of something that I read. Basically, I am just wanting a better understanding of how unRAID works.

 

But I am doing something similar. I am copying files from a single disk to a share. The share I am copying to does not include the source disk and never has. I believe that is completely safe.

[garycase]

... when copying from a single disk to a share that includes that disk, you must exclude that disk globally, and not by just using the individual share settings.

 

NO!  If you exclude the disk globally it won't be included in any share ... even if it's listed in that share's "Includes"

 

 

... Also, it seems that when I exclude a disk, the files that are contained on that disk are still available through the share. Does the exclusion only apply to writes, but not reads?

 

Correct ... Includes/Excludes -- whether local or global -- only apply to writes.    No disks are excluded when showing the contents of a share for reading.

 

[gundamguy]

Ok, now that I have a better understanding of what you are asking.

 

At a basic level user shares work by aggregating the content of the multiple disks into a user share.

From a Linux prospective it's as simple as this.

 

/mnt/user/Movies

 

Is just the sum of ALL /mnt/diskX/Movies folders for disks1-diskX where X is all your disks + Cache

 

/mnt/user0/XXXX

 

Is the same as /mnt/user/XXXX but without your Cache.

 

Fill settings such as "Most Free", "Fill Up" or "Min Free Space" apply logic that helps decide which disk from 1 to X is written to when you write to /mnt/user/Movies because a write to /mnt/user/Movies is actually just a write to /mnt/diskX/Movies (where X is the desk that the logic decides is best.)

 

Include / Exclude (don't use both at the same time) is logic that says never write to the selected disks, or only write to the selected disks when you write to /mnt/user/Movies

 

Split Level is also logic that affects writes, but split level is more about keeping sets of folders and subfolders on the same disk.

 

 

[huladaddy]

Thanks gundamguy. That wa exactly my understanding.

 

... when copying from a single disk to a share that includes that disk, you must exclude that disk globally, and not by just using the individual share settings.

 

NO!  If you exclude the disk globally it won't be included in any share ... even if it's listed in that share's "Includes"

Arg. This is frustrating. I'm quite sure I have a thorough understanding, but then I read your comment and don't understand what you are trying to say.

 

Let me try this. This is what I said in the part that you quoted by me above. If you have this:

 

/mnt/disk1/media/stuff/a

/mnt/disk2/media/stuff/b

 

you had better not do this:

 

mv /mnt/disk1/media/stuff/a /mnt/user/media/stuff/

 

but apparently, what I have read elsewhere, if you exclude disk1 globally, not in media's share settings, then you can do the mv above.

 

So then you say "NO!", but proceed to agree with me, I think?

[itimpi]

Another point that is important is that if unRAID finds a file already exists I believe that it writes the updated file to that disk regardless of which disk it is on (although I could be wrong about that).  That means that if you copy from diskX/sharenameY/fileZ  to user/sharenameY/fileZ then you end up destroying the copy of fileZ on diskX.  The safe thing to do is to rename the sharenameY folder on diskX so that it no longer corresponds to the user/sharenameY

 

Is that too confusing?

[huladaddy]

Another point that is important is that if unRAID finds a file already exists I believe that it writes the updated file to that disk regardless of which disk it is on (although I could be wrong about that).  That means that if you copy from diskX/sharenameY/fileZ  to user/sharenameY/fileZ then you end up destroying the copy of fileZ on diskX.  The safe thing to do is to rename the sharenameY folder on diskX so that it no longer corresponds to the user/sharenameY

 

Is that too confusing?

Nope. Not confusing. I am doing something similar. I have made a new, temp share and am moving everything into it, one disk at a time. At the end I will rename the share back to the original.

[garycase]

What you're referring to is the "user share copy bug".

 

You do NOT want to write to a user share from a component of that share.  The result will be a complete loss of the data for the files involved in the write.

 

THAT is what you can carefully eliminate by removing the disks you want to write to from ANY use in a share -- i.e. by adding them to the global exclude list.

 

A better approach is to NEVER do that write -- i.e. do NOT write to the share.

 

i.e. never do what you noted:  "mv /mnt/disk1/media/stuff/a /mnt/user/media/stuff/"

 

If you want to move files writhing a share to a different disk, use the DISK reference ... NOT the share reference.  e.g. it's okay to do this:  "mv /mnt/disk1/media/stuff/a /mnt/diskX/media/stuff/"

 

[huladaddy]

Yup. Completely tracking with you.

 

There is a case where moving from a disk to a share is safe, as long as its a different share, such as:

 

mv /mnt/disk1/media/stuff/a /mnt/user/different_share/stuff/

 

What you're referring to is the "user share copy bug".

 

You do NOT want to write to a user share from a component of that share.  The result will be a complete loss of the data for the files involved in the write.

 

THAT is what you can carefully eliminate by removing the disks you want to write to from ANY use in a share -- i.e. by adding them to the global exclude list.

 

A better approach is to NEVER do that write -- i.e. do NOT write to the share.

 

i.e. never do what you noted:  "mv /mnt/disk1/media/stuff/a /mnt/user/media/stuff/"

 

If you want to move files writhing a share to a different disk, use the DISK reference ... NOT the share reference.  e.g. it's okay to do this:  "mv /mnt/disk1/media/stuff/a /mnt/diskX/media/stuff/"

[gundamguy]

Yup. Completely tracking with you.

 

There is a case where moving from a disk to a share is safe, as long as its a different share, such as:

 

mv /mnt/disk1/media/stuff/a /mnt/user/different_share/stuff/

 

I mean that should work, but to be honest with you, I avoid doing that too, for the simple reason that I don't want to get in the habit of doing something that with a few minor changes can have major consequences.

 

Also I wasn't trying to be pedantic earlyer, really wasn't sure how much you knew.

 

All this said, I'm not really sure what why you would want to do a move like mv /mnt/disk1/media/stuff to /mnt/user/media/stuff

 

The only time I use user share in my paths is when I'm doing config on an application that I want to read the data from whatever disk it's on (like giving plex /mnt/user/movies instead of giving it /mnt/disk1/movies /mnt/disk2/movies /mnt/disk3/movies and so on...) because this is more flexible, if I add a disk 4 at some point and I write to the movies share and files wind up on disk4 plex wouldn't need to be reconfigured to find them...)

 

Now I do write to user shares a lot via Windows, but only when I don't care what disk in the share the data actually goes too.

 

So when you say you want to move data around in your share, I just wonder why you don't just go to /mnt/disk to /mnt/disk?

 

Would that cause problems because of balance reasons or because you are trying to move data from a share where you don't know what disk it's on to a specific disk, or move it off that disk? Just trying to get an understanding of what the goal of your planned move is, because if it's to consolidate data in a share that is on multiple disks to one disk or something we might be able to give you a better safer process.

[huladaddy]

All this said, I'm not really sure what why you would want to do a move like mv /mnt/disk1/media/stuff to /mnt/user/media/stuff

 

I agree! I have no idea why anyone would ever want to do this!

 

So when you say you want to move data around in your share, I just wonder why you don't just go to /mnt/disk to /mnt/disk? Would that cause problems because of balance reasons or because you are trying to move data from a share where you don't know what disk it's on to a specific disk, or move it off that disk?

 

Right. The second reason. I just created my unRAID machine. Copied a ton of stuff off another old NAS, and stuff from elsewhere. Once everything was copied to unRAID, and all my old disks were precleared and added, I ended up with 4 empty disks, and a bunch of nearly full disks, and the data wasn't well organized. Of course, it is just working fine as is, but I wanted to move stuff onto certain disks or groups of disks according to data type, and I wanting unRAID's allocation methods to be involved. So, therefore, I'm moving data, disk by disk, to either new temporary shares, or directly to a new disk.

 

My plan seems to be working just fine. Especially that I just solved an annoyance the rsync was giving me. All good.

[garycase]

The moving around you just did is what gets folks into trouble with the "user share copy bug."

 

e.g. somebody will add a disk to the "exclude" list for a share; and then do a move (or copy) from the disk that's no longer included [e.g. \\Tower\disk1\media\<file to copy> ]  to the share [e.g. \\Tower\media\ ], expecting that this will now put the file on one of the disks still included for the share.

 

What happens, however, is that the file contents are lost, as it becomes a zero length file [This is the "user share copy bug"].    The BEST workaround is to never do this !!

 

Personally, I've never moved any data around on the server => it's completely irrelevant where data is stored within a share, except perhaps for keeping videos that are spread over multiple files (e.g. DVD .VOB's) together so minimize drive spinups and consequent delays in playback -- which can easily be controller via the split level setting.  If I DID want to move some data, however, I'd absolutely only use disk references ... NOT user share references.

 

 

[huladaddy]

Sure. You can certainly get into trouble if you don't know what you're doing! And as I mentioned above, I don't really need to do any of this. The array is working just fine... Different personalities have different needs...

[SSD]

I believe excluding a drive from the GLOBAL setting would cause that drive to not participate in user shares (SHFS) at all, for reading, writing, or overwriting of files. But I have never tried it, and, in fact, have never heard any user exclude a disk in this manner. But if it works as I say, excluding a disk globally would make files on that disk immune to the user share copy bug.

 

If someone has time and interest they could try and confirm.

 

I don't recommend using this method, as it seems like playing with a live hand grenade. But just trying to explain what the OP read about global share settings, which I think was based on something I wrote.

[gundamguy]

I believe excluding a drive from the GLOBAL setting would cause that drive to not participate in user shares (SHFS) at all, for reading, writing, or overwriting of files. But I have never tried it, and, in fact, have never heard any user exclude a disk in this manner. But if it works as I say, excluding a disk globally would make files on that disk immune to the user share copy bug.

 

If someone has time and interest they could try and confirm.

 

I don't recommend using this method, as it seems like playing with a live hand grenade. But just trying to explain what the OP read about global share settings, which I think was based on something I wrote.

 

I didn't test this, but I would assume if it works like the normal exclude it would prevent all writing and overwriting, but wouldn't prevent reading... likely for the same reason that excluding a disk doesn't prevent existing files / folders on that disk to show up in your user share... to avoid "losing" data that isn't lost.

 

If it does prevent reading from that disk and removes it from the aggregation that could lead to some duplication issues (not really a issue as much as waste of space) and it would be good to know if that's the actual behavior because it is different then the lower level exclude include.

[SSD]

I believe excluding a drive from the GLOBAL setting would cause that drive to not participate in user shares (SHFS) at all, for reading, writing, or overwriting of files. But I have never tried it, and, in fact, have never heard any user exclude a disk in this manner. But if it works as I say, excluding a disk globally would make files on that disk immune to the user share copy bug.

 

If someone has time and interest they could try and confirm.

 

I don't recommend using this method, as it seems like playing with a live hand grenade. But just trying to explain what the OP read about global share settings, which I think was based on something I wrote.

 

I didn't test this, but I would assume if it works like the normal exclude it would prevent all writing and overwriting, but wouldn't prevent reading... likely for the same reason that excluding a disk doesn't prevent existing files / folders on that disk to show up in your user share... to avoid "losing" data that isn't lost.

 

If it does prevent reading from that disk and removes it from the aggregation that could lead to some duplication issues (not really a issue as much as waste of space) and it would be good to know if that's the actual behavior because it is different then the lower level exclude include.

 

My understanding is that the global feature literally exclude a disk from participating in the shared fine system feature of unRaid at a technical level, based on feedback from Tom. It's as though the disk isn't there for any share purposes. You can confirm and I will apologize if you find otherwise, but I am pretty certain.

[cinereus]

On 2/3/2016 at 6:54 AM, garycase said:

The moving around you just did is what gets folks into trouble with the "user share copy bug."

 

e.g. somebody will add a disk to the "exclude" list for a share; and then do a move (or copy) from the disk that's no longer included [e.g. \\Tower\disk1\media\<file to copy> ]  to the share [e.g. \\Tower\media\ ], expecting that this will now put the file on one of the disks still included for the share.

 

What happens, however, is that the file contents are lost, as it becomes a zero length file [This is the "user share copy bug"].    The BEST workaround is to never do this !!

 

Personally, I've never moved any data around on the server => it's completely irrelevant where data is stored within a share, except perhaps for keeping videos that are spread over multiple files (e.g. DVD .VOB's) together so minimize drive spinups and consequent delays in playback -- which can easily be controller via the split level setting.  If I DID want to move some data, however, I'd absolutely only use disk references ... NOT user share references.

 

 

Hang on. A "bug" this big doesn't seem to be mentioned often! Is this still an issue in 2020? I have been reading so much about unraid for the last months and have never heard of it.

 

Would I be right in thinking move/copy commands explicitly mentioning the disk numbers would be a workaround?

[cinereus]

Another couple of questions.

 

1. What happens when there's no free space on the disks enabled for a share and you try to write? Does it automatically include another disk? Does it tell the user there's no space? Does it silently fail and lose the data?

 

2. What about if you explicitly copy to a disk where that share is no allowed?

 

3. What happens to files on a disk that is then excluded from that share?

Edited April 28, 2020 by cinereus
add

[itimpi]

1 hour ago, cinereus said:

Hang on. A "bug" this big doesn't seem to be mentioned often! Is this still an issue in 2020? I have been reading so much about unraid for the last months and have never heard of it.

It is a Linux level architectural limitation so not really fixable!  This is one of the reasons that dirk shares are not enabled by default as that makes it harder for a naïve user to mic dirk and user shares in the same copy/move command.

1 hour ago, cinereus said:

would I be right in thinking move/copy commands explicitly mentioning the disk numbers would be a workaround?

That is fine The issue only arises if y/u try and mix disk shares and user shares in the same copy/move command.  Using just disk shares or just user shares is fine.

1 hour ago, cinereus said:

1. What happens when there's no free space on the disks enabled for a share and you try to write? Does it automatically include another disk? Does it tell the user there's no space? Does it silently fail and lose the data?

You need to read up on Allocation Method, Split Level and Minimum Free Space settings for a share that are used to determine which drive Unraid will select for a new file (note that this selection does NOT take into account the size of the file you are trying to write)    After selecting a drive, if it turns out these is not enough space then you will get an "out of space" failure when free space is exhausted.

 

1 hour ago, cinereus said:

2. What about if you explicitly copy to a disk where that share is no allowed?

That by-passes the user share level so will go ahead.

 

1 hour ago, cinereus said:

3. What happens to files on a disk that is then excluded from that share?

The exclude just applies to where new files will be placed.  They will still be found for read purposes.

[cinereus]

7 hours ago, itimpi said:

It is a Linux level architectural limitation so not really fixable!  This is one of the reasons that dirk shares are not enabled by default as that makes it harder for a naïve user to mic dirk and user shares in the same copy/move command.

That is fine The issue only arises if y/u try and mix disk shares and user shares in the same copy/move command.  Using just disk shares or just user shares is fine.

You need to read up on Allocation Method, Split Level and Minimum Free Space settings for a share that are used to determine which drive Unraid will select for a new file (note that this selection does NOT take into account the size of the file you are trying to write)    After selecting a drive, if it turns out these is not enough space then you will get an "out of space" failure when free space is exhausted.

 

That by-passes the user share level so will go ahead.

 

The exclude just applies to where new files will be placed.  They will still be found for read purposes.

Okay, thanks for a comprehensive reply.

 

So the includes/excludes seem to only affect disk writes and only when using the /mnt/user/share form not the /mnt/diskx/share form?

 

What's a "dirk share"?

[trurl]

Typo, he meant disk share. If you aren't sharing disks on the network might as well just call it disks. In either case, you shouldn't mix disks and user shares when moving/copying.

 

In general, the user share settings only apply to new writes to the share, except the Use cache setting of course. Any read of a share will include any disks that include the folder for that share.

 

Another thing that happens down at the linux level that can be surprising is that if you try to move something from one user share to another, it will just get "repathed" on the same disk. This is because the linux command for move and rename are the same, mv. So, even if destination user share excludes the disk, it still winds up there. The workaround for this is to copy from source to destination, so a new file gets written according to the user share settings, then delete from the source.

[cinereus]

What happens if you have a file e.g. user/share/file (stored as /disk1/share/file) and you then create a file at /disk2/share/file?