unRAID OS Version: 6.0-beta6

 

Description: If a cache-only share is created, and the same directory exists on an array disk, the mover script will try to move the data from the cache disk to the array disk, ignoring the fact that it is a cache-only share.

 

How to reproduce:

See description

 

Other information:

The bug here is that a non-cache-only share is still shown as 'cache-only' in the webGui.

 

The fact that a share exists only on the cache is the definition of a cache-only share.

 

You change a cache-only share to a non-cache-only share precisely by creating the share on an array disk.

The fact that user shares = top level directories across all the disks (cache included) is not clear. The special configuration screen lets the user include/exclude/indicate if cache is included, etc. leads users to believe that their configurations, and not purely the existence/nonexistence of root level directories on the disks, defines the behavior. This is a big problem! It probably lead to the user share copy bug being encountered.

 

Dynamically updating the share definition as you create directories might add to the confusion as a user goes back and sees their cache only share become not cache only and don't connect it to the fact that they accidentally made a directory on an array disk with the same name as the cache directory name.

 

You can't just configure one or two user shares and expect all other root directories to be treated as though user shares are turned off.  So you have to be very very careful with the top level directory names!

 

Perhaps the setting page should populate based on directory existance and be read only, with a button to create the user share directory on a new disk.

 

The fact that user shares = top level directories across all the disks (cache included) is not clear. The special configuration screen lets the user include/exclude/indicate if cache is included, etc. leads users to believe that their configurations, and not purely the existence/nonexistence of root level directories on the disks, defines the behavior. This is a big problem! It probably lead to the user share copy bug being encountered.

 

Dynamically updating the share definition as you create directories might add to the confusion as a user goes back and sees their cache only share become not cache only and don't connect it to the fact that they accidentally made a directory on an array disk with the same name as the cache directory name.

 

You can't just configure one or two user shares and expect all other root directories to be treated as though user shares are turned off.  So you have to be very very careful with the top level directory names!

 

Perhaps the setting page should populate based on directory existance and be read only, with a button to create the user share directory on a new disk.

 

The key concept is that the user share file system is simply a union of all storage devices.  When you browse a share you will always see all files and subdir's that exist under that top-level directory name on each array and cache device (but see ** below).

 

All these fields:

- min free space

- split level

- included disks

- excluded disk

- use cache

only have to do with deciding where to put new files being written to the share.  They have nothing to do with whether any device is included or excluded from the union view of the share when reading directories.

 

With 'use cache' you have 3 options:

- no: do not consider using the cache when a new object is created

- yes: ok to consider using the cache when a new object is created (not guaranteed though)

- only: create all new objects only on the cache  (despite if share is on array too)

 

If 'use cache' is set to 'only' and there is no more space on the cache, then the create will fail with "out of space" error.

 

Now let's talk about the mover.  The mover simply looks at each top-level directory on the cache.  For each directory, if that directory also exists on any array device, then the mover will move the contents of that directory on the cache to the array (and then delete the directory on the cache).  If a top-level directory exists on the cache but not on the array, then nothing is moved - the mover will not create top-level directories on the array.

 

Now combine how the mover works with idea of 'use cache' = 'only'.  When you create a new share and fill in the info such as the name, etc., if you set 'use cache' to 'only', then the top-level directory will be created on the cache.  In this case, all data written to that share will remain on the cache.  If instead, you set 'use cache' to 'no' or 'yes', then the top-level directory will be created on an array disk.

 

So you see, if you start out with a 'cache only share', then later manually create the same-name directory on an array disk, when the mover runs it will see this and proceed to move data off the cache and onto the array for that share.  This is a feature not a bug.

 

This to me seems pretty clear. 

 

All I did to "fix" this bug was add another color indicator (red) to the share list page:

green - all data for this share is on the array

orange - some or all data for this share is currently on the cache (but can get moved)

red - all data is on the cache and will not get moved

 

I guess we could add:

"convert red to orange" - not sure what to call this, but click the button and it creates share directory on the array.

 

Another feature easy to add is this: the mover could look for the existence of a certain file in a top-level directory, and if exists, skip moving.  Maybe call this ".nomove" or ".moverignore".  Really needed though?  Also, though such a file would be hidden in linux, it's visible in windows.  To hide we'd have to set the 'hidden' bit, but then that adds complications to Active Directory set ups, so you see this gets complicated real fast.  Could also do it with an extended attribute - but again, just adds unnecessary complexity imho.

 

**  Actually you can specifically "hide" array devices from being seen by shfs using the include/exclude masks on the Share Settings page; but can't hide cache.  This was added to "partition" the array into a set of disks which participate in user shares, and a set that don't.  The idea being that the set that don't are used for AFP (time machine in particular).  This is because netatalk doesn't lend itself very well to our union-like shfs file system because of it's cnid database.  Unfortunately when I added AFP support I meant to prohibit user share export via AFP but forgot to turn this off & people starting using it so I said, ok, well why not?

 

*** Finally one more mover feature... At some point we will implement idea of copying (instead of moving) files from the cache to the array.  This will result in "duplicates", aka, "backups".

 

I agree with Tom -- this isn't a "bug" => it's an education problem.

 

There needs to be more clarity on the Shares page that explains that shares are simply top level folders that are merged ... and that all the various share settings are only used when determining where to write data.

 

It's really very simple:  Don't use the same name for folders that you expect to be treated differently -- and remember that UnRAID's shares merge all of your disks, so when considering folder names, you need to be sure you don't duplicate them on different disks UNLESS you expect them to be treated as a single entity.

 

I think the color indicators Tom added is all the "fix" that's needed for this.

 

Tom -

 

Your writeup of the defect is better than the documentation.

 

But the configuration page certainly leads users to believe that IT controls user shares and that simply isn't the case. Maybe the configuration page should only show up ONCE, when the share is created. And going forward they get a different screen that only allows them to edit a couple of fields (included disks for write, excluded disks for wrete, and upgrade share to be cached).

 

Or you could relabel some of the fields (e.g., Use Cache Disk Initially?"), and "Included disks for write". But as it is, users would be easily lead to believe the fields are actually relevant. I still think it would be very useful on the shares screen to list the disks that are participating in the share.

 

Call it a bug. Call it poorly documented. At the end of the day they are one in the same.

 

What happens if you take an existing share and set it to not use the cache. What would it do? Would it give an error that it can't do it? Would it move the remaining files on the cache to the array and then not cache anything else? Would it delete everything from the cache? Would it do absolutely nothing? I'm guessing the latter but I know for sure that is not what a typical user would believe.

 

And what would happen with a share configured to not use the cache and someone creates a directory on the cache disk that is the same as the name of a user share? Would the mover ignore it since the share is configured to not use the cache? Now I think not, but previously I thought so. But when a user brings up the configuration page and it says not to use the cache disk, but then he sees the files being cached, do you think he might be confused?

 

Not trying to be snarky, but just trying to represent the users here. Having the mods kiss butt is not going to help anyone here.

Your writeup of the defect is better than the documentation.

I'll be copy/pasting text from these threads into the documentation.  That is one reason why I'm trying to be thorough.  Sometimes I forget exactly how everything works because there are dozens of different subsystems that make up unraid and I can no longer keep them all in my head at the same time.

 

What happens if you take an existing share and set it to not use the cache. What would it do? Would it give an error that it can't do it? Would it move the remaining files on the cache to the array and then not cache anything else? Would it delete everything from the cache? Would it do absolutely nothing?

If you set 'use cache' to No and click Apply nothing would happen immediately, other than setting the new value.  The next time you go to create an object on that share however, it will create it on the array, even if the share previously had 'use cache' set to Only.

 

Files on the cache will simply remain there until next time the mover runs, which can be immediate if you click the 'Move Now' button.

 

But... in the case of a true 'cache-only' share, even if you change 'use cache' to No, the mover still won't move stuff to the array because the top-level dir does not exist there.  Workaround is to create a file in the share (and you can immediately delete it); this will cause the share directory to get created on the array.

 

This issue is on a real old bug list I have and I went and changed the message icon for this post so that I don't forget again to fix it.

 

I'm guessing the latter but I know for sure that is not what a typical user would believe.

Well yes depending on preconceived assumptions it can be confusing.

 

And what would happen with a share configured to not use the cache and someone creates a directory on the cache disk that is the same as the name of a user share?

Any objects that exist in that directory will show up along with objects from the same-named directory on all the other disks.

 

Would the mover ignore it since the share is configured to not use the cache? Now I think not, but previously I thought so.

You know the answer to this.  The mover doesn't know about any of the "cache settings" for a share.  It's just a script that operates on top-level directory names.  For each name on cache, if same name exists on array, objects are moved.  If same name does not exist on array, nothing is moved.  It's just that simple.

 

But when a user brings up the configuration page and it says not to use the cache disk, but then he sees the files being cached, do you think he might be confused?

Not if he understands what's happening under the hood.

 

Not trying to be snarky, but just trying to represent the users here. Having the mods kiss butt is not going to help anyone here.

I respect you guys a lot and appreciate all the insight you can offer.

 

 

These "issues" [cache-only share "bug" and "user share copy" bug] are all caused by the fact that UnRAID effectively has two different "views" of the disks -- a unified view that treats the entire array as a big "disk"; and a disk view that is specific to a specified disk.  Consequently, things that you CAN'T do on a single disk can suddenly be attempted without realizing that this is what's happening.

 

I'm not sure there's any real "solution" to it other than educating the users to understand the potential consequences.  I suppose it's possible to come up with workarounds that would eliminate the possibility of attempting these operations -- but many of those have undesirable side-effects.  [i.e. making shares "read only".]

 

One thing (easy?) that could help clarify the UI is to make it so Use Cache: Only is not an option for shares that already have a folder on the array.

One thing (easy?) that could help clarify the UI is to make it so Use Cache: Only is not an option for shares that already have a folder on the array.

 

Right, do that and then  pop up a notice saying a new cache-only share must be created and everything moved over manually.  A savvy user could create the top level on the cache, move all the files, then delete the top-level on the array and if I understand everything Tom says it should all work.  A less savvy user would need to create a new share, move everything via their desktop, remove the old share, and then rename the new share to the old share's name (assuming they had mappings to maintain).

 

Or create a "convert to cache-only share" script button that will do all that for the user.  i mean the script should be pretty easy, since it is basically a reverse mover right?

Remember this is the defect forum, not the enhancement forum.

 

I agree that there are enhancements that could be made to make user shares more user friendly and automate functions.  You can add them as suggestions.

 

For fixing this defect and others related to confusing user share behavior, I think the prompts needs to be more precise.

 

Here are my suggestions:

 

1 - On the user share global screen, it is not clear what included disks, and excluded disks mean relative to the individual share screen. If the included disks are the disks that can participate (for read or write) in ANY user share, and the excluded disks are disks that CANNOT PARTICIPATE (for read or for write) in ANY user share, the labels should be updated. (e.g., "Disks available for user shares", and "Disks unavailable for user shares"). [if any of these are selected on the individual user share page, it should generate an error], and similarly [if one is attempted to be excluded that is already being used in a user share, it should generate an error].

 

2 - On the same user share global screen, the "use cache disk" is also confusing. Should be labeled or explained with something like "Allow option for cache disk to buffer user share writes".

 

3 - On the specific share setting screen, the include and exclude disks labels should be changed to "Disks used for new files" and "Disks not used for new files". The "Use cache disk" if the "only" setting is selected, it should contain a warning if an array disk is in the share  indicating that an array disk exists for the user share and are acting like "yes" unless the array directory is renamed.

 

4 - A bit of an enhancement, but something that would clear up a lot of confusion, list the disks that are currently participating (i.e., contain files) in the user share for read purposes at that moment in a read only field.

 

I think this would clear up much of the confusion. This requires no change in functionality of the user share feature - just label changes.

Remember this is the defect forum, not the enhancement forum.

 

I agree that there are enhancements that could be made to make user shares more user friendly and automate functions.  You can add them as suggestions.

 

For fixing this defect and others related to confusing user share behavior, I think the prompts needs to be more precise.

 

Here are my suggestions:

 

1 - On the user share global screen, it is not clear what included disks, and excluded disks mean relative to the individual share screen. If the included disks are the disks that can participate (for read or write) in ANY user share, and the excluded disks are disks that CANNOT PARTICIPATE (for read or for write) in ANY user share, the labels should be updated. (e.g., "Disks available for user shares", and "Disks unavailable for user shares"). [if any of these are selected on the individual user share page, it should generate an error], and similarly [if one is attempted to be excluded that is already being used in a user share, it should generate an error].

 

2 - On the same user share global screen, the "use cache disk" is also confusing. Should be labeled or explained with something like "Allow option for cache disk to buffer user share writes".

 

3 - On the specific share setting screen, the include and exclude disks labels should be changed to "Disks used for new files" and "Disks not used for new files". The "Use cache disk" if the "only" setting is selected, it should contain a warning if an array disk is in the share  indicating that an array disk exists for the user share and are acting like "yes" unless the array directory is renamed.

 

4 - A bit of an enhancement, but something that would clear up a lot of confusion, list the disks that are currently participating (i.e., contain files) in the user share for read purposes at that moment in a read only field.

 

I think this would clear up much of the confusion. This requires no change in functionality of the user share feature - just label changes.

 

 

Agree these would improve the clarity of how the shares function ...

 

However, r.e.

 

This requires no change in functionality of the user share feature - just label changes.

 

... it would require additional logic to generate the lists of participating disks;  check for existing top level folders (to generate the warnings); etc.  --- so it's not quite "... just label changes."

 

Agree it should be relatively simple to do, however.    ... and would be a nice improvement.  Coupled with a CLEAR warning about the circumstances that generate the "user share" copy issue, this would likely go a long way to eliminating the problem ... at least for those users who pay attention.

 

 

I agree that the global include and exclude and that the cache only setting constantly screw with people. I like the suggestions to improve the global share settings page and I agree that selecting a disk in a share setting that isn't allowed by the global settings should give a warning.

 

I would also suggest putting a read-only warning box (possibly with a link to a better Wiki explanation) beside the cache setting that gives a warning when the user has the cache only share setting with a disk share existing. Even if they manually create an array share directory and have the issue they're likely going to double check the share page and see the warning.

 

I like the idea of a field listing the disks included in a share. There should be a list of disks that can be used for future writes. So maybe 2 new read-only fields would be appropriate, one with disks that contain share data and the other with disks that can be written to.

 

You want to allow users to exclude a disk that currently contains data so you can't give an error if that is done. Many users do this as each disk gets full to "lock" the contents into a read-only state. A warning might be appropriate though just as a FYI to the user.

 

 

Remember this is the defect forum, not the enhancement forum.

 

TomAto / Tamaato ... I was suggesting a way to deal with your "bug" ;-)

Remember this is the defect forum, not the enhancement forum.

 

TomAto / Tamaato ... I was suggesting a way to deal with your "bug" ;-)

 

No worries guys.  Sometimes there is a very fuzzy line between what's a bug and what's a feature and what's an enhancement.  This is definitely one such case.