Application Data: Cache-only Share or Hidden Folder?

[trurl]

It seems a lot of users are not getting the message about cache-only shares and their apps get broken by mover. I don't know how many posts I have made about this trying to help people.

 

It seems PhAzE is still using hidden folders on cache for this, and I am beginning to wonder if this is not a better approach.

 

Hidden folders do have the disadvantage of not being shown sometimes, for example when using some GUI to browse. Just tried it with docker manager and it does browse hidden folders.

 

But hidden folders also have the advantage of automatically being ignored by mover. I wasn't sure whether hidden folders were aggregated in /mnt/user but I just tried from both the command line and by creating a .test share in the GUI and it looks like they are.

 

I know I was involved in some discussion probably a few months ago where it was proposed that top level cache folders would not be shares unless the user explicitly made them in the GUI. A quick search didn't turn up the thread though. I remember questioning that idea since auto creation of shares is also a good feature.

 

Anyway just thought I would put this out there for further discussion. If someone can find the other thread I referred to please post a link, or to any other threads that might be relevant.

 

Mostly just wanted some ideas on how we can help users do this the right way, or prevent them from doing it the wrong way, or maybe even change the way it's done to help with this problem.

[trurl]

One thing about hidden folders I just noticed is that they don't show up on the shares page even though they show up in /mnt/user. Even the .test share I created in the GUI does not get listed in the shares page, but it is in /mnt/user. I also created a .folder in /mnt/cache, and that shows up in /mnt/user also, but not on the shares page. So the shares page is one place where hidden folders cannot be seen, but docker manager will let you browse to them when selecting host paths.

[dmacias]

I have always used .appdata. I have several hidden folders on my cache drive. I do have a few cache only shares too though, like for downloads. I think they both have their place.  Hidden won't show up under smb in Windows by default. I don't want or need network access to my hidden folders anyway. I figure ubuntu uses hidden folders for app data so why not. I also prefer the auto creation of shares even though I don't always want that.

[trurl]

I have always used .appdata. I have several hidden folders on my cache drive. I do have a few cache only shares too though, like for downloads. I think they both have their place.  Hidden won't show up under smb in Windows by default. I don't want or need network access to my hidden folders anyway. I figure ubuntu uses hidden folders for app data so why not. I also prefer the auto creation of shares even though I don't always want that.

I am starting to lean toward the hidden folder as well, even though I have spent countless posts telling users about cache-only shares. There are some docker templates that default to /mnt/cache/appdata though, with nothing to tell users that this is a potential problem.

 

Maybe we could get the developers to change the templates to use /mnt/cache/.appdata . Then the users wouldn't shoot themselves in the foot because they don't know about cache-only shares.

[Squid]

My point of view is that I hate hidden folders / files with a passion.  Nothing aggravates me more than programs writing files that I cannot (under normal circumstances) see.

 

Perhaps a quick sanity check when creating the container to see if the folder is not set to cache only and toss up a warning.

[trurl]

My point of view is that I hate hidden folders / files with a passion.  Nothing aggravates me more than programs writing files that I cannot (under normal circumstances) see.

 

Perhaps a quick sanity check when creating the container to see if the folder is not set to cache only and toss up a warning.

Which folder should be cache-only will vary from docker to docker though. Maybe plugin authors could do this check, but you would need more logic in dockerman (and templates) to make this happen with dockers. And having plugin authors implement this individually would also not be ideal.

[Squid]

My point of view is that I hate hidden folders / files with a passion.  Nothing aggravates me more than programs writing files that I cannot (under normal circumstances) see.

 

Perhaps a quick sanity check when creating the container to see if the folder is not set to cache only and toss up a warning.

Which folder should be cache-only will vary from docker to docker though. Maybe plugin authors could do this check, but you would need more logic in dockerman (and templates) to make this happen with dockers. And having plugin authors implement this individually would also not be ideal.

Absolutely an XML change would be required to implement this.  It was however my 2cents worth of input. 

 

For new users coming to docker (and now that we're in RC stage, and soon Final), this problem is going to happen over and over again, and some sort of solution would be required.  Maybe if I get bored someday I'll take at look at the mover script and see about the changes of ignoring top-level shares.

[trurl]

...  Maybe if I get bored someday I'll take at look at the mover script and see about the changes of ignoring top-level shares.

Don't understand this statement. Moving top level folders from cache is what mover does.

[Squid]

On a quick thought, seems to me to be easier to have ignore top level cache folders that aren't shares.  Not sure however when the cfg file for a share is actually created however, so may be all for naught.

[trurl]

On a quick thought, seems to me to be easier to have ignore top level cache folders that aren't shares.  Not sure however when the cfg file for a share is actually created however, so may be all for naught.

From reading log files, the way I think it works currently is when the array is started, unRAID looks at all top level folders on cache and array, then looks in /config/shares for a .cfg for that folder. If there isn't a .cfg for the folder then a share is created with default settings. If there is a .cfg for the folder then a share is created using the settings in the .cfg . So either way a share is created.

[bonienl]

On a quick thought, seems to me to be easier to have ignore top level cache folders that aren't shares.

 

This is going back how it used to be ...

 

[Squid]

Probably hasn't changed which means that approaching this problem via mover is a non starter

[trurl]

On a quick thought, seems to me to be easier to have ignore top level cache folders that aren't shares.

 

This is going back how it used to be ...

Could you elaborate (extensively) on this please? I was unaware there was any change, and how it was before / is now is mostly folklore.

[bonienl]

It used to be that top folders present on the cache drive which were not a share, did not get moved (at least that was my observation).

 

[trurl]

It used to be that top folders present on the cache drive which were not a share, did not get moved (at least that was my observation).

Looking at mover in it's current incarnation (if my understanding is correct), it skips cache folders that have a .cfg with the cache-only setting and moves all others.

 

And the way it should work (may have at one time): Only move it if it already exists in /mnt/user0 or if there is a .cfg file for it without the cache-only setting.

 

This does seem like it would avoid handing the users a cocked and loaded gun to use on their foot.

 

So this is going to happen?

[lionelhutz]

Only mover change I remember before the addition of ignoring hidden files was making it delete the cache directories. It didn't delete the leftover directories at one time.

 

If it used to ignore directories that didn't have a cfg file then the addition of ignoring hidden files really wouldn't have been necessary.

[gundamguy]

 

And the way it should work (may have at one time): Only move it if it already exists in /mnt/user0 or if there is a .cfg file for it without the cache-only setting.

 

 

So isn't this basically setting the default setting for new cache shares to cache-only?

 

This seems like a good move and would help avoid the current user error issue.

[trurl]

...setting the default setting for new cache shares to cache-only...

This I think captures the concept in its simplest terms, and we can leave the implementation details up to limetech.

[Freddie]

For the historically curious, the relevant mover change happened at v6beta14. Here is a little bit of explanation:

http://lime-technology.com/forum/index.php?topic=38353.msg356217#msg356217

[lionelhutz]

Interesting. Then might as well just add back in the check to not move if there is no matching directory in /mnt/user0. instead of replacing that check it should have been OR'd with the check for a cache only share.

[trurl]

I have attached a zip with mover from 6b12 and 6rc3 for comparison

mover.zip

[danioj]

For me, in my new setup Cache drive is a Cache drive and thats all.

 

Thanks to the plugin 'Unassigned Devices' (it seems yet to be named) created by the awesome @gfjardim. BTW: this functionality HAS to be included natively into Unraid - it's THAT good!

 

http://lime-technology.com/forum/index.php?topic=38635.0

 

I bought an additional drive which this plugin lets me mount outside the protected array (as in "nothing" to do with it at all") and mount it and share it as "app". Anyway, I have placed my Docker.img onto it (literally 5 mis ago) and plan to do the same thing with my VM's.

 

I will back this up to the protected array at a suitable frequency but all in all great!

 

 

[limetech]

How about, instead of this:

 

#!/bin/bash

# This is the 'mover' script used for moving files from the cache disk to the
# main array.  It is typically invoked via cron.

# After checking if it's valid for this script to run, we check each of the top-level
# directories (shares) on the cache disk: if the share is marked "cache-only" then we
# skip this directory; otherwise, we use the 'find' command to process the objects (files
# and directories) of that directory, moving them to the array.

# The script is set up so that hidden directories (i.e., directory names beginning with a '.'
# character) at the topmost level of the cache drive are also not moved.  This behavior can be
# turned off by uncommenting the following line:
# shopt -s dotglob

# Files at the top level of the cache disk are never moved to the array.

# The 'find' command generates a list of all files and directories on the cache disk.
# For each file, if the file is not "in use" by any process (as detected by 'fuser' command),
# then the file is copied to the array, and upon success, deleted from the cache disk.
# For each directory, if the directory is empty, then the directory is created on the array,
# and upon success, deleted from the cache disk.

# For each file or directory, we use 'rsync' to copy the file or directory to the array.
# We specify the proper options to rsync so that files and directories get copied to the
# array while preserving ownership, permissions, access times, and extended attributes (this
# is why we use rsync: a simple mv command will not preserve all metadata properly).

# If an error occurs in copying (or overwriting) a file from the cache disk to the array, the
# file on the array, if present, is deleted and the operation continues on to the next file.

# Only run script if cache disk enabled and in use
if [ ! -d /mnt/cache -o ! -d /mnt/user0 ]; then
  exit 0
fi

# If a previous invokation of this script is already running, exit
if [ -f /var/run/mover.pid ]; then
  if ps h `cat /var/run/mover.pid` | grep mover ; then
      echo "mover already running"
      exit 0
  fi
fi
echo $$ >/var/run/mover.pid
echo "mover started"

cd /mnt/cache
shopt -s nullglob
for Share in */ ; do
  if grep -qs 'shareUseCache="only"' "/boot/config/shares/${Share%/}.cfg" ; then
    echo "skipping \"${Share%/}\""
  else
    echo "moving \"${Share%/}\""
    find "./$Share" -depth \( \( -type f ! -exec fuser -s {} \; \) -o \( -type d -empty \) \) -print \
      \( -exec rsync -i -dIWRpEAXogt --numeric-ids --inplace {} /mnt/user0/ \; -delete \) -o \( -type f -exec rm -f /mnt/user0/{} \; \)
  fi
done

rm /var/run/mover.pid
echo "mover finished"

 

We have this:

 

#!/bin/bash

# This is the 'mover' script used for moving files from the cache disk to the
# main array.  It is typically invoked via cron.

# After checking if it's valid for this script to run, we check each of the top-level
# directories (shares) on the cache disk: if the share is marked "cache-only" then we
# skip this directory; otherwise, we use the 'find' command to process the objects (files
# and directories) of that directory, moving them to the array.

# The script is set up so that hidden directories (i.e., directory names beginning with a '.'
# character) at the topmost level of the cache drive are also not moved.  This behavior can be
# turned off by uncommenting the following line:
# shopt -s dotglob

# Files at the top level of the cache disk are never moved to the array.

# The 'find' command generates a list of all files and directories on the cache disk.
# For each file, if the file is not "in use" by any process (as detected by 'fuser' command),
# then the file is copied to the array, and upon success, deleted from the cache disk.
# For each directory, if the directory is empty, then the directory is created on the array,
# and upon success, deleted from the cache disk.

# For each file or directory, we use 'rsync' to copy the file or directory to the array.
# We specify the proper options to rsync so that files and directories get copied to the
# array while preserving ownership, permissions, access times, and extended attributes (this
# is why we use rsync: a simple mv command will not preserve all metadata properly).

# If an error occurs in copying (or overwriting) a file from the cache disk to the array, the
# file on the array, if present, is deleted and the operation continues on to the next file.

# Only run script if cache disk enabled and in use
if [ ! -d /mnt/cache -o ! -d /mnt/user0 ]; then
  exit 0
fi

# If a previous invokation of this script is already running, exit
if [ -f /var/run/mover.pid ]; then
  if ps h `cat /var/run/mover.pid` | grep mover ; then
      echo "mover already running"
      exit 0
  fi
fi
echo $$ >/var/run/mover.pid
echo "mover started"

cd /mnt/cache
shopt -s nullglob
for Share in */ ; do
  if grep -qs 'shareUseCache="yes"' "/boot/config/shares/${Share%/}.cfg" ; then
    echo "moving \"${Share%/}\""
    find "./$Share" -depth \( \( -type f ! -exec fuser -s {} \; \) -o \( -type d -empty \) \) -print \
      \( -exec rsync -i -dIWRpEAXogt --numeric-ids --inplace {} /mnt/user0/ \; -delete \) -o \( -type f -exec rm -f /mnt/user0/{} \; \)
  else
    echo "skipping \"${Share%/}\""
  fi
done

rm /var/run/mover.pid
echo "mover finished"

[trurl]

I wanted to sleep on this before responding, and I don't see any downsides to this idea. So instead of skipping cache-only shares, mover would only move cache-yes shares. I think that would probably fix this problem.

 

We would still have people accidentally (or at least not explicitly) creating shares by specifying top level folders. But if they create a share on cache by specifying /mnt/cache/whatever, then they shouldn't be surprised that it doesn't get moved. And if they create a share on the array with /mnt/user/whatever, then they shouldn't be surprised that it won't use cache unless they configure it to.

 

And changing a cache-no (or not configured) share to a cache-yes share wouldn't require any additional cleanup. I think it would be a lot easier and make more sense to tell someone that they have to configure a share to use cache if they want it to, instead of telling them that when they created a top level folder on cache without configuring it, it got moved by default.

 

I like it!

 

[bonienl]

In combination with this suggestion, wouldn't it be better as well to change the default value for Use cache disk to Yes instead of No when creating a new user share AND a cache drive is present?