Description:

The "default appdata storage location" is incorrectly set to reference the "user" share, this can cause fuse to crash under certain circumstances when starting a docker container, the resolution to this is to switch the default location from /mnt/user/appdata to /mnt/cache/appdata.

 

Whilst i understand not everybody has a cache drive and thus i understand the decision to set it to /mnt/user, i think the fact that setting it to use a fuse mounted share causes fuse to crash is not worth the risk and thus recommend that LT either removes this option or detects the presence of a cache drive and sets it to use the cache disk share instead.

 

How to reproduce:

1. Set the default appdata storage location to /mnt/user/appdata

2. start several docker containers that reference this path

3. watch out for random fuse crashes, signified by /mnt/user/* disappearing

 

Expected results:

Starting and stopping of docker containers does not crash fuse.

 

Actual results:

Starting and stopping of docker containers that reference /mnt/user/appdata can cause fuse to crash

 

Other information:

The docker im using to reproduce this is my new shiny docker image binhex-koel, this DOES include soft links to /config (which would reference /mnt/user/appdata), so the issue im seeing could well be linked with using a fuse share AND also soft linking from the container.

 

Link to post just showing the default location set for app data:-

http://lime-technology.com/forum/index.php?topic=51971.msg498936#msg498936

As far as I know the /mnt/user/appdata is now meant to be the default location regardless of whether you have a cache drive.  This became the default when support for hard links in User shares was supported.

 

It seems to me that the defect here is fuse crashing in the first place, and this is what should be fixed?

In my mind, this is something specific to your setup, as I've been using /mnt/user as a destination for Appdata backups since rc3 which is fundamentally the same as using /mnt/cache as the Appdata location

 

Sent from my LG-D852 using Tapatalk

 

 

In my mind, this is something specific to your setup, as I've been using /mnt/user as a destination for Appdata backups since rc3 which is fundamentally the same as using /mnt/cache as the Appdata location

 

Sent from my LG-D852 using Tapatalk

 

hi squid, it is entirely possible that its because i have a couple of docker images which do soft links to /config and the combination of fuser and soft linking from container causes fuser to crash, i dunno at this stage, all i i do know is that switching it from /mnt/user/appdata to /mnt/cache/appdata prevents fuser from crashing.

It seems to me that the defect here is fuser crashing in the first place, and this is what should be fixed?

 

This is the defect your right, but i believe the reason fuser crashes is due to the use of /mnt/user/appdata and soft linking, switching to /mnt/cache/appdata (in my testing so far) looks to fix it.

I'll try it out tonight with koel. Will it fail right away or is there more I have to do?

 

Sent from my LG-D852 using Tapatalk

 

I'll try it out tonight with koel. Will it fail right away or is there more I have to do?

 

Sent from my LG-D852 using Tapatalk

 

it was pretty consistent, i wouldnt say it ALWAYS blew up fuser, but more often than not it would. One thing you can do is let the docker container start, then stop it, then delete folder /config/mysql/database/ and start it again, this will force the container to do some intensive soft links whilst it rebuilds the database (system and koel), this seemed to trigger it for me.

 

thanks Squid, i appreciate you taking a look, it would be nice to know somebody else is seeing this too.

first thing i did when the option to set a default appdata folder was change to /mnt/cache

 

no end of problems were caused by /mnt/user and some dockers plain refusing to work because of fuse. i've seen no good reason to change my settings now.

I agree with Sparklyballs and binhex on this one, I have all my appdata at /mnt/cache/.appdata/ and have no plans to change either.

 

Sent from my LG-H815 using Tapatalk

 

 

first thing i did when the option to set a default appdata folder was change to /mnt/cache

 

no end of problems were caused by /mnt/user and some dockers plain refusing to work because of fuse. i've seen no good reason to change my settings now.

 

ah ha!, i have somebody else seeing issues around fuser and docker, that means we are now officially a crowd :-) *riot!!*

I'll try it out tonight with koel. Will it fail right away or is there more I have to do?

 

Sent from my LG-D852 using Tapatalk

 

it was pretty consistent, i wouldnt say it ALWAYS blew up fuser, but more often than not it would. One thing you can do is let the docker container start, then stop it, then delete folder /config/mysql/database/ and start it again, this will force the container to do some intensive soft links whilst it rebuilds the database (system and koel), this seemed to trigger it for me.

 

thanks Squid, i appreciate you taking a look, it would be nice to know somebody else is seeing this too.

 

Can you provide a syslog after the 'crash' please?  Do you know if it's hard links or soft links causing issues?  Also, it's "fuse" not "fuser", though actually it's "shfs", FUSE is the name of the linux technology used by "shfs".

first thing i did when the option to set a default appdata folder was change to /mnt/cache

 

no end of problems were caused by /mnt/user and some dockers plain refusing to work because of fuse. i've seen no good reason to change my settings now.

I'm not suggesting that, but you would have changed it prior to rc3.  After rc3 is when hardlinks started working.  And I've have zero problems with backups to user shares and will soon be deprecating the ability to backup to disk shares

 

Sent from my LG-D852 using Tapatalk

 

 

I'll try it out tonight with koel. Will it fail right away or is there more I have to do?

 

Sent from my LG-D852 using Tapatalk

 

it was pretty consistent, i wouldnt say it ALWAYS blew up fuser, but more often than not it would. One thing you can do is let the docker container start, then stop it, then delete folder /config/mysql/database/ and start it again, this will force the container to do some intensive soft links whilst it rebuilds the database (system and koel), this seemed to trigger it for me.

 

thanks Squid, i appreciate you taking a look, it would be nice to know somebody else is seeing this too.

 

Can you provide a syslog after the 'crash' please?  Do you know if it's hard links or soft links causing issues?  Also, it's "fuse" not "fuser", though actually it's "shfs", FUSE is the name of the linux technology used by "shfs".

It's soft links in my case, I will try and reproduce it tonight and post the log.

 

Sent from my SM-G900F using Tapatalk

 

 

i saw 6.2.1 just dropped, assumed this is a sign from the gods that i should update, reboot and get that syslog posted 

 

so it was insanely easy to replicate the issue:-

 

1. set koel container to use /mnt/user/appdata for /config volume mapping

2. start koel container

3. stop the koel container

4. delete /config/mysql/database/ folder

5. start koel container

6. try navigating to any fuse based share and watch it fail

 

if i then reboot the server, switch /config to use /mnt/cache/appdata instead, delete the partially created  /config/mysql/database/ and start koel then it successfully re-creates the database and starts correctly.

syslog.txt

i saw 6.2.1 just dropped, assumed this is a sign from the gods that i should update, reboot and get that syslog posted 

 

so it was insanely easy to replicate the issue:-

 

1. set koel container to use /mnt/user/appdata for /config volume mapping

2. start koel container

3. stop the koel container

4. delete /config/mysql/database/ folder

5. start koel container

6. try navigating to any fuse based share and watch it fail

 

if i then reboot the server, switch /config to use /mnt/cache/appdata instead, delete the partially created  /config/mysql/database/ and start koel then it successfully re-creates the database and starts correctly.

 

Hi binhex,

 

I just followed your instructions and I'm not having any issues.  Can you clarify what you mean by "try navigating to any fuse based share"?  Are you talking about over SMB, from within a shell session, within the container, or something else?

 

Also, we saw a segfault in your logs relating to Plex.  Is that the container variant?  Can you try to recreate this issue with Koel being the only container running on the system?

i saw 6.2.1 just dropped, assumed this is a sign from the gods that i should update, reboot and get that syslog posted 

 

so it was insanely easy to replicate the issue:-

 

1. set koel container to use /mnt/user/appdata for /config volume mapping

2. start koel container

3. stop the koel container

4. delete /config/mysql/database/ folder

5. start koel container

6. try navigating to any fuse based share and watch it fail

 

if i then reboot the server, switch /config to use /mnt/cache/appdata instead, delete the partially created  /config/mysql/database/ and start koel then it successfully re-creates the database and starts correctly.

 

Hi binhex,

 

I just followed your instructions and I'm not having any issues.  Can you clarify what you mean by "try navigating to any fuse based share"?  Are you talking about over SMB, from within a shell session, within the container, or something else?

 

Also, we saw a segfault in your logs relating to Plex.  Is that the container variant?  Can you try to recreate this issue with Koel being the only container running on the system?

 

hey jon, ok so to clarify i cannot connect to any fuse share via smb, also if i ssh in and look for /mnt/user it no longer exists. i did have several other docker containers running (production system), so i can turn them off and try and get you a cleaner syslog.

 

just a quick thought, could this be linked to filesystem perhaps?, just as a fyi my cache drive is using btrfs, array is all reiserfs.

i saw 6.2.1 just dropped, assumed this is a sign from the gods that i should update, reboot and get that syslog posted 

 

so it was insanely easy to replicate the issue:-

 

1. set koel container to use /mnt/user/appdata for /config volume mapping

2. start koel container

3. stop the koel container

4. delete /config/mysql/database/ folder

5. start koel container

6. try navigating to any fuse based share and watch it fail

 

if i then reboot the server, switch /config to use /mnt/cache/appdata instead, delete the partially created  /config/mysql/database/ and start koel then it successfully re-creates the database and starts correctly.

 

Hi binhex,

 

I just followed your instructions and I'm not having any issues.  Can you clarify what you mean by "try navigating to any fuse based share"?  Are you talking about over SMB, from within a shell session, within the container, or something else?

 

Also, we saw a segfault in your logs relating to Plex.  Is that the container variant?  Can you try to recreate this issue with Koel being the only container running on the system?

 

hey jon, ok so to clarify i cannot connect to any fuse share via smb, also if i ssh in and look for /mnt/user it no longer exists. i did have several other docker containers running (production system), so i can turn them off and try and get you a cleaner syslog.

 

just a quick thought, could this be linked to filesystem perhaps?, just as a fyi my cache drive is using btrfs, array is all reiserfs.

 

Drop the attached file (extra.cfg) into the 'config' directory on your flash and then stop/start array.  This will result in a huge amount of debug info from 'shfs' being recorded in your syslog.  For that reason, prob better to temp remove/disable all your other containers.  Next, make the problem happen and post log here or send to me: [email protected]

 

To restore normal operation, delete that file and stop/start the array again.

extra.cfg

i saw 6.2.1 just dropped, assumed this is a sign from the gods that i should update, reboot and get that syslog posted 

 

so it was insanely easy to replicate the issue:-

 

1. set koel container to use /mnt/user/appdata for /config volume mapping

2. start koel container

3. stop the koel container

4. delete /config/mysql/database/ folder

5. start koel container

6. try navigating to any fuse based share and watch it fail

 

if i then reboot the server, switch /config to use /mnt/cache/appdata instead, delete the partially created  /config/mysql/database/ and start koel then it successfully re-creates the database and starts correctly.

 

Hi binhex,

 

I just followed your instructions and I'm not having any issues.  Can you clarify what you mean by "try navigating to any fuse based share"?  Are you talking about over SMB, from within a shell session, within the container, or something else?

 

Also, we saw a segfault in your logs relating to Plex.  Is that the container variant?  Can you try to recreate this issue with Koel being the only container running on the system?

 

hey jon, ok so to clarify i cannot connect to any fuse share via smb, also if i ssh in and look for /mnt/user it no longer exists. i did have several other docker containers running (production system), so i can turn them off and try and get you a cleaner syslog.

 

just a quick thought, could this be linked to filesystem perhaps?, just as a fyi my cache drive is using btrfs, array is all reiserfs.

 

Drop the attached file (extra.cfg) into the 'config' directory on your flash and then stop/start array.  This will result in a huge amount of debug info from 'shfs' being recorded in your syslog.  For that reason, prob better to temp remove/disable all your other containers.  Next, make the problem happen and post log here or send to me: [email protected]

 

To restore normal operation, delete that file and stop/start the array again.

 

Hi tom, sorry for the delay, real life getting in the way. so i did as instructed, i put the extra.cfg in /config on flash, set all docker containers to not auto start on boot, rebooted host, then delete mysql database folder for koel and started it, first run resulted in it starting no probs, fuse was still holding up, then stopped the container, removed mysql database again and started it and boom it killed fuse. so i hope the attached log tells you a bit more as to what is going on, i had a quick look through it and didnt see anything too interesting, but hey im no expert on fuse (or fuser hehe).

 

syslog.zip

I don't see anything wrong in there. 

 

When it's in this state, can you browse user shares normally, eg, via file browser in webGui?  Please try that and if doesn't work there should be more stuff in the system log.

 

Also, is your appdata share entirely residing on the cache?

I don't see anything wrong in there. 

 

When it's in this state, can you browse user shares normally, eg, via file browser in webGui?  Please try that and if doesn't work there should be more stuff in the system log.

 

Also, is your appdata share entirely residing on the cache?

 

im afraid there are no shares to browse once its in this state (see attached screenshot), also see a dir listing of /mnt, note the fact it does show /mnt/user but its not accessible (second screenshot).

 

appdata is a cache only share.

 

attached latest syslog, just incase the steps above caused anything interesting to get written to the log.

syslog.zip

I don't see anything wrong in there. 

 

When it's in this state, can you browse user shares normally, eg, via file browser in webGui?  Please try that and if doesn't work there should be more stuff in the system log.

 

Also, is your appdata share entirely residing on the cache?

 

im afraid there are no shares to browse once its in this state (see attached screenshot), also see a dir listing of /mnt, note the fact it does show /mnt/user but its not accessible (second screenshot).

 

appdata is a cache only share.

 

attached latest syslog, just incase the steps above caused anything interesting to get written to the log.

 

Nothing wrong in that log.  In that extra.cfg file change the logging numeric value from 2 to 3, this will produce even more chatter 

 

Also is anything getting output on the console (physical monitor attached to motherboard)?