1 hour ago, limetech said:

 

Yeah haven't looked in this topic for a while but working on this issue:

 

Will be back in here soon.

 

 

Appreciate the update.

Please test out 6.5.0-rc1.  We didn't find any obvious memory leak in the code.  But here are some observations.

 

Any time you reference files in user share file system FUSE allocates a structure called a node and also can allocate some heap memory using malloc().  The node contains info that describes the file and servers as a kind of interface between the FUSE kernel layer and user space layer.  These nodes contain a reference count which is incremented, eg, when a file is opened.  Normally these nodes are deallocated (along with any heap memory free()'ed) after returning request information to client and/or after file is closed.  There is an internal FUSE '"clean" thread which wakes up every 10 seconds that does the actual deallocation.  HOWEVER, the internal FUSE "inode number" of these nodes is what's used as the NFS file handle when accessing a user share via NFS.  For this reason we cannot permit these nodes to expire so quickly or else NFS clients will get "stale file handle". This is the purpose the "fuse_remember" tunable on the NFS Setings page.  The value there of 330 (when NFS is enabled) tells FUSE to keep these things in memory for min 5 1/2 minutes.  This was chosen because the typical NFS client side handle cache is 5 minutes.

 

So... If you have an application which is constantly referencing a huge number of files/directories within /mnt/user, the 'shfs' memory footprint is going to grow, seemingly unbounded, especially if you have NFS enabled and fuse_remember is set to 330.

 

In our testing we have noticed that we can do something like:

 

find /mnt/user/  >/dev/null

 

And watch shfs memory usage grow via htop.  Finally when the 'find' command exits it appears that 'shfs' memory is never deallocated.  But this is not actually the case.  If you type this:

 

echo 3 >/proc/sys/vm/drop_caches

 

It tells Linux to forcibly mark "available" pages as 'free" pages.  Normally Linux would do this on-demand.

 

Bottom line: we continue to work on this issue...

56 minutes ago, limetech said:

Please test out 6.5.0-rc1.  We didn't find any obvious memory leak in the code.  But here are some observations.

 

Any time you reference files in user share file system FUSE allocates a structure called a node and also can allocate some heap memory using malloc().  The node contains info that describes the file and servers as a kind of interface between the FUSE kernel layer and user space layer.  These nodes contain a reference count which is incremented, eg, when a file is opened.  Normally these nodes are deallocated (along with any heap memory free()'ed) after returning request information to client and/or after file is closed.  There is an internal FUSE '"clean" thread which wakes up every 10 seconds that does the actual deallocation.  HOWEVER, the internal FUSE "inode number" of these nodes is what's used as the NFS file handle when accessing a user share via NFS.  For this reason we cannot permit these nodes to expire so quickly or else NFS clients will get "stale file handle". This is the purpose the "fuse_remember" tunable on the NFS Setings page.  The value there of 330 (when NFS is enabled) tells FUSE to keep these things in memory for min 5 1/2 minutes.  This was chosen because the typical NFS client side handle cache is 5 minutes.

 

So... If you have an application which is constantly referencing a huge number of files/directories within /mnt/user, the 'shfs' memory footprint is going to grow, seemingly unbounded, especially if you have NFS enabled and fuse_remember is set to 330.

 

In our testing we have noticed that we can do something like:

 

find /mnt/user/  >/dev/null

 

And watch shfs memory usage grow via htop.  Finally when the 'find' command exits it appears that 'shfs' memory is never deallocated.  But this is not actually the case.  If you type this:

 

echo 3 >/proc/sys/vm/drop_caches

 

It tells Linux to forcibly mark "available" pages as 'free" pages.  Normally Linux would do this on-demand.

 

Bottom line: we continue to work on this issue...

 

I will try to do some test in the next days/weekend and get the debugging log (I couldn't the past week), a few tidbits:

 

- I found out about the issue because it couldn't free avaible ram (if it was marked as avaible) on demand (error allocating 2GB for a virtual machine in a 8GB server with just 3 containers started (emby, let'sencrypt and transmission)) and also If started (after reboots) and then "leaking" I was getting bizarre behavior in the VM (maybe the VM ram been overwrited, maybe a series of strange coincidences)

 

- In my test, with just 4-10 files seeding, was enough to appreciate the growth (I didn't try with less), just transmission working on its share (no other plugin/docker). So, not a huge number of files/directories.

 

- I'm guessing the NFS is a reference example but just in case, I have NFS disabled.

 

- The command I was using to check is "ps -e -orss=,args= | sort -b -k1,1n"

4 hours ago, limetech said:

Please test out 6.5.0-rc1.  We didn't find any obvious memory leak in the code.  But here are some observations.

 

Can't go further than "Loading bzroot.. ok" with blinking cursor on 6.5.0-rc1.

Replaced bz files from the "previous" folder and booting OK to 6.4.0

On 3/1/2018 at 7:55 AM, Jeronyson said:

 

Can't go further than "Loading bzroot.. ok" with blinking cursor on 6.5.0-rc1.

Replaced bz files from the "previous" folder and booting OK to 6.4.0

 

If you are still having an issue with 6.5.0-rc1 not booting, I would suggest you start a new thread in this section of the forum.  Your problem at this point is not a SHFS memory Leak.