[6.8.3] SHFS process consuming all memory over time

Wanted to add another diagnostics and screen shot of htop showing the memory usage.  Here you can see unraid running with Docker turned off, no plugins installed and no VMs.  Fresh USB drive also.  It is using 55GB of memory, the only thing I did was start the array.  To my knowledge nothing else is running.  Any help would be appreciated.

tower-diagnostics-20200402-2057.zip

I just wanted to report I am still doing daily restarts as SHFS still eats up all my memory.  If there is any other data I can provide let me know.  For the time being I have docker running so the kids can watch their movies while we are stuck at home.  

Still doing daily restarts, my monthly parity check just started but I will probably have to cancel it because it will not complete before SHFS uses all available ram.  I'm just thankful I bought the server with 192GB of ram, if I had something reasonable like 32 or 64 GB I would have to restart 2 or 3 times a day and that would just be a deal breaker.   

 

I will add that if I forget to restart the server, SHFS will drive my RAM usage to 100% then it eventually stops running and i guess the process kills itself, it no longer shows in htop and ram used drops back to normal.  Once this happens all my shares disappear.  

 

At this point I am trying to decide if I should just build a new server, switch away from unraid, or use a script to restart daily.

I don't know what causes your issue.

 

My main server is up and running for more than 32 days and memory consumption stays steady all the time.

 

It's one of these things that some users see on occasion (along with shfs consuming all CPU resources and wsdd consuming all CPU resources) that the majority of users do not see for some unknown reason.

1 hour ago, bonienl said:

I don't know what causes your issue.

 

My main server is up and running for more than 32 days and memory consumption stays steady all the time.

 

Mine use to do this, Would go for months solid as a rock.

 

1 hour ago, DarphBobo said:

It's one of these things that some users see on occasion (along with shfs consuming all CPU resources and wsdd consuming all CPU resources) that the majority of users do not see for some unknown reason.

 

I understand that, and to me that makes me think I have something hardware related not playing nice with unraid.  I haven't made and changes to the hardware but I wonder if something is failing.  My first thought was memory, but there are no errors in the log.  I tried taking out half the memory and restarting.  Same behavior, shfs starting eating memory.   Then I took that half out and replaced with the half I pulled before.  Again, same behavior.  In my mind that rules out a single stick of memory being the problem.  Of course I could have multiple sticks failing so its not a perfect test.  My other thought is if shfs has to do with my shares and disks, maybe it is something in the HBA card.  I have a LSI 9211-4I.  Could it possibly be firmware corruption?  Im know I'm throwing darts here but just brainstorming.

 

Last question, is there any way to see what is being written to memory?

 

Also, this week I think I may try the following just to see what happens, again throwing more darts but I'm working from home and have the time so why not.

 

1. swap my windows PC hardware into the server, reuse HBA, backplane, and all disks.

2. Order a new USB drive, maybe mine is on the way out so when I start the server something corrupted gets loaded?  Is that even possible?  

 

I may also start researching a new server build.  I keep reading about these Ryzen builds and how great they are.  My current dual E5-2670 server can handle everything I use it for easily but its loud, power hungry, and puts out a lot of heat.  

 

Thanks for any help and/or feedback!  

 

P.S.  Parity check is at 65% and ram utilization is only at 56%.  Maybe the parity check will finish before shfs crashes!

 

I thought I would provide an update.  I have narrowed down the source of what is triggering the shfs processes to consume all my memory.  Today a read a thread on the forums here explaining how shfs has to do with reading files in the different shares.  I decided to start turning off everything that would be accessing the shares to see if it would make a difference in shfs consuming the memory.  Up until this point I had only turned off docker and plugins which had no effect.  I went into the unraid settings and turned off all the network services. (AFP, NFS, SMB, FTP, and Wireguard VPN).  Reboot and boom, no shfs processes consuming memory.  I let it run like that for 30 minutes and restarted the server again and let it run again for 30 minutes just to make sure.  Then one by one, i turned those services on again.  It turns out SMB is my problem.  So I then turned on SMB but set all the options under SMB to no.  Everything is working fine now for a few hours and I can still browse my mapped network drives from my PC.  Tomorrow when I have time I will turn each of those settings (WDS, NetBIOS, and Enhanced macOS interoperability) back on to see which one causes my issues.

 

I have always had issues viewing my unraid shares on my PC and I know there is a great video out there from spaceinvaderone showing how to fix it I just havent taken the time to do it.  I just login with "NOBODY" each time it asks.  Im hoping my problem has something to do with that and SMB and the WSD setting or something.  

 

So keeps your fingers crossed that I have this figured out or at least know where to start trouble shooting.

On 5/26/2020 at 5:27 PM, user20C said:

I thought I would provide an update.  I have narrowed down the source of what is triggering the shfs processes to consume all my memory.  Today a read a thread on the forums here explaining how shfs has to do with reading files in the different shares.  I decided to start turning off everything that would be accessing the shares to see if it would make a difference in shfs consuming the memory.  Up until this point I had only turned off docker and plugins which had no effect.  I went into the unraid settings and turned off all the network services. (AFP, NFS, SMB, FTP, and Wireguard VPN).  Reboot and boom, no shfs processes consuming memory.  I let it run like that for 30 minutes and restarted the server again and let it run again for 30 minutes just to make sure.  Then one by one, i turned those services on again.  It turns out SMB is my problem.  So I then turned on SMB but set all the options under SMB to no.  Everything is working fine now for a few hours and I can still browse my mapped network drives from my PC.  Tomorrow when I have time I will turn each of those settings (WDS, NetBIOS, and Enhanced macOS interoperability) back on to see which one causes my issues.

 

I have always had issues viewing my unraid shares on my PC and I know there is a great video out there from spaceinvaderone showing how to fix it I just havent taken the time to do it.  I just login with "NOBODY" each time it asks.  Im hoping my problem has something to do with that and SMB and the WSD setting or something.  

 

So keeps your fingers crossed that I have this figured out or at least know where to start trouble shooting.

Hi - did you have a chance to identify what could cause the issue?

It was my windows 10 machine trying to communicate with unraid over SMB.  I don't know what actually fixed my problem.  I updated my backup software which was really out of date, updated windows, and then followed spaceinvaderone's youtube video on optimizing windows settings.  1 of those 3 things fixed my issue and the server has been doing great ever since.