I upgraded to 7.0.0 last week. It was quick and painless. However, I've noticed high RAM usage since then. Specifically it looks like shfs is eating up a lot -- almost 50% of my RAM (40 GB total in the system). It never used anything like that before the upgrade.

 

root 10093 107 48.8 20396412 19922852 ? Ssl 06:39 119:10 /usr/libexec/unraid/shfs /mnt/user -disks 63 -o default_permissions,allow_other,noatime -o remember=0

 

Anyone seen this before and have ideas about how to fix it? My system becomes unresponsive over time (which seems like a memory leak), requiring a hard reboot.

Attach Diagnostics to your NEXT post in this thread.

On 1/20/2025 at 8:59 AM, trurl said:

Attach Diagnostics to your NEXT post in this thread.

Edit: Removed diagnostics as they ended up not being important for this particular issue.

 

Edited March 26, 20251 yr by Neural42

Just for kicks, I downgraded back to 6.12.14. I'm still getting high RAM utilization from shfs (currently 54.3% of my 40 GB). This didn't happen before the upgrade to 7.0.0, so something weird is happening. I'm hoping I don't have to do a fresh install but that might be what's needed.

 

root     11030  111 54.3 22537388 22176676 ?   Ssl  Jan20 809:34 /usr/local/bin/shfs /mnt/user -disks 63 -o default_permissions,allow_other,noatime -o remember=0

 

Edited January 21, 20251 yr by Neural42
Fixed word for readability.

Some probably unrelated things I noticed in your diagnostics.

 

appdata, domains, and system shares have files on the array. Ideally, those shares would have all files on a pool and none on the array, so Dockers/VMs will perform better, and so array disks can spin down, since these files are always open.

 

Why do you have 150G docker.img? Has usage been growing? Making it larger won't fix that. The usual cause of docker.img growing is an application writing to a path that isn't mapped.

 

You were only using 12G of that 150G when diagnostics were done, so you can probably reduce that to the default 20G and get those shares moved while you're at it.

 

Another way dockers can be misconfigured, which would cause them to write to RAM, is a host path that isn't actually storage. That would write into rootfs, but I don't see any evidence that is happening.

 

I noticed a dump at the end of syslog in those diagnostics you posted. Don't know the cause. Are you still getting that after downgrade?

1 hour ago, Neural42 said:

I'm hoping I don't have to do a fresh install but that might be what's needed.

Each reboot is a fresh install. Unraid is installed fresh into RAM at each boot, from the archives on flash, and it runs completely in RAM. Think of it as firmware.

 

Those archives are not changed in any way during usage. They are exactly as they were from the initial download of whatever version you are running.

 

Your configuration (everything set in the webUI) is stored and updated in the config folder on the flash drive so your configuration can be reapplied at boot.

Thank you for the response about the other issues. I'm working on fixing the appdata, domains, and system issues. The files are moved now.

 

Thanks for the reminder about my docker.img size! I was using a neuroscience research-specific docker image that was large and really needed the high I/O of the store pool so I needed a large vDisk, but I've stopped using that on this machine and have now reverted my vDisk size to the default.

 

As for a fresh install, I meant more in the sense of starting from scratch with 7.0 and not import any configurations other than making sure to set up the array as it currently is. Then I'd install containers one by one to see what happens.

 

I have a hunch that it might be related to Syncthing, but that could be an incorrect hunch. I started using that recently and maybe it's related to having to copy tens of millions of files (these are some of my neuroscience data) to another device. Syncthing uses its own RAM, but maybe because of the number of files, shfs is also spiking up in usage??? Syncthing is the only other recent change other than the OS upgrade.

 

Thank you again for your help so far! I appreciate it.

46 minutes ago, Neural42 said:

As for a fresh install, I meant more in the sense of starting from scratch with 7.0 and not import any configurations other than making sure to set up the array as it currently is. Then I'd install containers one by one to see what happens.

Take a look at the Dynamix Factory Reset plugin. 

47 minutes ago, Neural42 said:

I'm working on fixing the appdata, domains, and system issues. The files are moved now.

You can see how much of each disk or pool is used by each share by clicking Compute... for the share on the User Shares page. 

I'm not sure what caused the issue but I'm still going with Syncthing as the most likely culprit. I deleted my Docker vDisk, upgraded to 7.0 and am just using my most needed containers (Jellyfin and Plex) before adding more. Right now I'm sitting at 2.66 GiB of system RAM usage, which is way less than before where system was in the 25 GiB range. I'm going to mark this as solved for now.

 

Edit: I checked in the morning and am at 2.68 GiB of system RAM used, which is excellent.

 

2nd Edit: Still at only 3.19 GiB of system RAM with more containers running and an rsync of millions of files running. I'm almost certain now the problem was due to a Syncthing issue.

 

3rd edit: Now it's sitting at 1.5 GiB of system RAM used after more days. Syncthing has not been reinstalled and there have not been any RAM issues.

Edited January 23, 20251 yr by Neural42
Adding more info

Something is definitely going on here. I have had exactly the same behavior replicated on one of my servers (but interestingly not the other one). I'm getting a mix of just lockups (where I can ping the server but cannot access SSH or webUI) or the array crashing and shares disappearing until a reboot.

 

1. 6.12.x with Syncthing - no issues 

2. Upgrade to 7.0.0 - begin having shfs memory blowouts any time there is heavy Syncthing activity

3. Revert back to 6.12.x - still having shfs memory blowouts

 

Disabling Syncthing does fix the problem, however, it seems like something happens in the 7.0.0 upgrade that causes this issue to occur permanently on the array.

 

I have tried to cap the RAM usage of the docker image and add a swapfile to the Unraid server, but ultimately this shfs memory leak or whatever we want to call it is a new issue.

2 hours ago, nethead25 said:

Something is definitely going on here. I have had exactly the same behavior replicated on one of my servers (but interestingly not the other one). I'm getting a mix of just lockups (where I can ping the server but cannot access SSH or webUI) or the array crashing and shares disappearing until a reboot.

 

1. 6.12.x with Syncthing - no issues 

2. Upgrade to 7.0.0 - begin having shfs memory blowouts any time there is heavy Syncthing activity

3. Revert back to 6.12.x - still having shfs memory blowouts

 

Disabling Syncthing does fix the problem, however, it seems like something happens in the 7.0.0 upgrade that causes this issue to occur permanently on the array.

 

I have tried to cap the RAM usage of the docker image and add a swapfile to the Unraid server, but ultimately this shfs memory leak or whatever we want to call it is a new issue.

I'm glad I'm not the only one seeing this. What you wrote is exactly what happened on my server. That means it is most likely some odd issue with Syncthing and unRAID when upgrading to 7.

Edited February 24, 20251 yr by Neural42

When you notice the problem, what do you get from command line with this?

df -h /

 

Here's the df -h / output:

 

 

Meanwhile shfs at the moment is consuming 70% of RAM & a bunch of CPU; current RAM consumption on the server is 91% out of 16GB.

root 3875 89.4 70.2 11283308 11092196 ? Ssl 05:57 145:13 /usr/local/bin/shfs /mnt/user -disks 3 -o default_permissions,allow_other,noatime -o remember=0

 

Interestingly I have another server that is almost identically configured with similar RAM, shares, data volumes, Syncthing, and array setup. (though it is in a proxmox VM rather than bare metal). This server is still on 7.0 but shows 2.7% RAM usage for the same process and is rock solid. 

 

The other interesting observation is that the shfs process fluctuates in RAM usage -- it does sometimes go down a bit, so I'm not sure it's a classic memory leak per se. 

 

Edited February 24, 20251 yr by nethead25

This is still happening in 7.0.1. 

After boot, and before the Syncthing image is started, the server shows about  consuming about 6% RAM usage.

 

Within 5 minutes of starting the Syncthing container, system RAM balloons, increasing about 1% every 10 seconds or so until the growth slows around this point:

 

Once the Docker image is stopped, the high system RAM usage persists until after a reboot. 

If it only happens when running Syncthing, there may be a memory leak with that container.

Hi @JorgeB, thanks for responding. That would make sense, except that the memory expansion is happening in the shfs process (see screenshots previously) that runs outside the container. The Docker container itself is only using 1GB of RAM, as you can see in the 7.0.1 screenshot. I have also capped the memory within the container and the amount consumed by shfs exceeds that.

 

The memory consumption in shfs also persists after the docker service is stopped entirely.

Edited February 27, 20251 yr by nethead25

Though technically that should not happen, containers have been known to cause issues outside them, even crash servers, but retest using rsync for example, to sync the data, instead of the container, and see if you observe the same.

I wonder if it's related to this issue? No Syncthing in that thread, but a similar issue of SHFS consuming more and more RAM given certain circumstances.

 

Maybe it's a side effect of how both Syncthing and Unraid manage file I/O and memory allocation under heavy load.

Edited February 27, 20251 yr by Neural42

Perhaps -- it looks very similar, and sorry @Neural42 for hijacking your thread. 

 

@JorgeBI am not disputing that Syncthing at some level the cause of the problem. I can't really practically switch to rsync at the moment given the volumes involved, but that's eventually my plan. I'm raising this only because I know Syncthing is a pretty common container to run on these.

 

The issue is that I have two Unraid boxes (different config, one is Coffee Lake, the other is Ryzen). Both running the same version of the syncthing docker container. The arrays are actually mirroring each other (hence the syncthing use case) so the data is virtually identical. The drives are exactly the same model and array config. In other words, not a perfect control, but close enough to be relevant to the conversation.

 

Using the same version of the container on the same data and share configuration, I do not observe the issue with the other box, but this server (the Ryzen one) immediately started crashing when it was upgraded to 7.0 and continued to do so after it was downgraded. Using this same Syncthing container and version that was working fine before. Meanwhile the other server was also upgraded and does not exhibit the problem at all.

 

So I think it's the unfortunately something about this specific combination of factors that is causing the issue. The fact that it persists after a downgrade is also weird -- is there a setting, process or binary in the 7.0 upgrade process that would theoretically stick after a downgrade? It seems like that would be a good place to start, but I wouldn't know where to look. If it were something to do with a kernel bug with the AMD PCIe controller or something, I would have thought that reverting back would have solved it. Does the 7.0 upgrade change any persistent settings by default, or perform any actions on an array?

 

 

31 minutes ago, nethead25 said:

Perhaps -- it looks very similar, and sorry @Neural42 for hijacking your thread.

No worries! I'm hoping some issue is figured out with this. I want to add that my setup is with a UGREEN NAS DXP 6800 Pro, which has a 12th gen i5 processor so it's not likely an AMD-specific issue.

Edited February 27, 20251 yr by Neural42

40 minutes ago, nethead25 said:

The fact that it persists after a downgrade is also weird -- is there a setting, process or binary in the 7.0 upgrade process that would theoretically stick after a downgrade?

Nope, the OS loads to RAM every boot from the flash drive, so also pointing to the issue not being Unraid, or downgrading should have fixed it.

 

 

same issue. disabled syncthing and its rebooting. will report back and follow the discussion for a solution.

On 3/25/2025 at 7:20 PM, BelgarionNL said:

same issue. disabled syncthing and its rebooting. will report back and follow the discussion for a solution.

Did the issue return just by disabling Syncthing and rebooting? I'm checking because that didn't solve the issue for me. Only deleting Syncthing and my Docker vDisk and then not reinstalling Syncthing solved it.

no synching is not an issue for me. the UI still hangs after 4 days where I have to reboot

I will try deleting syncthing and vdisk

Edited March 28, 20251 yr by BelgarionNL