nethead25

Perhaps -- it looks very similar, and sorry @Neural42 for hijacking your thread. @JorgeBI am not disputing that Syncthing at some level the cause of the problem. I can't really practically switch to rsync at the moment given the volumes involved, but that's eventually my plan. I'm raising this only because I know Syncthing is a pretty common container to run on these. The issue is that I have two Unraid boxes (different config, one is Coffee Lake, the other is Ryzen). Both running the same version of the syncthing docker container. The arrays are actually mirroring each other (hence the syncthing use case) so the data is virtually identical. The drives are exactly the same model and array config. In other words, not a perfect control, but close enough to be relevant to the conversation. Using the same version of the container on the same data and share configuration, I do not observe the issue with the other box, but this server (the Ryzen one) immediately started crashing when it was upgraded to 7.0 and continued to do so after it was downgraded. Using this same Syncthing container and version that was working fine before. Meanwhile the other server was also upgraded and does not exhibit the problem at all. So I think it's the unfortunately something about this specific combination of factors that is causing the issue. The fact that it persists after a downgrade is also weird -- is there a setting, process or binary in the 7.0 upgrade process that would theoretically stick after a downgrade? It seems like that would be a good place to start, but I wouldn't know where to look. If it were something to do with a kernel bug with the AMD PCIe controller or something, I would have thought that reverting back would have solved it. Does the 7.0 upgrade change any persistent settings by default, or perform any actions on an array?

Hi @JorgeB, thanks for responding. That would make sense, except that the memory expansion is happening in the shfs process (see screenshots previously) that runs outside the container. The Docker container itself is only using 1GB of RAM, as you can see in the 7.0.1 screenshot. I have also capped the memory within the container and the amount consumed by shfs exceeds that. The memory consumption in shfs also persists after the docker service is stopped entirely.

This is still happening in 7.0.1. After boot, and before the Syncthing image is started, the server shows about consuming about 6% RAM usage. Within 5 minutes of starting the Syncthing container, system RAM balloons, increasing about 1% every 10 seconds or so until the growth slows around this point: Once the Docker image is stopped, the high system RAM usage persists until after a reboot.

Here's the df -h / output: Meanwhile shfs at the moment is consuming 70% of RAM & a bunch of CPU; current RAM consumption on the server is 91% out of 16GB. root 3875 89.4 70.2 11283308 11092196 ? Ssl 05:57 145:13 /usr/local/bin/shfs /mnt/user -disks 3 -o default_permissions,allow_other,noatime -o remember=0 Interestingly I have another server that is almost identically configured with similar RAM, shares, data volumes, Syncthing, and array setup. (though it is in a proxmox VM rather than bare metal). This server is still on 7.0 but shows 2.7% RAM usage for the same process and is rock solid. The other interesting observation is that the shfs process fluctuates in RAM usage -- it does sometimes go down a bit, so I'm not sure it's a classic memory leak per se.

Something is definitely going on here. I have had exactly the same behavior replicated on one of my servers (but interestingly not the other one). I'm getting a mix of just lockups (where I can ping the server but cannot access SSH or webUI) or the array crashing and shares disappearing until a reboot. 1. 6.12.x with Syncthing - no issues 2. Upgrade to 7.0.0 - begin having shfs memory blowouts any time there is heavy Syncthing activity 3. Revert back to 6.12.x - still having shfs memory blowouts Disabling Syncthing does fix the problem, however, it seems like something happens in the 7.0.0 upgrade that causes this issue to occur permanently on the array. I have tried to cap the RAM usage of the docker image and add a swapfile to the Unraid server, but ultimately this shfs memory leak or whatever we want to call it is a new issue.