So leave this thread so the people who are still having the issue can get the attention of developers instead of wasting time talking to you. I've been more than patient.

Is it so HARD to disable all plugins and dockers and restart your server? 

 

The devs wont do anything, if you cant provide ANYTHING.

In a previous reply I said that I disabled everything and it still happens. It seems like simple read/write via samba/cifs will still cause the issue. 

Edit: 
I provided full diagnostics. I have yet to see a dev request anything else. 

Edited February 14, 20188 yr by ffiarpg

Did you restart after disabling everything? And did you check everything wont start automatic on restart?

Edited February 14, 20188 yr by nuhll

Most probably this has something to do with the change introduced.

 

Quote

 

Improved shfs/mover (-rc1)

 

The LimeTech user share file system (shfs) has been improved in two areas.  First, we now make use of FUSE read_buf/write_buf methods.  This should result in significant throughput increases.  Second, the mover script/move program no longer uses rsync to move files/directories between the cache pool and the parity array.  Instead the move program invokes a new shfs ioctl() call.  This should result in complete preservation of all metadata including atime and mtime.

 

 

The use of ioctl() to move files from the cache indicates that shfs have received some threading support to perform a file copy. That would be a prime suspect - and the leak would then depend on amount of files moving between cache pool and array.

I had to restart just to enable safe mode disabling everything. And yes I already don't allow it to auto start the array on restart. I have to start the array to access via samba/cifs so obviously it wont increase until then. I can't really not have an array though as a long term fix.

7 minutes ago, nuhll said:

Is it so HARD to disable all plugins and dockers and restart your server? 

 

The devs wont do anything, if you cant provide ANYTHING.

Is it so HARD to realize that a leak is a leak and if this affects what unRAID can do then it's a big problem?

1 minute ago, ffiarpg said:

I had to restart just to enable safe mode disabling everything. And yes I already don't allow it to auto start the array on restart. I have to start the array to access via samba/cifs so obviously it wont increase until then. I can't really not have an array though as a long term fix.

 

So, just to understand it correct.

 

Youve disabled all plugins and dockers, and it didnt happend? You can ofc start the array. 

1 minute ago, pwm said:

Most probably this has something to do with the change introduced.

 

 

The use of ioctl() to move files from the cache indicates that shfs have received some threading support to perform a file copy. That would be a prime suspect - and the leak would then depend on amount of files moving between cache pool and array.

I agree with you, these changes seem very likely to be the culprit. In my case though, I have cache drives but I actually don't use the mover since I have previous issues with it. I have my "share" setup to NOT use cache and I have some other shares like "appdata" setup to ONLY use cache. 

Just now, nuhll said:

 

So, just to understand it correct.

 

Youve disabled all plugins and dockers, and it didnt happend? You can ofc start the array. 

I disabled all plugins and dockers, via restarting in safe mode, started the array with no dockers, read and wrote to my share drive via samba share and the memory usage still increased, just at a slower rate. My guess, fewer reads/writes. If I were to guess this issue is directly corrolated with reads and/or writes but I don't have any tools to troubleshoot the closed source proprietary shfs program.

Hm, okay, youve uploaded a diagnostic, was this while dockers/plugins are disabled, array startet and the memory leak happend? If so, then the devs should really look into this.

 

You may also sent it direct via email, so they see it, bc as far as i saw it on the forums, this got posted servival times, but no one provided (as far as i know) a diagnostic without plugins/dockers.

I sent a non anonymous diagnostic via the built in bug reporting but still no reply yet. At some point I will get sick of restarting and either try to hotswap the old shfs into my updated install or downgrade to an older version of unraid.

Maybe they are working on it. I sent one time a mail thru the built in system and got a very fast response.

What i read about ioctl is that it adds a "uncontrolled way of system calls" - what ever that means  so i guess, if its bugged, its not so easy to find out where the problem is.

Edited February 14, 20188 yr by nuhll

I will add some information.

 

The secondary system, the one i'm using to test, I had cache_dirs installed but disabled, I removed  4 or 5 plugins, and leave it all night with deluge container with 3 or 4 torrents active, the usage went up more or less the same rate, so, adding this to transmission behavior (transmission not guilty ) it looks like que issue is with heavy file access, It just went up when the torrent is active and stops when it's finished and not sharing. I didn't reboot the server (big mistake), I will do it now and test again. I will test too something to access files over the server with all plugins and dockers removed and probably same tramsission test avoiding user shares.

 

I did find out about this issue when I was trying to power up a VM and the server was out of memory to do it, in fact the bizarre behaviour I was having with that VM was the only reason I was looking, in normal conditions that VM is in autostart, then just to compare I checked the main server and there it was, 11Gb used at the moment. But I didnt notice problems on this server (dockers and VM on autostart, no reason to check). So its posible it's flying under the radar for a lot of people, adding that probably without a heavy usage on the user share (ex. torrent) no one will notice for a long time whitout reboots.

 

 

 

Edited February 16, 20188 yr by cferrero
Strike bad info

Same here.

 

I will post this here too, but I think tis better to continue on the other thread.

 

A few more test after  ….

 

I have been able to reproduce this with just transmission container, vm engine disabled and no plugins, docker image is loaded from a ssd outside of the array manually mounted before array start.

 

The test was:

All plugins removed, vm engine disabled, server rebooted to clean shfs from other test, ssd manually mounted, array started, transmission started, a few torrents seeding and/or downloading, in just a few minutes its clear that shfs memory is growing fast, but to be sure, waited 2+ hours  and check again to see the ram over 200+ Mb and not getting lower even after stopping transmission.

 

The exact same test with deluge instead transmission, it never went over 30Mb after 15h and loads of torrents.

 

So, recap, the ¿leak? looks triggered by transmision container (linuxserver.io version) and needs a reboot to clear it (if the container is stopped and deluge started the ram usage continues to grow).

 

What is the exact problem, no idea at the moment, as Jeronyson noted, it's an issue not present on unraid 6.3.5

 

Now that I have isolated the issue I will validate it on the main server, changing transmission to deluge while I think what more test to do.

On 2/17/2018 at 6:35 AM, cferrero said:

So, recap, the ¿leak? looks triggered by transmision container (linuxserver.io version) and needs a reboot to clear it (if the container is stopped and deluge started the ram usage continues to grow).

 

I haven't had a good chance to do troubleshooting yet. I'm also experiencing this issue, but it seems to be caused by the binhex-SABnzbd docker. I don't have Transmission or Deluge installed at this time, so it seems like more of a general symptom of heavy writing done by any docker.

I've been in the process of switching from transmission to qbittorrent over the last few months (transmission docker doesnt fully utilize gigabit for some reason) and finally turned off transmission completely. The memory still increases but much slower (instead of several % of my 16GB per day it is 0.1% every few days). I don't have hundreds of torrents in qbittorrent yet but it pushes out my required restarts from twice a week to likely every few months at my current rate. Definitely some issue with shfs and the more dockers we find that trigger it the easier it should be for devs to isolate the problem. Would be nice to get a developer response at some point. It would be one thing if this was open source but this huge issue being in a paid product with no reponse for weeks is pretty terrible IMO.

Edited February 22, 20188 yr by ffiarpg

2 minutes ago, mudsloth said:

 

I haven't had a good chance to do troubleshooting yet. I'm also experiencing this issue, but it seems to be caused by the binhex-SABnzbd docker. I don't have Transmission or Deluge installed at this time, so it seems like more of a general symptom of heavy writing done by any docker.

 

Well, not any heavy writing docker, I have deluge up in two servers, 5-6 days uptime, downloading new files each day, no issue, but two different containers, that's someting interesting.

6 minutes ago, ffiarpg said:

I've been in the process of switching from transmission to qbittorrent over the last few months (transmission docker doesnt fully utilize gigabit for some reason) and finally turned off transmission completely. The memory still increases but much slower (instead of several % of my 16GB per day it is 0.1% every few days). I don't have hundreds of torrents in qbittorrent yet but it pushes out my required restarts from twice a week to likely every few months at my current rate. Definitely some issue with shfs and the more dockers we find that trigger it the easier it should be for devs to isolate the problem. Would be nice to get a developer response at some point. It would be one thing if this was open source but this huge issue being in a paid product with no reponse for weeks is pretty terrible IMO.

 

the assigned ram can and will go up, the normal behaivor it's to drop it when it's no need. No developer will look just on "something is wrong", luckily I only needed a few days to isolate the issue on my systems, and now, mudsloth added a new piece of information, not just transmission. Now there is something that can be reapeated and some clues to allow people to look and probably find something.

1 minute ago, cferrero said:

No developer will look just on "something is wrong",

 

We provided far more than that even from post 1. shfs keeps consuming memory until it consumes all of it and then the kernel kills it and shares get dropped. That is enough for a developer to start digging. 

2 minutes ago, cferrero said:

allow people to look and probably find something.

 

Yes if by people you mean developers since shfs is closed source. Those developers have yet to comment, which is my point.

I have a suggestion. You're discussing this problem in a thread within a topic called Prerelease 6.4 Support. unRAID 6.4 has already been released so it's unlikely that the developers are keeping a close eye on this topic any longer. If you haven't already done so I recommend that you gather your evidence and file a defect report here.

 

2 minutes ago, tjb_altf4 said:

 

Thanks for adding that, I've thrown in a small comment saying I have the same issue. 

 

Still, no dev response in that thread either... 2 almost 3 weeks!

No response doesn't mean it's being ignored, just that there's nothing to add to the discussion. Wait for new releases and check the release notes.

On 2/23/2018 at 3:52 PM, ffiarpg said:

Still, no dev response in that thread either... 2 almost 3 weeks!

 

Yeah haven't looked in this topic for a while but working on this issue:

 

Will be back in here soon.