axel

Perfect that URL worked ..... the documentation page needs updating to the new URL... https://wiki.unraid.net/UnRAID_6/Upgrade_Instructions#Possible_upgrade_issues Thanks for all upoir help 🙂

Hi All, when I click updates on the plugins page ... I dont see any updates fro the OS running 6.2.4. When I manually try the URL https://raw.githubusercontent.com/limetech/unRAIDServer/master/unRAIDServer.plg It fails with the following log: "plugin: installing: https://raw.githubusercontent.com/limetech/unRAIDServer/master/unRAIDServer.plg plugin: downloading https://raw.githubusercontent.com/limetech/unRAIDServer/master/unRAIDServer.plg plugin: downloading: https://raw.githubusercontent.com/limetech/unRAIDServer/master/unRAIDServer.plg ... done plugin: downloading: https://s3.amazonaws.com/dnld.lime-technology.com/stable/unRAIDServer-6.5.3-x86_64.zip ... failed (Invalid URL / Server error response) plugin: wget: https://s3.amazonaws.com/dnld.lime-technology.com/stable/unRAIDServer-6.5.3-x86_64.zip download failure (Invalid URL / Server error response)" Its looking for a file: https://s3.amazonaws.com/dnld.lime-technology.com/stable/unRAIDServer-6.5.3-x86_64.zip And that gets access denied and causes the upgrade to fail ... so It looks like somebody has messed up granting access to that file from their S3 Servers. Thx Axel

This solved the transport end point problem for me when running plex media server; I configured Plex Media server on unRAID to point to a fuse user share!! Simply reconfigure it to point to a native real File system mount to avoid using fuse user shares; /mnt/disk1/plex/tmp/Library instead of /mnt/user/plex/tmp/Library /mnt/disk1/plex/tmp instead of /mnt/user/plex/tmp The problem seems to be triggered when Plex tries to access its transcoder cache via a fuse user share.

I'm wondering what ever made you think that there is a connection between the nfs stale file handles and the transport endpoint problems. To be fair ... I was thinking in my head about some plex forum posts trying to link the unRAID Plex media server crash with NFS fuse related issues and accessing large number of files. In Retrospect these *could* be 2 unrelated issues with fuse that result in loss of NFS Access; 1). Accessing user share folders with large numbers of files via NFS => fuse triggers stale_file_handles => NFS loss of access. 2). Accessing user share folders with large numbers of files (via Remote Plex Clients or local http Consol) That triggers plex media server running in unRAID user space to access the files locally => fuse goes la la and NFS mounts to the user shares become unavailable as a side effect. I can trigger the problem with the following load in 5min 58 sec where I believe that PID 15887 was plex media server 21178 shfs_getattr: ,PID: 15887 20206 shfs_write: ,PID: 19517 20206 shfs_getxattr: ,PID: 19517 18418 shfs_getxattr: ,PID: 15887 8900 shfs_read: ,PID: 15887 7292 shfs_release: ,PID: 0 5640 shfs_getattr: ,PID: 19490 4914 shfs_getattr: ,PID: 19344 4560 shfs_open: ,PID: 15887 4555 shfs_flush: ,PID: 15887 3517 shfs_getattr: ,PID: 19350 3154 shfs_getattr: ,PID: 15942 2376 shfs_read: ,PID: 19490 2284 shfs_open: ,PID: 19490 2284 shfs_flush: ,PID: 19490 2208 shfs_readdir: ,PID: 15887 812 shfs_readdir: ,PID: 19490 I have emailed the log file to Tom (thanks for your help m8 its appreciated). I'll download the source for fuse and start to educate myself about it; One thing I have spotted is that just before fuse appears to barf .. there is a lot of the following syslog entries; Jun 7 11:52:38 unRAID shfs/user: assign_disk_high_water: disk1 size 122092910 free 68001250 Jun 7 11:52:38 unRAID shfs/user: assign_disk_high_water: disk2 size 122092910 free 93972738 Jun 7 11:52:38 unRAID shfs/user: assign_disk: disk_path: /mnt/disk1 ##################################### UPDATE: SOLVED ? It looks like its *not* Plex Media Servers access to the Folders with large number of files in it that triggers the plex\fuse problem. Its Plex Media servers "own" access to its own local cache that triggers the problem when accessing the phototranscoder cache; plex/tmp/Library/Application Support/Plex Media Server/Cache/PhotoTranscoder/ It seems to be a problem when plex tries to write to the fuse user share "shfs_create: real_path" This is what happens just before the NFS\Fuse loss of access and Plex hang\crash, all this activity happened in 1 second in the plex transcoder cache folder (user share) 10 shfs/user:,shfs_create:,pid: 10 shfs/user:,shfs_create:,real_path: 16 shfs/user:,shfs_flush:,pid: 31 shfs/user:,shfs_getattr:,lookup: 64 shfs/user:,shfs_getattr:,pid: 9 shfs/user:,shfs_getxattr:,getxattr: 9 shfs/user:,shfs_getxattr:,pid: 10 shfs/user:,shfs_open:,pid: 16 shfs/user:,shfs_release:,pid: 6 shfs/user:,shfs_rename:,pid: 10 shfs/user:,shfs_truncate:,pid: 9 shfs/user:,shfs_write:,pid: >> When I configure plex media server's library and temp folders at the native disk mount /mnt/disk1/plex/tmp/Library instead of /mnt/user/plex/tmp/Library /mnt/disk1/plex/tmp instead of /mnt/user/plex/tmp I can no longer recreate the problem of fuse user shares going la la and killing the NFS mounts ! Yea :-) All I have to do now is run fuse in debug and see why plex is able to killl the user share with seemingly such little activity !!!!

Guys FYI on the NFS Stale handles issue .... I think there is a tangent going on here ..... most of the "emhttp: get_filesystem_status: statfs: /mnt/user/ftp Transport endpoint is not connected" error messages are reported by *plex* media center plugin users! The problem occurs whether you run NFS or not! The problem is simple to recreate at will. 1). Unraid 5.0-rc13 2). PlexMediaServer-0.9.7.22.511-4b5280f-unRAID.txz 3). Create a photo album folder on a user share 4). dump about 100 photos in the user share folder 5). launch plex media web console and add the user share photo folder to your photo library and then watch while it index's\creates meta data. 6) tails syslog and just wait for fuse to go bonkers. basically rc 13 is a lot better than 12 I did upgraded my HP Micro server from 2GB to 4GB of ram to see if it helped (it didn’t). also all of the other suggestions i). /boot/config/extra.cfg ii). oom_score_adj = -1000 for the emhttp pid did not make a blind bit of difference! This is driving me nuts and stopping me taking this POC and buying Pro version + cache drive to replace my Media Center. Its also so easy to recreate ... just install plex and adds some music photos. happy to do any debugging with what little spare time I have. It must be something to do with fuse and a spike of files handles that are created when you add large number of files to the library. I don’t know why this is just with photo's ..... music does not seem to be a problem. Only difference ? a). Size of files. b). number of files in a folder (more in photo folders).

See the other post about transport errors ..... FYI Disabling NFS does not make any difference in my case.

Right as well as increasing the max number of files Samba can open as above... I also; 1). Completely removed the Plex Media Server package. 2). Rebooted. 3). Reinstalled the Plex Media Server package. 4). Deleted all my media Libraries form Plex Media. 5). Re Added my media folders back into Plex one by one and let them rebuild. i) Music Folder => OK ii). Movies Folder => OK. iii). Home Videos => Transport Errors iv). Photos Folder => Transport Errors Free memory steadily decreases over this time frame as Plex adds more files to the Library, and it looks like memory used by Linux for cache is not being released for other apps. Kill off Plex and Free memory returns back to before Plex was started. Plex logs do not indicate anything unusual up until it simply looses access to the user folder. At the same time in the syslog you get the usual; emhttp: get_filesystem_status: statfs: /mnt/user/ftp Transport endpoint is not connected To progress I suspect I am going to have to see if I can put the FUSE user space file system in debug mode to figure out what's going on. It looks like a resource management issue that is triggered over time, probably memory (e.g. why is cached memory not being freed up for apps ?). The question is .. which component is not doing its job!

To reply back to limetech .... I have no idea where the 20000 value came from for open files .... This latest release of unRaid 5.0-rc12a is unusable AFAIC, I only use unRaid for Plex media. This problem can be created at will for me by simply starting PlexMedia via the WebGui, and launching the PlexWeb Management GUI and simply selecting my music library. It triggers the Transport endpoint problem immediately. This should not happen on a system designed to be a file server! I suspect that the Plex Web GUI gui would need to display the thumbnails of the music albums it may be opening lots of files. Does Lime stress test samba access in their product before releasing? or is what Plex is doing unreasonable ? I can not recreate the issue from other clients (e.g. LG TV) and seeing as I did not have issues with earlier releases unRAID and the same music database it points towards a shoddy release from Lime. My Music library has 7793 MP3's but you expect that Plex will be opening jpg's for the music albums as well. So if you double that (worst case scenario + other Plex clients accessing my shares) => you are looking at ~ 15k max possible file items open. This is suspiciously close to the samba ulimit setting "testparm -sv | egrep open" Output "max open files = 16404" setting ulimit to 20,000 and then 40,000 did not make any difference !! (I am running a HP Microserver with 2GB DDR3 Memory single channel). Also setting the oom_score values to -1000 does not fix this issue either. What does appear to have delayed the time the problem takes to occur after clicking on the Music Folder in the Management Web GUI of Plex is simply increasing the samba max open files param ! To do this create a file called: /boot/config/smb-extra.conf And add this text; [global] max open files = 32808 >> I get more icons on the web page before the Transport errors occur .......... Still digging into this issue ..... its driving me NUTS!! :-0