limetech Posted June 7, 2013 Share Posted June 7, 2013 Plex running locally to populate it's metadata database accessing files located on a 'user share' caused "transport endpoint not connected" errors. Refer to: http://lime-technology.com/forum/index.php?topic=27720.msg244725#msg244725 http://lime-technology.com/forum/index.php?topic=27720.msg245497#msg245497 http://lime-technology.com/forum/index.php?topic=27720.msg245510#msg245510 Link to comment
kontu Posted June 7, 2013 Share Posted June 7, 2013 My reference thread where I've been logging any and all data I've collected trying to resolve this for myself, including adding a change to logging to capture more info. Might have some extra information for you guys consolidated, thought I'd share. http://lime-technology.com/forum/index.php?topic=27752.0 Link to comment
axel Posted June 7, 2013 Share Posted June 7, 2013 Hi Guy's I have a work around that made my unRAID platform stable This solved the transport end point problem for me when running plex media server; Problem: I configured Plex Media server on unRAID to point to a fuse user share!! Simply reconfigure Plex Media Server on unRAID to point to a native real File system mount to avoid using fuse user shares; /mnt/disk1/plex/tmp/Library instead of /mnt/user/plex/tmp/Library /mnt/disk1/plex/tmp instead of /mnt/user/plex/tmp It looks like its *not* Plex Media Servers access to the Folders with large number of files in it that triggers the plex\fuse problem. Its Plex Media servers "own" access to its own local cache that triggers the problem when accessing the phototranscoder cache; plex/tmp/Library/Application Support/Plex Media Server/Cache/PhotoTranscoder/ It seems to be a problem when plex tries to write to the fuse user share "shfs_create: real_path" This is what happens just before the NFS\Fuse loss of access and Plex hang\crash, all this activity happened in 1 second in the plex transcoder cache folder (user share) 10 shfs/user:,shfs_create:,pid: 10 shfs/user:,shfs_create:,real_path: 16 shfs/user:,shfs_flush:,pid: 31 shfs/user:,shfs_getattr:,lookup: 64 shfs/user:,shfs_getattr:,pid: 9 shfs/user:,shfs_getxattr:,getxattr: 9 shfs/user:,shfs_getxattr:,pid: 10 shfs/user:,shfs_open:,pid: 16 shfs/user:,shfs_release:,pid: 6 shfs/user:,shfs_rename:,pid: 10 shfs/user:,shfs_truncate:,pid: 9 shfs/user:,shfs_write:,pid: I can no longer recreate the problem of fuse user shares going la la and killing the unRAID NFS mounts, when Plex iterates through new Photo Folders with large number of items in it ! Yea ! :-) All I have to do now is run fuse in debug and see why plex is able to killl the fuse user share with seemingly such little activity !!!! Link to comment
kontu Posted June 9, 2013 Share Posted June 9, 2013 Confirmed Axel's workaroudn on my setup as well - using a disk share instead of a user share for the plex library and temp stopped the problems in this thread. Link to comment
lanthade Posted June 16, 2013 Share Posted June 16, 2013 I'm gonna call it not solved. I was experiencing the problem until I removed my plex photos share. Unraid 5.0-rc15 Plex v0.9.7.22.511-4b5280f Plex Library directory is set to /mnt/user/AppData/plex Plex temp directory is set to /tmp My library is 4.1GB in size and I would rather have it on a user share so I don't have to worry about disk usage for an individual disk. So while axel's workaround may work, I wouldn't call it a fix or call the issue solved. Additionally I'm only using AFP and SMB on my setup, not sure if it matters. ~Erik Link to comment
limetech Posted June 17, 2013 Author Share Posted June 17, 2013 I'm gonna call it not solved. I was experiencing the problem until I removed my plex photos share. Unraid 5.0-rc15 Plex v0.9.7.22.511-4b5280f Plex Library directory is set to /mnt/user/AppData/plex Plex temp directory is set to /tmp My library is 4.1GB in size and I would rather have it on a user share so I don't have to worry about disk usage for an individual disk. So while axel's workaround may work, I wouldn't call it a fix or call the issue solved. Additionally I'm only using AFP and SMB on my setup, not sure if it matters. ~Erik Please "make it happen" and then post your system log. thanks Link to comment
lanthade Posted June 23, 2013 Share Posted June 23, 2013 Will do. Do I need to do anything to turn on the necessary logging or is it already on as part of the stock build? Link to comment
Frank1940 Posted June 23, 2013 Share Posted June 23, 2013 Will do. Do I need to do anything to turn on the necessary logging or is it already on as part of the stock build? Logging is a part of the stock build. Please read this for the exact procedure to get a syslog to attach your posting: http://lime-technology.com/forum/index.php?topic=9880.0 Link to comment
11rcombs Posted June 27, 2013 Share Posted June 27, 2013 I have this issue when using Plex on a user share (cannot move it off; the meta library is distributed and the media is well above the size of any one of my disks). There's nothing in my syslog when the issue happens apart from Transmission lines before it (normal) and various lines after from various plugins (esp. Transmission, which is log-happy) reporting "Transport endpoint is not connected". I don't even have a photos section, though the PhotoTranscoder is used for cover art for TV and Movies sections, so it could still be related. Stopping and starting the array does not solve the problem, though rebooting does. EDIT: A few extra facts: 1. I'm running 15a, but this has happened for quite some time. 2. It looks like the shfs process that runs /mnt/user dies, and for some reason can't be brought back online. 3. /mnt/user0 is unimpaired by all of this. Link to comment
limetech Posted June 27, 2013 Author Share Posted June 27, 2013 I have this issue when using Plex on a user share (cannot move it off; the meta library is distributed and the media is well above the size of any one of my disks). There's nothing in my syslog when the issue happens apart from Transmission lines before it (normal) and various lines after from various plugins (esp. Transmission, which is log-happy) reporting "Transport endpoint is not connected". I don't even have a photos section, though the PhotoTranscoder is used for cover art for TV and Movies sections, so it could still be related. Stopping and starting the array does not solve the problem, though rebooting does. EDIT: A few extra facts: 1. I'm running 15a, but this has happened for quite some time. 2. It looks like the shfs process that runs /mnt/user dies, and for some reason can't be brought back online. 3. /mnt/user0 is unimpaired by all of this. Can you make this happen "quickly" and "at will"? Link to comment
whiteatom Posted June 27, 2013 Share Posted June 27, 2013 How often does this happen and where are you seeing the error? I have very random (3-4 time a month) lock-ups in plex clients where the video stream just dies. I assumed it was some weird client or network error, but it could easily be a server issue... Never though about that. I have a 14 TB media library, so using a disk share isn't really an option Link to comment
subieriot Posted June 27, 2013 Share Posted June 27, 2013 Hello guys- I was having the same problem and this thread helped solved it. Just to be clear (because it wasn't clear to me at first), only the Library which contains the meta data directory="Application Support" needs to be on the physical disk rather than the user share. All of my media (music/movies) still resides on the user share. The only downside to this workaround that I can see is my Plex metadata (which does take a while to collect) isn't protected. It would be nice if this was still fixed, because something is still very wrong with fuse/plex. Take care! Link to comment
lanthade Posted June 28, 2013 Share Posted June 28, 2013 I was waiting till I'd finished backup of my data before causing system problems again. So that's done and now I'm back to "crashing" the system out. All I had to do was add more music to my library and it seems to have pushed it back over the edge, didn't even need to re-add the photos. System log is attached. Please let me know if you need more information and thank you for the help/work/etc. Oh, and this happens like clockwork every time I reboot the system. Not sure if it's happening when I attempt to access the data or on it's own, but it's pretty reliable. ~Erik unraid_log.txt Link to comment
WizADSL Posted June 28, 2013 Share Posted June 28, 2013 Hello guys- I was having the same problem and this thread helped solved it. Just to be clear (because it wasn't clear to me at first), only the Library which contains the meta data directory="Application Support" needs to be on the physical disk rather than the user share. All of my media (music/movies) still resides on the user share. The only downside to this workaround that I can see is my Plex metadata (which does take a while to collect) isn't protected. It would be nice if this was still fixed, because something is still very wrong with fuse/plex. Take care! Above you mention that not using a user share means that the data is unprotected, this is incorrect. /mnt/user and /mnt/disk1, /mnt/disk2 etc. all write to the array and are protected by parity, the only difference is that by using /mnt/disk1 you are placing your data specifically on disk1 and it could run out of space whereas with /mnt/user you get the benefit of using split levels and having unraid automatically spread your data out for you. Link to comment
limetech Posted June 28, 2013 Author Share Posted June 28, 2013 I was waiting till I'd finished backup of my data before causing system problems again. So that's done and now I'm back to "crashing" the system out. All I had to do was add more music to my library and it seems to have pushed it back over the edge, didn't even need to re-add the photos. System log is attached. Please let me know if you need more information and thank you for the help/work/etc. Oh, and this happens like clockwork every time I reboot the system. Not sure if it's happening when I attempt to access the data or on it's own, but it's pretty reliable. ~Erik After it 'crashes' please post the output of the "free" command. Link to comment
lanthade Posted June 28, 2013 Share Posted June 28, 2013 Just rebooted, took about 6 minutes for failure to occur this time. The only thing I was actively doing was browsing to a share via afp and ssh to system. root@Tower:/mnt# free total used free shared buffers cached Mem: 4072092 1096348 2975744 0 101772 777052 -/+ buffers/cache: 217524 3854568 Swap: 0 0 0 root@Tower:/mnt# Link to comment
lanthade Posted June 28, 2013 Share Posted June 28, 2013 Just had my system fail again - without the Plex plugin installed. I was ssh'ed into the system and the only thing that had changed recently was that I had started crashplan. Here's the result of the free command: total used free shared buffers cached Mem: 4072092 736892 3335200 0 93916 473340 -/+ buffers/cache: 169636 3902456 Swap: 0 0 0 Attached is my latest system log unraid_log.txt Link to comment
limetech Posted June 29, 2013 Author Share Posted June 29, 2013 I got plex installed and what do you know, I managed to reproduce this "Transport endpoint not connected". What's happening is the 'shfs' file system user space process is getting killed for some reason (and not an OOM situation). Bad news is it seems to take a long time to happen and I've only been able to make it happen once so far. So this bug might not get fixed before 5.0 'final' is released. Link to comment
subieriot Posted June 29, 2013 Share Posted June 29, 2013 I didn't start having the problem until I started using it more with a larger library. You should be able to recreate the issue more often if you add some music to your Plex library, do a scan which will pull the album covers, and try to browse your music (with the album pictures) in the plex web ui. I had over 200 albums and it started occurring everytime I browsed my music/album covers.. Thanks for your support on this! Link to comment
ajax3712 Posted July 3, 2013 Share Posted July 3, 2013 I used to be able to reproduce "at will", with Plex. I have a sizable library, with files number the 10s of thousands (audio, videos, etc.). It seemed to be related to the number of open files (ulimit issue). However, since changing the value (while I was still on rc12) in /boot/config/plugins/plexmediaserver/settings.ini to 40960 (yes, even higher than what I think later rc's used - 20000), I have not seen the error. ... #ulimit -n PLEX_MEDIA_SERVER_MAX_OPEN_FILES=40960 ... Link to comment
KillerCookie Posted July 3, 2013 Share Posted July 3, 2013 I had Plex running on RC12 and it immediatelly crashed while scanning my library with about 50 movies. I raised the ulimit and the inotify watchlimit to get Plex running well. Now on RC15 Plex is running well without any workarounds - i just thought you did some hidden magic and the bug is already fixed. Just wanted to give some feedback. Link to comment
11rcombs Posted July 4, 2013 Share Posted July 4, 2013 I have this issue when using Plex on a user share (cannot move it off; the meta library is distributed and the media is well above the size of any one of my disks). There's nothing in my syslog when the issue happens apart from Transmission lines before it (normal) and various lines after from various plugins (esp. Transmission, which is log-happy) reporting "Transport endpoint is not connected". I don't even have a photos section, though the PhotoTranscoder is used for cover art for TV and Movies sections, so it could still be related. Stopping and starting the array does not solve the problem, though rebooting does. EDIT: A few extra facts: 1. I'm running 15a, but this has happened for quite some time. 2. It looks like the shfs process that runs /mnt/user dies, and for some reason can't be brought back online. 3. /mnt/user0 is unimpaired by all of this. Can you make this happen "quickly" and "at will"? It doesn't seem to happen after a particular amount of time; I think refreshing a large amount of cover art may trigger it, but I'm not sure. Link to comment
11rcombs Posted July 4, 2013 Share Posted July 4, 2013 I got plex installed and what do you know, I managed to reproduce this "Transport endpoint not connected". What's happening is the 'shfs' file system user space process is getting killed for some reason (and not an OOM situation). Bad news is it seems to take a long time to happen and I've only been able to make it happen once so far. So this bug might not get fixed before 5.0 'final' is released. 1. You may be able to make the problem happen quicker by setting the PLEX_MEDIA_SERVER_MAX_OPEN_FILES to an unreasonably low setting (haven't tried this, but it seems logical that it'd trigger the issue faster since a higher setting seems to delay it). 2. I'm not sure exactly what, but something made me think the shfs process was getting SIGSEGV [segmentation fault] (maybe some weird log setting I was screwing with a while ago; I'm not sure). It could be wrong, and I could be thinking of something else; it's just something I seem to remember from trying to work around this a while back, but it could be worth looking into. Link to comment
Tophicles Posted July 4, 2013 Share Posted July 4, 2013 I got plex installed and what do you know, I managed to reproduce this "Transport endpoint not connected". What's happening is the 'shfs' file system user space process is getting killed for some reason (and not an OOM situation). Bad news is it seems to take a long time to happen and I've only been able to make it happen once so far. So this bug might not get fixed before 5.0 'final' is released. Hopefully this will lead you down a path that will correct this for those of us who do not use Plex and still get the errors Until then, I'm happy to stick to rc8a and try each subsequent rc. Thanks for all your hard work Tom! Link to comment
sacretagent Posted July 4, 2013 Share Posted July 4, 2013 ok guys, i have 2 unraid servers one for my movies which has about 6000 movies (according couchpotato 5000 movies but not everything is in there) and one for tv shows which has roughly 795 shows (224 active) | 37266/40888 episodes downloaded according to sickbeard i have on each server plex running and i don't run into this issue .... BUT i know i had a very weird issue when i moved the movies when i took the second server in service i couldn't import all movies in one time .... plex seemed to quit all the time ... i resolved that by importing by folder (in my case this was easy as i have my movies more or less separated by genre) which was going better but still had some issues getting thumbs and mediainfo from the bigger folders at that time i always thought it was a plex issue and didn't even bother mentioning it here .... but still it doesn't seem to be exactly the same as what you guys are seeing here .... only thing in plex i never really got working is the once a day auto update .... on the tv server it works as he is 24/7 on .. but the movie server who goes off at night it never really was updating daily... so i just do it manually need to move stuff in the right folders anyway so when i finished doing that i click the update library button edited for typo's Link to comment
Recommended Posts