[[SOLVED] Plex - Docker Stability issues (Ubiquiti Edgerouter networking related!)]

Final Update Aug 26 - This is all due to Plex and their new Network code that was introduced after 1.13.2. If you're having server connectivity/stability problems, try backing out to this version.

 

Update (after wasting 3 evenings on this, turns out it's a Ubiquit problem)!!!!! (PFsense may just become my new firewall)

 

It's a known problem with newer versions of Ubiquiti firmware. It will be addressed in upcoming releases until then, disable HWNAT hardware offloading (for a significant performance hit on fast internet connections)

 

https://community.ubnt.com/t5/EdgeRouter/ER-L-Plex-I-HATE-THIS-ROUTER/td-p/2305201/page/2

 

 

Update:

 

The Plex server will be intermittently up for about 40-50 seconds the second this message hits the log, the server is unresponsive:

 

Aug 05, 2018 14:48:07.584 [0x1540cbdfe700] DEBUG - MyPlex: We appear to have regained Internet connectivity.

 

It comes online, times-out and does it all over again.

 

 

I editied the /etc/dnsmasq.conf file in the router, and I had it workign flawlessly for a few hours... Not realizing the configuration must be set through CLI not through the file. Rebooted the router and working configs were lost. (Running config is not saved in backup)

 

NOTE: IF YOU DECIDE TO GO WITH PREVENTING DNS-REBIND ATTACKS ON YOUR UBIQUITI EDGEROUTER YOU *MUST* DO THE FOLLOWING IN THE CLI OF YOUR EDGEROUTER:

 

configure

set service dns forwarding options rebind-domain-ok=/plex.direct/

commit

save

exit

 

Citing this as reference: https://community.ubnt.com/t5/EdgeRouter/need-to-add-a-dns-bounce/td-p/1580827

 

Otherwise, expect your server to yo-yo constantly.

 

<Original Post begins here>

I'm having major stability (and sanity) issues at the moment getting unRAID to run Plex in my environment.

 

Full disclosure:

1) I've recently switch ISP's (I'm hoping this isn't the issue, but seeing things like 443s times for resolving an IP address makes me wonder)

2) I have been trying setting up an internal local domain and as such have my primary DNS running on a Synology DNS server, the secondary DNS server is the router itself

3) There is a third Plex Server (different Plex account) run in my environment under ESXi, it seems to be okay, it's user hasn't complained about any issues this week.

4) Apparently there are 8 instances in Plex server listings when my friends try and connect to the server. I've been dilligent about Deleting the servers under devices after I do a rebuild.

 

Symptoms:

unRAID Plex docker server takes about 2-5 minutes to come online, it shows up, shows library content, then seems to disappear. I'm unable to do anything more with it. The Plex docker simply goes offline. I can't see (but I probably don't know what I'm looking for) any obvious crashes.

I've tried this with at least four different docker images, binhex's (both of them), Linixserver.io's and the native Unraid. All unstable.

All other dockers seem fine. However, Tautulli seems to have issues sometimes with log files. 

Trakt plugin seems to make it worse. I've only been trying to install it after the Plex server seems stable.

I've tried to keep the AppData permissions to nobody:users 

 

Other times, it will stay up all night and then go offline at a moment's notice.

 

Machine #1 Bellerophon (log attached)

Lenovo P410 ThinkStation

E5-1620V4

32GB Cruiser Fit

64GB DDR4 ECC RAM

2 * 256 SATA SSD's cache

1 * 512 SATA SSD, download/transcode directory

LSI Raid controller AT Mode

4TB * 2 Parity drives and 9 Drives of various sizes with an overall data size of 26TB.

 

Machine #2 Chimera (no log)

 

Lenovo ThinkStation P700

128GB DDR4 ECC

LSI RAID controller (native Lenovo RAID firmware, no IT Mode firmware available, as a result, drives can never be spun down) in JBOD mode.

7 * 6TB WD RED's

(375W @ idle.

I have performed:

1) re-install of all dockers after removing 20GB Docker file

2) Multiple re-installs of Plex docker, after removing the appdata folder.

 

The issues appeared last week after running this server for 2-3 months? (Trying to decommission Chimera as it's a MASSIVE power beast) I thought I would fire up Chimera (the old one) and it's also unstable after being offline for so long.

 

I love unRAID and especially the docker support. It's been amazing, but clearly, I've done something really stupid. 

 

Any help would be appreciated.

 

I see this message about "regained internet connectivity" a lot.

Aug 04, 2018 10:34:03.120 [0x151e51fff700] DEBUG - EventSource: Resolving 184.105.148.108 port 443s
Aug 04, 2018 10:34:03.120 [0x151e51fff700] DEBUG - EventSource: Resolved 184.105.148.108 to 184.105.148.108
Aug 04, 2018 10:34:03.120 [0x151e51dfe700] DEBUG - EventSource: Resolving 184.105.148.108 port 443s
Aug 04, 2018 10:34:03.120 [0x151e51dfe700] DEBUG - EventSource: Resolved 184.105.148.108 to 184.105.148.108
Aug 04, 2018 10:34:03.200 [0x151e51fff700] DEBUG - EventSource: Connected in 55 ms.
Aug 04, 2018 10:34:03.200 [0x151e51dfe700] DEBUG - EventSource: Connected in 55 ms.
Aug 04, 2018 10:34:03.200 [0x151e51dfe700] DEBUG - EventSource: Wrote data, reading reply.
Aug 04, 2018 10:34:03.200 [0x151e51fff700] DEBUG - EventSource: Wrote data, reading reply.
Aug 04, 2018 10:34:03.754 [0x151e51fff700] DEBUG - EventSource: Read HTTP reply header.
Aug 04, 2018 10:34:03.754 [0x151e51fff700] DEBUG - MyPlex: We appear to have regained Internet connectivity.
Aug 04, 2018 10:34:03.755 [0x151e51dfe700] DEBUG - EventSource: Read HTTP reply header.
Aug 04, 2018 10:34:03.755 [0x151e51dfe700] DEBUG - MyPlex: We appear to have regained Internet connectivity.
Aug 04, 2018 10:34:07.996 [0x151e46bff700] DEBUG - NetworkServiceBrowser: SSDP departed after not being seen for 21.990171 seconds: 172.16.0.42

 

If I have to sacrifice a rubber chicken or perform allegiance to a different hockey team, I'd be game at this point.

 

Thanks for any assistance!

 

 

 

bellerophon-diagnostics-20180804-1039.zip

[[Support] binhex - DelugeVPN]

ISSUE IDENTIFIED: Ubiquiti EdgeRouter ER-X is causing the issue. 

 

UPDATE: When I remove ER-X and substitute a different router, traffic is fine. Unknown what is causing the issue at this time. (I had tried two ER-X with identical configuration)

 

 

I'm having a bizarre network issue and while it's related to this, I'm not sure what's causing it.

 

I have a 150/15 Mbps Down/Up internet link and when I run Speedtest from my cable provider it works just fine. (180/17 Mbps usually)

 

If I have *ANY* torrent running in DelugeVPN container my ping spikes to 120+ from 16, and performance from everything else on the network plummets and the speed goes to 5Mpbs down and 7Mbps up. I discovered that my Usenet downloads went from 11Mbps (throttled by choice) to 0.3 while torrents were running. My Usenet test is a direct Windows download from a different device on the network.

 

I've throttled the number of connections, tried changing speeds, even PIA locations. This drop starts with even one torrent, with speed limits normally set to 300K down and 30K up, Even tried like 10K / 5K. Watching the unRAID traffic and the container traffic and overhead in Netdata doesn't show any unusual spikes.

 

For those in the "Network know." Traffic goes through Cisco SG300-10 * 2, Ubiquiti ER-X router, the cable modem is in Bridge mode. 

 

This has been working for months, and when I first installed it I never saw any bottlenecks like this. I was rather impressed that it all worked together. I'd say this slowdown has been occurring for at least the last few weeks and I couldn't figure out why. Now I know the "what" but I have no idea why.

 

unRAID is running on an overkill dual Xeon box.