Poor bandwidth to unRAID server and severe packet loss

[Herdo]

I'm not sure if this is a problem with my unRAID server, or something else, but I figured I'd ask here as this forum has always been very helpful.

 

I've been having trouble streaming Plex to my parents house (2 miles away).  I have an asymmetric gigabit connection (1000/35), and my parents have ADSL (15/1).  I've run various speed tests from my desktop and found I should have no problem meeting the 720p 4mbps stream requirements, as I consistently get 30mbps upload, with maybe 20mbps at the slowest.

 

This evening I went to my parents house to check their connection.  They are consistently getting 10mbps+ and have no problem streaming full HD Netflix, Hulu, and other services from the same device running Plex (Samsung Smart TV).  I then decided to run some iperf3 tests.  I found that from my desktop running iperf3 to their house, I was getting 8mbps - 10mbps.  Again, this should be fine, and it seems to be maxing out their downstream connection.

 

 

After several hours I left their house puzzled, with no idea what the problem could be.  When I got home I had the idea of running iperf3 directly on the unRAID server instead on my desktop.  This told a very different story.  Now I was only getting about 500kbps to their house, with spikes maxing out around 700kbps.  I decided to download the speedtest plugin onto my unRAID server.  Again, this was showing something very different than when running a speed test from my desktop.

 

 

On my desktop, I consistently get about 800/25.  I just ran 5 speed tests in quick secession  and they were all under 20ms latency, 750 - 850 down, and 20 - 30 up.  I did the same thing on the unRAID server, and here are the results.

 

 

2018-01-14 04:24 Sun	Speedtest.net (Phoenix, AZ)	23.86 km	199.151 ms	226.18 Mbit/s	4.11 Mbit/s	http://www.speedtest.net/result/6964850039.png
2018-01-14 04:12 Sun	Speedtest.net (Phoenix, AZ)	23.86 km	539.298 ms	44 Mbit/s	3.29 Mbit/s	http://www.speedtest.net/result/6964828188.png
2018-01-14 04:11 Sun	Speedtest.net (Phoenix, AZ)	23.86 km	23.062 ms	53.86 Mbit/s	6.11 Mbit/s	http://www.speedtest.net/result/6964825532.png
2018-01-14 04:10 Sun	Speedtest.net (Phoenix, AZ)	23.86 km	61.116 ms	44.67 Mbit/s	4.43 Mbit/s	http://www.speedtest.net/result/6964823300.png
2018-01-14 04:05 Sun	Speedtest.net (Phoenix, AZ)	23.86 km	23.684 ms	274.93 Mbit/s	6.03 Mbit/s	http://www.speedtest.net/result/6964815609.png

 

 

 

As you can see, it's not only very slow, it's wildly inconsistent.  This got me thinking, and I realized that almost every time I ssh into the unRAID server I notice "lag".  I'll go to type a command and mid typing the cursor would lock up and then a few seconds later the text would quickly appear on the screen.  I assumed it was just a process running in the background occupying the CPU temporarily, but now I realized it may be something more sinister.

 

 

I have a mikrotik hEX router, which includes a handy little ping tool so I pinged the unRAID server.  As I suspected, there is some serious packet loss from the router to the unRAID server.  I've attached an image showing how bad it is.  It isn't rhythmic at all, just random packet loss from what I can tell.  It's upwards of 40% sometimes.

 

 

 

 

I just upgraded to this gigabit connection about a week ago, which is when I noticed this problem, but I'm thinking it's just a coincidence.  The modem was upgraded to a DOCSIS 3.1 modem, and then the router was upgraded a few days ago.  I first noticed it sometime between getting the new modem/gigabit and getting the new router.  I just set my parents up on Plex right around the time I got the new modem, but I've noticed inconsistent speeds for months, maybe more.  For instance, sometimes syncthing will pull down files from my remote server at 18Mbps, and sometimes at 200kbps, and like I said, I first noticed that months ago.  The problem with syncthing would then somehow fix itself the next day, but then I'd usually notice very low download speeds again soon after.

 

 

So I guess my reason for posting this is advice on where to start.  I'm not sure why only my unRAID server would be dropping packets, when everything else on the network is working fine.  My desktop and the unRAID server are both plugged in the same switch in my office and are sitting about 8 feet away from each other.  I'd really appreciate any ideas at all.  Any other tests/diagnostics I could run?  Has anyone run into a problem like this before?

 

If more information is needed, please just let me know.

 

 

EDIT:  I just realized I should try swapping the port on my switch.  I tried a different port, and a different cat5 cable with no luck.  Still getting packet loss.

 

EDIT 2: I've solved the issue.  My comment below explains what I did to fix it. 

Edited January 14, 2018 by Herdo

[bonienl]

Start with the obvious

 

1. Move the ethernet cable of your server to a different port on your router

2. Replace the ethernet cable of your server

3. Install the "Tips and Tweaks" plugin of dlandon,  it allows changing various interface settings

 

Note: rebooting your network gear (router, switch) may help too.

[Herdo]

2 minutes ago, bonienl said:

Start with the obvious

 

1. Move the ethernet cable of your server to a different port on your router

2. Replace the ethernet cable of your server

3. Install the "Tips and Tweaks" plugin of dlandon,  it allows changing various interface settings

 

Note: rebooting your network gear (router, switch) may help too.

 

 

Thanks for the reply bonienl.  I updated the post while you were typing this reply.  I've tried swapping the port and ethernet cable with no luck.  I just tried pinging my server from my desktop and had 30% packet loss.  However, I thought I'd ssh into the unRAID server and try pinging my desktop, and the packet loss was very minimal; 3% after about 3 minutes.  I don't think that is considered "acceptable", but it's certainly better than 30%.

 

I'll check out the Tips and Tweaks plugin and reboot my equipment.  Thanks again.

[bonienl]

3 minutes ago, Herdo said:

I don't think that is considered "acceptable", but it's certainly better than 30%

 

If you are doing this on your local LAN then there should be 0% packet loss.

[Herdo]

I had another NIC on my motherboard so I just tested that.  I thought I fixed it because it seemed to work fine at first, but after a longer ping test I can see I am definitely still dropping packets.  24% packet loss as of the latest test.

 

Is it possible the CPU can't keep up?  I don't know how much the CPU would affect something like this, but it's definitely the weak link in my server.  It's a Pentium G4400.

[Herdo]

OK, I've solved it.  

 

 

Disabling the "Static" IP setting in unRAID under "Network Settings", and then assigning the IP as static in my router instead seems to have completely solved the packet loss issue.  Just did a test of 500 50 byte packets, and I had 0 lost packets.

 

 

The speeds I'm getting to the unRAID server are still lower than expected.  It's anywhere from 1/2 to 1/3 what I am getting on my desktop, but at least the Plex stream should work now.  The latency is also quite a bit higher.  It's also possible the cli version of speedtest tests differently.

 

I'm not sure why this solved the issue, I just saw it mentioned on a thread where someone was having a similar problem.  I'm guessing my routers DHCP server was getting confused?

 

 

Thanks again for the help bonienl.

 

 

EDIT:  Nevermind about the latency and speeds being worse.  I must have reset the server settings and I wasn't comparing the same speedtest server between my desktop and server tests.  They are testing about equal now.

Edited January 14, 2018 by Herdo

[pwm]

4 hours ago, Herdo said:

 

 

Thanks for the reply bonienl.  I updated the post while you were typing this reply.  I've tried swapping the port and ethernet cable with no luck.  I just tried pinging my server from my desktop and had 30% packet loss.  However, I thought I'd ssh into the unRAID server and try pinging my desktop, and the packet loss was very minimal; 3% after about 3 minutes.  I don't think that is considered "acceptable", but it's certainly better than 30%.

 

I'll check out the Tips and Tweaks plugin and reboot my equipment.  Thanks again.

I think if you specify larger ping packets, your ping will also reach 30% packet loss.

 

I don't think your packet loss has with CPU load to do - it's more likely you have bit errors and with larger packets you get more bit errors and so a higher percent packets thrown away.

 

Somewhere, you have a bad cable or a bad network card/port. Or maybe some mixup with 100 Mbit/s half/full duplex.

 

Can you see packet counters on your switch and/or router? If it shows that it receives errors or framing?

[pwm]

1 hour ago, Herdo said:

Disabling the "Static" IP setting in unRAID under "Network Settings", and then assigning the IP as static in my router instead seems to have completely solved the packet loss issue.  Just did a test of 500 50 byte packets, and I had 0 lost packets.

Didn't see this - you might have had a IP address collision with some other device. Your unRAID used a fixed IP that the DHCP server had given out to some other device.

 

If you specify a static IP in unRAID, then that IP must be an IP that the DHCP server doesn't think it's allowed to send out to some other equipment. So you need to MAC-lock the IP in the DHCP server or alternatively reserve a range of IP for static assignments.

[Herdo]

4 hours ago, pwm said:

Didn't see this - you might have had a IP address collision with some other device. Your unRAID used a fixed IP that the DHCP server had given out to some other device.

 

If you specify a static IP in unRAID, then that IP must be an IP that the DHCP server doesn't think it's allowed to send out to some other equipment. So you need to MAC-lock the IP in the DHCP server or alternatively reserve a range of IP for static assignments.

 

Hey thanks for the reply.  That makes perfect sense. I've been testing it periodically and it's still all good and my parents can stream without any buffering.