[6.7.0-RC5] Post upgrade R/W speed troubleshooting - SOLVED

[weirdcrap]

My primary server "NODE" recently got an upgrade of sorts and I am now noticing disk performance has been degraded. Running the latest RC5.

 

 

For the longest time I have been running 2x Marvel Chipset 9215 (IIRC) PCIE x1 4 port SATA HBAs without much fuss. Recently I decided to swap them out (to prevent future UnRAID issues) with a newly flashed Dell H200. 

 

My disks were split between my 6 onboard SATA ports and 6 ports spread between the two Marvell controllers. One Marvell in the PCIE x16 slot and one in the first PCIE x1 slot. With this I would see a consistent 100-120Mbps.

 

Now I have 7 disks (including parity) running off the H200 in the PCIE x16 slot and the 4 Mobo SATA III ports all filled up. All disks are successfully reporting 6Gbps speeds. With this I am seeing sustained speeds of only about 50-60Mbps.

 

I ran a parity check right off the bat just to see how everything would perform under load. Speeds are good (around ~150 - ~100) but I noticed Plex would throw errors about not being fast enough for playback. At first I just wrote it off until I started trying to import a movie via Radarr. Radarr downloads via NZBGet to my cache drive and it is picking up and moving from their to my array cache which should normally only take ~30 seconds for ~5GB movie. When I logged in I had 4 movies all queued up and not moving at all until I cancelled the parity check. A few minutes after the parity check was cancelled everything imported just fine. I would only see small intermittent bursts of write activity in the system stats.

 

 

This led me to check speeds copying between disks which is where I got more concerned. Whether I am copying between disks connected to the H200 or between disks on the H200 and the Mobo controller disks I see bursts starting at ~100ish but dropping down to a sustainable ~50ish. This is significantly lower than I was expecting to see. 

 

Copying a 4.5GB ISO from an array disk on the mobo controller to an array disk on the H200 is around 50-60Mbps. Copying array disk to array disk on the H200 is around 60Mbps. 

I already tried adjusting my disk tuneables to what @johnnie.black recommends (up until today I had been running at the defaults on this server). Sorry for the wordiness, just trying to be thorough in my description of the issue. Attached is my diagnostics from the server.

 

I am considering going back to my Marvell controllers just to regain performance since this is my primary heavily used/shared server. I have an H310 that should be arriving tomorrow that I could try swapping out in it's place once I get it flashed.

 

What can I do to improve my performance beyond switch back to my old hardware? 

 

To summarize: I used to be able to have downloads continue to be imported in the background and/or watch Plex during parity checks and such but since putting in the H200 it seems to cause pretty significant performance issues when doing simultaneous R/W operations.

 

node-diagnostics-20190225-0140.zip

Edited March 5, 2019 by weirdcrap
grammar and such | added a TL;DR

[JorgeB]

6 hours ago, weirdcrap said:

This led me to check speeds copying between disks which is where I got more concerned. Whether I am copying between disks connected to the H200 or between disks on the H200 and the Mobo controller disks I see bursts starting at ~100ish but dropping down to a sustainable ~50ish. This is significantly lower than I was expecting to see. 

This is perfectly normal with default parity writing mode, and you should have similar performance before changing the controller, you can try turbo write.

[weirdcrap]

Interesting, you are correct about those speeds being normal, copying between my two H200s on my home server produces the same speeds. My Parity writing mode is set to Auto.

 

So what about the "blocking" of writes to the cache drive from docker? When the parity check was running I had movies hung up and not importing in Radarr. I SSH'd into the machine and manually copied one of the movies with MC and it copied no problem...

 

This is all anecdotal of course without having to put my old hardware back in and doing way more testing. I don't recall ever having performance issues importing downloads via docker containers or watching Plex regardless of whether a parity check was running or not. This is the biggest concern for me.

 

I'm doing some testing this morning and will come back and edit my post with results.

 

EDIT: Well after a reboot things seem to be in order...Radarr is picking up movies from NZBGet with no delay unlike last night's 30 minutes to import a local movie file. Plex hasn't hung up or thrown a to slow error yet.

Edited February 25, 2019 by weirdcrap

[weirdcrap]

A couple follow up questions as I have been doing reading on the turbo mode and the plugin for auto-enabling it that Squid made.

 

First, It sounds like this plugin is still useful as UnRAID's current "Auto" setting is to just leave turbo mode off, correct?

 

Second, the discussion in the thread makes it sound like this is ok to use with Docker applications as long as I have a cache drive and all my shares are set to "Cache: Yes." Otherwise the disks won't ever spin down and turbo mode will always be enabled (this is what I got from the discussion).

 

Sadly, based on some recent posts in the plugin thread it sounds like Dell H310s (and probably H200s since they use same chipset?) may return some odd statues to hdparm that are causing the plugin to not function as expected. So I may not be able to use it after all...

 

 

Edited February 25, 2019 by weirdcrap

[JorgeB]

9 minutes ago, weirdcrap said:

as UnRAID's current "Auto" setting is to just leave turbo mode off, correct?

Yes.

 

11 minutes ago, weirdcrap said:

it sounds like Dell H310s (and probably H200s since they use same chipset?)

Yes.

[weirdcrap]

After using it for a few weeks I am satisfied with how everything is working, as always thanks for your help Johnnie.