Where to start for Performance Issues?

[Tweak3D]

I've been more heavily utilizing my unraid setup recently and loading lots of media onto my array while also utilizing dockers and the webgui within the system. I've been running into a lot of lag though whenever a few things start happening at once, but I cannot pin down the cause. Based on my observations, i think this is related to my Cache disk not being able to keep up but wanted to get some ideas of where to check as i'm not that familiar with non-windows os's.

 

So, first off, my hardware

Lenovo TD340

2x Intel E5-2420V2

96gb DDR3 ECC 1600 ram

5x 3TB WD Red Data Drives

3x 8Tb WD Red Data drives

512gb Samsung 850 Pro Cache Drive

256gb Samsung 850 Pro Unassigned (Plex DB)

 

I'm running the SSD's through the motherboard SATA connectors and the harddrives are driven through an IT mode raid controller.

 

So my primary issue is say i'm copying down a lot of large media thats going to take some time. but if plex or sickrage decide to also go out and update their DB, basically unraid becomes a slide show and is nearly unusable until i can get in and stop one of the outstanding tasks. I am not, however seeing any significant usage on my CPU or memory, so unless the items causing me pain are all single threaded, i'm at a loss. Where can I start looking to help diagnose this performance? I am considering replacing my SSD's with an m.2 (970 pro likely), but don't want to unless I am able to confirm its the iops limiting me currently.

 

Things I've tried so far:

Moving plex to its own SSD

Moving deluge to its own dedicated unassigned drive

Moving my surveillance (unifi) to its own dedicated harddrive.

 

While the above help, none have made the problem go away. Any thoughts?

hoofs-diagnostics-20180612-1531.zip

Edited June 12, 2018 by Tweak3D
diag information attached

[Tweak3D]

wanted to add an example. Copying some large files from my computer to the array and copying files from an unassigned disk to my computer. Either copy alone will saturate the gigabit nic no problem, but with both going, they both drop to 10 mb/s or less and very often both go to 0 for a bit and then bounce back up to 10ish. If I pause either copy, within seconds the other copy jumps back to saturating the nic. During the time that both are slow, the entire array and and any apps are slow from every device on my network. 

 

Perhaps its a networking issue?