So over the last couple of days I've been upgrading the system but since the upgrade the system seems to be pretty unresponsive. As an example my transmission docker used to time out every now and then for up to 60 seconds but since the "upgrade" its almost always on time out. 

The things that have changed are 

Motherboard MSI z270m Mortar >>>>> Gigabyte X399 Aorus Xtreme

CPU Intel G4600 >>>>> Threadripper 1900x

Cache Drive Samsung Evo 970 500GB >>>>> Corsair MP510 480GB

Ram Dell Samsung B die 2x 16GB >>>>> Corsair Samsung B die 8x 8GB

 

The systems seems be getting hit pretty hard by my dockers but in the past it wasn't unusual for me to see 100% load on all cores/threads with the intel CPU, currently the 1900x is about 50% average load with a few cores/threads hitting 100%. Disk reads/writes seems quite high and to be honest I'm not too sure why currently transmission is downloading at maybe 20MB/s but I'm hitting 70+MB/s reads and writes on the array.

 

If anyones got any ideas would be grateful for some help here since this was supposed to be an upgrade but so far general performance seems to be down.  I do also have a 2nd crosier MP510 that's currently unassigned was planning on using it for VM's but would I be better off putting it into a cache pool and setting my transmission downloads to use cache. I also have another Corsair MP510 that has a baremetal windows install on there that I am planning to passthrough to a vm but really need to get these performance issues in check before I start adding VM's into the mix.

I would suggest you check and redo all the core pinning (including isolcpus, docker core pin, VM core pin etc.).

1 hour ago, testdasi said:

I would suggest you check and redo all the core pinning (including isolcpus, docker core pin, VM core pin etc.).

The only problem with that is I don't have any cores pinned lol the only time I had any cores pinned were on VM which I haven't set up yet. The problem seems to be massive amounts of read/writes for no particular reason. I installed the netdata docker which doesn't seem to help much since most of the read/writes are displayed as "other". Now when I stop all my dockers the read/writes stop but I still can't figure out why my read/writes keep spiking since enabling one docker at a time multiple times it seems like it can start on any given docker it doesn't seem like a specific docker starts the problem. 

Are you writing temp data to array? Torrents (e.g. Transmission) do not like to write to the array due to their IO patterns. They should be on cache or unassigned.

Is parity build / check running? Btrfs scrub running? They would also cause slow down.

When the array is busy (e.g. parity build / torrents etc.), any attempt to access it will load up the CPU cores due to high IO Wait.

 

The other typical cause of 100% load on a few cores is if you isolcpus them and have dockers using those cores. That's why I said you need to check your core pinnings.

1 minute ago, testdasi said:

Are you writing temp data to array? Torrents (e.g. Transmission) do not like to write to the array due to their IO patterns. They should be on cache or unassigned.

Is parity build / check running? Btrfs scrub running? They would also cause slow down.

When the array is busy (e.g. parity build / torrents etc.), any attempt to access it will load up the CPU cores due to high IO Wait.

 

The other typical cause of 100% load on a few cores is if you isolcpus them and have dockers using those cores. That's why I said you need to check your core pinnings.

Yes I have torrents writing to the array and in the past when downloading I have had transmission time out etc but once the torrent has downloaded the connection comes back up most of my torrents are inactive or very slow at the minute like under 1MB/s but my reads/writes are 200+MB/s. Parity check isn't active at the minute. BTRFS also isn't active. Thats what I figured about the CPU load being due to high IO but still trying to figure out what's pushing the read/writes up. So for my downloads share which is where transmission downloads to that should be set to cache=yes? Any ideas if I would be best to set my other 480GB NVME as a second cache drive or should I keep it exclusively for VM's?

Download share should have Cache = Only or on an unassigned device.

 

Before making drastic config changes, just try having your torrent stuff on cache and see if it helps.

From my experience, torrents just plain kill it when having IO on array, doesn't matter how much IO.

1 hour ago, testdasi said:

Download share should have Cache = Only or on an unassigned device.

 

Before making drastic config changes, just try having your torrent stuff on cache and see if it helps.

From my experience, torrents just plain kill it when having IO on array, doesn't matter how much IO.

Quick update I figured out what the problem was but also made a new share that is cache only for my incomplete downloads. So the problem was my unpackerr docker I figured with the old cpu G4600 the cpu was slow enough that it couldn't saturate the drive io but since upgrading to threadripper files are being extracted quick enough that it can saturate the drives io and make everything lock up due to high io wait.