Increasing cache performance?

[Kuroyukihimeeee]

Hey all,

 

Having a small issue. Basically im trying to work out where my system is having the bottleneck for its cache speeds. Currently the cache is running 3x 850 120gb SSD's setup to use btrfs RAID0 with the -dconvert=raid0 command.

 

No matter what i try though, i seem to keep hitting this limit of about 458MB/s write and around 355MB/s read. Tried using different HBA controllers for SSD's, different PCIE sockets and still no change. My HBA cards currently are only Dell H200 passing the disks directly to unRAID, i appreciate these are not the most powerful cards on the planet so suspect these could be the issue, but i had the same problem when running directly off the SATA2 ports on the motherboard too.

 

Tried using hdparm -tT on one of the SSD's and it reports that it can do  Timing cached reads:  16800 MB in  2.00 seconds = 8409.01 MB/sec and  Timing buffered disk reads: 1394 MB in  3.00 seconds = 464.27 MB/sec. Not sure if this is useful at all, just that them buffered disk reads seem to be very very close to the ramdisk to cache array speeds.

 

Both clients have a Intel X540-T2 10GBe network card and are connected using a CAT7 cable. Both have been tried using 1500MTU up to 9014, virtually no change. unRAID has 8 cores, 16 threads and gets up to about 30% usage when copying over, the other system has a i7 4790k and uses about 15% when copying so im fairly sure its not a processor issue.

 

Again im not sure if this is just a blatant message pointing out the issue, but syslog does have these messages about the H200 and the X540-T2 when booting.

 

Jul 18 23:21:48 AVALON kernel: ixgbe 0000:04:00.0 eth0: Intel(R) 10 Gigabit Network Connection
Jul 18 23:21:49 AVALON kernel: ixgbe 0000:04:00.1: PCI Express bandwidth of 8GT/s available
Jul 18 23:21:49 AVALON kernel: ixgbe 0000:04:00.1: (Speed:2.5GT/s, Width: x4, Encoding Loss:20%)
Jul 18 23:21:49 AVALON kernel: ixgbe 0000:04:00.1: This is not sufficient for optimal performance of this card.
Jul 18 23:21:49 AVALON kernel: ixgbe 0000:04:00.1: For optimal performance, at least 20GT/s of bandwidth is required.
Jul 18 23:21:49 AVALON kernel: ixgbe 0000:04:00.1: A slot with more lanes and/or higher speed is suggested.

 

Just wondering if anybody has any tips to break this barrier i seem to be hitting? Or is there just a system bottleneck i haven't identified yet? Its almost like unRAID is ignoring the RAID0 of the cache and only writing/reading from a single SSD at any one time?

 

avalon-diagnostics-20160725-1550.zip

[JorgeB]

Looks to me like network bottleneck, unRAID first caches all writes to RAM (20% of free RAM by default), then speed drops to the actual device write speed, so the first few seconds of any write should be at maximum network speed (as long as the source is fast enough), your speed is constant, it looks like the cache can keep up with the writes, i.e., RAM cache never fills, you can easily confirm this by copying to an array disk, you have to use a big enough file, but if you see the same speed in the beginning then decreasing to the normal disk write speed it's your LAN, try the NIC on a PCIe 2.0 slot if available.

 

Below is what I get on my server, I usually use more RAM for cache but purposely decreased it for this example, first 10GB are cached to RAM at ~600/700MB/s then it decreases to my SSD actual write speed.

[Kuroyukihimeeee]

Just gave it a go with a 15.5GB file. Still solid transfer speeds (from about 525MB/s to 485MB/s so i dont think its NIC related at the moment, might try tomorrow making sure my HBA and NIC are both on PCIE2 16x slots to see if i can find any performance gain.

 

I mean its still a super clean 5GBe transfer, but still half what it should be.

 

[JorgeB]

I was saying you should test writing to an array disk, bypassing the cache.

[Kuroyukihimeeee]

So directly to array with no cache SSD looks like this. Starts at about 200MB/s solid then drops down to the slow speeds. Im assuming parity calculations, HBA,etc and all that is why its so slow right to the disk. Array is all WD Red 3TB's. 3 data, 1 parity.

 

 

 

Tried taking out all other add-in cards so only cards left are the NIC and HBA which should free up more lanes and CPU resources for them, both in 16x PCIE2 slots. Both of them however only have 8x connections on the bottom so will be limited to the 8x speeds, but still this should be more than enough. I highly doubt intel would make a 2x 10gbe connection NIC that does not have the bandwidth available to it. Same for the LSI based HBA.

 

Since swapping PCI slots and removing other cards, i see a different message in the syslog about bandwidth though, no warning this time.

 

Jul 25 22:40:55 AVALON kernel: Intel(R) 10 Gigabit PCI Express Network Driver - version 4.3.13
Jul 25 22:40:55 AVALON kernel: Copyright (c) 1999-2015 Intel Corporation.
Jul 25 22:40:56 AVALON kernel: ixgbe 0000:02:00.0: PCI Express bandwidth of 32GT/s available
Jul 25 22:40:56 AVALON kernel: ixgbe 0000:02:00.0: (Speed:5.0GT/s, Width: x8, Encoding Loss:20%)
Jul 25 22:40:56 AVALON kernel: ixgbe 0000:02:00.0 eth0: MAC: 3, PHY: 3, PBA No: G36748-005
Jul 25 22:40:56 AVALON kernel: ixgbe 0000:02:00.0: a0:36:9f:3c:59:88
Jul 25 22:40:56 AVALON kernel: ixgbe 0000:02:00.0 eth0: Enabled Features: RxQ: 16 TxQ: 16 FdirHash 
Jul 25 22:40:56 AVALON kernel: ixgbe 0000:02:00.0 eth0: Intel(R) 10 Gigabit Network Connection
Jul 25 22:40:57 AVALON kernel: ixgbe 0000:02:00.1: PCI Express bandwidth of 32GT/s available
Jul 25 22:40:57 AVALON kernel: ixgbe 0000:02:00.1: (Speed:5.0GT/s, Width: x8, Encoding Loss:20%)

[JorgeB]

The lower speed is normal writing to the protected array  but I was expecting the initial speed to be the same as when writing to the cache, since it's using RAM cache, stange...

[Kuroyukihimeeee]

Did a little more tweaking on this using a 10GB test file. Now managing to get about 700MB/s reads from the cache to a ramdisk but its not sustained, seems to go up for a second or so then go back to standard 350-400MB/s. Top screenshot below.

 

Writing still seems to be capped at 525MB/s solid. No drops or anything, it just refuses to go higher.

 

Still stumped by this, seems to be showing improvement from changes i have made, but still not what it should be.

 

 

Made these tweaks below from this link to my sysctl.conf file - http://dak1n1.com/blog/7-performance-tuning-intel-10gbe/

 

 # -- tuning -- #
# Increase system file descriptor limit
fs.file-max = 65535

# Increase system IP port range to allow for more concurrent connections
net.ipv4.ip_local_port_range = 1024 65000

# -- 10gbe tuning from Intel ixgb driver README -- #

# turn off selective ACK and timestamps
net.ipv4.tcp_sack = 0
net.ipv4.tcp_timestamps = 0

# memory allocation min/pressure/max.
# read buffer, write buffer, and buffer space
net.ipv4.tcp_rmem = 10000000 10000000 10000000
net.ipv4.tcp_wmem = 10000000 10000000 10000000
net.ipv4.tcp_mem = 10000000 10000000 10000000

net.core.rmem_max = 524287
net.core.wmem_max = 524287
net.core.rmem_default = 524287
net.core.wmem_default = 524287
net.core.optmem_max = 524287
net.core.netdev_max_backlog = 300000

[JorgeB]

I posted a few days ago about different speeds when writing to a disk or user share, you may want to try and see if there's any difference.

 

Still, on my other 10GbE server there's a much smaller difference, and these are the speeds I get without any tuning, writes are only being cached for about 2GB, then it's the actual cache speed, Samsung 950 Pro NVMe 250Gb, reads are slower, and max below 700MB/s, but are steady.

[Kuroyukihimeeee]

Might have to look into them. Might take a while, (9TB of data would need to be placed somewhere)

 

Still something seems off. Thinking in the next few months of building an entirely new server anyway so might just throw in the towel for now and see if the new server performs better.