LAN is the bottleneck. So what can we do about it?

[NAS]

Interestingly we all know/assume that the LAN is the main day to day performance bottleneck in unRAID for most people but we have nothing on the cards that will addresses this.

 

It probably makes sense to have a discussion about it in one place and perhaps link to any realistic suggestions that have been made in the past.

 

Other ideas are obviously welcome as well.

[boof]

I would disagree, copying from ram disk on a client -> ramdisk in unraid I get much better, almost wire speed, throughputs on a gigabit lan.

 

I still see the physical disks at either end of the equation being the limiting factor especially on parity writes. SSD's might help in theory but in practice probably not so much and it would have to be end to end for source and target.

 

To address any network issue would be as easy as enabling the kernel bonding drivers (if not done already?) and editing the web config pages to allow configuration. This would allow either easy client side bonding (which wouldn't help with a single transfer but would provide better throughput for multiple clients) or LACP compatibility if your switch can do it.

[starcat]

There is no bonding and no LACP implemented within unRAID. You just get one interface, may enable the second but that's it. No idea if you have custom unRAID installed on a full Slackware distro, anyone tried this with LACP?

 

Just use the best possible Intel GigNICs. But the bottleneck isn't at the NIC (80-95 MB/s for Intel NICs) but in unRAID itself and especially when writing to it.

[boof]

There is no bonding and no LACP implemented within unRAID. You just get one interface, may enable the second but that's it. No idea if you have custom unRAID installed on a full Slackware distro, anyone tried this with LACP?

 

Sorry my post was poorly worded. I meant that bonding would need to be included in the unraid kernel (if it's not already..which you've confirmed it isn't) and the web management pages would need to be updated to allow it to be configured easily.

[bubbaQ]

I would disagree, copying from ram disk on a client -> ramdisk in unraid I get much better, almost wire speed, throughputs on a gigabit lan.

 

Possible on an optimum infrastructure ... not generally what you find in a typical consumer setup.

 

Is speed writing really a big deal?  I mean, certainly a snail's pace of 5NB/sec is bad and worth working on, but with latest versions typically deliver better than 40MB/sec writing to protected storage, and can deliver writes at 80MB/sec to a cache drive, the need to do better than that seems rare.  It is not like we have to babysit and wait for a copy job to finish before we can use the system.

 

And how much could that be improved... at what cost... for what size?

 

Say an SSD for cache can do full wireline speed.  But when are you copying FROM another SSD or RAMdisk?  SSD cache won't help if the source you are copying from is a spinning disk in your workstation.

 

Even if you are copying from SSD workstation to SSD cache, and you get 125MB/sec (full gigabit wireline), how much are you copying?  Remember, your copy volume, per day, can't exceed your cache drive size or else you lose its benefit.  

 

So you want to copy 100GB a day... that's two 100GB SSDs... or a striped RAID in your workstation or for cache.

 

And where do you get the source for the files on the workstation end?  You certainly are not going to copy from a spinning disk in your workstation, to a SSD in the workstation, just to get faster wireline transfers.... so where's the benefit?  Are you generating 100GB of new data a day, originally written to the SSD in the workstation?  I didn't think so.

 

For most people, using an SSD for cache will not help throughput more than a marginal amount.

For most people, improving LAN speed will not help throughput more than a marginal amount.

 

If your writes to protected array are 35MB/sec or better, what's to be gained?

If your writes to unprotected cache are 70MB/sec or better, what's to be gained?  

 

Suppose you can do NIC bonding or eliminate the LAN bottleneck.... then what?  You eliminate one limit, but the next one is only 10% higher.

 

The bottom line is 1) what will it cost in $$$ and 2) how much actual impact will it be to the work you can do a day?

 

 

[boof]

I would disagree, copying from ram disk on a client -> ramdisk in unraid I get much better, almost wire speed, throughputs on a gigabit lan.

 

Possible on an optimum infrastructure ... not generally what you find in a typical consumer setup.

 

The above statement is true for my cheapo $25 5 port gigabit switch and whatever gig eth chipsets are on my desktop and unraid servers motherboards.

 

Pretty consumer.

[bubbaQ]

Hey, anything can be proven with a statistical sample of one!

 

But the point remains.... what does it cost and what do you get for that cost?

 

So if LAN is faster than a spinning disk, as in your case, when copying from a RAMdisk..... what can you do with that? 

 

Faster spinning disk for cache will get you what improvement?  At what cost?

Striped RAID for cache will get you what improvement?  At what cost?

SSD for cache will get you what improvement?  At what cost?

 

These all assume you are always copying from a RAMdisk in your workstation to unRAID.  Is that realistic?

 

I always tell people to do a test with RAMdisk to RAMdisk, because that is an isolation test -- it isolates the LAN from other potential bottlenecks so you get a legitimate test of what is possible with that LAN.  But RAMdisk to RAMdisk is not a realistic operating mode for daily use.

[WeeboTech]

There are a number of performance tweeks that can occur in the kernel to use buffering and caching which can significantly improve writes to the parity protected array.

 

With one set of tweaks I was getting consistent transmission of 55MB/s from a lil itx machine (intel nic) to X7SBE (intel nic) on a parity protected drive.

 

Granted I'm using RAID0 on my parity with a raid card.

I've bumped up the settable config options in the settings page.

I've bumped up how the kernel caches the data.

 

Last night I bumped ram up to 8GB.

I re-tweaked and tested "local" on my fastest drives.

I burst at 80MB/s for the first 500MB. It dropped slowly over time between 66MB/s and 75MB/s.

 

It's all a balancing act.

Where do you cache to improve throughput so the drive stays busy and is not starved for reads/writes.

 

It will be interesting when my next test  uses jumbo packets.

 

[bubbaQ]

 

It will be interesting when my next test  uses jumbo packets.

 

Measure your latency first.  A large MTU won't have much impact if latency is 1ms or less.

 

 

[starcat]

I've bumped up the settable config options in the settings page.

I've bumped up how the kernel caches the data.

 

WeeboTech, are you referring to the three tunable disk settings on the Settings page, md_num_stripes, md_write_limit and md_sync_window and could you give some more hints as to how to tune them or which values do you use? I think we have pretty similar X7SBE configs. Thanks much!

[NAS]

I have purposely been quiet in this thread so far to see where it was going but what I am hearing is:

 

if you have a properly tuned system LAN may be the a bottleneck but effort vs performance gains might not be worth it

in other scenario LAN could be a serious bottleneck

 

I think we should probably define a simple test of unprotected write to disk locally to gauge a maximum projected speed, perhaps a similar test but via a loopback interface and the same test done remotely (somehow making sure its not a client PC problem)

 

The key is that more often than not if you read back on advice we as a community say "LAN is the bottleneck". What we should be saying is "LAN could be the bottle neck check this or are your stats in the ballpark of xyz"

[WeeboTech]

I've bumped up the settable config options in the settings page.

I've bumped up how the kernel caches the data.

 

WeeboTech, are you referring to the three tunable disk settings on the Settings page, md_num_stripes, md_write_limit and md_sync_window and could you give some more hints as to how to tune them or which values do you use? I think we have pretty similar X7SBE configs. Thanks much!

 

I set mine from 3 to 5 times the initial values.

 

here is the script I used to test with.

#!/bin/bash

md_num_stripes=2784
md_write_limit=1536
md_sync_window=576

md_num_stripes=1280
md_write_limit=768
md_sync_window=288

(( md_num_stripes*=3  ))
(( md_write_limit*=3  ))
(( md_sync_window*=3  ))

sysctl vm.vfs_cache_pressure=100
sysctl vm.swappiness=50
sysctl vm.dirty_ratio=80

# To free pagecache, dentries and inodes: 
echo 3 > /proc/sys/vm/drop_caches

/root/mdcmd set md_num_stripes ${md_num_stripes}
/root/mdcmd set md_write_limit ${md_write_limit}
/root/mdcmd set md_sync_window ${md_sync_window}

 

#cat /boot/config/extra.cfg 

md_num_stripes=2784

md_write_limit=1536

md_sync_window=576

 

 

You have to stop the array and start the array anytime you adjust the extra.cfg from the array.

[starcat]

md_num_stripes=2784

md_write_limit=1536

md_sync_window=576

 

Changed those and restarted the array without any change in write speed to a NFS share on the protected volume!? Do you have NCQ disabled?