I think we have established that non-verbal communication can be easily taken the wrong way, and then blown out of proportion.  From what I have seen there is a lot of knowledge and genuine desire to help, assuming the normal resources have already been exhausted (wiki, forum searches, etc).  Obviously this thread turned around and you were able to get some information that will help you accomplish what you originally intended.  I would suggest you not abandon this forum, as there is simply too much experience on here to ignore, despite its possible stalwart initial response.  At any rate, I'm looking forward to your providing some serious benchmarks using hardware configurations outside the norm for most of us.  Good luck, and happy testing!

I do hold 5 US patents for a fortune 50 company, so I'm not completely crazy!  (Disclaimer:  Not saying this to pat myself on the back - so please don't flame me anymore!)

 

No, how could we interpret it as such?

I want to take a moment to clarify where I am with this test.  

 

I am going to refresh my UNRAID box over the next two months and want to make sure I am making the fastest practical system I can so that I can have another 5 years of trouble-free use at the best level of performance I can achieve with the priorities I had set in the first post.

 

The discussions led me to the conclusion that any modern processor will run straight UNRAID equally well for read/write performance.

 

Since the motherboard will be the foundation of the system, I want to start with something that will not be a bottleneck.  For that reason, I STILL plan to test one of today's most powerful offerings (i7 920) against today's most efficient/basic offering (Dual Core Atom).  Measuring these extremes will put to bed exactly what a user can expect to tradeoff when working at either end of the performance spectrum for new purchases.  I don't doubt what everyone is telling me, but I want to confirm whether the difference is 0kB/s or 5MB/s in average sequential read/write transfers of various sizes.  The architecture of these systems are quite a bit different - this is not just a comparison of processor clock rate, so the answer may be a little surprising.  I really don't plan to permanently use an i7 for this purpose even if it is a little faster.  This is a proof of concept.

 

The advent of the cache drive is new since my last system update.  Unfortunately, I am not running a pro license so cannot currently enable this functionality because I have never used more than 3 disks.  Sounds like there were some bargains offered over the holidays which I missed out on, unfortunately.

 

I will read up on the cache drive since it sounds like this should be a variable in my experiment.  I wish I had a spare Intel SSD on hand to stick alongside the two Velociraptors.  

 

Testbeds:

 

Atom 330 System

945GCD-M Chipset

1GB DDR2 667 - single channel

 

Core i7 920 System (I may use the newer 930 chip for the tests, depending on availability)

x58 Chipset

6GB DDR3 1600 - tripple channel

 

Both systems will have these common components:

(2) WD Velociraptor 300GB 10,000 RPM HDDs (1 for parity, 1 for data)

Corsair high speed USB stick for UNRAID system

 

Mesurements:

 

IOzone measurements of 4k - 4GB file sizes - Sequential read and write in the following configurations:

 

• Direct crossover cable connection to desktop PC
  
• Connection through a high end 24 port enterprise switch
  
• Connection through a basic D-Link DGS-2208 8 port Gigabit Switch
  
• If I can swing a PRO license: Above 3 tests repeated with cache drive enabled - an Intel SSD.
  

 

The desktop system from which these measurements will be made is a 4GHz Core i7 machine with 12GB DDR3 and a pair of Intel SSD (2nd gen) in a RAID 0 stripe set.  It should not be a bottleneck in the test.

I read the previous posts on link aggregation.  Based on those posts, It doesn't sound like UNRAID supports it yet and most felt it would not change performance.

 

Has the support for aggregation changed since those posts?  Any opportunity here if a user has a PC and switch which support the standard?

I just read up on the cache drive feature.  Not sure how I feel about this.  I don't normally utilize user shares, only disk shares.  It would be nice if unraid could utilize the cache drive for disk shares. 

 

For example: 

 

When a user writes a file to the disk share, it is instead written to the high speed cache drive (if enabled and space permits).  After network transfer is completed, the system would then immediately cleanup the cache drive local to the server to restore those contents to both the parity and intended disk.

I just read up on the cache drive feature.  Not sure how I feel about this.  I don't normally utilize user shares, only disk shares.  It would be nice if unraid could utilize the cache drive for disk shares. 

 

For example: 

 

When a user writes a file to the disk share, it is instead written to the high speed cache drive (if enabled and space permits).  After network transfer is completed, the system would then immediately cleanup the cache drive local to the server to restore those contents to both the parity and intended disk.

 

I think the cache drive should be left where it is right now.

A User share has a fuse layer between samba and the disk subsystem. This adds a small performance penalty.

Right now they are directly accessible to samba and nfs, i would prefer not having a software layer between the disk shares and the network protocols.

 

I suppose we could create a poor mans cache subsystem for disk shares. Instead of using a layer between disk shares, we could create a new directory structure on the cache disk named disk1, disk2, disk3 ,etc, etc. and write to it directly.

After that create our own scanner which moves the files on a specified schedule.

 

 

FWIW, your benchmarks could test disk shares and a user share to show the difference.

 

I think the cache drive should be left where it is right now.

A User share has a fuse layer between samba and the disk subsystem. This adds a small performance penalty.

Right now they are directly accessible to samba and nfs, i would prefer not having a software layer between the disk shares and the network protocols.

 

I suppose we could create a poor mans cache subsystem for disk shares. Instead of using a layer between disk shares, we could create a new directory structure on the cache disk named disk1, disk2, disk3 ,etc, etc. and write to it directly.

After that create our own scanner which moves the files on a specified schedule.

 

FWIW, your benchmarks could test disk shares and a user share to show the difference.

 

Perhaps you're right, it seems like this is something best implemented in hardware or in the base operating system without all of the middleware.  It just seems like there should be a way to get rid of the write bottleneck imposed by the parity process by buffering up the data in high speed storage.  This would be a great use of additional system memory without having to invest in hard drive controllers with huge cache built into the hardware.  But I'm sure this opens up a can of worms with data integrity and issues imposed by power failures, etc.

By the way, with all of the discussion around switch performance - if anyone who is about to buy a 24 port switch for their house, this seems to be a fairly good deal on a high performance switch:

 

Dell Powerconnect 2824 24 port web-managed gigabit switch: $180 (they ship configured for unmanaged if you don't want to do anything fancy):

 

http://www.dell.com/us/en/business/networking/switch-powerconnect-28xx/ct.aspx?refid=switch-powerconnect-28xx&s=bsd&cs=04&dgc=BF&cid=7421&lid=197378&acd=10466193-1225267-u0t1809391f0fp0c0s558

 

I wouldn't use this in an enterprise environment, but it would be excellent for whole-home ethernet.

 

This would be a great use of additional system memory without having to invest in hard drive controllers with huge cache built into the hardware.

 

Filesystem writes are already buffered in available RAM.  And it won't help much when dumping 400GB images and hidef ISOs to unRAID (or anytime the size of what you are writing significantly exceeds buffers).

 

You don't need a huge cache.  A mere 250MB of cache ram on the controller, with intelligent write reordering and read-ahead that takes into account to disk geometry, can make a MUCH larger difference.

 

The advent of the cache drive is new since my last system update.  Unfortunately, I am not running a pro license so cannot currently enable this functionality because I have never used more than 3 disks.  Sounds like there were some bargains offered over the holidays which I missed out on, unfortunately.

 

 

Maybe we can all chip in and get you fifty bucks for a Plus license for your tests?

That seems like a good cause.  After all, price is no object for the Ultimate Unraid Server.

 

 

After all, price is no object for the Ultimate Unraid Server.

Sorry, but I was lol'ing at that.

Unfortunately, I am not running a pro license so cannot currently enable this functionality because I have never used more than 3 disks.  Sounds like there were some bargains offered over the holidays which I missed out on, unfortunately.

 

Some of those bargains are still there.

 

" $10 off Registration Key order!

For limited time, get $10 off any Registration Key order.

Use coupon code KEY10 at checkout."

You should also try the KEY50 Code.

Don't forget the infamous KEY50.  That one gets Tom's attention every time.

This would be a great use of additional system memory without having to invest in hard drive controllers with huge cache built into the hardware.

 

Filesystem writes are already buffered in available RAM.  And it won't help much when dumping 400GB images and hidef ISOs to unRAID (or anytime the size of what you are writing significantly exceeds buffers).

 

You don't need a huge cache.  A mere 250MB of cache ram on the controller, with intelligent write reordering and read-ahead that takes into account to disk geometry, can make a MUCH larger difference.

 

You're right about HD ISO transfer - no practical amount of cache will cover a 50GB file movement except for a hard drive.  But 6GB of RAM cache would cover most file tranfers or "normal" use - even DVD ISO transfers.

 

However, my system is primarily for backing up my desktop systems daily which frequently exceeds 10-20GB each night.

 

While reading the WIKI, onboard SATA is stated to be faster than PCIe adapters.  I like using the onboard controllers if this is the case - since this results in faster, simpler, less expensive system with less likelihood of failure.

 

Do PCIe controllers with cache outperform onboard controllers (ich10R) for large (4GB+) file transfers?

You are missing the trees for the forest.

 

You must understand how writing to unRAID works -- otherwise attempts at benchmarking/optimization are just shooting in the dark.  It is very different from any typical desktop or a server.  A good analogy would be someone saying "Hey, this drug helps emphysema, so lets try it on asthma since they are similar problems."  Whereas anyone with knowledge of the difference in the diseases and the way the drug works could tell you that because of the way the drug works to help emphysema, it will have no effect on asthma.

 

Buffers only cache filesystem writes.  Once they are full, they do no more good until they are flushed and free again.  Buffers have no knowledge of disk geometry.  There is no effective reordering.  So if you write a file that covers 100 sectors, the 100 sectors will be written to the controller in the order that the OS sends them.  100 reads and 100 writes.

 

Now it would be MUCH faster to do 100 reads, keep them in cache, then do 100 writes to cache, have those 100 writes reordered so they can be written sequentially as the disk spins.  But unRAID itself can not do that.  Buffers can't do that either because they don't know anything about disk geometry, although certain read-ahead algorithms can help on the read side.

 

Now with a (good, configurable) caching controller, after the first read preceding the write, the whole track or cylinder will be read into the cache.  Subsequent reads don't have to wait on the disk to spin around to the right sector.  Writes go to the controller cache.  And because the controller does know the disk geometry, it can reorder the writes so they get written sequentially, potentially all in one rotation.  Thus a small amount of cache on the controller can have a much larger impact on write speed that OS buffers 10 times as large.  Cache on disk can do the same thing, but it is not going to be configurable to the degree a caching controller is.  Cache on disk is also going to generally be smaller.

 

But on-disk cache will help -- A LOT -- as it works like controller cache described above.  That is one likely explanation why "slower" (7200 RPM and even 5600 RPM drives with 32M or 64M of cache (like the new WD green with 4K sectors)) can out-perform a VRaptor w/ 8M cache at 10K RPM in unRAID, while the opposite will be true in a desktop or traditional file server.

 

Lets say you had 8GB of OS buffers, and were writing a 4GB file and had a blazingly fast LAN and source drive.  Yes, the copy from your source to unRAID would be over in a few seconds, as the whole file will fit in the OS buffers.  But it would take 3 minutes (just a number) to actually write the 4GB to disk... so 3 minutes before all 4 GB of buffers are available, and 3 minutes before your data is safely written to disk.

 

With 256MB of cache on the controller, that 4GB of data could easily be written to the disk in close to the large-file sequential read/write benchmark of the disk -- in under a minute.

 

This enormous impact would not be felt on a desktop or traditional server however.... you only get this huge impact because or the unique read-then-write operation of unRAID.  You have to forget/ignore much of the traditional wisdom of optimization in other environment.

 

Fragmentation also has a much larger effect on unRAID.... which is one reason Reiser FS is a good choice because it's handling (prevention) of fragmentation is better than several other FS.

 

Onboard controllers are favored over PCI or PCIe controllers because of bandwidth necessary for doing a parity rebuild.  When writing a single file to one disk, the difference would be negligible (unless both are managed by the same chip).

 

You want a test?  Get 2 of these:

 

  http://www.newegg.com/Product/Product.aspx?Item=N82E16820227393

 

and set up one as data and the other as cache.  This will give you the practical maximum your testbed can produce.  Note you will need to use a RAMdisk or another SSD as source in order not to starve the data.  Many benchmarkers use the Fusion IO card (800MB/s reads) as a source for such tests.

 

Then get 2 of the WD EARS drives (64M cache):

 

  http://www.newegg.com/Product/Product.aspx?Item=N82E16822136490

 

... then compare those results to an unRAID system with the precious VRaptors that only have 16MB of cache.

 

Because of the large cache, I predict the EARS will perform similar to or even better than the raptor in unRAID large-file transfers -- despite the EARS costing half the price and having 3 times the space and half the rotations speed.

 

Onboard controllers are favored over PCI or PCIe controllers because of bandwidth necessary for doing a parity rebuild.  When writing a single file to one disk, the difference would be negligible (unless both are managed by the same chip).

 

You want a test?  Get 2 of these:

 

   http://www.newegg.com/Product/Product.aspx?Item=N82E16820227393

 

and set up one as data and the other as cache.  This will give you the practical maximum your testbed can produce.  Note you will need to use a RAMdisk or another SSD as source in order not to starve the data.  Many benchmarkers use the Fusion IO card (800MB/s reads) as a source for such tests.

 

Then get 2 of the WD EARS drives (64M cache):

 

   http://www.newegg.com/Product/Product.aspx?Item=N82E16822136490

 

... then compare those results to an unRAID system with the precious VRaptors that only have 16MB of cache.

 

Because of the large cache, I predict the EARS will perform similar to or even better than the raptor in unRAID large-file transfers -- despite the EARS costing half the price and having 3 times the space and half the rotations speed.

 

Good information, bubbaQ... Unfortunately, I don't have this hardware available to me at present to test your theory.  I wouldn’t mind picking up another SSD if I find a good deal on an Intel drive (I know I will have use for it), but wouldn’t want to buy into the other drives right now. 

 

Here’s what I do have if we want to structure a test around these options:

 

I have (2) WD VelociRaptor 10,000 RPM 300GB, 16MB Cache Drives

I have (2) WD Raptor X WD1500AHFD 10,000 RPM 150GB, 16MB Cache Drives

 

I also have a few assorted individual drives available for test:

Western Digital Black WD6401AALS, 640GB, 7200 RPM, 32MB Cache Drive (New)

Hitachi  Deskstar HDT721010SLA360, 1TB, 7200RPM, 16MB Cache Drive (New)

Hitachi Deskstar HDS721075KLA330, 750GB, 7200 RPM, 32MB Cache Drive (Vintage Mar-2008)

 

My current server has the following, but I am actively using these volumes so can only test in place:

Western Digital RE3 WD1002FBYS 7200 RPM, 1TB, 32MB Cache Drive (Data)

Seagate TSD-1500AS SY 7200 RPM, 1.5TB, 32MB Cache Drives (1 Parity and 1 Data)

 

As mentioned earlier in my post, the data source will be a RAID 0 stripe set consisting of 2 Intel X25 (Gen2) SSD’s, so I don’t believe I will have a source bottleneck for the writes.  However, they are in my primary desktop and I can’t use them for a test on the server side.

 

What did you mean when you said “unless both are managed by the same chip”?  Your earlier statements about a caching controller lead me to believe that you felt a caching controller would be superior to on disk caching. 

 

You and others are referencing 8MB cache on the VelociRaptors.  Notice above that both the VelociRaptor and the older 150GB Raptor have 16MB cache.  However, I don’t know how this will stack up to the large 64MB cache on the EARS drives.  I was actually hoping that the 150G raptors did have 8MB, because I could compare them to the 300G with 16MB and essentially measure the cache impact when spindle speed is constant.

 

As mentioned earlier in my post, the data source will be a RAID 0 stripe set consisting of 2 Intel X25 (Gen2) SSD’s, so I don’t believe I will have a source bottleneck for the writes.  However, they are in my primary desktop and I can’t use them for a test on the server side.

 

So you are copying over the wire?  that won't tell you anything except where you saturate the Ethernet.

 

You can do the same thing by just writing over the wire to a RAMdisk on unRAID.

 

You need to do all the testing with a RAMdisk or SSD outside the array but physically in the unRAID server.... no LAN involved.

bubbaQ: 

 

The LAN transfer is the output variable I am most concerned with.  I can do additional testing at a later date from within the UNRAID box if we can learn more from that venture.

If you don't test the drives internally w/o the LAN, you won't know if the test is testing the LAN or the drives!  And if you are intending on testing the LAN, you should not be using drives but writing to RAM on unRAID.

 

Only after establishing benchmarks for individual components do you benchmark the combined system.

 

If you don't want to use a RAMdisk, at least test the drives with dd using /dev/zero.

 

I had some time this afternoon, so I went ahead and completed the tests with the hardware I had on hand.

 

The test was run with direct cable connection between the NAS and a Core i7 system running at 4.0GHz from an Intel SSD Raid 0 Stripe Set (Data Source) and the following command line argument for IOZone:

 

iozone -Rab RESULTS.xls -i 0 -i 1 -+u -f \\[iNSERT SERVER NAME]\disk1\filetest -y 64k -q 64k -n 64k -g 4G -z

 

 

Baseline System (My current NAS):

•ASUS P5LD2-VM R2.0

•Intel Pentium D 820 Dual Core 2.8GHz, 800MHz FSB

•1GB Dual Channel Corsair XMS2 PC5400 DDR2 675 SDRAM

•380 Watt Cooler Master Power Supply

•QTY 1: Western Digital RE3 WD1002FBYS 7200 RPM, 1TB, 32MB Cache Drive (Data Disk1)

•QTY 2: Seagate TSD-1500AS SY 7200 RPM, 1.5TB, 32MB Cache Drives (Parity and Data Disk2)

 

Power Consumption (Disks Spun Up):  110 Watts

Power Consumption (Disks Spun Down):  86 Watts

Power Consumption (Peak During 1.4GB File Transfer):  132 Watts

 

 

DISK 1 (WD RE3) Write Performance for 64KB Record Length at Various File Sizes:

1MB:  683 MB/S

32MB:  82 MB/S

512MB:  64 MB/S

1GB:  17.8 MB/S

2GB:  13.7 MB/S

4GB:  12.6 MB/S

 

DISK 1 (WD RE3) Read Performance for 64KB Record Length at Various File Sizes:

1MB:  68.4 MB/S

32MB:  81.9 MB/S

512MB:  45.1 MB/S

1GB:  31.0 MB/S

2GB:  30.7 MB/S

4GB:  30.5 MB/S

 

DISK 2 (Seagate) Write Performance for 64KB Record Length at Various File Sizes:

1MB:  24.7 MB/S

32MB:  19.2 MB/S

512MB:  51.6 MB/S

1GB:  18.6 MB/S

2GB:  16.6 MB/S

4GB:  13.9 MB/S

 

DISK 2 (Seagate) Read Performance for 64KB Record Length at Various File Sizes:

1MB:  70.1 MB/S

32MB:  82.5 MB/S

512MB:  45.5 MB/S

1GB:  31.3 MB/S

2GB:  32.7 MB/S

4GB:  31.2 MB/S

 

 

Atom System (Direct Cable Connection):

•Atom 330 with 945GCD-M Chipset (ECS 945GCD-M(1.0))

•1GB PC5400 DDR2 SDRAM (2 512MB Sticks)

•500W, 70% Efficient CoolerMaster Extreme Power Plus

•QTY 2: Western Digital VelociRaptor 300GB, 10,000 RPM, 16MB Cache (Parity and Data Disk1)

 

Power Consumption (Disks Spun Up):  47 Watts

Power Consumption (Disks Spun Down):  38 Watts

Power Consumption (Peak During 1.4GB File Transfer):  56 Watts

 

Write Performance:

1MB:  626.5 MB/S   

32MB:  65.8 MB/S

512MB:  44.9 MB/S

1GB:  30.2 MB/S

2GB:  22.9 MB/S

4GB:  21.7 MB/S  (56% Faster than Best Drive on Current System)

 

Read Performance:

1MB:  57.9 MB/S

32MB:  77.2 MB/S

512MB:  31.8 MB/S

1GB:  29.2 MB/S

2GB:  29.3 MB/S

4GB:  29.6 MB/S  (5% Slower than Best Drive on Current System)

 

Atom System (Through Inexpensive D-Link DGS-2208 8 Port Gigabit Switch):

 

Write Performance:

1MB:  663.4 MB/S

32MB:  65.7 MB/S

512MB:  48.1 MB/S

1GB:  29.4 MB/S

2GB:  23.1 MB/S

4GB:  21.6 MB/S  (Less than 1% slower than direct cable connection)

 

Read Performance:

1MB:  64.2 MB/S

32MB:  71.8 MB/S

512MB:  32.5 MB/S

1GB:  28.1 MB/S

2GB:  28.5 MB/S

4GB:  29.4 MB/S  (Less than 1% slower than direct cable connection)

 

That 48W idle power difference with the disks spun down is a pretty big deal (and the gap gets wider under load).  My Atom test system is even running on an extremely oversized power supply for the application (it is all I had available for the test).  That's a reduction of 420kw-hours per year which is about $48 for the average household ($.113/kWH).

The Core i7 system results will be forthcoming when I get a processor in stock.

Different drives, controllers, chipsets, and NICs.... in addition to different CPUs.

 

So which is responsible for the delta in performance?  You have to change only 1 thing at a time.

Different drives, controllers, chipsets, and NICs.... in addition to different CPUs.

 

So which is responsible for the delta in performance?  You have to change only 1 thing at a time.

 

BubbaQ: 

 

I will get there as time permits and the remaining hardware is available.  My plan is to first test the envelope of performance to determine the boudary conditions. 

 

To get the maximum amount of knowlege, I plan to test the UNRAID system using a DOE (Design of Experiments).  This is many times faster than perfoming an OFAT experiment (One Factor at a Time) as you suggest.  Another benefit is that it will tell me how different inputs interact with one another (for example, Disk RPM, Disk Cache and System Memory may only produce good results in a certain combination of the three).  If you are unfamiliar with this testing method, you can read about it at any web site that discusses statistical problem solving (Six Sigma).

I didn't mention it above, but my 4 year old "baseline" system and the new ECS Atom 330 board both use ICH7 controllers. 

 

I didn't time it, but the Atom system doese take quite a while to load up the UNRAID OS from the flash drive.  My old system is online in just a few seconds after power-up (less than a minute).  The atom took several minutes to become available on the network.

I didn't mention it above, but my 4 year old "baseline" system and the new ECS Atom 330 board both use ICH7 controllers. 

 

I didn't time it, but the Atom system doese take quite a while to load up the UNRAID OS from the flash drive.  My old system is online in just a few seconds after power-up (less than a minute).  The atom took several minutes to become available on the network.

 

Loading the OS from a USB flash has a number factors which affect the speed. The point most users will care about is general operation and write speed.  if load speed were an issue for people, the OS could be loaded onto a cache drive in a /.boot folder. Grub4dos can be used to boot the OS from hard disk. As long as /boot is on your flash, the system will register and use that folder.  The only issue would be keeping the flash and /.boot on your cache in sync.  I'm veering off topic here, but the core issue many users have is write speed. I would suggest not going to crazy on figuring out the boot speed issue between the atom and other machine unless you have spare time.

 

 

A couple of betas ago many of us did a very basic DD test local in the machine.

This taxed the machine directly without the network layer.

It showed the basis of what the maximum your write speed could ever be.

 

It proved to be very enlightening when limetech adjusted the md driver parameters for better write speed.

 

Please see thread:  http://lime-technology.com/forum/index.php?topic=4625.0

 

It would be helpful if you did this on your test systems to provide a bast case baseline.

 

#!/bin/bash

if [ ! -z "${1}" ]
   then TMPFILE="${1}"
   else TMPFILE="test.$$"
fi

trap "rm -f ${TMPFILE}" EXIT HUP INT QUIT TERM

echo "`date` writing to: ${TMPFILE}"
dd if=/dev/zero of="${TMPFILE}" count=4000000 bs=1024
echo "`date` Done."
ls -l --si ${TMPFILE}

rm -f "${TMPFILE}"

 

run it with the name of the script and argument 1 as the path to write.