Jump to content
henris

Performance benchmarking using IOzone

7 posts in this topic Last Reply

Recommended Posts

Quick links

[iurl=#anch1]1. Background[/iurl]

[iurl=#anch2]2. Test aspects[/iurl]

[iurl=#anch3]3. Test setup[/iurl]

[iurl=#anch4]4. Test results[/iurl]

[iurl=#anch5]5. Conclusion[/iurl]

[iurl=#anch6]6. Update history[/iurl]

 

[anchor=anch1]1. Background[/anchor]

I started looking for a scalable and reliable storage solution at the end of last year when all my so-called NAS's and USB-disks were reaching their maximum capacity. I had been come very aware of the limitations of the solutions I had acquired over the past 3-4 years. Both the functionality and performance of those were insufficient to my liking. Also the move from DVD to Blu-Ray meant I would be needing tens of terabytes of storage space to host my movie collection in the upcoming  years.

 

It was obvious to me to go for a raid-based file server solution. I had always wanted to build a dedicated server and this was a perfect excuse for it. I got my first touch to UnRAID when visiting a friend of mine who had built an UnRAID server by combining existing workstation hardware with a mix of IDE- and SATA-drives. I was especially impressed by the non-striping RAID concept which allows flexible addition of new disks with no size limitations. Also the straightforward setup and intuitive management interface made it easy for me to select Unraid over alternatives like OpenSolaris RaidZ or FreeNAS. UnRAID also allowed the extension of the server functionalities by running additional software within the same system.

 

From my past experience of several PC and HTPC builds I had learned that time spent on selecting proper HW- and SW-components and understanding their features and cons/pros will pay itself back in the future. So I spent quite some time in forums like Unraid, AVSForums and similar before making final decision on the HW-components. I wanted to understand what kind of performance I should be expecting from UnRAID server and what of kind factors could affect performance or reliablity. I soon realised that there are several things which might affect performance and there were partial performance reports of invidual people. I also found some more serious benchmarking done but these were a bit outdated.

 

So I decided to make benchmarking on my own. I got usefull information and hints from the UnRAID forum and ended up with the test setup used in these tests. I wanted the tests to be repeatable and mostly automated so I could easily re-run them and compare the results to the original in the future when upgrading UnRAID software or adding disks. I didn't want to spend too much time on this project so I didn't try to make tests to be generic in a way an average joe could install a package, run the tests and easily see the results. The testing was done primarily for myself but as there were no or little real benchmarking information available I thought I could spend some time to document and publish the results. I hope it helps someone.

 

At this time only the tests performed inside the UnRAID server are finished and published. This is mostly due to my personal time constraints. These tests have been performed already back in February 2009 and it was kind of a last chance to publish the results before they become outdated.

 

[anchor=anch2]2. Test aspects[/anchor]

The following questions were the primary aspects for the testing to be performed.

 

Q1: What is the theoretical maximum performance of the used hard drives?

 

Q2: What is the performance of the chosen hard drives on the Linux platform without the effect of UnRAID?

 

Q3: What is the internal performance of UnRAID server?

 

Q4: Is there a difference between internal and over-the-network client performance?

 

Q5: How do different configuration settings (read-ahead cache size, NCQ) affect performance?

 

Q6: Does disk fill ratio affect performance?

 

Q7: What is the performance difference between disk shares and user shares?

 

Q8: Does enabling of user shares affect disk share performance?

 

Q9: How do synthetic tests correlate to real life performance?

 

Q10: How is the UnRAID server in comparison to previously used solutions (NAS-devices, USB-disks, local hard drives)?

 

[anchor=anch3]3. Test setup[/anchor]

3.1 UnRAID server hardware[/b][/size]

As this post is not a build story the components are listed here only for reference. Although I'm very satisfied with the overall system I would now choose a case with easier expansion capabilities (all HDs available at the front). Also two of the six sata-ports on the mb are not supported by the UnRAID.

 

Case: Antec Performance One P180 Grey

Processor: AMD Athlon 64 X2 4850E 2.5Ghz

Power Supply: Antec TP3-650 EC TruePower Trio

Memory: Kingston HyperX 2048MB (2pcs) => 4GB total memory

Hard drives: Samsung Spinpoint F1 1TB SATA 3.0 Gb/s (4pcs)

Motherboard: MSI K9A2 Platinum

Display card: ATI X1300XT

 

3.2 UnRAID setup

UnRAID software version is v4.4.2. No additional packages installed besides UnMenu and test software.

 

Disk setup:

- Disk0 = Parity

- Disk1 = Data disk 1 (900GB disk space used)

- Disk2 = Data disk 2 (~0GB disk space used)

- Disk3 = Cache (~0GB disk space used)

 

3.3 Test tools

The primary test tool used was IOzone v3.221. The test scheme was adopted from SmallNetBuilder (Tom's NAS charts). The only modification was the exclusion of file sizes below 32MB and inclusion of file sizes upto 32GB since the original maximum of 4GB was too close to the installed system memory size thus producing mainly cached reads/writes. One could quite easily build a simple performance benchmark tool to be integrated to some of the customised web guis.

 

IOzone is normally run on the client side. In the internal tests IOzone was running inside UnRAID server thus stressing the same system which might lower the results. On the other hand file caching might have the opposite effect. However CPU was never maxed out during the tests and the test results seemed reasonable enough so it was concluded that running IOzone in the same machine did not considerably affect results.

 

Also some other operating system tools were used for both testing and changing system configuration. Below is a list of all tools, used command lines and related links. The whole test script is attached to this post for analysis purposes only (it contain some dependencies which I didn't want to document properly at this point). Even though it does not contain anything destructive I advice not to run as it is. The script contain also functions for hdparm and dd tests, these are currently disabled in the test script.

 

The charts were produced with MSExcel.

 

IOzone v3.221:

http://www.iozone.org/

http://www.linuxpackages.net/pkg_details.php?id=4277

http://www.smallnetbuilder.com/content/view/30682/77/

http://www.smallnetbuilder.com/component/option,com_nas/Itemid,190

iozone -Rab result.xls -i 0 -i 1 -+u -f /mnt/#####/iozone.dat -y 64k -q 64k -n 32M -g 32G -z

 

Set read ahead cache:

blockdev --setra $1 /dev/md*

 

Clear disk caches:

sync

echo 3 > /proc/sys/vm/drop_caches

 

Enable/disable NCQ:

echo 31 > /sys/block/sd#/device/queue_depth

echo 1 > /sys/block/sd#/device/queue_depth

 

 

[anchor=anch4]4 Test results[/anchor]

Summary chart for read performance (see the linked images in each test case for detailed results)

UnRAID_test_results_Chart_5i.png

 

Summary chart for write performance (see the linked images in each test case for detailed results)

UnRAID_test_results_Chart_6e.png

 

4.1 Test 1 - Read ahead cache effect on read performance

* See Charts 1a-h

Charts 1a to 1c contain read speeds of invidual disks with different read ahead cache sizes.

Charts 1d to 1f contain the same information as 1a to 1c but the visible Mbps is limited (Zoomed) to 0-150Mbps area.

Chart 1g contain read speeds of all disks with static 2048 read ahead cache size

Chart 1h contain the same information as 1g but the visible Mbps is limited (Zoomed) to 0-150Mbps area.

 

4.2 Test 2 - Read ahead cache effect on write performance

* See Charts 2-d

Charts 2a to 2c contain write speeds of invidual disks with different read ahead cache sizes.

Chart 2d contain read speeds of all disks with static 2048 read ahead cache size

 

4.3 Test 3 - NCQ effect on read performance

* See Charts 3a-h

Charts 3a to 3c contain read speeds of invidual disks with different read ahead cache sizes and NCQ enabled/disabled.

Charts 3d to 3f contain the same information as 3a to 3c but the visible Mbps is limited (Zoomed) to 0-150Mbps area.

Chart 3g contain read speeds of all disks with static 2048 read ahead cache size and NCQ enabled/disabled.

Chart 3h contain the same information as 3g but the visible Mbps is limited (Zoomed) to 0-150Mbps area.

 

4.4 Test 4 - NCQ on write performance

* See Charts 4a-d

Charts 4a to 4c contain write speeds of invidual disks with different read ahead cache sizes and NCQ enabled/disabled

Chart 4d contain read speeds of all disks with static 2048 read ahead cache size and NCQ enabled/disabled

 

4.5 Test 5 - User share read performance and effect on disk share read performance

* See Charts 5a-k

Charts 5a to 5c contain read speeds of invidual disks with different read ahead cache sizes, NCQ enabled/disabled and user shares enabled.

Charts 5d to 5f contain the same information as 5a to 5c but the visible Mbps is limited (Zoomed) to 0-150Mbps area.

Chart 5g contain read speed of a user share with different read ahead cache sizes and NCQ enabled/disabled.

Chart 5h contain the same information as 5g but the visible Mbps is limited (Zoomed) to 0-150Mbps area.

Chart 5i contain read speeds of all disks and user share with static 2048 read ahead cache and NCQ enabled/disabled

Chart 5j contain the same information as 5i but the visible Mbps is limited (Zoomed) to 0-150Mbps area.

Chart 5k contain read speeds of all disks with static 2048 read ahead cache, NCQ enabled and user shares enabled/disabled

 

4.6 Test 6 - User share write performance and effect on disk share write performance

* See Charts 6a-g

Charts 6a to 6c contain write speeds of invidual disks with different read ahead cache sizes, NCQ enabled/disabled and user shares enabled.

Chart 6d contain write speed of a user share with different read ahead cache sizes and NCQ enabled/disabled.

Chart 6e contain write speeds of all disks and user share with static 2048 read ahead cache and NCQ enabled/disabled

Chart 6f contain the same information as 6e but the visible Mbps is limited (Zoomed) to 0-150Mbps area.

Chart 6g contain write speeds of all disks with static 2048 read ahead cache, NCQ enabled and user shares enabled/disabled

 

[anchor=anch5]5. Conclusions[/anchor]

Coming back to the original test aspects:

Q1: What is the theoretical maximum performance of the used hard drives?

Maximum interface bandwidth (measured by Toms hardware, link below) for Samsung F1 1TB (HD103UJ) is 170MB/s. The same source measured 120MB/s as maximum transfer rate (60MB/s minimum).

http://www.tomshardware.com/reviews/samsung-overtakes-a-bang,1730-8.html

 

Q2: What is the performance of the chosen hard drives on the Linux platform without the effect of UnRAID?

From chart 1d the read speed for Cache disk is ~120MB/s.

From chart 2a the write speed for Cache disk is ~100MB/s.

 

Q3: What is the internal performance of UnRAID server?

From charts 1e and 1f the read speed for disk shares is ~115MB/s

From charts 2d the write speed for disk shares is ~45MB/s

These results are in the expected range since writing requires parity calculation which affects performance. The read speed difference compared to cache disk read speed is surprisingly small.

 

Q4: Is there a difference between internal and over-the-network client performance?

Not tested at the moment. From real life usage over-the-network performance is about 25% less than internal performance. Writing speed to disk share from Windows XP machine has been around 30-35MB/s.

 

Q5: How do different configuration settings (read-ahead cache size, NCQ) affect performance?

Reading speeds (chart 1e and 1f) for disk shares are almost identical with read-ahead cache sizes above 512.

Same applies to user shares (chart 5g). It seems that the default 2048 is sufficient and that is what I'm using in my production system.

 

Disabling or enabling NCQ (charts 3 and 4) didn't have any effect on performance. But this might be due to the nature of the tests, enabling NCQ might benefit an environment where multiple concurrent reads and writes are taking place. I left NCQ enabled.

 

Q6: Does disk fill ratio affect performance?

There is no difference in performance (charts 1h and 2d) between disks 1 and 2, even though disk 1 is almost full and disk 2 almost empty. This is contrary to Tom's Hardware's results where the performance diminishes towards the end of the medium (from 120Mb/s to 60Mb/s). The reason for contradiction might be the test setup, I have actually no idea where in the physical disk the IOZone test data is being written to. So final result is inconclusive. My assumption is that in any case fill ratio would limit reading speed only, since writing is already limited to 45MB/s by parity calculation overhead.

http://www.tomshardware.com/reviews/samsung-overtakes-a-bang,1730-8.html

 

Q7: What is the performance difference between disk shares and user shares?

Disk share performance was covered in Q3 (115MB/s read, 45MB/s write). User share read speed (chart 5h) is 70MB/s and writing speed 30MB/s. So the user performance is around 60% of disk share performance.

 

Q8: Does enabling of user shares affect disk share performance?

No it doesn't (chart 5k for read, 6g for write)

 

Q9: How do synthetic tests correlate to real life performance?

Partly covered in answer to Q4.

 

Q10: How is the UnRAID server in comparison to previously used solutions (NAS-devices, USB-disks, local hard drives)?

Not properly tested at the moment. However UnRAID is lightning fast compared to my Lacie Ethernet Disk mini and Buffalo Drive station (USB).

 

[anchor=anch6]6. Update history[/anchor]

28.06.2009 - First release

14.01.2009 - Added "embedded" summary charts in test results, detailed results as linked images in each test case as before

Share this post


Link to post

Terrific work!  Can't wait to see your next set of test results!  A few comments:

 

I too have found little difference with NCQ on or off, but others have seen a significant drop in performance with NCQ on.  I believe certain drives are particularly negatively affected by NCQ, and I have yet to see any evidence of significant improvement, in normal unRAID usage, by turning NCQ on.  Currently, the default therefore is to disable NCQ, since at best it has not been shown to be helpful, and at worst it really slows some transfer speeds down.

 

Your 'fill ratio' results do seem suspect.  I have found a significant difference in both reading and writing, between inner and outer tracks.  This is subjective only though, nothing like your excellent testing.

 

Easily the most surprising result is the difference in read speed between the Cache disk and a data disk, with the Cache disk being over twice as fast as data disks.  I'm wondering if any others have noticed anything like that?  I find it a little hard to believe.  If true, then it looks like an excellent candidate for optimization, something for Tom to take a look at.

 

Edit:  written while dead tired at 3am.  I completely misinterpreted and mixed up the charts.

Share this post


Link to post

I'm thinking of using my spare test UnRAID system as a "client" to make the first client-side tests. That way I could easily re-use the existing script and also to be free of any Windows peculiarities. I would get a cleaner baseline for Windows client tests. However I found this handy tool called TST10 for making telnet based scripting so I could run the full set automatically also on the client side (to set read ahead cache and others in between tests). Since I'm a little green on Linux I don't know yet how to accomplish the same from Linux to Linux. I suspect it should be quite trivial (or at least easier than from Windows).

 

You noticed that reading speed for Cache disk was ~120MB/s, for disk share ~115MB/s and for user share ~70MB/s. So only user shares are getting a performance hit (as expected due to the extra file system layer).

Share this post


Link to post

Easily the most surprising result is the difference in read speed between the Cache disk and a data disk, with the Cache disk being over twice as fast as data disks.  I'm wondering if any others have noticed anything like that?  I find it a little hard to believe.  If true, then it looks like an excellent candidate for optimization, something for Tom to take a look at.

I think you intended to remark about the difference in read speed between user-shares and disk-shares.  As henris said:

reading speed for Cache disk was ~120MB/s, for disk share ~115MB/s and for user share ~70MB/s. So only user shares are getting a performance hit (as expected due to the extra file system layer).

The "Cache Disk" has almost identical performance to the disk-shares  (120MB/s vs. 115MB/s). 

 

The user-file-system has always had about half the "read" speed of the disks themselves in my experience, as noted almost two years ago in this post:

http://lime-technology.com/forum/index.php?topic=965.msg6563#msg6563

 

At that point in time, with a read-ahead buffer size of 2048k, I was getting a read speed of 47.0 MB/s from /mnt/disk8 vs. 23.5 MB/s from /mnt/user.

 

Today, I'm seeing improved performance from the user file-system, but still less than reading directly from the disk.  48.6 MB/s vs. 37.6 MB/s

[pre]

root@Tower:/mnt/disk11# dd if=/mnt/user/Movies/WIMBLEDON.ISO of=/dev/null

10127896+0 records in

10127896+0 records out

5185482752 bytes (5.2 GB) copied, 138.092 s, 37.6 MB/s

root@Tower:/mnt/disk11# dd if=/mnt/disk11/Movies/WIMBLEDON.ISO of=/dev/null

10127896+0 records in

10127896+0 records out

5185482752 bytes (5.2 GB) copied, 106.776 s, 48.6 MB/s

[/pre]

 

Joe L.

 

Share this post


Link to post

That will teach me to analyze charts while dead tired at 3am!  I was thinking of the *Write* speed difference (in Chart 6g and others) of about 100MB/s for the Cache drive, and about 46MB/s for the data drives, and inexplicably (apart from being tired and stupid) completely forgetting about the 'parity penalty'.

Share this post


Link to post

;D No problem, also the charts can be(well, are) a little bit exhaustive. I tried to make the charts as easile readable as possible but since there are several attributes to be changed the total number became too large even for a statistics fan like me. Since read-ahead cache size (above 512) or NCQ enabling/disabling didn't affect the internal performance I could make single overview chart with read and write speeds for disks and user share. This could be then directly embedded to the first post. I could also add the exact IOzone command line to run that test on any disk.

 

At that point in time, with a read-ahead buffer size of 2048k, I was getting a read speed of 47.0 MB/s from /mnt/disk8 vs. 23.5 MB/s from /mnt/user.

 

Today, I'm seeing improved performance from the user file-system, but still less than reading directly from the disk.   48.6 MB/s vs. 37.6 MB/s

From your previous posts I picked that you have atleast one PCI based setup, is this the same system as above? What type of disk you have? Even with PCI you should be able to get around 80-90MB/s if there is no other traffic in PCI bus. In case of external read speeds, and if the NIC is also attached to the PCI-bus, the throughput would be less than half of that.

Share this post


Link to post

Added two embedded summary charts for read and write in the first post for those unable to find the linked test results. Still haven't found time to finish client side testing properly. In the mean time I have added a HighPoint RocketRAID 2310 and few new disks to the server. Everything is working very nicely.

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.