Ultimate Unraid Server - The Sequel

[PhilH]

For any who are interested I will wait for the i7 processor I have on order to arrive and then give it a shot against the dual core Atom board with a pair of WD Raptor 300 drives installed (one parity and one data - I believe this is still one of the fastest rotating drives, so it will show us the best practical speed we could expect).  The i7 will have DDR3 tripple channel memory and the atom will have DDR2.  Everything else will be the same between the two systems.

 

Actually I think a more practical test is with Seagate 7200 RPM 1.5TB drives.

From what I've read, these have been benchmarked as fast, if not faster, then raptors.

In addition, I doubt people will ever put a raptor in an unraid system, but many will use the larger drives (5400,5900 & 7200 rpm).

 

Trainwreck? Hmm.. I did not think so. There was just a bit of tudes floating around.

The thread does not need to be locked. Part of the discussion had some points.

I would like to see Gary back up the theories with numbers from the familiar benchmarks when hardware is present. My own anecdotal tests revealed that a 1.6 and a 2.6 CPU provided the same throughput.

An Atom vs an i7 will be an interesting test.

 

I remember Tom saying that this was the nicest community he's had the pleasure of working with.

So for those who post/respond. Please re-read your replies and consider others before clicking that post button.

If there's any chance that someone may take offense to what or how something is said. you can be sure of a similar response back.

 

Someone once posted in another forum.  "Attack idea's, not people".  Consider that in the discussion.

 

Well said WeeboTech. This thread was starting to get a little dark. Maybe we need to lighten up a bit & discuss the benchmarks that GaryMaster is interested in.. 

[agw]

My own anecdotal tests revealed that a 1.6 and a 2.6 CPU provided the same throughput.

An Atom vs an i7 will be an interesting test.

 

Ditto - my testing was probably not as controlled as what GaryMaster has planned.  But replacing my BE-2400 2.3GHz dual core with a 1.6GHz single core had no measurable effect on parity speed or copy speed.

[boof]

If you took your old / current system and created a 'normal' raid out of the disks you would easily saturate your existing gig-e connection.

 

This should tell you your current system is fast enough to max out gige in terms of cpu and memory.

 

Switching to unraid leads to, by design, far far fewer IOPs and so worse performance. This isn't cpu or memory bound.

 

Memory and cpu can become important if you start using your unraid box for *other* things like vm's, torrent clients etc etc but thats outwith the core unraid storage requirements.

 

I'm an advocate of if you're building a new system then you may as well purchase with a view for the future and theres no harm buying a bit more power than you absolutely need presuming you're happy with the consequences in terms of power usage and price.

 

However if you have a perfectly good system already but are looking to improve unraid performance by throwing cpu and memory horsepower at it you will be dissapointed and may well regret the significant spend.

 

To improve unraid performance in your new machine the best you can do is buy the fastest disks possible and make sure you have the fastest bus available to each disk. So stuff your motherboard full of pci-e sata controllers to couple your fast disks to.

 

That will give you the improvement in performance you seek (although you'll still be capped by your network throughput. I do agree with you regarding the gigabit speeds quoted elsewhere. 120 megs a second from a gigabit link is easily do-able and I see it on a daily basis).

 

What might be a nice middle ground solution would be for you to keep your current system but add a high end SSD (intel X25 or similar) as a cache drive. This would give you immediate write speeds to the array at a very high level approaching your network limit. It would be moved onto your array at the lower speeds eventually but this would be 'in the background' and so your perception as a user would only be of the high speed writes.

 

Doing this would be significantly cheaper than buying a new system and would likely yeild you the same results perceptually.

 

It would be my approach in your situation.

 

I'm afraid I don't have numbers to give you. Nor the inclination to take the time to make a lab to do so. What you're hitting here is common sense about 'modern' disk speeds, cpu speeds etc. Numbers and figures bore out all these sentiments years ago but it's now perceived as common knowledge.

 

You can easily test it yourself by grabbing some spare disks putting them into your server and creating a true raid stripe and testing performance. I've comfortably had gigabit speed raid arrays on hardware similar to yours.

 

Your new server will certainly be more powerful but I don't think it will help with raw throughput. So long as you spend your money with your expectations set accordingly so as not to dissapoint..

[bubbaQ]

From what I've read, these have been benchmarked as fast, if not faster, then raptors.

 

It is possible.  I just purchased the latest WD Black 1TB (7200 RPM) for a desktop.  It benchmarks *faster* that a WD 150GB 10K Raptor in the same system.  Digging into the diagnostics, the WD Black has more cache 32M vs 16M) and the Black is ATA8-ACS compatible, whereas the Raptor is only ATA7.  I seem to recalling that the track buffering and read-ahead algorithms are different too.  The ATA8-ACS spec includes non-volatile write-cache in the drive (hybrids) but I could not find any documentation that states whether or not the WD Black incorporates this feature.

 

Read ahead, whole-track buffering, and write cache, can have more impact than rotation speed depending on how you use unRAID.  For example if you do a lot of random small reads/writes, rotation speed will make a bigger difference than cache or read-ahead (assuming modern seek times).  For large sequential reads/writes, read-ahead and cache will have a larger impact than rotational speed.... and this advantage will be magnified in unRAID, because of the way it works (read a sector, then write the same sector back).  Indeed, if non-volatile caching of writes is implemented, and the cache is large enough to hold several full tracks (or even full cylinders) then even with unRAID's read then write the same sector paradigm, write speed could increase dramatically.... and even approach single-drive performance.

 

If ATA8-ACS compatible drives large enough to be useful in unRAID are indeed shipping with nonvolatile cache, or even with volatile cache that is large enough and you are willing to use it for writes (turning write-through off), then anyone who does large-file reads/writes with unRAID should pay close attention to them.

[aiden]

Sorry to all for the offense.  Obviously it wasn't intended as such, more to underscore the drumbeat of locking the thread.

 

I'm not usually one to complain. I understand the above saying about beating a dead horse is a old one, but I could have done without the graphic. As a horse owner, I find it disturbing. In fact one of my horses (my personal horse is a grey) is pure white.

 

Again sorry to complain but imho it is in bad taste..

 

Phil

 

Phil, you're right, and I apologize.

[VampyreGTX]

Bubba.... what's the best way to determing latency?  When I ping my server from my desktop... I get a whopping 0ms response time.  Granted, it's through a Netgear Green 1Gb switch and running 5e for a total of about 20 feet (to the room directly upstairs where the unRaid box is...) So I'm guessing that better than expected?

 

Also, I was looking into adding a cache disk, so based on the discussions within here, a 10K would be a waste of money?  I was looking at a 360gb or so 10K, should I just settle for a smaller WD Black drive for that? I was also thinking a 2.5" 7200rpm drive as well, how would those compare?

[boof]

Also, I was looking into adding a cache disk, so based on the discussions within here, a 10K would be a waste of money?  I was looking at a 360gb or so 10K, should I just settle for a smaller WD Black drive for that? I was also thinking a 2.5" 7200rpm drive as well, how would those compare?

 

What is your target throughput? I dropped a 2-3 year old WD 500GB disk in mine as a cache disk and get sustained transfer rates off 60-64 megabytes per second across the gig-e link.

[bubbaQ]

Bubba.... what's the best way to determing latency?  When I ping my server from my desktop... I get a whopping 0ms response time. 

 

I believe that switch uses cut-through mode, which helps latency significantly.  But you can't have 0ms, because there has to be some finite amount of time so I am suspicious.  Try pinging the other way (from unRAID to your desktop) since the Linux ping will give you 4 decimal places.  Using my cut-through switch I can get about 0.18 ms.

 

Also when doing it from Linux, note the latency of the FIRST ping compared to the rest.  If there is a large variation, then your switch is probably doing cut-through (i.e. once it learns where the destination is, it will cut-through future packets after just reading the header).

[PhilH]

Sorry to all for the offense.  Obviously it wasn't intended as such, more to underscore the drumbeat of locking the thread.

 

I'm not usually one to complain. I understand the above saying about beating a dead horse is a old one, but I could have done without the graphic. As a horse owner, I find it disturbing. In fact one of my horses (my personal horse is a grey) is pure white.

 

Again sorry to complain but imho it is in bad taste..

 

Phil

 

Phil, you're right, and I apologize.

 

Apology accepted. I must apologize also. I was never upset with you. Just wanted you to know that..

 

Phil

 

P.S. You didn't need to remove your post. I had no problems with the words, just the graphic.. Before I owned horses I probably would have laughed about it. But since I have owned them they get into your blood. They are strange creatures.. 

[SSD]

Sorry I'm late to the fight (I mean debate) about unRAID performance.

 

Thought I'd make just a few comments:

 

1 - There are several references in here to PIII processors being plenty fast enough for unRAID.  Although true in some sense, and there are some users running them, the motherboards won't support PCIe (or PCI-X) required to add more disks without saturating the PCI bus.  Also, these CPUs and motherboards are getting old, and the risk of failure is high.  I would not recommend going earlier that Pentium 4 or AMD equivalent for a serious array.  Low end CORE2 (or AMD equivalent) CPUs are inexpensive enough for most.

 

2 - Saying that adding a faster CPU / memory subsystem will not speed up the unRAID server is not true at all.  If you have your unRAID server transcoding video, performing scientific simulations, or otherwise performing CPU intensive activities, these types of speed enhancement can and will improve performance.  But for core unRAID file reads and writes - it isn't going to do much.

 

3 - In the early days of PCs, Peter Norton wrote a book "Inside the IBM PC" (or some such) where he discussed performance.  He ran some benchmarks and proved that when trying to improve performance of a closed system, that SPEEDING UP the already fastest component yielded little performance improvement, while SLOWING DOWN the already slowest component yielded large performance degradation.  The section showed how important finding the slowest component was to improving performance.  With unRAID the disks themselves are a potential bottleneck, as are the bus speeds and network speed.  Even a modest CPU is not the performance limiter, as evidenced by the low CPU utilization through all common usage models of unRAID.

 

4 - With time, most here yawn when tweaking of unRAID performance is discussed.  It just doesn't make that much difference for normal day to day activities.  (The one exception has been array write performance, which has been the Achilles heel of unRAID until the most recent improvements).  Without fail, new users arrive on the scene asking about how to max out their servers to make them faster, and soon discover there are more important things to worry about on, what has become, their most valuable data asset.

 

5 - Final comment.  Those that defy conventional wisdom and follow their own logic have sometimes made unexpected and dramatic discoveries.  (Others have failed miserably and died as paupers). Although conventional wisdom would say that an exhaustive performance benchmark of core unRAID activity would be a useless exercise and waste of time, the inspired innovator may yet discover something incredibly useful in the process!

[VampyreGTX]

What is your target throughput? I dropped a 2-3 year old WD 500GB disk in mine as a cache disk and get sustained transfer rates off 60-64 megabytes per second across the gig-e link.

 

I'm looking for the fastest throughput possible for the initial write to the server.  I do have some older 300gb Maxtor's laying around, I may try those and see what I get.  Currently, my transfers run about 20-22'ish on most transfers w/parity so it's not too bad.  I'll add those just to get them into the user benchmarks page.  I've already added everything I can there.  Doing what I can for the community!  

 

[Also when doing it from Linux, note the latency of the FIRST ping compared to the rest.  If there is a large variation, then your switch is probably doing cut-through (i.e. once it learns where the destination is, it will cut-through future packets after just reading the header).

 

So do I just type Ping xxx.xxx.xxx.xxx as well into the terminal to initiate the ping from the server side?  I thought something was odd with just 0, but all 4 pings came across as 0.  I'll let you know how my server side pings respond tonight.

[330T]

Try using this drive.  Its about the fastest drive you can find.  I have one as my cache drive.  All my other drives are WD Green 1TB drives.

[Chris Pollard]

Also when doing it from Linux, note the latency of the FIRST ping compared to the rest.  If there is a large variation, then your switch is probably doing cut-through (i.e. once it learns where the destination is, it will cut-through future packets after just reading the header).

First ping the machine will have to arp for the mac address too so this can be misleading.

[VampyreGTX]

Try using this drive.  Its about the fastest drive you can find.  I have one as my cache drive.  All my other drives are WD Green 1TB drives.

 

If you look at Bubba's post quoted below , it sounds like the WD Black may be preferential for me as most of my files will be large BD and DVD .iso's:

 

From what I've read, these have been benchmarked as fast, if not faster, then raptors.

 

It is possible.  I just purchased the latest WD Black 1TB (7200 RPM) for a desktop.  It benchmarks *faster* that a WD 150GB 10K Raptor in the same system.  Digging into the diagnostics, the WD Black has more cache 32M vs 16M) and the Black is ATA8-ACS compatible, whereas the Raptor is only ATA7.  I seem to recalling that the track buffering and read-ahead algorithms are different too.  The ATA8-ACS spec includes non-volatile write-cache in the drive (hybrids) but I could not find any documentation that states whether or not the WD Black incorporates this feature.

[boof]

Try using this drive.  Its about the fastest drive you can find.  I have one as my cache drive.  All my other drives are WD Green 1TB drives.

 

If you look at Bubba's post quoted below , it sounds like the WD Black may be preferential for me as most of my files will be large BD and DVD .iso's:

 

From what I've read, these have been benchmarked as fast, if not faster, then raptors.

 

It is possible.  I just purchased the latest WD Black 1TB (7200 RPM) for a desktop.  It benchmarks *faster* that a WD 150GB 10K Raptor in the same system.  Digging into the diagnostics, the WD Black has more cache 32M vs 16M) and the Black is ATA8-ACS compatible, whereas the Raptor is only ATA7.  I seem to recalling that the track buffering and read-ahead algorithms are different too.  The ATA8-ACS spec includes non-volatile write-cache in the drive (hybrids) but I could not find any documentation that states whether or not the WD Black incorporates this feature.

 

330T posted the *velociraptor* not the normal raptor so slightly different.

 

I have no idea on the performance comparison though, but it is a different drive.

[VampyreGTX]

Hahaha.... nice catch, the raptor is pre 2008, velociraptor is post 2008... I'll have to look into those, but not sure if the price premium is worth it for a 10K over a 7200. especially since the velociraptor is supposed to be 35% faster than the raptors.  I may just settle for the cheaper 7200's and deal with that.  there's a limit to the price I'm willing to pay for a small performance gain.

[bubbaQ]

330T posted the *velociraptor* not the normal raptor so slightly different.

 

Correct -- my Raptor is not a VRaptor, which I believe is ATA8, not ATA7.

 

But the 16K cache on the VRaptor compared to the 32MB on the Black, as well as the differences in caching algorithms, may be a bigger factor than the rotational speed.

[purko]

Bubba.... what's the best way to determing latency?  When I ping my server from my desktop... I get a whopping 0ms response time. 

 

I believe that switch uses cut-through mode, which helps latency significantly.  But you can't have 0ms, because there has to be some finite amount of time so I am suspicious.  Try pinging the other way (from unRAID to your desktop) since the Linux ping will give you 4 decimal places.

 

Here's my ping "the other way":

(The switch is a WRT310N)

root@Tower:~# ping 10.22.33.5

PING 10.22.33.5 (10.22.33.5) 56(84) bytes of data.

64 bytes from 10.22.33.5: icmp_seq=1 ttl=128 time=0.000 ms

64 bytes from 10.22.33.5: icmp_seq=2 ttl=128 time=0.000 ms

64 bytes from 10.22.33.5: icmp_seq=3 ttl=128 time=0.000 ms

64 bytes from 10.22.33.5: icmp_seq=4 ttl=128 time=0.000 ms

64 bytes from 10.22.33.5: icmp_seq=5 ttl=128 time=0.000 ms

64 bytes from 10.22.33.5: icmp_seq=6 ttl=128 time=0.000 ms

64 bytes from 10.22.33.5: icmp_seq=7 ttl=128 time=0.000 ms

64 bytes from 10.22.33.5: icmp_seq=8 ttl=128 time=0.000 ms

64 bytes from 10.22.33.5: icmp_seq=9 ttl=128 time=0.000 ms

64 bytes from 10.22.33.5: icmp_seq=10 ttl=128 time=0.000 ms

64 bytes from 10.22.33.5: icmp_seq=11 ttl=128 time=0.000 ms

64 bytes from 10.22.33.5: icmp_seq=12 ttl=128 time=0.000 ms

64 bytes from 10.22.33.5: icmp_seq=13 ttl=128 time=0.000 ms

64 bytes from 10.22.33.5: icmp_seq=14 ttl=128 time=0.000 ms

64 bytes from 10.22.33.5: icmp_seq=15 ttl=128 time=0.000 ms

^C

--- 10.22.33.5 ping statistics ---

15 packets transmitted, 15 received, 0% packet loss, time 14000ms

rtt min/avg/max/mdev = 0.000/0.000/0.000/0.000 ms

root@Tower:~#

 

[Chris Pollard]

I can't find the article now but we had this on some Redhat servers at work....  anything below 10ms would report 0.000ms maybe this is something similar.  Seem to remember it being something to do with 64bit registers as it only happened on their x64 boxes.  Also here is the effect of having to arp.  Cut through is disabled :-

 

root@ubuntu:~$ ping 192.168.0.2
PING 192.168.0.2 (192.168.0.2) 56(84) bytes of data.
64 bytes from 192.168.0.2: icmp_seq=1 ttl=255 time=3.14 ms
64 bytes from 192.168.0.2: icmp_seq=2 ttl=255 time=0.335 ms
64 bytes from 192.168.0.2: icmp_seq=3 ttl=255 time=0.420 ms
64 bytes from 192.168.0.2: icmp_seq=4 ttl=255 time=0.302 ms
64 bytes from 192.168.0.2: icmp_seq=5 ttl=255 time=0.410 ms
^C
--- 192.168.0.2 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4001ms
rtt min/avg/max/mdev = 0.302/0.922/3.145/1.112 ms


root@ubuntu:~$ ping 192.168.0.2
PING 192.168.0.2 (192.168.0.2) 56(84) bytes of data.
64 bytes from 192.168.0.2: icmp_seq=1 ttl=255 time=0.384 ms
64 bytes from 192.168.0.2: icmp_seq=2 ttl=255 time=0.357 ms
64 bytes from 192.168.0.2: icmp_seq=3 ttl=255 time=0.281 ms
64 bytes from 192.168.0.2: icmp_seq=4 ttl=255 time=0.349 ms
^C
--- 192.168.0.2 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 2997ms
rtt min/avg/max/mdev = 0.281/0.342/0.384/0.044 ms


root@ubuntu:~$ sudo arp -d 192.168.0.2
root@ubuntu:~$ ping 192.168.0.2
PING 192.168.0.2 (192.168.0.2) 56(84) bytes of data.
64 bytes from 192.168.0.2: icmp_seq=1 ttl=255 time=3.15 ms
64 bytes from 192.168.0.2: icmp_seq=2 ttl=255 time=0.308 ms
64 bytes from 192.168.0.2: icmp_seq=3 ttl=255 time=0.484 ms
64 bytes from 192.168.0.2: icmp_seq=4 ttl=255 time=0.314 ms
^C
--- 192.168.0.2 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3001ms
rtt min/avg/max/mdev = 0.308/1.065/3.156/1.209 ms

[WeeboTech]

Try using this drive.  Its about the fastest drive you can find.  I have one as my cache drive.  All my other drives are WD Green 1TB drives.

 

Hahaha.... nice catch, the raptor is pre 2008, velociraptor is post 2008... I'll have to look into those, but not sure if the price premium is worth it for a 10K over a 7200. especially since the velociraptor is supposed to be 35% faster than the raptors.  I may just settle for the cheaper 7200's and deal with that.  there's a limit to the price I'm willing to pay for a small performance gain.

 

Ha,  I thought there was no limit?  I.E. this is the Ultimate unRaid Server - The Sequel thread. <GRIN>

 

Kidding aside, newegg has the VelociRaptor cheaper then the original link posted.

 

In addition there is a promo code today, so the cost would be $189.

 

http://www.newegg.com/Product/Product.aspx?Item=N82E16822136322

 

As I made mention of earlier, the outer tracks of a 1.5TB 7200 RPM or a 2TB 7200 RPM drive can provide up to 120Mb/s.  In this case it may be more cost effective for future needs to go that route with a cache drive.

[yp_1]

A high load average does not mean that it's CPU bound. It's mostly waiting on I/O to complete.

 

I willl bite...

According to my Cacti graphs, system portion of CPU time is 20-30% at the time I copy data to a user share.

I can see shfs is constantly on the top, and there is no I/O wait or it is negligible (a few percent).

 

[VampyreGTX]

FYI, to follow up on the ping, my average was .104, not shabby!

 

[purko]

I am still wondering as to why my ping times show as zero.

[GaryMaster]

I really wasn't planning on coming back to this forum, but all of the activity compelled me to look at what was going on.  Thanks for all of those who have come back with voices of reason and encouragement.  It may not have been noticed by many, but I was extremely put off by this early response and took a level of offense that I should not have:

 

So far, I haven't learned much

 

That, is an understatement.

 

My brash responses thereafter just added fuel to the fire.  I appologize to everyone for my over reacting.  It was childish.  And again, thanks to everyone who commented on innovation and learning something from the excercise.  None of you know me, but I am very passionate about innovation.  Many of you will still think I'm a quack.  I do hold 5 US patents for a fortune 50 company, so I'm not completely crazy!  (Disclaimer:  Not saying this to pat myself on the back - so please don't flame me anymore!)

[GaryMaster]

Try using this drive.  Its about the fastest drive you can find.  I have one as my cache drive.  All my other drives are WD Green 1TB drives.

 

Note that the 300GB Velociraptor is the drive I mentioned I would be using on my testbed.  I have two of them sitting here that I can use.  One will be Parity and the other the Data drive.