March 17, 201016 yr Rather than hijack a thread, I'm creating another. I'm also looking to build a very high-performance unraid server for primary use, as well as a cheapie for backup use. The cheapie is easy and can be scrounged from old parts; the high performance always-on one is trickier. Primary use case is movie storage / large files. The recent kernel changes seem to make this a practical reality, and I'd like to use this thread to coalesce knowledge as to how to do it. Goals: Moderate number of drives (6-8 to start) and flexibility regarding this to go up to "really big" Low power use Fast (40mb/sec write); at this level I can stomach getting rid of my hw raid-5. Resliency High density (2tb drives; 1.5tb at minimum); lower density exposes you to greater risk (more spindles) and cost--I believe it is a false economy Automated fan-control I worry about later. It's not altogether obvious what will accomplish this; best tip I've gotten is fast drives, on a good MB. Core i3 seems to actually be an unusual beast; tons of CPU, great available south bridge (more or less ICH10: 6 sata, good reported unraid speed), budget price because of "cut features" (no firmware-raid), and atom-like power usage. Yesterday I bought a Fry's core i3-530 / MSI h55m-e33 combo for $150. Requires DDR3, unsure if it can take ECC (unfortunate if not; mixed reports). Bottlenecks / Limiters: PCI-E bandwidth; must be shared by all connected HBAs Drive rotational speed (parity and/or data) Drive on-board cache (parity and/or data) As for drives, there don't seem to be many acceptable choices at the moment: DriveRPMCachePrice (as of 3/15/2010)Notes Caviar Black 2tb720064mb$270Ideal but expensive. WD GP EARS590064mb$150Might be price/perf sweet spot Samsung F3E5 (ecogreen)540032mb$180Unconfirmed: disappointing performance Hitachi 7k200 2tb720032mb$150May not perform any better than WD EARS b/c of 32mb cache WD, Hitachi, and Seagate seem to have shoddy quality control at the moment (25% failure rates). Open issues: How does unraid handle TLER (specifically, when it tries to read/write a block, and the drive takes up to 2 mins to respond)? Has the typical ~6 second limit been softened in the driver? I would like to try to benchmark unraid using a pair of the caviar black drives, but the cost is prohibitive for a pure experiment as this is beyond my pain point for purchasing. I currently have some older 1.5TB Seagate 7200rpm drives (unsure of cache) I will try.
March 17, 201016 yr probably go with hitachi deskstar 7k2000's. these give me the 30-40MB/sec write speeds which seem to be tops for most unraid uders. a cache drive would be faster though. i did a paritycheck on my unraid which are 7k2000's and the final speed was 95MB/sec. the wd blacks are very good for single sequential operations, but the deskstar shares the same multi io capability that the ultrastars have. while theyr eslower at 120MB vs the 150MB that blacks can do, when you start performing mutliple io operations the deskstars are faster. not qute sure you need thermal fan control, hdds get away with the lowest amount of airflow. as long as air is moving they stay cool enough. also, i3 is a cpu, the p/h55 provides teh ports and features laong with the motherboards, not the cpu.
March 17, 201016 yr Author Those hitachis are reported to be unbelieveably loud, and have similar, high, failure rates as the commodity seagate/wd. Noise control would be nice, but I suppose it's secondary to my other goals. Failure rate, however, tends to be a deal killer. 32mb cache. I've read other threads that theorize that the WD 5900rpm drives perform on par (for unraid) with 7200 rpm drives because of the 64mb cache. I'd rather have the WD in that case.
March 17, 201016 yr Aren't the WD Intellipower drives (EADS/EARS) spinning at 5400RPM? Its the Seagate LP 5900 drives that spin at 5900RPM.
March 17, 201016 yr Author Not sure. Marketing garbage. WD deliberately obscured the rotational speed of their drives and permitted this "dynamic adjustment" meme to spread. Scummy. Serious journalists/reviewers dug in when the products were new and found out the speed was actually 5900rpm. At some level, I don't care. I only care how it performs in Unraid.
March 17, 201016 yr I am almost positive that the WD greens are spinning at 5400RPM. If you have a link that says otherwise I would love to read it. Silent PC Review did a comparison and the drives were supposedly running at 5400RPM. I run 2 Hitachi 2TB drivers in my server and have had no problem with them, noise or otherwise. They precleared for 3 cycles and did not show any signs of problems.
March 17, 201016 yr Author One problem that we face in this is how to quantify performance objectively. I may end up doing it on my own by buying a slew of random drives (worst case scenario, to use in my low-perf backup array). I'm toying with the idea of building a simple web app to collect and aggregate configurations so that conclusions can be drawn based on stats. The problem will be this: are there enough people here that are willing to enter configurations, such that we can draw conclusions? Anecdotes suggest the WD greenpower 2tb performs far beyond what its rotational speed would suggest. I have 2 on their way to my door, so we shall find out (if they're not DOA).
March 17, 201016 yr Anecdotes suggest the WD greenpower 2tb performs far beyond what its rotational speed would suggest. I have 2 on their way to my door, so we shall find out (if they're not DOA). Any drive will look good if the data being read is cached. (because the drive is not supplying the data) You are purposely being misled by marketing numbers. For many unRAID users it is the sustained throughput that is going to determine their array's performance. Not some "burst" mode that has data in memory still waiting to be written to the disk. That rate is somewhere near 60-100 MB/s for modern drives. It is faster on the outer tracks where bit density is greater per revolution, and less on inner tracks where there is less room sectors. (Let's pretend that 70MB/s is average) In one minute I could read 70*60 MB, or 4200MB (4.2Gig) I think SATA is limited to 300MB/s, even if the disk could be read faster. Many unRAID users write huge files, far bigger than any memory buffer. In the same way, when checking parity we read the entire disk linearly, from start to end, and no "cache" will help, since we do not go back and re-read what we've previously read. If anything, a cache slows you down, since you're constantly searching to see if the data being requested is in the cache, and it never is. All that said, if I invented an infinitely dense recording technology that could store 1000TB in a single track on a disk's surface, that disk could spin at 1 RPM and I'd have read/write performance far better than any currently available disk spinning at 7200 RPM, where the data density per track is much lower. On my theoretical disk, in one minute I can read 1000TB, even though the disk rotated only once. The tradeoff is the seek time to a specific block is terrible. It could be as long as 1 minute to get to a specific block, and if used in unRAID, another minute to get back to it before it could be written to after reading it. (The 7200 RPM drive can get back to a specific block in 1/7200th of a second) In general, faster rotational speed disks will perform better than slower rotational speed disks using the same technology. The internal cache only helps a tiny bit so it can attempt to queue the request for data better as the disk is spinning under the read/write heads. In interpreting the disk manufacturer's specs, look for sustained read and sustained write speeds, in addition to rotational speed (the real limit on how fast you can write to an unRAID array) With luck your new drives will arrive safely, and work well. as you said, we'll soon find out. Joe L.
March 17, 201016 yr probably go with hitachi deskstar 7k2000's. these give me the 30-40MB/sec write speeds which seem to be tops for most unraid uders. a cache drive would be faster though. i did a paritycheck on my unraid which are 7k2000's and the final speed was 95MB/sec. For performance comparison, my parity check using 3 2TB WD Green (EADS) (1 parity+2 data) has a final speed of 75MB/sec.
March 17, 201016 yr Author probably go with hitachi deskstar 7k2000's. these give me the 30-40MB/sec write speeds which seem to be tops for most unraid uders. a cache drive would be faster though. i did a paritycheck on my unraid which are 7k2000's and the final speed was 95MB/sec. For performance comparison, my parity check using 3 2TB WD Green (EADS) (1 parity+2 data) has a final speed of 75MB/sec. What typical throughput do you get while transferring large files--read speed? Write speed? Are you on the latest version (4.5.3)?
March 17, 201016 yr I am running the emhttp/shfs from 4.5.3. My unRAID server is running Gigabit NIC on a full Slackware-Current distro (13.1 if you will) which is running Linux kernel 2.6.33. I have an upgrade to 2.6.33.1 kernel scheduled for tonight. I will try to fit in some performance tests this week and let you know the results. The parity check starts off with speeds of ~ 100MB/sec. I've been very happy with the balance of power and performance. My usage pattern is very few network writes with high network reads. The network read speeds are plenty fine for playing 1080p x264s.
March 17, 201016 yr Author The parity check starts off with speeds of ~ 100MB/sec. I've been very happy with the balance of power and performance. My usage pattern is very few network writes with high network reads. The network read speeds are plenty fine for playing 1080p x264s. Don't kill yourself, but every data point would be useful. What this community seems to be missing (probably because the high speed is new/recent) is a "recipe" for constructing the highest-possible-speed-Unraid. I am willing to do some organizational work if it gets me that recipe I would guess Parity checking is probably indicative of read speed, or maybe even less-than-read speed, if the pci-e bus is choked for bandwidth. I'm more concerned about write speed, and how to push that up as high as it can go (absent a cache drive). I'm willing to do driver work to get it.
March 17, 201016 yr Author I am running the emhttp/shfs from 4.5.3. My unRAID server is running Gigabit NIC on a full Slackware-Current distro (13.1 if you will) which is running Linux kernel 2.6.33. I saw that you mentioned this before. How did you do this? Manually plucking the driver and pieces and setting it up? What was the purpose--to increase utilization of the NAS box overall (running daemons, etc)?
March 17, 201016 yr Those hitachis are reported to be unbelieveably loud, and have similar, high, failure rates as the commodity seagate/wd. Noise control would be nice, but I suppose it's secondary to my other goals. Failure rate, however, tends to be a deal killer. 32mb cache. I've read other threads that theorize that the WD 5900rpm drives perform on par (for unraid) with 7200 rpm drives because of the 64mb cache. I'd rather have the WD in that case. loud? i cant hear them over the 22 hdds in my main pc if you want the fastest speed, youll probably want to use a fusion io drive as your parity disk
March 17, 201016 yr I'm more concerned about write speed, and how to push that up as high as it can go (absent a cache drive). I'm willing to do driver work to get it. Why the refusal to use a cache drive? I consistently see 50 - 60 mb/s write speeds to my older, slower cache drive. My recent tests with SSD drives pushed that up to over 70 mb/s. That's nearly double the fastest speeds other users have reported without a cache drive. If you are looking for outstanding write performance, use a cache drive. No amount of tweaking will come even close to the speeds a cache drive will give you. If you are concerned with your data sitting on the cache drive unprotected for a time, you can edit the mover script to run more often, say once per hour.
March 18, 201016 yr I only had time to run some rough tests locally on the server using a 50% filled disk, writing a 8.5GB file measured 31.0 MB/s, reading measured 80.2 MB/s. Using a 10% filled 1TB WD FALS unprotected drive measured 80.3 MB/s writes, and 97.4 MB/s reads. Some items to note... Writing to a lesser filled drive will result in higher writes and reads. Writing or reading an image that is not twice the size of physical memory installed will produce results that score significantly and artificially higher. If you're after absolute performance, grab a cheap Dell PERC 5i, flash it to MSI firmware and run RAID5/6. If you're after a balance of performance and power for mostly media retrieval, give unRAID a shot. If you want the highest performance with unRAID, use a cache drive. If a cache drive isn't suitable, then use 7200 RPM drives for parity AND data. dd if=/dev/zero of=/mnt/disk1/Media/test.wtf bs=4194304 count=2022 8480882688 bytes (8.5 GB) copied, 273.619 s, 31.0 MB/s dd of=/dev/null if=/mnt/disk1/Media/test.wtf bs=4194304 count=2022 8480882688 bytes (8.5 GB) copied, 105.757 s, 80.2 MB/s ~~~ dd if=/dev/zero of=/cache/.downloads/extracted/test.wtf bs=4194304 count=2022 8480882688 bytes (8.5 GB) copied, 105.671 s, 80.3 MB/s dd of=/dev/null if=/cache/.downloads/extracted/test.wtf bs=4194304 count=2022 8480882688 bytes (8.5 GB) copied, 87.1096 s, 97.4 MB/s
March 18, 201016 yr Author I'm more concerned about write speed, and how to push that up as high as it can go (absent a cache drive). I'm willing to do driver work to get it. Why the refusal to use a cache drive? I consistently see 50 - 60 mb/s write speeds to my older, slower cache drive. My recent tests with SSD drives pushed that up to over 70 mb/s. That's nearly double the fastest speeds other users have reported without a cache drive. Maybe you're right, and this is all somewhat moot. I had been initially avoiding it because it chews up a SATA port, and adds an always-on drive (heat, power, space). My needs put me at about 10 TB, which is 6-8 drives, depending. This is an easy number to deal with; but it's just short of the cusp of inexpensive expandability. I don't need 20 drives, but in 2 years I might need 10. But maybe dealing with an SSD + a cheap 2-port PCI-e adapter actually would be far cheaper (and better) than the alternative. I'll look into it. My preference, anyway, would be to populate with slower, lower power drives and mitigate cooling concerns (during parity checks in particular) signficantly. I know, I know: "Buy a HW raid-5!!!". I actually own one, and want to get rid of it in favor of Unraid, because almost every characteristic of Unraid is a better fit for my needs than HW raid-5.
March 18, 201016 yr Author Why the refusal to use a cache drive? I remember now, more of my rationale: Because it's another moving part that can break. There's a reason that I use Teracopy to move files - it verifies (by hash) that a file made it to the destination unchanged. If I use a cache drive, I'm taking a risk that the file makes it to the cache drive unchanged, but breaks during the move. You may think that's crazy, but it's not. Move enough files and it will happen to you, and you will be shocked and surprised, and then realize I'm right ;-) Is the mover part a script? If I can enhance it to copy-verify-delete, that would solve it.
March 18, 201016 yr Why the refusal to use a cache drive? I remember now, more of my rationale: Because it's another moving part that can break. There's a reason that I use Teracopy to move files - it verifies (by hash) that a file made it to the destination unchanged. If I use a cache drive, I'm taking a risk that the file makes it to the cache drive unchanged, but breaks during the move. You may think that's crazy, but it's not. Move enough files and it will happen to you, and you will be shocked and surprised, and then realize I'm right ;-) Is the mover part a script? If I can enhance it to copy-verify-delete, that would solve it. The "mover" is a script, and it uses rsync to perform the move and rsync does verify via checksum that the file move from the cache drive to the protected array is complete and accurate. You can find the "mover" script at /usr/local/sbin/mover The risk of using a cache disk is that of the cache disk failing between the copy of the file to it and the move of the file from it to the protected disk when the move script runs. Granted, the risk is small, but there is a risk. You might lose the file entirely in that situation if the drive were to die completely and stop spinning.
March 19, 201016 yr I know, I know: "Buy a HW raid-5!!!". I actually own one, and want to get rid of it in favor of Unraid, because almost every characteristic of Unraid is a better fit for my needs than HW raid-5. I think most of the people here have this same sentiment. Having moved from H/W or S/W raid arrays or NAS devices etc...
March 19, 201016 yr For me, perfection would be if Tom were able to implement some kind of software raid 1 into the cache drive. I know there are hardware devices you can purchase that would emulate this. But the people who have posted about that say you lose some of the unraid drive mngt features when you do that. If Tom were to incorporate the raid1 into unraid, seems like it would resolve that problem.
March 19, 201016 yr Author writing to a raid1 is slower than writing to a single drive. Are you sure about that? What are you basing that on? I actually don't believe that's the case (in any significant amount. If you tell me it dings 1 MB/sec in overhead or adds some small level of throughtputvariance, ok). When writing blocks out, there is no dependency between the drives from a driver's perspective. Ability to soft raid-1 the cache data would be ideal, yeah. Kills another sata port, but I'd probably still do it. I was more worried about corner (inflection point) cases inside the mover script: Start, but don't finish transfering a file, mover "moves" incomplete file, file xfer finishes, mover declares victory, deletes complete file. But, having looked at the mover script, I don't think that can happen. I'm going to test it a few times with big files though ;-)
March 19, 201016 yr writing to a raid1 is slower than writing to a single drive. It's the speed of your slowest drive, plus a small factor. Chances are it is still faster then writing to the parity protected array. Plus reads are usually faster because they are usually interleaved between two drives.
Archived
This topic is now archived and is closed to further replies.