So i've been struggling with trying to get a baseline on what's considered the "average" read/write speed out of UnRaid.

 

I'm splitting this off of CaptainTivo's Thread .

 

I get 25 MB/s to perhaps 35 MB/s when writing to the array. Reading - I max out my gigabit connection, easily. Even if I have the parity drive disabled. Here are two different inputs of hdparm and dd like OP did:

 

One is from running the ever popular TAMs machine with 3x AOC-SAS cards.

 

root@Tower:~# hdparm -tT /dev/sdd

 

/dev/sdd:

Timing cached reads:  1812 MB in  2.00 seconds = 906.62 MB/sec

Timing buffered disk reads: 562 MB in  3.01 seconds = 186.72 MB/sec

root@Tower:~# dd if=/dev/zero of=/mnt/disk1/test.dd count=8192000

8192000+0 records in

8192000+0 records out

4194304000 bytes (4.2 GB) copied, 61.6544 s, 68.0 MB/s

root@Tower:~#

 

that's my test array. My production array is actually pitiful. This is a VM, with a Dell H310 HBA; the drives are connected to a Norco ss-400.

 

root@Rigel:~# hdparm -tT /dev/sdg

 

/dev/sdg:

Timing cached reads:  17096 MB in  1.99 seconds = 8570.08 MB/sec

Timing buffered disk reads: 370 MB in  3.00 seconds = 123.22 MB/sec

root@Rigel:~# dd if=/dev/zero of=/mnt/disk1/Videos/test.dd count=8192000

8192000+0 records in

8192000+0 records out

4194304000 bytes (4.2 GB) copied, 149.959 s, 28.0 MB/s

 

the above is to a wd green drive.

 

here's to a 7200 spinner on the same machine:

root@Rigel:~# hdparm -tT /dev/sdb

 

/dev/sdb:

Timing cached reads:  16526 MB in  2.00 seconds = 8278.69 MB/sec

Timing buffered disk reads: 318 MB in  3.01 seconds = 105.79 MB/sec

root@Rigel:~# dd if=/dev/zero of=/mnt/disk7/test.dd count=8192000

8192000+0 records in

8192000+0 records out

4194304000 bytes (4.2 GB) copied, 119.682 s, 35.0 MB/s

 

Both are running running latest 5.0.4.

 

The Production machine (Rigel) has  a 2TB 7200RPM Parity drive (Toshiba, w. 64MB Ram);

The Test Machine (Tower) has a 4TB 5900RPM Parity drive.

 

now that's dd tests... but the weird thing is when i'm copying over the network, the test machine is actually slower than the production machine. by about 10MB/s. i'm guessing this has to do with the spindle speed on the parity disk

 

 

 

 

just adding more info from the previous thread:

 

i've tried:

 

[*]different HBA PERC H310 and BR10i

[*]increased VM ram from 4GB to 8GB with nothing else running

[*]running unraid on baremetal (instead of inside VM).

[*]changing BIOS settings to ensure AHCI (for built-in ports which I am not using

 

None of these yielded any discernible difference outside of a +/- 3MB/s so I didn't really record the results.

 

So we are left with: Motherboard/PCI-e issue (ports are set to run at x16, x8 and x8; hba being in x16 slot), Parity Drive issue (2TB, 7200 with 64MB cache Link ) or the  Norco SS-400

 

Anything else I can/should check ?

 

I'm tempted to do a survey to see what people are getting and what's acceptable, etc. there are so many combinations and what nots that a centralized "this is what my speed is" would be handy. Is there interest in something like this?

 

See here: http://lime-technology.com/forum/index.php?topic=9989.msg95514#msg95514

 

Version 5.0.4 includes a write option that requires all drives to be read but is much faster. With all drives spinning write speed approaches the parity check speed.

I get 25 MB/s to perhaps 35 MB/s when writing to the array. Reading - I max out my gigabit connection, easily.

 

Those are reasonable write speeds ... 25MB/s to a bit over 40MB/s is typically what I've seen, depending on the areal density of the drives; the rotational speed; and where on the drive the write is taking place (outer cylinders are much faster than inner cylinders).

 

As already noted, remember that there are 4 disk operations required for each write -- two reads and two writes -- and a full rotation of the drives is needed between the read and the write.

 

v5.04 supports a "reconstruct" write mode, whereby the reads are no longer required, as it reconstructs the parity by reading all of the drives for each write => so instead of 2 disk I/O's on the disk being written to and 2 disk I/O's on the parity disk, there is one disk I/O on EVERY disk in the array [a write to the parity disk and the disk being written to; a read for all other disks].    If all the drives are already spinning, this is notably quicker than the normal read/modify/write cycle used for writes to the array.

 

 

thanks - i did see the release note about this - but also saw that for larger arrays, there's a point of diminishing returns. I have 8 drives in my array now, and within a week or so, will increase to 12.

 

i'm still confused about why the Test array ran a DD faster, and a Parity check faster (160MB/s on Test vs about 80MB/s on PROD) but still transfers are faster to the Prod array.

 

The reason i'm trying to nail this down now is when i do increase to the 12 drives, I will be using a SAS Expander - where i expect a performance hit. far less moving pieces if I can get a solid baseline before that would be great.

 

 

Should the output of my DD be higher? I'm still confused on why one system reports 68MB/s while the other reports about half that. The one reporting 35MB/s actually has a faster (7200RPM) parity drive as compared to the 68MB/s machine, which has a parity drive that spins at 5900RPM.

 

 

Your 5900rpm 4TB drives are almost certainly 1TB/platter drives, while the 2TB drive on your other array likely has a lower areal density, so even though it's spinning faster, it's got a slower sustained access rate.    This is why parity checks will run faster on the array with the 5900rpm drives.

 

But network writes are small enough that the driving factor isn't sustained transfer rate, but access times for the 4 required disk accesses for each write (2 reads, 2 writes) ... so your 7200rpm parity drive results in faster writes on that system.

 

When you're testing drive speeds with DD, the higher sustained access speed of the drives with the 1TB platters results in notably better speeds (as you've noted).

 

 

I've found that you will get some benefit to write speed if you adjust the disk tunable settings from the defaults. I discovered this while trying out the unraid-tunables-tester script. While the test itself focuses on parity check speed you will see an increase in the write speeds too. I'm currently seeing write speeds in 35-40MBps range with WD Green 3TB disks.

 

Here is the wiki link to the subject: http://lime-technology.com/wiki/index.php/Improving_unRAID_Performance#User_Tunables

And some more explanation on the topic: http://lime-technology.com/forum/index.php?topic=15224.msg142767#msg142767

 

I'm using the following values on all my systems (4GB or more memory on all of them).

Tunable (md_num_stripes): 4096

Tunable (md_write_limit): 2048

Tunable (md_sync_window): 1280

The tunables adjust the number of stripes available for both read and write buffering.  Since every write involves 2 reads and 2 writes, clearly the efficiency of buffering can impact the overall speed (in conjunction with the disk's own buffers).

 

Especially when writes are of consecutive sectors, the likelihood of a buffer "hit" on a read can be fairly high ... effectively eliminating one or two disk operations from many writes (i.e. one or both of the reads).    This is likely why you're seeing a notable improvement in overall write speeds to the array.  Consecutive writes may also be buffered, which, in conjunction with AHCI, can also be accounting for some of the speed gains.

 

 

 

I've found that you will get some benefit to write speed if you adjust the disk tunable settings from the defaults. I discovered this while trying out the unraid-tunables-tester script. While the test itself focuses on parity check speed you will see an increase in the write speeds too. I'm currently seeing write speeds in 35-40MBps range with WD Green 3TB disks.

 

Someone more knowledgeable might explain why these settings affect the write speed. Here is the wiki link: http://lime-technology.com/wiki/index.php/Improving_unRAID_Performance#User_Tunables

 

I'm using the following values on all my systems.

Tunable (md_num_stripes): 4096

Tunable (md_write_limit): 2048

Tunable (md_sync_window): 1280

 

Thanks - I always thought those were only tweaked if/when you suffer massive slowdowns (like that one where people were getting like 12KB/s writes).

Your 5900rpm 4TB drives are almost certainly 1TB/platter drives, while the 2TB drive on your other array likely has a lower areal density, so even though it's spinning faster, it's got a slower sustained access rate.    This is why parity checks will run faster on the array with the 5900rpm drives.

 

But network writes are small enough that the driving factor isn't sustained transfer rate, but access times for the 4 required disk accesses for each write (2 reads, 2 writes) ... so your 7200rpm parity drive results in faster writes on that system.

 

When you're testing drive speeds with DD, the higher sustained access speed of the drives with the 1TB platters results in notably better speeds (as you've noted).

 

Thanks Gary - this makes a lot of sense. In the next day or so, I am going to move the 4TB, 5900RPM disk to my production array; i'll run more tests on that configuration and see how the numbers look.

 

Dying for a decent 4TB at 7200 RPM - but not looking likely anytime soon.

Dying for a decent 4TB at 7200 RPM - but not looking likely anytime soon.

 

Both the 4TB WD Black and the 4TB WD RE series are 7200rpm drives ... but they're both 5-platter, 800MB/platter densities.    I'm sure by "decent 4TB" you mean one with 1TB/platter areal density ... which I've also been watching for 

 

Personally, I'm more anxious for the 5TB drives to come out 

(What would REALLY be nice is if they were 1.25TB/platter units ... but I'm anticipating they'll be 5-platter, 1TB/platter units.)

 

Dying for a decent 4TB at 7200 RPM - but not looking likely anytime soon.

 

Both the 4TB WD Black and the 4TB WD RE series are 7200rpm drives ... but they're both 5-platter, 800MB/platter densities.    I'm sure by "decent 4TB" you mean one with 1TB/platter areal density ... which I've also been watching for 

 

Personally, I'm more anxious for the 5TB drives to come out 

(What would REALLY be nice is if they were 1.25TB/platter units ... but I'm anticipating they'll be 5-platter, 1TB/platter units.)

 

ah didn't realize that about the WDs - good to know. the 5's and 6's wont be here for at least a year; my new 24-bay case will have to deal with 4's until i fill 'em up (or it is no longer economically viable to buy that "small").

the 5's and 6's wont be here for at least a year ...

 

I think the 5's will be available much quicker than that.  WD had projected 5TB WD Reds by "4th quarter 2013" ... so they've already missed that target.    Not sure just how late they'll be, but I doubt it will be more a small number of months.    Seagate's announcement for their 5TB drives just said "in 2014" ... so that may indeed be the end of the year.

 

The helium-filled 6TB HGST drives are already shipping for large data centers, but the pricing is well beyond what a typical consumer would be willing to pay