Poor performance with my new Lime server


Recommended Posts

  • Replies 83
  • Created
  • Last Reply

Top Posters In This Topic

If the logic stated by Joe L. previously on this thread about parity writes is correct there should no performance gain if you only change the parity drive; since reads/writes are performed simultaneously to data and parity disk, the slower one dictates overall write performance.

 

Not true, adding a 7200 RPM parity drive will increase performance. Unraid writes the byte to the array disk being written to, at them same time performs a read of the same byte on the parity disk, xors the result to calculate parity and then writes the byte to the parity disk. So the parity disk does a read then a write to the same sector, meaning the parity disks platter needs to do a whole platter rotation (after the onboard cache is exhausted).

unRAID reads both the data disk being written to, AND the parity disk being written to. It is not just the parity disk that must rotate a full revolution, both do.

With the data disks 5400/5900 RPM or "green disks" probably wont affect performance dramatically as long as they can sustain good sequential write performance. Data disks are either being read from or written too, not both like a parity drive is. Ideally we (unRaid) would want 15K parity drives to even the balance and match a 5400 RPM data drive.

Your statement is incorrect.

 

Let's take it on a single bit level.   We want to write a single bit to the 1000000th bit on a data drive.   For argument sake, we want to write a 0.  Currently, it is a (well, we don't know, do we... not yet)

We might need to flip the parity bit on the parity drive from its current state, to the opposite to maintain parity, but... we don't know yet if we need to... not yet anyway.    Remember, parity is simply and only the fact that we will always store an "even" number of bits set to "1" in a bit position across all the disks.    If the old bit at position 1000000 was a "0" and the new (the one we are writing) is also a "0" we do not need to change the parity bit at all, as the total number of "1"s is not changing.

 

Ok...  To learn if we need to "flip" the parity bit to its opposite state (to maintain an even number of "1s" in that bit-position across all the drives) we must first read the existing 1000000th bit on the data drive.   If it is also a "0", the same as the bit we want to write, then no change is needed to the parity drive, and in fact, if we operated at the single bit level, no write to the parity drive is needed at all since it is not being changed (flipped from a one to a zero, or from a zero to a one)

 

Sooo... we must first read the data disk bit at position 1000000.

 

Now, let's assume the existing data bit is a "1" on the data disk.  That would change the number of "1s" in the 1000000th bit position on all the drives in the array.  Since it previously was an even number of "1s" when the bit was a "1", and now we will be changing it to a zero, we must "flip" the parity bit to its opposite to maintain an even number of "1s"  (The parity bit could currently be either a zero, or a one, depending on the values on the other data drives.)

 

If there was only one other data drive, and it had a "1" in the 1000000th bit position, then the current parity bit would have been set to a "0" for an even number of "1s" (one each in each of the two data drives)  If the other data drive had a "0" in the 1000000th bit position, then the parity bit would have been set to a "1". (It, together with the "1" on one data drive and the "0" on the other data drive would have resulted in an even number of "1s")

 

So, without spinning up and reading ALL the other data drives (and there could be 18 other data drives) we have no way to know the existing state of the parity bit in bit position 1000000 until we read its old contents before writing the new bit.   We would not want to wait until after the "read" of the data disk being written occurred to then request a read from the parity disk (as it would slow things down to half the rate) therefore unRAID issues the read of the parity disk bit at the same time as the read of the data disk bit we will be writing to.  In effect, the two disks will be read in parallel.

 

Ok, back to the data disk we are writing to.  We know the existing bit is changing from a "1" to a zero.  We know we must flip the parity bit from its current value to the opposite... and we issued a "read" of bit 1000000 on the parity disk so we can determine its existing value.   We can then do the math and using an xor (exclusive or) operation determine the new value to write to the parity disk based on its prior value.

 

In actuality, unRAID doesn't act on a single bit at a time, but multiple whole sectors at a time.  so it always writes the parity sector with the calculated value, even if the possibility exists is that it really does not change, since most of the time it does.  It issues the write to the parity disk in parallel with the write to the data disk.

 

As bubbaQ said:

Each write to an unRAID data disk results in 4 disk ops... 2 reads and 2 writes.  a read and write for parity, and a read and write for data, with each disk having its platter make a full revolution after reading to position the disk head back over the sector being written.  Always.   The time to rotate a disk platter will limit you if nothing else does when writing to the array.

 

Joe L.

Link to comment

For an immediate performance boost, adding a cache drive.

 

In order for the average unraid users to see a performance boost, they would need to upgrade the parity drive AND data drive(s). The rare case where only upgrading the parity drive helps out is when the system performs concurrent writes to multiple data drives.

Link to comment

For an immediate performance boost, adding a cache drive.

 

In order for the average unraid users to see a performance boost, they would need to upgrade the parity drive AND data drive(s). The rare case where only upgrading the parity drive helps out is when the system performs concurrent writes to multiple data drives.

 

Thanks. That's what I will do probably.

Again, which drive is the fastest for writes under 200$?

Link to comment

So, what is better - upgrading parity drive or adding cache drive or.. both?

Which 1.5TB/2TB drives are considered to give best stability and performance (under 200$)?

If you are looking for a maximum performance RAID array, then unRAID is the wrong solution.  It can never be as fast as an array where only "writes" are performed.  (Classical RAID5 where writes to ALL drives are always needed, and when reading, all drives MUST be always spinning, as all are read to reconstruct any given movie.)

 

For unRAID I would suggest using any 7200 RPM 1.5 or 2 TB drive that has been on the market for 5 or more years. In that time frame the drive will have had the kinks worked out.   Of course, going by that suggestion severely limits your choice of drives.   ;)

 

Second best is to go with a name brand with as good a replacement warranty as you can find, spinning as fast as possible, priced as cheaply as iyour budget can afford.

 

All hard disks fail... and ALL will... it is only a matter of time.  There are only two types of hard disks in the world. 

 

No, not IDE and SATA...

 

The two types are

1. Those disks that have already failed.

2. Those disks that have not yet failed... but just need a bit more time until they do)

 

I just purchased a pair of 2TB Hitachi disks, retail kits, 3 year warranty, 7200 RPM, 32MB cache...  They were on sale at Frys.com for $159.99 each a few days ago.   They should arrive here on Monday or Tuesday if I'm lucky.

 

      Qty:2 Item#:5947254     HIT.HD32000IDK7/7K 2TB   SEIAL ATA/300 RETAIL HD      319.98

 

These are not the slower "green" drives, but the faster 7200 RPM drives.  The retail kit will come with the cables... You just need to watch the sales. 

I'll use one for parity, and the other for data.

 

Joe L.

Link to comment

Wow, that sucks. I can speed up unraid easily, if that is the logic, why the f do we spend literally hours clearing drives?

 

The only time unraid should be "reading" the data drive is if the sector has already been written too, ie a file is being overwritten. Otherwise we know the byte is all zeroes, we need to read the parity, xor the new byte being written. Job done and only one read operation. Did I miss something?

 

 

Link to comment

Then I do not understand why to use "Green" drives on a pre-built system at all since you are only saving a few watts per drive either when idle or active.

 

It isn't just Watts, but also heat and noise.

 

And don't discount the Watts.  Each Watt saved when saved in an always-on box like a server, saves $1/year.  5 Watts x 10 drives saves $50 per year, every year.  Saving 40 watts with a low-wattage mobo and underclocked/undervolted CPU saves $40/year.

This is definitely true but by reading this forum quite many are using the UnRAID for permanent storage adding up new disks as old ones are filling up. So they are mostly writing to the newest drive (or to the cache drive). And if you didn´t happen to have a big family in a big house you are most likely reading simultaneously only from few drives. So most of the time disks are either spun down or reading/writing but rarely idle.

 

Don´t take me wrong, I also selected mb and overall system with power usage and heat in mind. But at least currently the largest differences are in idle mode and if you take a look on the power consumption when streaming data (link below) the differences are much smaller (with latest versions of 7200rpm and green drives). E.g. the Samsung 7200rpm 0,5TB F3 consumes only 0,1W more than the WD 5400RPM 1,5TB Green. Ok I admit it´s a bit unfair comparison due to difference in platter count but it´s also only 1,38W difference to Samsung 7200rpm 1,0TB F1.

http://www.tomshardware.com/charts/2009-3.5-desktop-hard-drive-charts/Streaming-Read-Operations,1028.html

 

The only reason why I´m still on this topic is the alleged (I don´t know if it really exists or not ) huge difference in write performance between systems equipped with 7200rpm or 5400rpm green drives. 10MB/s vs 30MB/s is huge in my mind, even if most users never need such performance. Somehow I just can´t believe that the read/write twice logic combined with difference in rpms can fully explain the difference. I would like get to the bottom of all this and have proper information available for future system builders on which drives to get for which use scenario.

 

Link to comment

Wow, that sucks. I can speed up unraid easily, if that is the logic, why the f do we spend literally hours clearing drives?

 

The only time unraid should be "reading" the data drive is if the sector has already been written too, ie a file is being overwritten. Otherwise we know the byte is all zeroes, we need to read the parity, xor the new byte being written. Job done and only one read operation. Did I miss something?

Even with that logic you would still be reading and only after that writing to the same (parity) drive. And that takes time. The other read and write on the data disk is basically irrelevant from performance point of view.

 

To my understanding UnRAID does not know which sectors have been used so it has to always read the sector to compare it to the new sector data and to make xor on the parity sector with changed bits.

 

Link to comment

The only reason why I´m still on this topic is the alleged (I don´t know if it really exists or not ) huge difference in write performance between systems equipped with 7200rpm or 5400rpm green drives. 10MB/s vs 30MB/s is huge in my mind, even if most users never need such performance. Somehow I just can´t believe that the read/write twice logic combined with difference in rpms can fully explain the difference. I would like get to the bottom of all this and have proper information available for future system builders on which drives to get for which use scenario.

 

The difference in performance on my side (without the network and using rsync from 1 drive to another) is 8-12Mb/s up to 13-20Mb/s.

The difference in performance with my rtorrent client running all the time was quote noticeable.

Especially if I started to download a fast torrent. I could get over 2.5Mb/s download speed with the use of faster drives.

 

The only two that would matter is a fast parity drive or a fast cache drive.

I run allot of applications directly on unRAID so I have a fast parity drive.

 

To my understanding UnRAID does not know which sectors have been used so it has to always read the sector to compare it to the new sector data and to make xor on the parity sector with changed bits.

 

This is correct.

Link to comment

Slower write speed is primarily caused by two factors:

1. Any time a data disk is written, the Parity disk needs to be updated as well.  This is true of all RAID-4/5/6 systems.

2. Disk rotation speed - the faster the rotation speed, the faster new data can be written.

 

An easy way to think about this is in terms of spindle rotations.

 

For example, suppose we want to write 128K to a track that can store 160K (that is we are going to write an amount of data which less than a full track).  Because Parity must be udpated here is the sequence that takes place (from disk point of view):

 

1. Disk seeks to desired track, then waits for starting sector to come under read/write head.

2. Disk starts reading "old data".  This will take place at rate disk is spinning.

3. Disk now wants to write "new data" to same sectors where it just read "old data", hence it has to wait for starting sector to rotate under read/write head.

4. Disk starts writing "new data".

 

Now let's count the disk revolutions.  We'll start at step 2 (ie, assume seek is done & first sector has rotated under r/w head).

It's going to take 1 revolution to read the "old data" and then position the starting sector back under the r/w head.

It's going to take a partial revolution to write the "new data" (ie, 80% of a revolution because 128K is 80% of 160K).

 

But we have to assume that this is a series of 128K requests resulting from writing a large file.  So we have to count the time it's going to take to get the next sequential sector back under the r/w head.  Because this next sector will be the one immediately following the last one we just wrote, the disk has to wait another full revolution for it to come under the r/w head because there is not enough time between sectors for the disk to process the command and change from write mode to read mode on the head.  Hence add 1 more revolution.

 

So if we are storing the data of 80% of a track, it is going to take 2.8 revolutions to do it.  In other words, it is taking 2.8 times longer vs. just straight writing to the disk.  So if you have disk that can write at say 72MB/sec, the BEST case write will be 72/2.8 = 25MB/sec.  But we never really see the best case because in reality there are several other factors that mess us up:

 

a) Files are not completely contiguous.  Even though reiserfs does a pretty good job at keeping things contiguous, there are time where a large seek must take place - this causes us to lose even more revolutions.

 

b) The amount of data you can store on a track varies considerably depending on disk model and where on the disk we're writing - tracks toward the outside of the disk store more than tracks on the inside.

 

c) There are discontinuities in the smooth read then write process due to queuing many 128K requests to the unraid driver.

 

d) Some disks may perform various retries that are never reported to the operating system, these result in more lost revolutions.

 

One more thing I should add: the actual parity calculation - that is performing the required XOR operations in memory, has a negligible effect on write throughput in an unRAID server.

Link to comment

One potential improvement is an aggressive read-ahead cache, so some percentage of reads will be found in the cache and not need a physical read. This also would benefit from a FS that is aggressive about using free contiguous space first for writes.

 

I'd like to see a test with two Ocz Vertex SSDs - one for parity and one for data - eliminates seek and rotational latency.

Link to comment

So if we are storing the data of 80% of a track, it is going to take 2.8 revolutions to do it.  In other words, it is taking 2.8 times longer vs. just straight writing to the disk.  So if you have disk that can write at say 72MB/sec, the BEST case write will be 72/2.8 = 25MB/sec. 

There is a around 33% difference in RPM between the WD 1,5TB Green drive and Samsung 1TB F1, otherwise their performance benchmarks reported are quite similar. I would thus expect a write performance difference in the ball park of 25-50%, however it is now 300%.

 

I took a closer look on the WD Green drive specs. WD isn´t actually listing a rpm value at all; in place of the value they have IntelliPower which they have explained to be variable rpm between 5400 and 7200 (closer to 5400 unofficially). Could it be that they have optimised the drive behaviour so far towards "normal" operations that the "unusual" drive usage by UnRaid causes severe problems? It wouldn´t also be the first case when manufacturers optimise their products to improve synthetic benchmark results even sacrificing real life performance (but I doubt that´s the case this time).

 

To get some definite answers we would need to have different types of drives benchmarked on the same reference system. This is out of the scope of any one user but could Tom as part of the full system deliveries make some benchmarks? The other alternative would be some easy test package provided by one of the power users (perhaps integrated to UnMenu) which could easily be run on any user system providing simple results  (like parity check speed result) with system info. This way we could get information on which drives get the most out of UnRaid.

Link to comment

The RPM of the drive does *not* vary

http://www.seagate.com/docs/pdf/whitepaper/mb_intellipower_exposed.pdf

 

"Nothing intelli about it" class quote from Seagate.

Well at least I wasn't the only mislead person in the world since Seagate saw it worthy of a marketing bulleting  :D It would have been cool to introduce a hard drive with settable (not dynamic while in operation like intellipower was rumoured to be) rpm thoug;, if you needed top performance you would configure the drive for 7200 rpm and if you needed low power consumption and quietness you would configure it to 5400 rpm. I would settle even for a physical jumper but software configurability would of course be more flexible.

Link to comment

Well at least I wasn't the only mislead person in the world since Seagate saw it worthy of a marketing bulletin

You weren't even I was perplexed by it. I kept questioning how they could vary the RPM and still be reliable.

It was marketing's spin on the engineering bulletin that said something like :

"We'll be able to offer this drive in a 7200 RPM version if we can ever get one to work reliably at that speed...

Until then, we best not limit the "Intelipower" marketing materials to just one speed...

advertise that "Intelipower" will cover a range of speeds (whatever we can get to work),  otherwise we'll have to scrap the advertising campaign.

Also... advertise it as a "green" drive, since that is popular these days.

 

Signed...

  Pointy Haired Boss.

Link to comment

Well I couldn't leave this open so I went and bought a Samsung F2 1,5TB Green drive (5400rpm). I'll be able to install it in the next few days. I'll report back on the  performance once I got the results.

::) I thought I could just add a 1,5TB drive to my all 1TB drive array as a data disk but would only see 1TB out of it. Obviously this isn't the case, rather you have to have the largest disk always as the parity disk. Thinking aftwerwards this is very logical and provides incomplexity and robustness. So I went on and performed the parity switch (aka parity upgrade) procedure by simply first unassigning the old parity drive and then assigning the disk as the parity. It took some 6 hours to complete.

 

Even though my original idea was to test the green, 5400rpm, drive as data disk, it shouldn't make a difference using it as the parity drive. This is because the data is written in the same manner both to data and parity drives without any delays. So slower parity drive should affect data disk write performance almost identically compared to a situation where they were in switched positions.

 

Below is the results of the dd write test:

root@Viper_UnRAID1:/mnt/disk4/blu-ray# dd if=Knowing.iso of=/mnt/disk5/blu-ray/t
est.iso
24976043+0 records in
24976043+0 records out
12787734016 bytes (13 GB) copied, 477.604 s, 26.8 MB/s

This is only few MB/s slower than the previous test with Samsung F1 7200rpm drive. According to Tom's Hardware the WD 1,5TB green drive is faster than the Samsung F2 green drive. This makes me wonder why michael123 is seeing so slow performance. The only thing left to test is to have a green drive also as the data disk. This should not make any difference or our logic about simultaneous reads/writes have been wrong from the beginning. As I don't currently have a need for more storage it will take some time before I can come back to this topic with "all green" test results.

 

The positive thing about this is that I can keep the new green drive as the parity since it doesn't degrade performance. I also started parity check, it seems that also the read performance of the green drive is quite nice; parity check is currently hovering around 100MB/s (as it was also with F1 7200rpm parity), it will propably end up in the area of 90MB/s.

 

 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.