30% Read Performance Improvement Tweak... Still works in unRaid 4.2

[Billped]

you were already running at over 80MB/s? SWEET!!! 

 

I'm not sure what that test is telling us.  With one drive, there is no parity, though by pushing to /dev/null it may not invoke parity anyway.  Is this giving us maximum possible read independent of network?

 

With my five SATA drive setup running 4.0, that same test moved me from 25.0MB/sec to 45.7MB/sec.  As an FYI, moving from 2048 to 8192 yielded 49.1MB/sec - I may play with a more complete set of values later.

 

 

Bill

 

Update: OK, I see conflicting info - more than one drive or one drive?

[Joe L.]

you were already running at over 80MB/s? SWEET!!! 

 

I'm not sure what that test is telling us.  With one drive, there is no parity, though by pushing to /dev/null it may not invoke parity anyway.  Is this giving us maximum possible read independent of network?

 

With my five SATA drive setup running 4.0, that same test moved me from 25.0MB/sec to 45.7MB/sec.  As an FYI, moving from 2048 to 8192 yielded 49.1MB/sec - I may play with a more complete set of values later.

 

 

Bill

 

Update: OK, I see conflicting info - more than one drive or one drive?

He was reading from one drive and parity would not have been involved at all.  So basically it is showing the " maximum possible read independent of network," exactly as you described.  Yes, 80MB/s is very impressive compared to my system's performance.

 

My own tests have shown that reading from the user-shares to /dev/null is considerably slower than reading from the disk shares.  There is additional overhead involved in going through the user-share abstraction layer.  (and lots of room for improvement there, in my opinion)

 

If that Abit AB9 Pro motherboard has equally impressive network performance, it will certainly be one to consider if max performance is a requirement.

 

Joe L.

[RADIatiON]

Hey gang,

 

Just wanted to post my results... unRAID ver. 4.2.1, SATA drives on TX4 controllers, with parity on on-board controller.

 

root@Tower:/boot# blockdev --getra /dev/md7
256
root@Tower:/boot# dd if=/mnt/disk7/HTPC/store7/Movies/movie.mkv of=/dev/null
8977745+1 records in
8977745+1 records out
4596605839 bytes (4.6 GB) copied, 184.03 seconds, 25.0 MB/s
root@Tower:/boot# blockdev --setra 2048 /dev/md7
root@Tower:/boot# blockdev --getra /dev/md7
2048
root@Tower:/boot# dd if=/mnt/disk7/HTPC/store7/Movies/movie.mkv of=/dev/null
8977745+1 records in
8977745+1 records out
4596605839 bytes (4.6 GB) copied, 69.9131 seconds, 65.7 MB/s

 

65MB up from 25MB... that's pretty good in my books !

 

 

[NLS]

Is there a reason this is not included as standard in new builds? (or is it)

 

 

[Jon]

It's not included on new builds.  4.3 Beta3 and I went from 24.8MB/s to 61.4MB/s on a 2.1GB file.

[Joe L.]

It's not included on new builds.  4.3 Beta3 and I went from 24.8MB/s to 61.4MB/s on a 2.1GB file.

My rusty old math skills may not be perfect, but an increase from 24.8 to 61.4 MB/s is a tad bit more than a 30% increase in speed.

 

Joe L.

[NLS]

So is there a reason it is not default that way? (it's been a long ago discovered improvement and Tom has added other much more complex things - like MC)...

Maybe there is an possible problem with the patch?

 

Tom?

 

 

[SSD]

Maybe Tom wants to leave a few nuggets in the forums to encourage participation!  I've noticed that Tom seems to enjoy working on the big enhancements, leaving the "low hanging fruit" enhancements undone.

 

I just did a big upgrade and went from a PCI bus limited Pentium 4 motherboard to a P5B VM D0 w/ Intel E2140 and 2G RAM.

 

I never did this on my old setup, but with the new I went from 24.9 MB/sec to 78 on a 1G file (Seagate 500GB SATA / 16Meg buffer).  (I had to keep picking different 1G files because otherwise, the second time through, it went to 280 MB/Sec!  I guess that 2G of memory is being used for big disk buffers.)

 

I have a stand alone disk (300G IDE) on the unRAID server that is shared manually through Samba. It seems totally unaffected by changing the read ahead buffer size from 256 to 2048 (about 44MB/sec either way).  I also experimented setting the /dev/md? back to 256 and doing the --setra on the underlying device (/dev/sdc) to 2048.  I got no benefit (back to 24.9 MB/Sec).  I guess I am surprised.

 

Thanks Joe L. for this find!

[NLS]

Since he is selling complete systems, I doubt he wouldn't like to give the best performance to his clients. After all you don't HAVE to participate in the forum.

 

 

[Lohkay]

humm... very strange. Only running 2 data drives, 1 sata (WD 500gb 16mb cache) and 1 IDE (WD 250gb) on v4.2.4. The SATA went from 28.2 MB/s to 36.2MB/s which is pretty bad but the IDE went from 26MB/s to 52.3MB/s.

 

I was expecting better from my SATA drive since its brand new.... would the fact that its full change anything?

[NLS]

I doubt - you should look into it further (check syslogs etc.)

 

 

[NLS]

So will this work with 4.3? (latest beta)

 

 

[Joe L.]

yes, still works in 4.3

[leemik]

I just did this..

 

BEFORE:

I copied a large file off unRAID to a SATA drive on my Vista SP1 PC.. was copying at 20-22MB/s (that's what Vista said anyways)

 

AFTER:

copied same file off unRAID to same local SATA drive.. was copying at ~34MB/s

 

So i saw an improvement in reads as well.. 34MB/s is more than enough bandwidth to stream media to my PC but I do still see better results if I stream the file from another PC on same network to my Vista machine rather than unRAID..  maybe Vista SP1 is still not optimal with Samba shares..

 

--mike

 

[jjchambers]

how is everyone editing the "go" file on the flash drive?  Are you text editing over telnet or just opening the file from network explorer?

 

Thanks!

 

J

[erikatcuse]

I went to the flash share via windows and then opened the go file with text editor, saved and then rebooted.

[stephenm00]

Fixed

Edited April 21, 2017 by stephenm00

[Joe L.]

 

NAS login: root

Linux 2.6.24.4-unRAID.

root@NAS:~# dd if=/mnt/disk2/Videos/Movies/b-sahws.avi of=/dev/null

1427024+0 records in

1427024+0 records out

730636288 bytes (731 MB) copied, 29.299 s, 24.9 MB/s

root@NAS:~#

 

NAS login: root

Linux 2.6.24.4-unRAID.

root@NAS:~# dd if=/mnt/disk2/Videos/Movies/b-sahws.avi of=/dev/null

1427024+0 records in

1427024+0 records out

730636288 bytes (731 MB) copied, 9.69916 s, 75.3 MB/s

root@NAS:~#

 

 

A 300% increase isnt to bad...

True, but you can't give all the credit in your case to the read-ahead buffer setting.  You need to try the same experiment with a much larger file than 731 Meg.  Most of that will fairly small file still be in the disk buffer cache when you perform the test the second time, so it will not need to read the physical disk to get it, but it will read it directly from memory.    (of course, if you re-booted between the two tests, and cleared the buffer cache, then congratulations.... it really helps in your server.)

 

Joe L.

[stephenm00]

Fixed

Edited April 21, 2017 by stephenm00

[Joe L.]

Well the I did reboot between tests and there was maby 12 hours in between the first and second test. I am not sure if it would still be in the buffer (I did not manually clear it, I am not sure if it does it on reboot). However I am happy.

Memory is completely cleared when you reboot, and the buffer cache I was referring to is in memory.  It was cleared...  In fact, I know of no other way to "clear" it. 

 

It you did not reboot, and did not play other media files that would have been cached by the buffer pool, then even 12 hours later it is likely the server would not need to read the physical disk.  The disk buffer cache is freed only when something else needs the space.  Or rather, the blocks of data in the disk cache are replaced with newer data being read from the disks, usually it replaces the oldest blocks of data first, working its way towards those accessed more recently.  A block of data accessed very-very frequently would stay in the buffer cache and likely be read only once from the hard disk itself.  A block of data read once, and never again would be eventually replaced with a block of data being read that was not in the disk buffer cache. 

 

When we play movies, we tend to not re-read the same block of data again and again, (unless we re-wind to watch again) so the buffer cache is less useful.

 

With all that in mind, enjoy your 300% increase in speed. ;D

 

Joe L.

[Billped]

The mathematicians in the group are fidgeting.

 

It's a 200%, not 300%, increase.  Otherwise, going from 24.9 to 24.9 would be a 100% increase.

 

Regardless, nice work.

 

 

Bill

[Joe L.]

The mathematicians in the group are fidgeting.

 

It's a 200%, not 300%, increase.  Otherwise, going from 24.9 to 24.9 would be a 100% increase.

 

Regardless, nice work.

 

 

Bill

You can tell it has been a LONG time since I was in a math class... 

Regardless, it is transferring bits about three times as fast as before, and that is good.  (or would a mathematician say that 75.3 is only twice as fast as 24.9?  )

 

 

[crankbearing]

OK - Is this even possible - I was set at 256 and then copied a 1GB VOB file and the system reported 25MB/s changed it 2048 and copied the same file and it jumps to 283MB/s - What gives. I just installed a ASUS M2NPV-VM with an AMD dual core 4200 and 2GB of PC-6400. I have all WD SATA drives 10 of them 1 1TB as Parity some 500 and 750 and one 250 for my music. I ran this on the drive 9 which is a 750Gb. 4 of the drives are on the board and the other 6 are spread across two SATA300 TX4 cards.

 

Sorry for the attachment - I cannot figure out how you are attaching the code. If I do a printscrn in the telnet window it give me the whole desktop.

 

 

Regards,

 

Dave

[crankbearing]

Ok - I set up the go script and rebooted the server and now the readings seem somewhat more in line. The same file at 256 was 25MB/s and at 2048 it was 65MB/s then I tried another file on another drive and got 24MB/s and 55MB/s but the cool news is these are all sata drives and it is making a difference for me.

 

Thanks for the cool tweak....

 

Regards,

 

Dave

[Joe L.]

OK - Is this even possible - I was set at 256 and then copied a 1GB VOB file and the system reported 25MB/s changed it 2048 and copied the same file and it jumps to 283MB/s - What gives. I just installed a ASUS M2NPV-VM with an AMD dual core 4200 and 2GB of PC-6400. I have all WD SATA drives 10 of them 1 1TB as Parity some 500 and 750 and one 250 for my music. I ran this on the drive 9 which is a 750Gb. 4 of the drives are on the board and the other 6 are spread across two SATA300 TX4 cards.

 

Sorry for the attachment - I cannot figure out how you are attaching the code. If I do a printscrn in the telnet window it give me the whole desktop.

 

 

Regards,

 

Dave

Once you read a 1Gig file its blocks of data are in the disk buffer cache.  This is basically memory.  Since you have 2Gig of ram, it all fits.  Unless something else needs that memory, it is there when you attempt to do another speed test.  To make a valid test, you need to use a much bigger file, that can't possibly fit in RAM all at once, or use a different file for each test, or reboot (which clears out all of the ram)

 

The second time you did your test, I doubt if the physical disk was accessed at all, the whole file was read from RAM and it is much faster than reading from a spinning disk.

 

Use "Alt-Printscreen and you will get just th window that has "focus."  I usually then paste it into "paint" and crop it to get rid of the junk I don't need, and then save it as a .jpg file, then attach it.

 

Joe L.