Poor performance with my new Lime server

[Kaygee]

You do have Gb NICs and a Gb switch ?

 

As for directory reads, leave the disks spun up or install cache_dirs. unRAID appears to read the directory contents then send them to the client so when browsing a top level directory for the first time it reads the contents, on really large directories this can take a while, second access is nearly instant. Again accessing a user share spread over several disks means spinning up the disks and reading their content.

 

On my music directory it can take upto a minute to read the directory/file contents (it has a lot of directories/files in it). I can only assume this is a memory saving technique so systems can run with minimum amount of memory, I have 1GB ram which I would of thought would be more than enough to cache the entire directory.

 

Cache_dirs attempts to cache all the drives contents into memory (attempts because you need enough RAM to do this). It works well, however there are a couple of operation issues, stopping the array takes longer and may require a couple of presses of stop. If the system is busy with a parity check or similar, stopping the array is a headache. You need to stop cache_dirs first, this means telnet to the box. Accessing content can take a couple of attempts from something like a media box as cache_dirs allows the drives to stay spun down until access is required.

 

I really like cache_dirs, if you have plenty of memory in the box it's a must have IMO.

[michael123]

yes, of course, all equipment is 1000Mbos

I have 50MB/sec read rate. Only write performance is poor.. and directory access

 

I will add "cache_dirs -w" to the go script, and see if that helps for the directory access (it should, right?)

I have 4GB on my storage box, so I believe I shall not add any other constraints, just cache everything..

[Joe L.]

 

VAULT login: root

Linux 2.6.27.7-unRAID.

root@VAULT:~# ifconfig

eth0      Link encap:Ethernet  HWaddr 00:30:48:b0:ad:08

          inet addr:192.168.3.102  Bcast:192.168.3.255  Mask:255.255.255.0

          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

          RX packets:3287130583 errors:0 dropped:0 overruns:0 frame:0

          TX packets:1123489433 errors:0 dropped:46 overruns:0 carrier:0

          collisions:0 txqueuelen:1000

          RX bytes:3299461697 (3.0 GiB)  TX bytes:2256782712 (2.1 GiB)

          Interrupt:219 Base address:0xa000

 

lo        Link encap:Local Loopback

          inet addr:127.0.0.1  Mask:255.0.0.0

          UP LOOPBACK RUNNING  MTU:16436  Metric:1

          RX packets:31 errors:0 dropped:0 overruns:0 frame:0

          TX packets:31 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:0

          RX bytes:6432 (6.2 KiB)  TX bytes:6432 (6.2 KiB)

 

root@VAULT:~#

 

 

 

 

BTW, why the directory is so slow? I had 4GB installed by Tom, shall not all the files to reside in RAM? How can I speed up opening the directory? S-L-O-W!!

The buffer cache is shared by all the data you read or write to the unRAID server.  As soon as you play ANY file more than a few gig, the oldest un-used blocks in buffer cache are re-used for more current data.  Unfortunately, the newest data is your movie, or music, and the oldest un-used data are the directory listings.

 

To re-read them, when you ask for a directory listing, often unRAID will need to spin up a disk... this takes, as you said, forever.

 

There is only one way to solve this, and that is to keep the directory entries in memory... unfortunately, Linux does not make this easy.  

 

But... there is hope... We can add a script that constantly reads the directory entries into memory.  That way they are not the oldest un-used blocks in the buffer cache and directory listings are nearly instant.

 

That script is called

cache_dirs

and it can be found in this thread: http://lime-technology.com/forum/index.php?topic=3666.msg36387#new

 

You can read this thread to see its evolution and instructions on how to use it: http://lime-technology.com/forum/index.php?topic=3666.0

 

also, I noticed some "dropped" packets in your ifconfig output.  Looks like your router can't keep up, or you have some cabling issues, or mis-matched MTU sizes.  Also, it shows an MTU of 1500... that is NOT jumbo frames...  You probably have a ton of re-transmissions from the router to your unRAID server because of the mismatch.

 

Joe L.

 

 

[michael123]

Thanks

I added your script

 

 

[henris]

I would try to eliminate possible causes by starting from the source side. You can quite easily run a rough read/write performance benchmark inside your unRaid server. You just need a large testfile (>10GB) to avoid any caching effects.

For server internal read speed:

dd if=/mnt/disk1/insertyourfilehere of=/dev/null

 

And for server internal write speed (you have to have at least two disks for this)

dd if=/mnt/disk1/insertyourfilehere of=/mnt/disk2/insertyourfilehere

 

I have a newish system with five identical Samsung F1 1TB models which give me with above tests ~80MB/s reading speed and ~35MB/s writing speed (with disk having 50% of free space). I would expect your figures to be in the same ball park even considering your 5400RPM drives.  Parity check ends in 87MB/s on my system.

 

Once you get these figures, you can continue from the client side. Once again there is a simple repeatable way to roughly measure the read/write performance in Windows (and then do some calculation).

For client side read speed I use:

echo Start time %time%
copy //tower/disk1/insertyourfilehere nul
echo End time %time%

 

For client side write speed (you need a mapped drive for command prompt):

echo Start time %time%
copy c:\insertyourfilehere Z:\insertyourfilehere
echo End time %time%

 

My results were with Windows XP system 47MB/s read speed and 30MB/s write speed. I'm still a bit confused why so many say that 10MB/s writing speed is ok. To my opinion (and according to all the test results) you can easily achieve way higher writing speeds with modern hardware (PCI-E connected SATA hard drives). I'm actually wondering why the difference between parity protected and cache drive write speeds is so huge (35MB/s vs ~80MB/s) since the CPU is no way near maxed out (~25%) during parity writes. Time is obviosly spent somewhere else than in actual parity calculation.

[Kaygee]

Henris,

 

I think of it like this, when doing a disk share write operation you are doing the disk write and a parity calculation (requires reading all disks in the array) and then writing parity bit to the parity disk. When writing to a cache disk it is only doing one write operation. Therfore a disk share write has to be slower, however in theory writing to a disk share should be only slightly slower than a parity build (since it has to write parity not just check it) operation, since this is what it is actually doing on the new bits being written. So it should be parity build + a disk write operation. On my system a parity build runs at 58MB/s but a disk share write is around 35MB/s.

 

The 10MB/s figure I believe relates to user shares, not disk shares, here you have the additional overhead of user share layer, security layer and the disk allocation calculations.

 

I never write to user shares, I always use disk shares. I get similar results to you, unsuprisingly since I'm using the same disks :-).

The newer hd103sj 1TB version is supposed to be faster still 125Mb/s vs 115Mb/s.

 

   

[Joe L.]

I'm actually wondering why the difference between parity protected and cache drive write speeds is so huge (35MB/s vs ~80MB/s) since the CPU is no way near maxed out (~25%) during parity writes. Time is obviosly spent somewhere else than in actual parity calculation.

That is simple to explain, to write to a parity protected drive unRAID must read a sector on both the data disk and the corresponding sector on the parity disk (these are done in parallel) and then write back the same sector on both the data disk and parity disk.  (These too are done in parallel)

 

In between each read and write, the respective disks must rotate at least one revolution to position the same sector back under the read/write head.

 

The huge difference in "write" speed is the time it takes for the disk platter to rotate a full revolution between reading and subsequently writing any given sector on a cylinder.   Your "write" speed is limited by basic physics and the rotational speed of the disk.  7200 RPM drives will do much better than 5400 RPM drives.   

 

The cache on the disk helps a tiny bit on the writing of a single sector, or even a few, but it is far smaller than most of the files we write, and therefore not as useful when we are writing a movie sized file to the disk.

 

Perhaps the 10MB/s benchmark was with 5400 RPM drives and smaller on-disk cache sizes.  It is good you can get 30MB/s over your LAN with modern hardware.

 

Joe L.

[henris]

That is simple to explain, to write to a parity protected drive unRAID must read a sector on both the data disk and the corresponding sector on the parity disk (these are done in parallel) and then write back the same sector on both the data disk and parity disk.  (These too are done in parallel)

 

In between each read and write, the respective disks must rotate at least one revolution to position the same sector back under the read/write head.

 

The huge difference in "write" speed is the time it takes for the disk platter to rotate a full revolution between reading and subsequently writing any given sector on a cylinder.   Your "write" speed is limited by basic physics and the rotational speed of the disk.  7200 RPM drives will do much better than 5400 RPM drives.   

 

The cache on the disk helps a tiny bit on the writing of a single sector, or even a few, but it is far smaller than most of the files we write, and therefore not as useful when we are writing a movie sized file to the disk.

This is exactly the part which I didn't think through and fully explains the hit on write performance. I was thinking it in bit level which of course is nothing like real life with 512 byte sectors or so.

 

Perhaps the 10MB/s benchmark was with 5400 RPM drives and smaller on-disk cache sizes.  It is good you can get 30MB/s over your LAN with modern hardware.

I was only referring to the initiator of this thread who had a brand new unRAID system from Limetech and was wondering about 10MB/s write speeds. According to their specs, they are equipped with Adaptec 1430SA PCI-E SATA-cards and thus should easily break the 20MB/s barrier, even with 5400RPM disks.

 

I'm very satisfied with the unRAID system and all the above figures are well over any real life requirements (apart from occasional mass transfers). I'm yet to try real multiclient scenarios but I suspect the system to be able to feed several (3-5) Blu-ray movies to clients simultaneously without problems.

[michael123]

henris

 

thank you for your benchmarking tips, it was very helpful as I am not a Linux guy

 

What I received is following

 

***READ ***

 

root@VAULT:~# dd if="/mnt/disk3/Movies/Music/Test.mkv" of=/dev/null

73184338+1 records in

73184338+1 records out

37470381528 bytes (37 GB) copied, 609.037 s, 61.5 MB/s

 

 

*** WRITE ***

 

root@VAULT:~# dd if="/mnt/disk3/Movies/Music/Test.mkv" of=/mnt/disk4/Movies/Rush.mkv

73184338+1 records in

73184338+1 records out

37470381528 bytes (37 GB) copied, 3073.51 s, 12.2 MB/s

 

 

Does it mean that the these WD disks just have poor write performance?

[BRiT]

Here's some numbers from my system with 3 of the 2TB WD Green drives.

 

Reads from User Shares:

root@Reaver:# dd if=/mnt/user/Anime/Test.mkv of=/dev/null

9141555+1 records in

9141555+1 records out

4680476463 bytes (4.7 GB) copied, 52.6067 s, 89.0 MB/s

 

 

Reads from Disk Shares:

root@Reaver:# dd if=/mnt/disk1/Test/Test.mkv of=/dev/null

9141555+1 records in

9141555+1 records out

4680476463 bytes (4.7 GB) copied, 52.4691 s, 89.2 MB/s

 

Write to Disk Shares:

root@Reaver:# dd if=Test.mkv of=/mnt/disk1/Test/Test.mkv

9141555+1 records in

9141555+1 records out

4680476463 bytes (4.7 GB) copied, 264.975 s, 17.7 MB/s

 

 

For comparison, Reads from cache drive:

root@Reaver:# dd if=Test.mkv of=/dev/null

9141555+1 records in

9141555+1 records out

4680476463 bytes (4.7 GB) copied, 47.0591 s, 99.5 MB/s

 

 

 

 

[michael123]

so why do I have so low write rate?

[henris]

henris

 

thank you for your benchmarking tips, it was very helpful as I am not a Linux guy

 

What I received is following

 

***READ ***

 

root@VAULT:~# dd if="/mnt/disk3/Movies/Music/Test.mkv" of=/dev/null

73184338+1 records in

73184338+1 records out

37470381528 bytes (37 GB) copied, 609.037 s, 61.5 MB/s

 

 

*** WRITE ***

 

root@VAULT:~# dd if="/mnt/disk3/Movies/Music/Test.mkv" of=/mnt/disk4/Movies/Rush.mkv

73184338+1 records in

73184338+1 records out

37470381528 bytes (37 GB) copied, 3073.51 s, 12.2 MB/s

 

 

Does it mean that the these WD disks just have poor write performance?

I just re-ran the write test and received the following:

root@Viper_UnRAID1:/mnt/disk4/blu-ray# dd if=test.iso of=/mnt/disk5/blu-ray/t
est.iso
40677267+0 records in
40677267+0 records out
20826760704 bytes (21 GB) copied, 637.36 s, 32.7 MB/s

 

What is the exact make and model of your drive? You can get easily with the following command (you also get some additional info):

hdparm -i /dev/sda

 

My Samsung F1 1TB states the following:

/dev/sda:

Model=SAMSUNG HD103UJ                         , FwRev=1AA01118, SerialNo=S13PJD
WS507284
Config={ Fixed }
RawCHS=16383/16/63, TrkSize=34902, SectSize=554, ECCbytes=4
BuffType=DualPortCache, BuffSize=32767kB, MaxMultSect=16, MultSect=?0?
CurCHS=16383/16/63, CurSects=16514064, LBA=yes, LBAsects=1953525168
IORDY=on/off, tPIO={min:120,w/IORDY:120}, tDMA={min:120,rec:120}
PIO modes:  pio0 pio1 pio2 pio3 pio4
DMA modes:  mdma0 mdma1 mdma2
UDMA modes: udma0 udma1 udma2 udma3 udma4 udma5 *udma6
AdvancedPM=yes: disabled (255) WriteCache=enabled
Drive conforms to: unknown:  ATA/ATAPI-3,4,5,6,7

* signifies the current active mode

[michael123]

I get an error

 

root@VAULT:~# hdparm -i /dev/sda

/dev/sda:
HDIO_GET_IDENTITY failed: Invalid argument
root@VAULT:~#

 

My drives are

http://www.wdc.com/en/products/products.asp?driveid=575

 

 

 

[henris]

My mistake. You should first find out which device name is actually used. You can see this in the web management interface´s Devices tab. Eg.

disk1 device:  	pci-0000:00:12.0-scsi-1:0:0:0 (sda) ata-SAMSUNG_HD103UJ_S13PJDWS107228

 

Then just replace sda in the command parameter with correct device name. The other info provided by the hdparm, like the used udma mode, can help to make sure bios settings are as they should be.

[henris]

My drives are

http://www.wdc.com/en/products/products.asp?driveid=575

The exact model is thus WD15EADS. Taking a look on Tom´s Hardwares benchmark results (link below), the maximum write throughput of this drive is over 100MB/s. This is quite close to Samsung F1.

http://www.tomshardware.com/charts/2009-3.5-desktop-hard-drive-charts/h2benchw-3.12-Max-Write-Throughput,1012.html

 

Could you repeat the dd-test on cache drive (if you do not have installed you can temporarily assign one of your empty data disks as cache disk)? This would once again narrow the problem area. Also run the hdparm so that we can see the other info on drive and bios settings.

[michael123]

This is the info of 'disk4':

 

root@VAULT:~# hdparm -i /dev/sdf

/dev/sdf:

Model=WDC WD15EADS-00P8B0                     , FwRev=01.00A01, SerialNo=     WD-WMAVU0068527
Config={ HardSect NotMFM HdSw>15uSec SpinMotCtl Fixed DTR>5Mbs FmtGapReq }
RawCHS=16383/16/63, TrkSize=0, SectSize=0, ECCbytes=50
BuffType=unknown, BuffSize=32767kB, MaxMultSect=16, MultSect=?16?
CurCHS=16383/16/63, CurSects=16514064, LBA=yes, LBAsects=18446744072344861488
IORDY=on/off, tPIO={min:120,w/IORDY:120}, tDMA={min:120,rec:120}
PIO modes:  pio0 pio3 pio4
DMA modes:  mdma0 mdma1 mdma2
UDMA modes: udma0 udma1 udma2 udma3 udma4 udma5 *udma6
AdvancedPM=no WriteCache=enabled
Drive conforms to: Unspecified:  ATA/ATAPI-1,2,3,4,5,6,7

* signifies the current active mode

[henris]

Nothing suspicious in that report. This was expected as the reading speed is ok. The only thing left is to verify write performance on a non-parity protected drive. Easiest is to use cache drive. However I suspect that it will provide normal (60-80MB/s) results. I just noticed that Brit reported 17MB/s writing speed with his 2TB WD Green drives. Taking into account the speed difference between 1,5TB and 2TB models and system dependent variables, 10MB/s is plausible. It might be that 5400RPM drives take much severe hit when doing parity protected writes than 7200RPM drives do. Then I do not understand why to use "Green" drives on a pre-built system at all since you are only saving a few watts per drive either when idle or active.

[michael123]

Why are there some unknown fields, like

 

BuffType=unknown

and

Drive conforms to: Unspecified: 

??

 

I got the pre-built system without drives. I asked Tom regarding this model of the disks (this is what my localshop had in stock), and Tom confirmed that they are OK.

Frankly, I do not perform many 'writes' on the array, which are mostly taken off-line.

 

So, do you think that if I buy faster parity drive, I will get better writer performance?

Which model can you recommend? (I need either 1.5TB or 2TB)

Do you think if Seagate Barracuda 7200.11 (1.5TB 7200 RPM, 32MB Sata II) ST31500341AS? will be better than WD?

 

 

[BRiT]

FWIW, here's the hdparm info from my WDs. It displays some of the same weirdness for "BuffType", and  "Drive conforms to: Unspecified: ".

 

root@Reaver:~# hdparm -i /dev/sdc

Model=WDC WD20EADS-00R6B0                     , FwRev=01.00A01, SerialNo=     WD-WCAVY0247937
Config={ HardSect NotMFM HdSw>15uSec SpinMotCtl Fixed DTR>5Mbs FmtGapReq }
RawCHS=16383/16/63, TrkSize=0, SectSize=0, ECCbytes=50
BuffType=unknown, BuffSize=32767kB, MaxMultSect=16, MultSect=?0?
CurCHS=16383/16/63, CurSects=16514064, LBA=yes, LBAsects=18446744073321613488
IORDY=on/off, tPIO={min:120,w/IORDY:120}, tDMA={min:120,rec:120}
PIO modes:  pio0 pio3 pio4
DMA modes:  mdma0 mdma1 mdma2
UDMA modes: udma0 udma1 udma2 udma3 udma4 udma5 *udma6
AdvancedPM=no WriteCache=enabled
Drive conforms to: Unspecified:  ATA/ATAPI-1,2,3,4,5,6,7

* signifies the current active mode

root@Reaver:~# hdparm -i /dev/sdd

Model=WDC WD20EADS-00R6B0                     , FwRev=01.00A01, SerialNo=     WD-WCAVY0252670
Config={ HardSect NotMFM HdSw>15uSec SpinMotCtl Fixed DTR>5Mbs FmtGapReq }
RawCHS=16383/16/63, TrkSize=0, SectSize=0, ECCbytes=50
BuffType=unknown, BuffSize=32767kB, MaxMultSect=16, MultSect=?0?
CurCHS=16383/16/63, CurSects=16514064, LBA=yes, LBAsects=18446744073321613488
IORDY=on/off, tPIO={min:120,w/IORDY:120}, tDMA={min:120,rec:120}
PIO modes:  pio0 pio3 pio4
DMA modes:  mdma0 mdma1 mdma2
UDMA modes: udma0 udma1 udma2 udma3 udma4 udma5 *udma6
AdvancedPM=no WriteCache=enabled
Drive conforms to: Unspecified:  ATA/ATAPI-1,2,3,4,5,6,7

* signifies the current active mode

root@Reaver:~# hdparm -i /dev/sdb

Model=WDC WD20EADS-00R6B0                     , FwRev=01.00A01, SerialNo=     WD-WCAVY0211284
Config={ HardSect NotMFM HdSw>15uSec SpinMotCtl Fixed DTR>5Mbs FmtGapReq }
RawCHS=16383/16/63, TrkSize=0, SectSize=0, ECCbytes=50
BuffType=unknown, BuffSize=32767kB, MaxMultSect=16, MultSect=?0?
CurCHS=16383/16/63, CurSects=16514064, LBA=yes, LBAsects=18446744073321613488
IORDY=on/off, tPIO={min:120,w/IORDY:120}, tDMA={min:120,rec:120}
PIO modes:  pio0 pio3 pio4
DMA modes:  mdma0 mdma1 mdma2
UDMA modes: udma0 udma1 udma2 udma3 udma4 udma5 *udma6
AdvancedPM=no WriteCache=enabled
Drive conforms to: Unspecified:  ATA/ATAPI-1,2,3,4,5,6,7

* signifies the current active mode

[Kaygee]

My seagate drives show unknown for buffer type, Drive conforms to: Unspecified:  ATA/ATAPI-1,2,3,4,5,6,7, is because drives aren't ata/atapi.

[henris]

Why are there some unknown fields, like

 

BuffType=unknown

and

Drive conforms to: Unspecified: 

??

I don´t know for sure but based on some googling it seems to be ok to have unknown reported as bufftype. Same applies to "Drive conforms" value.

I got the pre-built system without drives. I asked Tom regarding this model of the disks (this is what my localshop had in stock), and Tom confirmed that they are OK.

Frankly, I do not perform many 'writes' on the array, which are mostly taken off-line.

 

So, do you think that if I buy faster parity drive, I will get better writer performance?

Which model can you recommend? (I need either 1.5TB or 2TB)

Do you think if Seagate Barracuda 7200.11 (1.5TB 7200 RPM, 32MB Sata II) ST31500341AS? will be better than WD?

If the logic stated by Joe L. previously on this thread about parity writes is correct there should no performance gain if you only change the parity drive; since reads/writes are performed simultaneously to data and parity disk, the slower one dictates overall write performance. I myself am all Samsung guy, they have never failed on me and usually perform above average. But that´s just me, others have similar good experience with Seagate. I usually use Tom´s hardware´s reviews and benchmarks as guidelines. I also almost always buy models which have been on the market for quite some time to avoid any early adaptor problems.

 

[bubbaQ]

Then I do not understand why to use "Green" drives on a pre-built system at all since you are only saving a few watts per drive either when idle or active.

 

It isn't just Watts, but also heat and noise.

 

And don't discount the Watts.  Each Watt saved when saved in an always-on box like a server, saves $1/year.  5 Watts x 10 drives saves $50 per year, every year.  Saving 40 watts with a low-wattage mobo and underclocked/undervolted CPU saves $40/year.

[WeeboTech]

Why are there some unknown fields, like

 

BuffType=unknown

and

Drive conforms to: Unspecified: 

??

I don´t know for sure but based on some googling it seems to be ok to have unknown reported as bufftype. Same applies to "Drive conforms" value.

I got the pre-built system without drives. I asked Tom regarding this model of the disks (this is what my localshop had in stock), and Tom confirmed that they are OK.

Frankly, I do not perform many 'writes' on the array, which are mostly taken off-line.

 

So, do you think that if I buy faster parity drive, I will get better writer performance?

Which model can you recommend? (I need either 1.5TB or 2TB)

Do you think if Seagate Barracuda 7200.11 (1.5TB 7200 RPM, 32MB Sata II) ST31500341AS? will be better than WD?

If the logic stated by Joe L. previously on this thread about parity writes is correct there should no performance gain if you only change the parity drive; since reads/writes are performed simultaneously to data and parity disk, the slower one dictates overall write performance.

 

If you happen to do multiple writes to multiple drives, the faster parity drive helps.

Every write operation to the protected array depends on the parity drive.

So even though you have a slower data drive,  the faster parity drive has the potential to handle more traffic per second.

 

The other solution is to get a smaller, but fairly fast cache drive and use that for writes.

This might be the cheapest and fastest solution.

[Kaygee]

If the logic stated by Joe L. previously on this thread about parity writes is correct there should no performance gain if you only change the parity drive; since reads/writes are performed simultaneously to data and parity disk, the slower one dictates overall write performance.

 

Not true, adding a 7200 RPM parity drive will increase performance. Unraid writes the byte to the array disk being written to, at them same time performs a read of the same byte on the parity disk, xors the result to calculate parity and then writes the byte to the parity disk. So the parity disk does a read then a write to the same sector, meaning the parity disks platter needs to do a whole platter rotation (after the onboard cache is exhausted).

 

With the data disks 5400/5900 RPM or "green disks" probably wont affect performance dramatically as long as they can sustain good sequential write performance. Data disks are either being read from or written too, not both like a parity drive is. Ideally we (unRaid) would want 15K parity drives to even the balance and match a 5400 RPM data drive.

 

 

 

 

[bubbaQ]

Unraid writes the byte to the array disk being written to, at them same time performs a read of the same byte on the parity disk, xors the result to calculate parity and then writes the byte to the parity disk.

 

No.  unRAID has to read the sector about to be written on the data disk, in order to remove it from the parity calc, and then it adds the new sector of data to the parity.

 

Each sector written to data results in reading the data and parity for that sector (2 reads) and writing that sector to data and parity (2 writes)

 

Each write to an unRAID data disk results in 4 disk ops... 2 reads and 2 writes.  a read and write for parity and a read and write for data.  Always.