February 6, 201214 yr HI Guys, Copying a 22GB 1080p file from my cache drive (7200RPM at 30% full) to a 2TB WD Green at 87.5% full and I'm only getting ~18 MB/sec average over 90 seconds. Command: bwm-ng -Ibwm-ng -A 90 -T avg -R 1 -i disk -c 1 -I sdb,sdc,sdd,sde,sdf,sdg,sdh I will do the same to a disk that isn't quite as full but this seems... slow!? The Parity drive is a brand new 2TB WD Black 7200RPM on a PCI-E Sil3132 controller (no other disks on this). The data and cache drive is on the mobos controller. Ideas?
February 6, 201214 yr Perhaps you already ruled this out, but I get similar performance when I just use windows to manage the file copying. When you do that it has to copy twice, once to the pc and then back to the server on a new disk. I usually average 35MBps so seems correct to me getting about half that when I copy from 1 disk to another. My anecdotal evidence anyway.
February 7, 201214 yr That's speed is not unheard of for parity protected drives near the end of capacity. to see what your maximum potential would be, download the writeread10gb shell from my googlecode page. do a chmod u+x ./writeread10gb ./writeread10gb /mnt/disk?/test.dd where /mnt/disk? is the disk you want to test. This will show you the maximim potential write speed you will get from the parity protected data drive. I estimate this would be around 30-35MB/s on the outer tracks so 18MB/s isn't all that bad on a nearly full filesystem forcing allocation and writes to the inside tracks.
February 7, 201214 yr Author Now down to 14.5MB/sec to a disk that is 70% full. Will run your script when I get a minute. Ran your script to a disk that is 33% full (2TB WD Green) and the result was 21.5 MB/sec. I'm 99% sure I used to get results that were twice that. WTF is slowing it down so much!?!? Edit: Fixed it! My md_write_xxx was set at 1000 instead of 10000! There we go!
February 7, 201214 yr What is your motherboard? How Much Ram do you have? Have you tried tweaking the MD driver configuration any? I took all the settings and doubled them (but it requires ram to buffer the stripes). Also, the SIL3132 is a nice controller, I'm not so sure it's the best controller for the parity disk. If you look at your syslog, it may say it enabled the drive at DMA/100 rather then DMA/133. If you have a port on an X4 controller or on your motherboard ports, that is where I might switch the parity. The motherboard chipset usually has a bandwidth of around 256MB/s to 384MB/s. I might try putting reversing the drives and putting the cache on the SIL3132. There's no real basis for these hypothesis, but it's worth a try.
February 7, 201214 yr Author What is your motherboard? How Much Ram do you have? Have you tried tweaking the MD driver configuration any? I took all the settings and doubled them (but it requires ram to buffer the stripes). Also, the SIL3132 is a nice controller, I'm not so sure it's the best controller for the parity disk. If you look at your syslog, it may say it enabled the drive at DMA/100 rather then DMA/133. If you have a port on an X4 controller or on your motherboard ports, that is where I might switch the parity. The motherboard chipset usually has a bandwidth of around 256MB/s to 384MB/s. I might try putting reversing the drives and putting the cache on the SIL3132. There's no real basis for these hypothesis, but it's worth a try. Turns out it was a stupid mistake my me. I'd made my md_write_stripes as 1000 not 10000. I'm going to play around with NCQ on/NCQ off and all the settings to see what I can get up to. I have 8GB of ram and a M4A78LT-M with a quad Phenom @ 3.2.
February 7, 201214 yr Add this tweak too after emhttp is launched and the /dev/md1 device comes online. for disk in /dev/md* do blockdev --setra 2048 $disk done I have FORCE NCQ disabled. These are my md settings. Tunable (md_num_stripes): 3840 Tunable (md_write_limit): 2304 Tunable (md_sync_window): 864 I did have then set to 3x default, but that caused the kernel to run out of memory when I did a find down the whole system tree. (Even with 8GB of ram.. I think it's low ram that really matters).
February 7, 201214 yr Author NCQ/md_write_limit/md_num_stripes/read/write Off/10000 20000/121/37.86 On/10000 20000/123/37.53 Turning NCQ didn't really do a lot it seems! Restarting with your tweak! Edit: Increased it a tad. Averaging (over all the results) of about 38 MB/sec write and 123MB/sec read Shame I didn't run this when the WD Green was the Parity drive. Oh well. Solved! Edit2: Outer sector read/write (2TB WD Green @ 30% full): 123 MB/sec / 37-38MB/sec Inner sector read/write (1TB WD Green @ 97% full): 99.4MB/sec / 34-35MB/sec So essentially, the parity drive is the limiting factor, as I can write to even the inner sectors of the 1TB yet still manage to write at 95% full speed! All is good!
February 8, 201214 yr The values in edit2 are normal. yes the parity is the limiting factor. read parity, read "current" data, XOR out current data from parity, XOR in current data, write data, write parity. maybe not in the exact order but that's pretty much it. So for a drive that reads @120MB/s this is why you're seeing the write speed at about 25%. A caching controller helps this, but not all that much. I get from 53MB/s burst to 43MB/s down to 35MB/s at the end of 10GB. I tweaked my kernel to cache more then the normal unraid settings, so the burst is as the highest possible speed. For me it's needed as I use my fileserver for allot of small files. I.E. source repository over NFS. This is why when people ask about a cache drive that only yields 30MB/s, I mention that the array is capable of 30MB/s so a cache drive may not be all that effective.
February 8, 201214 yr Try changing the scheduling from CFQ to others to see how it affects your setup. I think Purko explored that one.
February 14, 201214 yr hey guys, i'm having a similar problem on mine.. I ran the writeread10gb and it starts out OK at 33.9 MB/sec but each and every test after that is slower and slower.. is that normal? The last portion of the test was 20.0 MB/sec and every update inbetween was a steady decline..
February 14, 201214 yr hey guys, i'm having a similar problem on mine.. I ran the writeread10gb and it starts out OK at 33.9 MB/sec but each and every test after that is slower and slower.. is that normal? The last portion of the test was 20.0 MB/sec and every update inbetween was a steady decline.. it;s normal for it to decline, however depending on your hardware, the average is 22-32MB/s on a parity protected drive. Depending on drive, it could be 30-60 even 90-135 on a modern cache disk on the outside. How full your drive is also determines speed. a drive at 90% will be slower then at 0%.
Archived
This topic is now archived and is closed to further replies.