October 25, 200916 yr I've just finished building my unraid machine and have a query regarding write speeds on it. Specs are as follows: Asus P5Q-Deluxe - AHCI mode enabled E6400 (currently underclocked to 1.6ghz) 2gb G.Skill Ram 2xAdapatec 1430SA cards - BIOS disabled. Seasonic M12-700W PSU Lian Li PC-A17B 3xNorco SS-500 5in3 hotswap modules Lexar JumpDrive Firefly 4gb Unraid 4.4.2 I currently have 5xWD10EADS 1tb drives connected as data drives on the Adaptec cards. I copied a lot of data to them prior to putting in the parity drive and I got speed varying from 65-80MB/s when copying the data to them. I installed a Seagate 1.5TB drive as parity last night after running it through the preclear script. This drive is connected to one of the motherboard ports. During Parity Sync i was getting speeds of between 55-60MB/s which seemed to be pretty good from what i've read here. This morning I've just gone to copy some small files (350mb vid files) to the array. When copying to the array using cut/paste method in Vista64 my write speed to the array starts around 15mb/s and rapidly drops off to 11MB/s for the first part of the transfer but tails off to just 5-6MB/s by midway through the transfer. Consequently it takes about 50 seconds to transfer that 350mb file. Other threads i've seen talking about write speed indicate ~1gb/min is about what I should be looking at? I haven't setup user shares yet so I'm just writing to the //Tower/Disk4 location under Network locations in Windows Explorer. I've been reading people getting ~15MB's writing to their unraid servers so am just wondering if there something in particular I should be looking at or are these 5-6MB/s speeds normal and what I should be expecting from the server? Syslog is attached although it doesn't seem to have any entries from today showing the file transfers. Any advice would be appreciated as I still have a hell of a lot of data to transfer to the server and don't particularly relish the idea of doing it at 5MB/s.
October 25, 200916 yr I got speed varying from 65-80MB/s when copying the data to them. I sincerely doubt that. I think you may be mis-remembering or confused MB with Mb. 80MB/s = 640Mb/sec = 4.8 GB/min.... that is an incredibly high number that your LAN is not likely able to do. What SP level of Vista? You may try copying them from another machine (with XP).
October 25, 200916 yr Author mm.. Maybe my MB's should be Mb's then.. I'm pretty sure that it was MB/s though. It was peaking at 80 but sitting more consistently in the 65-70mb range. Either way the speeds were exceptional when transferring without parity which is to be expected. I moved ~2tb in a few hours across the network. All computers on my network are gbit enabled and i have gbit routing. Thats a side note to the query on the ~50% speed others seem to be getting though. I'm thinking I might try moving the parity onto the Adaptec card tonight when I get home from work and see if that affects speeds. Does this seem like something that may affect performance? Its Vista64 SP2 in answer to your edit. I will try from one of my Win7 machine tonight also.
October 25, 200916 yr If you have a PCIe card, parity should usually be in that. Second best is (usually, not always) on a mobo SATA port. You are right that 15MB/sec (1GB/min) is about what good systems average...a few can beat that but not many.... a lot are about 20% under that mark too. 5MB/sec is indeed sub-par and warrants investigation. I would add a cache disk, so I can time xfers and eliminate the unRAID parity/driver/system elements.
October 26, 200916 yr Author If you have a PCIe card, parity should usually be in that. Second best is (usually, not always) on a mobo SATA port. You are right that 15MB/sec (1GB/min) is about what good systems average...a few can beat that but not many.... a lot are about 20% under that mark too. 5MB/sec is indeed sub-par and warrants investigation. I would add a cache disk, so I can time xfers and eliminate the unRAID parity/driver/system elements. To be honest I'd be happy with 12MB/s (the 20% under you indicate). The parity is currently on one of the mobo SATA ports but I will move it to the adaptec when I get home from work. I intend to allocate a slot for a cache disk but the drive I intend to use there is reasonably small compared to the rest of the array so doesnt really lend itself to use while copying the several terabytes of data I will need to in order to get the array in its start position. Hence I havent installed the drive yet. I fully intend to make use of its increased speed once I have it all setup though. Should I be using a mode other than AHCI on the mobo ports?
October 26, 200916 yr I have also had what I called "Curious performance in copying to unRAID" (http://lime-technology.com/forum/index.php?topic=4541.0) which appears similar to your observations below when copying large volumes of data. WeeboTech and RobJ have provided some interesting feedback there which I am working through at the moment
October 26, 200916 yr Should I be using a mode other than AHCI on the mobo ports? Yes, I had a 20%+ improvement (from ~50MB/s to 60+MB/s) in parity calculation performance by moving from the MB bios SATA configuration to the AHCI configuration.
October 26, 200916 yr I intend to allocate a slot for a cache disk but the drive I intend to use there is reasonably small compared to the rest of the array so doesnt really lend itself to use while copying the several terabytes of data I will need to in order to get the array in its start position. I was suggesting a cache disk for troubleshooting purposes, not as a solution. With a cache disk you can do more detailed testing.
October 26, 200916 yr Author I was suggesting a cache disk for troubleshooting purposes, not as a solution. With a cache disk you can do more detailed testing. Ah ok I misunderstood. I will try moving the parity drive to the 1430SA card first then I'll try the cache drive if that makes no difference.
October 26, 200916 yr Author OK ive moved the parity drive onto the 1430SA card now. There are now 5x1TB WD10EADS and 1x1.5TB ST31500341AS all hanging off the two 1430SA cards. I've also installed Teracopy and am using that for comparing file copy speeds. As a test ive made 10 identical copies of a 350mb AVI file and copied those 10 files as a batch across the network to the unraid server. The total file size was 3.41GB. Total transfer time was 6 minutes 28 seconds for an average speed of 9.0MB/s. During that time I saw the reported transfer speed drop as low as 638KB/s and go as high as 22MB/s. As a 2nd test ive made 10 identical copies of 1.1gb MKV file and copied those 10 files as a batch across the network to the unraid server. The total file size was 10.92GB. Total transfer time was 20 minutes 37 seconds for an average speed of again 9.0MB/s. During that time I saw the reported transfer speed drop as low as 638KB/s and go as high as 33MB/s. So i'm getting exactly 9.0mb's on both tests. This still seems a few MB/s short of what I should be expecting. When its copying there seems to be bursts of copying instead of a constant stream. It will write for a few seconds at high speed then drop to about 1.5-3KB/s then speed back up, then slow down. Its almost like its filling a cache then holding off till that empties and rinse and repeat. Is this because it writes the file to HD then updates the parity drive or is it something else bottleknecking the system? When there was no Parity drive in the system the speeds stayed constant throughout the whole file copy process when I was doing the initial copying to the 5xWD10EADS drives. Should I look at adding more ram (4gb instead of 2gb) or putting the CPU back to normal speed instead of underclocking it? Attached is a new syslog.txt in case there is anything useful in there.
October 26, 200916 yr More RAM will probably not help, nor will a faster CPU clock speed, as the process is most likely limited by your "green" drives slow rotational speed. See this thread for what "might" help some. http://lime-technology.com/forum/index.php?topic=4520.0 No matter what you do, basic physics restricts the best/fastest "write" speed with parity enabled to about 2.8 time slower than when it is not enabled. See this post for an explanation: http://lime-technology.com/forum/index.php?topic=4390.msg40684#msg40684 Your WD drives are all 5400 RPM drives, with a purposely slower speed to use less energy. They will always be slower than the 7200 RPM drives, regardless of any marketing claims, when you have to read and then write the identical sector with the disk spinning in between. Your faster parity drive helps, but it is going to be waiting on the data disks to rotate between their alternate read and writes of sectors. If you can "tune" the sizes of the writes to the disks or use an alternate method of queuing the data to them, you might benefit. That first thread I pointed you to might help you experiment with different disk buffer sizes and disk I/O scheduler modes. Joe L.
October 26, 200916 yr Author mm some interesting reading there Joe.. Thx for the pointers.. shame that the green-power drives can be written to singularly in excess of 65MB/s yet cant muster more than 9MB/s in the "raid" config. I would have thought that the parity drive would have been the one slowing things down as it is read/written more than the data drive? I notice when copying data to the drive that the main unraid reports approximately twice as many read and write operations on the parity drive over the data drive. Unfortunately it seems not to be the case. 9MB/s isnt too bad i guess.. thankfully teracopy can queue transfers..
October 26, 200916 yr I would have thought that the parity drive would have been the one slowing things down as it is read/written more than the data drive? I notice when copying data to the drive that the main unraid reports approximately twice as many read and write operations on the parity drive over the data drive. That entirely depends on the sizes of the "writes" based on your physical disks and the controllers. Twice the "writes" can represent the same data, (unless there is some kind of bug that writes twice... but that would affect all of us) I can write 1000 bytes once, or 500 twice, and the same work gets done... BUT 500 twice will require the disks to rotate around even more. In other words (using 500 bytes only for this example, I've no idea of the actual sizes of the "writes" and "reads "of your disks) rotate to desired sector. read 500 bytes rotate once to get back to same sector write 500 bytes rotate once to get back to same sector read 500 bytes rotate once to get back to same sector write 500 bytes If your disk read and wrote 1000 bytes at a time (twice the amount) it would not need as many revolutions to complete the operation. Unfortunately it seems not to be the case. 9MB/s isnt too bad i guess.. thankfully teracopy can queue transfers..
October 26, 200916 yr mm some interesting reading there Joe.. Thx for the pointers.. shame that the green-power drives can be written to singularly in excess of 65MB/s yet cant muster more than 9MB/s in the "raid" config. I would have thought that the parity drive would have been the one slowing things down as it is read/written more than the data drive? I notice when copying data to the drive that the main unraid reports approximately twice as many read and write operations on the parity drive over the data drive. Unfortunately it seems not to be the case. 9MB/s isnt too bad i guess.. thankfully teracopy can queue transfers.. What is missing here is the data drive has to be read and re-written also. For every write to a data drive there is a read of the data drive (and a slower RPM will delay this) a read from parity, a write to the data drive, then a write to parity. If you need to move allot of data at the fastest possible rate, use a cache drive. I would have thought that the parity drive would have been the one slowing things down as it is read/written more than the data drive? I notice when copying data to the drive that the main unraid reports approximately twice as many read and write operations on the parity drive over the data drive. That entirely depends on the sizes of the "writes" based on your physical disks and the controllers. Twice the "writes" can represent the same data, (unless there is some kind of bug that writes twice... but that would affect all of us) I can write 1000 bytes once, or 500 twice, and the same work gets done... BUT 500 twice will require the disks to rotate around even more. In other words (using 500 bytes only for this example, I've no idea of the actual sizes of the "writes" and "reads "of your disks) rotate to desired sector. read 500 bytes rotate once to get back to same sector write 500 bytes rotate once to get back to same sector read 500 bytes rotate once to get back to same sector write 500 bytes If your disk read and wrote 1000 bytes at a time (twice the amount) it would not need as many revolutions to complete the operation. I do not totally agree with this. Modern hard drives use a fixed sector size (512 bytes) and have large cache and logic to minimze head movement and rotational delays. What comes into play is rotational speed for filling up the hard drive's cache, elevator sorting logic for aligning sectors to minimize head movement and rotational speed for dumping the hard drive's cache. Two 500 bytes writes or a 1000 byte write are going to be cached and adjusted for sectors either way. Timing is everything, but logic and caching play a huge role in this too. I'm also wondering if reiserfs, kernel locking and journaling play a role in this. If the journal is constantly being re-written, I wonder if a caching controller will help with this It would be interesting to see how ext2fs would perform in this scenario.
October 26, 200916 yr I do not totally agree with this. Modern hard drives use a fixed sector size (512 bytes) and have large cache and logic to minimze head movement and rotational delays. True, but, this is not likely the size of the "writes" to the drives from the linux disk driver. The large "cache" is 32 meg... and that is FAR less than an entire cylinder. It is also shared when reading and writing, so it might be able to coalessce the reading of the first set of 512 byte sectors, but will not be able to hold an entire cylinder's worth of sectors. If it could, there would not be any need to spin around once more to read the next sector being written. NCQ might help, if the buffering does not get in the way... or it might just make things worse. Drives give their best performance when reading sequential sectors... unRAID just does not work that way. It reads, then writes the same sector, then reads and writes the next. If you could prevent the "write" from destroying what is in the 32 meg disk drive buffer, it would go more quickly, but we don't get that level of control of the disks. What comes into play is rotational speed for filling up the hard drive's cache, elevator sorting logic for aligning sectors to minimize head movement and rotational speed for dumping the hard drive's cache. Exactly Two 500 bytes writes or a 1000 byte write are going to be cached and adjusted for sectors either way. Timing is everything, but logic and caching play a huge role in this too. Also true, but I think Tom has said unRAID reads and writes 128k "stripes" so no operation fits in the disk drive buffer. I'm also wondering if reiserfs, kernel locking and journaling play a role in this. If the journal is constantly being re-written, I wonder if a caching controller will help with this It would be interesting to see how ext2fs would perform in this scenario. I suspect the kernel locking, including the locking of interrupts during disk operations is potentially one reason for some of the start/stop data rate seen on the LAN when disks are being written to, or when spinning up. Check out the hdparm command and you'll see unRAID uses interrupt request locking (at least for IDE disks) I don't know if something similar is involved for SATA disks. root@Tower:# hdparm /dev/hdb /dev/hdb: multcount = 16 (on) IO_support = 0 (default - 16-bit) unmaskirq = 0 (off) using_dma = 1 (on) keepsettings = 0 (off) readonly = 0 (off) readahead = 256 (on) geometry = 60801/255/63, sectors = 976773168, start = 0 When a disk operation is being performed, I/O interrupts from the LAN chipset might not be possible. Here is a very early discussion of the same issue, but with a serial port and concurrent disk access. http://lkml.indiana.edu/hypermail/linux/kernel/9811.2/0399.html The interrupt request masking has nothing to do with the type of file-system on the disk, it is much lower level than that, so changing to a different file-system may not have the effect you predict. (Only time will tell, when hopefully at some point in time we might be able to use alternate file-system types. Locking during file-system operations can certainly be involved, but they would be to keep the data and the journal entries on a disk sane.) Joe L.
October 26, 200916 yr Do we think 4096 byte sectors will become the "next standard" now SSDs are becoming more of a mainstream technology? I seem to remember HDD folks talking about moving to 4096 byte sectors several years ago yet 512 bytes still seems to be the defacto standard. As for the "stalling" question make sure the client or server isnt running another application like a torrent application or a background update (windows update) that is writing or reading from the disk. This causes exactly the behaviour you are describing.
October 26, 200916 yr shame that the green-power drives can be written to singularly in excess of 65MB/s yet cant muster more than 9MB/s in the "raid" config. It all depends on which green drive. My 2TB WD Green drives perform 15-18 MB/s in my unRAID config when copying Gigs of data, and yes, it's always at least double the size of physical memory. I don't know why the 1.5TB are that significantly slower than the 2TB.
October 26, 200916 yr Author As for the "stalling" question make sure the client or server isnt running another application like a torrent application or a background update (windows update) that is writing or reading from the disk. This causes exactly the behaviour you are describing. There is nothing running on the unraid server. its a vanilla 4.4.2 install with no other scripts installed yet. I disabled all other running applications on this machine when I was copying across to the server to eliminate the problem being at this end.
October 26, 200916 yr Author It all depends on which green drive. My 2TB WD Green drives perform 15-18 MB/s in my unRAID config when copying Gigs of data, and yes, it's always at least double the size of physical memory. I don't know why the 1.5TB are that significantly slower than the 2TB. Mine are the 1TB WD10EADS not the 1.5TB. Ive seen other users with these drives claiming 15+mb's so i'm not entirely sure this comes down to just the slower rotational speed of the green drives. I am going to try some of the tweaks Joe suggested earlier in this thread and see if that improves anything.
October 26, 200916 yr For comparison, I get 18MB/sec (1GB/min) over the LAN to an array drive disk share (WD 5400 rpm green 1TB) and parity is also WD 5400 RPM green 1TB. Data drive is on an 8-Port supermicro PCI (not PCIe) card and Parity is on the mobo. I get twice that writing from the same PC to the same server on the same LAN when writing to Cache and a little better writing to a RAMDisk. Yes, transfers to the array drive are very "spikey" and IIRC, using renice on pdflush may have helped that out in the past.
October 27, 200916 yr Author For comparison, I get 18MB/sec (1GB/min) over the LAN to an array drive disk share (WD 5400 rpm green 1TB) and parity is also WD 5400 RPM green 1TB. MM.. So much for slower rotational speed affecting performance. Wondering if my Seagate 1.5TB drive is one of the dodgy ones? Might pick up another 1TB WD10EADS today and see if I get any difference in performance from it as the parity drive. Worst case scenario is I end up with another 1TB data drive to replace one of the smaller ones I was going to put into the array anyway. Data drive is on an 8-Port supermicro PCI (not PCIe) card and Parity is on the mobo. I get twice that writing from the same PC to the same server on the same LAN when writing to Cache and a little better writing to a RAMDisk. Seeing as your data drive is on the PCI bus and parity on the mobo this "should" be a slower combo than mine with both drives on the PCI-e yet your speeds are double what mine are. Yes, transfers to the array drive are very "spikey" and IIRC, using renice on pdflush may have helped that out in the past. Yeah im noticing that. It peaks and ebbs as it transfers when its under parity.
October 27, 200916 yr When I had WD 5400 1TB drives as parity and data I only got around 8-10Mb/s. This was on the motherboard ports. Later I moved parity to a SIL3132 port and it still was not any faster. So motherboard and chipset could have something to do with it too.
October 27, 200916 yr Author When I had WD 5400 1TB drives as parity and data I only got around 8-10Mb/s. This was on the motherboard ports. Later I moved parity to a SIL3132 port and it still was not any faster. So motherboard and chipset could have something to do with it too. The P5Q-Deluxe uses the Intel P45/ICH10R chipset combo for its raid ports and a Silicon Image Sil5723 for the two "Drive-Expert" ports. When the parity drive was on the mobo I had it connected to a ICH10R port, now its on the Adaptec 1430SA PCI-e and while it seems to be marginally faster on the pci-e bus its still nowhere near what others are getting with similar spec'd setups. Both the ICH10R and Adaptec 1430SA seem to be pretty common components among builds here so I would have thought this to be a relatively safe combination for compatability. Ive just ducked across to my local PC store and they had no WD10EADS drives available so I grabbed a Samsung HD103SI "Eco Power" drive to try as parity. Will run it through the pre-clear overnight and see how it goes.
October 27, 200916 yr I grabbed a Samsung HD103SI "Eco Power" drive to try as parity. Since you're planning to swap your parity drive, use the chance to see what your write speeds will look like without the parity disk installed.
Archived
This topic is now archived and is closed to further replies.