January 16, 200917 yr I am experiencing slow write performance inside my unRAID server. Read performance is good, parity check speed is good, writes to the cache are fast. But when I try to copy things from cache to one of the data disks, I am only getting about 3.75 MB/sec. Slow writes to the array are largely invisible to me, because I typically put stuff on the cache disk and manually initiate a move onto the array overnight. But I looked last night and thought the stats on the Performance view of myMain looked substantially slower than the 4.3.3 days (I think I was getting close to 12-14 MB/sec - maybe even a bit more). And last night's move didn't finish overnight. I know for a fact that I have moved far more data overnight in the past. Anyone else noticing something similar. Any idea of things to tweak. I'm thinking of reloading 4.3.3 to confirm the slowdown. I do not know if writes to the array via Samba are also slow, but expect that it is likely that they are. Final note, I am copying directly to the disk mount point. No user shares or Samba involved. I am running a pretty standard configuration P5B VM DO 2G Ram 2 Adaptec 1430SA Controllers 16 disks (14 data + parity + cache) unRAID v4.4
January 16, 200917 yr I've had 4mbps writes since October when I switched to 5 1.5tb drives all on the ICH8 controller. I don't recall it being like that before, and I ran the beta version all summer for dual core support. Since I get 80 MBps to the cache drive I haven't looked into it at all.
January 16, 200917 yr Check the syslogs for slowdowns, a slowing of the DMA mode from the starting UDMA/133 or UDMA/100 all the way down to any PIO mode. It sure sounds like a PIO mode at that speed. The error messages just prior to the slowing should give you a hint as to what is wrong.
January 16, 200917 yr Author Please take a look at the log (attached). I couldn't find anything unusual. I ran a test to copy a 1G file from my Windows workstation to the unRAID server. In each case I did exactly the same thing and copied the exact same file to the exact same disk after a reboot. ... With unRAID version 4.4 it took 5:38 With unRAID version 4.4.2 it took 5:45 With unRAID version 4.3.3 it took 1:52 4.3.3 is 3x faster!
January 17, 200917 yr I moved all 7 of my disks onto 2 1430SA controllers and am now seeing around 12 MBps instead of 4 MBps in Unraid 4.4.2. The transfer speed isn't as fast as 4.3.3 (16 MBps) was though. I also went back to 4gb from 2gb, but I had speed issues before I had to borrow 2gb of the ram.
January 17, 200917 yr Please take a look at the log (attached). I couldn't find anything unusual. Except for a couple of network hiccups, worth checking out, I don't see anything unusual either. But this syslog only covers about 8 and a half minutes, and the timestamp of the file when saved is only a couple of minutes later, so it doesn't seem like this syslog covered one of the slow transfers. I don't know if it will be different, but try checking the syslog after you do one of those too-slow copies, to see if anything unusual shows up. And check an ifconfig afterward, to see if it is clean too. The network did not initially detect a link beat for several seconds, which is longer than usual, then got it for a few seconds, lost it again, then regained it. That seems a little odd, although it did negotiate a gigabit connection. I don't have any other ideas yet. Make sure those SMART commands in the startup aren't initiating SMART tests! All those add ons and startup stuff make me a little bit nervous. It is always standard in troubleshooting, to simplify, cut out all the extras temporarily, just in case... You might try, by renaming files, to put the stock go script on line and test a transfer, not that we expect to see a difference, but for completeness in troubleshooting. Of course, the stock go file doesn't have the read-ahead setra optimization, so speed might be even worse. You might want to try a stock go plus a single blockdev --setra 2048 /dev/md1 or whatever the testing destination drive is.
January 17, 200917 yr Author Except for a couple of network hiccups, worth checking out, I don't see anything unusual either. But this syslog only covers about 8 and a half minutes, and the timestamp of the file when saved is only a couple of minutes later, so it doesn't seem like this syslog covered one of the slow transfers. I don't know if it will be different, but try checking the syslog after you do one of those too-slow copies, to see if anything unusual shows up. And check an ifconfig afterward, to see if it is clean too. The network did not initially detect a link beat for several seconds, which is longer than usual, then got it for a few seconds, lost it again, then regained it. That seems a little odd, although it did negotiate a gigabit connection. I don't have any other ideas yet. Make sure those SMART commands in the startup aren't initiating SMART tests! All those add ons and startup stuff make me a little bit nervous. It is always standard in troubleshooting, to simplify, cut out all the extras temporarily, just in case... You might try, by renaming files, to put the stock go script on line and test a transfer, not that we expect to see a difference, but for completeness in troubleshooting. Of course, the stock go file doesn't have the read-ahead setra optimization, so speed might be even worse. You might want to try a stock go plus a single blockdev --setra 2048 /dev/md1 or whatever the testing destination drive is. Thanks for taking a look RobJ. Here are my comments to your comments ... 1. I was having some problems with my Internet connection during that time, but it did not seem to affect intranet activity. Will keep an eye on syslogs to see if networking issues continue. I was noticing the slowdown with networked I/O but also with direct copy within the unRAID server itself, so unlikely that the network hiccup is causing the slowdown (but it still may be a problem). I have been having some issues with my router recently (it is the DHCP server too) that has required reboots every week or two. I may need to update its firmware. 2. This syslog DOES cover the period of time of the copy. There were no log messages of any kind. I initiated the copy almost immediately after bootup, let it finish, captured the log, and then shutdown. 3. I will try taking the extra commands out of my bootup, but would be surprised if they are causing a problem. Remember, if I boot with 4.3.3 and run EXACTLY the same boot procedure, performance is normal. It is NOT possible that the smart procedure initiates tests, but an excellent thought! I guess bwm-ng could be causing some conflict but I doubt it (again works fine with 4.3.3). There is really nothing else much happening (copying custom timezone file, creating a few custom shares with Samba and reloading config, doing the setra tweak to 2k, and loading the addons for unmenu). I will try to boot plain vanilla and see if it makes any difference. 4. I think I may try moving my parity drive onto a motherboard SATA port (ICH8R) rather than the Adaptec 1430SA. Have a hunch that might make a difference. Romir and I discussed via PM and he realized a common element is that we are both running parity off a 1430SA. Thanks again. I will keep at it. Anyone other ideas are welcome!
January 17, 200917 yr Author Making progress ... I put 4.4.2 back in place and switched my parity drive to a motherboard SATA port (from the Adaptec 1430SA). Copying my 1 gig file went to 6:15. Even slower! I then rebooted and went into the BIOS changing the setting from AHCI to IDE for the ICH. The JMicron is still AHCI. I thien rebooted and copied the file in 1:30! There is definitely something wrong with the AHCI driver in the 4.4x releases. I remember all the hoopla about how important enabling AHCI was in the BIOS, and at the time I remember switching but losing a couple meg per sec on my parity check speed. But I figured the NCQ support was worth it. For now I think I will leave it in IDE mode. Romir - can you see if this change has an impact on your performance with 4.4? I have included another syslog with the IDE mode enabled. RobJ, if you have a chance to give it a once over I'd appreciate it. Thanks!!
January 17, 200917 yr Other than the drive moves and the change from the ahci driver to ata_piix, I don't see anything different. I don't know what to say about the AHCI change, it's not supposed to happen that way, but your improvement is noteworthy. It will be interesting to hear from anyone else, with different motherboards and chipsets, that can confirm this kind of improvement, by avoiding the ahci module. v4.4.2 feels about the same for me, as to performance, but I've only now installed it. So far, it's much better for stability, I've removed all boot options, and haven't had an exception yet, with any of the v4.4's with the 2.6.27.7 kernel.
January 17, 200917 yr Author I ran a parity check and saw a ~4MB/sec improvement. Not nearly as significant as the 3 TIMES performance difference on writes.
January 17, 200917 yr I started searching the net and I'm not sure if I found the complete problem but I think NCQ is the issue. I don't know if this is a chipset problem, a motherboard not being able to support NCQ or a driver issue but when I turned off NCQ I saw a huge performance increase. First make sure you're in AHCI mode and then disable NCQ with the following. To turn off NCQ do the following for each disk echo 1 > /sys/block/sdX/device/queue_depth you can than use cat /sys/block/sdX/device/queue_depth to see if it was set. So if you get a number greater than 1 it is enabled if its a 1 its disabled. Then do a parity check. I just started one and I had been seeing speeds of 55MB/Sec its now at 80MB/Sec. I haven't tried to write to the array yet but the results should be better too! I believe when you change to the IDE mode the ata_piix driver does not have NCQ support so it's disabled by default resulting in better performance. Ok as the parity check was taking place I enabled NCQ on one of the data drives and my parity speed dropped back to 55MB/Sec if you want to test this as well you can enable NCQ with the following command echo 31 > /sys/block/sdX/device/queue_depth I guess we need to edit the go script to disable NCQ every time we boot our systems. Erik EDIT: I thought I'd add I only have 6 SATA drives in my ARRAY all on the onboard ICH10 controller. Parity speeds might differ if you have more drives.
January 17, 200917 yr bjp999: Can you post syslog from a boot with AHCI enabled, and another with it disabled? I'd like to diff them and see all differences that flow from the change.
January 17, 200917 yr Author bjp999: Can you post syslog from a boot with AHCI enabled, and another with it disabled? I'd like to diff them and see all differences that flow from the change. I have posted 2 syslogs in this thread. The first one is with AHCI enabled and the second is with AHCI disabled (IDE mode on motherboard SATA ports). Both are with 4.4.2. If you need another syslog just let me know what version of unRAID, what setup you want (BIOS settings etc.), and what, if anything, you want me to do in unRAID before capturing a log. I'll be happy to assist any way I can.
January 17, 200917 yr bjp999: Can you post syslog from a boot with AHCI enabled, and another with it disabled? I'd like to diff them and see all differences that flow from the change. Here's another set. My 6 data & parity drives are connected to ICH8R and the cache drive is on the jmicron. I'm getting better speeds with AHCI now, around 11-12 MB/s, but IDE mode hovered around 17-18 MB/s. Edit: While in AHCI "mode" I disabled the NCQ queue following erikatcuse's instructions. I'm now seeing 17-18 MB/s copying from disk to disk in midnight commander.
January 17, 200917 yr Author I started searching the net and I'm not sure if I found the complete problem but I think NCQ is the issue. I don't know if this is a chipset problem, a motherboard not being able to support NCQ or a driver issue but when I turned off NCQ I saw a huge performance increase. First make sure you're in AHCI mode and then disable NCQ with the following. To turn off NCQ do the following for each disk echo 1 > /sys/block/sdX/device/queue_depth you can than use cat /sys/block/sdX/device/queue_depth to see if it was set. So if you get a number greater than 1 it is enabled if its a 1 its disabled. Then do a parity check. I just started one and I had been seeing speeds of 55MB/Sec its now at 80MB/Sec I haven't tried to write to the array yet but the results should be better too! I believe when you change to the IDE mode the ata_piix driver does not have NCQ support so it's disabled by default resulting in better performance. Ok as the parity check was taking place I enabled NCQ on one of the data drives and my parity speed dropped back to 55MB/Sec if you want to test this as well you can enable NCQ with the following command echo 31 > /sys/block/sdX/device/queue_depth I guess we need to edit the go script to disable NCQ every time we boot our systems. Erik Great detective work Erik! I hope this will help Tom I tried the same. With NCQ queue depth set to 1 on all drives, my performance copying that 1G file was 1:31 seconds! Virtually the same was with IDE mode. Parity check speeds were similar to IDE mode (about 4 MB/sec faster than with queue depth at 31) - about 57,000 KB/sec. Did not get the dramatic increase in speed to 80 MB/sec that I was hoping for based on Erik's post. Not sure if the WD GP drives are slowing me down, but that's what I am guessing. Still, no complaints! I moved the parity drive between the motherboard and the Adaptec controller and got virtually the same results.
January 17, 200917 yr Author The plot thickens ... I rebooted into 4.3.3 to see if reducing the queue depth had any impact. I found a very strange thing. When I checked the queue depth, I found that the first 8 hard disks had it ALREADY set to 1. The last 8 were set to 31. There was no consistency with the controller (the first 8 were on a mix of controllers). The parity disk is "sdl", so not in the first 8. (BTW, the USB boot disk always seems to have this set to 1.) So I ran a shell script to set them all to 1 - and got "Permission Denied" errors on the first 8 (I was just setting them back to 1, so not a big deal. But this would stop any attempt to set them to a higher level in 4.3.3). The last 8, however, did get reduced to queue depth of 1. I ran a parity check and the speeds were around 53 MB/sec. (About the same as I've been seeing with 4.3.3 with in AHCI mode.) BTW, the 4.4.2 parity check with NCQ set to 1 was running at 60/61 MB/sec when I stopped it. That is faster than I have ever seen with 14 drives being checked. Although not 80, a substantial bump from 53/54 I had been seeing.
January 17, 200917 yr Great detective work Erik! I hope this will help Tom I tried the same. With NCQ queue depth set to 1 on all drives, my performance copying that 1G file was 1:31 seconds! Virtually the same was with IDE mode. Parity check speeds were similar to IDE mode (about 4 MB/sec faster than with queue depth at 31) - about 57,000 KB/sec. Did not get the dramatic increase in speed to 80 MB/sec that I was hoping for based on Erik's post. Not sure if the WD GP drives are slowing me down, but that's what I am guessing. Still, no complaints! I moved the parity drive between the motherboard and the Adaptec controller and got virtually the same results. What brand is your parity drive? You could try a hdparm -tT /dev/sdb on all your drives to see which gives you the best performance and change that to your parity drive. That what I had done prior to finding the NCQ problem and I saw a slight increase. You're only interested in is the very last number, in MB/sec. Link to Wikihttp://www.lime-technology.com/wiki/index.php?title=Check_Harddrive_Speed Also I only have 6 drives in my array so parity speeds will probably be faster since I don't have as much overhead.
January 31, 200917 yr Just quoting myself from here, As to the AHCI/NCQ issues related above, I have done some limited research, and found hints that it may be related to certain WD drives. That is, certain WD drives on detecting NCQ, will disable read-ahead, which obviously will severely impact sequential read performance. It basically negates the drive cache. I can't help thinking this may be what is causing the slowdowns reported above. We need more testing, to determine if it really has anything to do with AHCI, and if not, is it just a specific WD drive problem. For those with the slowdowns, the workaround suggested above seems like a very good idea, especially if it can be confirmed that it is not *directly* related to AHCI. If it proves to be just a WD / read-ahead / NCQ issue, then I would suggest users putting some pressure on WD to provide a firmware release without that behavior. We clearly need more research and testing first. Too late, but I should have written it here, and put a link there... I do agree, good detective work. More detective work is needed now. For one, I need to find those WD comments, and which WD drives were mentioned, but I'm almost positive I remember that they included WD drives mentioned above. The site where I *think* I found those comments is down today.
February 12, 200917 yr This has made a big difference for me in unRAID 4.4.2 - informal testing with big files (~2GB) indicates write speeds with queue depth of 31 at ~4MB/sec, jumping to ~12MB/sec when queue length is reduced to 1 for both drives. System specs: ICH10 in AHCI mode with Seagate 7200.11 1TB and WD 1001FALS 1TB (Caviar Black). Intel DG43NB mobo (integrated intel NIC) with 2GB DDR2 and a 1.6GHz dual-core Celeron.
February 13, 200917 yr Not sure if my post here is warranted. But maybe someone can let me know if my speeds seem about right. Unraid 4.4.2 Intel D975XBX2 (using all 8 onboard controllers) 3 Adaptec 1430SA PCI Express controllers. Cache Drive - ST3500641AS 500GB Parity Drive - ST31500341AS 1.5TB 15 Drives in Array - ST31500341AS 1.5TB With AHCI turned off (IDE Mode) - parity check at 33MB / Sec With AHCI turned on - parity check at 38MB / Sec Confirmed all drives are set to NCQ of 1. (None of my drives are WD - so I did not expect to see that fix work) hdparm test results - 500GB Cache drive - Timing Cache Reads 1760 MB/sec, Timing Buffered Disk Reads = 59 MB/sec All 1.5 Drives test similar - Timing Cache Reads 1700-1800 MB/sec, Timing Buffered Disk Reads = 120-128 MB/sec I am posting my information - because at one time I was getting 80-100MB / Sec parity check (my first build using Super Micro board and only 7 1.5 TB drives) I did expect a drop off in performance with a maxed out array - but is 38MB/Sec parity check about right? When transfering files to the array - (no cache drive) - I get about 8mb / sec to some drives and 12mb / sec to other drives. Any tips? Or are these results to be expected?
February 13, 200917 yr This link no longer seems to work: http://www.mediafire.com/?sharekey=a16ecda04074777d111096d429abd360e04e75f6e8ebb871 Edit: upgraded qd to version 3, added logging to syslog, packed in qd3.zip Above Attached below is qd3.zip containing qd, a simple script to display and set the queue_depth's for drives with NCQ support. The first run of it saves the original values, if they have not been modified!! Probably best run after a fresh boot, with no go script modifications. It is designed only for the current need, of testing how different queue_depth values affect performance. Syntax: qd displays all of the current and original queue_depths of drives with NCQ support qd sdX displays the current and original queue_depth of sdX (sda to sdz) qd off disable all NCQ by limiting all queue_depths to 1 qd on restore the original NCQ (queue_depth) values Note: you should not set the queue_depth higher than it was originally set by the kernel. The Linux kernel at boot time determined the highest depth supported by that drive. Partly why I created qd was to enforce that, as well as make an easier and safer way to manage the qd values.
February 13, 200917 yr Author Not sure if my post here is warranted. But maybe someone can let me know if my speeds seem about right. Unraid 4.4.2 Intel D975XBX2 (using all 8 onboard controllers) 3 Adaptec 1430SA PCI Express controllers. Cache Drive - ST3500641AS 500GB Parity Drive - ST31500341AS 1.5TB 15 Drives in Array - ST31500341AS 1.5TB With AHCI turned off (IDE Mode) - parity check at 33MB / Sec With AHCI turned on - parity check at 38MB / Sec Confirmed all drives are set to NCQ of 1. (None of my drives are WD - so I did not expect to see that fix work) hdparm test results - 500GB Cache drive - Timing Cache Reads 1760 MB/sec, Timing Buffered Disk Reads = 59 MB/sec All 1.5 Drives test similar - Timing Cache Reads 1700-1800 MB/sec, Timing Buffered Disk Reads = 120-128 MB/sec I am posting my information - because at one time I was getting 80-100MB / Sec parity check (my first build using Super Micro board and only 7 1.5 TB drives) I did expect a drop off in performance with a maxed out array - but is 38MB/Sec parity check about right? When transfering files to the array - (no cache drive) - I get about 8mb / sec to some drives and 12mb / sec to other drives. Any tips? Or are these results to be expected? This seems slow to me. I have 15 drives (14 data + parity) on my P5B VM DO (motherboard + 2 1430SA). I get 50-60 MB/sec speeds on parity checks. Several of my disks are the older WD GP drives and older Seagates (7200.10) with lower densities. I would not expect that you'd be slower than me with those drives. (Not that 38 MB/sec is bad, just that I don't understand why it is that speed and not faster). I would suggest that you boot 4.3.3 and compare your results. General note - the changing of the queue_depth may have some minor impact on my READ performance. but the SEVERE slowdowns are only present on WRITES.
February 13, 200917 yr http://www.mediafire.com/?sharekey=a16ecda04074777d111096d429abd3602b04092a0e308ed35be6ba49b5870170 Above is qd.zip containing qd, a simple script to display and set the queue_depth's for drives with NCQ support. The first run of it saves the original values, if they have not been modified!! Probably best run after a fresh boot, with no go script modifications. It is designed only for the current need, of testing how different queue_depth values affect performance. Syntax: qd displays all of the current and original queue_depths of drives with NCQ support qd sdX displays the current and original queue_depth of sdX (sda to sdz) qd off disable all NCQ by limiting all queue_depths to 1 qd on restore the original NCQ (queue_depth) values Note: you should not set the queue_depth higher than it was originally set by the kernel. The Linux kernel at boot time determined the highest depth supported by that drive. Partly why I created qd was to enforce that, as well as make an easier and safer way to manage the qd values. Very cool RobJ. I did some initial testing with changing the queue_depth and it did not affect my speeds much. I will be doing some more rigorous testing on how this affects my drives (reading and writing) and then see if this tool is necessary. One thing I would love to see is for this tool to become an addon to unMenu. Perhaps in the Package Manager section, or in the User Scripts section.
February 13, 200917 yr Not sure if my post here is warranted. But maybe someone can let me know if my speeds seem about right. Unraid 4.4.2 Intel D975XBX2 (using all 8 onboard controllers) 3 Adaptec 1430SA PCI Express controllers. Cache Drive - ST3500641AS 500GB Parity Drive - ST31500341AS 1.5TB 15 Drives in Array - ST31500341AS 1.5TB With AHCI turned off (IDE Mode) - parity check at 33MB / Sec With AHCI turned on - parity check at 38MB / Sec Confirmed all drives are set to NCQ of 1. (None of my drives are WD - so I did not expect to see that fix work) hdparm test results - 500GB Cache drive - Timing Cache Reads 1760 MB/sec, Timing Buffered Disk Reads = 59 MB/sec All 1.5 Drives test similar - Timing Cache Reads 1700-1800 MB/sec, Timing Buffered Disk Reads = 120-128 MB/sec I am posting my information - because at one time I was getting 80-100MB / Sec parity check (my first build using Super Micro board and only 7 1.5 TB drives) I did expect a drop off in performance with a maxed out array - but is 38MB/Sec parity check about right? When transfering files to the array - (no cache drive) - I get about 8mb / sec to some drives and 12mb / sec to other drives. Any tips? Or are these results to be expected? This seems slow to me. I have 15 drives (14 data + parity) on my P5B VM DO (motherboard + 2 1430SA). I get 50-60 MB/sec speeds on parity checks. Several of my disks are the older WD GP drives and older Seagates (7200.10) with lower densities. I would not expect that you'd be slower than me with those drives. (Not that 38 MB/sec is bad, just that I don't understand why it is that speed and not faster). I would suggest that you boot 4.3.3 and compare your results. General note - the changing of the queue_depth may have some minor impact on my READ performance. but the SEVERE slowdowns are only present on WRITES. I am a little raw when it comes to unraid - do I need to do anything special to revert to 4.3.3? I first started with 4.4.2.
Archived
This topic is now archived and is closed to further replies.