Pauven

I found this related to the three md_ parameters: http://lime-technology.com/forum/index.php?topic=15224.msg142767#msg142767 In summary, in seems that increasing all three values is a good thing, as long as you have enough memory. The default settings were optimized for servers with 512MB ram, which makes sense because stock, I typically see about 400mb used for parity checks. With 4GB, I have a bit of spare room to play. A couple notes: md_sync_window + md_write_limit should be < md_num_stripes. Currently I'm not configured correctly, as my md_sync_windows (896) + md_write_limit (768) = 1664, which is 384 more than my md_num_stripes(1280). Some people are blindly increasing these numbers to the theoretical maximum (i.e. I have 4GB RAM, which is 8x more than the default server size of 512MB, so simply increase all numbers x8). This is a very bad idea. Here's why: Certain tasks on a server will consume much more memory, beyond what is used for parity checks/rebuilds. A good example is the linux mv command (move). Sometimes I like to telnet onto the box and run a mv to move folders from one physical drive to another. I notice my memory utilization skyrockets from 400 MB to pretty much 4GB, 100% utilization. I've even seen my server come to a crawl when doing mv's at the same time I was pre-clearing a couple drives. True, a cp (copy) followed by a rm (delete) is more memory efficient, so I do that now, but the point is, you need to leave lots of memory for other processes (some Linux, some unMENU, some Simple-Features, some other plugins). Do not blindly try to increase these values to use up all memory, as you will probably only crash your server. After my rebuild is done (running great by the way), I plan to double md_num_stripes to 2560, and increase md_write_limit to 1024, and increase md_sync_window to 1280. Then I'll do some parity speed tests again. Now, more than ever, I see a strong need to go 64-bit for more memory. Perhaps 5.1....

I have excellent news to report: the Parity Check finished in 10h30m. My last rebuild took 10h45m, so sure enough the md_sync_window settings allowed the Parity Check to run a little bit faster than the rebuild. My next test will be to upgrade a 1.5TB disk to 3TB, and measure how long the rebuild takes, to see if it was affected by the change. I wish I had more answers regarding the correct settings for md_sync_window, for the most part I'm just happy it is working. I don't imagine every server would need the same settings (after all, this is a tunable parameter for a reason). I think that SAS/SATA controller chipset is the main impactor, as I simply migrated my current HD over to my new server build with the 2760A when the problems arose. Things to think about: If the SAS/SATA controller chipset does have an impact, server builds that use multiple different chipsets (i.e. MB SATA ports + add-in card) might not have a 'sweet spot'. The md_num_stripes is also a tunable... what does it do? We do know that md_sync_window can't be set larger than md_num_stripes, but there's a lot more to learn. Same for the md_write_limit. I wonder if this was controlling rebuild speed... Why was the md_sync_window value so low to start with? Is there a danger in having a higher default value? The defaults work fine for many controller cards (like my Adaptec 1430SA) and MB ports. I did see user jbartlett mention he encountered CPU stalls (whatever those are) with higher values. I wouldn't change any values without beforehand knowledge or guidance, either from Tom or an experienced user. I do recall a lot of posts regarding the three md_ settings in the past on this forum. These were values that expert users wanted, and Tom opened them up. Some savvy searching and you might find the answers to all our questions already hiding in the forum somewhere. I'm also somewhat surprised that such a simple solution to the SAS2LP/88SE9485 performance issues was not unearthed before now. Regardless, I'm pleased that other users are seeing positive results. -Paul

Tom, I completed my preliminary testing, and so far the results are very promising! Thank you. Details are here: http://lime-technology.com/forum/index.php?topic=27460.msg244639#msg244639 It would be very nice if there was an "AutoTune" button in unRAID, it could run a script to perform the tests that I just did manually. A nice to have for the future, and it might even cut down on support requests... I'm performing a full Parity Check now with the best settings from my preliminary results, and after that a full Rebuild to upgrade a drive. -Paul

Tom has directed me to test both "md_sync_window" and "Force NCQ disabled" to see if parity check speeds improve on the HighPoint 2760A. I increased md_sync_window from a default of 384 all the way to 1280, in increments of 128. Note, I am not sure if md_sync_window needs to be in increments of 128, but that's what I used in my testing. All of my tests were performed to a Parity Check (NoCorrect) position of 1.0% only. I would start the Parity Check, then monitor on the unMENU>MyMain>Detail status screen, manually refreshing to see the counter increment from 0.9% to 1.0%, then immediately cancel the Parity Check. I would pull the start and end times from the log. I found this method to be accurate and repeatable (and a whole lot quicker than waiting 10-14 hours for a full Parity Check). Time in Seconds Time in Seconds md_sync_window No NCQ W/ NCQ 384 439 395 512 296 299 640 275 280 768 270 274 896 269 269 1024 270 270 1280 271 275 I found the best results with a md_sync_window value of 896, regardless of whether NCQ was enabled or disabled. At this value, the Parity Check speed appeared to be equivalent to a Rebuild, possibly event a bit faster. Memory utilization did increase less than 1% from increasing the md_sync_window value. I typically use a little over 10% of my 4GB of RAM, and I noticed almost 11% used with a md_sync_window value of 1280. According to Tom, each increment uses 4K of memory per drive, so going from 384 to 1280 on my 16 drive server would only use an additional 56 MB. In general, NCQ didn't seem to have an impact except at the default md_sync_window value of 384, where having NCQ on seemed to help. Since it doesn't seem to help when md_sync_window is properly tuned, I plan to leave it off, since I have (possibly unfounded) fears about NCQ interfering with a parity check or rebuild when I'm simultaneously writing to the array, which does happen from time to time. I will now kick off a full Parity Check with md_sync_window set to 896 and NCQ disabled. Once that is complete, I will upgrade another 1.5TB drive to a WD Red 3TB drive. Hopefully the server will produce equivalent run times for both jobs.

Tom, I've conducted the test 3 times now, switching back and forth between Force NCQ Disabled, and rebooting after each change. Each time I let the Parity Check NOCORRECT run until exactly 1.0% complete (measured on an unMenu screen that does not show any SMART data), then immediately stop it, and measure the elapsed time in the log. Obviously not a full-scale test, but should be representative of the impact of the change. After we've tweaked a few more settings to hone in on a good configuration, I'll run a 100% parity check. I'm getting approximately 10% shorter times with NCQ ON (not disabled). This is a step in the right direction, though I would expect that this may have a similar result during a rebuild. IF NCQ improves rebuild speed by 10%, then the 30% speed gap during rebuild and parity checks would remain. I have one last 3TB drive waiting in the wings to upgrade one of my 1.5TB drives. After we've gone through all the tweaks and tests on the parity check speed, and settled on a correct configuration, I will do a rebuild to upgrade and see how the changes affected the rebuild speed.

Here I thought I was the only one that thought like that! I was considering building a new house and was wanting to build it with double offset wall to eliminate thermal bridging which could get the side walls around R47 (4" Closed cell foam and 6" R19 bat), of crouse the house would be sealed so tight that it would cost me an air exchanger! I also paid full price for my Galaxy Note 2 smart phone, so that I could have a cell bill that was less then $20 a month with Ting. That one actually does pay back in less then a year. Wow, you guys are great. I upgraded two contractor grade 10-SEER heat pumps to 15-SEER a year ago. I expected to see huge energy savings, but somehow they never materialized. I feel a little jaded. I also have a TED (The Energy Detective) installed. If neither of you have one of these, you are truly missing out. I remember the first time I saw the washer and dryer (elec) running at the same time and I saw 12KW being pumped into my clothes! Makes the 10W savings for a computer component seem like chump change. The TED helps you save energy by educating you on your own energy use. Though oddly, one day I turned off and unplugged everything I could find in the house, and I still had about 600W going somewhere. I need to go through that process again, as whatever is causing that energy drain appears to be going 24/7, and costing me about $600 year. Bleh, I feel sick just talking about it. Time to go hunting...

I read that as a maximum power rating. The 2760A was rated 55W, if I recall correctly. If it idles at 50% like the 2760A, then the Areca might have a 14W idle. Purely theoretical, of course, but certainly has an opportunity to save a good number of watts.

No, I've never touched these before. I will go ahead and test, just need some clarification below. Thank you. What is an increment? Is 384 to 512 an increment or 128 increments? If 128 increments, then going from 384 to 512 on a 16 drive server will use an additional 8MB? Do I need to increase to specific intervals (by 128, by powers of 2, by 8, etc.) or to specific targets (384, 512, 768 only)? Testing now, thanks.

Thanks for the tip garycase! I replied to Tom's post. Fingers crossed.

Tom, thank you for taking a look at the parity check issues with the SAS2LP. I have some relevant info, and since the forum is such a large place, I'm sure my post went unnoticed by your eyes. I have a HighPoint 2760A, which is notable in that it uses the same Marvel 88SE9485 SAS/SATA controller chip as the SAS2LP, the same mvsas driver, and in that I am experiencing the exact same parity check performance issue. My first awareness of the issue in my brand new build: http://lime-technology.com/forum/index.php?topic=27460.msg242873#msg242873 My verification that the 2760A was connecting at PCIe 2.0 x16: http://lime-technology.com/forum/index.php?topic=27460.msg243813#msg243813 dskt.sh test results showing controller achieves max throughput: http://lime-technology.com/forum/index.php?topic=27460.msg244049#msg244049 My conclusion of the source of the issue: http://lime-technology.com/forum/index.php?topic=27460.msg244334#msg244334 All of the links are from the same thread, so you could just read through from the first link. There's other nuggets of info in that thread that I didn't specifically link to, which may be of interest to you. Please let me know if I may be of assistance. -Paul

As a quick follow-up, the above test resulted in the exact same processing speed. I think my previous adjustment, moving the 3TB Red drives onto the same controller, had an impact because of the large speed differential between the Reds (~140MB/s) and my Samsung F2/F3's (~105MB/s). Since the Samsung drives are about the same speed, regardless of size, splitting them onto separate controllers had no benefit.

I finally looked up the price of the Adaptec 72405... Yikes! That's a 50% premium over the 2760A. The cables are more expensive too. If it saves you 10W over the 2760A, it's going to take about 30 yrs (at 11 cents KWh) to break even on the price premium. If that 18W figure is the maximum power (I don't know if it is) and it idles even lower (sure would be nice), then we might be looking at a 20W savings on average. Still, 15 years for ROI just isn't cost effective over the 2760A. As garycase points out, the lower heat output sounds desirable, but I also noticed in the article they mentioned throttling if the 72405 doesn't get enough airflow, which makes me wonder if the cooling solution has cut some corners. Now, if you want to save electricity at any cost, then I salute you, as you are a better man than I am. I certainly had a goal of making a low power 24-bay server, but at a certain point chasing a few more watts just becomes too expensive for me. I've also had to redefine what a low-power 24-drive server even means. When I started my quest, I thought <20W was the target. After my build is done, and I've seen where the power flows, I would say anything in the 50W range is world-class. I don't think you are going to get better performance with the 72405, as the 2760A is already faster than the hard drives it is connected to, so the HD's are the bottleneck. I say that assuming that one day unRAID won't have the parity check throughput issue that is afflicting the 2760A. That Supermicro X9XPV-M3-3UE also looks interesting... how much does it cost and how much power will it consume at idle? I picked up the Celeron G1610 and Foxconn H61S for about $100 total. That's cheap! I still got the benefits of a PCIe 3.0 x16 slot, it has plenty of horsepower, and I was able to achieve a 17W-18W idle (not counting case fans, controller card or HD's). There's not much room to save more power on the CPU/MB combo. 10W saved is about $10/yr, so you might be looking at a very long return on investment. An in case you're curious, I've calculated the idle wattage of my build, once scaled up to 24 WD Red 3TB drives, to be 76 Watts. It's surprising how fast the watt counts goes up once you add in things like case fans and backplanes (I see 10W going to three 120mm case fans, and about 6W going to the SAS backplanes).

I've still got a couple weeks to decide to keep or return the 2760A. I'm a little on the fence. Pros: It works with unRAID out of the box! It's very fast, as long as you're not doing a parity check It's a single-slot solution to a 24 bay server It is somewhat price competitive, on a per drive basis It boots incredibly quick (no 5-minute bios boot sequence) Marvell says their 88SE9485 SAS/SATA chips are "the most energy efficient in their class" Parity Checks/Rebuild energy consumption is way down from previous build (may also be CPU choice) Cons: 28 Watt Idle power consumption (measured) 55 Watt Max power consumption (theoretical) Runs hot, even at idle 160+ degrees measured under load in a well ventilated, high airflow case Exposes an issue with unRAID during Parity Checks - ~30% slower Probably still way overkill for an unRAID box It's not cheap I don't feel like I got my money's worth due to power consumption and unRAID parity check issues I found your link while looking for other options to consider. I was one of the few unRAID users to try the Adaptec 1430SA adapter, and to be honest it was an excellent controller and worked great with unRAID 4.3 through the current versions. There is a risk with the 3.4.x Linux kernel versions not having driver support out of the box, since it is already a little old, so if you do try it out (and I hope you do) definitely purchase from a shop that will let you return it. I would actually try it myself if only it had the SFF-8087 mini-SAS connectors; I would have to buy $$ cables to try this card. Did you find any other options for 24 port controller cards? I'm willing to give an 8-port HBA with port multipliers built in a try.

For the first time I really feel every AOC-SAS2LP-MV8 owner's frustration with unRAID and Tom. Even a day ago I was willing to go to bat for Tom, saying there's no way he can be held responsible for the quality of the drivers being integrated into the various Linux kernels. I thought he was just along for the ride like the rest of us. But after a series of tests, I believe I can now conclusively say that 'something' is wrong with unRAID. The parity check performance issues I am experiencing are not a driver issue. First I used UhClem's handy script to evaluate controller performance, and while I was expecting to see some type of bottleneck, the 2760A performed flawlessly. It delivers 100% of single drive performance even when all drives are accessed simultaneously. Kudos to HighPoint, Marvel, and the various developers of the mvsas driver. I just ran a rebuild to upgrade a disk, and the rebuild screamed along at max throughput, only limited by the speed of my slowest drive. In my dskt.sh tests, the slowest drive was about 102 MB/s, and sure enough the rebuild started at about 102 MB/s, and at 15% was still close to 100 MB/s. Kudos to Tom and unRAID, rebuilds are performing perfectly. But then there are parity checks. In theory, parity checks should be pretty much the same speed as a rebuild. The only difference is that one drive is written to rather than read from. In my previous build, using the Adaptec 1430SA controller cards, a full rebuild and a full parity check both completed within less than a minute of each other. That's right, for an 11+ hour job, both completed with less than one minute of variance. But the controller cards using the Marvell 88SE9485 SAS/SATA controller are dramatically slower on a parity check. How much slower? The rebuild I just completed took 10h45m (~77 MB/s), while the parity check took 14h17m (~56 MB/s). 27% slower processing speed, resulting in a 33% longer running time. On the Adaptec 1430SA controller cards, both rebuild and parity took 11h17m (~72 MB/s). The variance doesn't make sense. It is the same hard drives, the same motherboard, the same memory, the same cpu, the same cables, the same controller card, even the same drivers, and the jobs being executed are very similar. But the parity check is significantly slower than a parity rebuild. At this point, the only culprit left is the unRAID parity check code. And this is why I'm now joining the camp of frustrated Marvell 88SExxxx series controller card owners. For a very long time now, years if I'm not mistaken, users have been telling Tom there is a problem. More unRAID users are running Marvell powered add-in cards than any other provider. Even unRAID's new shipping servers utilize a Marvell 88SExxxx series hd controller chip, though perhaps not a model affected by this issue. Tom Harms, aka Tom#2, in your daily/weekly/monthly status reports with Tom#1, please help illustrate that this is a long running code issue in unRAID that needs to be addressed. Please feel free to reach out to me for troubleshooting assistance. I'm willing to help in any way possible. And I'm not the only one. And Tom#2, in case you are unsure why this matters, this is more than a convenience factor. Parity checks are the most common reason that all drives are spun up for long periods of time, generating high heat, and accelerating wear and tear. When a parity check runs slow, these drives are spun up longer, significantly increasing wear and tear, reducing lifespan, and wasting electricity, and ultimately costing the users more money. As long as parity checks to secure our data are a necessary evil of the great product that is unRAID, we need efficient code that doesn't contribute towards premature hd failures.

Thanks for the links electron286! Those first 4 links talk about mvsas changes between kernels 3.5 and 3.7. I don't think any unRAID beta/RC has ventured beyond 3.4, so I'm not sure that info is relevant (but perhaps a little scary about going to newer kernel releases!). I'm a bit confused, though, as I only see mvsas 0.8.16 mentioned in this timeframe - are the mvsas version numbers not changing with the changes? That last link, from Marvell, doesn't have any date or solid release info, but since it mentions 2.6.5, I think it is old old info.

© was in reference to the PCIe bandwidth and/or system bus bandwidth between the PCIe bus and the CPU, not the CPU horsepower itself. I mentioned that because in my previous server I had an older AMD CPU AM2 that only supported HT 1.0, and parity checks on all the drives at the same time appeared to hit a throughput brick wall. I was actually thinking of swapping the processor for an AM2+/AM3 model that supports HT 3.0, which should have removed that bandwidth bottleneck. I agree 100% that the CPU is not being maxed out by parity checks.

Below are my results from using the dskt.sh script mentioned in my previous post. My original intention was to simply test a single Marvel 88SE9485 SAS/SATA controller on the 2760A, and I thought I would get the most informative results by putting all of the same drive model on a single controller. Since I have 8 3TB Red data drives, I moved them all to the first controller. I also moved the remaining 7 data drives (Samsung 1.5TB & 2TB) onto the second controller chip, leaving only the parity drive on the third controller chip. I then ran dskt.sh both with and without the X parameter, getting individual drive max speed and concurrent drive max speed. Both tests were run 3 times, and the totals below are the average of the 3 runs. I tested all 15 drives concurrently, as my test showed I didn't need to focus in on the 8 Red 3TB drives on the single controller chip. Somewhat surprisingly, a performance issue didn't show up. As you can see, the tested concurrent drive throughput is about 99% of max throughput, and the 1% delta is well within the margin of error. This leads me to believe I am getting max throughput. Confused by the results, I started up a parity check with the new configuration, and again I was surprised to see speeds in the 60 MB/s range. I had expected to see speeds drop to 40 MB/s, like my previous tests with all data drives onto the first two controllers. Apparently each Marvel 88SE9485 works better with all matching drives. Since my parity check times are still over 2 hours longer than my previous build (using the exact same drives) I'm going to do one more test. I will move the parity drive to the motherboard's onboard SATA port, and move the Samsung 2TB drives to the 3rd controller. That way, each controller only has one drive type/size. Then I'll test again. DRIVE NAME, CONTROLLER AND TYPE INDIVIDUAL DRIVE SPEED CONCURRENT DRIVES SPEED sdc - Ctrl 1 - 3TB Red 141 MB/s sdd - Ctrl 1 - 3TB Red 151 MB/s 138 MB/s sde - Ctrl 1 - 3TB Red 146 MB/s 146 MB/s sdf - Ctrl 1 - 3TB Red 138 MB/s 134 MB/s sdg - Ctrl 1 - 3TB Red 140 MB/s 138 MB/s sdh - Ctrl 1 - 3TB Red 140 MB/s 139 MB/s sdi - Ctrl 1 - 3TB Red 145 MB/s 144 MB/s sdj - Ctrl 1 - 3TB Red 143 MB/s 142 MB/s sdk - Ctrl 2 - Smsng 1.5TB 107 MB/s 107 MB/s sdl - Ctrl 2 - Smsng 1.5TB 106 MB/s 105 MB/s sdm - Ctrl 2 - Smsng 1.5TB 104 MB/s 101 MB/s sdn - Ctrl 2 - Smsng 1.5TB 102 MB/s 103 MB/s sdo - Ctrl 2 - Smsng 2TB 102 MB/s 101 MB/s sdp - Ctrl 2 - Smsng 2TB 107 MB/s 106 MB/s sdq - Ctrl 2 - Smsng 2TB 114 MB/s 114 MB/s TOTAL 1887 MB/s 1859 MB/s 142 MB/s

garycase thank you for the info. I theorize that parity speeds improve beyond 1.5TB and 2TB for three reasons: A) Smaller drives on the slow inner cylinders drop out of the mix (well documented phenomenon) B) The # of drives a controller sees is reduced, reducing overall workload C) The total volume of data on the system bus is reduced as drives drop out of the mix Part of my theory is that, for any given SATA/SAS controller, it is capable of max work Z. That maximum amount of work can be divided between actual throughput X, and overhead Y, but X+Y can never be greater than Z. In different kernel and driver revisions, some code changes are causing additional overhead, taking away max throughput. Some of these code changes have been to address critical bugs (well noted in the mvsas drivers), and may very well be required for stable operation. It's also possible that the overhead code is inefficient, stealing away processing cycles from throughput to implement the the overhead code. This thread gives an excellent way to test for controller throughput saturation outside of unRAID: http://lime-technology.com/forum/index.php?topic=25676.0 That thread gets really interesting on the third page. They are using the dskt.sh script to call hdparm and test both hd maximum speed individually (with the X parameter), and controller max speed by testing all hd on that controller at the same time (without the X parameter). The test script is written by UhClem and available here: http://lime-technology.com/forum/index.php?topic=17851.0 I am going to perform these tests now to see if there is a trend.

The AOC-SASLP-MV8 card is based upon the older Marvell 6480 host controller. Luckily it uses the same mvsas driver, but not exactly an apples to apples situation. Since the mvsas driver is used, then I'm hopeful that Tom might focus some attention on it. Of course, the server specs have this little disclaimer at the bottom: Tom could simply give up and try a different controller. I agree with your assessment of the other thread, though many people did point out that they had issues even without Simple Features, so it should be obvious to Tom that the core issue seems to linked to the Linux kernel, and possibly the mvsas driver, in each unRAID build.

I have recently built a 24-bay server using a HighPoint 2760A 24-drive controller card. This card uses the same Marvell 88SE9485 controller chip as the AOC-SAS2LP-MV8. Actually, the 2760A is like three AOC-SAS2LP-MV8's in one, as the 2760A has 3 Marvell 88SE9485 controller chips (instead of just one) which are connected by a PLX PCI bridge. I'm mentioning all this because the 2760A is having a similar parity speed performance issue as described for the AOC-SAS2LP-MV8. Interestingly, I've been able to conduct a special test on the 2760A that is difficult or impossible with the AOC-SAS2LP-MV8, and I wanted to share those results. I currently only have 16 drives in the server (15 data, 1 parity). When I first installed the drives in the server, I simply filled up the drive bays starting at the top and working down. That resulted in the first 88SE9485 chip having 8 data drives on it, the second 88SE9485 having 7 data drives on it, and the third 88SE9485 only having the one parity drive on it. In that configuration, I had parity check speeds of 4MB/s under 5RC6, and 40MB/s under 5RC12a. As a troubleshooting step, I moved the drives around on the controller to spread out the load more evenly. The first two 88SE9485 chips now had 6 drives on them, and the third 88SE9485 now had 4 drives. My parity speeds jumped from ~40MB/s to ~70MB/s. I didn't reboot to make the change. All drives were still connected to the 2760A. I simply put the drives on different ports, making sure that each 88SE9485 chip didn't have more than 6 drives to manage. An earlier test with only 3 drives produced parity speeds well in excess of 110MB/s. PCIe bandwidth is not an issue, as each drive had a full PCIe 2.0 lane available in my tests. I'm running my test bare metal with zero plug-ins. So, my observations are that: The Marvel 88SE9485 controller seems to be sensitive to the number of drives it is managing, slowing down with each new drive added Different unRAID builds have had a very dramatic influence on the speed, so the limiting factor appears to be software based Hopefully these results shed new light on an issue my fellow unRAIDers have been struggling with for over a year. You can see more of my results here: http://lime-technology.com/forum/index.php?topic=27460.msg243897#msg243897 Surprisingly I haven't seen any mention of the mvsas driver, which is used on both of these cards. I can't help but think that different mvsas driver revisions, shipped inside the different Linux kernels Tom has incorporated into the various Beta/RC builds, should be a prime suspect. Does anyone have the changelog for the mvsas driver?

I found this unRAID forum post regarding slow parity checks with the AOC-SAS2LP-MV8. http://lime-technology.com/forum/index.php?topic=25890.0 Using the same controller chip, I think the 2760A may be experiencing identical issues. I noticed parity speeds went from 4MB/s on RC6 to 40MB/s on RC 12a, results not all that different from what some have been experiencing with the AOC-SAS2LP-MV8. Ironically, I purposely wanted to stay away from the AOC-SAS2LP-MV8 because I had often read of these issues, and now it seems that's exactly what I bought with the 2760A. I think the 2760A has allowed me to perform one test that no one else has performed, redistributing the drives across the three controllers on the 2760A. The fact that dropping the number of drives per controller chip gave a nice performance boost might be an indication of where the problem lies. This also gives me restrained hope for the future. Assuming the underlying software issue is ever resolved, I think that this could be a very fast controller card. I've been trying to find the mvsas driver revision history/changelog, as this should be a main culprit. I'd like to see what has been changing in this driver over the various Linux kernel builds unRAID has been slogging through. Anyone have a link?

Just for grins, what is the exact mix of drives you have connected right now (that produced the results you just listed) ?? I added the "After" drive distribution to the table above. Looks like I lost track somewhere, I'm up to 9 of the 3TB Reds now. In looking around, it seems that the Marvel 88SE9485 is used on the AOC-SAS2LP-MV8, correct? It seems to me that this is one of the most common controllers used on unRAID builds, so I'm wondering - has anyone noticed a performance hit just from adding more drives?

Wow, this had a dramatic impact! My parity speeds jumped from ~40MB/s to the high 60's! I just saw it peak at 75MB/s, almost double what I've seen before the change. Before After -------------------------------------- Port 1 | 4 drives | 3 drives Port 2 | 4 drives | 3 drives Port 3 | 4 drives | 3 drives Port 4 | 3 drives | 3 drives Port 5 | 0 drives | 3 drives Port 6 | 1 drives | 1 drives Before After --------------------------------------- 9485 1 | 8 drives | 6 drives 9485 2 | 7 drives | 6 drives 9485 3 | 1 drives | 4 drives There is 4GB/s of bandwidth between the PLX PCI bridge chip and each Marvel 88SE9485 controller chip. With 8 drives max on each segment, that is 500 MB/s of bandwidth per drive (aka 1 whole PCIe 2.0 lane). There is no way the PCIe bridge is the bottleneck. Do you think the Marvell 88SE9485 controller chips are being overworked? I need to look up the specs on this chip.

I've been thinking of how I could execute this test without putting my data at risk. I don't have 7 'spare' drives, these are real drives as part of a larger array, and I don't want to throw away my parity to conduct this test. Here's what I'm thinking - pull my current parity drive and put it someplace safe, create a new array definition that only has the Red 3TB drives, and put a spare WD Red 3TB drive in for parity. Run my parity tests. After my tests are done, I can revert back to the original configuration. If a drive fails during my testing, I can reinsert my original parity, revert to the original array definition, and rebuild. I'm thinking I need to back up the array configuration files before I take this path, that way I can simply restore the configuration files and reinsert the parity drive, reboot and I'm right back where I started. Does this sound safe? Another test I'm also thinking of conducting is moving the drives around on the controller. Currently I have 16 data drives and they are all installed on SAS ports 1-4. SAS port 5 is unused, and SAS port 6 has just the parity drive. That means two of the Marvell controllers are fully saturated, while the third controller only has the parity drive on it. I was thinking of moving one drive from each port (1-4) down to ports 5 and 6, making everything distributed more evenly among the Marvell drive controllers. This is such an easy test, I think I'll go ahead and do it now.

Not a stupid question at all. I've never ever installed Simple Features (it's on my list to try someday). I do run unMENU, but I've tested with it disabled with no variance in speed. At this point, I am running a stock 'go' file.