June 4, 201313 yr WD RED + LSI2308 Interesting statistics -- this is perhaps possible for a very short interval when a serendipitous combination of disk buffer -> UnRAID buffers -> parity check logic happens to occur for a few seconds; but clearly it can't be maintained, as Western Digital's own specifications for the drive indicate a "... maximum sustained internal (platter-to-buffer) transfer rate of 145MB/s ..." ==> and that, of course, will only be on the outer cylinders [which is, of course where it's operating with only 1.5% done ]
June 4, 201313 yr yes that was little trick disks was empty and unraid was in maintaince mode but even on 20% still had 260mb/s hihi Now when i have some data, and unraid working in normal mode i get almost full possible speed, something arround 140mb/s almost all time of process parity sync
June 4, 201313 yr Author I have excellent news to report: the Parity Check finished in 10h30m. My last rebuild took 10h45m, so sure enough the md_sync_window settings allowed the Parity Check to run a little bit faster than the rebuild. My next test will be to upgrade a 1.5TB disk to 3TB, and measure how long the rebuild takes, to see if it was affected by the change. I wish I had more answers regarding the correct settings for md_sync_window, for the most part I'm just happy it is working. I don't imagine every server would need the same settings (after all, this is a tunable parameter for a reason). I think that SAS/SATA controller chipset is the main impactor, as I simply migrated my current HD over to my new server build with the 2760A when the problems arose. Things to think about: If the SAS/SATA controller chipset does have an impact, server builds that use multiple different chipsets (i.e. MB SATA ports + add-in card) might not have a 'sweet spot'. The md_num_stripes is also a tunable... what does it do? We do know that md_sync_window can't be set larger than md_num_stripes, but there's a lot more to learn. Same for the md_write_limit. I wonder if this was controlling rebuild speed... Why was the md_sync_window value so low to start with? Is there a danger in having a higher default value? The defaults work fine for many controller cards (like my Adaptec 1430SA) and MB ports. I did see user jbartlett mention he encountered CPU stalls (whatever those are) with higher values. I wouldn't change any values without beforehand knowledge or guidance, either from Tom or an experienced user. I do recall a lot of posts regarding the three md_ settings in the past on this forum. These were values that expert users wanted, and Tom opened them up. Some savvy searching and you might find the answers to all our questions already hiding in the forum somewhere. I'm also somewhat surprised that such a simple solution to the SAS2LP/88SE9485 performance issues was not unearthed before now. Regardless, I'm pleased that other users are seeing positive results. -Paul
June 4, 201313 yr Excellent news. Seems you've not only resolved the issue for your controller, but have solved the elusive SAS2LP-MV8 issue that's been ongoing for a long time !! Like you, I'm reluctant to start experimenting without some assurance that wrong values won't "hurt" anything ==> running slower is okay; losing data is not !! I'm going to search the forums a bit to see if I can find the discussions that led to the parameters being "opened up".
June 4, 201313 yr Author I found this related to the three md_ parameters: http://lime-technology.com/forum/index.php?topic=15224.msg142767#msg142767 In summary, in seems that increasing all three values is a good thing, as long as you have enough memory. The default settings were optimized for servers with 512MB ram, which makes sense because stock, I typically see about 400mb used for parity checks. With 4GB, I have a bit of spare room to play. A couple notes: md_sync_window + md_write_limit should be < md_num_stripes. Currently I'm not configured correctly, as my md_sync_windows (896) + md_write_limit (768) = 1664, which is 384 more than my md_num_stripes(1280). Some people are blindly increasing these numbers to the theoretical maximum (i.e. I have 4GB RAM, which is 8x more than the default server size of 512MB, so simply increase all numbers x8). This is a very bad idea. Here's why: Certain tasks on a server will consume much more memory, beyond what is used for parity checks/rebuilds. A good example is the linux mv command (move). Sometimes I like to telnet onto the box and run a mv to move folders from one physical drive to another. I notice my memory utilization skyrockets from 400 MB to pretty much 4GB, 100% utilization. I've even seen my server come to a crawl when doing mv's at the same time I was pre-clearing a couple drives. True, a cp (copy) followed by a rm (delete) is more memory efficient, so I do that now, but the point is, you need to leave lots of memory for other processes (some Linux, some unMENU, some Simple-Features, some other plugins). Do not blindly try to increase these values to use up all memory, as you will probably only crash your server. After my rebuild is done (running great by the way), I plan to double md_num_stripes to 2560, and increase md_write_limit to 1024, and increase md_sync_window to 1280. Then I'll do some parity speed tests again. Now, more than ever, I see a strong need to go 64-bit for more memory. Perhaps 5.1....
June 4, 201313 yr I found this related to the three md_ parameters: http://lime-technology.com/forum/index.php?topic=15224.msg142767#msg142767 In summary, in seems that increasing all three values is a good thing, as long as you have enough memory. The default settings were optimized for servers with 512MB ram, which makes sense because stock, I typically see about 400mb used for parity checks. With 4GB, I have a bit of spare room to play. A couple notes: md_sync_window + md_write_limit should be < md_num_stripes. Currently I'm not configured correctly, as my md_sync_windows (896) + md_write_limit (768) = 1664, which is 384 more than my md_num_stripes(1280). Some people are blindly increasing these numbers to the theoretical maximum (i.e. I have 4GB RAM, which is 8x more than the default server size of 512MB, so simply increase all numbers x8). This is a very bad idea. Here's why: Certain tasks on a server will consume much more memory, beyond what is used for parity checks/rebuilds. A good example is the linux mv command (move). Sometimes I like to telnet onto the box and run a mv to move folders from one physical drive to another. I notice my memory utilization skyrockets from 400 MB to pretty much 4GB, 100% utilization. I've even seen my server come to a crawl when doing mv's at the same time I was pre-clearing a couple drives. True, a cp (copy) followed by a rm (delete) is more memory efficient, so I do that now, but the point is, you need to leave lots of memory for other processes (some Linux, some unMENU, some Simple-Features, some other plugins). Do not blindly try to increase these values to use up all memory, as you will probably only crash your server. After my rebuild is done (running great by the way), I plan to double md_num_stripes to 2560, and increase md_write_limit to 1024, and increase md_sync_window to 1280. Then I'll do some parity speed tests again. Now, more than ever, I see a strong need to go 64-bit for more memory. Perhaps 5.1.... I had a problem years ago when these parameters were exposed and I increased them. I believe the issue I had is because these buffers are in low memory. This is where the kernel stores allot of other critical structures. Even though I had 8GB of ram, if I ran an rsync over a very large directory I would crash with an OOM situation. Granted I increased the parameters by x3 or x4, but I also had 8GB. My filesystem on 1 drive had over 100,000 small files. Therefore keep in mind the potential low memory issues that could occur when increasing these values too much.
June 4, 201313 yr I found this related to the three md_ parameters: http://lime-technology.com/forum/index.php?topic=15224.msg142767#msg142767 In summary, in seems that increasing all three values is a good thing, as long as you have enough memory. The default settings were optimized for servers with 512MB ram, which makes sense because stock, I typically see about 400mb used for parity checks. With 4GB, I have a bit of spare room to play. A couple notes: md_sync_window + md_write_limit should be < md_num_stripes. Currently I'm not configured correctly, as my md_sync_windows (896) + md_write_limit (768) = 1664, which is 384 more than my md_num_stripes(1280). Some people are blindly increasing these numbers to the theoretical maximum (i.e. I have 4GB RAM, which is 8x more than the default server size of 512MB, so simply increase all numbers x8). This is a very bad idea. Here's why: Certain tasks on a server will consume much more memory, beyond what is used for parity checks/rebuilds. A good example is the linux mv command (move). Sometimes I like to telnet onto the box and run a mv to move folders from one physical drive to another. I notice my memory utilization skyrockets from 400 MB to pretty much 4GB, 100% utilization. I've even seen my server come to a crawl when doing mv's at the same time I was pre-clearing a couple drives. True, a cp (copy) followed by a rm (delete) is more memory efficient, so I do that now, but the point is, you need to leave lots of memory for other processes (some Linux, some unMENU, some Simple-Features, some other plugins). Do not blindly try to increase these values to use up all memory, as you will probably only crash your server. After my rebuild is done (running great by the way), I plan to double md_num_stripes to 2560, and increase md_write_limit to 1024, and increase md_sync_window to 1280. Then I'll do some parity speed tests again. Now, more than ever, I see a strong need to go 64-bit for more memory. Perhaps 5.1.... Before you do that (doubling, etc...), keep it at your sweet setting (896) that you have found with the array mounted and running (your normal routine), and try to preclear a spare drive lying around.
June 4, 201313 yr Author I had a problem years ago when these parameters were exposed and I increased them. I believe the issue I had is because these buffers are in low memory. This is where the kernel stores allot of other critical structures. Very good point, thanks WeeboTech! I hadn't been paying attention to LowMem. Here's my current memory stats: from /usr/bin/free -l) total used free shared buffers cached Mem: 4044604 470304 3574300 0 50760 248420 Low: 868268 195024 673244 High: 3176336 275280 2901056 -/+ buffers/cache: 171124 3873480 Swap: 0 0 0 So I have 848 MB of LowMem on RC12a, of which 190 MB are used and 658 are free. This is while a rebuild is running. I will check again both with both stock values and my target values to see how it affects LowMem. Any recommendations on how much LowMem I need to keep free? -Paul
June 4, 201313 yr Author Before you do that (doubling, etc...), keep it at your sweet setting (896) that you have found with the array mounted and running (your normal routine), and try to preclear a spare drive lying around. Currently my sweet setting of 896 is too high, as md_sync_window + md_write_limit are higher than md_num_stripes. I need to fix this before I do anything else - I'm just waiting for my data rebuild to finish, and I'm not touching anything else while it is running. I'm also thinking my sweet spot could be a touch higher after correctly adjusting the md_num_stripes parameter. I like the idea of testing a preclear on a spare drive while running a Parity Check, but I might as well try it with the doubled numbers. I would only test with a NoCorrect Parity Check, though, to be safe. And if that works, I might try a NoCorrect Parity Check + multi-drive Preclear + folder move using 'mv', simultaneously! This will eat memory like a banshee. Might as well see what breaks, I'll just have to use replaceable data.
June 4, 201313 yr Thats was not my goal for you LOL. Ok, when you get it to your sweet spot (or with 896 for the time being) and no parity checks running just your normal routine, kick off a preclear. I ask because whenever I attempted to play with these settings, no matter what gains I made, I could never preclear after changing the defaults crashes... (sometimes it where oops errors. Sometimes emhttp crashes, other time all seemed well, until you tried to stop and/or reboot the server, crashed) I guess one could revert the settings back to defaults on the rare occasions you need to preclear a drive or two. Or a cheesy second system lying around just to preclear the drives. thereby keeping the gains. You test properly, so I wanted to see if you end up with the same behaviour. No rush whenever you get the chance.
June 4, 201313 yr Author I ask because whenever I attempted to play with these settings, no matter what gains I made, I could never preclear after changing the defaults crashes... (sometimes it where oops errors. Sometimes emhttp crashes, other time all seemed well, until you tried to stop and/or reboot the server, crashed) Yikes! Good to know, thanks. I'll still fix my md_ settings first, but I'll take testing a bit slower. Did you ever check your LowMem when running those preclears? Were the crashes consistent and timely (i.e. always in the first 5 minutes, or always in the last 10%, or always during the post read)? Or completely random in a way that I'll have to let the whole thing run and then try to stop/reboot to test the whole cycle?
June 4, 201313 yr If it crashes while you do a preclear, It will crash on you at some unexpected point. So raise your values slowly. In my particular situation, all I had to do was a very large rsync or a find down a disk that had tons of files. if I happened do that them in parallel, I would surely start getting OOM errors. I think I eventually resorted to dropping the cache before my rsync backup jobs, then dropping the cache after. It's hard to predict what your system needs for low memory since each system has a variable number of disks, variable number of files and the speed at which you can write to one disk comes into play when trying to flush the buffers. This is one of the reasons I tried to lobby for root ram disk being on a tmpfs rather then rootfs. tmpfs can be swapped out when needed. rootfs can never be swapped. Running the cool dir_cache tool holds "low" ram in the dentry table. Keep in mind any plugins you load, use up ram being installed on rootfs and then application ram when running. Some packages which are installed can put other files such as man,include,etc... which take away ram by being stored on rootfs.
June 4, 201313 yr I ask because whenever I attempted to play with these settings, no matter what gains I made, I could never preclear after changing the defaults crashes... (sometimes it where oops errors. Sometimes emhttp crashes, other time all seemed well, until you tried to stop and/or reboot the server, crashed) Yikes! Good to know, thanks. I'll still fix my md_ settings first, but I'll take testing a bit slower. Did you ever check your LowMem when running those preclears? Were the crashes consistent and timely (i.e. always in the first 5 minutes, or always in the last 10%, or always during the post read)? Or completely random in a way that I'll have to let the whole thing run and then try to stop/reboot to test the whole cycle? I very much do believe that it's do to lowmem as no matter how much physical ram I added It never helped. The crashes were not consistent (between different settings of these values), I had (actually always have) a putty window to the unRAID server with a tail command running at all times. I remember some oops and emhttp crashing very early on, say within 10-20 minutes of a preclear. With other values, all was well (I was not doing anything). But after a preclear finished I decided to stop the array and reboot, and you could see the oops and other crashes upon shutdown. Repeated the exact same under the default settings no issues at all. Let us know your findings. What's a bit frustrating is no one really wants to jump to these hurdles of per system tweaks and ram disks, etc. at least not manually. I guess if it were a script or add-on by a knowledgeable linux person that could figure things out for you (by running some tests maybe) and it set's it all up for you, it would be a nice to have. I guess... even if you would have to re-run it once in awhile as your system changes (say disk controller(s), additional drive(s), additional plug-in(s)). I guess what I am saying is its ok if our system were 10-20 mb different from one another but not to the effect we are seeing for some, I think that it is a bug, just saying.
June 4, 201313 yr Thanks for finding the details Paul ... you save me a bit of search time this evening WeeboTech ==> When you used values that caused crashes, did that have any impact on data integrity? Just curious if "playing" with the settings is reasonably "data safe".
June 4, 201313 yr My settings on both servers are: Tunable (md_num_stripes): 2048 Tunable (md_write_limit): 768 Tunable (md_sync_window): 1024 Which leaves 256 for reads if I start writing something during a parity check. I had to back down sync_window to 1024, since in the past I would crash setting num_stripes too high. No crashing yet. I generally preclear 3-4 drives at the same time, but I don't have any spares at the moment. I'd also like to add i'm not so sure about num_stripes needing to be higher than the others combined. With 1280 sync_window, and only 1024 num_stripes, I am able to parity check while transferring to and from server with no problems. This shouldn't be possible if that's how it worked?
June 4, 201313 yr Author I'd also like to add i'm not so sure about num_stripes needing to be higher than the others combined. With 1280 sync_window, and only 1024 num_stripes, I am able to parity check while transferring to and from server with no problems. This shouldn't be possible if that's how it worked? Thanks for sharing your info tyrindor. From what I understand, it would still be possible, it is just that the requests get queued up, which could lead to delays. But I see your point.
June 5, 201313 yr Author My drive upgrade/rebuild completed in 10h45m, same as my last one, so rebuild performance was completely unaffected by the change to md_sync_window. This is a perfect result. Sure, it would have been amazing if it somehow magically went faster, but this is the limit of my hard drives. The Results: Disk Rebuild: 10h45m - Unchanged Parity Check: 14h17m with md_sync_window set to 384 (unRAID default) Parity Check: 10h30m with md_sync_window set to 896 At this point, I think I can safely agree that the md_sync_window parameter is the "slow parity check" fix for Marvel 88SE9485 based controllers, like the HighPoint 2760A and the AOC-SAS2LP-MV8, and may also have good results for other controllers. Big THANKS to Tom#1 for pointing us in the right direction! Over the next few days I will perform additional testing, evaluating changes to md_num_stripes, md_write_limit and md_sync_window. I will be looking for out of memory (OOM) issues and Kernel OOPS that other users have reported when tweaking these settings, and any other issues that may crop up. I will actively attempt to stress test the server by performing multiple tasks simultaneously. Rather than start of with mild settings, I will go straight to aggressive settings (within reason) and attempt to quickly break the system. If the system breaks, I will back off the settings by 10% and try again. Keep in mind, my results will not be directly applicable to your system (unless you built a clone of my server). That means you have to do your own testing (sorry), but hopefully my test results will give you an idea of values to try, how to test, and what to look for. Oh, and if your system performs fine with the defaults, stop reading and go watch a movie or something. This isn't fun, it's a necessary evil for the group of us afflicted with performance issues. -Paul
June 5, 201313 yr WeeboTech ==> When you used values that caused crashes, did that have any impact on data integrity? Just curious if "playing" with the settings is reasonably "data safe". I did not find corruption. It was mainly the rsyncs or emhttp that was failing with the OOM issue. if the rsync was killed, I had to restart it. same with emhttp or any other app.
June 5, 201313 yr WeeboTech ==> When you used values that caused crashes, did that have any impact on data integrity? Just curious if "playing" with the settings is reasonably "data safe". I did not find corruption. It was mainly the rsyncs or emhttp that was failing with the OOM issue. if the rsync was killed, I had to restart it. same with emhttp or any other app. Thanks -- running a parity check at the moment (seems ~ the same speed as previous ones, so no impact with RC13) ... but when it completes I do plan to experiment a bit to see if I can get the kind of nifty improvements Paul did by tweaking the parameters. I'm not unhappy with my 8 hrs, but of course 6 or 7 would be even better
June 5, 201313 yr I found this related to the three md_ parameters: http://lime-technology.com/forum/index.php?topic=15224.msg142767#msg142767 Very good fine -- this thread does an excellent job of outlining just what the 3 key parameters are. Currently I'm not configured correctly, as my md_sync_windows (896) + md_write_limit (768) = 1664, which is 384 more than my md_num_stripes(1280).[/i] Note that the reason for that constraint is: "... You want to make sure the sum of md_write_limit+md_sync_window < md_num_stripes so that reads do not get starved if you starting writing a large file while a parity-sync/check is in process." If you're like me, you probably don't do ANY writes to the array during a parity check. I don't use the array for anything -- reads or writes -- during parity checks; although I HAVE streamed a movie when I had a parity check in progress and the wife wanted to watch something (I didn't time it, but I'm sure this significantly slowed down the parity check). If you don't do writes during parity checks, then clearly md_sync_window can be much larger, since effectively the "md_write_limit" during that operation is zero. So if you have the same constraint I do, then your 896 value is well below 1280. After my rebuild is done (running great by the way), I plan to double md_num_stripes to 2560, and increase md_write_limit to 1024, and increase md_sync_window to 1280. Then I'll do some parity speed tests again. Certainly won't hurt to experiment, but given what I just noted above, I think your earlier testing showing diminishing gains above 896 will probably still be true. The only unknown is whether or not significantly bumping the total number of stripes allowed (doubling it) will have some impact ... but I doubt it, as you probably didn't have either reads or writes in progress when you tested earlier (is that true??) ... so I think the md_sync_window was the only operable constraint => and you've already shown that advancing it beyond 896 didn't help. One unknown is what might happen if you try some notably higher md_sync_window settings (i.e. something like 1536) ... which you could do with total stripes at 2560. I'm looking forward to testing some higher numbers on my Atom board in the next day or so
June 5, 201313 yr I HAVE streamed a movie when I had a parity check in progress and the wife wanted to watch something (I didn't time it, but I'm sure this significantly slowed down the parity check). Sometimes what I do is read the whole file once via application i.e. cp file /dev/null. This puts it in the buffer cache so subsequent reads are from ram. Food for thought next time you want to stream a movie while doing a long parity operation.
June 5, 201313 yr I HAVE streamed a movie when I had a parity check in progress and the wife wanted to watch something (I didn't time it, but I'm sure this significantly slowed down the parity check). Sometimes what I do is read the whole file once via application i.e. cp file /dev/null. This puts it in the buffer cache so subsequent reads are from ram. Food for thought next time you want to stream a movie while doing a long parity operation. That works for a 45GB BluRay MKV?
June 5, 201313 yr I HAVE streamed a movie when I had a parity check in progress and the wife wanted to watch something (I didn't time it, but I'm sure this significantly slowed down the parity check). Sometimes what I do is read the whole file once via application i.e. cp file /dev/null. This puts it in the buffer cache so subsequent reads are from ram. Food for thought next time you want to stream a movie while doing a long parity operation. That works for a 45GB BluRay MKV? Oh Jeeze, probably not. I forget that most people watch huge blueray movies while I watch compressed DVD's and downloaded compressed bluerays. It's what ever fits in ram. I suppose if I were going to watch something that large I would copy it to the cache drive somewhere.
June 5, 201313 yr Oh Jeeze, probably not. I forget that most people watch huge blueray movies while I watch compressed DVD's and downloaded compressed bluerays. It's what ever fits in ram. I suppose if I were going to watch something that large I would copy it to the cache drive somewhere. Thanks. I thought you knew something I didn't since I'm Linux illiterate. And technically I watch mostly recorded HD programming. But 2 hour movies off my HD-PVR are ~11GB in size so would still apply for me.
Archived
This topic is now archived and is closed to further replies.