garycase Posted June 6, 2013 Share Posted June 6, 2013 I feel the same way about RC13, it's a pass for me as well. I don't think Paul's skipping RC13 due to any particular issue ... but (apparently) just to be consistent with his testing, so it's all done on the same baseline. FWIW I've been doing mine (which I just started yesterday) all on RC13 ... but had I started on RC12a I'd have stayed with it. From what I've read on the RC13 feedback section, however, there are definitely a few issues that would lead me to suggest that if you're stable on the version you're running now, just stay with it !! Link to comment
tyrindor Posted June 6, 2013 Share Posted June 6, 2013 With these commands making such a difference, I hope we can get some kind of database on the wiki or something for recommended (or at least ballpark) settings for certain builds. I'm still using this on both of my 3x SAS2LP servers: Tunable (md_num_stripes): 2048 Tunable (md_write_limit): 768 Tunable (md_sync_window): 1024 No crashing yet, and my parity checks are completed in about 9 hours and 11 hours (down from 18 and 21 hours on defaults). Needless to say, with results this massive, tweaking these settings should be something every unRAID users does (Especially SAS2LP owners)... or something unRAID does automatically based on your hardware config. I'd also like to add that I adjusted these values back on an older RC and it did not help. I'm not sure if RC13's kernel update is required to get these variables to work with SAS2LP drivers or not, but if you do adjust them and it doesn't seem to make a difference, maybe try updating to RC13. Link to comment
WeeboTech Posted June 6, 2013 Share Posted June 6, 2013 Check your memory with free and free -l Low memory exhaustion can cause OOM issues. Link to comment
Pauven Posted June 6, 2013 Author Share Posted June 6, 2013 I'd also like to add that I adjusted these values back on an older RC and it did not help. I'm not sure if RC13's kernel update is required to get these variables to work with SAS2LP drivers or not, but if you do adjust them and it doesn't seem to make a difference, maybe try updating to RC13. What older versions did you try? I'm using the parameters with RC12a to great result. Link to comment
Pauven Posted June 6, 2013 Author Share Posted June 6, 2013 Check your memory with free and free -l Low memory exhaustion can cause OOM issues. I'm doing exactly that. I'm testing using settings md_num_stripes = 2560, md_write_limit = 1024, and md_sync_window = 1024. Running a NoCorrect Parity Check: total used free shared buffers cached Mem: 4044604 532032 3512572 0 4876 247580 Low: 868268 253280 614988 High: 3176336 278752 2897584 -/+ buffers/cache: 279576 3765028 Swap: 0 0 0 Running a NoCorrect Parity Check & 2 Pre-Clears: total used free shared buffers cached Mem: 4044604 893756 3150848 0 357260 247528 Low: 868268 610788 257480 High: 3176336 282968 2893368 -/+ buffers/cache: 288968 3755636 Swap: 0 0 0 Running a NoCorrect Parity Check & 3 Pre-Clears: total used free shared buffers cached Mem: 4044604 1081324 2963280 0 539984 247544 Low: 868268 796372 71896 High: 3176336 284952 2891384 -/+ buffers/cache: 293796 3750808 Swap: 0 0 0 Observations: When just the Parity Check was running, the memory stats were fairly stable. I could click refresh once a second, and the values changed very little from refresh to refresh. As soon as I started running a Pre-Clear, the numbers jump wildly up and down. The memory values I'm showing are the highest values I noticed, but may not represent the absolute highest values the server is experiencing. I don't know if this is garbage collection or or paging or caching or inefficient coding. I would say it would be nice if the Pre-Clear script was a little more stingy with memory use, or perhaps had some parameters to keep it in check, like the md_ parameters for the parity check. Maybe Pre-Clear has these settings and I've just never noticed. I feel if I started up a 4th Pre-Clear I would see OOM errors immediately. I'm leaving it at 3 Pre-Clears to see how far they get; the numbers are already dancing dangerously close to peaking out LowMem. Currently these Pre-Clears are in the "Pre-Read" stage, and I anticipate that memory usage will change during the write and post-read stages. I also feel that if I do anything else on the server right now, like playing a movie or copying some files, I would see OOM errors. Had I not increased the md_ parameters, I would have a little more LowMem to play around with, so raising these values is definitely having an impact. Obviously I'm pushing the server hard, running simultaneously process through the front end GUI and on the back end through command lines, doing things unRAID isn't capable of out-of-the-box, so I can't hold Tom responsible if the server craps its pants. I know I said this once before recently, but it bears repeating: I see the need for a 64-bit unRAID build now more than ever. AMD brought 64-bit computing to the masses just over 10-years ago now, and today it is actually rather challenging to buy a non-64-bit x86 processor. As commonplace as x64 is today, and in light of the 32-bit LowMem limitations that seem ridiculously easy to hit, 64-bit is not a nice to have, but a requirement. Link to comment
tyrindor Posted June 6, 2013 Share Posted June 6, 2013 I'd also like to add that I adjusted these values back on an older RC and it did not help. I'm not sure if RC13's kernel update is required to get these variables to work with SAS2LP drivers or not, but if you do adjust them and it doesn't seem to make a difference, maybe try updating to RC13. What older versions did you try? I'm using the parameters with RC12a to great result. Don't remember, but it was before RC12. Check your memory with free and free -l Low memory exhaustion can cause OOM issues. I'm doing exactly that. I'm testing using settings md_num_stripes = 2560, md_write_limit = 1024, and md_sync_window = 1024. Thanks for testing, I don't see myself ever doing 3 preclears during a parity check. I don't even write to the server during parity check, but I will watch a BD ISO or two sometimes (Which only translates to about 3-5MB/s of reads). If you do something like.. 4-5 preclears without anything else going on, what's it look like? I think that's when I would load my server the most. Link to comment
WeeboTech Posted June 6, 2013 Share Posted June 6, 2013 While I agree with the 64 bit comments, unRAID has some issues to take care of now. However at some point it HAS to go 64 bit. Another choice is to put / on a tmpfs rather then rootfs. Then unused parts of the filesystem can be swapped out. I think having preclear use badblocks to do the read/write of 0's to the drive might help alleviate all the buffering that goes on with dd. I thought I saw something in the code that accessed the drive more directly. I'll have to revisit that one day. Where I had issues with the expanded tunings is with large rsyncs or finds down disks with a huge number of small files. Especially on my download and bittorrent drive. a mishandled find or rsync would start to create a chain of OOM errors. Dropping the buffer cache helped. Perhaps doing that in preclear would help too. drop cache before writing, drop cache before reading, etc, etc. Link to comment
Pauven Posted June 6, 2013 Author Share Posted June 6, 2013 Thanks for testing, I don't see myself ever doing 3 preclears during a parity check. If you do something like.. 4-5 preclears without anything else going on, what's it look like? Wow, you must think I have stacks of spare drives sitting around doing nothing [he says staring blankly at large piles of unused, spare drives... ], uhm... yeah I guess I could do that. But not until my current test is done. Link to comment
madburg Posted June 6, 2013 Share Posted June 6, 2013 While I agree with the 64 bit comments, unRAID has some issues to take care of now. However at some point it HAS to go 64 bit. Another choice is to put / on a tmpfs rather then rootfs. Then unused parts of the filesystem can be swapped out. I think having preclear use badblocks to do the read/write of 0's to the drive might help alleviate all the buffering that goes on with dd. I thought I saw something in the code that accessed the drive more directly. I'll have to revisit that one day. Where I had issues with the expanded tunings is with large rsyncs or finds down disks with a huge number of small files. Especially on my download and bittorrent drive. a mishandled find or rsync would start to create a chain of OOM errors. Dropping the buffer cache helped. Perhaps doing that in preclear would help too. drop cache before writing, drop cache before reading, etc, etc. Right, so I should have been more clear on my experiences Pauven. I never ran a parity check AND preclear. I had things like an Plex server scrapping media shares, handbrake server crushing BD's and spitting out mkv's, other servers extracting BD ISO, media clients play various media, SAB 5.1MB DL (never ending) /sickbeard stuff, kicking of mover to free up 100's of GBs to the array nightly. I am sure I forgot some others things going on. So at some point I would need more room, so I would start a preclear while this was all going on, AND if I had tweaked those disk settings, then it was all downhill. To be clear all server/services are external to unraid, not running as plug-ins. unRAID being the data hub. But if I left the defaults you could see some slowness kick-in, but nothing would fail. The latest RC are slower though and can't do as much as I used too. I setup an old rig and do preclear's from it now, and test older drivers, etc... so I may have to revisit this, now that I see how you are checking things; to see what a sweet spot would be with my config. And just know NOT to preclear from it... is an option for me. Thanks again. Link to comment
UhClem Posted June 6, 2013 Share Posted June 6, 2013 ... I think having preclear use badblocks to do the read/write of 0's to the drive might help alleviate all the buffering that goes on with dd. I thought I saw something in the [badblocks] code that accessed the drive more directly. Yeah, it's called Direct I/O. badblocks uses it by default, but it's disabled by -B dd doesn't use it by default, but it's enabled by iflag=direct and/or oflag=direct Preclear could invoke dd using Direct I/O during pre-read & zeroing with no ill-effect (but significant reduction in buffer/memory use). However, it can't do so during the post-read phase because of all it needs to accomplish. But there's little benefit from reducing the memory usage of Preclear during only 2 of its 3 phases; it's really all or nothing, especially since the post-read phase typically takes the longest. None of my comments here should be taken as critical of Preclear. It is a superb tool and an excellent example of shell programming. JoeL deserves your continued appreciation and support for his efforts and contributions. Why don't any of you help--by building upon that excellent foundation, and adding these improvements? Reminds me of that Little Red Hen children's story. [Note: I don't use unRAID myself, so I don't think I'm being a hypocrite here. (I don't want to eat the bread )] For any who missed it, I'd like to repeat a "challenge" I made at the end of this posting[link]: Add 10-20 lines to the preclear script that will cause the post-read phase to run just as fast as the pre-read phase for many/most users. For the rest, no change (but there would be such clamoring that they could easily/quickly join the party). [Same exact functionality/results as now.] Think about it-- several hours saved per Preclear cycle for a 2TB drive. Note: if you combine the correct solution to the "challenge", with the simple addition of having dd use Direct I/O during the pre-read and zeroing phases, you get both better performance and a significant reduction in memory usage. --UhClem Link to comment
UhClem Posted June 6, 2013 Share Posted June 6, 2013 My X7SPA does a parity check in 8:07 with six 3TB WD Reds ... and I suspect would do it in ~ the same time with 14 of them if I added an 8-port controller. Like I just said, your board makes an excellent 6-drive energy efficient server, But scale it up to 16 drives and I think it will start to choke. 16?? Hah!! You're far too kind. Try 11 or 12 !! But, I commend you. You do appreciate the basics of performance (measurement/analysis), and seem to take the right approach to learning, and understanding, the concepts behind it. (Certain others will hopefully follow your example.) --UhClem Link to comment
garycase Posted June 6, 2013 Share Posted June 6, 2013 My X7SPA does a parity check in 8:07 with six 3TB WD Reds ... and I suspect would do it in ~ the same time with 14 of them if I added an 8-port controller. Like I just said, your board makes an excellent 6-drive energy efficient server, But scale it up to 16 drives and I think it will start to choke. 16?? Hah!! You're far too kind. Try 11 or 12 !! But, I commend you. You do appreciate the basics of performance (measurement/analysis), and seem to take the right approach to learning, and understanding, the concepts behind it. (Certain others will hopefully follow your example.) --UhClem I don't think 11 or 12 is an issue at all. As I noted in an earlier post, I'm confident it would perform very sell with a SASLP-MV8 and 8 more drives ... total of 14. Beyond that, there would clearly be bandwidth issues. Note that this is the exact configuration used in Limetech's new Atom-based server Incidentally, the 8:07 parity check time I noted in my earlier post is now down to 7:33 after adjusting the tunable disk parameters. I'm still experimenting a bit, but that seems to be the best it's going to be [md_sync_window at 1024; stripes limit 1920; write limit 768 if you're interested] Without actually buying a SASLP-MV8 and 8 more 3TB WD Reds I can't confirm what that would do to the timings; but based on the bandwidth of the x4 slot; the bandwidth of the drives; and the simple fact that parity checks aren't particularly CPU-intensive, I wouldn't expect the time to be much greater than what I'm seeing now. It'll be interesting to get some performance feedback from early users of Tom's new server who have fully populated it with 8 3TB Reds and 4 other drives in the 2.5" slots. Link to comment
UhClem Posted June 7, 2013 Share Posted June 7, 2013 I don't think 11 or 12 is an issue at all. As I noted in an earlier post, I'm confident it would perform very sell with a SASLP-MV8 and 8 more drives ... total of 14. Your problem, regarding disk subsystem performance, is that you don't know how little you know. For example, you haven't even progressed past the arithmetic-based-on-([raw] PCIe)spec stage. Note that this is the exact configuration used in Limetech's new Atom-based server New? Feh! Based on a 5+ years old mobo chipset and a 4+ years old controller chip. They just barely kept up with disk drives of that era. Like I said, you don't know ... Link to comment
garycase Posted June 7, 2013 Share Posted June 7, 2013 I don't think 11 or 12 is an issue at all. As I noted in an earlier post, I'm confident it would perform very sell with a SASLP-MV8 and 8 more drives ... total of 14. Your problem, regarding disk subsystem performance, is that you don't know how little you know. For example, you haven't even progressed past the arithmetic-based-on-([raw] PCIe)spec stage. Note that this is the exact configuration used in Limetech's new Atom-based server New? Feh! Based on a 5+ years old mobo chipset and a 4+ years old controller chip. They just barely kept up with disk drives of that era. Like I said, you don't know ... Well, when I get a bit past the novice stage maybe I'll learn a bit more I've only been doing this stuff for ~ 50 years or so I didn't mean to imply Limetech's "new Atom-based server" used new technology; it's simply a new server in Limetech's offerings. But nevertheless it's an excellent system which uses TRIED and TRUE technology that runs very well. Parity checks averaging over 110MB/s are certainly good enough for me Link to comment
Pauven Posted June 7, 2013 Author Share Posted June 7, 2013 It looks like everything completed perfectly in my latest test, though there was one anomaly: the system log restarted at 4:40 this morning. Because I don't write out the log to a file, or tail it in a console, I have no idea what was in the log before 4:40am. As a reminder, I tested using settings md_num_stripes = 2560, md_write_limit = 1024, and md_sync_window = 1024. The Parity Check completed in ~10h27m (since I lost the start time in the log, I'm approximating here) This is a few minutes faster than my last Parity Check, which clocked in a 10h30m using md_sync_window of 896 and the other values stock. During the Parity Check, I also ran three simultaneous 1 pass Pre-Clears on Samsung F2 1.5TB drives. These also completed without error, and in excellent time (for a Pre-Clear). As far as I could tell, they ran at full speed of the HD. That means the server was accessing 19 drives simultaneously at full speed (1 Parity, 15 Data, 3 Pre-Clear), and all jobs/tasks completed. I'm extremely pleased. I saw extremely minor benefit to the higher md_ values, so I may back them down a bit for some server headroom. I see absolutely no reason to explore even higher values at this time. I will revisit this in the future as I expand from 16 drives to 24, as I expect that the number of drives is directly related to how much memory needs to be allocated to the process. I also was able to watch a Blu-Ray ISO off the server last night, after the Parity Check but during the Zeroing/Post-Read phase of the 3 Pre-Clears. I monitored for OOM & OOPS and saw none during the time I monitored, but there is a 6-hour stretch of time in the logs that I lost so I can't say for sure. In fact, the restart of the syslogd might be indicative of an OOPS/OOM/Something. I monitored memory use up to the Zeroing phase of the Pre-Clear, and LowMem was obviously taxed: total used free shared buffers cached Mem: 4044604 1148944 2895660 0 599204 249608 Low: 868268 859908 8360 High: 3176336 289036 2887300 -/+ buffers/cache: 300132 3744472 Swap: 0 0 0 As you can see, LowMem is basically all used up. I never saw any values below 8192, but it was hovering just above that area, and I think memory was being paged to HighMem (don't know how to tell for sure) based upon the behavior I saw. I still had plenty of HighMem available. This morning, after all processing was complete, the LowMem had a good amount freed up, but the HighMem was fairly exhausted: total used free shared buffers cached Mem: 4044604 3384584 660020 0 20900 3072484 Low: 868268 283004 585264 High: 3176336 3101580 74756 -/+ buffers/cache: 291200 3753404 Swap: 0 0 0 A quick Stop/Start of the array (no reboot) freed up the memory: total used free shared buffers cached Mem: 4044604 532120 3512484 0 4872 249028 Low: 868268 254516 613752 High: 3176336 277604 2898732 -/+ buffers/cache: 278220 3766384 Swap: 0 0 0 I wish I had mem stats from the Post-Read phase of the Pre-Clear, as that is probably the most taxing phase of the Pre-Clear script. The Parity Check had already completed before the Post-Read, though, so any issues would have been purely on the Pre-Clear script. I said in an earlier post that perhaps Pre-Clear has some parameters to control memory usage. I checked out Joe L.'s excellent documentation, and sure enough there are some parameters to reduce the memory footprint. Joe cautions that limiting memory will increase run-time, and for those that need it this seems like an acceptable compromise. After giving it careful thought, I think my testing is wrapped up for now. I know other users have experienced OOM issues with modified md_ settings, but they are doing things that I don't do, would never have a reason to do, and don't have the data to recreate. For example, WeeboTech is doing large rsyncs and finds down drives with large numbers of small files (I don't have this condition to recreate the test). And madburg is doing this: Right, so I should have been more clear on my experiences Pauven. I never ran a parity check AND preclear. I had things like an Plex server scrapping media shares, handbrake server crushing BD's and spitting out mkv's, other servers extracting BD ISO, media clients play various media, SAB 5.1MB DL (never ending) /sickbeard stuff, kicking of mover to free up 100's of GBs to the array nightly. I am sure I forgot some others things going on. So at some point I would need more room, so I would start a preclear while this was all going on, AND if I had tweaked those disk settings, then it was all downhill. To be clear all server/services are external to unraid, not running as plug-ins. unRAID being the data hub. But if I left the defaults you could see some slowness kick-in, but nothing would fail. The latest RC are slower though and can't do as much as I used too. madburg, you're not running just an unRAID server, you're running something like 6 servers on shared hardware (and you're not doing it through a VM either, from what I can tell). I admire your ambition, and the fact that this works at all is remarkable, but this configuration just isn't supportable. I would say that unRAID and Linux are both doing what they were designed to do, and even as LowMem was depleted by the Pre-Clear script, Linux managed the memory effectively. At this point, I'm returning back to the original topic: an "Ultra Low Power 24-Bay Server". I consider the matter of slow parity checks with Marvel 88SE9485 controllers resolved to my satisfaction (and I'm glad others have benefited as well). I think any further discussion on that topic should be moved to a dedicated thread, as it is getting lost in this mammoth thread and is technically off-topic here. -Paul Link to comment
Pauven Posted June 7, 2013 Author Share Posted June 7, 2013 For any who missed it, I'd like to repeat a "challenge" I made at the end of this posting[link]: Add 10-20 lines to the preclear script that will cause the post-read phase to run just as fast as the pre-read phase for many/most users. For the rest, no change (but there would be such clamoring that they could easily/quickly join the party). [Same exact functionality/results as now.] Think about it-- several hours saved per Preclear cycle for a 2TB drive. Note: if you combine the correct solution to the "challenge", with the simple addition of having dd use Direct I/O during the pre-read and zeroing phases, you get both better performance and a significant reduction in memory usage. --UhClem I wish I had any inkling of what any of that meant, because I'm the type of guy who can't back down from a challenge. Since you know, UhClem, I'm surprised you don't take your own challenge. Think about it-- everyone would benefit! -Paul Link to comment
garycase Posted June 7, 2013 Share Posted June 7, 2013 For any who missed it, I'd like to repeat a "challenge" I made at the end of this posting[link]: Add 10-20 lines to the preclear script that will cause the post-read phase to run just as fast as the pre-read phase for many/most users. For the rest, no change (but there would be such clamoring that they could easily/quickly join the party). [Same exact functionality/results as now.] Think about it-- several hours saved per Preclear cycle for a 2TB drive. Note: if you combine the correct solution to the "challenge", with the simple addition of having dd use Direct I/O during the pre-read and zeroing phases, you get both better performance and a significant reduction in memory usage. --UhClem I wish I had any inkling of what any of that meant, because I'm the type of guy who can't back down from a challenge. Since you know, UhClem, I'm surprised you don't take your own challenge. Think about it-- everyone would benefit! -Paul I'd put this in the "so what" category => the time it takes to do a pre-clear is very much irrelevant. Running a pre-clear has virtually no impact on the array (unless you're running so many that the memory impact is an issue), so the time it takes simply doesn't seem to be an issue. I believe Joe L intentionally does a bunch of additional seeking during the post-read phase to intentionally "work out" the drive ... these seeks are obviously what causes the notably longer timing, but are also adding to the "confidence level" you can have in the drive. To quote from Joe L's description: "... I purposely keep the disk head moving a LOT more than if just reading each block in turn. This is to identify hardware that is marginal. " Bottom line: Certainly not worth the effort to speed it up. Link to comment
madburg Posted June 7, 2013 Share Posted June 7, 2013 Your problem, regarding disk subsystem performance, is that you don't know how little you know. Well, when I get a bit past the novice stage maybe I'll learn a bit more I've only been doing this stuff for ~ 50 years or so I remember as a teen working for a construction company to make some extra cash. Two guys got heated. The sheet rocker, said "Dude, I've been framing and rocking for 20 years! don't tell me..." The foremen said "Just because you have been doing something for a long time, doesn't mean you have been doing it right!" there is lot of truth in that statement... And I also learned when people are quick to start off with how long they have been doing something, versus factual information that proves them right, etc.. it's usually not a good sign. Sorry for polluting the thread Pauven. I was just laughing reading those posts and had to drop this one. Link to comment
garycase Posted June 7, 2013 Share Posted June 7, 2013 I saw extremely minor benefit to the higher md_ values, so I may back them down a bit for some server headroom. I see absolutely no reason to explore even higher values at this time. I've seen the same thing. I think 896 or 1024 is the optimal point for the md_sync_window parameter. I've done a LOT of parity checks on both my v5 system and my v4.7 system the past few days, and have found that dramatically raising the stripes limit and sync_window beyond those values has virtually no benefit ... but the 896 range definitely helped a lot. I've reduced my v5 parity checks to 7:33 (a bit over a half-hour improvement); and my v4.7 system (14 drives of mixed sizes) dropped by ~ 45 minutes (to 8:30). I've got both systems set at 1920/768/1024 and will probably just leave them there [although I plan to do one final test at 1792/768/896]. After giving it careful thought, I think my testing is wrapped up for now. I know other users have experienced OOM issues with modified md_ settings, but they are doing things that I don't do, would never have a reason to do, and don't have the data to recreate. Agree ... with regard to both the amount of testing you've done and the fact that memory issues are simply not a problem if you're not doing other things during the parity tests [i.e. not "... doing things that I don't do, would never have a reason to do, and don't have the data to recreate .."] When I run a parity check, I don't do anything else, so I'm not at all concerned about the ancillary activities having any impact. At this point, I'm returning back to the original topic: an "Ultra Low Power 24-Bay Server". Excellent idea to re-focus on the original topic !! I consider the matter of slow parity checks with Marvel 88SE9485 controllers resolved to my satisfaction (and I'm glad others have benefited as well). Completely agree ... and you've also as a major side-benefit resolved the long-standing issue folks have been complaining about r.e. performance with the SAS2LP-MV8's Link to comment
garycase Posted June 7, 2013 Share Posted June 7, 2013 Your problem, regarding disk subsystem performance, is that you don't know how little you know. Well, when I get a bit past the novice stage maybe I'll learn a bit more I've only been doing this stuff for ~ 50 years or so I remember as a teen working for a construction company to make some extra cash. Two guys got heated. The sheet rocker, said "Dude, I've been framing and rocking for 20 years! don't tell me..." The foremen said "Just because you have been doing something for a long time, doesn't mean you have been doing it right!" there is lot of truth in that statement... And I also learned when people are quick to start off with how long they have been doing something, versus factual information that proves them right, etc.. it's usually not a good sign. Sorry for polluting the thread Pauven. I was just laughing reading those posts and had to drop this one. I agree -- I shouldn't have bitten at the insult I completely agree that longevity doesn't always mean much ... in fact it can mean complacency. Ditto academic credentials, etc. I recall a few years ago when I asked an installer to test a mod he'd done to my wife's car and he was "insulted" that I'd question whether it was done correctly and that it needed to be tested ... but when he actually (reluctantly) tested it, it didn't work !! He blamed the part (a spoiler), went to the trouble of removing it and installing another one; and when that also didn't work he THEN checked his wiring ... which he'd gotten wrong. Link to comment
madburg Posted June 7, 2013 Share Posted June 7, 2013 It looks like everything completed perfectly in my latest test, though there was one anomaly: the system log restarted at 4:40 this morning. Because I don't write out the log to a file, or tail it in a console, I have no idea what was in the log before 4:40am. -Paul That's awesome, I have learned a lot from your tests (and how to go about it). So I will find and set aside some time in the near future to give this a go again. Thanks man! madburg, you're not running just an unRAID server, you're running something like 6 servers on shared hardware (and you're not doing it through a VM either, from what I can tell). I admire your ambition, and the fact that this works at all is remarkable, but this configuration just isn't supportable. P.S. it seems like a lot what I was doing, but it really isn't. We run (at work) massive esx clusters with larger amounts of vm's, databases, transactions, bandwidth, etc.. that far exceeds what I am doing with my 2 node vsphere cluster at home. I have way more than 6 virtual servers, but only a select few work off the data housed on unRAID. unRAID owns the controllers and HD that are allotted to it, no sharing there. It's vmdk and private mount (another vmdk) are stored on a RAID 10 esx datastore, no performance issues there either. All the connection to unRAID also work of the cache drive not the array & am most of the connections are from the same node so no traffic transverses a physical nic for that matter, so its not really what is seems. I know once I can get (for $$ not $$$$$$) a 2TB SSD or (4) 512 SSD in a future cache pool feature. Sparks will fly Link to comment
madburg Posted June 7, 2013 Share Posted June 7, 2013 Your problem, regarding disk subsystem performance, is that you don't know how little you know. Well, when I get a bit past the novice stage maybe I'll learn a bit more I've only been doing this stuff for ~ 50 years or so I remember as a teen working for a construction company to make some extra cash. Two guys got heated. The sheet rocker, said "Dude, I've been framing and rocking for 20 years! don't tell me..." The foremen said "Just because you have been doing something for a long time, doesn't mean you have been doing it right!" there is lot of truth in that statement... And I also learned when people are quick to start off with how long they have been doing something, versus factual information that proves them right, etc.. it's usually not a good sign. Sorry for polluting the thread Pauven. I was just laughing reading those posts and had to drop this one. I agree -- I shouldn't have bitten at the insult I completely agree that longevity doesn't always mean much ... in fact it can mean complacency. Ditto academic credentials, etc. I recall a few years ago when I asked an installer to test a mod he'd done to my wife's car and he was "insulted" that I'd question whether it was done correctly and that it needed to be tested ... but when he actually (reluctantly) tested it, it didn't work !! He blamed the part (a spoiler), went to the trouble of removing it and installing another one; and when that also didn't work he THEN checked his wiring ... which he'd gotten wrong. No worries, I didn't imply you did or not know what your talking about, just that I got a good laugh from u 2 and brought back a memory Link to comment
garycase Posted June 7, 2013 Share Posted June 7, 2013 No worries, I didn't imply you did or not know what your talking about, just that I got a good laugh from u 2 and brought back a memory I chuckled at bit at my own post after I'd originally written it. I've had quite a few of those kind of experiences over the years ... and shouldn't have "bitten" at the provocation [i think the "... you haven't even progressed past the arithmetic ..." just hit a nerve ... but I should have simply ignored it (FWIW ... which I agree is not much ... I completed a PhD in math when I was 20) ] Link to comment
Pauven Posted June 7, 2013 Author Share Posted June 7, 2013 So I did a bit of poking around. I know on my current AMD based unRAID server, the PowerNow cpu scaling features are not enabled by default. I load the CPU Frequency Scaling Driving by running: modprobe powernow-k8 in my 'go' file which saves a lot of power. From what I can tell, the Intel SpeedStep functionality is enabled by: modprobe acpi-cpufreq Newer version of Linux have more power saving features built in, especially with kernel version 3.4 (which unRAID current RC is on). More info here: https://wiki.archlinux.org/index.php/CPU_Frequency_Scaling From what I've read, I think the default scaling governor settings are appropriate for an unRAID server. It looks like Intel has incorporated some driver rewrites into kernel 3.6, including Ivy Bridge support: http://www.phoronix.com/scan.php?page=news_item&px=MTE0NzY So it seems really easy to get the CPU power saving features enabled and optimized, but I might need the latest drivers from 3.6 for my new Ivy Bridge based Celeron. From additional research, Linux is reported to fall behind Windows on 3rd party driver power saving functionality. For example, in my new build the following devices all have drivers that may or may not enable power saving functionality in Linux: HD2500 graphics (though since this is integrated into the CPU, maybe it works with above) HighPoint RocketRAID 2760A (I really hope it works out of the box, otherwise it will consume 45W all the time) Realtek 8112B Gigabit LAN Not sure if I missed any other biggies. I plan to disable in the BIOS all features that won't be used for unRAID (i.e. Audio, SATA ports, COM ports, etc). I need to do some more research to determine if I need to load special drivers for any of the above devices to enable power saving features, and if there are any special settings I have to set for the drivers to actually enable power saving. I didn't expect Tom to take unRAID off of Linux kernel 3.4 so soon. Now that unRAID 5RC13 is using kernel 3.9.3, it would seem that the Ivy Bridge support introduced back in 3.6.x are now baked into unRAID! As I wrap up my testing on this low power server build, I plan to evaluate the impact of the Ivy Bridge support, since the G1610 I'm using is an Ivy Bridge chip. As a refresher, I was seeing a 17W-18W idle with just the MB/CPU/MEM on unRAID 5RC12a (no drives, no 2760A controller, no case fans, no backplanes). To test, I will re-validate that 17W-18W figure under 5RC12a, then upgrade to 5RC13 (all on a test thumb drive without drives attached, I will not be testing 5RC13 with my Production hardware at this time) and see if the numbers stay the same, or if I can get the idle even lower. Link to comment
garycase Posted June 7, 2013 Share Posted June 7, 2013 To test, I will re-validate that 17W-18W figure under 5RC12a, then upgrade to 5RC13 (all on a test thumb drive without drives attached, I will not be testing 5RC13 with my Production hardware at this time) and see if the numbers stay the same, or if I can get the idle even lower. Lower than 17w ?? Indeed an admirable goal ... although as you know, your PSU efficiency is probably relatively poor at those power levels, so I'm not sure it's worth the bother, as there are other places that it's a lot easier to "find" a few watts to save. I think your two biggest consumers of "extra" watts are going to be the 2760a and the idle power of 24 hard drives ... together that's going to add at least 40 watts => even if you use all WD Reds, 24 of them will draw 14.4 watts in standby, so those plus the controller are hitting you for over 40 watts independent of everything else. Then there's the draw of all the various cooling fans (case fans, hot-swap cage fans, etc.) ... Bottom line: I think you'll be doing very well if you can get the idle down to ~ 70w for that size system. Link to comment
Recommended Posts
Archived
This topic is now archived and is closed to further replies.