May 20, 201313 yr What is your current power usage and how many hard drive are currently installed? READ the thread ... this whole thread is about low-power consumption => I and a few others have noted several very low power consumptions systems; and had a bit of a discussion about the PicoPSU possibility (this post was a follow-up to that discussion) ... the specific answer to your question is in reply #2
May 22, 201313 yr I did read the thread but there was no clear cut answer for 24 drive system and the Intel Dual Core Atom D525 is very slow (current unraid build) and it takes 11 days to do a parity check on 2tb!
May 22, 201313 yr I did read the thread but there was no clear cut answer for 24 drive system and the Intel Dual Core Atom D525 is very slow (current unraid build) and it takes 11 days to do a parity check on 2tb! If you're taking 11 days for a parity check you have something seriously wrong with your system -- and it has nothing to do with the CPU you're using My D525-based backup server has 6 3TB WD Reds and takes just over 8 hours to do a parity check, using a max of only 48watts (it idles at 20W). It has excellent read/write performance on a par with any other UnRAID system I've seen. Clearly it doesn't have the "horsepower" to run some of the CPU-intensive add-ons; but it would be fine for a 24-drive system. Parity checks aren't very CPU-intensive ... they're primarily driven by disk speed. Limetech is using the same motherboard in their new 14-drive server as I used in the build I did last year; and I doubt the parity check times on that will take any longer than mine do now ... or at most a few minutes longer.
May 22, 201313 yr hilljd00, Is that what you were going to run Unraid virtualized under ESXi? If that is the situation, that looks like it may be the problem...
May 22, 201313 yr Author I did read the thread but there was no clear cut answer for 24 drive system and the Intel Dual Core Atom D525 is very slow (current unraid build) and it takes 11 days to do a parity check on 2tb! Hi hilljd00, we are still in the process of researching the answer for a 24 drive system. I have actually purchased the components to make a test system, using a Celeron G1610 CPU, Foxconn H61S motherboard, and Adaptec 2760A SAS controller card. These parts should all arrive by Thursday, and I expect to have some initial test results shortly thereafter. A couple points: [*]The goal of this particular build was very low power consumption, not ESXi compatibility. unRAID will be running on bare metal, not virtualized. The components chosen are not suitable for an ESXi build (but neither is your Atom build) - in fact there may not be any support for the 2760A under ESXi. I am not sure that ESXi takes advantage of the latest power saving technologies (though I would be interested to know). [*]We have theorized that the minimum idle wattage on a 24 drive unRAID server is about 35-45 watts (15-20 watts for the CPU/MB, 5-10 watts for the SAS controller card, and 15 watts for the 24 hard drives themselves). In my build, careful selection of components was performed to balance performance with energy efficiency, but at a greater purchase $$$. I agree completely with garycase and electron286, you have a serious issue in your build. If you are attempting to use ESXi, that is most likely the root cause. Try running unRAID on bare metal (no ESXi) and see if your results improve. If you are not using ESXi, then you should start a new thread asking for help with your build. Good luck. -Paul
May 22, 201313 yr ...slightly OT, but inspiring......well, maybe. -> 8.5W Core i3-based desktop computer (English)
May 22, 201313 yr Author ...slightly OT, but inspiring......well, maybe. -> 8.5W Core i3-based desktop computer (English) I wouldn't call it off topic at all, just perhaps not applicable on a 24-drive unRAID server. I read that same article a few days ago in hopes of finding solutions. I was shocked at how far the author went to save an extra watt or two. He built his own micro power supply, because a PicoPSU wasn't good enough. He rewired the motherboard so that voltage was sourced from the most efficient point, even if that mean the resulting voltage was out of spec. He replaced some chips with lower power versions. He also cut power to many components altogether. None of those are things I want to do to a server I'm trusting with my valuable data. Doesn't seem worth it for a couple watts anyway. If I was that obsessive, I would simply rip the motherboard out of a laptop as my starting point. While it is wonderful to see such low power systems, it is doubtful that the solutions transfer directly to a 24-drive unRAID server. The biggest problem with storage servers is the amps it takes to spin-up a hard drive, combined with the fact that unRAID has the nasty behavior of spinning up all drives simultaneously. Using WD Red 3TB drives, the spin-up Amps per drive is 1.73 on the 12V line, so for 24 drives you will need 41.5 Amps, which is 500W on the 12V line. That's not even counting the power needed by the CPU, motherboard, or controller card. These power requirements far outstrip a PicoPSU's capability. We think that spinning up 5-6 3.5" drives may be possible by a PicoPSU, but that is right at the limit and we haven't seen any tests results. The relatively high spin-up Amps consumption means you have to buy a much larger power supply (I just picked up a 650W power supply) to be able to spin-up 24 drives and run the PC at the same time. Assuming you had a system capable of idling at 10W, that same 650W power supply would be outputting only 1.5% of its rated power, far below the 20% threshold used in 80+ drive certification, and the power supply would be very inefficient at this output level. Assuming it was 50% efficient, that 650W power supply would burn 10W as waste heat to produce 10W of usable electricity, for a total consumption of 20W. So for a 24-drive unRAID server, those ultra-ultra-low power levels just are not reachable. By my calculations, my new build will consume ~40W, not counting the power supply inefficiency. I picked a notably efficient Kingwin LZP-650, which looks to only have about 7W of energy waste at 40W (80%+ efficient at 6% load, truly remarkable!), so total power consumption will be closer to 50W. I appreciate you sharing the link Ford Prefect.
May 22, 201313 yr A very interesting link, but for all the reasons Pauven just noted, it isn't useful in the quest to build a low-power 24 drive system. And even if I wanted a super-low-power minimal UnRAID system, I wouldn't be inclined to start swapping parts on the motherboard; re-routing power; or using a custom power supply that generated the WRONG (out of spec) voltages !!
May 22, 201313 yr Thank you for your kind words Pauven. While it is wonderful to see such low power systems, it is doubtful that the solutions transfer directly to a 24-drive unRAID server. The biggest problem with storage servers is the amps it takes to spin-up a hard drive, combined with the fact that unRAID has the nasty behavior of spinning up all drives simultaneously. Using WD Red 3TB drives, the spin-up Amps per drive is 1.73 on the 12V line, so for 24 drives you will need 41.5 Amps, which is 500W on the 12V line. That's not even counting the power needed by the CPU, motherboard, or controller card. I just did some measurements with my X8SIL-F based, ESXi build....and found the startup wattage was way below (25%) of what I'd expected. Here's my config, in short: X8SIL-F, XEON 3426L , 32GB-RDIMM, 1xM1015, 1xRES2CV240, Norco4224 with 6x 80mm fans(LEPA-80D), 1x 3V 256GB SSD, 4x DVB-S2 tuner cards on 1x PCIe bridge with dual CI (empty). Disks: 7x WD-EARS 2TB, 7x 7200.14 3TB, 2x WD REDs 3TB, 3xToshiba 7.2k HGST 3TB All in idle: 75W (fans incl. CPU on max, disks spun down, all NICs up incl. IPMI, ESXi reporting 120MHz total "consumption) Doing a cold boot: 300W Max online load: 235W (ZFS scrub on all arrays, DVB recordings/EPG grabbing online - when I shutdown the DVB recorder-VM, wattage drops by 30W). PSU: SuperFlower 600W 80Plus-Platinum ...with 19 disks at 1.5A or higher according to datasheets, giving a total of 28.5A, I would have expected 350W to 400W or even more during boot. So with 24 disks, my setup would probably cope with little over 400W? A 1155 socket build would even be lower, I believe?
May 22, 201313 yr Author Thanks for the numbers Ford Prefect. As an interesting point, if I am able to get my server to idle below 45W, that would save me over $300/yr in energy costs compared to your 75W server (~$.11/KWh in my area). Seeing as my current server is consuming over 100W, my annual savings from the new build could be as much as $600/yr! My new hardware will pay for itself in a couple years, and after that it starts truly saving me money. Of course, all of this is assuming I can get the idle that low. ...with 19 disks at 1.5A or higher according to datasheets, giving a total of 28.5A, I would have expected 350W to 400W or even more during boot. So with 24 disks, my setup would probably cope with little over 400W? I think you may have just touched on a major variable in your test: boot spin-up vs. unRAID spin-up AHCI drives automatically (from what I've read) implement staggered spin-up. Most controller cards also implement staggered spin-up. However, the staggered spin-up functionality only works: On boot With software that correctly calls the functionality We are actually expecting peak spin-up amps to hit when all drives are spun-up simultaneously by unRAID, which does not implement staggered spin-up. Please do one additional test: Get your server idle with all drives spun down. Then issue a spin-up and monitor your power usage. Report back. I am also curious what your system would idle at if you were running unRAID on bare metal, without ESXi. Has anyone done such a comparison? Thanks! I look forward to hearing your results. -Paul
May 22, 201313 yr Thanks for the numbers Ford Prefect. As an interesting point, if I am able to get my server to idle below 45W, that would save me over $300/yr in energy costs compared to your 75W server (~$.11/KWh in my area). You might want to check your math 30 watts savings x (24 x 365) hours = 262,800 watt/hrs = 262.8 KWh x $0.11 = $28.91 ... seems a bit closer to $30/year than $300
May 22, 201313 yr I think you may have just touched on a major variable in your test: boot spin-up vs. unRAID spin-up Absolutely correct with this one. It would indeed be a good test to carefully monitor the consumption for a second or two immediately after clicking the "Spin Up" button in the Web GUI.
May 22, 201313 yr Thanks for the numbers Ford Prefect. As an interesting point, if I am able to get my server to idle below 45W, that would save me over $300/yr in energy costs compared to your 75W server (~$.11/KWh in my area). Seeing as my current server is consuming over 100W, my annual savings from the new build could be as much as $600/yr! My new hardware will pay for itself in a couple years, and after that it starts truly saving me money. Of course, all of this is assuming I can get the idle that low. Yes, but mind you that I have a quad-tuner card in it as well. i believe it will account for some watts...even when idle. The reason I measured the consumption was my last energy bill ..I am currently thinking to resize my box and move the DVB recorder to one that can hibernate/sleep when nothing is planned. I think you may have just touched on a major variable in your test: boot spin-up vs. unRAID spin-up AHCI drives automatically (from what I've read) implement staggered spin-up. Most controller cards also implement staggered spin-up. However, the staggered spin-up functionality only works: On boot With software that correctly calls the functionality We are actually expecting peak spin-up amps to hit when all drives are spun-up simultaneously by unRAID, which does not implement staggered spin-up. Please do one additional test: Get your server idle with all drives spun down. Then issue a spin-up and monitor your power usage. Report back. I am also curious what your system would idle at if you were running unRAID on bare metal, without ESXi. Has anyone done such a comparison? I used to go with staggered spinup in the past, when I was on a dual rail PSU....I have now deactivated it My results for "unRAID spinup" would be my measures with "load", since I started from idle and then fired a scrub on my arrays (not using unRAID (anymore) but Solaris, because I need full-disk-encryption). A scrub is similar to a full parity check...all drives spinning...I get around 600MB/sec throughput on each (unencrypted) array. So for the disks alone, without DVB, best assumption is 235W minus 30W from recorder VM = 205W. My "kill-a-watt" keeps min and max values and I did reset it when system was on idle with 75W..
May 22, 201313 yr I think you may have just touched on a major variable in your test: boot spin-up vs. unRAID spin-up Absolutely correct with this one. It would indeed be a good test to carefully monitor the consumption for a second or two immediately after clicking the "Spin Up" button in the Web GUI. My meter keeps min and max values and I did a reset for each turn. On cold boot, meter was on zero and showed 300W as max value when it showed 75W as actual (and min) when system was idle (after 2hrs)
May 22, 201313 yr Thanks for the numbers Ford Prefect. As an interesting point, if I am able to get my server to idle below 45W, that would save me over $300/yr in energy costs compared to your 75W server (~$.11/KWh in my area). You might want to check your math 30 watts savings x (24 x 365) hours = 262,800 watt/hrs = 262.8 KWh x $0.11 = $28.91 ... seems a bit closer to $30/year than $300 ...at this side of the pond, 1 KWh of electricity is worth 0.32USD (0.25 EURO) Edit: used currency converter for actual exchange rate.
May 22, 201313 yr Thanks for the numbers Ford Prefect. As an interesting point, if I am able to get my server to idle below 45W, that would save me over $300/yr in energy costs compared to your 75W server (~$.11/KWh in my area). You might want to check your math 30 watts savings x (24 x 365) hours = 262,800 watt/hrs = 262.8 KWh x $0.11 = $28.91 ... seems a bit closer to $30/year than $300 ...at this side of the pond, 1 KWh of electricity is worth 0.30USD (0.25 EURO) It's all those extra volts they're charging you for
May 22, 201313 yr Thanks for the numbers Ford Prefect. As an interesting point, if I am able to get my server to idle below 45W, that would save me over $300/yr in energy costs compared to your 75W server (~$.11/KWh in my area). You might want to check your math 30 watts savings x (24 x 365) hours = 262,800 watt/hrs = 262.8 KWh x $0.11 = $28.91 ... seems a bit closer to $30/year than $300 ...at this side of the pond, 1 KWh of electricity is worth 0.30USD (0.25 EURO) It's all those extra volts they're charging you for ...no, actually its the extra KWh from Distributed Energy Resources (producers of "green" energy get a premium and community of consumers do pay with their bill).
May 22, 201313 yr Author Thanks for the numbers Ford Prefect. As an interesting point, if I am able to get my server to idle below 45W, that would save me over $300/yr in energy costs compared to your 75W server (~$.11/KWh in my area). You might want to check your math 30 watts savings x (24 x 365) hours = 262,800 watt/hrs = 262.8 KWh x $0.11 = $28.91 ... seems a bit closer to $30/year than $300 Can't argue with you on that one. I need to learn not to multi-task... Thanks for the correction! Since I'm replacing a server, assuming I sell my current server to recoup some of the cost, that puts my return on investment closer to 7-10 years. If you're starting from scratch, the ROI is much better because A) you have to buy hardware anyway, B) the premium on energy efficient hardware is not that big. Hmmm, I really should stop talking until I get to test my new hardware tomorrow. My luck it will idle at 200,000 Watts and I'm gonna look like a big fool.
May 23, 201313 yr Author I used to go with staggered spinup in the past, when I was on a dual rail PSU....I have now deactivated it My results for "unRAID spinup" would be my measures with "load", since I started from idle and then fired a scrub on my arrays (not using unRAID (anymore) but Solaris, because I need full-disk-encryption). A scrub is similar to a full parity check...all drives spinning...I get around 600MB/sec throughput on each (unencrypted) array. So for the disks alone, without DVB, best assumption is 235W minus 30W from recorder VM = 205W. My "kill-a-watt" keeps min and max values and I did reset it when system was on idle with 75W.. Okay, I had to read this a few times to get it straight, mainly because what you are saying was completely unexpected. Let me see if I got it right. 1) You're not using unRAID. 2) The all drives spin up at the same time feature of unRAID does not apply to your system, because of #1 3) You ran a Scrub in Solaris to spin up the drives, and that peak consumption number is reported in your 'Load' test number. 4) We can't tell from your comments if the Solaris Scrub Spin-up implements a staggered spin-up or simultaneous spin-up, or even if the drives were spun-down when you started the Scrub "from idle". 5) You've (somehow) disabled the staggered spin-up feature of your HD controller card(s) 6) A cold boot causes all drives to spin up simultaneously, because of #5a 7) You're cold boot peak power consumption is only 300W, even with 19 drives and an estimated 28.5 Amps required for spin-up If that's the case, then I agree with you completely, I would have expected the number to be higher, especially with the hardware you are running. I can think of three possible explanations: A) The hard drive spin-ups are still staggered, even if only partially (this makes sense because the M1015 controller only has 8 ports, and the RES2CV240 card is just an expander. If you have a mix of drives across different controllers, they may not all be starting at exactly the same time). B) The spin-up event is too short for your kill-a-watt to catch it. You might think a spin-up takes a few seconds, but that initial Amp spike is probably a few 100th's of a second, and the kill-a-watt may not have the resolution to see it. C) Spin-Up Amps are overrated... Regardless, thanks for sharing your numbers Ford Prefect. It does seem like you could downsize your power supply a little bit, especially if you remove the tuners. I'll be curious to see how my results, with a simultaneous unRAID HD spin-up, compare with yours. -Paul
May 23, 201313 yr I'll be curious to see how my results, with a simultaneous unRAID HD spin-up, compare with yours. Me too ... I'm also very interested in confirming that the 2760A works with UnRAID !!
May 23, 201313 yr Okay, I had to read this a few times to get it straight, mainly because what you are saying was completely unexpected. Let me see if I got it right. Hmm...yes, sorry for any inconvenience..English is not my native tongue. Thanks for your analysis hence 1) You're not using unRAID. Yes, I own two plus licenses and have retired them a while ago. I need full disk encryption which is not envisioned to make it into unRAID in any future release. 2) The all drives spin up at the same time feature of unRAID does not apply to your system, because of #1 3) You ran a Scrub in Solaris to spin up the drives, and that peak consumption number is reported in your 'Load' test number. 4) We can't tell from your comments if the Solaris Scrub Spin-up implements a staggered spin-up or simultaneous spin-up, or even if the drives were spun-down when you started the Scrub "from idle". ...from the software functionality point of view, your are right....solaris has not a spinup/spindown button. But technically, in this respect a ZFS array is like any other, conventional soft- or hardware raid...all drives in the array need to spin when it is accessed by a process that reads data. My drives are spun down when idle...Solaris has a working power management feature...I can also tell from the sound, case "vibration" and inside temps when drives are down or up. Staggered spinup is not implemented in solaris AFAIK...it will be a feature of the controller and therefore transparent to the OS. If it works for spinup after idle, it will do that for unRAID as well as for any other OS....if it will do that on boot only, it will be the same regardless of the OS. 5) You['ve (somehow) disabled the staggered spin-up feature of your HD controller card(s) 6) A cold boot causes all drives to spin up simultaneously, because of #5a Yes, I think I just set the no. of disks to spinup in a group higher than the physical no. of disks mounted inside my case ...what I could try is to flash one of my spare M1015s without BIOS...assuming that this is a BIOS feature, it should be gone for sure. 7) You're cold boot peak power consumption is only 300W, even with 19 drives and an estimated 28.5 Amps required for spin-up If that's the case, then I agree with you completely, I would have expected the number to be higher, especially with the hardware you are running. I can think of three possible explanations: A) The hard drive spin-ups are still staggered, even if only partially (this makes sense because the M1015 controller only has 8 ports, and the RES2CV240 card is just an expander. If you have a mix of drives across different controllers, they may not all be starting at exactly the same time). B) The spin-up event is too short for your kill-a-watt to catch it. You might think a spin-up takes a few seconds, but that initial Amp spike is probably a few 100th's of a second, and the kill-a-watt may not have the resolution to see it. C) Spin-Up Amps are overrated... Regardless, thanks for sharing your numbers Ford Prefect. It does seem like you could downsize your power supply a little bit, especially if you remove the tuners. I'll be curious to see how my results, with a simultaneous unRAID HD spin-up, compare with yours. I agree on the sensitivity of the meter to be the weak point. AFAIK all "consumer grade" kill-a-watt meters will not implement sub-second meter readings...You'll need a professional logger for that, I think.
May 23, 201313 yr Author All the equipment came in today, so I've started assembly and testing. Early teaser results: MB+CPU (Foxconn H61S+Celeron G1610) idle: 17W-18W MB+CPU+2760A idle: 46W Haven't actually tested 2760A unRAID compatibility yet. My hunch is that the drivers are not loaded, so the chips are running hot (the heatsink is too hot to touch). I certainly hope the 2760A doesn't consume 29W on idle. So, I'm a bit of a LINUX idiot... how do I load drivers with unRAID? modprobe?
May 23, 201313 yr Yes it is modprobe <kernel-module_name>......but the driver (kernel module) should load automagically. Do a "lsmod" and see if a driver for it is loaded (don't know what the name of it will be). You could also try "dmesg" which will output the kernel log generated during boot. In there you should find a trace what hardware was recognized and what driver got loaded (if any).
May 23, 201313 yr Author Thanks Ford Prefect, that was helpful. A driver for the 2760A is being loaded, and I can see a drive attached, so far so good. The driver, to the best that I can tell, is 'mvsas', version 0.8.16. My gut is telling me this is a generic driver, just like many of the generic drivers that ship with Windows. Is this a correct assumption? Should I be loading a manufacturer driver to get full functionality? If so, I'm not sure which driver to start with... There seems to be nearly a hundred different drivers to pick from. They (HighPoint) have provided drivers for the following distros: Debian Fedora openSUSE RHEL_CentOS SLES Ubuntu What I don't see is SlackWare; isn't unRAID based on SlackWare? Also, for each distro, there are version options (i.e. for Ubuntu it has different drivers for Ubuntu versions 8.04, 8.10, 9.04, 9.10, and 10.04) Any suggestions on how to proceed? I really appreciate the help! -Paul
May 23, 201313 yr Author I should have put this in my last post, here's a link to the HighPoint drivers for the 2760A: http://www.highpoint-tech.com/USA_new/CS-PCI-E_2_0_x16_Configuration.html In addition to the previous distros I listed, I see a 'Linux Open Source' version... perhaps that's where I should be looking?
Archived
This topic is now archived and is closed to further replies.