For years my #1 "wish list" item for UnRAID has been staggered spinup support.    While this may seem like a "so what" item, it does have one fairly significant impact in designing modern low-power servers.

 

In fact, the new LimeTech server could benefit from this a LOT.  This server, in its Atom version, uses the same motherboard I used in my newest server last year.  My server idles at 20w, and only draws about 45w at full load (parity check) with the 6 3TB WD Reds I have installed.

 

The LimeTech server also has a AOC-SASLP-MV8 controller, so clearly it's going to draw a bit more, but I'd expect it still idles in the 35w range, and probably doesn't draw more than 100w or so at full load.

 

Assuming that 35w-100w range is the correct operating range for the new server, then a "typical" draw is probably around 50-60w (with 3 or 4 drives spinning).    The server is offered with your choice of a 520w or 650w Seasonic power supply.    Note that the minimum load these power supplies operate at their rated 80+% efficiency is 104w (for the 520w) or 130w (for the 650w) ... both are well above what the server likely draws, so its power supply is going to be operating well below 80% efficiency.    Power supply efficiencies drop off rapidly below the 20% load factor.

 

And many of us are focusing on building systems that draw as little power as possible.  For example, my Atom-based server would run very nicely on this little PicoPSU unit EXCEPT for drive spin-up requirements:    http://www.amazon.com/Mini-Box-picoPSU-160-XT-Power-Mini-ITX-Supply/dp/B005TWE6B8    And the new LimeTech server could use a 300w 80+ unit and at least be close to the efficient operating range of the supply.

 

What I would LOVE to see is a user-configurable "spinup controls" setting with two parameters:

 

(a) Maximum number of drives to spinup at once [Values from 1 to 24]

 

(b)  Delay between spinup requests [Values in 1/10 sec increments from 0.1 to 3.0]

 

This would let users set values that best matched the power demands of their drives and the capability of their PSU.

 

This would be a GREAT feature !!!  ... at least in my opinion 

 

I agree. This would be an excellent enhancement candidate for 5.1 (32-bit) / 6.1 (64-bit) Releases.

 

 

what happens when a drive fails and all drives have to be spinning to create the failed drive?  If the PSU in that situation does not have enough juice then you will be completely SOL and more stuff will "blow up"

what happens when a drive fails and all drives have to be spinning to create the failed drive?  If the PSU in that situation does not have enough juice then you will be completely SOL and more stuff will "blow up"

 

No problem -- the PSU has to have enough power for all drives to spin .. but that's FAR less than the spinup current, which is typically 2-3 times the operating current.

 

what happens when a drive fails and all drives have to be spinning to create the failed drive?  If the PSU in that situation does not have enough juice then you will be completely SOL and more stuff will "blow up"

 

No problem -- the PSU has to have enough power for all drives to spin .. but that's FAR less than the spinup current, which is typically 2-3 times the operating current.

 

I posted this in another thread that no doubt spawned Gary's desire to create this thread...

 

Here's a great breakdown of common high capacity drives and their power states (taken from Storage Review):

 

You can see just how much more drives consume at startup than during reads and writes. Staggered spinup would allow for smaller (by a factor of at least 2x smaller) and more efficient PSUs to be used in our systems.

There would need to be housekeeping in the kernel driver to do what you want.

How many drives have been spun up in the last n seconds.

 

Then you have to consider, would one of the unRAID drivers timeout because spin up is being purposely delayed.

Would your network layer time out while spin up of 24 drives is being delayed.

 

A long time back I wrote a shell agent to take spin up commands via inetd and a TCP command.

It incorporated a staggered spinup. 

It's possible to do it in emhttp, but when dealing with a real failure in the kernel driver, it may not be the best approach.

 

This would have to be something for Tom to address in the unRAID code.

Maybe do the spinups by host groups then pause.

As for initial power up, you can do this now with power up in standby.

Yet it's only effective for the initial power on.

I suggest you go through your drives and set it, then see how long it takes for your system to come up.

In my systems, there would always be one or two drives that required a retry, thus delaying it longer.

 

While I think staggered spinup is a good idea, I'm not sure it will be good for the really large arrays.

"IF there were a failure."

 

as I understand it they'll all be powered on at once if you click the "Spin Up" button;  start a Parity Check; or Stop the array.    It would be SO much nicer if UnRAID implemented a sequential startup ... a simple checkbox in Disk Settings would be fine;  a delay time setting (seconds between spinups) would be a nice "tweak".

 

This could be done in emhttp however the kernel driver,  at the current time, would not adhere to any rules unless that was changed too.

It's all possible, where is the biggest bang for the buck in limetech's time.

Staggered spinup or 64Bit?

what happens when a drive fails and all drives have to be spinning to create the failed drive?  If the PSU in that situation does not have enough juice then you will be completely SOL and more stuff will "blow up"

 

No problem -- the PSU has to have enough power for all drives to spin .. but that's FAR less than the spinup current, which is typically 2-3 times the operating current.

 

I posted this in another thread that no doubt spawned Gary's desire to create this thread...

 

Here's a great breakdown of common high capacity drives and their power states (taken from Storage Review):

 

You can see just how much more drives consume at startup than during reads and writes. Staggered spinup would allow for smaller (by a factor of at least 2x smaller) and more efficient PSUs to be used in our systems.

What I find interesting in this graph is the power needed by Western Digital drives when spinning up from the 5Volt supply. {Looks like roughly 10 watts.)  For a 5 volt supply, this is about 2 Amps.    For a full array of 24 WD drives, you could be drawing 48 Amps from the 5 volt rail in addition to the 60 or so Amp load on the 12 volt rail.

 

Now, we might think a really high wattage supply will be able to handle this... but.... perhaps not.

 

I looked at this 1200 watt CORSAIR supply.  (I picked one of the biggest I could find on newegg)

http://www.newegg.com/Product/Product.aspx?Item=N82E16817139014

Single 12 volt rail at 100 Amps...  But... 5Volts at only 30 Amps.  That is not enough amps on the 5volt rail to properly handle 24 WD drives spinning up.  (it would be fine with 24 Hitachi, Samsung, or Seagate drives)

 

In looking around, I did not find any power supply with more than 30 Amp capacity on the 5 volt rail... most were 25 Amps.

There is more to choosing a power supply than it looks on the surface... depending on the brand of the drives.  WD drives are power hungry when spinning up.

 

The staggered spinup being requested might be more useful than you think if you have all WD drives.

 

Joe L.

In looking around, I did not find any power supply with more than 30 Amp capacity on the 5 volt rail... most were 25 Amps.

There is more to choosing a power supply than it looks on the surface... depending on the brand of the drives.  WD drives are power hungry when spinning up.

 

The staggered spinup being requested might be more useful than you think if you have all WD drives.

 

Joe L.

 

 

It's useful, but let me quote another member who posted on the other version of this thread.

 

 

Regardless of efficiency ratings etc, it's a very bad idea to rely on software settings to not blow up your PSU if all your drives decide to spin up at once for some unknown reason. All it would take is one mistake in software, and POOF, dead drives and or PSU. The power supply should be sized for peak possible draw plus a small margin.Intentionally undersizing a PSU for a few watts (at most) of efficiency is penny wise and pound foolish in my opinion. Staggered spinups may be desirable for other reasons, but to intentionally allow for a smaller PSU isn't one.

 

I would like to add/ask, how many people have > 24 drives in a server on one PSU?

Is it prudent to configure your server that way?

Have we heard/seen people popping their power supplies because of it?

 

It's only food for thought.

 

For me the staggered spin up 'on power up' was of paramount importance.

It's probably one of the most stressful times for the server as everything is coming up to speed.

Temperatures might be cold. etc, etc.

I've lost more PSU's in that initial power on then anything else.

 

For clarity here, I'm not opposing staggered spin up. I think it could (should) be incorporated in emhttp.

Basic code for that probably wouldn't be that complex.

 

Possibly put a 'small' delay in the driver, but anything advanced in the driver will probably need housekeeping and allot of testing.

 

If the goal is to use a smaller PSU, then I think your machine will suffer the fate of Barjiza's where he said it blew out his PSU.

Remember that BackBlaze is currently using (2) 760w PSUs for their 45 drive Storage Pods, plus one more OS drive. From the blog post:

 

There is more to power than just Watts. ATX power supplies deliver power at several voltages or ‘rails’ (12V, 5V, 3.3V, etc). Each vendor imposes unique limits on the amount of power you can draw off of each rail and unused power on one rail cannot be used on another. In particular, most high end power supplies are designed to deliver most of their power on the 12V rail because that is what high end gamer PCs use. Unfortunately, hard drives draw a lot of power off the 5V rail and can easily overwhelm a high wattage power supply. You will hit serious problems if power requirements for each component are not met so be careful if you don’t use the power supplies we recommend.

 

The specs for their recommended power supply, the Zippy 760w PSM-5670V:

VOLTAGE:  90 ~ 264 VAC FULL RANGE

FREQUENCY : 47 ~ 63 HZ

INPUT CURRENT: 12 A ( RMS ) FOR 115VAC / 6A ( RMS ) FOR 230VAC

INRUSH CURRENT: 20A MAX. FOR 115 VAC / 40A MAX. FOR 230 VAC

PFC CAN REACH THE TARGET OF 95% @230V, FULL LOAD

 

Output VoltageOutput Current Min.Output Current Max.Output Current PeakRegulation LoadRegulation LineOutput Ripple & Noise Max.[P-P]

+5V0.525.00±5%±1%50mV

+12V155.00±5%±1%120mV

-12V00.8±5%±1%120mV

+3.3V0.525±5%±1%50mV

+5VSB0.13.5±5%±1%50mV

 

* TEMPERATURE RANGE: OPERATING 0C ~ 50C

* HOLD UP TIME: WHEN POWER SHUTDOWN DC OUTPUT 5V MUST BE MAINTAIN 16MSEC IN REGULATION LIMIT AT NORMAL INPUT VOLTAGE

* HUMIDITY: OPERATING:20%-80%, NON-OPERATING:10%-90%

* EFFICIENCY: TYPICAL 80-85% AT 25C 115V FULL LOAD

 

This is in a production environment designed for long term storage, where customers will not tolerate a lot of "poof" type failures.

 

They are effectively running twice as many drives as we can on one unRAID installation, on about 1500W worth of PSUs. So dividing their requirements in half effectively cuts it down to one sub 1000w PSU with a beefy 5v rail being sufficient for 20+ drives, the motherboard, and processor. Clearly they avoided the surprisingly 5v hungry WD drives.

 

Based on their recommendations, I would submit that the Zippy 860w PSM-5860V (the bigger brother) could probably handle the job of a 24 drive, non-WD based unRAID server, with a moderate processor.

 

There is wisdom in Jonathan's post about having enough power to cover a simultaneous startup of all components in the event the OS fails to stagger the drives.

 

Having said all of the above, it would be much less taxing on the system as a whole if the OS could natively stagger the startup of the drives.

I am not completely against a staggered spin up... but... I think anything over the top is asking for and opening a can of worms.

 

The power supply you chose should be enough to cover peak draw plus a little more.

 

If the PSU happens to not run at its optimum efficiency most of the time that is no skin off my back.  I know that if/when all drives come online shit is not going to hit the fan because I thought I was being smart and tried to "optimize" the server for the 99% use case.  That 1% use case will come back to bite you in the arse.

The WD REDs seems to have fixed the high 5v power consumption on startup (Stogare Review).

 

 

What I find interesting in this graph is the power needed by Western Digital drives when spinning up from the 5Volt supply. {Looks like roughly 10 watts.)  For a 5 volt supply, this is about 2 Amps.    For a full array of 24 WD drives, you could be drawing 48 Amps from the 5 volt rail in addition to the 60 or so Amp load on the 12 volt rail.

 

That's indeed a lot of power ... and it would be scary IF it was accurate 

 

But according to Western Digital's specifications, the two drives probably most used by folks here are nowhere near that:  According to WD's specs, a 2TB WD Green draws 10.1w at spinup ... 0.7a from the 5v bus and .55a from the 12v bus => or, for a 24-drive system, 16.8a from the 5v bus, and 13.2a from the 12v bus (a total load of 242.4w).    A 3TB WD Red draws 8.4w during spinup ... 0.6a from the 5v bus, and .45a from the 12v bus => or, for a 24-drive system, 14.4a from the 5v bus, and 10.8a from the 12v bus (a total load of 201.6w)

 

While these are based on WDs specifications, the figures for the green drives very closely match the measurements Storage Review did in a review of the greens last year, so I suspect they're pretty accurate for both.

 

 

The WD REDs seems to have fixed the high 5v power consumption on startup (Stogare Review).

 

 

I don't think WD "fixed" anything ... I think the first chart was simply wrong 

Looking at the total spinup requirements vs. modern PSU's, a high-quality 400-450w 80+ unit with a single-rail 12v supply easily meets a 24-drive spinup load as long as the system doesn't have a high-end GPU drawing a lot of power at the same time ... a VERY unlikely circumstance for an UnRAID server.    This wouldn't have been true with previous generations of hard drives, which had notably higher spinup current requirements, but all of the modern high-capacity drives are much more power-efficient.  The WD 2TB and 3TB units I noted above;  the 4GB Seagate DM (draws more than the WD Reds, but less than the WD Greens);  and I suspect the forthcoming 4TB and 5TB WD Reds will be in the same ballpark.

 

As an example of a 450w unit that should work fine, consider the Seasonic SSR-450RM.  It has a 20A 5v rail and 37A 12v rail.    And works at 80+% as low as 90w, so should still be above 70% down to about a 45w draw => so only about 8-9w of "wasted" power due to the lower efficiency.  Of course at those power levels, 8 wasted watts is 20% ... but I agree it's small enough to be irrelevant.    And a 24-drive system isn't going to idle below 45w, so it won't get any worse.   

Reading through my own post, I basically convinced myself that the wattage penalty for using a PSU that can handle the full spinup load isn't all that big a deal 

 

I'd still like to see staggered spinup, just to be "kind" to the PSU ... we don't use nice regulated linear supplies these days; and the switching PSUs are "stressed" when the power demands jump so rapidly to max loads.    But nevertheless, being able to cover the full spinup load isn't a bad idea, especially if it can be done with only a 10% efficiency "penalty".

 

Gary, you talk to yourself a lot, don't you?   

Gary, you talk to yourself a lot, don't you?   

 

A sign of age 

 

Actually, in this case, as I read through the posts (mine and others), I basically decided that the "penalty" for having a PSU large enough to spin up all the drives isn't all that big a deal [i had to switch my "mindset" from the 6-drive system I use for my low-power server to a 24-drive system];  so perhaps the staggered spinup isn't all that big a deal.

 

To wit:  If you assume a 50w minimum draw (reasonable for a 24-drive system), then using a 450w PSU at that level is probably going to be about a 9 watt "penalty" compared to a PSU that was running at 80%.    Even if you used a 750w unit (a popular size for large arrays), it'd only be about a 21 watt penalty.    A 24/7 system wasting 21 watts is costing about an extra $23/year at $0.125/kwh (typical US cost).

 

Clearly an insignificant cost.

 

... so much for staggered spinup 

(It would still be a nice feature ... but clearly isn't economically justified.  Just hate to waste 21 watts on a system that's only drawing 50 watts !!)

 

I think PUIS (power up in standby) is important.

 

A. It does stagger the spinups.

B. it's more for problem situations then anything.

 

Let's say you power up. Your boot up fails or you have to replace a hard drive.

Then it's not detected by the bios or something.

You power off.

You power on.

 

Maybe you do that a few times adjusting a cable or something.

 

Point is, I've crashed/ruined hard drives just from the power up and having to power it all down again.

At one point the drive head must have had some kind of power issue and it scrambled a few sectors.

 

I believe staggered spin up is important, just not important to save a few watts and allow a less then adequate power supply.

 

 

 

I believe staggered spin up is important, just not important to save a few watts and allow a less then adequate power supply.

 

Agree ... this discussion has convinced me the power savings aspect of it doesn't matter; but I agree there are still other very good reasons for it.

 

I'd still like to see staggered spinup, just to be "kind" to the PSU

Yes, this exactly. I'm for anything that can reduce failure modes.

I'd still like to see staggered spinup, just to be "kind" to the PSU

Yes, this exactly. I'm for anything that can reduce failure modes.

 

Does anyone have any statistics from forum members reporting PSU failure due to not having staggered spin up?

 

[that guy's] statement recently was the first I've heard.

 

My concern about the staggered spin up is, once adjusted, people may not see the need to size their PSU adequately. We'll start to see more intermittent failures being reported.

 

As it is now, if you spin down/spin up a few times and have no failures, you're probably OK.

If you spin up and something doesn't work, there could be an issue with the PSU somehow.

i.e. you'll know something is up before you start second guessing everything.

Just to provide some comic relief:

 

From ~2005-2007 I had a server with ~21 drives connected (never less then 12 never more than 21 all 200-500GBs each) to a 3.2Ghz Pentium4 478 MB.  The PSU was an OCZ 500/550 watt multi rail PSU.  Believe it was 18amp on each rail and had 3 rails - but could be off - definately was multi rail.  It ran fine once I got it going but it was a b#%$h to get started.  I had to switch the power switch on the PSU and press the power button on the case in just the right sequence multiple times to catch the drives spinning and requiring less power to complete spinup in order to boot up completely.  Needless to say I didn't turn it off if I could help it.  I got my first 750watt PC Power & Cooling Silencer (single rail) to replace the OCZ when after two years it refused to boot.  During all of this I never had a drive fail while booting.  I'm sure they aged quicker but other than dropping from my RAID 5 arrays had few drive failures.

 

Edit: $#% spelling errors.

In my situation, the last failure I had, the power off & on must have scrambled a sector.

 

I knew the drive was good before hand because I had just completed a parity check. 100% aok no errors.

I turned off the machine, upgraded unRAID from 4.5 to 4.7. It hung. I had to hard power off while the drives were spinning.

 

Powered on again.  I know my PSU was good. I was using it for quite some time 750 Corsair.

must have been an option in the syslinux.cfg and I manually adjusted.

In any case, the system came up and immediately started 'upgrading my parity'  WTF!!! (I was baffled)

 

Ok not so bad I thought. I just finished a parity check.

But lo and behold, a bad sector just happened to have developed on one of my most important drives.

So the parity upgrade failed.

 

I was lucky enough to find ddrescue which allowed me to copy all the readable sectors forward, skipping the bad sectors, then read the drive backwards retrying all the bad sectors. I recovered all but 1 sector out of 1GB.

 

This is why I stress power up in standby.

If you have to turn off and on again before you access the drives.

You just never know how the drives are going to handle any form of extra electricity or sudden power off. (for lack of a better term).

 

It's not the first time this has happened to me with WD green drives either.

They always seemed to suffer after a shut off (on multiple machines and different brands of PSU's).

I learned to use them to death and go to another brand, while always keeping a spare.

 

It's also another reason why I like to use the badblocks tool in multiple pass multiple pattern writes. It really tests out the sectors and/or refreshes/reassigns them.

 

My experience has been more like Bob's => I've seen PSU's that couldn't power up all of the drives that were attached, but it's never (at least not that I'm aware of) caused data errors.  With switching supplies (all modern PSUs), the transients caused by the high-demand ramp-up are probably more of an issue than the actual wattage capacity.    I suspect that using those SATA cables with built-in 2200uf capacitors that were discussed in another thread would be beneficial when connecting a large number of drives, as these would tend to smooth out those transients.

 

 

My supermicro cards already do staggered spin up.