[Plugin] NUT v2 - Network UPS Tools

[Rysz]

14 minutes ago, knights_of_pine said:

unfortunately, no change...

 

I'm very sorry, I really have zero clue why this doesn't work for you.

It works for me on my testing server and also for many other users who use these filters. 😕 

[Masterwishx]

@Rysz i was asked by NUT maintainer to check pr for test something for my model UPS , can you please say how better to compile this pr for plugin, or use like separate slackware package ?

https://github.com/networkupstools/nut/pull/2562

Edited July 28 by Masterwishx

[Rysz]

13 hours ago, Masterwishx said:

@Rysz i was asked by NUT maintainer to check pr for test something for my model UPS , can you please say how better to compile this pr for plugin, or use like separate slackware package ?

https://github.com/networkupstools/nut/pull/2562

 

I can compile you a test package, what version of Unraid are you on?

[Masterwishx]

56 minutes ago, Rysz said:

I can compile you a test package, what version of Unraid are you on?

6.12.11 , i tryed but missed slack-package.conf file, if you can will be awesome 

 

Edited July 29 by Masterwishx

[knights_of_pine]

On 7/27/2024 at 9:27 AM, Rysz said:

 

I'm very sorry, I really have zero clue why this doesn't work for you.

It works for me on my testing server and also for many other users who use these filters. 😕 

i notice my comments arent "green" but "white" for the filers i added.  could this be that they are not currently in affect?  i copied them exactly but maybe something else...?

 

 

[killua]

@Rysz
Can we have a setting to spin up drives 1 by 1 rather than the whole array?

In a way do a "simulated" staggered spinup by reading each disk when UPS battery powered is activated. Like how plex only spins up the video file on that 1 particular disk and not the whole array.

Current problem,

1. 12 drives spun DOWN
2. Power goes out
3. UPS Online and now array uses battery powered
4. Shutdown initiated
5. [PROBLEM HERE!!] All 12 disk spins up all AT THE SAME TIME
6. UPS chokes due to high power spike and shuts down the server prematurely RIP
7. A mini heart attack followed by praying my data is all fine

As you can see this can be avoided by spinning up the disks 1 by 1, it doesn't take long at all probably 30 seconds per drive to be safe WHEN SHUTDOWN IS INITIATED. Please incorporate this!!

Edited July 30 by killua

[Rysz]

3 hours ago, killua said:

@Rysz
Can we have a setting to spin up drives 1 by 1 rather than the whole array?

In a way do a "simulated" staggered spinup by reading each disk when UPS battery powered is activated. Like how plex only spins up the video file on that 1 particular disk and not the whole array.

Current problem,

1. 12 drives spun DOWN
2. Power goes out
3. UPS Online and now array uses battery powered
4. Shutdown initiated
5. [PROBLEM HERE!!] All 12 disk spins up all AT THE SAME TIME
6. UPS chokes due to high power spike and shuts down the server prematurely RIP
7. A mini heart attack followed by praying my data is all fine

As you can see this can be avoided by spinning up the disks 1 by 1, it doesn't take long at all probably 30 seconds per drive to be safe WHEN SHUTDOWN IS INITIATED. Please incorporate this!!

 

Sorry this is outside of the scope of NUT (and UPS software in general). All NUT does is invoke the operating system's shutdown command when the respective shutdown conditions are met. How the OS chooses the handle the shutdown procedure itself is up to the operating system. In any case, one should always dimension the UPS to be able to sustain the target system at least under maximum load, if it cannot sustain the system under maximum load then it will likely also not provide much "on battery" time under regular load - which is a problem in itself. The solution you are asking for is a band-aid for the problem of a too small UPS for a too large system and not something that I want to add as it masks that actual problem - it would be better to purchase a sufficiently dimensioned UPS.

[thecode]

3 hours ago, killua said:

 

As you can see this can be avoided by spinning up the disks 1 by 1, it doesn't take long at all probably 30 seconds per drive to be safe WHEN SHUTDOWN IS INITIATED. Please incorporate this!!

You need a larger UPS, I would also check the batteries, when they are at end of life the UPS will not hold it full load capacity.

I don't see any other solution that will work as you can't control what is on and what is not. Power failure can happen during parity check which has all the disks running anyhow.

Spinning the disks at shutdown is how all OSes works.

[Kilrah]

4 hours ago, killua said:

All 12 disk spins up all AT THE SAME TIME

 

 

4 hours ago, killua said:

it doesn't take long at all probably 30 seconds per drive

 

12x 30 seconds = 6 mins, which is already longer than a UPS that's loaded close enough for drive spinups to matter would be expected to be able to hold.

Edited July 30 by Kilrah

[Rysz]

On 7/29/2024 at 6:36 PM, knights_of_pine said:

i notice my comments arent "green" but "white" for the filers i added.  could this be that they are not currently in affect?  i copied them exactly but maybe something else...?

 

 

 

No, that's just the editor's syntax highlighting which is different in light and dark modes respectively.

[killua]

12 hours ago, Rysz said:

 

Sorry this is outside of the scope of NUT (and UPS software in general). All NUT does is invoke the operating system's shutdown command when the respective shutdown conditions are met. How the OS chooses the handle the shutdown procedure itself is up to the operating system. In any case, one should always dimension the UPS to be able to sustain the target system at least under maximum load, if it cannot sustain the system under maximum load then it will likely also not provide much "on battery" time under regular load - which is a problem in itself. The solution you are asking for is a band-aid for the problem of a too small UPS for a too large system and not something that I want to add as it masks that actual problem - it would be better to purchase a sufficiently dimensioned UPS.

I've tested plugging out wall power when all 12 drives are in "SPUN UP" state. It ran for 20 minutes and 50% of the UPS's power before I plugged it back in. So its definitely the spike in power that's causing my UPS to trip.
Anyway to have custom shutdown sequence?

 

12 hours ago, thecode said:

You need a larger UPS, I would also check the batteries, when they are at end of life the UPS will not hold it full load capacity.

I don't see any other solution that will work as you can't control what is on and what is not. Power failure can happen during parity check which has all the disks running anyhow.

Spinning the disks at shutdown is how all OSes works.

This is not true, tested fine when I unplug wall power when 12 disks were in "SPUN UP" state for 20 minutes and it only took 50% of the UPS's power.  So the batteries are fine, its the sudden surge of current that's causing it to fail (12 hdd spun up at once)

 

12 hours ago, Kilrah said:

 

 

 

12x 30 seconds = 6 mins, which is already longer than a UPS that's loaded close enough for drive spinups to matter would be expected to be able to hold.

I tested my ups by unplugging it while 12 disk were in "SPUN UP" state for 20 minutes, disabled spin down. 20 Minutes only took 50% of the UPS capacity. So 6 minutes is nothing to wait for every disk to spin up.
Do you have an unraid plugin or something to change the shutdown script? I want to turn it on 1 by 1

[Rysz]

31 minutes ago, killua said:

I've tested plugging out wall power when all 12 drives are in "SPUN UP" state. It ran for 20 minutes and 50% of the UPS's power before I plugged it back in. So its definitely the spike in power that's causing my UPS to trip.
Anyway to have custom shutdown sequence?

 

Sorry but this still makes no sense to me. You say that your UPS is dimensioned large enough to sustain 20 minutes of your server's maximum load and only lose 50% of its battery charge in the process. If it's able to hold the server's maximum load for 20 minutes, that is likely nowhere near the UPS rated maximum load... otherwise the UPS wouldn't be able to hold it for that long (20 minutes is a very long time in a high utilization scenario). So if the UPS is in fact dimensioned large enough to be able to hold the server's maximum load for 20 minutes and still have 50% battery charge remaining, it certainly shouldn't care for the increase from the server's regular load to the server's maximum load even if within a short time-frame. If it does care about that spike from e.g. 20% utilization to e.g. 50% utilization then this is a hardware issue of the UPS.

 

Such a feature would be a band-aid to cloak a UPS hardware issue at best, and not a good one at that, so I'm not going to add this to the plugin with it doing more damage than good. As others have pointed out, such a problem is not limited to shutdowns... a full disk spin-up can occur at any time (parity check, containers scanning files, search indexing, ...) and you can't plan for that. There's no point in having a UPS if it crashes your system anytime a full disk spin-up occurs in an on battery situation, which might well happen before the shutdown (especially if you have your UPS running for 20 minutes). A staggered spin-up at shutdown won't prevent anything else causing a spin-up.

[Masterwishx]

On 7/29/2024 at 1:36 PM, Rysz said:

 

I can compile you a test package, what version of Unraid are you on?

In case you will have no time for compile, I can try compile myself if you can provide slack-package.conf. 

[killua]

1 hour ago, Rysz said:

 

Sorry but this still makes no sense to me. You say that your UPS is dimensioned large enough to sustain 20 minutes of your server's maximum load and only lose 50% of its battery charge in the process. If it's able to hold the server's maximum load for 20 minutes, that is likely nowhere near the UPS rated maximum load... otherwise the UPS wouldn't be able to hold it for that long (20 minutes is a very long time in a high utilization scenario). So if the UPS is in fact dimensioned large enough to be able to hold the server's maximum load for 20 minutes and still have 50% battery charge remaining, it certainly shouldn't care for the increase from the server's regular load to the server's maximum load even if within a short time-frame. If it does care about that spike from e.g. 20% utilization to e.g. 50% utilization then this is a hardware issue of the UPS.

Hmm, let me help you make some sense to it.
When HDD spins up it takes 2A each for about 30 seconds.
After spinning, it runs 0.1A-0.3A. That 30 seconds is the main problem. 
This video sums it up, 

All drives spin up at THE SAME TIME 1215W for a split second.

 

Staggered spin up doesn't exceed 200W

As you can see, my UPS can't handle that much current even for a split second. BUT it can sustain loads for long periods.
I.e like this example, the 200W "staggered" spin up can sustain 20 mins for 50% no problems.
But when there's large current draw for a split second, UPS says NOPE and just dies.

I'd like to remind you, booting from wall power from cold boot isn't an issue since there's plenty of power from MAINS, its the UPS battery/inverter that can't handle that much surge of current.
This bandaid you speak of, how do I do that?

 

 

1 hour ago, Rysz said:

 

As others have pointed out, such a problem is not limited to shutdowns... a full disk spin-up can occur at any time (parity check, containers scanning files, search indexing, ...) and you can't plan for that. There's no point in having a UPS if it crashes your system anytime a full disk spin-up occurs in an on battery situation, which might well happen before the shutdown (especially if you have your UPS running for 20 minutes). A staggered spin-up at shutdown won't prevent anything else causing a spin-up.

Could you please please understand, that 20 minutes test already SPUN UP, ALL DISK ARE SPINNING when I unplug wall power. There's no spin up surge anymore, disk are already spinning, means each disk only takes 0.3A max. So there's no problems there.

The only problem I'm highlighting that you seem to not understand is when all disk are IDLE/SPUN DOWN when UPS is triggered. 

Full disk spin-up ALWAYS happen before shutdown when the disk is originally spun down.

 

Edited July 31 by killua

[Rysz]

1 hour ago, killua said:

Hmm, let me help you make some sense to it.
When HDD spins up it takes 2A each for about 30 seconds.
After spinning, it runs 0.1A-0.3A. That 30 seconds is the main problem. 
This video sums it up, 

All drives spin up at THE SAME TIME 1215W for a split second.

 

Staggered spin up doesn't exceed 200W

As you can see, my UPS can't handle that much current even for a split second. BUT it can sustain loads for long periods.
I.e like this example, the 200W "staggered" spin up can sustain 20 mins for 50% no problems.
But when there's large current draw for a split second, UPS says NOPE and just dies.

I'd like to remind you, booting from wall power from cold boot isn't an issue since there's plenty of power from MAINS, its the UPS battery/inverter that can't handle that much surge of current.
This bandaid you speak of, how do I do that?

 

 

Could you please please understand, that 20 minutes test already SPUN UP, ALL DISK ARE SPINNING when I unplug wall power. There's no spin up surge anymore, disk are already spinning, means each disk only takes 0.3A max. So there's no problems there.

The only problem I'm highlighting that you seem to not understand is when all disk are IDLE/SPUN DOWN when UPS is triggered. 

Full disk spin-up ALWAYS happen before shutdown when the disk is originally spun down.

 

 

I understand what you are saying: Your UPS is not able to sustain the peak load of all drives spinning up, but is able to sustain the load of all drives already spinning. But this is exactly the problem I was talking about, the maximum load you need to scale your UPS for is the actual maximum load that is theoretically possible on the target system and not anything below that.

 

Let's take the example you have given: If you must spin down your disks, you need a UPS to support 1215W and not 197W, even if it only reaches the 1215W during spin-ups. You simply cannot rely on your drives not spinning up all at once randomly and for whatever reason during battery operation.

 

Even if I were to implement the feature you are asking for, your UPS would still crash if literally anything else caused a spin-up of all disks while it is on battery power - there's just no point to only implement this in NUT. If you really want that, you better ask for a OS feature that allows staggered spin-ups if anything causes a spin-up of all disks during runtime, and not just NUT shutting down the server (which is in fact the OS shutting down the server, as NUT just sends the regular shutdown command).

 

No offence, you need a UPS that can support what you are calling a "surge" but is actually just the maximum load that you should have scaled your UPS for. You are running 12 disks and you are already running into this problem, how do you think some of the users with up to 60 disks are using their UPS and NUT without this feature you're asking for? This literally is the first time I'm hearing about this problem.

 

P.S.: A staggered spin-up only makes sense like that (as in the video above) if you set it up in your BIOS or HBA BIOS and keep the drives always spinning afterwards. Then you'd theoretically be able to run with a smaller UPS, but it's still bad practice in my opinion. This should only be for cases where the PSUs are not able to support the peak load of all drives spinning up at boot time and then keep spinning afterwards, as spin-ups are just not a well controllable situation within the OS (unless the drives are always spinning).

[Rysz]

On 7/29/2024 at 1:24 PM, Masterwishx said:

6.12.11 , i tryed but missed slack-package.conf file, if you can will be awesome 

 

 

Sorry I didn't forget about you, I'm just not near the computer I'm compiling on at the moment.

I'll try to compile you a package or provide you the configuration file this afternoon hopefully, sorry for the wait.

[Masterwishx]

17 minutes ago, Rysz said:

 

Sorry I didn't forget about you, I'm just not near the computer I'm compiling on at the moment.

I'll try to compile you a package or provide you the configuration file this afternoon hopefully, sorry for the wait.

 

its OK , i can wait or try compile myself , just was waiting for a long time this fix for my UPS , so hoping added code will fix the little  issues for it.  

[Rysz]

2 hours ago, Masterwishx said:

 

its OK , i can wait or try compile myself , just was waiting for a long time this fix for my UPS , so hoping added code will fix the little  issues for it.  

 

Here is the pull request #2562 compiled into a package:

nut-2.8.2-x86_64-1eapr.txz

 

1) Stop NUT in NUT Settings

2) Run this command:

removepkg nut-2.8.2-x86_64-1stable.ssl11.txz

3) Put the testing file onto the system somewhere

4) Run this command in the folder where you put the testing file:

installpkg nut-2.8.2-x86_64-1eapr.txz

5) Start NUT in NUT Settings

6) Confirm backend now shows as "nut-2.8.2-x86_64-1eapr" in NUT Settings

[Masterwishx]

17 minutes ago, Rysz said:

Here is the pull request #2562 compiled into a package:

 

Wow thanks a lot, it seems fixed the issues i had  will report to NUT devs, if something i will let you know ...

[Kilrah]

4 hours ago, killua said:

I'd like to remind you, booting from wall power from cold boot isn't an issue since there's plenty of power from MAINS, its the UPS battery/inverter that can't handle that much surge of current.

But say you lose power, before NUT initiates shutdown something else fires up all drives, you're toast. As mentioned several things can make drives spin up.

 

You really need a UPS that can handle the load of your setup in all cases, otherwise you might as well not have one since the whole point is to have a power supply system that's as reliable as possible. 

Edited July 31 by Kilrah

[Rysz]

On 7/30/2024 at 5:13 PM, killua said:

@Rysz
Can we have a setting to spin up drives 1 by 1 rather than the whole array?

In a way do a "simulated" staggered spinup by reading each disk when UPS battery powered is activated. Like how plex only spins up the video file on that 1 particular disk and not the whole array.

Current problem,

1. 12 drives spun DOWN
2. Power goes out
3. UPS Online and now array uses battery powered
4. Shutdown initiated
5. [PROBLEM HERE!!] All 12 disk spins up all AT THE SAME TIME
6. UPS chokes due to high power spike and shuts down the server prematurely RIP
7. A mini heart attack followed by praying my data is all fine

As you can see this can be avoided by spinning up the disks 1 by 1, it doesn't take long at all probably 30 seconds per drive to be safe WHEN SHUTDOWN IS INITIATED. Please incorporate this!!

 

On another note, just because I won't add this to NUT, doesn't mean you can't do it with NUT yourself.

You can always modify the SHUTDOWNCMD or better NOTIFYCMD to point to a custom script that does this.

https://networkupstools.org/docs/man/ups.conf.html

[Masterwishx]

@Rysz Sorry to ask you again , but can you when will have time to compile or provide me config file for https://github.com/jimklimov/nut/tree/issue-2562-retry-NULL-IDs ?

Thanks

[Rysz]

9 hours ago, Masterwishx said:

@Rysz Sorry to ask you again , but can you when will have time to compile or provide me config file for https://github.com/jimklimov/nut/tree/issue-2562-retry-NULL-IDs ?

Thanks

 

nut-2.8.2-x86_64-2eapr.txz

1) Stop NUT in NUT Settings

2) Run this command

(if you still have the testing package installed):

removepkg nut-2.8.2-x86_64-1eapr.txz

or (if you don't have the testing package installed anymore):

removepkg nut-2.8.2-x86_64-1stable.ssl11.txz

3) Put the testing file onto the system somewhere

4) Run this command in the folder where you put the testing file:

installpkg nut-2.8.2-x86_64-2eapr.txz

5) Start NUT in NUT Settings

6) Confirm backend now shows as "nut-2.8.2-x86_64-2eapr" in NUT Settings

[killua]

22 hours ago, Rysz said:

You simply cannot rely on your drives not spinning up all at once randomly and for whatever reason during battery operation.

 

18 hours ago, Kilrah said:

But say you lose power, before NUT initiates shutdown something else fires up all drives, you're toast. As mentioned several things can make drives spin up.

I don't see how this is possible, when power goes out, unraid instantly does these 3 processes. So there's no "unexpected spin up". Its all expected.



How do i change "emttpd: Spinning up all drives" to "emttpd: Spinning drives 1 by 1".
Or better yet have Unraid kernel mdcmd spinup with a 30 second delay.

 

22 hours ago, Rysz said:

Let's take the example you have given: If you must spin down your disks, you need a UPS to support 1215W and not 197W, even if it only reaches the 1215W during spin-ups. You simply cannot rely on your drives not spinning up all at once randomly and for whatever reason during battery operation.

I think I know what's the issue here, GEL BATTERY. My UPS uses Gel battery instead of lead-acid. It has long shelf life, 10-20 years vs 5 years (lead-acid) but with the cost of having low "SURGE" current.
Please do an option (staggered spinup) for us that uses Gel batteries!!

Edited August 1 by killua

[Masterwishx]

3 hours ago, Rysz said:

 

nut-2.8.2-x86_64-2eapr.txz 1.38 MB · 0 downloads

1) Stop NUT in NUT Settings

2) Run this command

(if you still have the testing package installed):

removepkg nut-2.8.2-x86_64-1eapr.txz

or (if you don't have the testing package installed anymore):

removepkg nut-2.8.2-x86_64-1stable.ssl11.txz

3) Put the testing file onto the system somewhere

4) Run this command in the folder where you put the testing file:

installpkg nut-2.8.2-x86_64-2eapr.txz

5) Start NUT in NUT Settings

6) Confirm backend now shows as "nut-2.8.2-x86_64-2eapr" in NUT Settings

Many Times Thanks