The UNRAID OS and community is amazing, anything is possible and it's really easy "sometimes" to figure some stuff out but not this time.

I have a Nvidia Quadro P2000 GPU and it has a max TDP of 70W and in UNRAID OS it idles at 20W (load = 1 - 2%) with Power State P0 all the time.

 

So in conclusion with Spaceinvader One's video on improving power usage the only solutions for low power usage at idle are:

• GUI Mode with no VM's activated, only use for transcoding.
• Headless Mode when being passed through to a VM that is idle.

 

I read somewhere that could force the power state from P0 to P8 for idle to lower the usage but how will that react when for example Plex Transcoding is activated?

• Will the Power state change so that the GPU actually transcodes?
• Or will it remain in P8 state because it has been forced and no transcodes?

 

The reason for this post is obvious, I want to lower the idle power usage from 20W to 6-10W

 

Best regards,

 

 

Edited June 4, 20215 yr by SimpleDino

The power state has is solved. I applied this solution with some help from the Unraid Reddit forum:

This solution changes the power state from P0 to P8 and 20W --> 8W. The P-state will change if the GPU is being passed through for transcoding or VMs. But the solution is temporary, it will reset and not persist after a reboot. Type in console/Terminal:

 

• This solution changes the power state from P0 to P8 and 20W --> 8W. The P-state will change if the GPU is being passed through for transcoding or VMs. But the solution is temporary, it will reset and not persist after a reboot.

  • Type in console/Terminal:

    • nvidia-smi --persistence-mode=1

 

• Solution #2. To make it permanent.
  • Download the "User Scripts" plugin.
  • make a new script to run at:
    • "First Array Start Only"
  • and put this into it:
    • #!/bin/bash #nvidia-smi --persistence-mode=1 nvidia-persistenced fuser -v /dev/nvidia*

For anyone looking for this in the future, I had to modify the formatting of the script to be the following.

#!/bin/bash
nvidia-smi --persistence-mode=1
nvidia-persistenced
fuser -v /dev/nvidia*

@SimpleDino and @SpaceInvaderOne Can you all confirm on a system on 6.10.0 RC4?

On 3/30/2022 at 12:00 AM, AnnabellaRenee87 said:

For anyone looking for this in the future, I had to modify the formatting of the script to be the following.

#!/bin/bash
nvidia-smi --persistence-mode=1
nvidia-persistenced
fuser -v /dev/nvidia*

@SimpleDino and @SpaceInvaderOne Can you all confirm on a system on 6.10.0 RC4?

 

I have not upgraded yet to RC4, still on RC2 Maybe @SpaceInvaderOne can confirm?

 

Br,

18 months from the OP, this thread just fixed my high idle power for a P2000 in Version: 6.11.5

~20W P0 to 7W P8

 

Thanks!

 

On 3/29/2022 at 5:00 PM, AnnabellaRenee87 said:

For anyone looking for this in the future, I had to modify the formatting of the script to be the following.

#!/bin/bash
nvidia-smi --persistence-mode=1
nvidia-persistenced
fuser -v /dev/nvidia*

@SimpleDino and @SpaceInvaderOne Can you all confirm on a system on 6.10.0 RC4?

Confirmed working on Unraid 6.11.5, got my Quadro t1000 down from max clock speed to 300 on core and memory! Temps are down as well from 42 to 35 degree. 

2 minutes ago, Qwenton said:

Confirmed working on Unraid 6.11.5, got my Quadro t1000 down from max clock speed to 300 on core and memory! Temps are down as well from 42 to 35 degree. 

Please don't use "nvidia-smi --persistence-mode=1" anymore, only use "nvidia-persistenced"!

12 hours ago, ich777 said:

Please don't use "nvidia-smi --persistence-mode=1" anymore, only use "nvidia-persistenced"!

I will get this changed, thank you

12 hours ago, ich777 said:

Please don't use "nvidia-smi --persistence-mode=1" anymore, only use "nvidia-persistenced"!

New script looking like this then?

#!/bin/bash
nvidia-smi
nvidia-persistenced
fuser -v /dev/nvidia

 

2 hours ago, Qwenton said:

New script looking like this then?

You don‘t need nvidia-smi at all.

On 2/21/2023 at 3:57 PM, ich777 said:

Please don't use "nvidia-smi --persistence-mode=1" anymore, only use "nvidia-persistenced"!

@ich777 Running 7.0.0 I get an error with these lines in my script. If I use nvidia-smi --persistence-mode=1 I do not get that error.

 

Error with nvidia-persistenced:

script location: /tmp/user.scripts/tmpScripts/Gpu idle/script
Note that closing this window will abort the execution of this script
nvidia-persistenced failed to initialize. Check syslog for more details.
USER PID ACCESS COMMAND
/dev/nvidia-modeset: root 13191 F.... nvidia-persiste
/dev/nvidia0: root 13191 F.... nvidia-persiste
/dev/nvidiactl: root 13191 F.... nvidia-persiste

 

No error with this line nvidia-smi --persistence-mode=1:

Script location: /tmp/user.scripts/tmpScripts/Gpu idle/script
Note that closing this window will abort the execution of this script
Persistence mode is already Enabled for GPU 00000000:04:00.0.
All done.
USER PID ACCESS COMMAND
/dev/nvidia-modeset: root 13191 F.... nvidia-persiste
/dev/nvidia0: root 13191 F.... nvidia-persiste
/dev/nvidiactl: root 13191 F.... nvidia-persiste

 

 

Edited January 29, 20251 yr by TallMan206

31 minutes ago, TallMan206 said:

Check syslog for more details.

Please post your Diagnostics since I can't tell why it's not working.

 

However you won't need such a script, it is enough to issue `nvidia-persistenced` on boot once since this is a daemon that will run in the background no matter what.

 

It should be basically be enough to just put `nvidia-persistenced` in your go file after the emhttp line.

49 minutes ago, ich777 said:

Please post your Diagnostics since I can't tell why it's not working.

 

However you won't need such a script, it is enough to issue `nvidia-persistenced` on boot once since this is a daemon that will run in the background no matter what.

 

It should be basically be enough to just put `nvidia-persistenced` in your go file after the emhttp line.

That worked. Removed the script, and added the line to the GO file. Thanks

So i can remove this whole script and just run 'nvidia-persistenced' in GO for my GPU to power save now?

 

#!/bin/bash
# check for driver
command -v nvidia-smi &> /dev/null || { echo >&2 "nvidia driver is not installed you will need to install this from community applications ... exiting."; exit 1; }
echo "Nvidia drivers are installed"
echo
echo "I can see these Nvidia gpus in your server"
echo
nvidia-smi --list-gpus
echo
echo "-------------------------------------------------------------"
# Start NVIDIA persistence daemon
if ! pgrep -f "nvidia-persistenced" > /dev/null; then
    echo "Starting nvidia-persistenced..."
    nvidia-persistenced --verbose
else
    echo "nvidia-persistenced is already running"
fi
# Query power state
gpu_pstate=$(nvidia-smi --query-gpu="pstate" --format=csv,noheader);
# Query running processes by pid using GPU
gpupid=$(nvidia-smi --query-compute-apps="pid" --format=csv,noheader);
# Check if pstate is zero and no processes are running by checking if any pid is in string
if [ "$gpu_pstate" == "P0" ] && [ -z "$gpupid" ]; then
echo "No pid in string so no processes are running"
fuser -kv /dev/nvidia*
echo "Power state is"
echo "$gpu_pstate" # show what power state is
else
echo "Power state is"
echo "$gpu_pstate" # show what power state is
fi
echo
echo "-------------------------------------------------------------"
echo
echo "Power draw is now"
# Check current power draw of GPU
nvidia-smi --query-gpu=power.draw --format=csv
exit

 

Is there a cleaner way to do this i.e other than using killall?

It works fine, just feel killall is a bit OTT

 

<?php
if ($argv[2] == 'prepare'){
        shell_exec('date +"%b %d %H:%M:%S libvirt hook: killall nvidia-persistenced" >> /var/log/syslog');
        shell_exec('killall nvidia-persistenced');
}

if ($argv[2] == 'release' && $argv[3] == 'end'){
        shell_exec('date +"%b %d %H:%M:%S libvirt hook: start nvidia-persistenced" >> /var/log/syslog');
        shell_exec('nvidia-persistenced');
}

 

Interestingly powertop shows the  GPU as bad under tunables even though I can see it's in P8 after the VM closes.

 

>> Bad           VM writeback timeout                                                                                   
   Bad           Runtime PM for I2C Adapter i2c-12 (NVIDIA i2c adapter 2 at 1:00.0)
   Bad           Runtime PM for I2C Adapter i2c-14 (NVIDIA i2c adapter 4 at 1:00.0)
   Bad           Runtime PM for I2C Adapter i2c-16 (NVIDIA i2c adapter 6 at 1:00.0)
   Bad           Runtime PM for I2C Adapter i2c-13 (NVIDIA i2c adapter 3 at 1:00.0)
   Bad           Runtime PM for I2C Adapter i2c-15 (NVIDIA i2c adapter 5 at 1:00.0)
   Bad           Runtime PM for PCI Device NVIDIA Corporation AD107 High Definition Audio Controller
   Bad           Runtime PM for PCI Device NVIDIA Corporation AD107 [GeForce RTX 4060]

 

Have ended up with this to sort that out

 

if ($argv[2] == 'prepare'){
        shell_exec('date +"%b %d %H:%M:%S libvirt hook: killall nvidia-persistenced" >> /var/log/syslog');
        shell_exec('killall nvidia-persistenced');
}

if ($argv[2] == 'release' && $argv[3] == 'end'){
        shell_exec('date +"%b %d %H:%M:%S libvirt hook: start nvidia-persistenced" >> /var/log/syslog');
        shell_exec('nvidia-persistenced');
        shell_exec('date +"%b %d %H:%M:%S libvirt hook: Runtime PM for PCI device" >> /var/log/syslog');
        shell_exec('echo auto | tee /sys/bus/pci/devices/????:??:??.?/power/control');
}

 

Actually the above didn't work too well. The shell_exec of nvidia-persistenced in /etc/libvirt/hooks/qemu kept a grip on the drive my libvirt.img resides on (cache) and prevented the array stopping, should I need it to. I had to killall nvidia-persistenced and umount /dev/loop3 as per losetup output

 

Have ended up with this is script in /etc/libvirt/hooks/qemu.d/ directory. Which works well. nvidia-persistenced stops and starts when  start/ stop a VM. GPU goes back to P8 when the VM is stopped.

 

CPU C states never seem to recover though. CPU can drop to C6 after a reboot/ fresh start of unraid. After start/ stopping a VM, CPU only drops to C2. Not the end of the world..

 

#!/bin/bash

if [ "$2" == "prepare" ]
then
killall nvidia-persistenced
elif [ "$2" == "release" ]
then
nvidia-persistenced

echo auto | tee /sys/bus/pci/devices/????:??:??.?/power/control

fi

exit 0 

 

 

i have always used this script :

 

#!/bin/bash
tmp=$(virsh list --all | grep " vmtest " | awk '{ print $3}')
if ([ "x$tmp" == "x" ] || [ "x$tmp" != "xrunning" ])
then
    echo "VM does not exist or is shut down!"
    nvidia-smi -pm 1
else
    echo "VM is running!"
fi

 

worked for years for me , i trigger it  every 5 seconds.

On 2/4/2025 at 11:07 PM, fka said:

Actually the above didn't work too well. The shell_exec of nvidia-persistenced in /etc/libvirt/hooks/qemu kept a grip on the drive my libvirt.img resides on (cache) and prevented the array stopping, should I need it to. I had to killall nvidia-persistenced and umount /dev/loop3 as per losetup output

 

Have ended up with this is script in /etc/libvirt/hooks/qemu.d/ directory. Which works well. nvidia-persistenced stops and starts when  start/ stop a VM. GPU goes back to P8 when the VM is stopped.

 

CPU C states never seem to recover though. CPU can drop to C6 after a reboot/ fresh start of unraid. After start/ stopping a VM, CPU only drops to C2. Not the end of the world..

 

#!/bin/bash

if [ "$2" == "prepare" ]
then
killall nvidia-persistenced
elif [ "$2" == "release" ]
then
nvidia-persistenced

echo auto | tee /sys/bus/pci/devices/????:??:??.?/power/control

fi

exit 0 

 

 

Recently experienced Kernel panics relating to the audio part of the GPU when the VM exited and the GPU was handed back to the host

Apr 15 09:24:07 myunraid kernel: vfio-pci 0000:01:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=io+mem:owns=none
Apr 15 09:24:07 myunraid kernel: nvidia 0000:01:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=none:owns=none
Apr 15 09:24:07 myunraid kernel: [drm] [nvidia-drm] [GPU ID 0x00000100] Loading driver
Apr 15 09:24:10 myunraid kernel: [drm] Initialized nvidia-drm 0.0.0 for 0000:01:00.0 on minor 2
Apr 15 09:24:10 myunraid kernel: nvidia 0000:01:00.0: vgaarb: deactivate vga console
Apr 15 09:24:10 myunraid kernel: fbcon: nvidia-drmdrmfb (fb0) is primary device
Apr 15 09:24:10 myunraid kernel: BUG: kernel NULL pointer dereference, address: 000000000000001c
Apr 15 09:24:10 myunraid kernel: #PF: supervisor read access in kernel mode
Apr 15 09:24:10 myunraid kernel: #PF: error_code(0x0000) - not-present page
Apr 15 09:24:10 myunraid kernel: PGD 0 P4D 0 
Apr 15 09:24:10 myunraid kernel: Oops: Oops: 0000 [#1] PREEMPT SMP NOPTI
Apr 15 09:24:10 myunraid kernel: CPU: 7 UID: 0 PID: 2118002 Comm: qemu-event Tainted: P     U     O       6.12.54-Unraid #1
Apr 15 09:24:10 myunraid kernel: Tainted: P]=PROPRIETARY_MODULE, [U]=USER, [O]=OOT_MODULE
Apr 15 09:24:10 myunraid kernel: Hardware name: Micro-Star International Co., Ltd. MS-7E13/MAG B760M MORTAR WIFI II (MS-7E13), BIOS 1.B0 03/04/2026
Apr 15 09:24:10 myunraid kernel: RIP: 0010:nv_audio_dynamic_power+0x8f/0x110 [nvidia]
Apr 15 09:24:10 myunraid kernel: Code: c0 74 7c f6 80 a4 01 00 00 10 75 73 48 8b 80 40 01 00 00 48 85 c0 74 67 48 8b 90 a8 01 00 00 48 05 a0 01 00 00 48 39 c2 74 55 <83> 7a 1c 03 75 10 48 8b 7a 20 48 83 bf 40 03 00 00 00 75 08 eb 3f
Apr 15 09:24:10 myunraid kernel: RSP: 0018:ffffc9000b126b98 EFLAGS: 00010207
Apr 15 09:24:10 myunraid kernel: RAX: ffff8881404d39a0 RBX: 0000000000000000 RCX: ffff888101570bb8
Apr 15 09:24:10 myunraid kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881017fc100
Apr 15 09:24:10 myunraid kernel: RBP: ffff888161c85bf0 R08: ffff8881017fb000 R09: 00000000ffffffff
Apr 15 09:24:10 myunraid kernel: R10: 00000000ffffffff R11: ffffffffa65b06a8 R12: ffff888161c85c50
Apr 15 09:24:10 myunraid kernel: R13: ffffffffa17c3dc0 R14: ffff888345c36820 R15: ffff888161c85d50
Apr 15 09:24:10 myunraid kernel: FS:  0000152e70eff6c0(0000) GS:ffff88885f5c0000(0000) knlGS:0000000000000000 .........

I've created /boot/config/modprobe.d/nvidia.conf containing "options nvidia NVreg DynamicPowerManagement=0x00" without ""

I've appended boot option in SysLinux Configuration "nvidia_drm.fbdev=0" without ""

Amended the hooks script to

#!/bin/bash
if [ "$2" == "prepare" ]; then
    killall nvidia-persistenced 2>/dev/null
elif [ "$2" == "release" ]; then
    nvidia-persistenced
fi
exit 0

Now all is well in the world :)

Edited April 15Apr 15 by fka

See here. Seems this latest driver 595.58.03 is a fault

https://forums.unraid.net/topic/98978-plugin-nvidia-driver/page/229/#findComment-1619400