Unraid power usage higher than Windows


Mr.Will

Recommended Posts

Hi!,

I'm new to Unraid and testing it. My current needs are a NAS and a couple of VMs in one box, as well as my main Windows gaming rig. The gaming rig can be virtual or physical.

 

Now, I have tested 2 different approaches and checked their power consumption, and here are the findings:

- Unraid (bare metal) + drive spin down + Windows rig VMs OFF: 66 Watts (sustained after a few minutes idle)

- Unraid (bare metal) + drive spin down + Windows rig VMs ON: 90 Watts (sustained after a few minutes idle)

- Windows (bare metal) + drive spin down + VMs off: 46 Watts (sustained after a few minutes idle. While working on the PC it's aroung 55)

 

The Windows VM has a Nvidia RTX 3070 passthrough. It took me some tweaking but now it's not using any apparent power if the VM is off.

The lowest I achieved with Unraid was 66W after a little tweaking in the BIOS. Before that it was 77 normally.

 

So, basically, bare metal Unraid uses 66 Watts while bare metal Windows 11 Pro uses less than 50. Reading the forums I understand that 66W is actually a good measure, but with the prices of electricity being crazy in Europe right now, I want to lower it. And the thing is, I KNOW this computer can run at 46W because that's what Windows uses!

 

What can I do to make Unraid use the same power as Windows? I'm guessing Windows drivers are better optimised than those in Unraid, so this may be impossible.

If I'm unable to lower it I may use bare metal Windows with Hyper-v and virtualize my NAS in it, but then that NAS can't be Unraid since it's not supported in Hyper-v :(

 

Any thoughts? Attached is the diagnostics just in case.

Thank you!

 

unraid-diagnostics-20220105-1828.zip

Edited by Mr.Will
Attached diagnostics
Link to comment
1 hour ago, Mr.Will said:

The Windows VM has a Nvidia RTX 3070 passthrough. It took me some tweaking but now it's not using any apparent power if the VM is off.

The lowest I achieved with Unraid was 66W after a little tweaking in the BIOS. Before that it was 77 normally.

Are you already using nvidia-persistenced?

 

As long your dGPU is active, you will never reach the Windows value in unRAID.

 

It's in German, but the most recent thread I know of:

https://forums.unraid.net/topic/118039-strom-sparen-ansätze/?tab=comments#comment-1081006

 

 

 

Link to comment
24 minutes ago, primeval_god said:

 

 

I forgot to mention that I already played with powertop a little, and also tried the auto-tune option, but it didn't make much of a difference. I noticed though there are only C1, C2 and C3 states, and not above that. I tried changing some related option sin the BIOS but that didn't show other states (C7, etc.). Then I read that those states only exist when using battery, so I left that appart. Should I be seeing states above C3?

 

15 minutes ago, mgutt said:

Are you already using nvidia-persistenced?

 

As long your dGPU is active, you will never reach the Windows value in unRAID.

 

It's in German, but the most recent thread I know of:

https://forums.unraid.net/topic/118039-strom-sparen-ansätze/?tab=comments#comment-1081006

 

 

 

 

Wow, that's quite a post. I will have to get deep into that, and first of all learn how to run scripts :)

No, I don't use nvidia-persistenced AFAIK, but I will defenitely give this a go.

 

Thank you!

Link to comment
1 hour ago, mgutt said:

powertop does not support most recent CPUs, so sadly you don't have a valid option to check higher states. I'm still investigating how to get package c-states without powertop.

Good to know. Thank you, and good luck! :)
I have tried what you suggested (I hope I'm not missing something). Setting the
intel_pstate=passive seems to use about 6W less, so that's good.
However "nvidia-persistenced failed to initialize"). I had installed the nvidia driver plugin but since the card is assigned to vfio nvidia-smi doesn't work. I'm guessing "nvidia-persistenced" is the same as "nvidia-smi -mp 1" (?). Do you know how the people in the German post did it to be able to set the nvidia persistence even when the graphics card is passed through? (VM off).

 

I don't know if it's relevant but in order to passthrough the nvidia card without issues I had to also set unraid boot to legacy, and for that I had to enable CSM (compatibility support module) in my Asus board. CSM disables the onboard graphics option, so I can't set the CPU graphics as my main output. See.

 

I tried different configurations like changing my monitor to the onboard HDMI (which is blank after reboot) or not connecting anything, but the nvidia plugin in Unraid still can't talk to the card. However, if I power ON the VM now (cores still in powersave) I think the driver in the VM is kicking in and after a few minutes idle the usage goes down to 48W!

Is there any way to have this same result but without turning on the VM? For that I think Unraid must be able to talk to the card while on vfio.

Finally, I have seen a couple times the VM getting unresponsive for about 20 seconds. Not sure if this is related :( Thanks again!
 

Link to comment
12 hours ago, mgutt said:

Solve this. Legacy shouldn't be needed as your dGPU supports UEFI.

 

Understood. But how do you think I should do that? Without legacy passthrough didn't work well. The computer hanged every few seconds when I had the GPU HDMI audio passed to the VM. If I removed the audio device then it was fine.

I spent 2 days dealing with this and only when I set it to Legacy I was able to use the HDMI audio without the VM freezing all the time.

 

Thanks again for your help!

Edited by Mr.Will
Link to comment

One solution is to boot a Linux VM with Nvidia passthrough everytime you don't use the Windows VM.

 

This could be done by script like this:

if [[ ! $(virsh list | grep Windows) ]] && [[ ! $(virsh list | grep Ubuntu ) ]]; then
  virsh start Ubuntu
fi

 

Execute it every 10 minutes or similar and it should be fine. You would only need to shutdown the Linux VM before starting the Windows VM.

 

Or, but I'm unsure about this, maybe @ich777 can contribute, is to use the vfio bound Nvidia card inside a docker container with nvidia drivers installed. By that their is absolutely no VM overhead. The idea would be to start a privileged Container with various additional capabilities to access the PCie devices. Maybe CAP_SYS_RAWIO?

 

EDIT: Should be possible by simply passing the /dev/vfio/X device to the container:

https://gist.github.com/egernst/aefc7cc70ab633e2a76c48061ac43d54

 

 

Link to comment
7 hours ago, mgutt said:

Or, but I'm unsure about this, maybe @ich777 can contribute, is to use the vfio bound Nvidia card inside a docker container with nvidia drivers installed.

This isn't possible from what I know.

 

On 1/6/2022 at 2:35 AM, Mr.Will said:

Setting the intel_pstate=passive seems to use about 6W less, so that's good.
However "nvidia-persistenced failed to initialize"). I had installed the nvidia driver plugin but since the card is assigned to vfio nvidia-smi doesn't work. I'm guessing "nvidia-persistenced" is the same as "nvidia-smi -mp 1" (?).

One thing that I could imagine is that you don't bind the card to VFIO (the GPU should work in the VM anyways without being bound to VFIO with the newer Nvidia drivers) install the Nvidia Driver plugin and always launch nvidia-persistenced with a hook when the VM is off and kill it when the VM is started.

If you do that you are making sure that the card goes to P8 (lowest idle power state) when the VM is off.

 

But first I would recommend that you check if the card is working without the VFIO binding.

 

7 hours ago, mgutt said:

EDIT: Should be possible by simply passing the /dev/vfio/X device to the container:

https://gist.github.com/egernst/aefc7cc70ab633e2a76c48061ac43d54

I think the easier way is to install the Nvidia Driver plugin and create the hook files for the VM.

Link to comment
5 minutes ago, ich777 said:

you don't bind the card to VFIO

Sadly this does not work for him:

12 hours ago, Mr.Will said:

Without legacy passthrough didn't work well. The computer hanged every few seconds when I had the GPU HDMI audio passed to the VM. If I removed the audio device then it was fine.

I spent 2 days dealing with this and only when I set it to Legacy I was able to use the HDMI audio without the VM freezing all the time.

 

Do you think passing /dev/vfio/X to a container is possible or not?

 

 

 

Link to comment
1 hour ago, mgutt said:

Sadly this does not work for him:

Doesn't he mean the boot mode itself? CSM (Legacy) or UEFI and that he had switch over to CSM?

 

1 hour ago, mgutt said:

Do you think passing /dev/vfio/X to a container is possible or not?

I don't think so, never tried that, because such a container also would need to run as privileged because this container then needs control over the full system.

Link to comment
1 hour ago, ich777 said:

Doesn't he mean the boot mode itself? CSM (Legacy) or UEFI and that he had switch over to CSM?

 

I don't think so, never tried that, because such a container also would need to run as privileged because this container then needs control over the full system.


My fault. I made so many tests that I think I'm not explaining myself well (and possibly mixing stuff).

 

Basically:

UEFI boot without audio device = everything fine (but no audio)

UEFI + Nvidia HDMI audio = continuous VM freezes

CSM + Unraid legacy boot + Nvidia HDMI audio = everything almost fine. VM got frozen just once

 

In every case, passthrough was done with Nvidia assigned to vfio. Without it, if I remember correctly, the VM could not boot and the UNRAID log said the Nvidia device was busy. Maybe I was doing something else wrong, like not setting the CPU graphics card as the main one (?).

 

Do you think that it should work with UEFI + no vfio + Nvidia graphics plugin + CPU graphics as main in BIOS? If this makes sense I can try this when I get back home in a few days.

Link to comment
17 hours ago, Mr.Will said:

CSM + Unraid legacy boot + Nvidia HDMI audio = everything almost fine. VM got frozen just once

This should also work but I'm really not that familiar with 11th Gen, on 10th Gen it works just fine.

 

I've heard that if you boot with legacy that your iGPU is disabled, but I'm not really sure if that's actually the case.

Link to comment
5 hours ago, ich777 said:

I've heard that if you boot with legacy that your iGPU is disabled, but I'm not really sure if that's actually the case.

 

Yes. If I enable CSM in the motherboard (required to boot unraid in Legacy mode) then iGpu can't be set as default graphics.

 

23 hours ago, mgutt said:

It should.

 

I will try this again beginning next week and report back.

 

Thank you both!

Link to comment

I tried this and it's "working". I can even get as low as 42W with only Unraid running, which is great with my current hardware.

 

However, there are 2 drawbacks:

- With primary display set to CPU in motheraboard, if no monitor is pluged in the mobo HDMI it will ball back to the PCIE, which does have the monitor installed. The only solution I found was to plug the monitor to the mobo while booting unraid, and then change it to the PCIE. I think maybe getting one of those "HDMI dummy emulators" may work. Any ideas?

 

- The most problematic: the VM gets stuck constantly as soon as I boot it up, which is the same behabior I saw before. The VM get's totally irresponsive for a few minutes, and when it comes back it only lasts a few seconds until it freezes again. How can I fix this? The only way I found to make it work was to either remove the audio device or boot everything in legacy :( I tried changing the bus and slot to use the same on GPU and Audio and it doesn't make any difference. I also tried to put the audio device in MSI mode but same thing.

Edited by Mr.Will
MSI mode
Link to comment

Just to make it clearer, when I tried the MSI "fix" I followed this guide, except that using the registry I used the MSI tool instead. Originally I got MSI enabled for the GPU but not the GPU's audio. After the "fix" I got MSI enabled in both as shown by the lspci, but the problem still occurs.

I also tried changing the audio to the onboard device and the audio works, but the quality changes. For example, in games some devices sound different, so I need to make that HDMI audio work. Other things I tried are changing the bios from OMFV to SeaBios, Machine from Q35 to i440fx, and creating the VM from scratch. None worked. Guest VM Window's log doesn't give anything either. Nothing while the PC is frozen.

 

Any ideas @ich777 or @mgutt?

Link to comment
  • 2 weeks later...

I found out that I can make the system work without aparent freezes just assigning 6 cores or less. If I add more cores then it just freezes constantly. My CPU is an intel 11600K  with 12 cores. 

Any ideas? I'm desperate! @ich777 @mgutt

 

Of course, I have core isolation stablished for all the cores I try to assign to the VM. And the cores I assign are never used for anything else, just this VM. Also, I always try to add consecutive cores and threads.

Link to comment
9 minutes ago, Mr.Will said:

If I add more cores then it just freezes constantly.

Which cores do you assign? Your CPU has 6 usual and 6 HT cores. You are not allowed to assign HT cores alone.

 

 

And do not isolote Core 0 and HT 6. The first core pair should be used by Unraid.

 

Link to comment

In the end it's not a matter of the number of cores, because with less cores it's happening also, although much much less frequently. However I think I found the issue.

I'm passing through an NVME and a RTX 3070 graphics card with it's audio. If I pass through the NVME and the audio device, the VM gents stuck. If I only pass through the NVME and not the audio, it's fine. Likewise, if I pass the audio and not the NVME it's fine as well. So basically it can't handle having both NVME and graphics card audio at the same time.

 

I don't know why this happens, but using a normal virtual disk inside the NVME works just fine. My suspicion is that my 11th gen Intel i5 has some sort of common PCIE bus for these devices that just doesn't work well with Unraid.

 

I spent almost a month trying to fix this issue and I'm quite angry at Unraid right now. I hope to recover the love after spinning a few containers and VMs. :)

I hope this helps others.

Link to comment
  • 8 months later...
On 1/5/2022 at 8:21 PM, mgutt said:

Are you already using nvidia-persistenced?

 

As long your dGPU is active, you will never reach the Windows value in unRAID.

 

It's in German, but the most recent thread I know of:

https://forums.unraid.net/topic/118039-strom-sparen-ansätze/?tab=comments#comment-1081006

 

 

 

 

@mgutt After a few months of this thread I wanted to share that I created my own user script that simply calls nvidia-persitenced when Unraid is started, and it modifies the qemu file to disable persistenced when a specific VM (Windows with passed through the nvidia card) is started. When the VM is off it enables persitenced again.

With this I managed to get quite some energy savings, from 100W to around 45W.

 

I'm thinking on converting this to a plugin so others can benefit from it, but I'm not sure if this solution I took maybe was overkill, or it would really benefit the comunity.

What's you opinion? Also interested in @ich777's feedback too.

I'm not sure if what I'm doing should be done automatically by the system, enabling nvidia-persistenced after some idle time and releasing it when we start the VM using the card. From my tests, if I try to start the VM without killing nvidia-persistenced before it just erros out.

 

Thanks!

Edited by Mr.Will
Link to comment
2 minutes ago, Mr.Will said:

 

@mgutt After a few months of this thread I wanted to share that I created my own user script that simply calls nvidia-persitenced when Unraid is started, and it modifies the qemu file to disable persistenced when a specific VM (Windows with passed through the nvidia card) is called. When the VM is off it enables persitenced again.

With this I managed to get quite some energy savings, from 100W to around 45W.

 

I'm thinking on converting this to a plugin so others can benefit from it, but I'm not sure if this solution I took maybe was overkill, or it would really benefit the comunity.

What's you opinion? Also interested in @ich777's feedback too.

I'm not sure if what I'm doing should be done automatically by the system, enabling nvidia-persistenced after some idle time and releasing it when we start the VM using the card. From my tests, if I try to start the VM without killing nvidia-persistenced before it just erros out.

 

Thanks!

How are you doing the change to qemu? from 6.10 you can create a file in qemu.d.

 

Here is the code I have for my plugin for usb devices.

 

 

https://github.com/SimonFair/USB_Manager/blob/main/source/scripts/install.sh

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.