[Solved] vfio: Unable to power on device, stuck in D3

[PeteyBoPetey]

Hi Guys,

 

Extremely frustrated and at wits end. VM wont output video. All 3 VMs were working on Unraid 6.9 but can't get them to work on 6.10. 
 

• Edited xml with multifunction='0n' and put all 3 functions into the same slot
• Download vbios from techpowerup and removed the Nvidia header

 

It is the primary gpu and it appears as though unraid is releasing it when I start the VM, but I'm not sure?. Do I need to edit the syslinix with apend....... to free up the gpu?. Is it a vbios problem? I'm at a loss. I need to get this up and running for my university studies. It's been down 3 weeks now :/.

 

UPDATE: The BIOS update included the addition of APM Advanced Power Management. The primary GPU would not be picked up as primary GPU and would shut down after two minutes and give a 'could not power on, stuck in D3' error which led me to believe it could be the power saving feature powering down the primary GPU. So I made sure APM was set to disabled, but it didn't fix the problem. I was about to pull the GPU out and send it back under warranty, as a last hail Mary I set APM to enable. F*ck me, everything is working!?. WTF??? It doesn't make sense?. You'd think that setting APM to disable would leave the GPU on all the time by default?.

tower-diagnostics-20220616-1349.zip

Edited July 22, 2022 by PeteyBoPetey

[PeteyBoPetey]

Found THIS thread relating to the "failing to mmap" but I don't understand it or the msi stuff?

[PeteyBoPetey]

Resolved by editing syslinux as shown HERE

[PeteyBoPetey]

Jumped the gun. VM 1 is working. VM 2 displays a screen but freezes and VM 3 unable to power on stuck in D3. I'm ready to give up. I'm at a loss?

[PeteyBoPetey]

UPDATE:

 

I searched google using "vfio unable to power on, stuck in D3" and there are numerous posts on numerous virt forums with the same problem that are all solved with a bios update. Now, I can't see how an upgrade from Unraid 6.9.2 to 6.10.3 can change something that would require a motherboard bios update. In the outback in Australia we have a saying "If it ain't broke, don't try and fix it". So I was very reluctant to touch my bios because, even though a lot of people have solved the problem with a bios update, I really couldn't see the connection or as a solution. Plus, a bios update has the potential to create more problems, on top of the problems I already have.

Updated the motherboard bios and now Unraid boots properly (sometimes it would not detect and boot from USB) and faster. In the past it would search all 3 nvme and 6 sata drives for a boot record before going to the Unraid USB, now it just goes straight the Unraid USB. Boots nearly as fast as a windows 10 on a nvme drive (I haven't enabled fast boot yet) .

 

VM1 works. VM2 boot into wrong resolution with a code 43 which was resolved with a driver update. VM3 is still failing to power on D3. Still investigating, but definitely making progress