Sporadic GPU passthrough issue with 560Ti

[Mythzz]

I`m seeing this strange issue when you start any sort of VM on UNRAID. If I run a passthrough my 560Ti, it will boot (and reboot) fine a few times. Post which it will blank out completely, and I notice the 560Ti GPU Fan kicks in at full throttle (Though nothings rendering on the screen). I DONT see this happening on my 980Ti.

If I reboot UNRAID, the issue goes away for some time, and then the problem comes back after a few VM reboots. Has anyone experienced this? 

 

My rig has the current steup:

 

• i5-6600K
• AsRock Fatal1ty Z170 Gaming K4/D3
• 32 GB DDR3 RAM
• Asus 980Ti Strix 6GB (Secondary Slot)
• Zotac 560Ti 1GB (Primary Slot)

 

MY IOMMU Groups have all been separated correctly. 

 

In the VM logs I dont see any error pointing to something obvious. However in the system logs I do see this line:

 

May 30 13:59:27 MythzUNRAID kernel: vfio-pci 0000:01:00.0: Invalid PCI ROM header signature: expecting 0xaa55, got 0xffff

 

0000:01:00.0 as per the IOMMU groups points to my 560Ti GPU.

 

IOMMU group 14:[10de:1200] 01:00.0 VGA compatible controller: NVIDIA Corporation GF114 [GeForce GTX 560 Ti] (rev a1)

 

Now the VBIOS`s Ive tried are ALL from Techpowerup, as well as a dump from GPU-Z. They all bring back the same line above.

 

MY IOMMU Groups have all been separated correctly. (It has ACS overide set to "BOTH".)

 

Any clues?

 

Edited June 1, 2020 by Mythzz

[Mythzz]

Been trying a few things over the week, to see if I could reduce the problem, but nothing so far:

 

When the VM boots fine I see the following line in the UNRAID system logs:

Jun 1 14:33:49 MythzUNRAID kernel: vfio-pci 0000:01:00.0: enabling device (0000 -> 0003)

 

No Clues found in the VM logs of late.

 

I wonder if this has anything to do with the monitor its connected to? Read that in a random forum while researching. But I did try connecting it to a different screen which didnt help.

 

I earlier thought that the error, pointed to some issue with the ROM I was using. Doesnt explain why it works sporadically.

 

If anyone knows of any logs I should be looking at, happy to oblige.

 

 

 

 

[Mythzz]

So far, I havent been able to pinpoint or workaround the problem.

 

Additionally I`m also seeing that any Windows VM I create (via the 560Ti or the 980Ti) seems to go into a "Paused" state if you leave it running for about 10 mins. It also causes UNRAID to hang, and become unresponsive. The screen on the Windows VM will go all Glitchy (White lines from top to bottom) before migrating to a black screen. I`m not sure if these 2 issues are related. Its stable if not passed through the GPU.

Edited June 2, 2020 by Mythzz

[1812]

5xx series is not recommended. Use 7xx and higher because these issues happen due to architecture and bios.

Edited June 2, 2020 by 1812

[Mythzz]

On 6/2/2020 at 9:40 PM, 1812 said:

5xx series is not recommended. Use 7xx and higher because these issues happen due to architecture and bios.

Thanks 1812. I did happen to bump into the Unraid wiki which clearly says 6xx and above. I should have paid more attention. I got so caught up in testing this, I missed that detail.

 

Btw any idea about the weird Windows VM issue i`m seeing? The one with the 980ti and white streaks across the screen a monute after booting? I can reproduce that on the fly, no matter what CPU core I assign to the VM. Or.....Shall I open this in a separate thread? Happy to.

Edited June 4, 2020 by Mythzz

[1812]

7 hours ago, Mythzz said:

Thanks 1812. I did happen to bump into the Unraid wiki which clearly says 6xx and above. I should have paid more attention. I got so caught up in testing this, I missed that detail.

 

Btw any idea about the weird Windows VM issue i`m seeing? The one with the 980ti and white streaks across the screen a monute after booting? I can reproduce that on the fly, no matter what CPU core I assign to the VM. Or.....Shall I open this in a separate thread? Happy to.

Maybe driver issue. Are you running the windows driver or nvidia?

Edited June 5, 2020 by 1812

[Mythzz]

6 hours ago, 1812 said:

Maybe driver issue. Are you running the windows driver or nvidia?

I remember it taking 10 - 15 mins to recognize the card AFTER attching the GPU. So I`m assuming Windows. I tried installing the Nvidia one, but th edarn thing keeps hanging before the install even finishes.