Binding VFIO is causing unraid to crash on array startup.

[forcalon]

Hello!
I am currently trying to passthrough an Elgato HD60 PCIe capture card to one of my VMs but I simply can't get it to work.

What I have tried:
Without doing any PCIe ACS Overrides my Elgato is in its own IOMMU group (I have also tried using all available override types just to rule that out)

 

 

But if I turn this on and reboot, the array will cause a crash/freeze when started. Another thing is when that group is turned on and I haven't tried to start the array yet, when I navigate to Tools > System Devices to see the IOMMUs to try and turn it off so I can use my unraid again, it crashes.

The one thing I can do to get Unraid working again is to manually remove the VFIO-bind entry from /boot/config/vfio-pci.cfg. When that cfg-file is empty, the array starts just fine.

I have tried checking the syslog-previous after a crash but it doesn't seem to be throwing any errors really, the crash seems to happen after the array is done starting, during the docker startup phase etc.

Relevant system specs:
mobo: z170a gaming m7

cpu: Intel 6700k (I remember reading somewhere that skylake processors have issues with passthrough, is this the culprit?)
 

Any advice is much appreciated, cheers.

 

EDIT:
Also worth mentioning is that I have Virtualization Technology and VT-d enabled. Unraid is recognizing this and shows:
IOMMU:Enabled in the info tab

 

I have also tried manually passing it through the xml with:
 

<hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
      </source>
</hostdev>

Which also caused a crash.

 

Edit 2:
I have also tried changing syslinux.cfg to include "vfio-pci.ids=12ab:0380", with and without "pci=realloc=off" and "iommu=pt". Still crashes.

Edited September 8, 2024 by forcalon

[SimonF]

5 hours ago, forcalon said:

Hello!
I am currently trying to passthrough an Elgato HD60 PCIe capture card to one of my VMs but I simply can't get it to work.

What I have tried:
Without doing any PCIe ACS Overrides my Elgato is in its own IOMMU group (I have also tried using all available override types just to rule that out)

 

 

But if I turn this on and reboot, the array will cause a crash/freeze when started. Another thing is when that group is turned on and I haven't tried to start the array yet, when I navigate to Tools > System Devices to see the IOMMUs to try and turn it off so I can use my unraid again, it crashes.

The one thing I can do to get Unraid working again is to manually remove the VFIO-bind entry from /boot/config/vfio-pci.cfg. When that cfg-file is empty, the array starts just fine.

I have tried checking the syslog-previous after a crash but it doesn't seem to be throwing any errors really, the crash seems to happen after the array is done starting, during the docker startup phase etc.

Relevant system specs:
mobo: z170a gaming m7

cpu: Intel 6700k (I remember reading somewhere that skylake processors have issues with passthrough, is this the culprit?)
 

Any advice is much appreciated, cheers.

 

EDIT:
Also worth mentioning is that I have Virtualization Technology and VT-d enabled. Unraid is recognizing this and shows:
IOMMU:Enabled in the info tab

 

I have also tried manually passing it through the xml with:
 

<hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
      </source>
</hostdev>

Which also caused a crash.

Are you able to add diagnostics?

[forcalon]

25 minutes ago, SimonF said:

Are you able to add diagnostics?

Yeah sorry, here it is.

 

kraken-diagnostics-20240908-2042.zip

[SimonF]

49 minutes ago, forcalon said:

Yeah sorry, here it is.

 

kraken-diagnostics-20240908-2042.zip 108.88 kB · 0 downloads

Looking at a previous post looks like the card does not support resets hence crash. I have done a quick check and the quirk patch does not seem to be in the kernel

 

 

[forcalon]

4 hours ago, SimonF said:

Looking at a previous post looks like the card does not support resets hence crash. I have done a quick check and the quirk patch does not seem to be in the kernel

 

 

I almost got it working with a patched kernel. I can have the IOMMU group active now and start the array. However, when I try to start a vm that uses the capture card it crashes after waiting for the power state to change:

Sep 9 02:37:43 Kraken kernel: vfio-pci 0000:04:00.0: enabling device (0000 -> 0002)
Sep 9 02:37:45 Kraken kernel: vfio-pci 0000:04:00.0: not ready 1023ms after PM D3hot->D0; waiting
Sep 9 02:37:46 Kraken kernel: vfio-pci 0000:04:00.0: not ready 2047ms after PM D3hot->D0; waiting
Sep 9 02:37:48 Kraken kernel: vfio-pci 0000:04:00.0: not ready 4095ms after PM D3hot->D0; waiting
Sep 9 02:37:52 Kraken kernel: vfio-pci 0000:04:00.0: not ready 8191ms after PM D3hot->D0; waiting
 

[SimonF]

5 hours ago, forcalon said:

I almost got it working with a patched kernel. I can have the IOMMU group active now and start the array. However, when I try to start a vm that uses the capture card it crashes after waiting for the power state to change:

Sep 9 02:37:43 Kraken kernel: vfio-pci 0000:04:00.0: enabling device (0000 -> 0002)
Sep 9 02:37:45 Kraken kernel: vfio-pci 0000:04:00.0: not ready 1023ms after PM D3hot->D0; waiting
Sep 9 02:37:46 Kraken kernel: vfio-pci 0000:04:00.0: not ready 2047ms after PM D3hot->D0; waiting
Sep 9 02:37:48 Kraken kernel: vfio-pci 0000:04:00.0: not ready 4095ms after PM D3hot->D0; waiting
Sep 9 02:37:52 Kraken kernel: vfio-pci 0000:04:00.0: not ready 8191ms after PM D3hot->D0; waiting
 

Did you add this? 

And, in /etc/modprobe.d/vfio.conf or similar:

options vfio-pci ids=12ab:0380 disable_idle_d3=1

 

[forcalon]

4 hours ago, SimonF said:

Did you add this? 

And, in /etc/modprobe.d/vfio.conf or similar:

options vfio-pci ids=12ab:0380 disable_idle_d3=1

 

Yes, I tried adding it in /etc/modprobe.d/vfio.conf
Also tried adding it to the startup parameters in syslinux.cfg

[forcalon]

Got it to boot up the vm and windows actually recognizes the capture card. Installed the drivers and software but no picture or sound

 

Edited September 9, 2024 by forcalon

[SimonF]

5 hours ago, forcalon said:

Got it to boot up the vm and windows actually recognizes the capture card. Installed the drivers and software but no picture or sound

 

Not sure if it will help but maybe try unsafe interupts in vm settings.