Passing A Capture Card to VM's

[RGB0b]

I've had mixed luck getting Datapath Vision E1s (or E2s) PCIe capture cards to pass through to an UnRAID VM.  Other cards work fine, but I've tried this model via multiple slots on multiple machines and while I've occasionally found a combination that works, I mostly just get this error:

 

internal error: qemu unexpectedly closed the monitor: 2022-03-08T19:37:22.659750Z qemu-system-x86_64: -device vfio-pci,host=0000:07:04.0,id=hostdev0,bus=pci.0,addr=0x6: vfio 0000:07:04.0: Failed to set up TRIGGER eventfd signaling for interrupt INTX-0: VFIO_DEVICE_SET_IRQS failure: Device or resource busy

 

Enabling PCIe ACS override to Downstream, Multi-Function or Both doesn't seem to make a difference.  I've also tried enabling "VFIO allow unsafe interrupts", but that also didn't help.  It's the same if I use Windows 7, 10, or Linux.

 

I'm actually working on documentation to show streamers how to use their UnRAID server as a dedicated streaming box via VM's and this is the final piece of the puzzle.  Would anyone be able to help me figure this one out?

 

nas-diagnostics-20220308-1445.zip

[ghost82]

18 hours ago, RGB0b said:

Failed to set up TRIGGER eventfd signaling for interrupt INTX-0: VFIO_DEVICE_SET_IRQS failure: Device or resource busy

 

You should check if this device supports 'INTx disable', I think it doesn't.

Without 'INTx disable' the device needs its own irq, if another device is using the same irq it won't work: that's probably why you have it working on some builds and not in others, because this depends on the mb layout and shared irqs.

You should check what device is using the same irq and disable it, or adjust your device layout so that this device has its own irq.

To remove the offending device you can do it from the terminal, something like:

echo -n 1 > "/sys/devices/pci0000:00/0000:00:1b.0/remove"

00:1b.0 being the offending device.

 

If I had to bet I would say some usb controller is sharing the same irq....

Edited March 9, 2022 by ghost82

[ghost82]

You can use this script to check if INTx disable is available or not:

#!/bin/sh
# Usage $0 <PCI device>, ex: 9:00.0

INTX=$(( 0x400 ))
ORIG=$(( 0x$(setpci -s $1 4.w) ))

if [ $(( $INTX & $ORIG )) -ne 0 ]; then
echo "INTx disable supported and enabled on $1"
exit 0
fi

NEW=$(printf %04x $(( $INTX | $ORIG )))
setpci -s $1 4.w=$NEW
NEW=$(( 0x$(setpci -s $1 4.w) ))

if [ $(( $INTX & $NEW )) -ne 0 ]; then
echo "INTx disable support available on $1"
else
echo "INTx disable support NOT available on $1"
fi

NEW=$(printf %04x $ORIG)
setpci -s $1 4.w=$NEW

 

Save it into an .sh file, chmod +x, and run it with:

./check.sh 07:04.0

 

[RGB0b]

Thanks so much for the help!  I ran your script and got the following on both; It's a *dual-input card, but shows up as two separate cards:

 

INTx disable support NOT available on 07:00.0

INTx disable support NOT available on 07:04.0

 

Would the next step be to check IRQ settings in the BIOS, or is it something I should be checking in the UnRAID configuration?

 

*The only times I was able to get it working previously, I could only have one of the two enabled, not both.  That would be an acceptable solution (and certainly better than nothing!), but having access to both would be ideal.

 

 

[ghost82]

1 hour ago, RGB0b said:

Would the next step be to check IRQ settings in the BIOS, or is it something I should be checking in the UnRAID configuration?

That is something you cannot fix in unraid or linux or windows, at least for me, and I think it cannot be fixed in the bios too, unless you have some settings to manually assign interrupts.

Irq interrupts are assigned automatically depending on your motherboard, some irq are shared, there are a lot of cases of irq shared with usb, ethernet, sata controllers.

In most cases thia is not an issue, sometimes it is, for example crackling audio could be caused by shared irq between audio and something else (usb, ethernet, etc), and in your specific case, since you can see that your capture card doesn't support INTx disable.

Sometimes, for pcie devices, changing the slot changes also the assigned irq.

Otherwise your only option is to remove the device sharing its irq with your capture card, from within linux, as I wrote.

I suggest you to read something abour irq if you don't know what they are, at least to know the basics.

[RGB0b]

2 hours ago, ghost82 said:

Otherwise your only option is to remove the device sharing its irq with your capture card, from within linux

Thanks very much for the response.  I know what IRQ's are, but to be honest, I haven't had to worry about them since the 90's...and never in Linux.  So, I'm almost worse than a newb, cause I have all that DOS/Windows experience confusing me.

 

I ran "cat /proc/interrupts" and got the attached results, which is not what I was expecting.  I'll do some more research and follow back up if I have any luck.

...and if there's any UnRAID experts here with a Datapath Vision card, maybe you can lend a hand?

[ghost82]

Ok, from your syslog you have:

genirq: Flags mismatch irq 16. 00000000 (vfio-intx(0000:07:00.0)) vs. 00000080 (ehci_hcd:usb1)

 

So, a usb 2.0 controller is using irq 16, confirmed by /proc/interrupts, the same irq that the card is trying to use.

 

P.S:

On 3/9/2022 at 8:51 AM, ghost82 said:

If I had to bet I would say some usb controller is sharing the same irq

I won the bet!!

 

From your lspci output you have 2 usb 2.0 controllers using the ehci driver:

00:1a.0 USB controller [0c03]: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 [8086:8c2d] (rev 04)
	Subsystem: Dell 8 Series/C220 Series Chipset Family USB EHCI [1028:05a5]
	Kernel driver in use: ehci-pci
00:1d.0 USB controller [0c03]: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 [8086:8c26] (rev 04)
	Subsystem: Dell 8 Series/C220 Series Chipset Family USB EHCI [1028:05a5]
	Kernel driver in use: ehci-pci

 

So try to remove one at a time in unraid terminal; try to start with 00:1d.0, since from your proc/interrupts we read usb1 and the output of lspci is USB EHCI #1 (but not sure if it's related or not..):

echo -n 1 > "/sys/devices/pci0000:00/0000:00:1d.0/remove"

 

Try to start the vm.

 

If it still output the error, try with the other one:

echo -n 1 > "/sys/devices/pci0000:00/0000:00:1a.0/remove"

 

Try to start the vm.

 

The other device, 07:04.0, is trying to be assigned to the same irq 16, so it should be ok (or not...I'm not sure, if this needs another irq and may conflict with 07:00.0):

genirq: Flags mismatch irq 16. 00000000 (vfio-intx(0000:07:04.0)) vs. 00000080 (ehci_hcd:usb1)

 

Note: take care to where your unraid usb is plugged in: if you want to avoid issue, plug in into a usb 3.0 port, otherwise you have to map the usb ports to know to which controller they belong, and don't plug in into a port belonging to a controller that you are going to disable/remove.

 

Note2: you may need to remove 07:00.0 and 07:04.0 from vfio-pci.cfg so that they don't attach to vfio at boot; that's because at boot you still have the problematic usb controller detected and I think that when they try to attach to vfio you will get the error.

If it's the case detach from vfio at boot and let them attach to it 'on the fly' (i.e. when you run the vm, after you disabled the usb controller).

Edited March 10, 2022 by ghost82

[RGB0b]

THAT WORKED!!!

 

- Rebooted with the USB stick in a USB 3.0 port
- Ran the first remove, still got the error.

- Ran the second remove and it booted with one input enabled.

 

It wouldn't boot with both inputs enabled.  Not a big deal, but I wanted to try your fix anyway and I got a syntax error.  Did I somehow copy and paste wrong?  I'm accessing the terminal through a browser on Windows, so that might be causing issues.  I posted the terminal window and error after trying to boot with both.

 

10 hours ago, ghost82 said:

you may need to remove 07:00.0 and 07:04.0 from vfio-pci.cfg so that they don't attach to vfio at boot; that's because at boot you still have the problematic usb controller detected and I think that when they try to attach to vfio you will get the error.

If it's the case detach from vfio at boot and let them attach to it 'on the fly' (i.e. when you run the vm, after you disabled the usb controller).

 

This is the one bit I'm unclear about:  I removed them from "vfio-pci.cfg" (as opposed to just unchecking in the GUI) and rebooted.  When it came back up, they were unchecked in Tools/SysDevs, but still showed up in the VM config.  I made sure only one input was checked off and got the same error.  I ran just the second command again (echo -n 1 > "/sys/devices/pci0000:00/0000:00:1a.0/remove") and it worked.

So, do I have to manually run command that every time I boot UnRAID?  Totally cool if that's the fix, I just wanted to double check, cause it feels like I missed something.

 

Thanks so much for your help!  You've given everyone with a Datapath Vision hope to use it in UnRAID...and that's a lot more people then you'd think!!

 

 

 

 

 

Edited March 10, 2022 by RGB0b

[ghost82]

On 3/10/2022 at 6:50 PM, RGB0b said:

but I wanted to try your fix anyway and I got a syntax error

On 3/10/2022 at 6:50 PM, RGB0b said:

 

 

 

Last command you typed, "genirq: Flags mismatch irq 16. 00000000 (vfio-intx(0000:07:00.0)) vs. 00000080 (ehci_hcd:usb1)", is not a command, that's why you got the syntax error, it is a line reported in your logs, to explain you what happened.

 

On 3/10/2022 at 6:50 PM, RGB0b said:

I removed them from "vfio-pci.cfg" (as opposed to just unchecking in the GUI) and rebooted

 

It should be the same, unchecking is fine too!

 

On 3/10/2022 at 6:50 PM, RGB0b said:

When it came back up, they were unchecked in Tools/SysDevs, but still showed up in the VM config

 

Yes, they have to be added to the vm (vfio), but not attached to vfio at boot.

With this config vfio doesn't claim that device at boot, so it attaches to the host driver (if any); when you boot the vm the device detaches from the host driver (if any) and attaches to vfio: that's what I mean for 'on the fly'.

 

On 3/10/2022 at 6:50 PM, RGB0b said:

So, do I have to manually run command that every time I boot UnRAID?

 

Yes, because everytime you boot the bios detects the usb controller (and so, unraid); you may use unraid user scripts to automate it.

Otherwise, you may have an option in bios to enable/disable usb controllers (I have it in my bios): by this way you can disable it in bios, so you don't have to type the command on every boot. But take care that disabling devices in bios may re-arrange all irqs, so the capture card may conflict with something else (or not...this must be checked directly).

 

On 3/10/2022 at 6:50 PM, RGB0b said:

It wouldn't boot with both inputs enabled

 

This could be because the "second device" is claiming the same irq as the first one, if this is the case I can't see any option to fix it.

Edited March 12, 2022 by ghost82

[RGB0b]

6 hours ago, ghost82 said:

Last command you typed, "genirq: Flags mismatch irq 16. 00000000 (vfio-intx(0000:07:00.0)) vs. 00000080 (ehci_hcd:usb1)", is not a command, that's why you got the syntax error, it is a line reported in your logs, to explain you what happened.

Hahahahaha, I knew I was missing something

 

6 hours ago, ghost82 said:

This could be because the "second device" is claiming the same irq as the first one, if this is the case I can't see any option to fix it.

This is what I assumed as well.  Totally fine, just wanted to check!

 

6 hours ago, ghost82 said:

you may use unraid user scripts to automate it.

I used User Scripts to run the second command you suggested every time the array starts:  echo -n 1 > "/sys/devices/pci0000:00/0000:00:1a.0/remove"

 

This is a great solution...and in fact, this entire thread is a great guide for people with Datapath cards to troubleshoot and find the same fix.  I'll eventually roll this into a guide for RetroRGB.com, but I'll absolutely credit you Ghost and reference this thread.  Thank you so much for your time!

[ghost82]

No problem man! happy you solved in some way your issue!

And happy that I helped a retro game fan!!I'm going for 40s and I miss that cool games played natively

 

Edited March 12, 2022 by ghost82