PCIe Passtrough "None Available"

[spourt38]

I am in Unraid version 6.12.2  because 6.12.4 make my unraid crash every day....

Good news for the next release 6.12.5

 

 

[mikey6283]

 I have followed  K Niesten  solution works as Coral TPU is now on the white list see below:

                $arrBlacklistClassIDregex = '/^(05|06|0a|0b|0c05)/';
                // Got Class IDs at the bottom of /usr/share/hwdata/pci.ids
                $arrWhitelistGPUClassIDregex = '/^(0001|03)/';
                $arrWhitelistAudioClassIDregex = '/^(0403)/';

                # "System peripheral [0880]" "Global unichip corp. [1ac1]" "Coral Edge Tpu [089a]" -pff "Global unichip corp. [1a>
                #                    typeid                                                                                      >
                # file is csv typeid:productid
                #
                if (is_file("/boot/config/VMPCIOverride.cfg")) {
                        $arrWhiteListOverride = str_getcsv(file_get_contents("/boot/config/VMPCIOverride.cfg")) ;
                }
                $arrWhiteListOverride[] = "0880:089a" ;

 

However,  the Coral TPU does not appear on the template to edit. Is there something that i am missing ?? Any advicse would be appreciated.

 

[SimonF]

8 hours ago, mikey6283 said:

 I have followed  K Niesten  solution works as Coral TPU is now on the white list see below:

                $arrBlacklistClassIDregex = '/^(05|06|0a|0b|0c05)/';
                // Got Class IDs at the bottom of /usr/share/hwdata/pci.ids
                $arrWhitelistGPUClassIDregex = '/^(0001|03)/';
                $arrWhitelistAudioClassIDregex = '/^(0403)/';

                # "System peripheral [0880]" "Global unichip corp. [1ac1]" "Coral Edge Tpu [089a]" -pff "Global unichip corp. [1a>
                #                    typeid                                                                                      >
                # file is csv typeid:productid
                #
                if (is_file("/boot/config/VMPCIOverride.cfg")) {
                        $arrWhiteListOverride = str_getcsv(file_get_contents("/boot/config/VMPCIOverride.cfg")) ;
                }
                $arrWhiteListOverride[] = "0880:089a" ;

 

However,  the Coral TPU does not appear on the template to edit. Is there something that i am missing ?? Any advicse would be appreciated.

 

Which version of unraid are using? Also have you bound to vfio

 

was add in a 6.12.6

 

You can create the following file

 

boot/config/VMPCIOverride.cfg

 

0880:089a,xxxx:yyyy

 

where x and y match vendor and product id.

[mikey6283]

35 minutes ago, SimonF said:

Which version of unraid are using?

Version 6.12.5

[SimonF]

20 minutes ago, mikey6283 said:

Version 6.12.5

updated some info above

[mikey6283]

1 hour ago, SimonF said:

updated some info above

Sorry please advise what information do you require?

 

I have an issue with Coral TPU no being found in HA after looking a the logs Frigate Logs .Previously it was ok without having to pass-through. I found this thread which prompted me to follow the solution. I removed /08 and save check to see if it was save and saw that /08 was added to the  whitelist. Went to HA to changed the template the Coral TPU was found so that i can add the PCIE.

 

The Coral TPU is greyed out there for it being used by unraid .

 

Hope this helps?

[SimonF]

21 minutes ago, mikey6283 said:

Sorry please advise what information do you require?

 

I have an issue with Coral TPU no being found in HA after looking a the logs Frigate Logs .Previously it was ok without having to pass-through. I found this thread which prompted me to follow the solution. I removed /08 and save check to see if it was save and saw that /08 was added to the  whitelist. Went to HA to changed the template the Coral TPU was found so that i can add the PCIE.

 

The Coral TPU is greyed out there for it being used by unraid .

 

Hope this helps?

Can you provide some screen prints. 

[mikey6283]

14 minutes ago, SimonF said:

Can you provide some screen prints. 

 Hope that this helps i appreciate your time. I did notice that you were dealing  with a similar siatutaion back in september 2023?

[SimonF]

6 minutes ago, mikey6283 said:

 Hope that this helps i appreciate your time. I did notice that you were dealing  with a similar siatutaion back in september 2023?

It needs lto be in its own iommu group. You could try using the pci  acs override option im the vm settings to see if that will split it up.

[mikey6283]

1 hour ago, SimonF said:

It needs lto be in its own iommu group. You could try using the pci  acs override option im the vm settings to see if that will split it up.

Hi i changed the Pcie  acs to Both and it worked  so i  wrongly changed it back to disabled  due to the error message that it shoud be set to disabled. Now  i cannot get back into unraid after the reboot  as there is an error message on  booting  unraid regardon etho. Not surew hat i have done wrong.  I get unraid running again  can i leave  pcie acs running  ?

 

 

[mikey6283]

 Error  message on loading says that no IPV4 is set. I assume it is due to ETH0 ? Any idea on resetting it?

 

 I later here  now so i will look into tomrrow . if you would n't mind replying ?

 

[SimonF]

8 hours ago, mikey6283 said:

 Error  message on loading says that no IPV4 is set. I assume it is due to ETH0 ? Any idea on resetting it?

 

 I later here  now so i will look into tomrrow . if you would n't mind replying ?

 

You can edit or delete

 

root@computenode:~# ls /boot/config/net*
/boot/config/network.cfg
root@computenode:~# 

 

Reboot and it will default to stock dchp.

[mikey6283]

On 7/19/2024 at 7:18 AM, SimonF said:

You can edit or delete

 

root@computenode:~# ls /boot/config/net*
/boot/config/network.cfg
root@computenode:~# 

 

Reboot and it will default to stock dchp.

 Hi

 

 Finally have Coral TPU bonded.,but still have the follwoing errors i have included  details of the installation. Any idea where i an going wrong ?

Logs showing error in container

[SimonF]

1 hour ago, mikey6283 said:

 Hi

 

 Finally have Coral TPU bonded.,but still have the follwoing errors i have included  details of the installation. Any idea where i an going wrong ?

Logs showing error in container 6.97 kB · 1 download

So you are not connecting to a VM? If you are running in a container you need to install this plugin. Also you cannot bond to vfio.

 

[mikey6283]

3 hours ago, SimonF said:

So you are not connecting to a VM? If you are running in a container you need to install this plugin. Also you cannot bond to vfio.

 

 My first issue was i couldn't  pass-thru my pci  to show up in HA vm.  as the pci card was not  showing up as avaiable in the VM even though it was on unraid. So i used pci  acs override option as mentioned and bonded the Coral to HA  as below.  immou no is the bridge for the Coral Do i tick these to be bonded to coral ?  I have Coral drivers in stalled but they too cannot find the Coral.

 

Any ideas on where i am going wrong? I had it all work  upto recently without have to bond the pci ? Your help is accpreciated.

[mikey6283]

On 7/18/2024 at 5:35 PM, SimonF said:

Which version of unraid are using? Also have you bound to vfio

 

was add in a 6.12.6

 

You can create the following file

 

boot/config/VMPCIOverride.cfg

 

0880:089a,xxxx:yyyy

 

where x and y match vendor and product id.

 Sorry i missed your reply, where would i write this file to ?  I am assuming that the  details are 0880:0089a,1ac1:089a ??

Version 6.12.10  installed now

 

I checked the libvirt_helpers.php file  it has been white listed :

 $arrBlacklistClassIDregex = '/^(05|06|08|0a|0b|0c05)/';
                // Got Class IDs at the bottom of /usr/share/hwdata/pci.ids
                $arrWhitelistGPUClassIDregex = '/^(0001|03)/';
                $arrWhitelistAudioClassIDregex = '/^(0403)/';

                # "System peripheral [0880]" "Global unichip corp. [1ac1]" "Coral Edge Tpu [089a]" -pff "Global unichip corp. [1a>
                #                    typeid                                                                                      >
                # file is csv typeid:productid
                #
                if (is_file("/boot/config/VMPCIOverride.cfg")) {
                        $arrWhiteListOverride = str_getcsv(file_get_contents("/boot/config/VMPCIOverride.cfg")) ;
                }
                $arrWhiteListOverride[] = "0880:089a" ;

[SimonF]

14 minutes ago, mikey6283 said:

 Sorry i missed your reply, where would i write this file to ?  I am assuming that the  details are 0880:0089a,1ac1:089a ??

Version 6.12.10  installed now

 

I checked the libvirt_helpers.php file  it has been white listed :

 $arrBlacklistClassIDregex = '/^(05|06|08|0a|0b|0c05)/';
                // Got Class IDs at the bottom of /usr/share/hwdata/pci.ids
                $arrWhitelistGPUClassIDregex = '/^(0001|03)/';
                $arrWhitelistAudioClassIDregex = '/^(0403)/';

                # "System peripheral [0880]" "Global unichip corp. [1ac1]" "Coral Edge Tpu [089a]" -pff "Global unichip corp. [1a>
                #                    typeid                                                                                      >
                # file is csv typeid:productid
                #
                if (is_file("/boot/config/VMPCIOverride.cfg")) {
                        $arrWhiteListOverride = str_getcsv(file_get_contents("/boot/config/VMPCIOverride.cfg")) ;
                }
                $arrWhiteListOverride[] = "0880:089a" ;

/boot/config/VMPCIOverride.cfg

[spotopolis]

I am having this same issue with an Intel QAT card on UNRAID 6.12.10. Im trying to pass it though to my pFsense VM. 

 

I tried adding the "boolBlacklist=false" for this device as shown in the first page of the thread, but it didnt work for me.

This is considered a system device as it is a 0b40. 

If I try and manually ad it to the VMs XML file, it says that the device is currently in use.

I have also tried to remove the whole of the "0b" class from the blacklist and saved the libvirt_helpers.php  file, and I still did not see it as an available options for my VM to select. Being how the system grabs this at boot, I figured it need to be configured to not grab it during the boot process. Trying to release it, or modify the config once the system is running didnt make sense to me. I saw that the libvirt_helpser.php changes are lost on reboot, and I did verify that is the case. 

What else could I try to get this device to pass though?

[SimonF]

45 minutes ago, spotopolis said:

I am having this same issue with an Intel QAT card on UNRAID 6.12.10. Im trying to pass it though to my pFsense VM. 

 

I tried adding the "boolBlacklist=false" for this device as shown in the first page of the thread, but it didnt work for me.

This is considered a system device as it is a 0b40. 

If I try and manually ad it to the VMs XML file, it says that the device is currently in use.

I have also tried to remove the whole of the "0b" class from the blacklist and saved the libvirt_helpers.php  file, and I still did not see it as an available options for my VM to select. Being how the system grabs this at boot, I figured it need to be configured to not grab it during the boot process. Trying to release it, or modify the config once the system is running didnt make sense to me. I saw that the libvirt_helpser.php changes are lost on reboot, and I did verify that is the case. 

What else could I try to get this device to pass though?

nano /boot/config/VMPCIOverride.cfg

 

and add 8086:0435 then save. Should show in VM Manager if bound to vfio now.

 

Looks like already bound to VFIO, have you rebooted?

 

also check to see if bound to a driver other than vfio.

 

Example

root@computenode:/mnt/user/VMbackup# lspci -s 0000:09:00.0 -k
09:00.0 VGA compatible controller: NVIDIA Corporation TU117GLM [Quadro T400 Mobile] (rev a1)
        Subsystem: NVIDIA Corporation Device 1489
        Kernel driver in use: nvidia
        Kernel modules: nvidia_drm, nvidia

 

run lspci -s 0000:83:00.0 -k

 

[spotopolis]

32 minutes ago, SimonF said:

nano /boot/config/VMPCIOverride.cfg

 

and add 8086:0435 then save. Should show in VM Manager if bound to vfio now.

 

Looks like already bound to VFIO, have you rebooted?

 

also check to see if bound to a driver other than vfio.

 

Example

root@computenode:/mnt/user/VMbackup# lspci -s 0000:09:00.0 -k
09:00.0 VGA compatible controller: NVIDIA Corporation TU117GLM [Quadro T400 Mobile] (rev a1)
        Subsystem: NVIDIA Corporation Device 1489
        Kernel driver in use: nvidia
        Kernel modules: nvidia_drm, nvidia

 

run lspci -s 0000:83:00.0 -k

 

Yes I have rebooted already. 
 

Do I need to add "BIND=" before the line because that file does not exist in my "/boot/config" folder so I would need to create it. 

Im assuming so, cause after adding just "8086:0435" fix common problems said "Your flash drive has possible corruption on /boot/config/VMPCIOverride.cfg."

This is the output of running lspci -m -nn


EDIT: I modified /boot/config/VMPCIOverride.cfg to read as below:



no more errors in fix common problems, however this device is still not listed as a device to select in the VM

Edited July 28 by spotopolis

[SimonF]

18 minutes ago, spotopolis said:

Yes I have rebooted already. 
 

Do I need to add "BIND=" before the line because that file does not exist in my "/boot/config" folder so I would need to create it. 

Im assuming so, cause after adding just "8086:0435" fix common problems said "Your flash drive has possible corruption on /boot/config/VMPCIOverride.cfg."

This is the output of running lspci -m -nn


EDIT: I modified /boot/config/VMPCIOverride.cfg to read as below:



no more errors in fix common problems, however this device is still not listed as a device to select in the VM

No does not need bind=

[spotopolis]

3 minutes ago, SimonF said:

No does not need bind=

So the Fix common problems error is safe to ignore in this case? Should this need a reboot to apply as well?

[SimonF]

7 minutes ago, spotopolis said:

So the Fix common problems error is safe to ignore in this case? Should this need a reboot to apply as well?

Yes it is fine it is just an override file for vmmanager. No reboot.

 

But you will need to blacklist the kernel driver as it is bound to the device. Will only show in vm manager if bound to vfio and the override file is set.

 

Tools->System Drivers and add blacklist to modprobe.d entry. Will need a reboot.

 

[spotopolis]

13 minutes ago, SimonF said:

Yes it is fine it is just an override file for vmmanager. No reboot.

 

But you will need to blacklist the kernel driver as it is bound to the device. Will only show in vm manager if bound to vfio and the override file is set.

Is the correct way to do this still to do this in modprobe.d ?

/boot/config/modprobe.d/qat_dh895xcc.conf

EDIT: Got it. Did it though the GUI. In process of rebooting now. Ill report back once I can determine if that has resolved the issue.

Edited July 28 by spotopolis

[SimonF]

8 minutes ago, spotopolis said:

Is the correct way to do this still to do this in modprobe.d ?

/boot/config/modprobe.d/qat_dh895xcc.conf

YES  but system drivers wil! create the file.