[SOLVED] Passthrough Solarflare Network Card

[happythatsme]

Hi all, 

 

I've beentrying to passthough a Solarflare Dual 10Gb Network card by following Spaceinvaders video here: https://www.youtube.com/watch?v=n2OPfALLqRA

 

I have two IOMMU groups, however, they both seem to have the same ID yet they are listed in two different IOMMU groups. 

IOMMU group 23:[1924:0a03] 04:00.0 Ethernet controller: Solarflare Communications SFC9220 10/40G Ethernet Controller (rev 02)

IOMMU group 24:[1924:0a03] 04:00.1 Ethernet controller: Solarflare Communications SFC9220 10/40G Ethernet Controller (rev 02)

I edited my Syslinux Configuration file adding:

append vfio-pci.ids=1924:0a03 initrd=/bzroot

Hit apply, done, restarted then went to edit my VM.

 

When i edit the VM settings ( ubuntu ) i can see two devices listed:

 

However, when i assign (1 or both) of them to the VM. I get the following error on startup:

 

 

 

I've included the diagnotics file below:

dl360-20core-diagnostics-20200420-2128.zip

 

I have a feeling this card is doing something weird and may not be supported, but i wanted to check if any experts could help before i move it into a dedicated machine. 

 

Thanks

Happy

 

 

 

Edited April 20, 2020 by happythatsme
typo

[happythatsme]

Tried everything i can find in the forums

 

I tried everything i could find in here, all three methods, nothing seemed to work. Giving up for now. 

 

 

[1812]

2 hours ago, happythatsme said:

Tried everything i can find in the forums

 

I tried everything i could find in here, all three methods, nothing seemed to work. Giving up for now. 

 

 

 

 

your log shows

 

vfio-pci 0000:04:00.0: Device is ineligible for IOMMU domain attach due to platform RMRR requirement.

 

you need

 

 

 

 

[happythatsme]

3 hours ago, 1812 said:

 

 

your log shows

 

vfio-pci 0000:04:00.0: Device is ineligible for IOMMU domain attach due to platform RMRR requirement.

 

you need

 

 

 

 

Hi, 

 

thanks very much for the tip! - I somehow missed this thread. 

 

I tried to follow the instructions, however, i still seem to have the same issue in my syslog.

 

dl360-20core-diagnostics-20200426-2354.zip

 

I'm running the latest verison 6.8.3

 

I copied the bzimage file from here:  https://github.com/AnnabellaRenee87/Unraid-HP-Proliant-Edition/tree/master/bzimage releases/Stable/6-8-3  onto my flash disk and rebooted. 

 

 

After rebooting,  i can see the new file is there with todays date, however the i still get "Device is ineligible".

Apr 26 08:52:42 DL360-20Core kernel: vfio-pci 0000:04:00.0: Device is ineligible for IOMMU domain attach due to platform RMRR requirement.  Contact your platform vendor.
Apr 26 08:52:43 DL360-20Core kernel: vfio_ecap_init: 0000:04:00.0 hiding ecap 0x19@0x168
Apr 26 08:52:43 DL360-20Core kernel: vfio_ecap_init: 0000:04:00.0 hiding ecap 0x1e@0x26c
Apr 26 08:52:43 DL360-20Core kernel: vfio-pci 0000:04:00.1: Device is ineligible for IOMMU domain attach due to platform RMRR requirement.  Contact your platform vendor.

I also compiled it on my server then used that version, however, same result.

mkdir -p /mnt/cache/.rmrr
cd /mnt/cache/.rmrr
wget https://raw.githubusercontent.com/AnnabellaRenee87/Unraid-HP-Proliant-Edition/master/build_script/kernel_compile.sh
chmod +x kernel_compile.sh
./kernel_compile.sh

 

Anything else i could be missing? Do i need to udpate bios or any other ideas? 

 

Thanks!

 

 

 

 

 

 

 

[1812]

3 hours ago, happythatsme said:

Hi, 

 

thanks very much for the tip! - I somehow missed this thread. 

 

I tried to follow the instructions, however, i still seem to have the same issue in my syslog.

 

dl360-20core-diagnostics-20200426-2354.zip 88.57 kB · 0 downloads

 

I'm running the latest verison 6.8.3

 

I copied the bzimage file from here:  https://github.com/AnnabellaRenee87/Unraid-HP-Proliant-Edition/tree/master/bzimage releases/Stable/6-8-3  onto my flash disk and rebooted. 

 

 

After rebooting,  i can see the new file is there with todays date, however the i still get "Device is ineligible".

Apr 26 08:52:42 DL360-20Core kernel: vfio-pci 0000:04:00.0: Device is ineligible for IOMMU domain attach due to platform RMRR requirement.  Contact your platform vendor.
Apr 26 08:52:43 DL360-20Core kernel: vfio_ecap_init: 0000:04:00.0 hiding ecap 0x19@0x168
Apr 26 08:52:43 DL360-20Core kernel: vfio_ecap_init: 0000:04:00.0 hiding ecap 0x1e@0x26c
Apr 26 08:52:43 DL360-20Core kernel: vfio-pci 0000:04:00.1: Device is ineligible for IOMMU domain attach due to platform RMRR requirement.  Contact your platform vendor.

I also compiled it on my server then used that version, however, same result.

mkdir -p /mnt/cache/.rmrr
cd /mnt/cache/.rmrr
wget https://raw.githubusercontent.com/AnnabellaRenee87/Unraid-HP-Proliant-Edition/master/build_script/kernel_compile.sh
chmod +x kernel_compile.sh
./kernel_compile.sh

 

Anything else i could be missing? Do i need to udpate bios or any other ideas? 

 

Thanks!

 

 

 

 

 

 

 

how interesting.

 

you typically don't need to update bios for the patch to work but in this case I would consider it. But before that,  I would also consider dropping back an Unraid version or two, maybe 6.8.2 or 6.8.1 and try with those patches. I don't have a need for the patch anymore so I have no way to test if it is the patch or not. But that will eliminate a variable.

[happythatsme]

9 hours ago, 1812 said:

how interesting.

 

you typically don't need to update bios for the patch to work but in this case I would consider it. But before that,  I would also consider dropping back an Unraid version or two, maybe 6.8.2 or 6.8.1 and try with those patches. I don't have a need for the patch anymore so I have no way to test if it is the patch or not. But that will eliminate a variable.

Apologies, I’m rather new to this. 

 

What is the "easiest" way to drop back to a previous version? 

 

Right now, i'd reformt the drive and copy over the files ?  Is there a way to retain my existing logical setup I have a lot of dockers etc installed.

Edited April 27, 2020 by happythatsme

[1812]

21 hours ago, happythatsme said:

Apologies, I’m rather new to this. 

 

What is the "easiest" way to drop back to a previous version? 

 

Right now, i'd reformt the drive and copy over the files ?  Is there a way to retain my existing logical setup I have a lot of dockers etc installed.

noooo, don't format the drive.

 

install this: 

 

 

then once installed (and possibly rebooting) go to settings>Unraid Nvidia. once there, look for the stock Unraid builds, pick one, let it install, reboot. then you'll need to go back to the github site for the right patch that aligns with the Unraid version you picked, copy that bzimage to the flash drive, reboot, and then report.

 

 

 

 

[happythatsme]

2 hours ago, 1812 said:

noooo, don't format the drive.

 

install this: 

 

 

then once installed (and possibly rebooting) go to settings>Unraid Nvidia. once there, look for the stock Unraid builds, pick one, let it install, reboot. then you'll need to go back to the github site for the right patch that aligns with the Unraid version you picked, copy that bzimage to the flash drive, reboot, and then report.

 

 

 

 

 

Thank You! - really useful tool. 

 

I Installed the plugin from https://raw.githubusercontent.com/linuxserver/Unraid-Nvidia-Plugin/master/plugins/Unraid-Nvidia.plg - Then rolled back to 6.8.0

 

 

Rebooted the server, everything seems to be working.

 

I had passed through the device.

 

dl360-20core-diagnostics-20200428-1014.zip

 

Strange i have a different error now: 

 

 

I moved on, i copied over https://github.com/AnnabellaRenee87/Unraid-HP-Proliant-Edition/blob/master/bzimage releases/Stable/6-8-0/bzimage to the flash drive, rebooted and everything started fine. 

 

Got the same error as above. 

dl360-20core-diagnostics-20200428-1026.zip

 

The logs do not mention anything relating to ineligible devices - perhaps the check did not happen in 6.8.0

vfio-pci 0000:04:00.0: Device is ineligible for IOMMU domain attach due to platform RMRR requirement.

I will research the error above

 

Thanks

 

 

Edited April 28, 2020 by happythatsme

[happythatsme]

I created a new VM - As i thought the XML from 6.8.3 may not be compatible with 6.8.0

 

I passed the card to the new VM and it booting right now... I will report back 

[happythatsme]

2 hours ago, happythatsme said:

I created a new VM - As i thought the XML from 6.8.3 may not be compatible with 6.8.0

 

I passed the card to the new VM and it booting right now... I will report back 

 

Passed through successfully using version 6.8.0 - thanks again!