Atheros AR9280 Wireless Card BAR error

[Arbadacarba]

I've looked and looked and only found references to this type of error with regards to passing through NVME drives.

 

I am trying to set up my pfsense VM to connect to a hotspot int the certain event that my point to point LTE connection fails. To do this I want to pass through a Wifi card.

 

I cannot use my built in AX card because FreeBSD does not support current cards. I bought a Ralink corp. RT3090 Wireless 802.11n card and the VM works but my whole server locks up every few days (Working on troubleshooting that currently)

 

So I bought a Atheros card and bound that to VFIO using the System Devices Tool.

 

I see it properly in my VM editor, and can select it and apply, but when I try to boot the VM I get:

 

 

I've looked and looked and found a couple of references:

PCIe Passthrough of Atheros AR9280

(OK, Looks like One reference)

 

I've tried 2 different versions of the same card, both giving the same error.. I've tried tow different slots in my mainboard...

 

I just don't know where else to look.

 

Thanks

 

Arbadacarba

 

 

 

[ghost82]

In that discussion, some user is reporting it working with the following mod, try to add custom qemu arg at the bottom of your xml, before </domain>.

From your picture your pci passthrough should have the alias hostdev1.

 

...
...
    <qemu:commandline>
      <qemu:arg value='-set'/>
      <qemu:arg value='device.hostdev1.x-msix-relocation=bar2'/>
    </qemu:commandline>
  </domain>

 

Edited February 20, 2022 by ghost82

[Arbadacarba]

I looked at that and could not figure out how to add that arg to the file...

 

I swapped the Wifi cards and gave it a try.

 

Other than having to realize that the hostdev changed to 0 it worked like a charm

 

Thank You VERY much.

 

Arbadacarba

[Arbadacarba]

I'm curious if there was a resource you can point me to for the conversion from Proxmox Scripting to Unraid's XML?

 

Or are you a witch?

 

Seriously appreciate your help on this... It worked perfectly.

[ghost82]

3 hours ago, Arbadacarba said:

I'm curious if there was a resource you can point me to for the conversion from Proxmox Scripting to Unraid's XML?

There's no "Unraid's xml", but there's libvirt (unraid is an interface for libvirt).

I don't know and never used proxmox, but I can image it's the same, with its configuration options.

Both proxmox and unraid are based on qemu and libvirt, libvirt being a sort of bridge to configure qemu vms (xml format).

So the basic documents you want to refer to are that for libvirt (and qemu).

You can run qemu vms from the command line terminal, qemu is a binary and you run it as a traditional binary, with options, parameters, etc.

Or you can run libvirt vms from the command line with the "virsh" command.

Most of the qemu commands/options are available in libvirt, so that options translate into xml format.

When such options are not available directly in libvirt you use <qemu:commandline>.

In the example above, in libvirt:

    <qemu:commandline>
      <qemu:arg value='-set'/>
      <qemu:arg value='device.hostdev1.x-msix-relocation=bar2'/>
    </qemu:commandline>

translates into qemu command line:

./qemubinary ...other options... -set device.hostdev1.x-msix-relocation=bar2

 

Edited February 22, 2022 by ghost82

[Arbadacarba]

So this worked GREAT right up until I installed RC3...

 

What do you mean you didn't see this scenario? There must be 1 of us doing this...

 

So after upgrading I rebooted and my VM would not start. I look and discover that my Atheros card is no longer included in the VFIO

 

I added it back in and rebooted.

 

Then on reboot I get a slightly different version of the message I got before... Only now when I look I don't see an appropriate alias assign to the wifi card.

 

 

And further if I add an alias, it's gone when I look again after I attempt to start the VM.

Edited March 12, 2022 by Arbadacarba

[ghost82]

4 hours ago, Arbadacarba said:

Only now when I look I don't see an appropriate alias assign to the wifi card

If this is the issue, it could be a bug (?) in the rc3.

I think you could always use terminal to edit the xml of the vm with the 'virsh edit' command.

If the alias is lacking try to manually add it with terminal:

virsh edit HEREVMNAME

 

Then in the editor add the alias in the appropriate device section:

<alias name='YOURALIASHERE'/>

 

Save and exit the editor.

Double check again with virsh edit that the alias has being added.

Close the editor and try.

[ghost82]

4 hours ago, Arbadacarba said:

What do you mean you didn't see this scenario?

I don't understand this, can you rephrase?

[Arbadacarba]

12 hours ago, ghost82 said:

I don't understand this, can you rephrase?

 

Failed sarcasm... Was supposed to show that I fully understood just how unique this question was and that I was in no way surprised that it wasn't foreseen and therefore fixed ahead of time...

 

WOW, I know I don't do sarcasm well... I shouldn't even have tried.

 

Really... that was intended as entirely self deprecating and to express my appreciation for the efforts to make this bizarre use case possible at all.

 

Thanks Again Ghost82

 

I did try editing it in the gui and did try inserting the alias there... but when I run the VM with the Atheros card connected (?) even the alias' for the other NIC vanish from the XML

 

Arbadacarba

 

[ghost82]

On 3/12/2022 at 10:35 PM, Arbadacarba said:

WOW, I know I don't do sarcasm well... I shouldn't even have tried.

Not your fault for sure, even with a translator I'm missing the meaning, even if explained ahah, but all I need to know it's that it was sarcasm

 

Did you try with the virsh edit command? It can be a bug in the gui stripping the aliases.

[Kilroy83]

Hi same problem here since 6.10rc3. With macinabox VM 

where i set this:

 

<qemu:arg value='device.sata0-0-1.rotation_rate=1'/>

 

The VM won’t start with error « no device sata0.0.1….

i try to had a label via webgui and virsh edit  but the setting not stay in place .

[ghost82]

Ok, open a bug in the appropriate forum section.

[Kilroy83]

 

23 minutes ago, ghost82 said:

Ok, open a bug in the appropriate forum section.

Ok Thanks for quick reply .

[Arbadacarba]

2 hours ago, ghost82 said:

Ok, open a bug in the appropriate forum section.

I think I did, and I think he already replied there

 

Thanks

[ghost82]

Can you please check in your diagnostics file, after setting a vm with aliases in unraid (don't bother if they are stripped or not) and run it then stop it:

 

1. Extract diagnostics zip and go to qemu/VMName.txt

2. Open VMName.txt and check if the aliases are there: for example for 'sata0-0-1' alias, you should have this in the device line:

id=sata0-0-1

 

Libvirt aliases translate to ids in qemu.

 

If they are not there too, as a temporary workaround you could run the vm directly from the command line, adding all the ids you need, calling directly the qemu binary.

Edited March 16, 2022 by ghost82

[kilroy]

Hi in my problematic vm i can find this in diagnostic file

 

-device '{"driver":"ide-hd","bus":"ide.2","drive":"libvirt-2-format","id":"sata0-0-2","bootindex":1,"write-cache":"on"}' \

 

-set device.sata0-0-2.rotation_rate=1 \

 

qemu-system-x86_64: -set device.sata0-0-2.rotation_rate=1: there is no device "sata0-0-2" defined

[ghost82]

Looking at the qemu bugtracker, it seems confirmed there too; the alias is there, it's not stripped, the issue is with the -set and json format.

https://gitlab.com/qemu-project/qemu/-/issues/787

 

Quote

This seems caused by libvirt trying to generate JSON syntax for -device by default for QEMU 6.2.0. However, QEMU 6.2.0 doesn't support -set option when JSON syntax is used for -device

 

Also here:

https://gitlab.com/libvirt/libvirt/-/issues/287

 

There is a patch too, but it hasn't been merged.

Not sure, how to eventually modify qemu.conf to add capability_filters; the only option could be to extract the unraid bzroot file, "patch" it, and recompress it, to have a persistent solution after a reboot. Moreover it's not clear if this is working or not.

...and:

Quote

# For debugging and testing purposes it's sometimes useful to be able to disable

# libvirt behaviour based on the capabilities of the qemu process. This option

# allows to do so. DO _NOT_ use in production and beaware that the behaviour

# may change across versions.

 

 

For rotation_rate I think you can actually add it to the libvirt xml without using the custom qemu arg (from libvirt 7.3.0, 6.10 RC3 should have >=7.8.0), something like this, in the target line of the disk section:

    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='path/to/vdisk.img'/>
      <target dev='hdc' bus='sata' rotation_rate='1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='2'/>
    </disk>

 

For x-msix-relocation the only option should be that described in the next post

Edited March 17, 2022 by ghost82

[ghost82]

Since this is an 'issue' specific with libvirt json format, running directly qemu from command line should work.

If you attach diagnostics to see the command we can try to translate it for qemu and see if that works.

Edited March 17, 2022 by ghost82

[kilroy]

11 hours ago, ghost82 said:

Looking at the qemu bugtracker, it seems confirmed there too; the alias is there, it's not stripped, the issue is with the -set and json format.

https://gitlab.com/qemu-project/qemu/-/issues/787

 

 

Also here:

https://gitlab.com/libvirt/libvirt/-/issues/287

 

There is a patch too, but it hasn't been merged.

Not sure, how to eventually modify qemu.conf to add capability_filters; the only option could be to extract the unraid bzroot file, "patch" it, and recompress it, to have a persistent solution after a reboot. Moreover it's not clear if this is working or not.

...and:

 

 

For rotation_rate I think you can actually add it to the libvirt xml without using the custom qemu arg (from libvirt 7.3.0, 6.10 RC3 should have >=7.8.0), something like this, in the target line of the disk section:

    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='path/to/vdisk.img'/>
      <target dev='hdc' bus='sata' rotation_rate='1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='2'/>
    </disk>

 

For x-msix-relocation the only option should be that described in the next post

Hi your solution "rotation rate in target line"  work perfectly for me. Thanks for all

[kilroy]

Here my diag file for problematic vm before modif if it can help

Macinabox HighSierra.txt

[ghost82]

Thank you @kilroyffor confirming.

Since your case can be easily solved by the above solution, I was more interested in the diagnostics from @Arbadacarba

[Kilroy83]

Ok Thanks for all

[Arbadacarba]

Sure... I appreciate the help.

 

jupiter-diagnostics-20220315-0016.zip

 

This is after trying several troubleshooting steps that I performed to write the post in the prerelease bug forum.

 

 

 

[ghost82]

6 hours ago, Arbadacarba said:

This is after trying several troubleshooting steps that I performed to write the post in the prerelease bug forum

Hi, I already downloaded it, but there is no trace of of passed through 05:00.0.

Please pass it through to a vm with the custom qemu arg, run it with the error and then reattach diagnostics.

[ghost82]

On 3/12/2022 at 5:04 AM, Arbadacarba said:

So this worked GREAT right up until I installed RC3

In RC3 add this:

  <qemu:capabilities>
    <qemu:del capability='device.json'/>
  </qemu:capabilities>

right after the </qemu:commandline> closing tag.

 

Please read also this, as this should be a temporary workaround:

https://forums.unraid.net/bug-reports/prereleases/rc3-vm-alias-handler-change-r1768/?do=findComment&comment=18036

 

Edited March 23, 2022 by ghost82