Hi Team,

 

I would like to raise a concern around my VFIO passthrough experience. May be this is a bug which can be addressed. Or this experience helps someone else from this pitfall.

 

I have a pfSense VM with 4 NIC's passthrough to the VM. The VM is set to autostart ofcourse.

The four NIC's are listed as below.

 

Now I decided to add a new Asus PCI M.2 X16 card which supports 4 M.2 drives with PCI bifurcation.

When I plugged in this device. My system would boot but freeze on startup. The error thrown out is a Filesystem corruption error displayed on the screenshot below.

 

After multiple hours of troubleshooting, I found that the PCI IDS have changed for the NIC's.

The four NIC's are now listed as below.

 

And if you see carefully, the old PCI IDS are now occupied by the SATA controllers.

 

I disabled Autostart of array, but could not disable Autostart of VM. Had to pull out the new Asus card, go back to old config. Set the pfSense VM not to autostart. Then plugged back the Asus card in. And finally I was able to start the VM Manager. At this point, if I manually start the VM without editing the passthrough devices, the same BTRFS corruption error happens and everything is frozen. So I hard rebooted the machine and before starting the pfSense VM, I edited the passthrough devices and everything works. Yey !!!

 

Can this be fixed in anyway ? The lockout situation is very bad, any check before starting the VM to see if the correct devices are passed through would be good. Then at least the VM manager starts and the VM config can be edited. My situation was severely bad as it was throwing out the array disks itself.

12 minutes ago, Jayant P Saikia said:

I disabled Autostart of array, but could not disable Autostart of VM

If you disable array auto start the VMs won't auto start after boot.

I did disable array autostart. And I was able to login. Then I disabled VM Manager start. Then I was able to start the array. But as soon as I enable VM Manager the error happend. The only way was to go back to old config without the Asus card and make the VM to not auto start.

3 minutes ago, Jayant P Saikia said:

But as soon as I enable VM Manager the error happend.

That should not happen, VMs should never autostart if array autostart is disabled, it's a constant user complain, i.e., "VMs don't start with autostart enable", because they only start at first boot if array auto start is enabled.

Well that's not what I experienced. 

 

Array Autostart Disabled.

VM Manager Enable VMs No.

pfSense VM Autostart On.

 

When I doVM Manager Enable VMs No to Yes.

The pfSense VM Autostarts irrespective Array Autostart was Disabled or Enabled.

Also, anyway to check [8086:10d3] 07:00.0 with [8086:1d6b] 07:00.0. and find a miss match. Which stops the PCI device reset ?

Seems to me that the VM start is only checking "07:00.0".

38 minutes ago, Jayant P Saikia said:

Well that's not what I experienced. 

You are correct, behavior changed in the current release, going to find out when and report it since like you've experience this is a problem.

 

32 minutes ago, Jayant P Saikia said:

Also, anyway to check [8086:10d3] 07:00.0 with [8086:1d6b] 07:00.0. and find a miss match. Which stops the PCI device reset ?

Not really my area but I'll also inquire about that, it would make things easier if that's possible to do.

Thanks for the update. Will keep this open to see if the PCI device id mapping can also be improved.