First the error:

internal error: qemu unexpectedly closed the monitor: 2017-07-08T22:34:48.518616Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.0,addr=0x9: vfio: failed to set iommu for container: Operation not permitted
2017-07-08T22:34:48.518646Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.0,addr=0x9: vfio: failed to setup container for group 44
2017-07-08T22:34:48.518652Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.0,addr=0x9: vfio: failed to get group 44
2017-07-08T22:34:48.518667Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.0,addr=0x9: Device initialization failed

 

My hardware: 

Asus Z10PE -D16 WS

dual intel xeon 2683 v3 (14 core cpu's)

128GB ECC memory (64GB per CPU in Quad Channel arrangement)

2 LSI 9300 HBA cards connected to 4 backplanes (16 bay) in those backplanes I have 5 6TB HDD (HBA cards in CPU 1 pci express slot)

1 intel single port 1GbE Network Card (in CPU 1 pci express slot)

1 intel Dual port 10GbE network card (in CPU 2 pci express slot)

1000W power supply platinum rating

 

My problem: 

I have a VM set up to run FreeNAS with HBA and Network card passthrough.  It functioned perfectly until I installed a second processor and moved the 10GbE network card to another PCI Express slot.

I can't passthrough PCI express cards to a VM without getting the error above.  I did some troubleshooting by trying no card passthrough and the VM starts up.  I have a windows 10 VM also running without any passthrough, and it starts up.  When I try to do any passing through of the HBA card or either of the network cards whether they are in CPU 1 or CPU 2's expansion slots, they don't start up.  

 

When I did tests with other cards essentially what ever device was first was the "group #" that showed up in the error report.

 

In addition I tried adding the PCI card to my Windows VM and the startup still failed.  I got the same error.  Again when I remove the passthrough option it boots up as normal.

 

Here's what I use when editing the XML, and it worked flawlessly when I had a single CPU:

This goes before the last "<Device>" text in the XML:

 

<hostdev mode='subsystem' type='pci' managed='yes'>
 <driver name='vfio'/>
 <source>
 <address domain='0x0000' bus='0x0a' slot='0x00' function='0x0'/>
 </source>
</hostdev>

 

I should note that I verified that all the devices I'm trying to pass through are on separate IOMMU Groups.  Example.  LSI 9300 are on IOMMU group 44 and 46, the 10GbE card is group 52 and 53 for each connection, and the additional network card is on group 45.

Edited July 8, 20178 yr by strikermed

So, this stumped me for hours, and after reading over my troubleshooting steps I came to the conclusion I hadn't gone through my BIOS settings after adding a new CPU.

 

Everything looked correct, but I had one setting that I thought was suspicious, since I was able to virtualize without any issue with a single cpu.

 

It turns out with 2 CPU's it actually complicates the whole IOMMU paths, so this setting is crucial to have enabled.  On my motherboard it was under advanced settings, and under PCI EX

 

"Single Root IO virtualization support" (SR-IOV)

 

Hopefully if anyone runs into this issue they can find this being their solution.