[pfSense in VM with PCI passthrough issues]

Hello,

 

Just wanted to post an update and say I've given up on the old NIC and picked up a new one.  The new one works perfectly with simply stubbing the new card and checking the boxes in the VM editor.  ACS override is set to no.  I followed the guide here and it worked like a charm:

 

 

I booted up pfSense, plugged in the cord and am getting <1ms ping response times on other parts of the network to my pfSense install and no packets dropped.  There are also no more watchdog errors.  I picked up a I350-T4 NIC and has the following details in lspci:

lspci

04:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
04:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
04:00.2 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
04:00.3 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)

 

lspci -n
04:00.0 0200: 8086:1521 (rev 01)
04:00.1 0200: 8086:1521 (rev 01)
04:00.2 0200: 8086:1521 (rev 01)
04:00.3 0200: 8086:1521 (rev 01)

 

 

 

 

 

The VM XML is of no interest as no manual configurations were done.  I appreciate all the suggestions and help.

[pfSense in VM with PCI passthrough issues]

Danioj: If I can't figure this out, that might be what I have to do. I liked the idea of the pci passthrough just a cleaner and more complete solution. It also played a large part in my hardware choice (vt-d). 

 

Neodude: Thanks for the XML. It looks like you are passing through a dual port yes?

 

 

[pfSense in VM with PCI passthrough issues]

A little bit of an update.  I tried turning on the ACS override with no luck, and I tried turning on the unsafe interrupts just for kicks and giggles.  Nothing seems to have an effect on it.

[pfSense in VM with PCI passthrough issues]

I have defaulted to always using seabios. I don't think I have successfully boot ovmf on any of my tests.

 

Here is the iommu groups. I have tried stubbing and using vfio-pci.ids for the nics and the other random device with no luck.

IOMMU group 15
    [111d:8018] 05:00.0 PCI bridge: Integrated Device Technology, Inc. [IDT] PES12N3A PCI Express Switch (rev 0e)
IOMMU group 16
    [111d:8018] 06:02.0 PCI bridge: Integrated Device Technology, Inc. [IDT] PES12N3A PCI Express Switch (rev 0e)
    [8086:10d6] 07:00.0 Ethernet controller: Intel Corporation 82575GB Gigabit Network Connection (rev 02)
    [8086:10d6] 07:00.1 Ethernet controller: Intel Corporation 82575GB Gigabit Network Connection (rev 02)
IOMMU group 17
    [111d:8018] 06:04.0 PCI bridge: Integrated Device Technology, Inc. [IDT] PES12N3A PCI Express Switch (rev 0e)
    [8086:10d6] 08:00.0 Ethernet controller: Intel Corporation 82575GB Gigabit Network Connection (rev 02)
    [8086:10d6] 08:00.1 Ethernet controller: Intel Corporation 82575GB Gigabit Network Connection (rev 02)

 

 

pfsense XML, note that the addressing on the PCI bus may not lineup perfectly since I tried swapping the nic to another slot, but I have always made sure the proper pci slot is used in the config for where it really is.

<domain type='kvm'>
  <name>pfSense</name>
  <uuid>cd844801-ca21-0131-62c4-4e12cff1b863</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="FreeBSD" icon="freebsd.png" os="freebsd"/>
  </metadata>
  <memory unit='KiB'>1572864</memory>
  <currentMemory unit='KiB'>1572864</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>2</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='0'/>
    <vcpupin vcpu='1' cpuset='8'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-q35-2.7'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu>
    <topology sockets='1' cores='2' threads='1'/>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <model name='i82801b11-bridge'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <model name='pci-bridge'/>
      <target chassisNr='2'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:06:ec:c9'/>
      <source bridge='virbr0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' websocket='-1' listen='0.0.0.0' keymap='en-us'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </video>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x06' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x04' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x05' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x07' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x06' function='0x0'/>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x07' function='0x0'/>
    </memballoon>
  </devices>
</domain>
 

 

 

Attached is also a diagnostics file.  T"hanks for your help.

 

tower-diagnostics-20170306-0547.zip

[pfSense in VM with PCI passthrough issues]

Hello all,

 

I am new, but have searched the forums a bit before posting to see if I could find help already given, but nothing seems to stand out to me at the moment.

 

Been banging my head against the desk for a bit now, and thought it might be worth reaching out to the community here in case someone has any pointers.  I think the issue is more with FreeBSD that pfSense is based on, but could sure use any help I can get.  I also posted in the pfSense forums to see if that leads anywhere.

 

I picked up a used server and have verified all the hardware through CPU stress tests, RAM testing, and even put a load on all the network interfaces with no issues at all.

I am trying to run unRaid(6.3.2) as the main OS on a server with pfSense as a VM.  The server I have is intel and supports VT-d.  I have configured unRaid to pass through the entire PCI card I am using for a NIC (Dell/Intel Quad port gigabit nic YT674).  The interfaces show up and configure just fine, but I keep seeing this message on the console:
igb2: Watchdog timeout -- resetting
igb2: Queue(0) tdh = 32, hw tdt = 39
igb2: TX(0) desc avail = 0,Next TX to Clean = 0

The port goes down and back up at this point.  I have noticed I only get the watchdog error when the port is idle.  If I run a continuous ping, there are no watchdog errors.  BUT the pings are also very erratic. I am pinging another host off the same switch as the server and the response times vary from <1ms to +3000ms and even loses packets.

I have spun-up an instance of CentOS and Ubuntu VM and forwarded the PCI card the exact same way and there are no problems at all. Port stays up, all pings <1ms.  So I am inclined to think the host OS and the VM config is proper (but there is a chance I might be missing something, I am a little new).

 

 

lspci from centos:
[root@centostest ~]# lspci
00:00.0 Host bridge: Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller
00:01.0 VGA compatible controller: Red Hat, Inc. Device 0100 (rev 04)
00:07.0 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 (rev 03)
00:07.1 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 (rev 03)
00:07.2 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3 (rev 03)
00:07.7 USB controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 (rev 03)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 92)
00:1f.0 ISA bridge: Intel Corporation 82801IB (ICH9) LPC Interface Controller (rev 02)
00:1f.2 SATA controller: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] (rev 02)
00:1f.3 SMBus: Intel Corporation 82801I (ICH9 Family) SMBus Controller (rev 02)
01:00.0 PCI bridge: Red Hat, Inc. Device 0001
02:01.0 Ethernet controller: Red Hat, Inc Virtio network device
02:02.0 Communication controller: Red Hat, Inc Virtio console
02:03.0 SCSI storage controller: Red Hat, Inc Virtio block device
02:04.0 Ethernet controller: Intel Corporation 82575GB Gigabit Network Connection (rev 02)
02:05.0 Ethernet controller: Intel Corporation 82575GB Gigabit Network Connection (rev 02)
02:06.0 Ethernet controller: Intel Corporation 82575GB Gigabit Network Connection (rev 02)
02:07.0 Ethernet controller: Intel Corporation 82575GB Gigabit Network Connection (rev 02)
02:08.0 Unclassified device [00ff]: Red Hat, Inc Virtio memory balloon
 

 

I have followed the tutorial that uses the pci stub for the VID/PID and I also followed the youtube video (https://www.youtube.com/watch?v=n2OPfALLqRA) with no change.  I have tried several versions and variations of pfSense, FreeBSD as well as played with the VM settings.  I did notice the IOMMU group for the NIC had another entry with the ethernet ports, so I tried adding both VID/PID of the ethernet and this other thing to the vfio-pci.ids and still didn't change anything.

 

I am using a SuperMicro X8DTU-F motherboard with a YT674 Dell/Intel Quad gigabit ethernet NIC.

 

Any suggestions?  Thanks for any help.