pfSense in VM with PCI passthrough issues

Kunzy · March 5, 2017

Hello all,

I am new, but have searched the forums a bit before posting to see if I could find help already given, but nothing seems to stand out to me at the moment.

Been banging my head against the desk for a bit now, and thought it might be worth reaching out to the community here in case someone has any pointers. I think the issue is more with FreeBSD that pfSense is based on, but could sure use any help I can get. I also posted in the pfSense forums to see if that leads anywhere.

I picked up a used server and have verified all the hardware through CPU stress tests, RAM testing, and even put a load on all the network interfaces with no issues at all.

I am trying to run unRaid(6.3.2) as the main OS on a server with pfSense as a VM. The server I have is intel and supports VT-d. I have configured unRaid to pass through the entire PCI card I am using for a NIC (Dell/Intel Quad port gigabit nic YT674). The interfaces show up and configure just fine, but I keep seeing this message on the console:
igb2: Watchdog timeout -- resetting
igb2: Queue(0) tdh = 32, hw tdt = 39
igb2: TX(0) desc avail = 0,Next TX to Clean = 0

The port goes down and back up at this point. I have noticed I only get the watchdog error when the port is idle. If I run a continuous ping, there are no watchdog errors. BUT the pings are also very erratic. I am pinging another host off the same switch as the server and the response times vary from <1ms to +3000ms and even loses packets.

I have spun-up an instance of CentOS and Ubuntu VM and forwarded the PCI card the exact same way and there are no problems at all. Port stays up, all pings <1ms. So I am inclined to think the host OS and the VM config is proper (but there is a chance I might be missing something, I am a little new).

lspci from centos:
[root@centostest ~]# lspci
00:00.0 Host bridge: Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller
00:01.0 VGA compatible controller: Red Hat, Inc. Device 0100 (rev 04)
00:07.0 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 (rev 03)
00:07.1 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 (rev 03)
00:07.2 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3 (rev 03)
00:07.7 USB controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 (rev 03)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 92)
00:1f.0 ISA bridge: Intel Corporation 82801IB (ICH9) LPC Interface Controller (rev 02)
00:1f.2 SATA controller: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] (rev 02)
00:1f.3 SMBus: Intel Corporation 82801I (ICH9 Family) SMBus Controller (rev 02)
01:00.0 PCI bridge: Red Hat, Inc. Device 0001
02:01.0 Ethernet controller: Red Hat, Inc Virtio network device
02:02.0 Communication controller: Red Hat, Inc Virtio console
02:03.0 SCSI storage controller: Red Hat, Inc Virtio block device
02:04.0 Ethernet controller: Intel Corporation 82575GB Gigabit Network Connection (rev 02)
02:05.0 Ethernet controller: Intel Corporation 82575GB Gigabit Network Connection (rev 02)
02:06.0 Ethernet controller: Intel Corporation 82575GB Gigabit Network Connection (rev 02)
02:07.0 Ethernet controller: Intel Corporation 82575GB Gigabit Network Connection (rev 02)
02:08.0 Unclassified device [00ff]: Red Hat, Inc Virtio memory balloon

I have followed the tutorial that uses the pci stub for the VID/PID and I also followed the youtube video (https://www.youtube.com/watch?v=n2OPfALLqRA) with no change. I have tried several versions and variations of pfSense, FreeBSD as well as played with the VM settings. I did notice the IOMMU group for the NIC had another entry with the ethernet ports, so I tried adding both VID/PID of the ethernet and this other thing to the vfio-pci.ids and still didn't change anything.

I am using a SuperMicro X8DTU-F motherboard with a YT674 Dell/Intel Quad gigabit ethernet NIC.

Any suggestions? Thanks for any help.

2.6k · March 6, 2017

post your iommu groupings, pfense xml, and if your'e using ovmf - change to seabios and try again.

Kunzy · March 6, 2017

I have defaulted to always using seabios. I don't think I have successfully boot ovmf on any of my tests.

Here is the iommu groups. I have tried stubbing and using vfio-pci.ids for the nics and the other random device with no luck.

IOMMU group 15
   [111d:8018] 05:00.0 PCI bridge: Integrated Device Technology, Inc. [IDT] PES12N3A PCI Express Switch (rev 0e)
IOMMU group 16
   [111d:8018] 06:02.0 PCI bridge: Integrated Device Technology, Inc. [IDT] PES12N3A PCI Express Switch (rev 0e)
   [8086:10d6] 07:00.0 Ethernet controller: Intel Corporation 82575GB Gigabit Network Connection (rev 02)
   [8086:10d6] 07:00.1 Ethernet controller: Intel Corporation 82575GB Gigabit Network Connection (rev 02)
IOMMU group 17
   [111d:8018] 06:04.0 PCI bridge: Integrated Device Technology, Inc. [IDT] PES12N3A PCI Express Switch (rev 0e)
   [8086:10d6] 08:00.0 Ethernet controller: Intel Corporation 82575GB Gigabit Network Connection (rev 02)
   [8086:10d6] 08:00.1 Ethernet controller: Intel Corporation 82575GB Gigabit Network Connection (rev 02)

pfsense XML, note that the addressing on the PCI bus may not lineup perfectly since I tried swapping the nic to another slot, but I have always made sure the proper pci slot is used in the config for where it really is.

<domain type='kvm'>
<name>pfSense</name>
<uuid>cd844801-ca21-0131-62c4-4e12cff1b863</uuid>
<metadata>
<vmtemplate xmlns="unraid" name="FreeBSD" icon="freebsd.png" os="freebsd"/>
</metadata>
<memory unit='KiB'>1572864</memory>
<currentMemory unit='KiB'>1572864</currentMemory>
<memoryBacking>
<nosharepages/>
</memoryBacking>
<vcpu placement='static'>2</vcpu>
<cputune>
<vcpupin vcpu='0' cpuset='0'/>
<vcpupin vcpu='1' cpuset='8'/>
</cputune>
<os>
<type arch='x86_64' machine='pc-q35-2.7'>hvm</type>
<boot dev='hd'/>
</os>
<features>
<acpi/>
<apic/>
</features>
<cpu>
<topology sockets='1' cores='2' threads='1'/>
</cpu>
<clock offset='utc'>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='pit' tickpolicy='delay'/>
<timer name='hpet' present='no'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>restart</on_crash>
<devices>
<emulator>/usr/local/sbin/qemu</emulator>
<controller type='usb' index='0' model='ich9-ehci1'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci1'>
<master startport='0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci2'>
<master startport='2'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci3'>
<master startport='4'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
</controller>
<controller type='sata' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
</controller>
<controller type='pci' index='0' model='pcie-root'/>
<controller type='pci' index='1' model='dmi-to-pci-bridge'>
<model name='i82801b11-bridge'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
</controller>
<controller type='pci' index='2' model='pci-bridge'>
<model name='pci-bridge'/>
<target chassisNr='2'/>
<address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</controller>
<controller type='virtio-serial' index='0'>
<address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/>
</controller>
<interface type='bridge'>
<mac address='52:54:00:06:ec:c9'/>
<source bridge='virbr0'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/>
</interface>
<serial type='pty'>
<target port='0'/>
</serial>
<console type='pty'>
<target type='serial' port='0'/>
</console>
<channel type='unix'>
<target type='virtio' name='org.qemu.guest_agent.0'/>
<address type='virtio-serial' controller='0' bus='0' port='1'/>
</channel>
<input type='tablet' bus='usb'>
<address type='usb' bus='0' port='1'/>
</input>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<graphics type='vnc' port='-1' autoport='yes' websocket='-1' listen='0.0.0.0' keymap='en-us'>
<listen type='address' address='0.0.0.0'/>
</graphics>
<video>
<model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
</video>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
</source>
<address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0x06' slot='0x00' function='0x1'/>
</source>
<address type='pci' domain='0x0000' bus='0x02' slot='0x04' function='0x0'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
</source>
<address type='pci' domain='0x0000' bus='0x02' slot='0x05' function='0x0'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0x07' slot='0x00' function='0x1'/>
</source>
<address type='pci' domain='0x0000' bus='0x02' slot='0x06' function='0x0'/>
</hostdev>
<memballoon model='virtio'>
<address type='pci' domain='0x0000' bus='0x02' slot='0x07' function='0x0'/>
</memballoon>
</devices>
</domain>

Attached is also a diagnostics file. T"hanks for your help.

tower-diagnostics-20170306-0547.zip

Kunzy · March 7, 2017

A little bit of an update. I tried turning on the ACS override with no luck, and I tried turning on the unsafe interrupts just for kicks and giggles. Nothing seems to have an effect on it.

danioj · March 7, 2017

I only ever had pfSense working in a VM when I created bridges for each interface rather than passthrough and assigned each bridge to the VM (In the past you had to do this manually in the go script but now unRAID can manage this for you). There was no performance penalty for doing this in my testing (although this has been noted by other users).

Edited March 7, 2017 by danioj

NeoDude · March 7, 2017

Not sure if it'll help at all but here's my working PfSense XML...

<domain type='kvm' id='1'>
  <name>pfSense</name>
  <uuid>6bfe1bd6-5089-c4c1-4932-403205c0c78a</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="FreeBSD" icon="freebsd.png" os="freebsd"/>
  </metadata>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>4</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='2'/>
    <vcpupin vcpu='1' cpuset='3'/>
    <vcpupin vcpu='2' cpuset='18'/>
    <vcpupin vcpu='3' cpuset='19'/>
  </cputune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-q35-2.5'>hvm</type>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-passthrough'>
    <topology sockets='1' cores='2' threads='2'/>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/user/vDisks/pfSense/vdisk1.img'/>
      <backingStore/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <alias name='virtio-disk2'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/ISOs/pfSense-CE-2.3.2-RELEASE-amd64.iso'/>
      <backingStore/>
      <target dev='hda' bus='sata'/>
      <readonly/>
      <boot order='2'/>
      <alias name='sata0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <alias name='usb'/>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <alias name='usb'/>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <alias name='usb'/>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <controller type='sata' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <model name='i82801b11-bridge'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <model name='pci-bridge'/>
      <target chassisNr='2'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:09:99:67'/>
      <source bridge='br0'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/0'/>
      <target port='0'/>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/0'>
      <source path='/dev/pts/0'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-1-pfSense/org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <alias name='input0'/>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'>
      <alias name='input1'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input2'/>
    </input>
    <graphics type='vnc' port='5900' autoport='yes' websocket='5700' listen='0.0.0.0' keymap='en-us'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </video>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x82' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x04' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x82' slot='0x00' function='0x1'/>
      </source>
      <alias name='hostdev1'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x05' function='0x0'/>
    </hostdev>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x06' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='none' model='none'/>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+0:+100</label>
    <imagelabel>+0:+100</imagelabel>
  </seclabel>
</domain>

I also notice that you have Core 0 assigned, I would change this as UnRaid prefers to use Core 0.

Kunzy · March 8, 2017

Danioj: If I can't figure this out, that might be what I have to do. I liked the idea of the pci passthrough just a cleaner and more complete solution. It also played a large part in my hardware choice (vt-d).

Neodude: Thanks for the XML. It looks like you are passing through a dual port yes?

NeoDude · March 8, 2017

Yep, It's a Dual Intel NIC that I have stubbed using ...

pci-stub.ids=8086:105e

within my Sys Config.

DZMM · March 11, 2017

On 07/03/2017 at 4:58 AM, danioj said:

I only ever had pfSense working in a VM when I created bridges for each interface rather than passthrough and assigned each bridge to the VM (In the past you had to do this manually in the go script but now unRAID can manage this for you). There was no performance penalty for doing this in my testing (although this has been noted by other users).

Hi I'm having similar problems passing a nice through to pfsense. Am i understanding this correctly and you created two virtual nics, and managed to get all wan traffic going through pfsense with one nic?

danioj · March 12, 2017

16 hours ago, DZMM said:

Hi I'm having similar problems passing a nice through to pfsense. Am i understanding this correctly and you created two virtual nics, and managed to get all wan traffic going through pfsense with one nic?

Firstly, I used a 2 port Intel NIC card.

I first ensured that the NIC was recognised by unRAID (Interestingly I found that with some INTEL NIC's their bios crapped out when I did a "Reboot". So I had to change my strategy when updating or resetting the server etc to shutting down and starting up from cold).

Then I created a bridge for each port on the NIC. I then created a virtual NIC for each port for the pfsense VM mapped to those bridges I had just created.

Then I set up pfSense as normal. Easy.

What I ended up with (I only used a 2 port card) was Modem into one port of NIC 16 port switch into the other. Then unRAID, wireless AP and other LAN devices into the switch.

DZMM · March 12, 2017

18 minutes ago, danioj said:

Firstly, I used a 2 port Intel NIC card.

I first ensured that the NIC was recognised by unRAID (Interestingly I found that with some INTEL NIC's their bios crapped out when I did a "Reboot". So I had to change my strategy when updating or resetting the server etc to shutting down and starting up from cold).

Then I created a bridge for each port on the NIC. I then created a virtual NIC for each port for the pfsense VM mapped to those bridges I had just created.

Then I set up pfSense as normal. Easy.

What I ended up with (I only used a 2 port card) was Modem into one port of NIC 16 port switch into the other. Then unRAID, wireless AP and other LAN devices into the switch.

I'm wondering if I'm having the same reboot problems - did your card become unusable when rebooted?

I think i know how to do what you did - did you create a bridge for each nic in network settings? Why did you do this way rather than passing through?

Thanks

2.6k · March 12, 2017

1 hour ago, danioj said:

Firstly, I used a 2 port Intel NIC card.

I first ensured that the NIC was recognised by unRAID (Interestingly I found that with some INTEL NIC's their bios crapped out when I did a "Reboot". So I had to change my strategy when updating or resetting the server etc to shutting down and starting up from cold).

Then I created a bridge for each port on the NIC. I then created a virtual NIC for each port for the pfsense VM mapped to those bridges I had just created.

Then I set up pfSense as normal. Easy.

What I ended up with (I only used a 2 port card) was Modem into one port of NIC 16 port switch into the other. Then unRAID, wireless AP and other LAN devices into the switch.

This makes sense. If you wanted to use a card not supported in pfense (or any other vm) like the mellanox connect2, you could either compile the driver in your own pfsense build, or make a bridge in the server to the card, and pass the virtual adapter to pfense. Smart.

unRaid doen't have this option in the GUI right?

danioj · March 16, 2017

On 12/03/2017 at 11:27 AM, DZMM said:

I'm wondering if I'm having the same reboot problems - did your card become unusable when rebooted?

I think i know how to do what you did - did you create a bridge for each nic in network settings? Why did you do this way rather than passing through?

Thanks

Yes, the card became completely unusable (ie not recognised) on reboot. However, from a cold startup, it was fine.

I think it worth noting that this was with a Supermicro board (https://www.supermicro.com/products/motherboard/xeon/c220/x10sl7-f.cfm) and I found elsewhere on the web that others had experienced similar issues. The Supermicro engineer I spoke to blamed the C222 Intel chipset - but I am not sure.

As for configuration, it was one bridge per port. I did this as I had problems passing the card through to the VM. So it went like this:

Onboard NIC: eth0 - part of bond0 used by unRAID

Onboard NIC: eth1 - part of bond0 used by unRAID

NIC on Intel Server Card: eth2 - create bridge in unRAID br1

NIC on Intel Server Card: eth3 - create bridge in unRAID br2

*Note, unRAID OS can see all the NIC's.

Create 2x virtual NIC for pfSense VM and assign them to br1 and br2. Done.

On 12/03/2017 at 0:33 PM, 1812 said:

This makes sense. If you wanted to use a card not supported in pfense (or any other vm) like the mellanox connect2, you could either compile the driver in your own pfsense build, or make a bridge in the server to the card, and pass the virtual adapter to pfense. Smart.

unRaid doen't have this option in the GUI right?

unRAID now has the ability to configure bridges in the GUI - Network Settings - so we don't have to do this in the Go file anymore! In fact, doing it in the Go file no longer works at all AFAIK.

2.6k · March 16, 2017

7 hours ago, danioj said:

unRAID now has the ability to configure bridges in the GUI - Network Settings - so we don't have to do this in the Go file anymore! In fact, doing it in the Go file no longer works at all AFAIK.

I see it, though it would be nice if the gui labeled them after creation. not a big deal to remember, but yet, another thing that you have to remember with unRaid.

DZMM · March 16, 2017

12 hours ago, danioj said:

Yes, the card became completely unusable (ie not recognised) on reboot. However, from a cold startup, it was fine.

Thanks - I got a new dual nic card that's working fine and I'm setting up pfsense. so far so good :-)

danioj · March 16, 2017

9 hours ago, 1812 said:

I see it, though it would be nice if the gui labeled them after creation. not a big deal to remember, but yet, another thing that you have to remember with unRaid.

I agree. In my Go file script I had them all named nicely like "brwan0 and brlan0 etc" but meh, not worth the energy to ask for naming to be allowed IMHO.

danioj · March 16, 2017

4 hours ago, DZMM said:

Thanks - I got a new dual nic card that's working fine and I'm setting up pfsense. so far so good :-)

Excellent. Happy days. Did you pass it through to the VM or use the method I used?

if you did pass it through, share your card specs and VM config for others please! ?

DZMM · March 17, 2017

On 3/16/2017 at 8:17 PM, danioj said:

Excellent. Happy days. Did you pass it through to the VM or use the method I used?

if you did pass it through, share your card specs and VM config for others please! ?

I've got the supermicro AOC-SGP-I2 card - it's one that pfsense sell themselves https://store.pfsense.org/AOC-SGP-I2/ but I got it for £60 on eBay.

There's nothing special in my VM - I just located the card in System Devices, stubbed it and then selected the adapters in the VM:

<domain type='kvm' id='1'>
  <name>Baymax</name>
  <uuid>dfbe0db8-84b8-f5a8-217b-be2d6b160dbd</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="FreeBSD" icon="freebsd.png" os="freebsd"/>
  </metadata>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>2</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='1'/>
    <vcpupin vcpu='1' cpuset='15'/>
  </cputune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-q35-2.7'>hvm</type>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-passthrough'>
    <topology sockets='1' cores='1' threads='2'/>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/disks/sm961/domains/Baymax/vdisk1.img'/>
      <backingStore/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <alias name='virtio-disk2'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/disks/sm961/isos/pfSense-CE-2.3.3-RELEASE-amd64.iso'/>
      <backingStore/>
      <target dev='hda' bus='sata'/>
      <readonly/>
      <boot order='2'/>
      <alias name='sata0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <alias name='usb'/>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <alias name='usb'/>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <alias name='usb'/>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <controller type='sata' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <model name='i82801b11-bridge'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <model name='pci-bridge'/>
      <target chassisNr='2'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:0f:10:40'/>
      <source bridge='br0'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/0'/>
      <target port='0'/>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/0'>
      <source path='/dev/pts/0'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-1-Baymax/org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <alias name='input0'/>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'>
      <alias name='input1'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input2'/>
    </input>
    <graphics type='vnc' port='5900' autoport='yes' websocket='5700' listen='0.0.0.0' keymap='en-gb'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </video>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0b' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x04' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0b' slot='0x00' function='0x1'/>
      </source>
      <alias name='hostdev1'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x05' function='0x0'/>
    </hostdev>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x06' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='none' model='none'/>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+0:+100</label>
    <imagelabel>+0:+100</imagelabel>
  </seclabel>
</domain>

Might have overdone it with Ram and HDD space, but I wasn't sure how much I'd need and luckily I've got enough to spare at the moment

Kunzy · March 23, 2017

Hello,

Just wanted to post an update and say I've given up on the old NIC and picked up a new one. The new one works perfectly with simply stubbing the new card and checking the boxes in the VM editor. ACS override is set to no. I followed the guide here and it worked like a charm:

I booted up pfSense, plugged in the cord and am getting <1ms ping response times on other parts of the network to my pfSense install and no packets dropped. There are also no more watchdog errors. I picked up a I350-T4 NIC and has the following details in lspci:

lspci

04:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
04:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
04:00.2 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
04:00.3 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)

lspci -n
04:00.0 0200: 8086:1521 (rev 01)
04:00.1 0200: 8086:1521 (rev 01)
04:00.2 0200: 8086:1521 (rev 01)
04:00.3 0200: 8086:1521 (rev 01)

The VM XML is of no interest as no manual configurations were done. I appreciate all the suggestions and help.

Edited March 23, 2017 by Kunzy

casperse · February 16, 2020

Hi All

Just found this post which looks allot like mine :-)

https://forums.unraid.net/topic/71071-video-guide-a-comprehensive-guide-to-pfsense-both-unraid-vm-and-physical/?do=findComment&comment=822768

But I did go out and bought a Intel Pro 1000 VT Quad Port NIC (EXPI9404VTG1P20) so I wouldn't have any problems LOL

So now I wonder if its about the firmware of the card? (Worked perfectly in Unraid before stubbing it and adding it to my VM)

Any suggestion?

pfSense in VM with PCI passthrough issues

Recommended Posts

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Join the conversation