pfSense VM lost access to Web GUI with no reason


PzrrL

Recommended Posts

 I have no idea what is going on. I followed Spaceinvader One pfSense Part 3 for the installation. I followed until around 19:57 (for changing from HTTPS to HTTP), then it suddenly doesn't work. During the setup wizard I used both dhcp and static ip and it was fine. I have no way to access the web GUI after changing to HTTP (I am not sure is this the point that I start to lose access to GUI). I think the problem does not related to HTTP or HTTPS as I remembered that after a restart of this fresh pfsense, I can no longer get access to the Web GUI. I actually have a workable pfSense before having the NIC in a different PCIe slot, but then I realize I need that slot for the other purpose, so I swap to another slot and reconfigure and reinstall the pfsense.

 

I tried reboot the VM, reinstall the VM, set static IP from cmd, enable or disable DHCP Server from pfSense, connect to pfSense GUI from my PC via router, directly connect my PC to the LAN port of pfSense...all of these don't work. I can't see the static IP on the router either, so I really need some guidance for this...

 

My syslinux config:

append vfio-pci.ids=10ec:8168,8086:1502 initrd=/bzroot

My IOMMU group:

IOMMU group 0:	[8086:0158] 00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v2/Ivy Bridge DRAM Controller (rev 09)
IOMMU group 1:	[8086:0151] 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09)
[8086:0155] 00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09)
[1000:0072] 02:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)
IOMMU group 2:	[8086:015d] 00:06.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09)
[1b21:1806] 03:00.0 PCI bridge: ASMedia Technology Inc. Device 1806 (rev 01)
[1b21:1806] 04:00.0 PCI bridge: ASMedia Technology Inc. Device 1806 (rev 01)
[1b21:1806] 04:02.0 PCI bridge: ASMedia Technology Inc. Device 1806 (rev 01)
[1b21:1806] 04:06.0 PCI bridge: ASMedia Technology Inc. Device 1806 (rev 01)
[1b21:1806] 04:0e.0 PCI bridge: ASMedia Technology Inc. Device 1806 (rev 01)
[10ec:8168] 05:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 0c)
[10ec:8168] 06:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 0c)
[10ec:8168] 07:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 0c)
[10ec:8168] 08:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 0c)
IOMMU group 3:	[8086:1502] 00:19.0 Ethernet controller: Intel Corporation 82579LM Gigabit Network Connection (Lewisville) (rev 05)
IOMMU group 4:	[8086:1c2d] 00:1a.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 (rev 05)
IOMMU group 5:	[8086:1c10] 00:1c.0 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 1 (rev b5)
IOMMU group 6:	[8086:1c18] 00:1c.4 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 5 (rev b5)
IOMMU group 7:	[8086:1c26] 00:1d.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #1 (rev 05)
IOMMU group 8:	[8086:244e] 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a5)
[102b:0532] 0b:03.0 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200eW WPCM450 (rev 0a)
IOMMU group 9:	[8086:1c54] 00:1f.0 ISA bridge: Intel Corporation C204 Chipset LPC Controller (rev 05)
[8086:1c02] 00:1f.2 SATA controller: Intel Corporation 6 Series/C200 Series Chipset Family 6 port Desktop SATA AHCI Controller (rev 05)
[8086:1c22] 00:1f.3 SMBus: Intel Corporation 6 Series/C200 Series Chipset Family SMBus Controller (rev 05)
IOMMU group 10:	[1912:0015] 09:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02)
IOMMU group 11:	[8086:10d3] 0a:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection

My pfSense VM XML:

<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm' id='1'>
  <name>pfSense</name>
  <uuid>18xxxxx-0000-aaaa-bbbb-cccdadee</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="FreeBSD" icon="pfsense.png" os="freebsd"/>
  </metadata>
  <memory unit='KiB'>3145728</memory>
  <currentMemory unit='KiB'>3145728</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>2</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='3'/>
    <vcpupin vcpu='1' cpuset='7'/>
  </cputune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-q35-3.1'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/18574f37-692d-a5eb-efd0-e4c5ca3e29c3_VARS-pure-efi.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-passthrough' check='none'>
    <topology sockets='1' cores='1' threads='2'/>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/pfSense-CE-2.4.4-RELEASE-p3-amd64.iso'/>
      <backingStore/>
      <target dev='hda' bus='sata'/>
      <readonly/>
      <boot order='2'/>
      <alias name='sata0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='writeback'/>
      <source file='/mnt/user/domains/pfSense/vdisk1.img'/>
      <backingStore/>
      <target dev='hdc' bus='sata'/>
      <boot order='1'/>
      <alias name='sata0-0-2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='2'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <alias name='usb'/>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <alias name='usb'/>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <alias name='usb'/>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <controller type='sata' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x11'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x12'/>
      <alias name='pci.3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-to-pci-bridge'>
      <model name='pcie-pci-bridge'/>
      <alias name='pci.4'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0x13'/>
      <alias name='pci.5'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0x14'/>
      <alias name='pci.6'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
    </controller>
    <controller type='pci' index='7' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='7' port='0x15'/>
      <alias name='pci.7'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/>
    </controller>
    <controller type='pci' index='8' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='8' port='0x16'/>
      <alias name='pci.8'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x6'/>
    </controller>
    <controller type='pci' index='9' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='9' port='0x17'/>
      <alias name='pci.9'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x7'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </controller>
    <serial type='pty'>
      <source path='/dev/pts/0'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/0'>
      <source path='/dev/pts/0'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-1-pfSense/org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <alias name='input0'/>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'>
      <alias name='input1'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input2'/>
    </input>
    <graphics type='vnc' port='5900' autoport='yes' websocket='5700' listen='0.0.0.0' keymap='en-us'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </video>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x00' slot='0x19' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x01' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev1'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev2'/>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev3'/>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev4'/>
      <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+0:+100</label>
    <imagelabel>+0:+100</imagelabel>
  </seclabel>
</domain>

Please help and advise thankss!!!

Link to comment

@PzrrL Changing the PCI slot will in most cases change the IOMMU groupings. In your case group 2 with the 4 nics "Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller" also has 5x "ASMedia Technology Inc. Device" (Asmedia usually USB controllers or controllers for SATA or IDE devices) in it and a "Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port".

 

I guess they are separate devices which can cause issues if they are not separated from the device you wanna passthrough. It might work on first boot of the the VM and the initial setup, but a reboot of the VM can end in a state where the passed through devices aren't reset properly. Only a restart of the whole server will help in this case. In general if you wanna passthrough a PCI device, you have to pass through every device in the same group. Depending of the devices in the group sometimes it works not handing over all devices because maybe there are devices in the same group that aren't used by anything else BUT you can't really predict what else in the backround triggers some device calls and might break your passed through devices.

 

3 Options:

  • Check how the groupings are if you place the network card in a different slot
  • passthrough all devices from that group (be careful, passing through the wrong devices can crash the whole server)
  • use the ACS override setting and check if you can split up the group
Link to comment
23 minutes ago, bastl said:

5x "ASMedia Technology Inc. Device" (Asmedia usually USB controllers or controllers for SATA or IDE devices)

Thanks for your reply! I suspect these controllers are actually belonging to the NIC which is in the same group. I only have one NIC with 4ports but before the changing of slots, they are still in the same group.

 

24 minutes ago, bastl said:

passthrough all devices from that group (be careful, passing through the wrong devices can crash the whole server)

You said I should passthrough the whole group, but for my group 2, all of them except the ports are PCI bridge, and I didn't put that PCI bridge's id into the syslinux config to stub it. Therefore I can't find them in the VM setting. Should I also stub all the PCI bridge (both ASMedia Technology Inc. Device and a "Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port")?

 

28 minutes ago, bastl said:

but a reboot of the VM can end in a state where the passed through devices aren't reset properly. Only a restart of the whole server will help in this case. 

I am not sure but I don't think rebooting the whole server is helping either.

 

29 minutes ago, bastl said:

use the ACS override setting and check if you can split up the group

Should this only be a testing method, or I can use it in long run? I know this would be the last resort.

 

Thanks!!!

Link to comment
3 minutes ago, PzrrL said:

You said I should passthrough the whole group, but for my group 2, all of them except the ports are PCI bridge, and I didn't put that PCI bridge's id into the syslinux config to stub it. Therefore I can't find them in the VM setting. Should I also stub all the PCI bridge (both ASMedia Technology Inc. Device and a "Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port")?

The XEON express root port definitly isn't part of your PCI device you wanna passthrough therefore my warning. Stubing it can already break something on your server. Same for the ASMedia. As long as you're not 100% sure where these devices come from, better don't stub or pass them through. You can remove the ethernet card and check if these devices also disappear. Don't only check group 2, search the complete device list if they still exist.

11 minutes ago, PzrrL said:

Should this only be a testing method, or I can use it in long run? I know this would be the last resort.

A lot of users mostly on mainstream hardware using this to be able to passthrough devices which are otherwise for example behind the chipset and grouped together with other devices. There is a slight chance that something become unstable by using it, but without breaking the groupings and passing a device which isn't in it's own group it's unstable in most cases anyways. It always depends on the setup. I use it for 2 years to be able to separate my onboard audio controller and one of my onboard USB controllers for passthrough. Often different motherboard BIOS versions produce different groupings, thats also a thing that you could test.

  • Thanks 1
Link to comment
15 minutes ago, bastl said:

The XEON express root port definitly isn't part of your PCI device you wanna passthrough therefore my warning.

So how should I break the IOMMU group 2 in order to get rid of this XEON express root port? ACS override?

 

In this video (Spaceinvader One pfSense Part 3) 7m54s, he said the PCI bridge usually does not matter and would passthrough anyway, why is this the case? 

 

And one more thing, is my concept correct that in order to passthrough something, I have to stub it first?

Edited by PzrrL
More info on the link
Link to comment
16 minutes ago, PzrrL said:

ACS override?

Try it out if it works for you. It's hard to predict. Each setup is different and for some it helps and some users have the same groupings as without.

21 minutes ago, PzrrL said:

And one more thing, is my concept correct that in order to passthrough something, I have to stub it first?

No you don't have to stub it. You only have to do this to prevent unraid to initialize it at server boot. For example you have only 1 GPU in your system and want to prevent Unraid to pick it up for itself. Some cards that at boot Unraid uses for itself can be handed over to a VM for others it doesn't work therefore you have to stub it. Without stubbing them you won't see them in the "Other PCI Devices" section when creating a VM but you're still be able to add them to the xml by hand. I think SpaceInvader shows this in a couple videos.

Link to comment

@bastl Hi, sorry I was quite busy these few days. I finally tried every PCIe Card in different slots, and I get the following result:

 

1. Without any PCIe card:

531645222_NoPCI.thumb.png.a6efff21d17e1ca1e2f78f9044f88248.png

 

2. Only LSI in the 2nd slot from top:

1089292117_OnlyLSI.thumb.png.83e45d69fe24750bd5b2cba1115785d6.png

 

3. Only USB 3.0 in 4th slot (last slot)

31017610_OnlyUSB3.0.thumb.png.7093eb51913c49189d44947045709e22.png

 

4. Only NIC in the 3rd slot

1407413055_OnlyNIC(in3rdslot).thumb.png.b6f070238921b6057c93d269286b9c13.png

 

5. Only NIC in the 4th slot (last slot)

162769482_OnlyNIC(in4thslot).thumb.png.f241146d378f666a0eea8d1c9aa08829.png

 

The 4th case is actually the current situation without LSI. Interestingly, the 5th case breaks all the NIC ports to different IOMMU group, and I am not sure if it is due to the following setting I made in the BIOS before:

- I changed PCI Express Port - Gen X from [Auto] to [Gen2]

1322320305_PCIExpressPortGenX.jpg.df740ebfaf9fb18e8392853ae5d9e65c.jpg

 

I am not sure if this affects all the PCIe slots, but the reason I need to change this is that when I am placing the card in the 2nd slot, my NIC cannot be detected, so I changed from [Auto] to [Gen2], and also [Auto] to [Enabled] for PCI Express Port. If I change to Gen3, my NIC is not detected in slot 2 as well. Besides, I don't have to change this setting if I am putting the NIC in the 3rd and 4th slot as I recall.

Edit: I am not sure if it was slot 2 or slot 1, but it should be slot 1 now cuz I am testing the NIC on slot 1 and 2. I which to Auto and it is detected on slot 2.

 

I think this BIOS option only affects the top 2 slots in the MB, please correct me if not, and actually I am not sure about what is the effect of this option. FYI:

imageproxy.php?img=&key=e5eec7c5c933ca161364335310_MBlayout.thumb.png.96849712ed85ee86f888655ee3650de7.png

 

Sorry for having so many questions. Is this BIOS option the reason affecting my main problem of the 1st post? And would you please advise me on how should I place the cards or what should I do?

 

N.B.: The first slot is reserved for using an extension cable to plug in my GPU. (I am going to try to use a x8 to x16 extension, not sure if it works or not).

 

btw, My current BIOS version is 2.3 which supports PCIe 3.0. It is not supported in version 1.x.

 

(I didn't try the ACS patch yet)

 

 

Edited by PzrrL
Add desc for BIOS setting
Link to comment

The picture for case 5 "Only NIC in the 4th slot (last slot)" looks fine to me. But keep in mind by adding devices in the other slots, it might be possible the groupings will change. It's possible slot 3 and 4 are grouped together. Test what happens if you add the USB card in 3rd slot whilst the 4th is populated with the 4 port nic. Is the USB controller in it's own group? Fine. If not, change the slots.

 

I know it's painful, but often the only way to check how the board behaves with different combinations of populated slots. Add your devices one by one and check how the IOMMU groups change.

 

Populating certain slots can also end up in a scenario where for example some sata ports or usb headers will be disabled. Therefore you have to check the manual for your board which devices share ressources. It's different for every board. It's often hard to find this info. Sometimes it's only mentioned in a tiny small text an can quickly be overseen.

 

36 minutes ago, PzrrL said:

- I changed PCI Express Port - Gen X from [Auto] to [Gen2]

Auto will detect the link speed of the connected devices and will set the optimal speeds for the slot. By forcing it into a lower speed can have the benefit of not wasting PCI lanes and use them for other slots which on auto might end up in grouped devices. Again, this is different for every manufacturer and often different BIOS version behave differently. It's all upon you to test it to find the best solution for your setup.

Link to comment
On 11/26/2019 at 7:10 PM, bastl said:

It might work on first boot of the the VM and the initial setup, but a reboot of the VM can end in a state where the passed through devices aren't reset properly. Only a restart of the whole server will help in this case.

Update on pfSense: If I use the 5th case to passthrough my NIC, it starts normally and I can access the web gui. After a restart of pfSense using VM manager, I can no longer access the Web GUI as you mentioned.

 

If I restart pfsense with the console in VNC, it still works after restart.

If I halt the pfsense in VNC (I guess it is the same as stopping from the VM manger) or restart from the VM manger, it doesn't work anymore until a full shutdown and startup (not restart) of server.

 

May I know if there is any fix for it? Thanks!

 

 

Link to comment
27 minutes ago, bastl said:

The picture for case 5 "Only NIC in the 4th slot (last slot)" looks fine to me. But keep in mind by adding devices in the other slots, it might be possible the groupings will change. It's possible slot 3 and 4 are grouped together. Test what happens if you add the USB card in 3rd slot whilst the 4th is populated with the 4 port nic. Is the USB controller in it's own group? Fine. If not, change the slots

 

The IOMMU group has the following grouping with case: 

IOMMU group 0:	[8086:0158] 00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v2/Ivy Bridge DRAM Controller (rev 09)
IOMMU group 1:	[8086:0151] 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09)
[8086:0155] 00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09)
[1000:0072] 02:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)
IOMMU group 2:	[8086:015d] 00:06.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09)
[1912:0015] 03:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02)
IOMMU group 3:	[8086:1502] 00:19.0 Ethernet controller: Intel Corporation 82579LM Gigabit Network Connection (Lewisville) (rev 05)
IOMMU group 4:	[8086:1c2d] 00:1a.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 (rev 05)
IOMMU group 5:	[8086:1c10] 00:1c.0 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 1 (rev b5)
IOMMU group 6:	[8086:1c18] 00:1c.4 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 5 (rev b5)
IOMMU group 7:	[8086:1c26] 00:1d.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #1 (rev 05)
IOMMU group 8:	[8086:244e] 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a5)
[102b:0532] 0b:03.0 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200eW WPCM450 (rev 0a)
IOMMU group 9:	[8086:1c54] 00:1f.0 ISA bridge: Intel Corporation C204 Chipset LPC Controller (rev 05)
[8086:1c02] 00:1f.2 SATA controller: Intel Corporation 6 Series/C200 Series Chipset Family 6 port Desktop SATA AHCI Controller (rev 05)
[8086:1c22] 00:1f.3 SMBus: Intel Corporation 6 Series/C200 Series Chipset Family SMBus Controller (rev 05)
IOMMU group 10:	[1b21:1806] 04:00.0 PCI bridge: ASMedia Technology Inc. Device 1806 (rev 01)
IOMMU group 11:	[1b21:1806] 05:00.0 PCI bridge: ASMedia Technology Inc. Device 1806 (rev 01)
IOMMU group 12:	[1b21:1806] 05:02.0 PCI bridge: ASMedia Technology Inc. Device 1806 (rev 01)
IOMMU group 13:	[1b21:1806] 05:06.0 PCI bridge: ASMedia Technology Inc. Device 1806 (rev 01)
IOMMU group 14:	[1b21:1806] 05:0e.0 PCI bridge: ASMedia Technology Inc. Device 1806 (rev 01)
IOMMU group 15:	[10ec:8168] 06:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 0c)
IOMMU group 16:	[10ec:8168] 07:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 0c)
IOMMU group 17:	[10ec:8168] 08:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 0c)
IOMMU group 18:	[10ec:8168] 09:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 0c)
IOMMU group 19:	[8086:10d3] 0a:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection

 

IOMMU group 1 has the LSI card in slot 2 (02:00.0).

IOMMU group 2 has the USB 3.0 card in slot 3 (03:00.0).

Group 10-18 has the NIC card in slot 4 (last slot).

 

From my observation, when plugging in PCIe card in slot 2, it will go to group 1, card in slot 3 will go to group 2.

 

34 minutes ago, bastl said:

By forcing it into a lower speed can have the benefit of not wasting PCI lanes and use them for other slots which on auto might end up in grouped devices

Thanks for this great advice!

Link to comment
9 minutes ago, PzrrL said:

If I halt the pfsense in VNC (I guess it is the same as stopping from the VM manger) or restart from the VM manger, it doesn't work anymore until a full shutdown and startup (not restart) of server.

Halt in pfsense = shutdown

For me it's the same as i STOP it from unraid ui. Both are working fine.

 

Restart from within Pfsense and Unraid both working fine. Make sure you have the following part for "poweroff/reboot/on_crash" in your XML bellow the cpu mode section. Should be their on default.

  <cpu mode='host-passthrough' check='none'>
    <topology sockets='1' cores='2' threads='1'/>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>

 

Don't passthrough group 10-14 "ASMedia Technology Inc. Device 1806". Only use the ethernet controllers for passthrough if you aren't already.

 

Link to comment
11 minutes ago, bastl said:

Restart from within Pfsense and Unraid both working fine. Make sure you have the following part for "poweroff/reboot/on_crash" in your XML bellow the cpu mode section. Should be their on default

This part is already there in the XML.

 

11 minutes ago, bastl said:

Don't passthrough group 10-14 "ASMedia Technology Inc. Device 1806". Only use the ethernet controllers for passthrough if you aren't already

And I didn't passthrough group 10-14 too. I didn't stub the ID of group 10-14 and it is not in the "Other PCI Devices" of the VM config.

 

That is, I already have the part in XML and didn't passthrough group 10-14, still the problem exists. (Problem: after restart, I can't access the web GUI).

Edited by PzrrL
Add info to the problem
Link to comment

Are you connecting from the WAN or the LAN side? Default Pfsense prevents you from having access from the WAN side. This shouldn't been changed for security reasons. Access to the web interface is limited to the LAN side. Make sure your network is setup correctly. WAN and LAN not on the same subnet!!!

Link to comment

I connected the LAN port (configured in pfSense) of the NIC to my main router's LAN port (which as DHCP), and I am accessing the Web GUI from the WiFi on that router. My computer's ip is 10.1.1.12. I have set up a static DHCP to the pfSense's LAN port's MAC Address.

1406645257_Screenshot2019-11-28at21_20_23.png.b3b1f2c59fb9645e37f25f8d9a0f75f9.png 

 

I haven't connect and cable to the wan port yet because I am just doing the config on the pfSense (not sure if this is the problem)

I am not sure if you are saying this is not the same subnet🤔Sorry I am a bit dumb.

 

Even when I connect the pfSense's LAN port directly to my computer and enable the DHCP on this LAN port, I still cannot access the Web GUI. (Mine is a macbook and it said "Thunderbolt Ethernet has a self-assigned IP address and will not be able to connect to the Internet." It is having a self-assigned IP of 169.254.131.242.

Edited by PzrrL
Link to comment

@bastl Sorry for this mess. Or maybe let me tell you the current situation.

 

My main router has a network of 10.1.1.1/24

I am configuring pfSense and will eventually replace the main router with my pfSense. Therefore, I am giving 10.1.1.4 to my pfSense by plugging in a cable to the LAN port of both my router to the pfSense's LAN port, and want to connect to my pfSense from my Macbook (10.1.1.12), which is connected to the WiFi of the router.

 

No matter if I connect to the pfSense via the router or directly from my Macbook, the Web GUI doesn't show up. I am kind of lost now🤦‍♂️

Link to comment

Pfsense runs it's own DHCP server on the LAN side. There is your problem. The router also has a DHCP server running on the same net. Plugin the Lan from the router to the WAN of Pfsense and change the IP from the LAN on Pfsense to something different than 10.1.1.0/24. Set it for exapmle to 192.168.0.1/24 for Pfsense LAN side. You should get an IP in that range connecting to that port.

 

The WAN on Pfsense should be set to get an IP via DHCP. Should be the standard config. No matter what you set in front of the WAN (Router, Modem) it should give Pfsense an IP.

Link to comment

I do get an IP for my WAN.

 

11 minutes ago, bastl said:

DHCP server on the LAN side. There is your problem. The router also has a DHCP server running on the same net

I understand this part, so I tried both enabled DHCP and disabled DHCP on the LAN side from the pfSense, still not working.

 

11 minutes ago, bastl said:

Plugin the Lan from the router to the WAN of Pfsense and change the IP from the LAN on Pfsense to something different than 10.1.1.0/24. Set it for exapmle to 192.168.0.1/24 for Pfsense LAN side. You should get an IP in that range connecting to that port.

I actually tried this setup at the very beginning. And now is forgotten LOL.

 

But still, after following your setup, it doesn't work.

544605092_Screenshot2019-11-28at21_55_36.thumb.png.4cb54258e3d717a3524f034164f8865b.png

Then the range is 192.168.0.100-192.168.0.200

 

After plugging in a cable from pfSense's LAN cable to my macbook, I still get a self-assigned IP.

Edited by PzrrL
Link to comment

Manually set an IP for you LAN port on your mac to 192.168.0.5 for example, gateway to 192.168.0.1, DNS server to 192.168.0.1. Not so familiar with MacOS how to do it or if there is something special. But you should be able to ping Pfsense on the LAN now. Disable the Wifi to be make sure you only have 1 active network connection on the Mac.

Link to comment
8 minutes ago, bastl said:

Have you restarted Pfsense after changing the IPs and the DHCP

No, I didnt. I thought it is ok not to do so as it didn't ask me to do so.

 

Now I shut down the unraid and started again. Connecting my mac directly from the pfsense's LAN, I still cannot get an IP, and cannot ping the pfsense when setting a manual ip. bimageproxy.php?img=&key=e5eec7c5c933ca16ut the console suddenly has the following "re3: watchdog timeout". 

 

154818623_Screenshot2019-11-28at22_40_12.thumb.png.42b8ae5e70e8e683bf10a0d2a66d281d.png

 

I also get this message when rebooting and at the line: Starting DNS Resolver...re3 watchdog timeout

Edited by PzrrL
Link to comment

Oh man. Pfsense is so easy to install, at least I thought it is. 😂

 

If it's your first install without any settings you might throw it away and start over again.

 

No cables plugged in. Use the autodetect feature for the ports. During setup it will ask you to plugin the WAN (should get an IP from router), LAN setup an ip 192.168.0.1, mask 24 and the rest on default, enable DHCP and basically thats it. It's possible that the web interface is screwed somehow by overlaping the DHCP ranges from before. Reset to factory defaults and restart the initial setup process should be enough.

 

Did you have a MAC adress reservation setup on your router for one of the 4 ports maybe and a switch between your Mac and the server itself where the router also has access to? If so, please separate the Pfsense LAN net completly from your main net. Connecting the Mac directly to the LAN port should work.

Link to comment
8 minutes ago, bastl said:

Pfsense is so easy to install, at least I thought it is.

I thought so too when I get into the Web GUI so quick for the first time, then all the nightmare...

 

8 minutes ago, bastl said:

 

No cables plugged in. Use the autodetect feature for the ports. During setup it will ask you to plugin the WAN (should get an IP from router), LAN setup an ip 192.168.0.1, mask 24 and the rest on default, enable DHCP and basically thats it. It's possible that the web interface is screwed somehow by overlaping the DHCP ranges from before. Reset to factory defaults and restart the initial setup process should be enough

Okay I will delete the VM and reinstall again. Do I actually have to delete the interface mentioned here (11:30)?

 

And just another curious question: For the primary vDisk, he said you should select SATA for installation, but can choose virtio afterwards here (10:43), after changing to virtio, all things crash. I know this is not sth I should worry about, I should get the pfSense setup first lol

 

8 minutes ago, bastl said:

Did you have a MAC adress reservation setup on your router for one of the 4 ports

I deleted the reservation already.

 

And there is no switch between my mac and the pfsense. My mac is connected to the router via WiFI on the router, and the server (Unraid GUI) is connected to one of the LAN port on router.

 

So I will directly connect my mac to the pfsense LAN port first. 

 

8 minutes ago, bastl said:

Connecting the Mac directly to the LAN port should work

I guess probably every first time it should work, and then somehow screwed up without knowing the reason lol

 

And btw, I also passthrough one of the onboard port as the WAN port for pfsense, that's why u see em0 in post my last post.

Edited by PzrrL
onboard LAN
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.