toolmanz Posted March 10, 2020 Share Posted March 10, 2020 (edited) Trying to Install PF Sense on latest Unraid as a VM. I am running on a Dell R710 server. 24 HT and 40 G memory. I have 4 NICS as part of the server and have added another Intel 4 port NIC for PF Sense. I entered the vfio command in Syslinux Configuration to remove the new card from Unraid. It was successfully removed and it turns up on the PF Sense config to include. IOMMU screen When I try to start it up, after removing virtual disk in XML, it gives this error message: Error Log: I have also tried the PCIe ACS Override in the VM Manager. It did split up the ports into their own IOMMU groups but get same error. I'm out of ideas. Anyone else get something similiar? I tried a few of the other suggestions in the forum but no joy! Could someone have a look at the particulars and suggest a direction to head in? Thanks in advance! Cheers ***Solution*** Sitrep 1. Dell R710 2U rackmount server 2. Onboard 4 port NIC 3. 23 HT’s and 46G memory 4. Unraid server with various dockers such as Nextcloud, NZBGet, Sonarr and Bitwarden 5. Server is stable and functioning as expected. 6. Unraid has claimed all 4 NIC’s for itself. Objective: To install PF Sense in a VM to handle network traffic from the server, various IOT devices, tablets and phones. Use existing 4 port NIC. 2 ethernet ports are required. Process: 1. Follow the process for downloading and preparing the iso for installation https://docs.netgate.com/pfsense/en/latest/install/installing-pfsense.html 2. Copy the iso you plan to use to your iso’s directory on the unraid server 3. Go to Tools/System Devices. It will provide a list of devices in your machine. Look for your ethernet adapters. I have 4 – but only need 2 so I used device 02:00:00 and 02:00:01 leaving the first to for Unraid to use. Jot down these numbers. 4. Go to Main/Flash and scroll to the bottom. Add the text shown Note where I used 02:00:0 and 02:00:1 Reboot the machine as these commands affect the entire machine and need to be in play for the next steps. 5. After reboot, go to VM tab in Unraid and click “add a VM”. 6. Choose FreeBSD 7. Fill in the template as shown in the attached “Tutorial PF Sense property sheet.PDF. Click on the yellow bubbles for additional info. Double check to make sure the “start vm after creation” is unticked. 8. Click on the VM and select edit. 9. In the top right there is a button that says “form view”. Click it and you should see something like this. (XML) 10. Scroll down to </video> near the bottom. 11. Add in the following text after </video>: <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </source> <alias name='hostdev0'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x02' slot='0x00' function='0x1'/> </source> <alias name='hostdev1'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </hostdev> Edit lines 4 and 12 – insert the device id’s from your machine that you wish to use. The supplied example is from my XML and note the usage of the 02:00:0 and 02:00:1 identifiers. Also, if you have more than 2 just add more. 12. Click save which should take you back to the VM page. 13. Start the VM and you now operational with PF Sense. I recommend you check out Spaceinvader One videos on setting up PF Sense. Next, since I have got this going, I am going to install a 4 port NIC into the server and try to get that working as it provides 4 instead of 2 NIC’s which was my original plan. Let me know if this is overkill - started my IT career as a technical writer. 🤔 Cheers Edited March 17, 2020 by toolmanz Solved Quote Link to comment
toolmanz Posted March 10, 2020 Author Share Posted March 10, 2020 (edited) Well it pays to be persistent! 😃 I played around with the settings and found a combo that works. I used Spaceinvader One video ... I guess the difference is the box I am running Unraid on .... only thing I can think was a difference from the video. Anyway .... I went with SEABIOS in the VM set up and turned on PCIe ACS Override in the VM Manager which separated all of the ethernet ports into their own IOMMU's. Not sure if it was one or both ?? Going with it for now. Cheers Edited March 10, 2020 by toolmanz Quote Link to comment
toolmanz Posted March 10, 2020 Author Share Posted March 10, 2020 Spoke too soon - the iso does install BUT it doesn't find any valid NIC's..... So removed PCIe ACS Override in the VM Manager as well and no joy! 😔 Ill keep at it...... Cheers Quote Link to comment
PeteAsking Posted March 10, 2020 Share Posted March 10, 2020 PFSense has major comparability issues with KVM, I always suggest just using OPNSense as its basically identical and doesnt have any problems. Just use I440fx, seabios and it works fine with the latest everything and even virtio drivers. Quote Link to comment
toolmanz Posted March 10, 2020 Author Share Posted March 10, 2020 Ok thanks for the suggestion. I did actually try installing OPNSense but without the seabios and I440fx. I will try that and report back. Quote Link to comment
toolmanz Posted March 10, 2020 Author Share Posted March 10, 2020 (edited) Well I tried your suggestion and I got the same result..... this is a really frustrating problem .... but I absolutely will not give up ...grrrrr😈 Looks like the same issue. IOMMU group 17 and 18 are where the newly installed NIC appears. When I look at settings/network the installed card does not show up (it did) before I executed the vfio-pci.ids= command in the flash drive. So it appears the card is properly excluded but the VM is not picking it up..... apparently you have to have an available NIC in order to start and install the VM. Back to testing I guess ...... Thanks in advance for any information you can provide..... Cheers Edited March 10, 2020 by toolmanz Quote Link to comment
PeteAsking Posted March 11, 2020 Share Posted March 11, 2020 2 hours ago, toolmanz said: Well I tried your suggestion and I got the same result..... this is a really frustrating problem .... but I absolutely will not give up ...grrrrr😈 Looks like the same issue. IOMMU group 17 and 18 are where the newly installed NIC appears. When I look at settings/network the installed card does not show up (it did) before I executed the vfio-pci.ids= command in the flash drive. So it appears the card is properly excluded but the VM is not picking it up..... apparently you have to have an available NIC in order to start and install the VM. Back to testing I guess ...... Thanks in advance for any information you can provide..... Cheers Did you test with ACS Override on as well when you did this? Quote Link to comment
toolmanz Posted March 11, 2020 Author Share Posted March 11, 2020 Thanks for the suggestion. Yup, with and without the ACS Override. ACS Override splits the 4 port NIC into 4 individual IOMMU's. It made no difference that I can detect. Hmmmm I am wondering if it is the card itself?? The card is an intel from a Dell server. Shouldn't be a problem...... I have changed around what riser the card is plugged into and seems to make no difference. The built in NIC is a broadcom...... I wonder if that is a problem. Well upwards and onwards.....🤪 Cheers Quote Link to comment
PeteAsking Posted March 11, 2020 Share Posted March 11, 2020 24 minutes ago, toolmanz said: Thanks for the suggestion. Yup, with and without the ACS Override. ACS Override splits the 4 port NIC into 4 individual IOMMU's. It made no difference that I can detect. Hmmmm I am wondering if it is the card itself?? The card is an intel from a Dell server. Shouldn't be a problem...... I have changed around what riser the card is plugged into and seems to make no difference. The built in NIC is a broadcom...... I wonder if that is a problem. Well upwards and onwards.....🤪 Cheers I believe you have to reboot when changing acs override. I might be wrong bit did you reboot between these tests? Quote Link to comment
toolmanz Posted March 11, 2020 Author Share Posted March 11, 2020 I did but double checked to make sure. No joy. On to investigating the IOMMU subject and investigate why the NIC's are not available to Unraid yet not available to anything else either.... ¯\\\_(ツ)_/¯ I am hoping that will yield some positive results. Cheers Quote Link to comment
PeteAsking Posted March 11, 2020 Share Posted March 11, 2020 Sorry man thats strange. You could always use a virtual nic in the meantime until you can fix it. Quote Link to comment
toolmanz Posted March 11, 2020 Author Share Posted March 11, 2020 Well, a little more info on the problem: The kernel error log states: It tells me to allow unsafe interrupts to clear the problem of "no interrupt remapping". Does anyone know where I can find the VFIO syntax listing? I have added the following to the flash drive kernel settings: kernel /bzimage append intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream,multifunction vfio-pci.ids=8086:10e8 initrd=/bzroot,/bzroot-gui OR kernel /bzimage append intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts pcie_acs_override=downstream,multifunction vfio-pci.ids=8086:10e8 initrd=/bzroot,/bzroot-gui Neither works to solve the problem. Note the missing "=1" in the first set of lines. I have included my complete IOMMU list - the added 4 port NIC is in found in groups 23, 24, 25 and 26. If I could find the syntax options perhaps there is another setting I'm missing or there is a syntax error in the lines. ¯\\\_(ツ)_/¯ Without checking I have no idea. It could also be the positioning of the statements too.. first, second, third.... Another lead ..... Cheers Tower_SysDevs_IOMMU List_ACS Override on 2.pdf Quote Link to comment
PeteAsking Posted March 11, 2020 Share Posted March 11, 2020 Is it similar to this guys issue? Quote Link to comment
toolmanz Posted March 11, 2020 Author Share Posted March 11, 2020 Ya I ran into that and doesn't seem to fit - but provides some clues. Also, I have discovered that my server, Dell R710, has problems with interrupt remapping because of the chipset (5520 I think). https://www.dell.com/community/PowerEdge-Hardware-General/R710-motherboard-without-interrupt-remapping-issues/td-p/7392044 So this may not work at all since there appears to be a history of problems. So, I am going to look at using 2 of the ports from my 4 port onboard NIC. Found these posts: and I hate starting from scratch .... but it seems my lot in life. Shame on Dell ...... I'll see what I can get done. It kinda restricts me to just the 2 ports but at least I will have it running (he hoped) 🤞 I'll report back when (not if) I get it running. Gotta be positive. Thanks again, peteasking, for staying with me on this "journey". You have provided lots to think about and new leads to follow. 👍 Cheers Quote Link to comment
PeteAsking Posted March 11, 2020 Share Posted March 11, 2020 Sorry I couldnt actually provide an answer. Hopefully you find a workable solution. Quote Link to comment
Joeri1594 Posted March 12, 2020 Share Posted March 12, 2020 Hello @toolmanz, i have made also a topic and i think we have the same kind of problem. only with different hardware. https://forums.unraid.net/topic/89686-cant-install-pfsense-opsense-and-sophos-xg/ Quote Link to comment
toolmanz Posted March 12, 2020 Author Share Posted March 12, 2020 A quick update ....I followed the instructions in the posts above for assigning 2 of the 4 on board NIC's to OPNSense. I actually got it to boot and get me to the menu. But, it didn't find any interfaces. Now the strange thing is that if the VM is shut down all 4 NICS appear in the network settings .... When I start the VM the last 2, the 2 I want, disappear in network settings .... ¯\\\_(ツ)_/¯ go figure. I have attached PDF's showing network settings with and without VM running. So tantilizingly close but so far ....🙄 Cheers Tower_NetworkSettings_only 2 nics show_VM running.pdf Tower_NetworkSettings_all 4 nics show_VM shutdown.pdf Quote Link to comment
PeteAsking Posted March 13, 2020 Share Posted March 13, 2020 The other guy had to use a 2.6 version of q35. Did you also do that? 1 Quote Link to comment
toolmanz Posted March 13, 2020 Author Share Posted March 13, 2020 OMG! OMG! It works. Changed to 2.6 version of q35 and it worked. I am currently using the 2 on board NICS and works fine. Not brave enough to try the 4 port addon card ...maybe later. I ain't gonna screw with it now that I have it working. PeteAsking you are da man! Who was the "other guy" you mentioned .... I want to leave some props there too! Cheers! and big thanks. Quote Link to comment
toolmanz Posted March 13, 2020 Author Share Posted March 13, 2020 Oh one last thing .... is there anyone who needs me to document how I got here? Quote Link to comment
bastl Posted March 14, 2020 Share Posted March 14, 2020 (edited) Oh, I didn't noticed this thread yesterday. I helped a guy yesterday in another thread with the hint which version of q35 to use. I knew i had to change it to an older version in the past, but couldn't remember which was the max version that works. @toolmanz Maybe edit the first post with the solution, so people can find it easier. Edited March 14, 2020 by bastl Quote Link to comment
toolmanz Posted March 14, 2020 Author Share Posted March 14, 2020 Thanks to you for getting this going with your tip on versions. Very happy camper here with much better internet throughput.😊😊 2x my speeds.... I will update first post with the solution.👍 Cheers 1 Quote Link to comment
PeteAsking Posted March 14, 2020 Share Posted March 14, 2020 Thanks all, glad this got resolved to a workable solution. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.