PF Sense VM - internal error: qemu unexpectedly closed the monitor - ***Solved***


Recommended Posts

Trying to Install PF Sense on latest Unraid as a VM.

I am running on a Dell R710 server. 24 HT and 40 G memory.

I have 4 NICS as part of the server and have added another Intel 4 port NIC for PF Sense.

I entered the vfio command in Syslinux Configuration to remove the new card from Unraid.

image.png.2ca95422dfcaa50e372f7138b8883ab5.png

It was successfully removed and it turns up on the PF Sense config to include.

image.png.a75eee60648c3d42a28f568784d0388f.png

image.png.ad3bd68d9c9c41d06d4e9c5d1c9d248a.png

IOMMU screen

image.png.717d0b51a1da36c15b3797da70b5d743.png

 

When I try to start it up, after removing virtual disk in XML, it gives this error message:

image.png.9f920573bb949db78101258e28dfebc1.png

 

Error Log:

image.thumb.png.d26795009bee15337d07985a22f95519.png

 

I have also tried the PCIe ACS Override in the VM Manager. It did split up the ports into their own IOMMU groups but get same error.

 

I'm out of ideas. Anyone else get something similiar?

 

I tried a few of the other suggestions in the forum but no joy!

 

Could someone have a look at the particulars and suggest a direction to head in?

 

Thanks in advance!

 

Cheers

 

***Solution***

 

Sitrep

1.       Dell R710 2U rackmount server

2.       Onboard 4 port NIC

3.       23 HT’s and 46G memory

4.       Unraid server with various dockers such as Nextcloud, NZBGet, Sonarr and Bitwarden

5.       Server is stable and functioning as expected.

6.       Unraid has claimed all 4 NIC’s for itself.

Objective:

To install PF Sense in a VM to handle network traffic from the server, various IOT devices, tablets and phones.

Use existing 4 port NIC. 2 ethernet ports are required.

Process:

1.       Follow the process for downloading and preparing the iso for installation

https://docs.netgate.com/pfsense/en/latest/install/installing-pfsense.html

2.       Copy the iso you plan to use to your iso’s directory on the unraid server

3.       Go to Tools/System Devices. It will provide a list of devices in your machine. Look for your ethernet adapters. I have 4 – but only need 2 so I used device 02:00:00 and 02:00:01 leaving the first to for Unraid to use. Jot down these numbers.

image.png.978f4c01ce6e7474cf0d6d345b8476ff.png

 

4.       Go to Main/Flash and scroll to the bottom. Add the text shown

 image.png.8f667c861d5caff74d0f3de15b8de8bc.png

 

Note where I used 02:00:0 and 02:00:1

Reboot the machine as these commands affect the entire machine and need to be in play for the next steps.

5.       After reboot, go to VM tab in Unraid and click “add a VM”.

 

 

6.       Choose FreeBSD

image.png.a3dae850da2aae973f1351fc14fb088a.png

 

7.       Fill in the template as shown in the attached “Tutorial PF Sense property sheet.PDF. Click on the yellow bubbles for additional info.

Double check to make sure the “start vm after creation” is unticked.

8.       Click on the VM and select edit.

9.       In the top right there is a button that says “form view”. Click it and you should see something like this. (XML)

image.png.62c1189c9f9c33b8899b83c32ab11877.png

10.   Scroll down to </video> near the bottom.

11.   Add in the following text after </video>:

 

<hostdev mode='subsystem' type='pci' managed='yes'>

      <driver name='vfio'/>

      <source>

        <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>

      </source>

      <alias name='hostdev0'/>

      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>

    </hostdev>

    <hostdev mode='subsystem' type='pci' managed='yes'>

      <driver name='vfio'/>

      <source>

        <address domain='0x0000' bus='0x02' slot='0x00' function='0x1'/>

      </source>

      <alias name='hostdev1'/>

      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>

    </hostdev>

 

Edit lines 4 and 12 – insert the device id’s from your machine that you wish to use. The supplied example is from my XML and note the usage of the 02:00:0 and 02:00:1 identifiers. Also, if you have more than 2 just add more.

 

12.   Click save which should take you back to the VM page.

13.   Start the VM and you now operational with PF Sense. I recommend you check out Spaceinvader One videos on setting up PF Sense.

 

Next, since I have got this going, I am going to install a 4 port NIC into the server and try to get that working as it provides 4 instead of 2 NIC’s which was my original plan.

 

Let me know if this is overkill - started my IT career as a technical writer. 🤔

 

Cheers

 

Edited by toolmanz
Solved
Link to comment

Well it pays to be persistent! 😃

 

I played around with the settings and found a combo that works.

 

I used 
Spaceinvader One video ...

 

I guess the difference is the box I am running Unraid on .... only thing I can think was a difference from the video.

 

Anyway ....

 

I went with SEABIOS in the VM set up and turned on PCIe ACS Override in the VM Manager which separated all of the ethernet ports into their own IOMMU's.

 

Not sure if it was one or both ??  Going with it for now.

 

Cheers

Edited by toolmanz
Link to comment

Well I tried your suggestion and I got the same result..... this is a really frustrating problem .... but I absolutely will not give up ...grrrrr😈

image.png.dbbcd9ab955f5d9a53c8993ff5710ed0.png

 

image.png.8d11098efa256ca2cd2ac678a3d4c7ac.png

 

Looks like the same issue. IOMMU group 17 and 18 are where the newly installed NIC appears.

 

When I look at settings/network the installed card does not show up (it did) before I executed the vfio-pci.ids= command in the flash drive.

 

So it appears the card is properly excluded but the VM is not picking it up..... apparently you have to have an available NIC in order to start and install the VM.

 

Back to testing I guess ......

 

Thanks in advance for any information you can provide.....

 

Cheers

Edited by toolmanz
Link to comment
2 hours ago, toolmanz said:

Well I tried your suggestion and I got the same result..... this is a really frustrating problem .... but I absolutely will not give up ...grrrrr😈

image.png.dbbcd9ab955f5d9a53c8993ff5710ed0.png

 

image.png.8d11098efa256ca2cd2ac678a3d4c7ac.png

 

Looks like the same issue. IOMMU group 17 and 18 are where the newly installed NIC appears.

 

When I look at settings/network the installed card does not show up (it did) before I executed the vfio-pci.ids= command in the flash drive.

 

So it appears the card is properly excluded but the VM is not picking it up..... apparently you have to have an available NIC in order to start and install the VM.

 

Back to testing I guess ......

 

Thanks in advance for any information you can provide.....

 

Cheers

Did you test with ACS Override on as well when you did this?

Link to comment

Thanks for the suggestion.

 

Yup, with and without the ACS Override.

 

ACS Override splits the 4 port NIC into 4 individual IOMMU's. 

 

It made no difference that I can detect.

 

Hmmmm I am wondering if it is the card itself?? The card is an intel from a Dell server. Shouldn't be a problem...... 

 

I have changed around what riser the card is plugged into and seems to make no difference.

 

The built in NIC is a broadcom...... I wonder if that is a problem.

 

Well upwards and onwards.....🤪

 

Cheers

 

Link to comment
24 minutes ago, toolmanz said:

Thanks for the suggestion.

 

Yup, with and without the ACS Override.

 

ACS Override splits the 4 port NIC into 4 individual IOMMU's. 

 

It made no difference that I can detect.

 

Hmmmm I am wondering if it is the card itself?? The card is an intel from a Dell server. Shouldn't be a problem...... 

 

I have changed around what riser the card is plugged into and seems to make no difference.

 

The built in NIC is a broadcom...... I wonder if that is a problem.

 

Well upwards and onwards.....🤪

 

Cheers

 

I believe you have to reboot when changing acs override. I might be wrong bit did you reboot between these tests?

Link to comment

I did but double checked to make sure.

 

No joy.

 

On to investigating the IOMMU subject and investigate why the NIC's are not available to Unraid yet not available to anything else either.... 

 

¯\\\_(ツ)_/¯

 

I am hoping that will yield some positive results.

 

Cheers

Link to comment

Well, a little more info on the problem:

The kernel error log states:

image.thumb.png.d78ce91cd8ef34d6f84478c643d60926.png

 

It tells me to allow unsafe interrupts to clear the problem of "no interrupt remapping".

 

Does anyone know where I can find the VFIO syntax listing? 

 

I have added the following to the flash drive kernel settings:

 

kernel /bzimage
append intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream,multifunction vfio-pci.ids=8086:10e8 initrd=/bzroot,/bzroot-gui

 

OR

 

kernel /bzimage
append intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts pcie_acs_override=downstream,multifunction vfio-pci.ids=8086:10e8 initrd=/bzroot,/bzroot-gui

 

Neither works to solve the problem. Note the missing "=1" in the first set of lines.

 

I have included my complete IOMMU list - the added 4 port NIC is in found in groups 23, 24, 25 and 26.

 

If I could find the syntax options perhaps there is another setting I'm missing or there is a syntax error in the lines.  ¯\\\_(ツ)_/¯

 

Without checking I have no idea. It could also be the positioning of the statements too..  first, second, third....

 

Another lead .....

 

Cheers

 

Tower_SysDevs_IOMMU List_ACS Override on 2.pdf

Link to comment

Ya I ran into that and doesn't seem to fit - but provides some clues.

 

Also, I have discovered that my server, Dell R710, has problems with interrupt remapping because of the chipset (5520 I think).

https://www.dell.com/community/PowerEdge-Hardware-General/R710-motherboard-without-interrupt-remapping-issues/td-p/7392044

 

So this may not work at all since there appears to be a history of problems.

 

So, I am going to look at using 2 of the ports from my 4 port onboard NIC. 

 

Found these posts:

and 

 

 

I hate starting from scratch .... but it seems my lot in life. Shame on Dell ......

 

I'll see what I can get done. 

 

It kinda restricts me to just the 2 ports but at least I will have it running (he hoped) 🤞

 

I'll report back when (not if) I get it running. Gotta be positive.

 

Thanks again, peteasking, for staying with me on this "journey". You have provided lots to think about and new leads to follow. 👍

 

Cheers

Link to comment

A quick update ....I followed the instructions in the posts above for assigning 2 of the 4 on board NIC's to OPNSense.

 

I actually got it to boot and get me to the menu.

 

But, it didn't find any interfaces.

 

Now the strange thing is that if the VM is shut down all 4 NICS appear in the network settings ....

 

When I start the VM the last 2, the 2 I want, disappear in network settings .... ¯\\\_(ツ)_/¯ go figure.

 

image.png.80a79719bbba3be57d14e35ca9c2785d.png

 

I have attached PDF's showing network settings with and without VM running.

 

So tantilizingly close but so far ....🙄

 

Cheers

Tower_NetworkSettings_only 2 nics show_VM running.pdf Tower_NetworkSettings_all 4 nics show_VM shutdown.pdf

Link to comment

OMG! OMG! It works.

 

Changed to 2.6 version of q35 and it worked.

 

I am currently using the 2 on board NICS and works fine.

 

Not brave enough to try the 4 port addon card ...maybe later.

 

I ain't gonna screw with it now that I have it working.

 

PeteAsking you are da man! 

 

Who was the "other guy" you mentioned .... I want to leave some props there too!

 

Cheers! and big thanks. 

 

 

Link to comment

Oh, I didn't noticed this thread yesterday. I helped a guy yesterday in another thread with the hint which version of q35 to use. I knew i had to change it to an older version in the past, but couldn't remember which was the max version that works.

 

@toolmanz Maybe edit the first post with the solution, so people can find it easier.

Edited by bastl
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.