HI everyone, I get this on my ubuntu VM in Unraid, attaching the urnaid diags. Really hard to google this, I couldn't find any resolution.. These are the last view lines of the VM log file. The VM crashes after around 8-10 hours being on. I have a lot of hard drives attached via 2 USB PCI-E passthrough cards, but it used to work reliably without crashing, not sure what's now causing this..

 

-device '{"driver":"qxl-vga","id":"video0","max_outputs":1,"ram_size":67108864,"vram_size":67108864,"vram64_size_mb":0,"vgamem_mb":16,"bus":"pcie.0","addr":"0x1"}' \
-device '{"driver":"vfio-pci","host":"0000:0e:00.0","id":"hostdev0","bus":"pci.5","addr":"0x0"}' \
-device '{"driver":"vfio-pci","host":"0000:04:00.0","id":"hostdev1","bus":"pci.6","addr":"0x0"}' \
-device '{"driver":"vfio-pci","host":"0000:05:00.0","id":"hostdev2","bus":"pci.7","addr":"0x0"}' \
-device '{"driver":"vfio-pci","host":"0000:06:00.0","id":"hostdev3","bus":"pci.8","addr":"0x0"}' \
-device '{"driver":"vfio-pci","host":"0000:07:00.0","id":"hostdev4","bus":"pci.9","addr":"0x0"}' \
-device '{"driver":"vfio-pci","host":"0000:0e:00.2","id":"hostdev5","bus":"pci.10","addr":"0x0"}' \
-device '{"driver":"vfio-pci","host":"0000:0e:00.3","id":"hostdev6","bus":"pci.11","addr":"0x0"}' \
-device '{"driver":"vfio-pci","host":"0000:10:00.0","id":"hostdev7","bus":"pci.12","addr":"0x0"}' \
-device '{"driver":"vfio-pci","host":"0000:88:00.0","id":"hostdev8","bus":"pci.13","addr":"0x0"}' \
-device '{"driver":"vfio-pci","host":"0000:89:00.0","id":"hostdev9","bus":"pci.14","addr":"0x0"}' \
-device '{"driver":"vfio-pci","host":"0000:8a:00.0","id":"hostdev10","bus":"pci.15","addr":"0x0"}' \
-device '{"driver":"vfio-pci","host":"0000:8b:00.0","id":"hostdev11","bus":"pci.16","addr":"0x0"}' \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
char device redirected to /dev/pts/0 (label charserial0)
qxl_send_events: spice-server bug: guest stopped, ignoring
failed to set up stack guard page: Cannot allocate memory
2023-12-23 01:13:59.993+0000: shutting down, reason=crashed

 

Thanks for any suggestions!

 

super-diagnostics-20231223-0917.zip

4 hours ago, theunraidhomeuser said:

HI everyone, I get this on my ubuntu VM in Unraid, attaching the urnaid diags. Really hard to google this, I couldn't find any resolution.. These are the last view lines of the VM log file. The VM crashes after around 8-10 hours being on. I have a lot of hard drives attached via 2 USB PCI-E passthrough cards, but it used to work reliably without crashing, not sure what's now causing this..

 

-device '{"driver":"qxl-vga","id":"video0","max_outputs":1,"ram_size":67108864,"vram_size":67108864,"vram64_size_mb":0,"vgamem_mb":16,"bus":"pcie.0","addr":"0x1"}' \
-device '{"driver":"vfio-pci","host":"0000:0e:00.0","id":"hostdev0","bus":"pci.5","addr":"0x0"}' \
-device '{"driver":"vfio-pci","host":"0000:04:00.0","id":"hostdev1","bus":"pci.6","addr":"0x0"}' \
-device '{"driver":"vfio-pci","host":"0000:05:00.0","id":"hostdev2","bus":"pci.7","addr":"0x0"}' \
-device '{"driver":"vfio-pci","host":"0000:06:00.0","id":"hostdev3","bus":"pci.8","addr":"0x0"}' \
-device '{"driver":"vfio-pci","host":"0000:07:00.0","id":"hostdev4","bus":"pci.9","addr":"0x0"}' \
-device '{"driver":"vfio-pci","host":"0000:0e:00.2","id":"hostdev5","bus":"pci.10","addr":"0x0"}' \
-device '{"driver":"vfio-pci","host":"0000:0e:00.3","id":"hostdev6","bus":"pci.11","addr":"0x0"}' \
-device '{"driver":"vfio-pci","host":"0000:10:00.0","id":"hostdev7","bus":"pci.12","addr":"0x0"}' \
-device '{"driver":"vfio-pci","host":"0000:88:00.0","id":"hostdev8","bus":"pci.13","addr":"0x0"}' \
-device '{"driver":"vfio-pci","host":"0000:89:00.0","id":"hostdev9","bus":"pci.14","addr":"0x0"}' \
-device '{"driver":"vfio-pci","host":"0000:8a:00.0","id":"hostdev10","bus":"pci.15","addr":"0x0"}' \
-device '{"driver":"vfio-pci","host":"0000:8b:00.0","id":"hostdev11","bus":"pci.16","addr":"0x0"}' \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
char device redirected to /dev/pts/0 (label charserial0)
qxl_send_events: spice-server bug: guest stopped, ignoring
failed to set up stack guard page: Cannot allocate memory
2023-12-23 01:13:59.993+0000: shutting down, reason=crashed

 

Thanks for any suggestions!

 

super-diagnostics-20231223-0917.zip 279.12 kB · 0 downloads

As you are not using virtiofs you could remove  the memfs and shared section, but not sure that will make more difference.  

 

 

You have 382G allocated is that spanning the NUMA nodes? Have you tried restricting memory allocation to memory allocated to the physical CPU that the VM is running on or are you spanning CPUs?

 

<memoryBacking>

<nosharepages/>

<source type="memfd"/>

<access mode="shared"/>

</memoryBacking>

Hi there and thanks for your help. I'll try and remove memfs and shared section as a first try.

I'm not an expert on numa nodes so not entirely sure what you mean by:
 

Quote

 

You have 382G allocated is that spanning the NUMA nodes? Have you tried restricting memory allocation to memory allocated to the physical CPU that the VM is running on or are you spanning CPUs?

 

 

Below is the CPU settings I have. as this is a VM, I don't know how you woulr restrict memory allocation to memory allocated to the CPU, if you could let me know what I need to do, I'd happily try that. I initally had 256GB assigned but then thought the machine might be running out of memory due to some leak but then that turned out not to be the cause.

 

Thanks!

 

 

and this is the VM setup but you will have seen that from the diags.

 

3 hours ago, SimonF said:

You have 382G allocated is that spanning the NUMA nodes? Have you tried restricting memory allocation to memory allocated to the physical CPU that the VM is running on or are you spanning CPUs?

I added this section from another forum post that I found:

 

  <numatune>
    <memory mode='preferred' nodeset='0'/>
  </numatune>

 

I've also changed the CPU pinning to the following... let's see..

 

the code above solved the issue! Thanks all!