Ubuntu 22.04 VM locks up and goes to 100% CPU useage about every 24 hours.

[relink]

I have no idea whats causing this so I'll try and provide as much information as possible.

 

I have a VM running Ubuntu Server 22.04, it seems to run just fine for hours on end, but not one day has gone by where I went to sleep or to work only to find the cores assigned to the VM at 100%, and the VM totally unresponsive.

 

Unraid Version 6.11.5

AMD Ryzen 5 2600

80GB DDR4

Diagnostics Attached (Taken while the VM was locked up)

 

VM Info:

OS: Ubuntu Server 22.04

CPU: 3C/6T Host Passthrough

RAM: 8GB

GPU:

1. VNC
2. Nvidia Quadro P400 (Passed through with it's audio controller)

Storage:

1. 40GB Virtio Vdisk qcow (On nvme cache)
2. Virtiofs mounted directory on a 2TB unassigned SSD.

 

VM Use:

The VM only runs Nextcloud 25.0.2 & NGINX. PostgreSQL and Redis are both running as docker containers on unraid. The Virtiofs storage is set as the Nextcloud data directory.

 

Aside from SSH & the Nvidia drivers there is nothing else running on this VM that isn't part of the standard Ubuntu Server installation.

 

I have completely formatted and re-installed the guest OS 4 times and this issue still happens. I'm really not sure why...

 

 

VM XML:

Spoiler

<?xml version='1.0' encoding='UTF-8'?>

<domain type='kvm' id='2'>

<name>NC-Ubuntu</name>

<uuid>2cce725e-4a5b-3f14-a49d-df77d39f892f</uuid>

<metadata>

<vmtemplate xmlns="unraid" name="Ubuntu" icon="ubuntu.png" os="ubuntu"/>

</metadata>

<memory unit='KiB'>8388608</memory>

<currentMemory unit='KiB'>8388608</currentMemory>

<memoryBacking>

<source type='memfd'/>

<access mode='shared'/>

</memoryBacking>

<vcpu placement='static'>6</vcpu>

<cputune>

<vcpupin vcpu='0' cpuset='0'/>

<vcpupin vcpu='1' cpuset='6'/>

<vcpupin vcpu='2' cpuset='2'/>

<vcpupin vcpu='3' cpuset='8'/>

<vcpupin vcpu='4' cpuset='4'/>

<vcpupin vcpu='5' cpuset='10'/>

</cputune>

<resource>

<partition>/machine</partition>

</resource>

<os>

<type arch='x86_64' machine='pc-q35-7.1'>hvm</type>

<loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>

<nvram>/etc/libvirt/qemu/nvram/2cce725e-4a5b-3f14-a49d-df77d39f892f_VARS-pure-efi.fd</nvram>

</os>

<features>

<acpi/>

<apic/>

</features>

<cpu mode='host-passthrough' check='none' migratable='on'>

<topology sockets='1' dies='1' cores='3' threads='2'/>

<cache mode='passthrough'/>

<feature policy='require' name='topoext'/>

</cpu>

<clock offset='utc'>

<timer name='rtc' tickpolicy='catchup'/>

<timer name='pit' tickpolicy='delay'/>

<timer name='hpet' present='no'/>

</clock>

<on_poweroff>destroy</on_poweroff>

<on_reboot>restart</on_reboot>

<on_crash>restart</on_crash>

<devices>

<emulator>/usr/local/sbin/qemu</emulator>

<disk type='file' device='disk'>

<driver name='qemu' type='qcow2' cache='writeback'/>

<source file='/mnt/cache-nvme/domains/NC-Ubuntu/vdisk1.img' index='2'/>

<backingStore/>

<target dev='hdc' bus='virtio'/>

<boot order='2'/>

<alias name='virtio-disk2'/>

<address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>

</disk>

<disk type='file' device='cdrom'>

<driver name='qemu' type='raw'/>

<source file='/mnt/cache-nvme/isos/ubuntu-22.04.1-live-server-amd64.iso' index='1'/>

<backingStore/>

<target dev='hda' bus='sata' tray='open'/>

<readonly/>

<boot order='1'/>

<alias name='sata0-0-0'/>

<address type='drive' controller='0' bus='0' target='0' unit='0'/>

</disk>

<controller type='usb' index='0' model='ich9-ehci1'>

<alias name='usb'/>

<address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>

</controller>

<controller type='usb' index='0' model='ich9-uhci1'>

<alias name='usb'/>

<master startport='0'/>

<address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>

</controller>

<controller type='usb' index='0' model='ich9-uhci2'>

<alias name='usb'/>

<master startport='2'/>

<address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>

</controller>

<controller type='usb' index='0' model='ich9-uhci3'>

<alias name='usb'/>

<master startport='4'/>

<address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>

</controller>

<controller type='pci' index='4' model='pcie-root-port'>

<model name='pcie-root-port'/>

<target chassis='4' port='0x13'/>

<alias name='pci.4'/>

<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>

</controller>

<controller type='pci' index='5' model='pcie-root-port'>

<model name='pcie-root-port'/>

<target chassis='5' port='0x14'/>

<alias name='pci.5'/>

<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>

</controller>

<controller type='pci' index='6' model='pcie-root-port'>

<model name='pcie-root-port'/>

<target chassis='6' port='0x15'/>

<alias name='pci.6'/>

<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/>

</controller>

<controller type='pci' index='7' model='pcie-root-port'>

<model name='pcie-root-port'/>

<target chassis='7' port='0x16'/>

<alias name='pci.7'/>

<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x6'/>

</controller>

<controller type='virtio-serial' index='0'>

<alias name='virtio-serial0'/>

<address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>

</controller>

<controller type='sata' index='0'>

<alias name='ide'/>

<address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>

</controller>

<controller type='pci' index='0' model='pcie-root'>

<alias name='pcie.0'/>

</controller>

<controller type='pci' index='1' model='pcie-root-port'>

<model name='pcie-root-port'/>

<target chassis='1' port='0x10'/>

<alias name='pci.1'/>

<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>

</controller>

<controller type='pci' index='2' model='pcie-root-port'>

<model name='pcie-root-port'/>

<target chassis='2' port='0x11'/>

<alias name='pci.2'/>

<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>

</controller>

<controller type='pci' index='3' model='pcie-root-port'>

<model name='pcie-root-port'/>

<target chassis='3' port='0x12'/>

<alias name='pci.3'/>

<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>

</controller>

<filesystem type='mount' accessmode='passthrough'>

<driver type='virtiofs' queue='1024'/>

<binary path='/usr/libexec/virtiofsd' xattr='on'>

<cache mode='always'/>

<sandbox mode='chroot'/>

<lock posix='on' flock='on'/>

</binary>

<source dir='/mnt/disks/ncdata-cache'/>

<target dir='ncdata'/>

<alias name='fs0'/>

<address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>

</filesystem>

<interface type='bridge'>

<mac address='52:54:00:d6:34:6b'/>

<source bridge='br0'/>

<target dev='vnet1'/>

<model type='virtio-net'/>

<alias name='net0'/>

<address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>

</interface>

<serial type='pty'>

<source path='/dev/pts/0'/>

<target type='isa-serial' port='0'>

<model name='isa-serial'/>

</target>

<alias name='serial0'/>

</serial>

<console type='pty' tty='/dev/pts/0'>

<source path='/dev/pts/0'/>

<target type='serial' port='0'/>

<alias name='serial0'/>

</console>

<channel type='unix'>

<source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-2-NC-Ubuntu/org.qemu.guest_agent.0'/>

<target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/>

<alias name='channel0'/>

<address type='virtio-serial' controller='0' bus='0' port='1'/>

</channel>

<input type='tablet' bus='usb'>

<alias name='input0'/>

<address type='usb' bus='0' port='1'/>

</input>

<input type='mouse' bus='ps2'>

<alias name='input1'/>

</input>

<input type='keyboard' bus='ps2'>

<alias name='input2'/>

</input>

<graphics type='vnc' port='5900' autoport='yes' websocket='5700' listen='0.0.0.0' keymap='en-us'>

<listen type='address' address='0.0.0.0'/>

</graphics>

<audio id='1' type='none'/>

<video>

<model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>

<alias name='video0'/>

<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>

</video>

<hostdev mode='subsystem' type='pci' managed='yes'>

<driver name='vfio'/>

<source>

<address domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>

</source>

<alias name='hostdev0'/>

<address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>

</hostdev>

<hostdev mode='subsystem' type='pci' managed='yes'>

<driver name='vfio'/>

<source>

<address domain='0x0000' bus='0x08' slot='0x00' function='0x1'/>

</source>

<alias name='hostdev1'/>

<address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>

</hostdev>

<memballoon model='none'/>

</devices>

<seclabel type='dynamic' model='dac' relabel='yes'>

<label>+0:+100</label>

<imagelabel>+0:+100</imagelabel>

</seclabel>

</domain>

 

 

 

 

 

serverus-diagnostics-20221227-1719.zip

[relink]

Anyone? Even any suggestions on what I could check? 
 

Im still dealing with this. And it actually seems to be less than 24 hours, maybe 12 or less. 

[SimonF]

18 minutes ago, relink said:

Anyone? Even any suggestions on what I could check? 
 

Im still dealing with this. And it actually seems to be less than 24 hours, maybe 12 or less. 

Possibly related to memoryBacking for virtiofs.  Maybe try 9p for the file share.

 

you would need to remove the memfd and  access mode

[relink]

Quote

Possibly related to memoryBacking for virtiofs.  Maybe try 9p for the file share.

I can certainly try it, can’t hurt. 
 

Quote

you would need to remove the memfd and  access mode

I’m not sure what your talking about here. All I did was add the share in the GUI when creating the VM and then added it to /etc/fstab. Maybe I missed an important step or option?

[SimonF]

25 minutes ago, relink said:

I can certainly try it, can’t hurt. 
 

I’m not sure what your talking about here. All I did was add the share in the GUI when creating the VM and then added it to /etc/fstab. Maybe I missed an important step or option?

Edit the VM goto XML mode.

 

Remove

<memoryBacking>
    <source type='memfd'/>
    <access mode='shared'/>
</memoryBacking>  

Replace with

<memoryBacking>
    <nosharepages/>
</memoryBacking>  

 

[relink]

Ahhh, ok. I did it through the GUI so I never saw that part. I’ll give it a shot as soon as I can get to my computer and report back. 
 

on a side note, isn’t 9p mode significantly slower? 

[SimonF]

1 hour ago, relink said:

Ahhh, ok. I did it through the GUI so I never saw that part. I’ll give it a shot as soon as I can get to my computer and report back. 
 

on a side note, isn’t 9p mode significantly slower? 

I have not done any performance tests. There are big improvements with QEMU 7.2 for 9P but still on 7.1 currently

[relink]

Damn, no go. I’ts still crashing, this time it was only running for about an hour. 
 

Completely reinstalled Ubuntu again, this time I changed the mount from virtiofs to 9p.

 

I also took the opportunity to expand my vdisk and switched to RAW instead of qcow. 
 

I have no idea what’s going on

[relink]

Ok, so this is definitely a KVM/Host issue. I decided to scrap Ubuntu entirely and installed Arch. I did nothing in Arch aside from setup user accounts, networking, time zone, and other basic stuff. I didn't even mount the share, it was in the VM config, but not mounted in the guest. Arch was literally just sitting there doing absolutely nothing and this issue still happened. 

[SimonF]

6 minutes ago, relink said:

Ok, so this is definitely a KVM/Host issue. I decided to scrap Ubuntu entirely and installed Arch. I did nothing in Arch aside from setup user accounts, networking, time zone, and other basic stuff. I didn't even mount the share, it was in the VM config, but not mounted in the guest. Arch was literally just sitting there doing absolutely nothing and this issue still happened. 

What do you hqve in the memoryBacking section in the xml?

[relink]

<memoryBacking>
    <source type='memfd'/>
    <access mode='shared'/>
  </memoryBacking>

In the context of the earlier comment I assumed this went along with the Unraid share setting. Should I ensure this is changed to `<nosharepages/>` regardless of what the share setting is? 

[SimonF]

5 minutes ago, relink said:

<memoryBacking>
    <source type='memfd'/>
    <access mode='shared'/>
  </memoryBacking>

In the context of the earlier comment I assumed this went along with the Unraid share setting. Should I ensure this is changed to `<nosharepages/>` regardless of what the share setting is? 

Yes set to nosharepages. Share setting will need to be 9p or no shared defined.

[relink]

Ok, im off to try again. Thank you for your help. *fingers Crossed*

[relink]

That definitely seemed to have been the culprit. I have been running on and off for over 2 days and that issue hasn't happened again since.