Ubuntu VM falls back to paused?


Recommended Posts

I have an Ubuntu 18 VM dedicated to running the unifi video application and taking care of 3 cameras. The VM doesn't do anything else. It writes data (video recordings) to an SSD (480GB) mounted using Unassigned Devices plugin. The drive is formatted as EXT4 and is not accessed by anything else but this VM. The cache pool where the Ubuntu install itself resides, is a BTRFS pool. The VM has two CPUs (from a Xeon E3-1230 v6) pegged to it (3/7) and 4GB of RAM (out of a total of 16GB). There are some dockers running as well, but utilization of CPU and RAM is rarely above 40%  There are no other VMs.

 

The VM now refuses to start properly, falling back to a Paused state all the time. When I force start it and VNC (from the VM context menu) to it to intercept the grub menu, I can choose recovery mode. The VM starts recovering journal for vda2 (see attached image)

 

The log shows from the VM this:

-uuid 58c18eaa-f238-265c-50e9-963c2f83c31b \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=33,server,nowait \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=utc,driftfix=slew \
-global kvm-pit.lost_tick_policy=delay \
-no-hpet \
-no-shutdown \
-boot strict=on \
-device pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2 \
-device pcie-root-port,port=0x11,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1 \
-device pcie-root-port,port=0x12,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2 \
-device pcie-root-port,port=0x13,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3 \
-device pcie-root-port,port=0x14,chassis=5,id=pci.5,bus=pcie.0,addr=0x2.0x4 \
-device pcie-root-port,port=0x15,chassis=6,id=pci.6,bus=pcie.0,addr=0x2.0x5 \
-device ich9-usb-ehci1,id=usb,bus=pcie.0,addr=0x7.0x7 \
-device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pcie.0,multifunction=on,addr=0x7 \
-device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pcie.0,addr=0x7.0x1 \
-device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pcie.0,addr=0x7.0x2 \
-device virtio-serial-pci,id=virtio-serial0,bus=pci.3,addr=0x0 \
-blockdev '{"driver":"file","filename":"/mnt/disks/INTEL480NVR/vdisk1.img","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":false,"cache":{"direct":false,"no-flush":false},"driver":"raw","file":"libvirt-1-storage"}' \
-device virtio-blk-pci,scsi=off,bus=pci.4,addr=0x0,drive=libvirt-1-format,id=virtio-disk2,bootindex=1,write-cache=on \
-netdev tap,fd=35,id=hostnet0,vhost=on,vhostfd=36 \
-device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:85:00:91,bus=pci.2,addr=0x0 \
-chardev pty,id=charserial0 \
-device isa-serial,chardev=charserial0,id=serial0 \
-chardev socket,id=charchannel0,fd=37,server,nowait \
-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 \
-device usb-tablet,id=input0,bus=usb.0,port=1 \
-vnc 0.0.0.0:0,websocket=5700 \
-k da \
-device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pcie.0,addr=0x1 \
-device virtio-balloon-pci,id=balloon0,bus=pci.5,addr=0x0 \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
2020-03-01 00:02:01.720+0000: Domain id=1 is tainted: high-privileges
2020-03-01 00:02:01.720+0000: Domain id=1 is tainted: host-cpu
char device redirected to /dev/pts/0 (label charserial0)

After a while, the VM then falls back to Paused state.

 

Not really sure how I should proceed to fix this. Any suggestions?

 

Thanks in advance.

 

2020-03-01 01_38_59-QEMU (Unifi Video Server) - noVNC – Google Chrome.png

Link to comment

That sounds right. The SSD is chucked full, which is weird since unifi video server i supposed to purge recordings after one week. 
 

But anyhow - is it possible to access the vdisk inage on the SSD and manually delete some files? Can I mount the vdisk outside the VM somehow?

Link to comment
20 minutes ago, t33j4y said:

But anyhow - is it possible to access the vdisk inage on the SSD and manually delete some files? Can I mount the vdisk outside the VM somehow?

Is the SSD passed-thought to the VM or there's a vdisk? If passed-through you should be able to mount it with UD, if a vdisk not sure what would be the best way, assuming there is one.

Link to comment
2 hours ago, jonathanm said:

See if this gets you anywhere.

https://forums.unraid.net/topic/54953-solved-how-to-mount-vdiskimg-files/?do=findComment&comment=537778

It's not a direct howto, but perhaps you may be able to modify the commands to fit your situation.

Thank you for this - I did this and the mounting worked. Deleted all of the recordings to make some room, but alas the VM now reports this when trying to start it:

1611224103_Screenshot2020-03-02at23_41_54.png.ff3127a4135d2370dd1e9494d17e5f05.png

 

The vdisk.img file is apparently 515G even though the disk is only a 480GB drive?

 

 

Link to comment
1 hour ago, t33j4y said:

The vdisk.img file is apparently 515G even though the disk is only a 480GB drive?

Yes, overprovisioning can bite you like that. Try copying the vdisk.img to another drive with enough free space and see if it runs. You will need to update the vm's xml to reflect the new location of the file.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.