I have an Ubuntu 18 VM dedicated to running the unifi video application and taking care of 3 cameras. The VM doesn't do anything else. It writes data (video recordings) to an SSD (480GB) mounted using Unassigned Devices plugin. The drive is formatted as EXT4 and is not accessed by anything else but this VM. The cache pool where the Ubuntu install itself resides, is a BTRFS pool. The VM has two CPUs (from a Xeon E3-1230 v6) pegged to it (3/7) and 4GB of RAM (out of a total of 16GB). There are some dockers running as well, but utilization of CPU and RAM is rarely above 40%  There are no other VMs.

 

The VM now refuses to start properly, falling back to a Paused state all the time. When I force start it and VNC (from the VM context menu) to it to intercept the grub menu, I can choose recovery mode. The VM starts recovering journal for vda2 (see attached image)

 

The log shows from the VM this:

-uuid 58c18eaa-f238-265c-50e9-963c2f83c31b \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=33,server,nowait \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=utc,driftfix=slew \
-global kvm-pit.lost_tick_policy=delay \
-no-hpet \
-no-shutdown \
-boot strict=on \
-device pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2 \
-device pcie-root-port,port=0x11,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1 \
-device pcie-root-port,port=0x12,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2 \
-device pcie-root-port,port=0x13,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3 \
-device pcie-root-port,port=0x14,chassis=5,id=pci.5,bus=pcie.0,addr=0x2.0x4 \
-device pcie-root-port,port=0x15,chassis=6,id=pci.6,bus=pcie.0,addr=0x2.0x5 \
-device ich9-usb-ehci1,id=usb,bus=pcie.0,addr=0x7.0x7 \
-device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pcie.0,multifunction=on,addr=0x7 \
-device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pcie.0,addr=0x7.0x1 \
-device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pcie.0,addr=0x7.0x2 \
-device virtio-serial-pci,id=virtio-serial0,bus=pci.3,addr=0x0 \
-blockdev '{"driver":"file","filename":"/mnt/disks/INTEL480NVR/vdisk1.img","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":false,"cache":{"direct":false,"no-flush":false},"driver":"raw","file":"libvirt-1-storage"}' \
-device virtio-blk-pci,scsi=off,bus=pci.4,addr=0x0,drive=libvirt-1-format,id=virtio-disk2,bootindex=1,write-cache=on \
-netdev tap,fd=35,id=hostnet0,vhost=on,vhostfd=36 \
-device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:85:00:91,bus=pci.2,addr=0x0 \
-chardev pty,id=charserial0 \
-device isa-serial,chardev=charserial0,id=serial0 \
-chardev socket,id=charchannel0,fd=37,server,nowait \
-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 \
-device usb-tablet,id=input0,bus=usb.0,port=1 \
-vnc 0.0.0.0:0,websocket=5700 \
-k da \
-device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pcie.0,addr=0x1 \
-device virtio-balloon-pci,id=balloon0,bus=pci.5,addr=0x0 \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
2020-03-01 00:02:01.720+0000: Domain id=1 is tainted: high-privileges
2020-03-01 00:02:01.720+0000: Domain id=1 is tainted: host-cpu
char device redirected to /dev/pts/0 (label charserial0)

After a while, the VM then falls back to Paused state.

 

Not really sure how I should proceed to fix this. Any suggestions?

 

Thanks in advance.

 

VMs usually pase when one of the storage devices is out of space/completely allocated, you can check yourself or post the diags.

That sounds right. The SSD is chucked full, which is weird since unifi video server i supposed to purge recordings after one week. 
 

But anyhow - is it possible to access the vdisk inage on the SSD and manually delete some files? Can I mount the vdisk outside the VM somehow?

20 minutes ago, t33j4y said:

But anyhow - is it possible to access the vdisk inage on the SSD and manually delete some files? Can I mount the vdisk outside the VM somehow?

Is the SSD passed-thought to the VM or there's a vdisk? If passed-through you should be able to mount it with UD, if a vdisk not sure what would be the best way, assuming there is one.

It’s mounted with UD and has a vdisk file on it 😕

2 hours ago, t33j4y said:

It’s mounted with UD and has a vdisk file on it 😕

See if this gets you anywhere.

https://forums.unraid.net/topic/54953-solved-how-to-mount-vdiskimg-files/?do=findComment&comment=537778

It's not a direct howto, but perhaps you may be able to modify the commands to fit your situation.

2 hours ago, jonathanm said:

See if this gets you anywhere.

https://forums.unraid.net/topic/54953-solved-how-to-mount-vdiskimg-files/?do=findComment&comment=537778

It's not a direct howto, but perhaps you may be able to modify the commands to fit your situation.

Thank you for this - I did this and the mounting worked. Deleted all of the recordings to make some room, but alas the VM now reports this when trying to start it:

 

The vdisk.img file is apparently 515G even though the disk is only a 480GB drive?

 

 

1 hour ago, t33j4y said:

The vdisk.img file is apparently 515G even though the disk is only a 480GB drive?

Yes, overprovisioning can bite you like that. Try copying the vdisk.img to another drive with enough free space and see if it runs. You will need to update the vm's xml to reflect the new location of the file.