[SOLVED] VM is always paused

cyberstyx

By cyberstyx
in VM Engine (KVM)

Recommended Posts

cyberstyx Rookie

cyberstyx

Posted
Posted

Hello all,

 

Returned today from work to find out my Windows 10 VM is inaccessible. Turns out it is "paused", but even resuming it from the Web GUI does nothing.

 

I did a Forced Stop and restarted it, it reached the Windows loading screen and then got paused again - same behavior as above.

 

The log from the VM tab is not helping me: 

2017-02-22 17:26:33.687+0000: starting up libvirt version: 1.3.1, qemu version: 2.5.1, hostname: Tower
LC_ALL=C PATH=/bin:/sbin:/usr/bin:/usr/sbin HOME=/ QEMU_AUDIO_DRV=none /usr/local/sbin/qemu -name Apps -S -machine pc-i440fx-2.5,accel=kvm,usb=off,mem-merge=off -cpu host,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff,hv_vendor_id=none -drive file=/boot/config/vm-ovmfs/OVMF-pure-efi-Apps_1.fd,if=pflash,format=raw,unit=0,readonly=on -drive file=/etc/libvirt/qemu/nvram/8ad559d1-4831-d65c-493f-4baaa9094b41_VARS-pure-efi.fd,if=pflash,format=raw,unit=1 -m 2560 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 8ad559d1-4831-d65c-493f-4baaa9094b41 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-Apps/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime -no-hpet -no-shutdown -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x7.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x7 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pcir=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/domain-Apps/org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -device usb-tablet,id=input0 -vnc 0.0.0.0:1,websocket=5701 -k en-us -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vgamem_mb=16,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 -msg timestamp=on
Domain id=2 is tainted: high-privileges
Domain id=2 is tainted: host-cpu
char device redirected to /dev/pts/1 (label charserial0)
ehci warning: guest updated active QH
ehci warning: guest updated active QH
ehci warning: guest updated active QH
ehci warning: guest updated active QH
ehci warning: guest updated active QH
2017-03-02T16:23:50.703615Z qemu-system-x86_64: terminating on signal 15 from pid 8085
2017-03-02 16:23:50.904+0000: shutting down
2017-03-02 16:23:54.572+0000: starting up libvirt version: 1.3.1, qemu version: 2.5.1, hostname: Tower
LC_ALL=C PATH=/bin:/sbin:/usr/bin:/usr/sbin HOME=/ QEMU_AUDIO_DRV=none /usr/local/sbin/qemu -name Apps -S -machine pc-i440fx-2.5,accel=kvm,usb=off,mem-merge=off -cpu host,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff,hv_vendor_id=none -drive file=/boot/config/vm-ovmfs/OVMF-pure-efi-Apps_1.fd,if=pflash,format=raw,unit=0,readonly=on -drive file=/etc/libvirt/qemu/nvram/8ad559d1-4831-d65c-493f-4baaa9094b41_VARS-pure-efi.fd,if=pflash,format=raw,unit=1 -m 2560 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 8ad559d1-4831-d65c-493f-4baaa9094b41 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-Apps/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime -no-hpet -no-shutdown -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x7.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x7 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pcir=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/domain-Apps/org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -device usb-tablet,id=input0 -vnc 0.0.0.0:1,websocket=5701 -k en-us -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vgamem_mb=16,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 -msg timestamp=on
Domain id=3 is tainted: high-privileges
Domain id=3 is tainted: host-cpu
char device redirected to /dev/pts/1 (label charserial0)

The message "ehci warning: guest updated active QH" did not appear again after the restart.

 

It's XML file is: 

<domain type='kvm' id='3'>
  <name>Apps</name>
  <uuid>8ad559d1-4831-d65c-493f-4baaa9094b41</uuid>
  <description>Windows Applications</description>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/>
  </metadata>
  <memory unit='KiB'>2621440</memory>
  <currentMemory unit='KiB'>2621440</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>1</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='1'/>
  </cputune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.5'>hvm</type>
    <loader readonly='yes' type='pflash'>/boot/config/vm-ovmfs/OVMF-pure-efi-Apps_1.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/8ad559d1-4831-d65c-493f-4baaa9094b41_VARS-pure-efi.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor id='none'/>
    </hyperv>
  </features>
  <cpu mode='host-passthrough'>
    <topology sockets='1' cores='1' threads='1'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/user/VMs/Apps/vdisk1.img'/>
      <backingStore/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <alias name='virtio-disk2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/Applications/System/Microsoft Windows/Windows 10/Microsoft Windows 10 Home and Pro x64 Clean ISO/en_windows_10_multiple_editions_x64_dvd_6846432.iso'/>
      <backingStore/>
      <target dev='hda' bus='sata'/>
      <readonly/>
      <boot order='2'/>
      <alias name='sata0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/Applications/unRAID/VIRTIO drivers/virtio-win-0.1.112.iso'/>
      <backingStore/>
      <target dev='hdb' bus='sata'/>
      <readonly/>
      <alias name='sata0-0-1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <alias name='usb'/>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <alias name='usb'/>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <alias name='usb'/>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>
    <controller type='sata' index='0'>
      <alias name='sata0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:72:8d:a8'/>
      <source bridge='br0'/>
      <target dev='vnet1'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/1'/>
      <target port='0'/>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/1'>
      <source path='/dev/pts/1'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-Apps/org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <alias name='input0'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='5901' autoport='yes' websocket='5701' listen='0.0.0.0' keymap='en-us'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </memballoon>
  </devices>
</domain>

 

I have a Windows Server 2012 running fine at the same time. Access to the host / server, file sharing, an existing docker, are all running fine. The host (6.2) was restarted a week ago ago due to its relocation to a different room.

 

I didn't fine something relevant by searching the forum.

 

Any thoughts would be appreciated!

 

Kind regards,

 

Christos.

Link to comment
cyberstyx Rookie

cyberstyx

Posted
  • Author
Posted

I tried to access my Win Server 2012 VM (through Web VNC). When I logged in and as Desktop appeared and started loading, this VM also got paused as well and could not resume it.

 

I tried to stop the array to do a system reboot, the Web GUI got in an infinite "Unmounting shares... Trying to unmount shares" loop. SSHed to the system, tried to powerdown, the command would not complete. Had to do a shutdown -r now.

 

After the restart, Win Server 2012 got up again (was on a auto-power-up mode), Win10 was paused again. Shares, docker, WinServer2012 MySQL connections etc, everything ok, no problems in the logs, as initially mentioned.

 

Trying to resume Win10 fails 9/10 of the tries (web GUI refreshes but the pause icon is still there). After a few tries the icon refreshes to "started", but refreshing the Web GUI with F5 shows it again as paused. When there is that very fast paused > resumed > paused again sequence, I can see from the VNC connection that the VM Win10 machines gets "one step" ahead: it shows the Windows logo, then it will show the progress bar / rotation, then it will update a bit the bar / rotation... it 's like the VM gets auto-paused, at some point it may resume for a fraction and then gets insta-paused again.

 

As this seems to be a system issue and not a VM issue, since Win10 is auto-blocked before the OS even loads and since WinServer2012 also got behaved similarly at least once, I think there is no point in trying to make more VMs with different OSes in case the OS has an issue. For the time being I am not accessing through VNC / RDP the WinServer2012 as, at least its MySQL DB services, are working fine.

Link to comment
ashman70 Proficient

ashman70

Posted
Posted

I haven't had time to do anything about it yet, its still in the same situation. I guess if we can't get it resolved I'll have to rebuild the VM, but its annoying not knowing what caused it in the first place and that it might happen again. The last thing I did was downgrade the server to unRAID 6.2.4 to see if it would make a difference as I had been on 6.3.1 and it didn't make any difference.

Link to comment
cyberstyx Rookie

cyberstyx

Posted
  • Author
Posted

I don't think it 's the actual VM due to its behavior, which seems to me OS-irrelevant.

 

It seems like it's either a VM configuration option (even though I didn't touch anything and it first happened when the VM was on and iddle for many days) or something going on with KVM.

 

I hope an expert user will have some time to check the logs or suggest something...

Link to comment
cyberstyx Rookie

cyberstyx

Posted
  • Author
Posted (edited)

Solved it today.

 

Did some more searching around, trying to use virtfs etc to get some more data about what is happening. Didn't progress so much with that, buy while reading about different solutions to this type of problem,

 

I found out that having your free disk space reduced below a certain point could create issues with VMs running that would instantly pause them without any appropriate messages whatsoever.

 

It seemed a bit strange as my VMs have all their disk size (40GB each) pre-allocated, but the problem is with the amount of free space the system uses for this operation. People had issues with those files being on Cache Disks which, when dropped below a certain point, could cause this.

 

My problem was a bit more obvious, if you knew where to look ofc. The drive hosting the VMs, and I suspect hosting other files for this reason, dropped to the amazing 20,5KB of free space. Which was strange as I had stopped all writes on this disk at 50+GB. That nice docker I have, which is a Minecraft server for the kids, ended up eating up all available free space due to daily backups.

 

Clearing that space and returning the disk to 50+ free GB allowed my VM to start correctly.

 

I can't tell you how nice it is to hit F5 and have the Web GUIs running on that VM to actual no say something rude to me...

 

I hope this situation is also helpful to someone else to, especially you ashman70

 

I will change the topic to solved, and I hope it stays like that ;)

Edited by cyberstyx
  • Like 1
  • Thanks 1
  • Upvote 1
Link to comment
  • cyberstyx changed the title to [SOLVED] VM is always paused
space Rookie

space

Posted
Posted

I had kind of the same issue.

My VM was running on the cache drive, and had free space to spare, when looking at it from within the VM, and the VM configuration left space on the cache drive for other cache activities. Or so I thought. Looking closer the cache drive had 0 bytes left. No wonder things were acting up.

 

Moved the VM to a disk share, and all worked well. Then swapped the cache drive for a new larger drive, and copied the VM back. No issues! 

Link to comment
  • 3 months later...
  • 1 year later...
Zer0Nin3r Apprentice

Zer0Nin3r

Posted
Posted (edited)
On 6/14/2017 at 11:50 AM, Zangief said:

I'm having this same issue... but I have a 500gb cache drive and there is 255gb free.... so its something else for me.... grrrr...

Is it a single cache drive? I have a 512GB SSD paired with a 1TB SSD in my cache pool and I received:

  

2018-10-30T07:01:08.933862Z qemu-system-x86_64: -drive file=/etc/libvirt/qemu/nvram/***_VARS-pure-efi.fd,if=pflash,format=raw,unit=1: Could not open '/etc/libvirt/qemu/nvram/***_VARS-pure-efi.fd': Read-only file system

Well, the 512GB drive is full even though the combined storage reads 768GB. So I suspect that is the root of my problem. Hope this helps you.

 

P.S. - I do wonder why my file transfer failed when the shares are marked as Yes. If the drive shows up full, then the files should have continued copying to the array.

Edited by Zer0Nin3r
Link to comment
Zer0Nin3r Apprentice

Zer0Nin3r

Posted
Posted

Good looking out @johnnie.black I appreciate it!

 

So, in the meantime, I should set my cache floor to 512GB. (It's currently set at the default 2MB.)

 

It's tough as I don't know what to do because I received a free upgrade on my SSD (1TB) and now I am left with mismatching drives.

A) I run a cache pool and have redundancy @ 512GB capacity.

B) Run 1TB un-protected. And find another use for the 512GB.

C) Sell the 512GB and get a 1TB.

Link to comment
  • 2 weeks later...
  • 2 years later...

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
Go to topic listing