KVM issues rc2

[Fizzyade]

I'm having real KVM issues since rc2, I've never had any issues but now the display virtual display driver seems to be crashing, the machine remains up but I can't connect to it by VNC (well, I can connect, but it's frozen) and under windows 10 I can't connect via RDP (it completely refuses when in this state).  The actual VM's remain functional in this state, both windows & linux are running buildcity agents which still quite happily build.

 

 

[jonp]

Hi @Fizzyade, this is going to be a tough one to solve.  I personally haven't seen any issues with virtual display adapters throughout testing the 6.9 series, and without more details, I really can't say why this is happening.  Do you see any messages in the server or QEMU's event logs?  What about Windows?  Are you using QXL drivers or no?  Latest VirtIO drivers in use?  If this is an issue with QEMU, then the best we can do is let the QEMU developers know about the bug.  We don't actually build the virtualization technology ourselves here at Lime Tech.  We leverage open source KVM / QEMU, which means issues relating to those components will have to be fixed by those developers.  Lastly, please report back after updating to 6.9.1 and let us know if these issues persist.

[Fizzyade]

3 hours ago, jonp said:

Hi @Fizzyade, this is going to be a tough one to solve.  I personally haven't seen any issues with virtual display adapters throughout testing the 6.9 series, and without more details, I really can't say why this is happening.  Do you see any messages in the server or QEMU's event logs?  What about Windows?  Are you using QXL drivers or no?  Latest VirtIO drivers in use?  If this is an issue with QEMU, then the best we can do is let the QEMU developers know about the bug.  We don't actually build the virtualization technology ourselves here at Lime Tech.  We leverage open source KVM / QEMU, which means issues relating to those components will have to be fixed by those developers.  Lastly, please report back after updating to 6.9.1 and let us know if these issues persist.

 

Hi Jon.  As I just mentioned in the other thread, I updated VirtIO to the latest, I'm using QXL.

 

I currently have my WIndows 10 VM running and although the display is working, the keyboard and mouse are not at the moment.  If I switch out to RDP and then go back into VNC they will start working again.

 

This is the same mechanism as the other problem I am happening when the screen stops responding, if I go back into RDP and then back out, VNC will start working again.

 

I would assume the same applies to Linux, except there's no RDP, so there's no real way of me fixing a Linux machine when it gets into this state, I just have to ssh in and reboot.

 

These issues started when I upgraded to 6.9 stable, I'd been using the release candidates for a quite a while and they gave me no issues like these, but now every VM is behaving the same.

[Fizzyade]

On 3/9/2021 at 8:38 PM, Fizzyade said:

 

Hi Jon.  As I just mentioned in the other thread, I updated VirtIO to the latest, I'm using QXL.

 

I currently have my WIndows 10 VM running and although the display is working, the keyboard and mouse are not at the moment.  If I switch out to RDP and then go back into VNC they will start working again.

 

This is the same mechanism as the other problem I am happening when the screen stops responding, if I go back into RDP and then back out, VNC will start working again.

 

I would assume the same applies to Linux, except there's no RDP, so there's no real way of me fixing a Linux machine when it gets into this state, I just have to ssh in and reboot.

 

These issues started when I upgraded to 6.9 stable, I'd been using the release candidates for a quite a while and they gave me no issues like these, but now every VM is behaving the same.

 

I updated to the latest release, but I'm still suffering with this issue.  Latest KVM drivers as well, it affects Windows and Linux, both will just stop providing graphics, the VM's themselves are still up and running.

 

I've checked logs in Windows/Linux and on Unraid and can't see any issues being reported, I'm baffled by this.  I rely on VNC when I am setting up my Windows TeamCity build agent as my code signing certificate is smartcard based and you cannot access the card via RDP (for security), so I have to ensure that the session is set to VNC.

I use the same VM to do test builds as well, the software I am working on (https://www.pingnoo.com) runs on macOS, Linux and Windows, so I have development environment VM's set up and can just pull the latest code and check it compiles and runs correctly under the various operating systems.

 

@jonpany ideas on any logs or info I can send your way?  It's really frustrating as it's worked perfectly for 2 years and suddenly all the VM's have started to misbehave in exactly the same way.  I just don't get any problems with Unraid normally (barring an issue I had with the cache transfer causing CPU load, but I managed to solve that quite a while ago).

[Fizzyade]

@jonpI'm at the end of my tether here, none of my virtual machines work properly anymore, I just rebooted the windows one and I managed to get to the desktop before KVM stopped working, went back in via RDP and I could use it, back into VNC and it works again, but if you drag a window or do anything intensive, it stops working again.

 

I'm happy to send you any logs necessary, but I need some input on what I can give you to try and figure out what is going on.

 

Doesn't matter if it's Windows or Linux, every VM exhibits this behaviour.

[Fizzyade]

@SpencerJ@jonp

Ok, I created a new VM, pointed it at the existing disk image and set the mac address to the previous VM and so far, Windows 10 seems to have held up and not fallen over.

 

I'm guessing I could post the XML of the VM that doesn't work?  I'm going to see how this holds up before doing the same procedure to all of my other VM's.

The only thing I can really see, is that the graphics freezing VM is 'pc-i440fx-4.2' and the newly generated VM is 'pc-i440fx-5.1'

Edited March 28, 2021 by Fizzyade

[jonp]

I'm really still at a loss on what's causing your issues.  I haven't been able to reproduce this in any of our testing systems.  I do have a VM running on pc-i440fx-4.2 but I would honestly be shocked if it started exhibiting issues as this is a fairly common scenario.  The VM's machine type version doesn't automatically get updated for the VM when we update QEMU, but QEMU can respect that.

 

Its even more odd since these are simply VNC-based virtual machines.  Typically when there are VM issues, its due to something hardware-specific with PCI-device pass through.  If that's not happening here and we're talking about just a basic headless VM, I'm at a loss.

 

What would be helpful is when these issues occur if we could get a fresh set of diagnostics with you on 6.9.1.