jonp Posted September 2, 2015 Share Posted September 2, 2015 We recently patched a bug in our VM manager code that had to do with creating a virtual machine using Q35 and OVMF. The bug was that an emulated IDE controller would get added to the domain XML and when OVMF is used with Q35, the use of an IDE controller isn't supported. This was resolved in unRAID 6.1, but apparently wasn't the end of our Q35 + OVMF woes. With that bug squashed, we ran into another discovery: Windows installations just flat out wouldn't work at all with OVMF + Q35 anymore. It didn't matter what we did, the end result was always the same. The VM would boot up, the installer would load, the virtio drivers would load, and the install would begin, but when the VM would go to reboot for the first time during the Windows install, graphics would appear to "hang". I literally tried every possible combination of emulated and physical graphics, every type of emulated storage, and everything in between. No matter what I did, it just wouldn't work. After having sent a detailed report to the OVMF / EDK2 team, I recently received a reply from one of the maintainers confirming I did indeed find a bug and they were able to recreate the issue. For those that are interested, here's the pertinent information: I managed to reproduce the issue in your report! (Using Windows 8.1.) After spending a few hours on it, I think I actually found the root cause. It has nothing to do with SATA. First, about the symptoms. When Windows is booted on UEFI, it inherits the video framebuffer address, size, and color depth / pixel format from the UEFI GOP. Until a Windows native video driver is loaded (which might not happen at all, if none is installed), Windows simply pokes graphics data into the video framebuffer. There is no acceleration, but this suffices for basic graphics. (This was one of the design ideas behind the GOP and behind exposing the framebuffer characteristics to the UEFI OS from UEFI.) What happens in my testing is that the FB base address is 0x8400_0000, according to how OVMF configures it. Unfortunately, during Windows boot, Windows reconfigures the legacy video RAM PCI BAR (= BAR 0) to a much higher address: 0xf800_0000. This means that in order to continue updating the display, Windows should poke to that new address range, not the one at 0x8400_0000. This doesn't happen however. After remapping the legacy VGA video RAM PCI BAR of the QXL card to 0xf800_0000, Windows continues massaging the old address range at 0x8400_0000. This is why the display freezes. Clearly this is not correct, so the question becomes why Windows does that. The answer is: misconfiguration in the \_SB.PCI0._CRS ACPI object that QEMU generates for the guest. This ACPI object describes for the runtime OS the various PCI apertures (ioport and MMIO ranges) that belong to the PCI host, with a list of resource descriptors. In case of i440fx, the relevant resource descriptor looks like this: DWordMemory (ResourceProducer, PosDecode, MinFixed, MaxFixed, NonCacheable, ReadWrite, 0x00000000, // Granularity 0x40000000, // Range Minimum 0xFEBFFFFF, // Range Maximum 0x00000000, // Translation Offset 0xBEC00000, // Length ,, , AddressRangeMemory, TypeStatic) The Range Minimum is 1 GB (which is the amount of RAM the guest had), and the Range Maximum is somewhat below 4GB. This means that in the i440fx case, the PCI MMIO range of the PCI host starts right above the end of guest RAM, and ends somewhat below 4GB. The video framebuffer (starting at 0x8400_0000) correctly falls *into* this aperture, ie. [0x4000_0000, 0xFEBF_FFFF]. However, look at the Q35 case: DWordMemory (ResourceProducer, PosDecode, MinFixed, MaxFixed, NonCacheable, ReadWrite, 0x00000000, // Granularity 0xB0000000, // Range Minimum 0xFEBFFFFF, // Range Maximum 0x00000000, // Translation Offset 0x4EC00000, // Length ,, , AddressRangeMemory, TypeStatic) }) In this case, the range minimum falls *above* the video framebuffer. In other words, the video FB falls *outside* of the advertized aperture of the PCI host. This is why Windows remaps the video card's BAR(s) into the [0xB000_0000, 0xFEBF_FFFF] range. ... But then it continues to write to the low address 0x8400_0000 that it inherited from the GOP. Why is that? Who is at fault? A disagreement between QEMU and OVMF is. Windows is not at fault, because it inherits an already inconsistent configuration -- a pre-configured PCI BAR that lives outside of the PCI host's aperture. BTW you can witness this on Linux too. Just check the dmesg: ... [ 0.400558] pci 0000:00:01.0: can't claim BAR 0 [mem 0x84000000-0x87ffffff]: no compatible bridge window ... Actually, this used to be a problem with i440fx too, but we fixed that: commit ddaaefb4dd427d6d2e41c1cfbe0cd8d8e8d6aad9 Author: Gerd Hoffmann <[email protected]> Date: Sat Dec 21 03:02:50 2013 +0100 piix: fix 32bit pci hole Make the 32bit pci hole start at end of ram, so all possible address space is covered. We used to try and make addresses aligned so they are easier to cover with MTRRs, but since they are cosmetic on KVM, this is probably not worth worrying about. Of course the firmware can use less than that. Leaving space unused is no problem, mapping pci bars outside the hole causes problems though. Signed-off-by: Gerd Hoffmann <[email protected]> Signed-off-by: Laszlo Ersek <[email protected]> Reviewed-by: Michael S. Tsirkin <[email protected]> Signed-off-by: Michael S. Tsirkin <[email protected]> Unfortunately, the situation is not that simple to fix on Q35. On Q35 the PCI config space can be made accessible through the MCFG range (an MMIO window) too, not just the IO ports 0xCF8 and 0xCFC. And Q35 wants to expose this range *between* the low system RAM and the PCI host MMIO aperture. So not only is the Q35-specific fix (that would parallel the i440fx patch "ddaaefb4") missing, we cannot even do the same, because there's the MCFG range for the firmware (OVMF) to jump over, above the top of the low RAM, when allocating resources for PCI devices. For a short while there used to be an fw_cfg file called "etc/pci-info" that was supposed to provide exactly this info to the guest firmware. That fw_cfg file was removed however in the qemu-1.7 time frame. For i440fx the above commit works well, but I have no clue how to solve this issue for Q35, in QEMU. The only solution I can see now is an OVMF patch. I'll post it soon to edk2-devel and I'll Cc you. With that patch I managed to install Windows 8.1 on Q35 and OVMF. Thanks for the report! Anyhow, just thought you guys might be interested in some of the stuff we work on when we're not on the forum and how we work with upstream maintainers on various components that we use in unRAID OS. Link to comment
archedraft Posted September 2, 2015 Share Posted September 2, 2015 Very impressive and thanks for the interesting behind the scenes. Link to comment
jonp Posted September 3, 2015 Author Share Posted September 3, 2015 Bringing this post full circle, a patch was issued for OVMF and I tested it today. Looks like I'll be updating the wiki again soon as all the Q35 issues with OVMF seem to be solved. I was able to create 3 VMs today using Q35 + OVMF with no issues. OVMF support will still be limited to guests that have UEFI support, but other than that, there should be no variance between Q35 and i440fx support (we hope). Link to comment
GHunter Posted September 7, 2015 Share Posted September 7, 2015 Thanks for the information. I appreciate the detailed report and know that you guys often spend time chasing these bugs down. It's great to see people working together so diligently to get them fixed. Was this the bug that was fixed in 6.1.1? Gary Link to comment
jonp Posted September 7, 2015 Author Share Posted September 7, 2015 Yes. I'll post later about the steps go perform to fix existing VMs, but new VMs should work just fine. Link to comment
goober07 Posted September 19, 2015 Share Posted September 19, 2015 Explains why I haven't been able to test Windows 10 with OVMF+Q35. Thanks for the update! Link to comment
Recommended Posts
Archived
This topic is now archived and is closed to further replies.