• Windows VM much slower after upgrade 6.10.RC[1-2] - Cyberpunk 2077 Videos


    agarkauskas
    • Annoyance



    User Feedback

    Recommended Comments

    Hi there,

     

    I've been trying to recreate these reported performance issues, but in my own testing, I'm not able to.  Can you confirm this performance drop is occurring with other games as well or just Cyberpunk?

    Link to comment
    4 hours ago, jonp said:

    Hi there,

     

    I've been trying to recreate these reported performance issues, but in my own testing, I'm not able to.  Can you confirm this performance drop is occurring with other games as well or just Cyberpunk?

     

    Hi jonp! Confirmed. Everything is slower. You can notice the degradation in Windows as well, browsing the web or using Word/Excel. Is there any specific test you want me to execute? I can run a Passmark complete test before and after.

     

    I can also try it on my other VMs (Win 2008, Ubuntu, VMWare ESXi and Proxmox):

     

    image.thumb.png.c08edc232a0de0797860a57a5cfb657e.png

    Link to comment

    And are both of your VMs experiencing this same issue with each GPU?  If you're having bad performance in Windows outside of games as well, this is unlikely a bug with GPU pass through, but rather, something amiss with CPU virtualization/pinning.  Can you confirm both of those Windows VMs you have experience the same issue and that the issue persists even with only one VM running at a time?

    Link to comment

    @agarkauskas really hoping to hear back from you on these questions.  In addition, we're starting to wonder if this might be due to the dual CPUs you're using in your setup.  Did you perhaps change any of your configurations with CPU assignments or isolations?  Did you ever align your memory allocation for your VMs to the right NUMA nodes in alignment with your GPUs?

    Link to comment

    Sorry for the delay jonp!

     

      <memoryBacking>
        <nosharepages/>
      </memoryBacking>
      <vcpu placement='static'>18</vcpu>
      <cputune>
        <vcpupin vcpu='0' cpuset='32'/>
        <vcpupin vcpu='1' cpuset='33'/>
        <vcpupin vcpu='2' cpuset='34'/>
        <vcpupin vcpu='3' cpuset='35'/>
        <vcpupin vcpu='4' cpuset='18'/>
        <vcpupin vcpu='5' cpuset='19'/>
        <vcpupin vcpu='6' cpuset='20'/>
        <vcpupin vcpu='7' cpuset='21'/>
        <vcpupin vcpu='8' cpuset='22'/>
        <vcpupin vcpu='9' cpuset='23'/>
        <vcpupin vcpu='10' cpuset='24'/>
        <vcpupin vcpu='11' cpuset='25'/>
        <vcpupin vcpu='12' cpuset='26'/>
        <vcpupin vcpu='13' cpuset='27'/>
        <vcpupin vcpu='14' cpuset='28'/>
        <vcpupin vcpu='15' cpuset='29'/>
        <vcpupin vcpu='16' cpuset='30'/>
        <vcpupin vcpu='17' cpuset='31'/>
      </cputune>
      <numatune>
        <memory mode='strict' nodeset='1'/>
      </numatune>

     

    I think it is aligned (i'm not using hyperthread - at least I am trying). I will run more tests.

    Link to comment

    This system has two physical socketed CPUs though, yes?  And did you have to do anything special to align the GPU and memory allocations on the previous versions of Unraid?

    Link to comment
    On 11/16/2021 at 5:02 PM, jonp said:

    This system has two physical socketed CPUs though, yes?  And did you have to do anything special to align the GPU and memory allocations on the previous versions of Unraid?

     

    Hi jonp! Thanks for being engaged with my issue. I scheduled a complete battery of tests for this weekend to retrieve more information for you.

     

    Right now I am finishing a heavy batch processing workload (need to buy milk for the children you know?) but I will finish tomorrow end of day.

     

    YES! I took special precautions due to the double sockets.

    My WorkGameStation VM is aligned with the CPU2, which is directly connected to the PCIEx slot connected to the GPU. Nothing else runs on CPU2. Another thing I did is to "disable" hyperthread on CPU2. I only selected one CPU core from a hyperthread pair to this VM. For some reason my 3DMark benchs are better this way.

     

    Another thing I did was to enforce memory allocation to my WorkGameStation to the RAM sticks tied to CPU2, like this:

     

    <numatune>
        <memory mode='strict' nodeset='1'/>
    </numatune>

     

    CPU1 is used with all my other VMs. The idea is to have the entire L3 cache available to my WorkGameStation from CPU2, while any heavy lifting processing running on VMs tied to CPU1 would not interfere. I dont want to loose precious FPSs while shooting bad guys in Cyberpunk and smashing CPU1 with heavy CPU consumption processes.

     

    Exception is my sound card, disk access and USB dedicated card. Both devices are connected to PCIEx slots connected to CPU1. For these devices the communication between my WorkGameStation VM will go thru CPU interconnect path.

    Edited by agarkauskas
    Link to comment

    Hi @jonp! Sorry for the delay, I'm running big experiments here in my rig.

    It took some time but I have great evidence for you. It seems to me that the problem is CPU related according to the benchmarks.

    I ran Passmark and Heaven Unigine on 8 different scenarios:

     

    - Each Windows vm running alone, without the other VM running

    - Both VMs running the bench at the same time.

     

    For the disks results: WIndows 10 VM has a dedicated NVME drive while Copilot is using a BTRFS disk "aided" by Unraid RAM.

     

    Results:

     

    image.thumb.png.77821146ca1bb73947d19f8d63277c42.png

     

    Evidence attached.

     

    FAQ:

    "Why dont you have Windows Copilot 6.10-rc2 Single VM running data for passmark?"

    "Because I forgot this one, :)"

     

    Benchmarks.zip

    Link to comment

    This is going to be a real bear to figure out ;-).  I definitely think this is related to CPU assignments and NUMA nodes.  Will keep investigating...

    Link to comment

    I noticed in the libvirt log that it is repeatedly trying to open remote connections from your other server, this only happens in the 6.10rc2 logs. 

    This might not resolve your issue, but worth removing these isos from the VM to rule it out in any case.

     

    2021-11-03 10:36:32.851+0000: 5854: error : virQEMUFileOpenAs:11262 : Failed to open file '/mnt/remotes/BABYBUS-GREEN_isos/template/iso/AOMEI_Windows.iso': No such file or directory
    2021-11-03 10:36:32.851+0000: 5858: error : virQEMUFileOpenAs:11262 : Failed to open file '/mnt/remotes/BABYBUS-GREEN_isos/template/iso/virtio-win-0.1.189-1.iso': No such file or directory
    2021-11-03 10:36:34.697+0000: 5855: error : virQEMUFileOpenAs:11262 : Failed to open file '/mnt/remotes/BABYBUS-GREEN_isos/template/iso/AOMEI_Windows.iso': No such file or directory
    2021-11-03 10:36:34.697+0000: 5854: error : virQEMUFileOpenAs:11262 : Failed to open file '/mnt/remotes/BABYBUS-GREEN_isos/template/iso/virtio-win-0.1.189-1.iso': No such file or directory
    2021-11-03 10:36:34.714+0000: 5858: error : virQEMUFileOpenAs:11262 : Failed to open file '/mnt/remotes/BABYBUS-GREEN_isos/template/iso/AOMEI_Windows.iso': No such file or directory
    2021-11-03 10:36:34.715+0000: 5857: error : virQEMUFileOpenAs:11262 : Failed to open file '/mnt/remotes/BABYBUS-GREEN_isos/template/iso/virtio-win-0.1.189-1.iso': No such file or directory
    2021-11-03 11:40:38.970+0000: 5858: error : virQEMUFileOpenAs:11262 : Failed to open file '/mnt/remotes/BABYBUS-GREEN_isos/template/iso/AOMEI_Windows.iso': No such file or directory
    2021-11-03 11:40:38.971+0000: 5857: error : virQEMUFileOpenAs:11262 : Failed to open file '/mnt/remotes/BABYBUS-GREEN_isos/template/iso/virtio-win-0.1.189-1.iso': No such file or directory
    2021-11-03 11:40:39.724+0000: 5856: error : virQEMUFileOpenAs:11262 : Failed to open file '/mnt/remotes/BABYBUS-GREEN_isos/template/iso/AOMEI_Windows.iso': No such file or directory
    2021-11-03 11:40:39.725+0000: 5855: error : virQEMUFileOpenAs:11262 : Failed to open file '/mnt/remotes/BABYBUS-GREEN_isos/template/iso/virtio-win-0.1.189-1.iso': No such file or directory
    2021-11-03 11:40:39.742+0000: 5854: error : virQEMUFileOpenAs:11262 : Failed to open file '/mnt/remotes/BABYBUS-GREEN_isos/template/iso/AOMEI_Windows.iso': No such file or directory
    2021-11-03 11:40:39.743+0000: 5858: error : virQEMUFileOpenAs:11262 : Failed to open file '/mnt/remotes/BABYBUS-GREEN_isos/template/iso/virtio-win-0.1.189-1.iso': No such file or directory
    2021-11-03 20:29:30.344+0000: 5857: error : virQEMUFileOpenAs:11262 : Failed to open file '/mnt/remotes/BABYBUS-GREEN_isos/template/iso/AOMEI_Windows.iso': No such file or directory
    2021-11-03 20:29:30.345+0000: 5856: error : virQEMUFileOpenAs:11262 : Failed to open file '/mnt/remotes/BABYBUS-GREEN_isos/template/iso/virtio-win-0.1.189-1.iso': No such file or directory
    2021-11-03 20:29:30.991+0000: 5858: error : virQEMUFileOpenAs:11262 : Failed to open file '/mnt/remotes/BABYBUS-GREEN_isos/template/iso/AOMEI_Windows.iso': No such file or directory
    2021-11-03 20:29:30.991+0000: 5857: error : virQEMUFileOpenAs:11262 : Failed to open file '/mnt/remotes/BABYBUS-GREEN_isos/template/iso/virtio-win-0.1.189-1.iso': No such file or directory
    2021-11-03 20:29:31.009+0000: 5856: error : virQEMUFileOpenAs:11262 : Failed to open file '/mnt/remotes/BABYBUS-GREEN_isos/template/iso/AOMEI_Windows.iso': No such file or directory
    2021-11-03 20:29:31.010+0000: 5855: error : virQEMUFileOpenAs:11262 : Failed to open file '/mnt/remotes/BABYBUS-GREEN_isos/template/iso/virtio-win-0.1.189-1.iso': No such file or directory
    2021-11-03 20:29:35.271+0000: 5854: error : virQEMUFileOpenAs:11262 : Failed to open file '/mnt/remotes/BABYBUS-GREEN_isos/template/iso/AOMEI_Windows.iso': No such file or directory
    2021-11-03 20:29:35.272+0000: 5858: error : virQEMUFileOpenAs:11262 : Failed to open file '/mnt/remotes/BABYBUS-GREEN_isos/template/iso/virtio-win-0.1.189-1.iso': No such file or directory
    2021-11-03 20:29:35.288+0000: 5857: error : virQEMUFileOpenAs:11262 : Failed to open file '/mnt/remotes/BABYBUS-GREEN_isos/template/iso/AOMEI_Windows.iso': No such file or directory
    2021-11-03 20:29:35.289+0000: 5856: error : virQEMUFileOpenAs:11262 : Failed to open file '/mnt/remotes/BABYBUS-GREEN_isos/template/iso/virtio-win-0.1.189-1.iso': No such file or directory

     

    Link to comment
    11 hours ago, tjb_altf4 said:

    I noticed in the libvirt log that it is repeatedly trying to open remote connections from your other server, this only happens in the 6.10rc2 logs. 

    This might not resolve your issue, but worth removing these isos from the VM to rule it out in any case.

     

    2021-11-03 10:36:32.851+0000: 5854: error : virQEMUFileOpenAs:11262 : Failed to open file '/mnt/remotes/BABYBUS-GREEN_isos/template/iso/AOMEI_Windows.iso': No such file or directory
    2021-11-03 10:36:32.851+0000: 5858: error : virQEMUFileOpenAs:11262 : Failed to open file '/mnt/remotes/BABYBUS-GREEN_isos/template/iso/virtio-win-0.1.189-1.iso': No such file or directory
    2021-11-03 10:36:34.697+0000: 5855: error : virQEMUFileOpenAs:11262 : Failed to open file '/mnt/remotes/BABYBUS-GREEN_isos/template/iso/AOMEI_Windows.iso': No such file or directory
    2021-11-03 10:36:34.697+0000: 5854: error : virQEMUFileOpenAs:11262 : Failed to open file '/mnt/remotes/BABYBUS-GREEN_isos/template/iso/virtio-win-0.1.189-1.iso': No such file or directory
    2021-11-03 10:36:34.714+0000: 5858: error : virQEMUFileOpenAs:11262 : Failed to open file '/mnt/remotes/BABYBUS-GREEN_isos/template/iso/AOMEI_Windows.iso': No such file or directory
    2021-11-03 10:36:34.715+0000: 5857: error : virQEMUFileOpenAs:11262 : Failed to open file '/mnt/remotes/BABYBUS-GREEN_isos/template/iso/virtio-win-0.1.189-1.iso': No such file or directory
    2021-11-03 11:40:38.970+0000: 5858: error : virQEMUFileOpenAs:11262 : Failed to open file '/mnt/remotes/BABYBUS-GREEN_isos/template/iso/AOMEI_Windows.iso': No such file or directory
    2021-11-03 11:40:38.971+0000: 5857: error : virQEMUFileOpenAs:11262 : Failed to open file '/mnt/remotes/BABYBUS-GREEN_isos/template/iso/virtio-win-0.1.189-1.iso': No such file or directory
    2021-11-03 11:40:39.724+0000: 5856: error : virQEMUFileOpenAs:11262 : Failed to open file '/mnt/remotes/BABYBUS-GREEN_isos/template/iso/AOMEI_Windows.iso': No such file or directory
    2021-11-03 11:40:39.725+0000: 5855: error : virQEMUFileOpenAs:11262 : Failed to open file '/mnt/remotes/BABYBUS-GREEN_isos/template/iso/virtio-win-0.1.189-1.iso': No such file or directory
    2021-11-03 11:40:39.742+0000: 5854: error : virQEMUFileOpenAs:11262 : Failed to open file '/mnt/remotes/BABYBUS-GREEN_isos/template/iso/AOMEI_Windows.iso': No such file or directory
    2021-11-03 11:40:39.743+0000: 5858: error : virQEMUFileOpenAs:11262 : Failed to open file '/mnt/remotes/BABYBUS-GREEN_isos/template/iso/virtio-win-0.1.189-1.iso': No such file or directory
    2021-11-03 20:29:30.344+0000: 5857: error : virQEMUFileOpenAs:11262 : Failed to open file '/mnt/remotes/BABYBUS-GREEN_isos/template/iso/AOMEI_Windows.iso': No such file or directory
    2021-11-03 20:29:30.345+0000: 5856: error : virQEMUFileOpenAs:11262 : Failed to open file '/mnt/remotes/BABYBUS-GREEN_isos/template/iso/virtio-win-0.1.189-1.iso': No such file or directory
    2021-11-03 20:29:30.991+0000: 5858: error : virQEMUFileOpenAs:11262 : Failed to open file '/mnt/remotes/BABYBUS-GREEN_isos/template/iso/AOMEI_Windows.iso': No such file or directory
    2021-11-03 20:29:30.991+0000: 5857: error : virQEMUFileOpenAs:11262 : Failed to open file '/mnt/remotes/BABYBUS-GREEN_isos/template/iso/virtio-win-0.1.189-1.iso': No such file or directory
    2021-11-03 20:29:31.009+0000: 5856: error : virQEMUFileOpenAs:11262 : Failed to open file '/mnt/remotes/BABYBUS-GREEN_isos/template/iso/AOMEI_Windows.iso': No such file or directory
    2021-11-03 20:29:31.010+0000: 5855: error : virQEMUFileOpenAs:11262 : Failed to open file '/mnt/remotes/BABYBUS-GREEN_isos/template/iso/virtio-win-0.1.189-1.iso': No such file or directory
    2021-11-03 20:29:35.271+0000: 5854: error : virQEMUFileOpenAs:11262 : Failed to open file '/mnt/remotes/BABYBUS-GREEN_isos/template/iso/AOMEI_Windows.iso': No such file or directory
    2021-11-03 20:29:35.272+0000: 5858: error : virQEMUFileOpenAs:11262 : Failed to open file '/mnt/remotes/BABYBUS-GREEN_isos/template/iso/virtio-win-0.1.189-1.iso': No such file or directory
    2021-11-03 20:29:35.288+0000: 5857: error : virQEMUFileOpenAs:11262 : Failed to open file '/mnt/remotes/BABYBUS-GREEN_isos/template/iso/AOMEI_Windows.iso': No such file or directory
    2021-11-03 20:29:35.289+0000: 5856: error : virQEMUFileOpenAs:11262 : Failed to open file '/mnt/remotes/BABYBUS-GREEN_isos/template/iso/virtio-win-0.1.189-1.iso': No such file or directory

     

     

    Thanks for your feedback @tjb_altf4! Really appreciate it.

     

    Indeed, I moved my iso storage from one of my unraid boxes (not the main one) to my WDEX4 NAS and I forgot to update this particular VM.

     

    Version 6.10 is probably checking the configured mountings even with VMs using them being offline. Weird.

     

    I corrected the mounting points.

    Link to comment
    13 hours ago, jonp said:

    This is going to be a real bear to figure out ;-).  I definitely think this is related to CPU assignments and NUMA nodes.  Will keep investigating...

     

    @jonp if you need me to run more tests please let me know. Not sure if there is another folk reporting a similar issue, I will be happy to continue this conversation in his/her topic or vice-versa.

    Link to comment

    Same here, 6.10.0-RC2 was very sluggysh, IO had to wait much longer for IO. I didn't notice much difference in games though, but working in Windows was more painful. Went back to 6.9.2 and it's all good again.

    (Ryzen9 3900x)

    • Like 1
    Link to comment
    On 12/15/2021 at 8:10 PM, Norbs said:

    Same here, 6.10.0-RC2 was very sluggysh, IO had to wait much longer for IO. I didn't notice much difference in games though, but working in Windows was more painful. Went back to 6.9.2 and it's all good again.

    (Ryzen9 3900x)

    I'm also sluggish on a Ryzen9 3900x Win10 VM with cpu host passthrough... If I roll back to 6.9.2 it's super fast.

    • Like 1
    Link to comment
    On 4/17/2022 at 8:17 AM, agarkauskas said:

    Ladies and gentleman! Problem solved with RC4. 😃

    @jonp

     

    WOO HOO!!  So glad to hear this!!

    Link to comment

    I'm experiencing the same issues with CB2077 with Ryzen 7 1700 and MSI RTX 3090 SUPRIM X with silent bios.

     

    I'm on 6.11.1. Windows install is on NVME passthrough to VM. No vDisks. 

     

    60 fps and ~95% GPU utilisation when booted into Windows and 15-20fps and 40-50% GPU utilisation in VM.

     

    Other games are not affected to the same extent. Have any other users experienced similar issues or found a solution?

    Edited by joncroweucl
    Link to comment
    3 hours ago, joncroweucl said:

    I'm experiencing the same issues with CB2077 with Ryzen 7 1700 and MSI RTX 3090 SUPRIM X with silent bios.

     

    I'm on 6.11.1. Windows install is on NVME passthrough to VM. No vDisks. 

     

    60 fps and ~95% GPU utilisation when booted into Windows and 15-20fps and 40-50% GPU utilisation in VM.

     

    Other games are not affected to the same extent. Have any other users experienced similar issues or found a solution?

     

    It seems stupid, but after a lot of trouble I figure it out some things:

     

    - Keep your Unraid updated

    - Keep your Windows updated (I kepp my Windows update disabled and only update when I am ready to)

    - Keep your NVidia drivers updated <- Usually a clean install helps a lot

    - Watch out your BIOS for Power States. Dial everything up for performance

    - Install Unraid Tips and Tweaks and, again, dial everything for performance

    - Try a different CPU pinning setting (other cores) - specially for Ryzen processors (chiplets -> trouble)

     

    Link to comment
    On 10/23/2022 at 10:41 PM, joncroweucl said:

    I'm experiencing the same issues with CB2077 with Ryzen 7 1700 and MSI RTX 3090 SUPRIM X with silent bios.

     

    I'm on 6.11.1. Windows install is on NVME passthrough to VM. No vDisks. 

     

    60 fps and ~95% GPU utilisation when booted into Windows and 15-20fps and 40-50% GPU utilisation in VM.

     

    Other games are not affected to the same extent. Have any other users experienced similar issues or found a solution?

    Hi, I had same issue. Resolved!

    After looooooong investigation, I remembered that I had modified my win10 system, long time ago, it was in order to make Halo infinite work. 

    So I reversed what I had done and bingo this command below worked for me :

    Disable-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V

     

    I have 3060Ti and Ryzen 5600X (using only 3cores and 6threads). Now I pass the Cyberpunk benchmark @60fps, before the command I was @19fps with GPU under-working (30% refering to msi-afterburner).

     

    This behaviour is odd because Halo worked fine and even 3dMark gave me good results with Supervisor activated. Is it a Cyberpunk bug? Maybe a feature.. 

    Link to comment


    Join the conversation

    You can post now and register later. If you have an account, sign in now to post with your account.
    Note: Your post will require moderator approval before it will be visible.

    Guest
    Add a comment...

    ×   Pasted as rich text.   Restore formatting

      Only 75 emoji are allowed.

    ×   Your link has been automatically embedded.   Display as a link instead

    ×   Your previous content has been restored.   Clear editor

    ×   You cannot paste images directly. Upload or insert images from URL.


  • Status Definitions

     

    Open = Under consideration.

     

    Solved = The issue has been resolved.

     

    Solved version = The issue has been resolved in the indicated release version.

     

    Closed = Feedback or opinion better posted on our forum for discussion. Also for reports we cannot reproduce or need more information. In this case just add a comment and we will review it again.

     

    Retest = Please retest in latest release.


    Priority Definitions

     

    Minor = Something not working correctly.

     

    Urgent = Server crash, data loss, or other showstopper.

     

    Annoyance = Doesn't affect functionality but should be fixed.

     

    Other = Announcement or other non-issue.