hi all.

 

just completed setting up a win10 vm and started to make some test and comparing the performance to baremetal.

 

i have two external usb 3.2 gen 2 nvme drives.

https://www.orico.me/product/nvme-m-2-ssd-enclosure-10gbps-tcm2-c3/

but when i run Crystaldiskmark test the win10 vm starts freeze up for a few sec and usb disconnect and reconnect sounds.

the cursor freezes and there are sound dropouts if a play some music while the test runs.

 

i noticed this only happened during the last two test.

RND4K Q32T1

RND4K Q1T1

this two test is were the performance differs the most comparing to baremetal.

 

are there any settings i could do to improve the performance and stabilty for these drives?

plan is to use them as a steam games library.

 

see att screenshot performance wise baremetal vs win10 vm

 

any help would be very much appreciated

Edited August 11, 20214 yr by TIE Fighter

Are you passing the drives or a whole USB controller. Can you show the rest of you template please.

40 minutes ago, gray squirrel said:

Are you passing the drives or a whole USB controller. Can you show the rest of you template please.

 

both two external usb 3.2 gen 2 nvme drives are connected to the usb ports for the same controller, passtrough to the vm as per below attached screenshot

 

So I think that is the usb controller from the cpu. If I remember correctly it sometimes struggles with passthough. On my Systems it locks the server up if I try to use it.

 

I notice you have another usb controller isolated. As a trouble shooting step can you try testing on that one and seeing if you get the same results?

 

edit, can as n you post you complete vfio settings.

Edited August 13, 20214 yr by gray squirrel

On 8/13/2021 at 8:09 AM, gray squirrel said:

So I think that is the usb controller from the cpu. If I remember correctly it sometimes struggles with passthough. On my Systems it locks the server up if I try to use it.

 

I notice you have another usb controller isolated. As a trouble shooting step can you try testing on that one and seeing if you get the same results?

 

edit, can as n you post you complete vfio settings.

 

correct this controller is integrated in the cpu.

the other usb controller is a usb 3.0 gen 1x pcie expansion card attached to the pcie 1x slot and this slot is intergrated to the chipset controller on the motherboard according to the manual.

anyway i did followed your advice and made the same test through this usb 3.0 controller expansion card, speed is only 5gbit but the results were the same last two test.

RND4K Q32T1

RND4K Q1T1

made the win10 vm freeze up a few sec and the external drives was disconnected and reconnected like previous.

Edited September 2, 20214 yr by TIE Fighter

So I’m still a little suspicious of the amd controller, can you try unbind it? And retest.

 

I notice on your template you have the USB emulation set to 3.0 (qemu XHCI). This is different from the default is 2.0 EHCI.

 

with a sata SSD in a USB caddy this made no difference to the speed. I don’t know but might be something to test.

 

I don’t have an NVME in a external drive to try and replicate unfortunately.

 

Are you seeing anything in unraid or VM logs, maybe try enabling allow unsafe interupts

Edited August 13, 20214 yr by gray squirrel

Also, are you able to boot your VM as a native machine. It might be worth checking you don't get the same behaviour in a non virtualized environment

 

edit: just re-read your OP, did you benchmark on the same hardwhere

Edited August 13, 20214 yr by gray squirrel

1 hour ago, gray squirrel said:

So I’m still a little suspicious of the amd controller, can you try unbind it? And retest.

 

I notice on your template you have the USB emulation set to 3.0 (qemu XHCI). This is different from the default is 2.0 EHCI.

 

with a sata SSD in a USB caddy this made no difference to the speed. I don’t know but might be something to test.

 

I don’t have an NVME in a external drive to try and replicate unfortunately.

 

Are you seeing anything in unraid or VM logs, maybe try enabling allow unsafe interupts

 

unbinded to usb controller intergrated to the cpu.

restart the VM & server

using only the pcie x1 usb 3.0 card controller and the two external drives attached and 2.0 EHCI usb emulation, same results

i did noticed demonic sound aswell using the 2.0 EHCI usb emulation.

reverted back to 3.0 QEMU no demonic sound but still usb disconnects during performance tests.

i did noticed something in the wm log however it did not shows up as a warning so maybe its not related.

 

1 hour ago, gray squirrel said:

Also, are you able to boot your VM as a native machine. It might be worth checking you don't get the same behaviour in a non virtualized environment

 

edit: just re-read your OP, did you benchmark on the same hardwhere

 

the vm is configured as a dual boot so i can restart the machine in baremetal win10, se first post.

in baremetal win10 everything works as expected and the performance is close to win10 vm except the last to tests

RND4K Q32T1

RND4K Q1T1

where the issues starts in the vitrual enviroment.

Have you got overrides enabled, was that device in another group?

 

you might have luck removing overrides and passing the non essential instrumentation part as well.

 

also, did you try “VFIO allow unsafe interrupts” in the VM manager.

23 hours ago, gray squirrel said:

Have you got overrides enabled, was that device in another group?

 

you might have luck removing overrides and passing the non essential instrumentation part as well.

 

also, did you try “VFIO allow unsafe interrupts” in the VM manager.

 

PCIe ACS override = disabled 

i tried to vfio bind the IMMOU group #35 and passtrough to the VM aswell thinking that it might be related to the sound stutters but no success so far the VM did not boot and the server froze forceing me to hard reset. and revert back the changes. *see att screenschots

 

VFIO allow unsafe interrupts = enabled 

again amd controller IOMMU group #37 vfio bind and passtrough to the VM

IOMMU group 32 pcie x1 usb 3.0 card controller vfio bind and passtrough to the VM

test 

 

RND4K Q32T1 = still the same issues with both amd usb controller and the pcie x1 usb 3.0 card controller

 

Edited August 14, 20214 yr by TIE Fighter

Try to passthrough also iommu 35 and 36, all 10:00.x devices

Edited August 14, 20214 yr by ghost82

36 minutes ago, ghost82 said:

Try to passthrough also iommu 35 and 36, all 10:00.x devices

when i tried to passtrough IOMMU group 35 and IOMMU group 36 and start the VM the monitor loose signal to the VM and the server stutters/ freeze only way is to hard reset the server.

binding works but passtrough those groups and start the VM freeze the server.

 

Try passing just one of the USB controllers only and nothing else. No GPU, no sound etc.

 

use VNC to run the test, then add one device at a time.

 

out of curiosity, why do you need the wifi and the Audio? I have just noticed that the Audio devise is not reset capable.

Edited August 14, 20214 yr by gray squirrel