midgard00

Members
  • Posts

    9
  • Joined

  • Last visited

midgard00's Achievements

Noob

Noob (1/14)

1

Reputation

  1. I was not able to get any display out when switching from the "radeon" driver to "amdgpu". I tried this for PopOS 22.04 and Ubuntu 20.04. In both cases after switching to "amdgpu" I can't shut down the VM properly and need to force stop it. After that, I don't get any display out on that GPU anymore, not even with a Linux with default "radeon" driver. Only a host reboot fixes that.
  2. Thanks for the tip. While that did not change anything regarding the GPU reset, it did massively improve GPU performance in PopOS. I will try the official AMD driver next to see if that changes anything.
  3. Hi, I have a weird issue with GPU passthrough. The short version: I have two VMs (Win 11 and PopOS) with the same GPU assigned (R9 290X). Each VM works fine, including reboots. However, once I started the PopOS VM once, I can no longer pass the GPU to the Win 11 VM (no display and Code 43 in RDP). The PopOS VM still works fine, but to get the Win 11 VM working again I need to reboot Unraid. The longer version: I have two R9 290X in my system with these settings: I have two Win 11 VMs, one for each GPU. These are the .xmls (identical apart from CPU pinning, vdisk and GPU address). These configurations contain tweaks from various sources for better performance: And one PopOS VM that uses the same GPU as the first Win 11 VM: Passthrough works for all three of these VMs, including reboots. Here is the first Win 11 VM working: And here is PopOS working (performance is not good, I assume this is down to the driver in use, haven't looked into that yet): The issue appears when I attempt to start the first Win 11 VM when the PopOS VM was started previously. I get no display out and when accessing the VM through RDP I can see a Code 43 on the GPU: PopOS as well as the second Win 11 VM with the other GPU still work fine. So the issue is only with the specific GPU that was passed to PopOS. I had the same issue with a Win 10/Ubuntu combination in the past. The GPU passthrough to Windows starts working again after a reboot of Unraid. I have attached diagnostics, but here is the syslog when I start/stop Win 11 -> start/stop PopOS -> start Win 11, with passthrough to Win 11 working fine the first time but Code 43 the second time. This appears to me as though the GPU is not properly reset when PopOS shuts down. As far as I understand this is different from the typical AMD reset bug, as restarting VMs works fine normally. Only switching from PopOS to Windows causes the issue. I did install the AMD vendor reset plugin, but that did not change anything. I would appreciate any tips on how to get the GPU to reset properly. Thanks x299-diagnostics-20230509-2031.zip
  4. I have no experience with php and Unraid plugin development, but I imagine the difficult thing would be to automatically detect ports where currently no device is connected, since there are no folders for those in the directories.
  5. Interesting. Last time I searched the forum I did not find this. It does however appear to work on a per device basis, as in you can define what actions shoud be taken if a certain device is detected, no matter the physical port. This method targets ports instead. So you could do something like attach the USB device to VM1 if inserted into port 1 and attach it to VM2 if inserted into port 2. That does not currently seem possible with that plugin.
  6. Hello, I want to demonstrate a method of enabling USB hotplugging on specific USB ports to specific VMs, that is only active while the VM is running. The behaviour is the similar to USB controller passthrough, but is applied to specific USB ports rather than all ports on the controller. It does not require passthrough or VFIO binding of the controller to which the port belongs. It can thus be used for USB ports connected to the controller that hosts the Unraid USB Drive. Calling this method "USB port passthrough" would be a fitting description of the behaviour, but this is not what is actually happening. The way this works is to use a script that monitors which VMs are currently running. This script enables (or disables) udev rules when a VM is started (or stopped). These udev rules manage the hotplugging and specify which USB port shall be "passed through" to which VM. They are automatically triggered when a device inserted into the USB port and then call another script that attaches the newly connected device to the specified VM. That means, while the VM is running, all USB devices plugged into the specified ports are automatically attached to the VM. While the VM is shut down, the USB devices are accessible to Unraid. It is also possible to have one USB port assigned to multiple VMs, if no more than one of these VMs is running simultaneously. When to use this method? I wanted to have a way to automatically pass any USB device connected to specific ports through to specific VMs. Passing an entire USB controller to a VM is the easiest way to get this to work. However, if we want to use more VMs than we have USB controllers, or we can't pass a controller through for some other reason, we need another option. I would recommend USB controller passthrough over this method, if possible, as it is much easier to set up or modify. This method takes some time and technical understanding to set up, but has now worked reliably for several months for me. There is only one thing that is currently not working: If a USB device is already plugged into a port designated for "passthrough" before the corresponding VM is started, it is not automatically attached to the VM, but needs to be unplugged and plugged in again. I have an idea on how to address this, but have not gotten around to testing it. Unfortunately, since it took me quite some time to develop this method I am not able to give full credit to all the sources that helped me. Among them were definitely these two posts: http://kicherer.org/joomla/index.php/en/blog/48-automatic-hotplugging-of-usb-devices-for-libvirt-managed-vms https://www.labsrc.com/unraid-automatic-usb-hotplugging-libvirt/ as well as this repository, which also contains the script I am using for attaching the USB devices to the VMs: https://github.com/olavmrk/usb-libvirt-hotplug How to set this up? The first and most complicated step is to identify the addresses of the USB ports that we want to "pass through". After that we can write the udev rules for these ports. The rules specify to which VM the USB device should be passed. After testing these rules, we will setup the script to enable or disable the rules when the corresponding VMs are started or stopped. Identifying USB controllers First, we need to get to know our motherboards USB layout. I am using an MSI Creator X299 and will use it as an example here to go through the process. First we can take a look at Unraids System Devices page to learn what USB controllers we have and what their addresses are: On this motherboard there are 3 USB controllers (marked red) with the addresses 00:14.0, 03:00.0 and 04:00.0. The latter two ( ASM3242 and ASM2142) are chips that bridge PCIe to USB. Thus, they can be passed through in the traditional way. From the motherboards manual I know that the ASM3242 provides the USB-C port on the rear I/O and the ASM2142 provides the internal Type-C header. The remaining controller is the one that is integrated into the X299 Chipset. Of the devices attached to it I can identify the Unraid USB drive (green), some other known USB devices (yellow) and two other ASM chips (blue). These chips are USB hubs (ASM1074) with one upstream and four downstream ports. The manual also tells me that the USB 3 ports on the rear I/O are connected to these ASM1074s, while the USB 2 ports and the internal headers are connected directly to the chipset. Thus the USB layout looks like this: To identify which port belongs to which controller you can also simply plug a USB device into the port in question, refresh the Unraid System Devices page and note under which controller the device is listed. You might have noticed that the two ASM1074 hubs were listed twice in the Unraid System Devices page. Once in bus 001 and once in bus 002. Bus 001 is the USB 2 bus, which i know because the two water pumps are connected to USB 2 headers. Bus 002 is the USB 3 bus, where also the Unraid flash drive is connected through one of the internal USB 3 headers. This will be important later on. Identifying USB ports Now we know which USB ports on the motherboard belong to which controller. Now we need to find out the device addresses of the USB ports. In an unraid terminal, navigate to /sys/devices. Among other things, there should be a folder called pci0000:XX, where XX are the first to digits in the address of the USB controller in the Unraid System Devices page. I will focus on the chipset controller here, so it is 00. Enter that folder. Inside that folder there should be more folders with names that match the addresses of several chipset devices from the Unraid System Devices page. We already know that the chipset USB controller has the address 00:14.0, so we enter the corresponding folder. In here vendor and device should contain the numbers shown in brackets before the address on the Unraid System Devices page. The usb1 and usb2 folders correspond to bus 001 and bus 002 from the Unraid System Devices page. Lets start with usb1. As mentioned above this is the USB 2 bus, so inside this folder we should be able to find all USB 2 ports connected to this controller (which in this example are all USB 2 ports on the motherboard). There are 6 folders inside (marked red), each corresponding to one of the 6 attached USB 2 devices. Note that the two ASM1074 hubs are included in this count, and any external USB hubs plugged into one of these ports would also be, because this is a tree-like structure. A folder corresponding to a hub will itself contain a folder for each port on the hub. Note also that we exclude the root hub and the folder 1-0:1.0 (meked yellow) because these are not of interest here. When I insert a USB stick into one of the USB 2 ports on the rear I/O, it now looks like this: The folder 1-11 has appeared. Any time I insert any device into that specific port, the folder 1-11 will appear here. I will refer to 1-11 as the address of that specific port, I don't know if that is technically the correct term, but the important thing is that it does not change. Even after a reboot, that number is specific to that port, unlike e.g. the "Device" number shown by lsusb, which changes each time the USB stick is plugged out and in again. We can also get some information about the connected devices: Each of the 1-XX folders will also contain another folder called 1-XX:1.0. With that, we have the full address of the USB port and the device that is connected to it. The USB drive I just inserted has the full address /sys/devices/pci0000:00/0000:00:14.0/usb1/1-11/1-11:1.0 In fact, any USB device connected to the same port will have that same address. Things get a little more complicated when the USB device is not as simple as a USB stick, a mouse or a keyboard: Some devices consist of several seperate devices, as is the case here with the Intel Bluetooth adapter on port 1-13 marked in green. For hubs like the ASM1074 on port 1-6, the folder 1-6:1.0 describes the hub itself, while the other folders 1-6.1 through 1-6.4 correspond to the ports provided by the hub and only appear, if a device is plugged into that port. I have here inserted devices into the ports 3 and 4 of the hub on port 1-6. The former has the complete address /sys/devices/pci0000:00/0000:00:14.0/usb1/1-6/1-6.3/1-6.3:1.0 If I were to insert a USB hub into that same port and the device into the first port of that hub, the address would look like this: /sys/devices/pci0000:00/0000:00:14.0/usb1/1-6/1-6.3/1-6.3.1/1-6.3.1:1.0 If we now take a look at the usb2 folder, the structure is the same: It is important to note that one physical USB 3 port has two separate addresses, one for operation in USB 3 mode and one for operation in USB 2 mode. My Unraid flash drive has the address /sys/devices/pci0000:00/0000:00:14.0/usb2/2-3/2-3:1.0 If I were plug a USB 2 stick into the same port, it would have the address /sys/devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3:1.0 I recommend pluggin one USB device into one port after another, checking which folders pop up in the usb1 and usb2 folders, and thus noting which USB port has which address. Writing udev rules Now that we know the addresses of each physical port on out motherboard, it is time to write the udev rules that define which ports should be "passed" to which VM. We will use the udev rule to automatically execute a script whenever a device is plugged in or out of the port specified in the rule. The script will attache (or detach) the device to the VM. I am using the "usb-libvirt-hotplug" script from here for this purpose. Download it and save it somewhere on the Unraid machine. The udev rule will look something like this: SUBSYSTEM=="usb",DEVPATH=="path_to_usb_port",RUN+="path_to_script VM_name" As previously determined, a USB device plugged into one of the USB 2 ports on my rear I/O has the address /sys/devices/pci0000:00/0000:00:14.0/usb1/1-11/1-11:1.0 If I want to automatically pass devices connected to that port through to a VM called "Ubuntu", the udev rule for that will look like this: SUBSYSTEM=="usb",DEVPATH=="/devices/pci0000:00/0000:00:14.0/usb1/1-11",RUN+="/mnt/user/appdata/scripts/usb-libvirt-hotplug.sh Ubuntu" Note that in the "path_to_usb_port" the leading /sys is excluded. Also the last section /1-11:1.0 is excluded, meaning we only use the address to the USB port and not to the device attached. If that port were a USB 3 port and I also wanted to attach USB 3 devices to the VM, a second rule would be required: SUBSYSTEM=="usb",DEVPATH=="/devices/pci0000:00/0000:00:14.0/usb2/2-11",RUN+="/mnt/user/appdata/scripts/usb-libvirt-hotplug.sh Ubuntu" Write this rule (or these two rules, each in one line) into a file called 90-testrule.rules and save it into the /etc/udev/rules.d/ directory. Then execute udevadm control --reload-rules This tells Unraid to reload the udev rules, activating this newly added one. Do this while the target VM is running. USB devices plugged into this port should now automatcally be attached to the VM. What about hubs? As mentioned above, my motherboard has two ASM1074 chips, so essentially two USB hubs directly on board. These have the addresses 2-5 (1-5 when operating at USB 2 speed) and 2-6 (1-6 when operating at USB 2 speed). The latter one hosts four USB 3 ports on my rear I/O. If I want to atutomatically attach all USB device inserted into these four ports to a VM called "Ubuntu", the .rules file for that would look like this: SUBSYSTEM=="usb",DEVPATH=="/devices/pci0000:00/0000:00:14.0/usb1/1-6/1-6.1",RUN+="/mnt/user/appdata/scripts/usb-libvirt-hotplug.sh Ubuntu" SUBSYSTEM=="usb",DEVPATH=="/devices/pci0000:00/0000:00:14.0/usb1/1-6/1-6.2",RUN+="/mnt/user/appdata/scripts/usb-libvirt-hotplug.sh Ubuntu" SUBSYSTEM=="usb",DEVPATH=="/devices/pci0000:00/0000:00:14.0/usb1/1-6/1-6.3",RUN+="/mnt/user/appdata/scripts/usb-libvirt-hotplug.sh Ubuntu" SUBSYSTEM=="usb",DEVPATH=="/devices/pci0000:00/0000:00:14.0/usb1/1-6/1-6.4",RUN+="/mnt/user/appdata/scripts/usb-libvirt-hotplug.sh Ubuntu" SUBSYSTEM=="usb",DEVPATH=="/devices/pci0000:00/0000:00:14.0/usb2/2-6/2-6.1",RUN+="/mnt/user/appdata/scripts/usb-libvirt-hotplug.sh Ubuntu" SUBSYSTEM=="usb",DEVPATH=="/devices/pci0000:00/0000:00:14.0/usb2/2-6/2-6.2",RUN+="/mnt/user/appdata/scripts/usb-libvirt-hotplug.sh Ubuntu" SUBSYSTEM=="usb",DEVPATH=="/devices/pci0000:00/0000:00:14.0/usb2/2-6/2-6.3",RUN+="/mnt/user/appdata/scripts/usb-libvirt-hotplug.sh Ubuntu" SUBSYSTEM=="usb",DEVPATH=="/devices/pci0000:00/0000:00:14.0/usb2/2-6/2-6.4",RUN+="/mnt/user/appdata/scripts/usb-libvirt-hotplug.sh Ubuntu" The first four lines handle USB 2 devices, the other handle USB 3 devices. Now say I know that I will additionally attach a USB 2 hub to port 2, which in turn hosts four USB 2 ports. The .rules file would look like this: SUBSYSTEM=="usb",DEVPATH=="/devices/pci0000:00/0000:00:14.0/usb1/1-6/1-6.1",RUN+="/mnt/user/appdata/scripts/usb-libvirt-hotplug.sh Ubuntu" SUBSYSTEM=="usb",DEVPATH=="/devices/pci0000:00/0000:00:14.0/usb1/1-6/1-6.2/1-6.2.1",RUN+="/mnt/user/appdata/scripts/usb-libvirt-hotplug.sh Ubuntu" SUBSYSTEM=="usb",DEVPATH=="/devices/pci0000:00/0000:00:14.0/usb1/1-6/1-6.2/1-6.2.2",RUN+="/mnt/user/appdata/scripts/usb-libvirt-hotplug.sh Ubuntu" SUBSYSTEM=="usb",DEVPATH=="/devices/pci0000:00/0000:00:14.0/usb1/1-6/1-6.2/1-6.2.3",RUN+="/mnt/user/appdata/scripts/usb-libvirt-hotplug.sh Ubuntu" SUBSYSTEM=="usb",DEVPATH=="/devices/pci0000:00/0000:00:14.0/usb1/1-6/1-6.2/1-6.2.4",RUN+="/mnt/user/appdata/scripts/usb-libvirt-hotplug.sh Ubuntu" SUBSYSTEM=="usb",DEVPATH=="/devices/pci0000:00/0000:00:14.0/usb1/1-6/1-6.3",RUN+="/mnt/user/appdata/scripts/usb-libvirt-hotplug.sh Ubuntu" SUBSYSTEM=="usb",DEVPATH=="/devices/pci0000:00/0000:00:14.0/usb1/1-6/1-6.4",RUN+="/mnt/user/appdata/scripts/usb-libvirt-hotplug.sh Ubuntu" SUBSYSTEM=="usb",DEVPATH=="/devices/pci0000:00/0000:00:14.0/usb2/2-6/2-6.1",RUN+="/mnt/user/appdata/scripts/usb-libvirt-hotplug.sh Ubuntu" SUBSYSTEM=="usb",DEVPATH=="/devices/pci0000:00/0000:00:14.0/usb2/2-6/2-6.3",RUN+="/mnt/user/appdata/scripts/usb-libvirt-hotplug.sh Ubuntu" SUBSYSTEM=="usb",DEVPATH=="/devices/pci0000:00/0000:00:14.0/usb2/2-6/2-6.4",RUN+="/mnt/user/appdata/scripts/usb-libvirt-hotplug.sh Ubuntu" The line for 2-6.2 is missing, because there wont be any USB 3 devices attached to that port. Also the four rules for the additional hub are only present for the USB 2 bus, because the hub only supports USB 2. (As far as I know adding more rules that will never trigger does not harm anything.) So for hubs this method is not very flexible, but as long as the hub stays in a fixed position it is fine. Enabling and disabling rules As we have seen before, activating a rule is simply done by copying the rules file into /etc/udev/rules.d/ and running udevadm control --reload-rules. Disabling the rule is equally simple: Remove the .rules file from /etc/udev/rules.d/ and run udevadm control --reload-rules again. If a rule is active but the corresponding VM is shut down and a USB device is inserted into a port managed by the rule, the script will attempt to attach the device to the VM and fail, because it is shut down. This will generate some error messages in the syslog but is otherwise not a problem. Still, it would be nice to have rules automatically enabled when their corresponding VM ist started and disabled when the VM is stopped. This would also make it possible to use the same ports for two VMs that only run mutually exclusive: VM1 starts -> rules for VM1 are loaded -> VM1 is shut down -> rules for VM1 are disabled -> VM2 starts -> rules for VM2 are loaded -> ... For this purpose I have written a script that does exactly that: You can specify the names of all VMs for which you want rules to be loaded and provide a .rules file for each of these VMs. The script monitors the state of these VMs and enables (or disables) the rules when the VMs are started (or stopped) by copying the corresponding .rules files into /etc/udev/rules.d/ and running udevadm control --reload-rules. The code: #!/usr/bin/python3 import subprocess import time import syslog ##################################### ### User configuation starts here ### ##################################### # This is the list of VMs managed by this script. # Add the names of the VMs for which you want to enable hotplugging here. # For each VM listed here a .rules file is required, see documentation for details. managed_vms = ['VM-name1','VM-name2','VM-name3'] # The path to the directory where the .rules files are stored. path_to_rules = '/mnt/user/appdata/udevrules/' # The .rules files should be named 'someprefix-vmname.rules'. The 'vmname' part must # be the same as in the 'managed_vms' list above and the prefix is defined here. rules_prefix = '90-usbhotplug-' ################################################# ### No changes should be necessary below here ### ################################################# # Uses the 'virsh list' command to get the state and names of all VMs on the host # and returns it as a formated list. def get_vms(): vms = [] # call 'virsh list --all' to get all stopped and running VMs p = subprocess.run(['virsh', 'list', '--all'], stdout=subprocess.PIPE, universal_newlines=True) # split the returned string into lines and exclude the header p = p.stdout.strip().split('\n')[2:] # for each line of the output for line in p: # add the name and the state to the return list vms.append(line.split()[1:3]) return vms # Initialize states of all managed VMs. The state is a tuple of the VM name and the # state string, which is either 'shut' when stopped or 'running' when running. vm_state = [] for vm in managed_vms: # initially all vms are stopped vm_state.append([vm,'shut']) syslog.syslog("VM USB hotplug monitor started.") while(True): # for each VM defined on the host for vm in get_vms(): # get previous state of the vm old_state = [] for v in vm_state: if vm[0] == v[0]: # match VM names old_state = v # if len(old_state) == 0, the VM is not in the managed_vms list if len(old_state) > 0: # if VM was previously running and is now shut down if old_state[1] == 'running' and vm[1] == 'shut': # remove udev rule for this VM subprocess.run(['udevadm', 'control', '--reload-rules']) syslog.syslog("Stopped " + vm[0] + " hotplugging.") # update vm_state vm_state[vm_state.index([vm[0], 'running'])][1] = 'shut' # if VM was previously shut down and is now running if old_state[1] == 'shut' and vm[1] == 'running': # copy the corresponding .rules file to '/etc/udev/rules.d' and activate it s = path_to_rules + rules_prefix + vm[0] + '.rules' subprocess.run(['cp', s, '/etc/udev/rules.d']) subprocess.run(['udevadm', 'control', '--reload-rules']) syslog.syslog("Started " + vm[0] + " hotplugging.") # update vm_state vm_state[vm_state.index([vm[0], 'shut'])][1] = 'running' # if no VM state change, do nothing # wait for 3 seconds before next check time.sleep(3) Setting up the script The script is meant for usage with CA User Scripts. You will also need python, which can be installed through NerdPack. Add a User Script and paste the code inside. You should only need to modify the three variables managed_vms, path_to_rules, and rules_prefix. managed_vms: This is the list of VMs for which you want to load udev rules. The names must be written in single quotes and comma separated. The name must be the exact name of the VM, i.e. what is shown on the Dashboard and in the "Name" field in the VM configuration. For each VM listed here a .rules file, that is to be loaded when the VM is started and disabled when the VM is stopped, is required. These files should contain the udev rules as shown above. path_to_rules must be the path to the directory where these rules are stored. (This must not be /etc/udev/rules.d/ !) The .rules files must be named "someprefix-vmname.rules", where "vmname" must be the name of the corresponding VM and "someprefix-" is defined by rules_prefix. So if I want to manage 3 VMs called "Win1", "Win2" and "Ubuntu", I have: managed_vms = ['Win1','Win2','Ubuntu'] I store my .rules files in /mnt/user/appdata/udevrules, so: path_to_rules = '/mnt/user/udevrules' I set the prefix to rules_prefix = '90-usbhotplug-' thus I need 3 files called 90-usbhotplug-Win1.rules, 90-usbhotplug-Win2.rules, and 90-usbhotplug-Ubuntu.rules in /mnt/user/udevrules. In User Scrips I set the script to run at startup of the array. In the last line of the script you can also modify the time between VM state checks if you desire. I have not tested this on other motherboards yet, but the procedure should be the same. I developed this to use it myself, but I figured there might be some other people interested in this method.
  7. I set both VMs to 6GB now and adjusted the windows pagefile settings. So far it seems to work. Thanks for the tip.
  8. Hi, I am new to Unraid and currently trying to set up two Win10 gaming VMs on my system. System specs: CPU: Ryzen 1700X RAM: 16GB Mobo: AsRock X370 Professional Gaming GPU 1: GTX 1080ti (for VM 1) GPU 2: RX 480 (for VM 2) PSU: beQuiet! 850W DPP 11 Unraid Version: 6.7.0-rc6 Plugins: Unassigned Devices (ver 2019.03.30) Community Applications (ver 2019.03.30a) Fix Common Problems (ver 2019.03.26) SwapFile (ver 2015.09.21) set to 16GB on the Array Tips and Tweaks (ver 2019.03.29) Dockers: binhex-krusader Drives: 1. Samsung EVO 256 GB as Array Disk, vDisk of VM 2 is located here 2. Samsung SM951 128GB, with Win10 installation, set as vDisk for VM1 3. 2 other SSDs, both passed to VM1 Before setting up Unraid, I had a Win10 installation on the SM951. The whole drive is set as vDisk for VM1, so that I can keep this installation. VM1 also gets two other SSDs passed, with games and programms on them. VM1 gets 8 Threads, 8GB of RAM and the 1080ti. VM2 gets 8 Threads, 7GB of RAM, the RX 480 and a 100GB Vdisk on the Array. VM2 works fine. VM1 works fine, until the system randomly freezes. Theese freezes happen only when VM1 is running. There doesn't appear to be any specific trigger for the freeze. Sometimes it happens while loading windows, sometimes while on desktop, sometimes while browsing or stress testing the GPU, with or without VM2 running. When the freeze happens, the whole system freezes, so both VMs freeze and the WebUI doesn't respond. I have to reset the system when that happens. Usually the freeze happens within 15 min from starting VM1, longest time I had it running is around 1:30 hours. What I have tried so far (chronologically): Installing SwapFile, so the system doesn't run out of memory. Installing Fix Common Problems, doesn't show anything. Installing Tips and Tweaks, set vm.dirty_background_ratio to 1 and vm.dirty_ratio to 2, read somewhere it helped with a similiar problem. Didn't. Updated Unraid from 6.6 to 6.7. Tried different combinations of i440fx/Q35 Machine and OVMF/SeaBIOS, all combinations that worked had freezing. Set up a fresh Win10 VM with vDisk on the Array, passed the 1080ti to it. Still freezes. Pass 1080ti without passing Audio Controller. Reducing PCIe speed to Gen 2 and tweaking some BIOS settings. Allow unsafe interrupts. Updated BIOS to latest Version. Some reasons I can rule out: PSU not sufficient: had both VMs running 3D stress tests, System was pulling 700W. System also freezes if VM2 is not running. Unstable OC: no OC on GPUs, 3D stress test running without problems. CPU and RAM are OCed, but have been running like this for more than a year without problems. I also tried lowering RAM speed, without effect. Both VMs run Prime95 without problems. Heat: Whole system is watercooled, also freezes happen when system is idleing too. Bad windows intallation: also happens on fresh Win10 VM. To me it appears that the problem has to be somehow connected to the 1080ti. In the attached logfile, these are the dated of the most recent freezes: Mar 31 19:20:34 Mar 31 18:35:42 Mar 31 17:58:15 There are older ones as well. Below are the VM configs. I hope someone can help me with this, thaks in advance. VM1: <?xml version='1.0' encoding='UTF-8'?> <domain type='kvm'> <name>Windows 10-1</name> <uuid>a1ce7404-65cd-43f6-c214-9aab2eeff8af</uuid> <description>Main System</description> <metadata> <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/> </metadata> <memory unit='KiB'>8388608</memory> <currentMemory unit='KiB'>8388608</currentMemory> <memoryBacking> <nosharepages/> </memoryBacking> <vcpu placement='static'>8</vcpu> <cputune> <vcpupin vcpu='0' cpuset='0'/> <vcpupin vcpu='1' cpuset='1'/> <vcpupin vcpu='2' cpuset='2'/> <vcpupin vcpu='3' cpuset='3'/> <vcpupin vcpu='4' cpuset='4'/> <vcpupin vcpu='5' cpuset='5'/> <vcpupin vcpu='6' cpuset='6'/> <vcpupin vcpu='7' cpuset='7'/> </cputune> <os> <type arch='x86_64' machine='pc-i440fx-3.1'>hvm</type> </os> <features> <acpi/> <apic/> <hyperv> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='8191'/> <vendor_id state='on' value='none'/> </hyperv> </features> <cpu mode='host-passthrough' check='none'> <topology sockets='1' cores='8' threads='1'/> </cpu> <clock offset='localtime'> <timer name='hypervclock' present='yes'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/local/sbin/qemu</emulator> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='writeback'/> <source dev='/dev/nvme0n1'/> <target dev='hdc' bus='sata'/> <boot order='1'/> <address type='drive' controller='0' bus='0' target='0' unit='2'/> </disk> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='writeback'/> <source dev='/dev/disk/by-id/ata-SMI_DISK_AA000000000000008737'/> <target dev='hdd' bus='sata'/> <address type='drive' controller='0' bus='0' target='0' unit='3'/> </disk> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='writeback'/> <source dev='/dev/disk/by-id/ata-CT1000MX500SSD1_1846E1D76760'/> <target dev='hde' bus='sata'/> <address type='drive' controller='0' bus='0' target='0' unit='4'/> </disk> <controller type='pci' index='0' model='pci-root'/> <controller type='sata' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </controller> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </controller> <controller type='usb' index='0' model='nec-xhci' ports='15'> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </controller> <interface type='bridge'> <mac address='52:54:00:ce:0e:35'/> <source bridge='br0'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </interface> <serial type='pty'> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='unix'> <target type='virtio' name='org.qemu.guest_agent.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <hostdev mode='subsystem' type='pci' managed='yes' xvga='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x2e' slot='0x00' function='0x0'/> </source> <rom file='/mnt/disk1/bios/MSIGTX1080Ti.rom'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x2e' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x0c70'/> <product id='0xf00b'/> </source> <address type='usb' bus='0' port='1'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x1b1c'/> <product id='0x1b34'/> </source> <address type='usb' bus='0' port='2'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x2516'/> <product id='0x0015'/> </source> <address type='usb' bus='0' port='3'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x8087'/> <product id='0x0aa7'/> </source> <address type='usb' bus='0' port='4'/> </hostdev> <memballoon model='none'/> </devices> </domain> VM2: <?xml version='1.0' encoding='UTF-8'?> <domain type='kvm'> <name>Windows 10-2</name> <uuid>feab9934-4f21-6ef2-b35e-0c82d7170468</uuid> <description>Secondary System</description> <metadata> <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/> </metadata> <memory unit='KiB'>7340032</memory> <currentMemory unit='KiB'>7340032</currentMemory> <memoryBacking> <nosharepages/> </memoryBacking> <vcpu placement='static'>8</vcpu> <cputune> <vcpupin vcpu='0' cpuset='8'/> <vcpupin vcpu='1' cpuset='9'/> <vcpupin vcpu='2' cpuset='10'/> <vcpupin vcpu='3' cpuset='11'/> <vcpupin vcpu='4' cpuset='12'/> <vcpupin vcpu='5' cpuset='13'/> <vcpupin vcpu='6' cpuset='14'/> <vcpupin vcpu='7' cpuset='15'/> </cputune> <os> <type arch='x86_64' machine='pc-i440fx-3.0'>hvm</type> </os> <features> <acpi/> <apic/> <hyperv> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='8191'/> <vendor_id state='on' value='none'/> </hyperv> </features> <cpu mode='host-passthrough' check='none'> <topology sockets='1' cores='8' threads='1'/> </cpu> <clock offset='localtime'> <timer name='hypervclock' present='yes'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/local/sbin/qemu</emulator> <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='writeback'/> <source file='/mnt/user/domains/Windows 10-2/vdisk1.img'/> <target dev='hdc' bus='sata'/> <boot order='1'/> <address type='drive' controller='0' bus='0' target='0' unit='2'/> </disk> <controller type='usb' index='0' model='nec-xhci' ports='15'> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </controller> <controller type='pci' index='0' model='pci-root'/> <controller type='ide' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='sata' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </controller> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </controller> <interface type='bridge'> <mac address='52:54:00:c5:25:dc'/> <source bridge='br0'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </interface> <serial type='pty'> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='unix'> <target type='virtio' name='org.qemu.guest_agent.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <hostdev mode='subsystem' type='pci' managed='yes' xvga='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x2f' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x2f' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x31' slot='0x00' function='0x3'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x046d'/> <product id='0xc52b'/> </source> <address type='usb' bus='0' port='1'/> </hostdev> <memballoon model='none'/> </devices> </domain> syslog(1)