bungee91

I think the best point to try and get this down to bare bones is to remove any variables that may be causing lag, or latency. Can you remove the USB card from the XML, and attempt this without the g933/hdd? You can always pass the onboard audio so that you have sound to test to see.

1st question, what are you using for sound in this VM? I don't see that the GPU sound is being passed (01:00.1), or the onboard sound (00.14.2), so I'm not seeing audio here at all. Some people have reported issues with passing a GPU without the HDMI sound also. Even if you don't plan to use the sound card part of the GPU, it seems to cause issues at times. You can add multiple sound devices to a VM with the + symbol next to the sound card option. 2nd question (just so I know), what 2 USB devices are you passing in the xml (specifically these) <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x1532'/> <product id='0x002e'/> <address bus='11' device='2'/> </source> <alias name='hostdev0'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x1532'/> <product id='0x011a'/> <address bus='10' device='2'/> </source> <alias name='hostdev1'/> </hostdev> Since you're passing this USB controller (04:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller [1b21:1042]), any reason you're also passing individual devices?

I'd like to take a look at your full diagnostics output, hoping I'll see something useful, please post that. (tools, diagnostics, download or from command line, type diagnostics, upload file on root of flash drive)

UnRAID doesn't require a GPU, you should be able to boot with a card, and then "steal" it away from UnRAID for your use within a VM. If this is the primary GPU, as of now that is recommended to be an AMD GPU, as Nvidia is said to have an issue being the primary card, and being repurposed for a VM. If you need to purchase a GPU, keep in mind some cards (especially the newer AMD ones) have reset issues that can make virtualizing them troublesome, so definitely research prior to choosing one.

I think the deal is this (speaking on your topic of Skylake): Most hardware is compatible with UnRAID, but there are caveats to each, and not everything is perfect. If we all want to have "servers" we should all have ECC memory, and Xeon's, or equivalent. However a lot of people don't, and are very happy with their setups. Skylake being the newest shiniest Intel architecture is definitely supported, however with all new hardware you're likely to be ahead of the curve at times with support in the kernel, drivers, etc.. If you're running multiple VM's, you're also likely to have an issue with IOMMU grouping. The ACS override patch is just that, a patch, and from what I understand will not go upstream in the Linux kernel, as it is not a recommended thing to enable (yes, you're likely fine, BUT that doesn't mean it may not cause issues), there is a reason you get that annoying warning in the log, and on the system devices page. Tell Intel to add ACS on root ports, and Skylake will likely be the new recommended "desktop" processor for KVM/QEMU. However unfortunately it doesn't, and for that reason the recommended (cover most bases) is i7 "E", or Xeon E5. I can't see that stance changing without the hardware changing, and the recommendation when a primary part of the recommendation is using vt-d and virtualization.

I actually don't have this issue, but I haven't gamed on this yet, just curious really. These recommendations comes from here: https://www.redhat.com/archives/vfio-users/2015-September/msg00041.html I played with some of these settings, pinning the emulator to the same pins as the vCPU, and to the host. Noticed a small (but negligible) difference in CPU mark rating, however this is still more for latency removal. Which I think is what is primarily causing your issues. I also have never isolated any cpu's from unRAID, just assign my VM's to cores that are not as likely to be used (for my main one I used the highest set 4-5, 10-11) as I haven't had an issue that I felt that it would help to resolve.

Try this: Edit your XML and assign the emulator to the thread pair (HT core) and the CPU to the VM. For an assignment of 3 cores to the VM, and 3 HT threads (for the emulator) it'd look as follows (for me) <vcpu placement='static'>3</vcpu> <cputune> <vcpupin vcpu='0' cpuset='0'/> <vcpupin vcpu='1' cpuset='1'/> <vcpupin vcpu='2' cpuset='2/> <emulatorpin cpuset='6-8'/> </cputune> <cpu mode='host-passthrough'> <topology sockets='1' cores='3' threads='1'/> </cpu>

We now have LOTS of ways to do this!.. You can also do this (credit to Samuel Holland on VFIOl): grep . /sys/bus/cpu/devices/*/topology/core_id Results: /sys/bus/cpu/devices/cpu0/topology/core_id:0 /sys/bus/cpu/devices/cpu1/topology/core_id:1 /sys/bus/cpu/devices/cpu10/topology/core_id:4 /sys/bus/cpu/devices/cpu11/topology/core_id:5 /sys/bus/cpu/devices/cpu2/topology/core_id:2 /sys/bus/cpu/devices/cpu3/topology/core_id:3 /sys/bus/cpu/devices/cpu4/topology/core_id:4 /sys/bus/cpu/devices/cpu5/topology/core_id:5 /sys/bus/cpu/devices/cpu6/topology/core_id:0 /sys/bus/cpu/devices/cpu7/topology/core_id:1 /sys/bus/cpu/devices/cpu8/topology/core_id:2 /sys/bus/cpu/devices/cpu9/topology/core_id:3 or this: grep . /sys/bus/cpu/devices/*/topology/thread_siblings_list Results: /sys/bus/cpu/devices/cpu0/topology/thread_siblings_list:0,6 /sys/bus/cpu/devices/cpu1/topology/thread_siblings_list:1,7 /sys/bus/cpu/devices/cpu10/topology/thread_siblings_list:4,10 /sys/bus/cpu/devices/cpu11/topology/thread_siblings_list:5,11 /sys/bus/cpu/devices/cpu2/topology/thread_siblings_list:2,8 /sys/bus/cpu/devices/cpu3/topology/thread_siblings_list:3,9 /sys/bus/cpu/devices/cpu4/topology/thread_siblings_list:4,10 /sys/bus/cpu/devices/cpu5/topology/thread_siblings_list:5,11 /sys/bus/cpu/devices/cpu6/topology/thread_siblings_list:0,6 /sys/bus/cpu/devices/cpu7/topology/thread_siblings_list:1,7 /sys/bus/cpu/devices/cpu8/topology/thread_siblings_list:2,8 /sys/bus/cpu/devices/cpu9/topology/thread_siblings_list:3,9 Both of these, plus the one mentioned a couple posts up, and lstopo are in agreement.. I consider this question now well covered to discover what the layout is. I think all of this has derailed this thread a bit, but certainly will help with lag, just not the only issue going on.

Installing packages by themselves without a plugin is not recommended, as there is no tracking of what is installed and could cause conflicts. Ok, I put the warning that others will mention if I didn't... Now that that's out of the way! If for whatever reason others are curious, and like that shiny picture representation, do the following. You need all the packages in the attached picture (I think I may have 1 unneeded one in the list), you can find them here for Slackware64 14.1 http://packages.slackware.com/ Then you can manually install each one (a pain/the way I did it) with: upgradepkg --install-new packagename Example: upgradepkg --install-new libXext-1.3.2-x86_64-1.txz You need to be in the same directory at the console/SSH as where you have the files stored, for me this was /boot/hwloc OR (much easier) put all the needed packages in the boot/extra folder on your USB drive (if you don't have one, create it) and reboot. Everything will be installed at next boot. Then SSH in (or at console) and type lstopo namethepic.png and it will output to the directory you are in at the command line. If you get an error about a shared library not available, you missed a package. You can search for the one it is asking for by using the search box at the top of the Slackware packages page. I removed all the packages afterwards with removepkg nameofpackage Example:removepkg cairo-1.12.16-x86_64-1_slack14.1.txz Also, this is a list of the commands/output you can do with lstopo http://manpages.ubuntu.com/manpages/quantal/man1/lstopo.1.html This was far too much hassle for the purpose of it, but I was determined and thought I'd share.

10+ packages later, and I have the lstopo graphic output (I had no idea it would take that many packages to make a png of my system!). Either way, pretty cool, but is the opposite of what virsh capabilities output displayed. Attached

Well I'm starting to find this interesting, primarily from an efficiency point of view! A couple of things, the <emulatorpin cpuset= > seems to be widely used to reduce latency. Maybe something that is worth trying within your XML. lstopo or also known as hwloc-ls displays the hierarchical topology map of the current system. Getting this to run in Slackware would give the output we want to see. hwloc-ls is available for Slackware https://www.open-mpi.org/projects/hwloc/ Lots of good info below, if someone has enough time to kill (I do not right now) you should follow the steps in the 2nd link. Primer: https://a20.net/bert/tag/libvirt/ Guide: http://docs.openstack.org/developer/nova/testing/libvirt-numa.html

Getting more info here, but still have some questions. Was advised to do the following: Which output the following (relevant information only): <topology> <cells num='1'> <cell id='0'> <memory unit='KiB'>32839588</memory> <cpus num='12'> <cpu id='0' socket_id='0' core_id='0' siblings='0'/> <cpu id='1' socket_id='0' core_id='0' siblings='1'/> <cpu id='2' socket_id='0' core_id='1' siblings='2'/> <cpu id='3' socket_id='0' core_id='1' siblings='3'/> <cpu id='4' socket_id='0' core_id='2' siblings='4'/> <cpu id='5' socket_id='0' core_id='2' siblings='5'/> <cpu id='6' socket_id='0' core_id='3' siblings='6'/> <cpu id='7' socket_id='0' core_id='3' siblings='7'/> <cpu id='8' socket_id='0' core_id='4' siblings='8'/> <cpu id='9' socket_id='0' core_id='4' siblings='9'/> <cpu id='10' socket_id='0' core_id='5' siblings='10'/> <cpu id='11' socket_id='0' core_id='5' siblings='11'/> </cpus> </cell> </cells> </topology> Now the siblings value = the cpu id value, so I didn't get the expected output as what was explained above. However the core_id value shows that 0-1, 2-3, 4-5 .... are the correct pairings. Checking to make sure that's right, and why my output is different than the cpu id 0 = siblings 1 (to align with what was explained). Edit: It is supposed to list as explained, but for whatever reason mine does not. Talk of its creation here:http://libvir-list.redhat.narkive.com/WBIG9szS/libvirt-add-hyperthreaded-sibling-info-to-virsh-capabilities

I understand X and Y coordinates here.. Yep, that was my thoughts also... Well, you confirmed my thoughts, will plan to post in the VFIO group to get some additional eyes and thoughts on it!

Any input on those who have tried it, with results that don't help to distinguish this? See here http://lime-technology.com/forum/index.php?topic=46664.msg450726#msg450726 Tried multiple times same results, that one was done at console with nothing else loaded (Docker/VM's), and no plugins (with exception to the ones needed to run this).

In my testing with a G210 Zotac, it was hit or miss. Driver install in Windows would cause it to lock up, can't remember if whole server or just VM, but seemed fine afterwards. OE required the legacy drivers, and at the time they weren't included in the LT image (may still not be). Anyhow, I had enough issues to replace it, hopefully yours will act differently (it can be different by brand). As far as alternative, I'd recommend the GT720, it serves me extremely well.

Interesting, yeah something is weird here... If I get REALLY bored, I could boot arch or Ubuntu and run this without UnRAID to find out for sure. I'm pretty certain mine are 0-6, 1-7, 2-8, etc... as I see no reason for Intel to change the grouping between the 4-core and 6-core CPU's, with the same basic architecture. No... They also didn't tell you about the terrible reset issues those R9 Nano cards have, and that the build is likely not reliable for reset/shutdown (kind of a deal breaker)... Info: http://vfio.blogspot.com/search?updated-min=2016-01-01T00:00:00-07:00&updated-max=2017-01-01T00:00:00-07:00&max-results=1 ....You can paint a picture anyway you'd like to make it look the way you prefer. However admittedly it was more a proof of concept, then meant for someone to actually do.

It sounds more CPU related than GPU, as 4k video shouldn't be hardware decoded (I believe) and rely on the CPU to decode. I assume that you have 8 cores to assign, correct? UnRAID likes to do tasks on core 0 (from what I've read), so I recommend not using that in the VM. Your XML shows 0,1,4,5 as assigned to this, any particular reason for choosing those? I'd try your top 4 cores (likely 4-7) or 2,3 & 6,7 if this is anything like Intel cores (I don't think it is, but I'm not certain on the recommendation for AMD assignments).

I decided to run the latency script to see if it gave me any insight into the groupings for my 5930k. I initially did this from SSH, but then decided it'd be best to do this from the console, no plugins, and certainly no VM's or Docker running. Anyhow, the results were the same regardless, and in my case aren't conclusive or helpful at all! For the most part (with higher being better here), the core to itself is a 10, everything else is a 4 or 5, without any differences in between. If anyone else has run this, did you see the expected 8 or 9's between logical and HT cores? I don't have any issues per say, however I expected this to show something that it is not. No special parameters in my syslinux file that would affect it.

I chose to do without, and the 5930k fit the bill quite nicely for my build. The additional 8 PCIe lanes I thought may be handy, so I sprung for the 5930 over the 5820, however it was likely unneeded, and certainly not needed for what your usage is stated to be. However, ECC ram is certainly a better proposition for stability, and gives a mechanism to detect single bit errors. I felt that Intel charged too much in terms of performance vs a "high end" desktop for the extra cost to be worth it for me. If the E3 supported ACS (I previously had a Z97 setup, with plenty of shortcomings for the amount of VM's with hardware assignment I planned to support), I would have likely gone that route, but it doesn't, and I couldn't justify the cost of an E5. If you do go X99 (5820k), you have a quad channel memory bus, so it is recommended to not exceed the 1 stick of RAM per channel. This fit my requirements fine (32GB total, 4X8GB), but keep that in mind if you plan to load up all 8 slots.

UnRaid doesn't require a GPU at all, so you shouldn't have an issue. Since these are AMD cards, you should be able to assign both to separate concurrently running VM's without issue. You should have no issue "stealing" the GPU from unRaid to the VM you choose to use it with (I do this now).

If you're doing hardware pass-through with more than 1 VM (really it should be any, but you'll likely be fine with just 1) it is recommended to get a CPU with support for ACS on root ports (this will ensure proper isolation of devices from the PCIe slots). If you want ECC - E5 Xeon or better. If you don't want ECC - i7"Extreme" (no Skylake products as of yet, only Haswell, Ivy/Sandy). Also, yes I seem to preach ACS, however it is both the stance of Limetech/JonP and also Alex Williamson who is kind of the "go to" for VFIO and updates regarding this kind of thing.

If it's an AMD card, you'll be able to "steal it" from the console just fine (I do exactly that). If you only have 1 card (no iGPU) and it's an Nvidia card, as of now your S.O.L., as there seems to be an issue with the only GPU being an Nvidia refusing to be used in this case. A lot of BIOS's have the option to choose which PCIe slot to use as the primary graphics output, mine lists all 4 16X (length, not wired) slots as an option. There is no reason to "never buy another ATI card again" unless you really have a tendency to be brand loyal (best bang/features for the $ at that time , plus good virtualization is my pick). AMD however does have many newer cards that don't properly "reset" which causes virtualization issues, or assignment/VM starting errors. Some reading on the VFIO mailing list https://www.redhat.com/archives/vfio-users/ or the many blogs put out by Alex Williamson will help to understand which AMD cards are recommended. I use my 260X without issue, so I know they're not all bad (plus a lot of people use the 6450 without issue). Nvidia is not without issue though, their attempt to stop us from virtualizing these resulting in an Error 43 in Windows, and needing to hide Hyper-v, are one example. Also all my Nvidia cards require the MSI interrupt fix to avoid the "demonic audio" issue at times.

You should do all of the post-install tuning things from this guide: http://lime-technology.com/wiki/index.php/UnRAID_6/VM_Guest_Support Also, no need to try to adjust SeaBIOS, it is not the issue.

I'd assume this functionality is how they solved the request of "how do I start a non running VM easily, without using another device to access the WebUI" I typically use my phone, as I'm now "all in" with the Server doing it all. It would also make sense why OVMF is now the recommended default, as you lose console output with a VM using SeaBIOS (this will seriously be a pain to switch all (my/SeaBIOS) Windows VM's to!).

A couple things to consider here, not all apply, but you should know the current shortcomings: If you only have Nvidia GPU's (and no iGPU), for whatever reason only 1 can typically be assigned to a running VM, and the other will output the console. There are many reports of people having this issue attempting to use the primary GPU (Nvidia only), and it not working. This is not the case with AMD, and you can steal it from the console typically without issue (I do exactly this, also X99). Make certain you are using the latest BIOS and Drivers for your cards. Some cards behave better than others, it is unfortunately just how it is. This can typically be seen with lag, freezing on guest reboots, unable to power on a previously shut down VM, etc... If you are now using SeaBIOS, you can try switching to OVMF, and see if this helps. For Windows installations, you cannot do this after installation (it will not boot). You can attempt to use Q35 instead of i440fx, however with Windows, this will likely prompt you to reactivate. You can try loading the rom file for your card in the xml. From http://lime-technology.com/wiki/index.php/UnRAID_6/VM_Management As a last resort, you can attempt to manually provide the ROM file for your video card by editing the XML for your VM (see below procedure). Edit XML for VM to supply GPU ROM manually[edit] From another PC, navigate to this webpage: http://www.techpowerup.com/vgabios/ Use the Refine Search Parameters section to locate your GPU from the database. Download the appropriate ROM file for your video card and store the file on any user share in unRAID. With your VM stopped, click the icon for your VM, then select Edit XML from the context menu. For SeaBIOS-based VMs Scroll to the bottom of the XML and locate this line (the host=##:##.# part may look different for you than from the example below): <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/> Modify this line to supply the ROM file to the VM, like so: <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=/mnt/user/sharename/foldername/rom.bin'/> Change the path after /mnt/user/ to the actual user share / sub-folder path to your romfile. For OVMF-based VMs Scroll to the bottom of the XML and locate this section (the <address> parts may look different for you than from the example below): <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </hostdev> After the </source> tag, add the following code: <rom file='/mnt/user/sharename/foldername/rom.bin'/> Change the path after /mnt/user/ to the actual user share / sub-folder path to your romfile. Once done editing the XML, click Update and try starting your VM again to see if GPU assignment works properly.