scorcho99

I have this working on my z370+i5 8400 machine with Windows 10 and linux VMs. Not sure what I'm doing that others aren't but I'm using the igpu has the host GPU (so it loses output when VM starts), the VMs have to be seabios, I have vesafb and efifb disabled but I doubt that is needed. I may have added i915.enable_gvt=1 to syslinux, but I think that was just in an attempt to get mediated passthrough to work. I can't remember if I blacklisted the i915 driver or not. I'm using 6.6.7 unraid

I don't have my notes that I use for this, but the command at least looks right at first glance to me. And that looks like the XML line.

I personally usually create the VM in unraid webUI and then use the command line to convert the disk type. Then I modify the XML to change the type from raw to qcow2. But qcow2 support in the webUI wasn't always available. If you already have a disk it will need to be converted, if starting fresh you probably wouldn't need to do this conversion.

This might help. https://blog.wikichoon.com/2014/03/snapshot-support-in-virt-manager.html It looks like your snapshot manager button is disabled, and you're using raw format disks rather than qcow2. It might be possible to snapshot with raw but I'm not sure how that would work.

Its not in front of me, but IIRC you have to open the virtual machine itself, not the host overview and its under one of the menu bar items.

I had a couple of questions related to this: Does this work with the mediated passthrough? (multiple virtual GPUs created from one real devices) or only GVT-d (single device)? Reading around it sounds like coffeelake support exists but its a pretty recent development and there aren't a lot of reports in the wild. Some people claim they have it working on arch though, I believe it requires 5.1 or 5.2 linux kernel. Are there plans to merge this in?

I'm curious if anyone else has any numbers on some modern GPUs when used with unraid. I've been testing this out the past couple days with my limited selection of cards. I've found a couple weird things, like my nvidia cards seem to use more power when the VM is shutdown than at the desktop. But my AMD cards seem to be at their lowest when they're unloaded, albeit not by much. Any observations are of interest. I'm testing with a killawatt and using subtraction with different cards swapped in. The idle power is a little noisy on this system but I'm pretty confident in the read +/-1watt This is with a gold Antec power supply. My GT710 seems to idle at 5watt, although I'm using it as a host card so perhaps it would do better with a driver loaded. My R7 250 DDR3 seems to idle at 6watt My 1060 3GB seems to idle at 14watt which seems high. Has anyone gotten zerocore on AMD to work in a satisfactory way? It didn't seem to make much difference to be versus shutting down the VM. It appears to be broken in newer drivers as well, the fan only shut down when I loaded an old 14.4 driver in Windows 7. There are forum posts indicating it doesn't even work in Windows 10.

I have an Asus H77 motherboard in my main rig that supports vt-d. IIRC there are a lot of forum posts complaining about it but I think it came to everything with bios updates eventually.

Believe it or not, I read this whole thread. I think I even maintained some of the information I read. Some one asked for only manual/scheduled hash creation, rather than using inotify to do the hashes when files are created. Was that feature ever added? I'd rather just swoop through nightly and do them then during the day when the server is busy doing other things. Plus, inotify-tools has issues where it can miss adding watches or even perform erroneous watches when files are moved. I think that might explain the issues some people have had while I was reading this thread.

I'll have to check my script at home. I've been running that for awhile. I suspect it just had a bug in it that I probably fixed at some point. Edit: I updated the original post with my new script, I think it corrects the error or at least its different.

I use virt-manager with spice VMs. I don't have my notes in front of me but I believe I used space invader one's youtube tutorial on virt-manager setup and then did similar to use to modify the VM to use spice display. AFAIK virt-manager is not available for Windows however.

Did this ever happen? I gather LVM is part of the unraid kernel but not bcache?

What was the usb reset issue people were having? I'm not seeing it with my ESXi6u1+6.1.7 combination I don't think. The guide says you can't do spindown or smart with RDM...mine seem to be working. I remember I had to create the RDM files using a different option, set the appropriate SCSI controller in the VM and then hack the ESXi startup script so it wouldn't poll the drives itself every hour or so causing them to spin up for no reason. I used to get all this junk in the direct console when it was running passed through disks but that seems to have disappeared with 6.1.7.

Well, that blacklist option doesn't seem to do anything. Does anyone who knows linux and unraid better than I know of a way to prevent the xhci / usb3 driver from ever loading? Its really the last idea I have after pci-stub failed. I don't need unraid to use any of the USB3 controllers at all. I'm wondering if others have problems? Can anyone test installing a USB3 device to a USB3 port (BEFORE turning on the host) that is passed through to a VM and tell me if the device appears OK in the VM? I ran a lot of tests and everything works perfectly if I have USB2 devices connected. I tested with external hard drive, flash drives, usb mouse and usb sound cards. On both the pci-e card and on the onboard controllers. I even used an extension cable to turn a USB3 device that had caused the problem into a 2.0 one and it worked fine. Its only USB3 devices that mess things up. Between that and the inability to get old PCI (classic) devices working I'm pretty much stuck on this project.

So I haven't played with this much recently. I still have that problem. I got ESXI working and tried to reproduce there and I could not. Both onboard and NEC card seem to work prefectly for me once I set it up correctly. I noticed that when VMs are down/off that use the passed through controller with the samsung external harddrive with ESXI the activity light remains off and the drive is off (this drive is weird and only turns on if it detects it is connected or something. With unraid, the drive shuts off when a VM shuts down but then it will come back on after. This leaves me leaning towards the idea that the host screwing around with the controllers and leaving them in a gummed up state so that they don't always work. I found a post about some one doing passthrough with an AMD videocard and having the passthrough messed up after he loaded the AMD drivers on the host (he had two AMD cards) even though he had assigned the device to pci-stub. He uninstalled the radeon driver from the host and it solved it. Its not the same thing but its sort of close. My motherboard is gigabyte so that weird on/off charge feature that has no options in the bios also seems like something I could blame. All I can think now is to try blacklisting the xhci module. I have yet to see the problem with only USB2 devices attached although I've only tried a few. I think I just need to add: append xhci_hcd.blacklist=yes initrd=bzroot to syslinux.cfg

Hmm, and it only happens when you use your one flash drive? Or do other devices give the code 10 if they are connected? It also happened with a USB3 hard drive, not not a usb2 flash drive. Have you tried to hide the controller from unraid with pci-stub.ids yet? Yes, I thought for sure that would fix it. I confirmed in the logs it was being assigned to pci-stub as well.

Well, I installed Windows 7 and preliminary tests show it behaves pretty much the same as XP. Code 10 if something is connected at startup.

Its even more specific, only one of my flash drives (USB3 Lexar) causes this problem if its installed prior to the VM starting. An older USB2 one is fine. I even tried adding the controllers to pci-stub but that didn't fix it either. I was going to say maybe unraid loads them first or something and they aren't getting cleanly unbound on VM start, but given it still happens when its stubbed and doesn't happen with an old sandisk cruizer that doesn't really add up to me. I ran a lot of tests and on both the NEC pci-e card and the onboard AMD USB controller it has the same effect on both and seems very reproducible once I figured out what was causing it. Since the AMD controller has two sets of ports only the one the flash drive is installed to gets a cannot start code 10. Weird. Now I should try this on bare metal to rule that out and maybe try a couple other devices. I wouldn't be surprised if esxi has the problem as well, I didn't own this flash drive when I was testing on that.

So I did a few more tests last night and what I found was that my original assumption was not correct. When I started the server yesterday I was surprised to find that one of the controllers had a code 10 again. I had previously thought this was cleared up when rebooting the hypervisor. I then ran a bunch of reboot, shutdown, restart tests with both controllers using a couple different USB flash drives on both passed through controllers. I had one weird situation where windows may not have yet polled the flash drive (had a drive present but said "please insert a drive" or something when I tried to open it immediately after boot) but no other problems at all. My running theory now is that if I have a flash drive installed on a passed through controller, unraid tries to do something with it on startup which causes the strange state. In theory the controller should just reset. In practice, maybe that doesn't always work. I didn't run a lot of hypervisor reboot tests so I'm still not sure I've nailed it down. I would think adding the devices to the syslinux.cfg file under pci-stub.ids would prevent unraid from ever looking at them directly but I didn't have a chance to try that either.

I can check it out at some point, but honestly running these devices on Windows 7 isn't really part of the end goal. I only have a few licenses for that and the installs take up to much space for what I want them to do. But it would be interesting to know regardless. I don't think I had any issues with the NEC card like this with ESXi but it wasn't possible to even pass through the onboard USB with ESXi.

So with the help of this guide and others I have essentially successfully passed through an onboard AMD USB3 controller and a NEC 720200? controller that worked well with ESXi. More testing is required. A problem I have though (this is in Windows XP by the way) is that when I shut down the VM and then start it again (not restart) both the AMD and NEC controllers get a code 10, cannot start problem. This seemed repeatable in limited testing. The safely remove hardware option actually lists all passed through devices and the virt devices as well. It seemed (tested once) if I safely removed the USB controller before shutting down it worked when started back up. I seem to recall people having this problem with video cards at one point. I suspect this is because FLR or something is not handled well. I know this implementation frees up devices when the machine is shut down so other machines could use them. I wonder if setting the managed option to "no" in the XML combined with blacklisting the device from unraid with that pci stub option would alleviate this problem. Weirdly, my HVR-1850 capture card seems to have no issues.

I had some questions about this guide. In the guide, there is a step about adding a line: pci-stub.ids=8086:153b to the syslinux.cfg I assume this line is "black listing" the device from unraid/hypervisor to make sure it is available to VMs. Edit: I saw in another post another user had this question as well. This guide is about network controllers which are sort of special compared to many devices in that unraid will grab them. Thus this command is needed to explicitly prevent that from happening. I suppose you might want it for SATA controllers...USB controllers? as well. But things like video and sound cards it should not be needed since unraid never tries to control them in the first place. Is this necessary? I've read other guides: https://lime-technology.com/forum/index.php?topic=38259.msg368555#msg368555 CHBMB that seem to suggest nothing being added to the syslinux.cfg or go file (where is the go file located btw) and it worked just editing the XML. I have devices that have "dependent" devices, like PCI devices behind a bridge or PCI-e cards that actually are PCI devices attached behind a bridge chip on board. Do I simply have to add all these devices to the XML together or does this complicate matters? Related question, when logged in with PuTTy to the server can you edit the syslinux.cfg or other files directly on the flash drive safely? What command should I use to do that. I apologize I am a linux noob.