scorcho99

Hmm, and it only happens when you use your one flash drive? Or do other devices give the code 10 if they are connected? It also happened with a USB3 hard drive, not not a usb2 flash drive. Have you tried to hide the controller from unraid with pci-stub.ids yet? Yes, I thought for sure that would fix it. I confirmed in the logs it was being assigned to pci-stub as well.

Well, I installed Windows 7 and preliminary tests show it behaves pretty much the same as XP. Code 10 if something is connected at startup.

Its even more specific, only one of my flash drives (USB3 Lexar) causes this problem if its installed prior to the VM starting. An older USB2 one is fine. I even tried adding the controllers to pci-stub but that didn't fix it either. I was going to say maybe unraid loads them first or something and they aren't getting cleanly unbound on VM start, but given it still happens when its stubbed and doesn't happen with an old sandisk cruizer that doesn't really add up to me. I ran a lot of tests and on both the NEC pci-e card and the onboard AMD USB controller it has the same effect on both and seems very reproducible once I figured out what was causing it. Since the AMD controller has two sets of ports only the one the flash drive is installed to gets a cannot start code 10. Weird. Now I should try this on bare metal to rule that out and maybe try a couple other devices. I wouldn't be surprised if esxi has the problem as well, I didn't own this flash drive when I was testing on that.

Last time I checked people were only able to passthrough nvidia quadro cards on ESXi or cards they had modded into Quadro. But perhaps that has changed? People seemed to have better luck with AMD/ATI cards on this platform. I had a 3450 and a 4850 passed through successfully with ESXi 5.5, but the cards were secondary adapters (you could still play games on them though). There was no way I could find to remove the VMWare VGA adapter from the VM, and I think I had problems if I disabled it in windows. The VM bios has no options for video card settings at all. Maybe they fixed that in 6 though. Unfortunately, I had a few weird driver bugs with the last legacy drivers which was a real bummer since they don't get updated. But I confirmed the problem still occurred on bare metal so I don't think it was related to pass through.

Booting off the card is nice option because the onboard USB3 in some cases has 4 ports whereas pci-e 1x cards only usually have 2. And I can't remember but I'm actually not sure if a 1x port is capable of providing full USB3 bandwidth to even the 2. That's a pretty good tip/idea by Zan.

So last night I tried to passthrough two cards that I've passed through before with ESXi 5.5. Sound Blaster Live! value PCI and a cheap PCI-e c-media sound card. These devices both have an additional gameport component (although the c-media card doesn't actually have it but that is a issue with bare metal as well) and they both are actually using a PCI to PCI-e bridge (the c-media appears to be a PCI device with an ASmedia bridge chip) I'm having a similar problem to the original one. I'm adding hostdev to the XML like I did with another USB3 card and the onboard AMD USB controller. But I get an "unexpected end of file..." message talking about the "device being in use". If I try to add the bridge device (which is listed in system/lspci) I get an error message that there is no such device...even though the error message indicates its using the right port id or whatever. In all cases I was also passing through the gameport device. Has anyone passed through traditional PCI devices? I guess I could try with the ACS override kernel setting/

So I did a few more tests last night and what I found was that my original assumption was not correct. When I started the server yesterday I was surprised to find that one of the controllers had a code 10 again. I had previously thought this was cleared up when rebooting the hypervisor. I then ran a bunch of reboot, shutdown, restart tests with both controllers using a couple different USB flash drives on both passed through controllers. I had one weird situation where windows may not have yet polled the flash drive (had a drive present but said "please insert a drive" or something when I tried to open it immediately after boot) but no other problems at all. My running theory now is that if I have a flash drive installed on a passed through controller, unraid tries to do something with it on startup which causes the strange state. In theory the controller should just reset. In practice, maybe that doesn't always work. I didn't run a lot of hypervisor reboot tests so I'm still not sure I've nailed it down. I would think adding the devices to the syslinux.cfg file under pci-stub.ids would prevent unraid from ever looking at them directly but I didn't have a chance to try that either.

I can check it out at some point, but honestly running these devices on Windows 7 isn't really part of the end goal. I only have a few licenses for that and the installs take up to much space for what I want them to do. But it would be interesting to know regardless. I don't think I had any issues with the NEC card like this with ESXi but it wasn't possible to even pass through the onboard USB with ESXi.

So with the help of this guide and others I have essentially successfully passed through an onboard AMD USB3 controller and a NEC 720200? controller that worked well with ESXi. More testing is required. A problem I have though (this is in Windows XP by the way) is that when I shut down the VM and then start it again (not restart) both the AMD and NEC controllers get a code 10, cannot start problem. This seemed repeatable in limited testing. The safely remove hardware option actually lists all passed through devices and the virt devices as well. It seemed (tested once) if I safely removed the USB controller before shutting down it worked when started back up. I seem to recall people having this problem with video cards at one point. I suspect this is because FLR or something is not handled well. I know this implementation frees up devices when the machine is shut down so other machines could use them. I wonder if setting the managed option to "no" in the XML combined with blacklisting the device from unraid with that pci stub option would alleviate this problem. Weirdly, my HVR-1850 capture card seems to have no issues.

It looks like the M and non-M board I have actually have different bios histories. The latest on mine is F6. Unfortunately, the newer bios from gigabyte returned a restart instead of shutdown on soft shutdown issue I had with the board with the F4 bios. That was a pretty unacceptable issue for a board that I was going to have hitched to a UPS so I can't use F6. I can't get a machine to start with passed through hardware without the allow_unsafe_interrupts option on. I never actually got to test hardware. XP is actually kind of a pain to install. You have to hand set the disk to IDE from virto to get the installer to see it. I tried the add a second disk and install the virto drivers before switching option, but windows never starts if that driver is running. The driver install seems to take forever and then when windows never boots, it just sits at the starting windows screen getting nowhere. The IDE problem might not be a problem since I could load the controller from the floppy virto driver image, but the guide suggests you should load other drivers first. There's no way to do that short of slipstreaming perhaps in the XP installer. Regardless, when the XP driver polls the disks before asking for a install location with the driver installed it also just gets stuck forever. Its up and running in IDE mode though. I just want to see if I can get a few pieces of hardware running that I had working with esxi.

I had some questions about this guide. In the guide, there is a step about adding a line: pci-stub.ids=8086:153b to the syslinux.cfg I assume this line is "black listing" the device from unraid/hypervisor to make sure it is available to VMs. Edit: I saw in another post another user had this question as well. This guide is about network controllers which are sort of special compared to many devices in that unraid will grab them. Thus this command is needed to explicitly prevent that from happening. I suppose you might want it for SATA controllers...USB controllers? as well. But things like video and sound cards it should not be needed since unraid never tries to control them in the first place. Is this necessary? I've read other guides: https://lime-technology.com/forum/index.php?topic=38259.msg368555#msg368555 CHBMB that seem to suggest nothing being added to the syslinux.cfg or go file (where is the go file located btw) and it worked just editing the XML. I have devices that have "dependent" devices, like PCI devices behind a bridge or PCI-e cards that actually are PCI devices attached behind a bridge chip on board. Do I simply have to add all these devices to the XML together or does this complicate matters? Related question, when logged in with PuTTy to the server can you edit the syslinux.cfg or other files directly on the flash drive safely? What command should I use to do that. I apologize I am a linux noob.

Thanks for the data point. What bios version are you running if you just happen to know? I'm going to update mine tonight. With the allow unsafe interrupts option I passed through a 3450 video card and I got no video when I swapped the monitor to that card. I've gotten the card to work as a secondary adapter under esxi before (there is no way I believe to remove the ESXI adapter from virtual machines) but I know that's a different setup. I'm almost positive that the same test failed without the interrupts thing the previous night but maybe not. I tried passing through the onboard video combined with the hdmi audio and it started but it seemed like I was still getting the unraid console. I'm not sure why that would be. I didn't have much time to play with this unfortunately. I still got errors when trying to run a VM with only the onboard AMD (not hdmi) sound and it failed with a message about iommu groups. But if I look at the groups I see that in addition an ISA bridge there is a SM bus controller in the group which of course aren't passed through. I haven't really gotten a good idea what this is from googling around but some suggested its a legacy interface. I could just pass it through if nothing is using it. I think my best bet is to update the bios (release notes say nothing about this, but last time I updated I fixed a shutdown issue that the release notes also said nothing about), drop the unsafe interrupts and try again. Honestly, the VGA card passthrough is academic for me. I need VMs with sound cards, TV cards and USB controllers. Unfortunately, two other sound cards I have installed (a crappy c-media pci-e one and an ancient SB live PCI one) do not show up in the drop down at all. They wouldn't work that way anyway since it seems that the UI doesn't automatically add grouped devices based on the AMD one. They're behind bridges. I'll have to hand edit the XML and I think there is a place you modify to blacklist devices? More research I guess.

Thanks for the ideas, I did some reading around this morning and came across a few of those things (probably following the same trail ntphil had). I had tried some of the options but not the allow unsafe interrupts one I don't think. I'm glad to see you don't have the acs option in your file, it didn't seem like I would need that. When I looked at the list of devices and iommu groups everything seemed to be cleanly separated and I have a lot of cards currently installed. akira62: Are the problems with the USB3.0 when doing a passthrough of the whole controller or just the USB device? I also think I might want to update the bios since there's a newer version from April. There was an option on ESXI I had to use for some devices...it was called something like MSIEnabled=false. I saw yesterday that something very similar sounded can be configured.

ntphil, do you remember what the kernel arguments you used were? I also sent you a PM. I have a Gigabyte GIGABYTE GA-F2A88X-D3H with a A4-6300 that I'm having trouble getting working. I get errors about IOMMU groups when trying to start a machine with something passed through. The reason I don't simply believe I'm stuck is I've had passthrough working with ESXI 5.5 using this hardware. I've passed through the pci-e TV Tuner, several PCI devices like sound cards, USB cards, a PCI-e USB3 card, and a couple old video cards and got them to work. It could be particular about hardware (I went through 3-4 USB3 cards before I found one that worked) it seemed pretty stable. I'm not at all a fan of ESXI admin tools for the most part and its really not the right tool for my job. Also, none of the AMD components were available to mark for passthrough and I'd like to at least access the onboard sound and maybe one of the USB controllers.