December 22, 201312 yr I am attempting to get pci passthrough working on KVM and am failing. I have an ASRock Z87 Extreme6 mobo and am attempting to passthrough one of the 2 SATA controllers. I am using the unRAID vhd from the opensuse thread on a debian server and this works fine until it comes to pci passthrough. My debugging so far.... I can confirm IOMMU is on # dmesg | grep IOMMU [ 0.000000] Intel-IOMMU: enabled I have 4 drives, 3 intended for unraid on one controller and 1 that is running the host on another # for each in a b c d;do udevadm info -q all -n /dev/sd${each} | grep DEVPATH;done E: DEVPATH=/devices/pci0000:00/0000:00:1f.2/ata2/host1/target1:0:0/1:0:0:0/block/sda E: DEVPATH=/devices/pci0000:00/0000:00:1f.2/ata4/host3/target3:0:0/3:0:0:0/block/sdb E: DEVPATH=/devices/pci0000:00/0000:00:1f.2/ata6/host5/target5:0:0/5:0:0:0/block/sdc E: DEVPATH=/devices/pci0000:00/0000:00:1c.2/0000:02:00.0/ata8/host7/target7:0:0/7:0:0:0/block/sdd /dev/sd[abc] are the drives I want to passthrough and they all reside on 1f.2 00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d5) Kernel driver in use: pcieport 00:1c.2 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #3 (rev d5) Kernel driver in use: pcieport 00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 (rev d5) Kernel driver in use: pcieport 00:1c.4 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #5 (rev d5) Kernel driver in use: pcieport 00:1f.0 ISA bridge: Intel Corporation Z87 Express LPC Controller (rev 05) Subsystem: ASRock Incorporation Device 8c44 Kernel driver in use: lpc_ich 00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05) Subsystem: ASRock Incorporation Device 8c02 Kernel driver in use: ahci 00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 05) If I attempt to pass 1f.2 to the unraid VM then I get the following in libvirtd.log (irrespective of whether I do a virsh nodedev-dettach pci_0000_00_1f_2 first or not) 2013-12-22 23:29:36.961+0000: 2454: info : libvirt version: 1.1.4 2013-12-22 23:29:36.961+0000: 2454: error : virPCIDeviceReset:933 : internal error: Unable to reset PCI device 0000:00:1f.2: no FLR, PM reset or bus reset available Looking at lspci -vv doesn't say anything about FLReset for the 1f.2 device (it does remark that the 1c device does not support). # lspci -vvs 00:1f.2 00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05) (prog-if 01 [AHCI 1.0]) Subsystem: ASRock Incorporation Device 8c02 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0 Interrupt: pin B routed to IRQ 42 Region 0: I/O ports at f0d0 [size=8] Region 1: I/O ports at f0c0 [size=4] Region 2: I/O ports at f0b0 [size=8] Region 3: I/O ports at f0a0 [size=4] Region 4: I/O ports at f060 [size=32] Region 5: Memory at f073a000 (32-bit, non-prefetchable) [size=2K] Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit- Address: fee002d8 Data: 0000 Capabilities: [70] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [a8] SATA HBA v1.0 BAR4 Offset=00000004 Kernel driver in use: ahci Google is not especially helpful (possibly pointing to a deficiency in the kvm stack somewhere that isn't present in xen? - https://bugzilla.kernel.org/show_bug.cgi?id=42636) but not really sure what to look for hence a bit stuck at this point. Ideas/suggestions would be appreciated. Cheers Matt
December 23, 201312 yr Author looking further into this, I suspect I will need to switch the sata ports around so that I passthrough the additional asmedia sata controller rather than the z87 one. The rationale being that the z87 sata controller is on a multifunction pci device which also contains some other stuff that I doubt I want to passthrough. 00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d5) 00:1c.2 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #3 (rev d5) 00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 (rev d5) 00:1c.4 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #5 (rev d5) 00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 05) 00:1f.0 ISA bridge: Intel Corporation Z87 Express LPC Controller (rev 05) 00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05) 00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 05) 02:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01) 03:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03) 04:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01) however I don't understand why a drive connected to the ASMedia controller shows up as connected to the pci bridge instead (and if that means I would need to passthrough the entire pci bridge then clearly that controller is going nowhere either).
December 23, 201312 yr Thanks for posting the update. While I don't have the same board as you, I'm new to pci-passthrough and am interested in what you find out.
December 23, 201312 yr Author I think this is actually a kvm defect. I have installed xen 4.3 and created a debian domU using xen-tools & confirmed it booted normally. I then detached the 00:1f.2 pci device and made it assignable (as per http://wiki.xen.org/wiki/Xen_PCI_Passthrough#Dynamic_assignment_with_xl) # modprobe xen-pciback # xl pci-assignable-add 00:1f.2 lspci -k shows that 1f.2 was now under pciback control. I then added the device to the xen cfg file in the usual way pci = [ '00:1f.2' ] and started the vm. xl did emit a warning about being unable to reset the device but I can see the 3 drives are available in the vm so it does appear to have carried on regardless and passed the devices through.
December 23, 201312 yr Author I have now extracted the unraid kernel out of the vhd from the xenserver thread and am hitting the problem described in this post - http://lime-technology.com/forum/index.php?topic=28665.msg273231#msg273231 I imported an unRAID VM from your 5.01 VHD, attached the USB drive and went on to boot. The unRAID kernel boots, but loops waiting for /dev/disk/by-label/UNRAID . Gives up after 30 seconds and moves on, but the system is defunct (/boot is not mounted). It turns out that the xen-aware kernel names the USB drive /dev/xvd.. , which seems to defeat either udev or emhttp, I haven't dug deep enough yet. At any rate, the USB drive (which is attached to the VM) is not seen in /dev/disk/by-label . the dom0 can see the device # ls -la /dev/disk/by-label/UNRAID lrwxrwxrwx 1 root root 10 Dec 23 17:31 /dev/disk/by-label/UNRAID -> ../../sde1 but passing this device to the unraid vm doesn't seem to show up. I am passing it by adding the following stanza to the cfg file disk = [ 'phy:/dev/sde1,xvda1,w' ] Any ideas (on what I'm doing wrong or how to move this forward) would be appreciated. Cheers Matt
December 23, 201312 yr Coincidence or not, I'm yet to see a report of success passing through a Haswell onboard SATA controller in ESXi. This used to be fairly usual in Ivy Bridge. You're the first guy i see trying that in Xen/KVM. I start to wonder if that's a Haswell related issue...
December 23, 201312 yr Author Hmmm file under "why didn't I buy a proven hardware combo" I have positively asserted the USB stick boots unraid OK as the host os so this is definitely a VM issue with unraid. I guess debugging this would be easier with access to an unraid recovery console. I don't know how to access this, if it exists, though.
December 24, 201312 yr Author a few more attempts to shave this yak.... I considered usb passthrough (http://wiki.xen.org/wiki/Xen_USB_Passthrough) but PVUSB is not available in my dom0 kernel so that's going nowhere. I then resorted to passing through the pci device the usb stick is on and this works fine. The downside on my mobo being, as per this thread http://lime-technology.com/forum/index.php?topic=30787.0 , that all the onboard ports are actually on the same pci bus so passing this through means losing all usb ports on the server. I guess I'll have to go down the pci usb card route or just wait for the "unraid on dom0" distro instead.
December 24, 201312 yr a few more attempts to shave this yak.... I considered usb passthrough (http://wiki.xen.org/wiki/Xen_USB_Passthrough) but PVUSB is not available in my dom0 kernel so that's going nowhere. I then resorted to passing through the pci device the usb stick is on and this works fine. The downside on my mobo being, as per this thread http://lime-technology.com/forum/index.php?topic=30787.0 , that all the onboard ports are actually on the same pci bus so passing this through means losing all usb ports on the server. I guess I'll have to go down the pci usb card route or just wait for the "unraid on dom0" distro instead. I have the extreme 4 but as you said, your newer motherboard shares a lot of stuff and doesn't have all the separate host controllers. Your solution of a $5 USB Controller card will work or wait till KVM / Xen are "backed" into unRAID itself. Just so you, your earlier issue with the USB showing up as xvda and not sda shouldn't matter. Mine shows up as xvda and works. Try changing this disk = [ 'phy:/dev/sde1,xvda1,w' ] To this disk = [ 'phy:/dev/sde1,sda1,w' ]
December 24, 201312 yr Author Ah ok, thanks for confirming it is not sensitive to the precise block device name. I missed this in the logs yesterday but now I see that it is rejecting names other than xvda if I use sda1 then I get [ 1.169626] blkfront: xvda2: flush diskcache: enabled; persistent grants: enabled; indirect descriptors: enabled; [ 1.179199] blkfront: xvda1: flush diskcache: enabled; persistent grants: enabled; indirect descriptors: enabled; [ 1.180806] vbd vbd-2049: 16 xlvbd_add at /local/domain/0/backend/vbd/14/2049 [ 1.180882] vbd vbd-2049: failed to write error node for device/vbd/2049 (16 xlvbd_add at /local/domain/0/backend/vbd/14/2049) and no mountable devices whereas if I pass xvda3 then I get [ 1.169422] blkfront: xvda2: flush diskcache: enabled; persistent grants: enabled; indirect descriptors: enabled; [ 1.172177] blkfront: xvda1: flush diskcache: enabled; persistent grants: enabled; indirect descriptors: enabled; [ 1.172922] blkfront: xvda3: barrier or flush: disabled; persistent grants: enabled; indirect descriptors: enabled; a mountable device at /dev/xvda3 which is /dev/sde1 from dom0 but nothing in /dev/disk/by-label I wonder what the difference between the 2 systems is.
December 27, 201312 yr I will assist tomorrow, but, for now, I can confirm that the Extreme6-Z87 will pass through the Intel-SATA controller to unraid in Xen. I am passing through the main six ports to unraid now. I am however, using a separate SATA card to boot Xen, and I have not tried to pass through the four remaining ports that run through the ASMedia controller. But the Extreme6-Z87 with an i5 is a confirmed go.
December 27, 201312 yr Author Good news. I look forward to seeing how you have configured it. I having trouble getting pciback loaded earlier enough on boot, this combined with the removal of managed domains from xl (when compared to xm and with no obvious replacement) mean id just going to avoid that problem and wrote an init script to detach it dynamically and start/stop the vm.
December 31, 201312 yr Matt, I was looking over the post and right away noticed something I missed before. I am using XenServer, and you are using KVM. That suggest the problem lies with KVM... I know that the board will pass through the two SATA controllers on board separately, as well as PCI cards (I am doing both things now). Since it is KVM, I am of no use. If you want to try setting up Xenserver, I have extensive files ;-)
January 1, 201412 yr Author Yeah I'm using xen now as well, kvm and passthrough didn't work so I gave up on it. I'm now have it running successfully on debian jessie with xen 4.3. The only thing I need to do now is get a usb pci card so I don't have to pass through the entire set of mobo usb controllers. I wrote my own init scripts to bring up the domains and deal with detaching the pci devices as fathoming how debian deals with module load order seems like a black art. I didn't use xend as I see it is deprecated so didn't want to use something that will disappear in the future. I can't use xcp-xapi as that seems tied to xen 4.1 and an earlier kernel revision atm. In case it's helpful to anyone, my init.d scripts to bring up a few domains in the current debian style. My unraid init.d, the DOMAIN_NAME var is the name var in the xen cfg file & so it expects to find it defined in /etc/xen/${DOMAIN_NAME}.cfg. You can then chain other services off this in usual debian way (https://wiki.debian.org/LSBInitScripts/DependencyBasedBoot) #!/bin/bash ### BEGIN INIT INFO # Provides: unraid # Required-Start: $syslog xen # Required-Stop: $syslog xen # Should-Start: $ups-monitor # Should-Stop: $ups-monitor # Default-Start: 2 3 4 5 # Default-Stop: 0 1 6 # Short-Description: Script to start unraid domU instance # Description: Service script for unraid, domU install of unRAID ### END INIT INFO DOMAIN_NAME="unraid" MAX_SHUTDOWN_WAIT_SECS=60 PCI_PT="00:14.0 00:1f.2" . /opt/xen-control/control-functions The last line sources a script that does most of the work, this is liberally based on the debian xendomains init.d script. #!/bin/bash -e # Provides ability to stop/start/reboot a domU domain. The control functions also support # dynamically allocating pci devices to the domain on startup. if [ -z "${MAX_SHUTDOWN_WAIT_SECS}" ] then MAX_SHUTDOWN_WAIT_SECS=30 fi . /lib/init/vars.sh . /lib/lsb/init-functions # determines if the domain is running by attempting to get the domid # for the named domain using xl domid # return value is the return value of the xl call hence 0 = is running is_running() { xl domid "${DOMAIN_NAME}" > /dev/null 2>&1 return $? } # checks if the devices listed in PCI_PT are assignable # if not and PCT_PT_DYNAMIC is set then the devices are dynamically # made assignable assign_pci_devices() { local RETURN="" if [ -n "${PCI_PT}" ] then local ASSIGNED=$(xl pci-assignable-list) if [ $? -eq 0 ] then for PCI_DEVICE in ${PCI_PT} do local IS_AVAILABLE=1 for ASSIGNED_DEVICE in "${ASSIGNED}" do [ "${ASSIGNED_DEVICE}" = "${PCI_DEVICE}" ] && IS_AVAILABLE=0 done if [ "${IS_AVAILABLE}" -eq 1 ] then xl pci-assignable-add "${PCI_DEVICE}" if [ $? -eq 0 ] then RETURN="${RETURN}${PCI_DEVICE}:SUCCESS " else RETURN="${RETURN}${PCI_DEVICE}:FAIL " fi fi done else RETURN="FAIL" fi fi echo "${RETURN}" } # starts the domain if it is not already running do_start() { log_action_begin_msg "domU ${DOMAIN_NAME} :: Preparing to start" # if running, abort if is_running then log_action_end_msg 1 "domU ${DOMAIN_NAME} :: Already running" else # assign any pci devices we need ASSIGNED="$(assign_pci_devices)" if [[ "${ASSIGNED}" == *FAIL* ]] then log_action_end_msg 1 "domU ${DOMAIN_NAME} :: PCI Devices [${PCI_PT}] are not available :: ${ASSIGNED}" else if [ -n "${ASSIGNED}" ] then log_action_cont_msg "PCI devices assigned ${ASSIGNED}" fi log_action_cont_msg "Creating domU" out=$(xl create -q "/etc/xen/${DOMAIN_NAME}.cfg" 2>&1 1>/dev/null) case "$?" in 0) log_action_end_msg 0 ;; *) log_action_end_msg 1 "${out}" ;; esac fi fi } await_shutdown() { for no in $(seq 0 ${MAX_SHUTDOWN_WAIT_SECS}) do if ! is_running then return 0 fi sleep 1 log_action_cont_msg done return 1 } do_stop() { log_action_begin_msg "domU ${DOMAIN_NAME} :: Preparing to shutdown" if is_running then xl shutdown "${DOMAIN_NAME}" 2>&1 1>/dev/null if [ $? -eq 0 ] then await_shutdown log_action_end_msg $? else log_action_end_msg 1 "domU ${DOMAIN_NAME} :: Unable to issue shutdown command" fi else log_action_end_msg 1 "domU ${DOMAIN_NAME} :: Not running" fi } do_reboot() { log_action_begin_msg "domU ${DOMAIN_NAME} :: Preparing to reboot" if is_running then xl reboot "${DOMAIN_NAME}" 2>&1 1>/dev/null if [ $? -eq 0 ] then # work out how to tell if a domain has rebooted log_action_end_msg 0 else log_action_end_msg 1 "domU ${DOMAIN_NAME} :: Unable to issue shutdown command" fi else log_action_end_msg 1 "domU ${DOMAIN_NAME} :: Not running" fi } do_status() { if is_running then log_success_msg "domU ${DOMAIN_NAME} :: Running" else log_failure_msg "domU ${DOMAIN_NAME} :: Not Running" fi } case "${1}" in start) do_start ;; stop) do_stop ;; status) do_status ;; restart|reload|force-reload) do_reboot ;; *) echo "Usage: $0 {start|stop|status|restart|reload|force-reload}" exit 1 ;; esac exit 0
January 3, 201412 yr Author One, hopefully final, piece of info on this mobo. After getting things running I then realised the config (stored on flash drive) was not persisting beyond reboots and the usb stick was being left in a bad state unless I unplugged it and reattached it. After ruling out a bad flash drive (via badblocks), I eventually realised this was in the syslog xhci_hcd 0000:00:00.0: xHCI host not responding to stop endpoint command. xhci_hcd 0000:00:00.0: Assuming host is dying, halting host. xhci_hcd 0000:00:00.0: HC died; cleaning up usb 1-3: USB disconnect, device number 2 usb 1-3.3: USB disconnect, device number 5 sd 6:0:0:0: Device offlined - not ready after error recovery the issue appears to have been fixed by a bios upgrade (from 2.1 to 2.3)
Archived
This topic is now archived and is closed to further replies.