So the goal was to first virtualize unRAID with an AMD setup and an IBM M1015 flashed in IT mode with arch linux (you can see my sig below for the complete specs.) At some later point, build a zfs vm (openindiana) with Xen with another M1015.

 

I just want to give a shout out to both ddeeds and ironicbadger for their help and support.

 

So I installed ARCH and followed ddeeds stellar thread on getting unRAID working. http://lime-technology.com/forum/index.php?topic=28878.0. Before that installed Xen 4.3, of course.

 

In short I was able to passthrough a AOC-SASLP-MV8 I had around with ease and did the same for the unRAID usb -- booted up in no time and I was able to see all the disks on the MV8 immediately. Did the same with the M1015 but encounter a kernel stack dump with the mpt2sas module in unRAID. Then tried compiling in the mpt2sas driver and was greeted with a KP.

 

mpt2sas version 14.100.00.00 loaded
scsi0 : Fusion MPT SAS Host
mpt2sas 0000:00:00.0: enabling device (0000 -> 0002)
mpt2sas 0000:00:00.0: Xen PCI mapped GSI28 to IRQ27
mpt2sas0: 32 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (1030492 kB)
BUG: unable to handle kernel paging request at edbfe00c
IP: [<c128b521>] pci_enable_msix+0x281/0x336
*pdpt = 0000000003a67027 *pde = 0000000003905067 *pte = 80000000d0542461
Oops: 0003 [#1] SMP
Modules linked in: mpt2sas(+) scsi_transport_sas raid_class
Pid: 650, comm: modprobe Not tainted 3.9.6p-unRAID #1
EIP: 0061:[<c128b521>] EFLAGS: 00010286 CPU: 0
EIP is at pci_enable_msix+0x281/0x336
EAX: edbfe00c EBX: ec7ce800 ECX: c390aa10 EDX: 00000001
ESI: c3aaa120 EDI: 00000000 EBP: c3a21ba4 ESP: c3a21b68
DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0069
CR0: 8005003b CR2: edbfe00c CR3: 03a71000 CR4: 00040660
DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
DR6: ffff0ff0 DR7: 00000400
Process modprobe (pid: 650, ti=c3a20000 task=ec7c6d80 task.ti=c3a20000)
Stack:
0000c00e 000000c0 00000000 ec494c38 ec7cebf0 ec494c38 000000c0 edbfe000
000000c2 0000000c 00002000 c00e4c38 d0540000 00000000 ec7e3394 c3a21c3c
edc939f9 edcaa523 ec7e33a8 edcaa520 000fb95c c3a21c10 c1050d77 e8d2332b
Call Trace:
[<edc939f9>] mpt2sas_base_map_resources+0x482/0x67f [mpt2sas]
[<c1050d77>] ? sched_clock_local+0x19/0x17c
[<edc94ba3>] ? mpt2sas_base_attach+0x7e/0x1844 [mpt2sas]
[<edc94c02>] mpt2sas_base_attach+0xdd/0x1844 [mpt2sas]
[<c104da53>] ? check_preempt_curr+0x29/0x64
[<c104da9d>] ? ttwu_do_wakeup+0xf/0xaa
[<c1050a39>] ? T.1963+0x31/0x35
[<c141fb7f>] ? _raw_spin_unlock_irqrestore+0x10/0x12
[<c1050be3>] ? try_to_wake_up+0x1a6/0x1b0
[<c1050c2f>] ? wake_up_process+0x29/0x2c
[<c1043cdd>] ? __alloc_workqueue_key+0x28c/0x37d
[<c1043dce>] ? __alloc_workqueue_key+0x37d/0x37d
[<edc9bde8>] _scsih_probe+0x3dc/0x4c6 [mpt2sas]
[<c10e2b38>] ? sysfs_do_create_link_sd+0x123/0x170
[<c104b542>] ? notifier_call_chain+0x26/0x48
[<c141fb7f>] ? _raw_spin_unlock_irqrestore+0x10/0x12
[<c12fa4f7>] ? __pm_runtime_resume+0x3b/0x43
[<c127fa54>] local_pci_probe+0x2d/0x51
[<c12806c2>] pci_device_probe+0x42/0x61
[<c12f2e6d>] driver_probe_device+0x90/0x183
[<c127f98e>] ? pci_match_device+0x8b/0x91
[<c12f2fa3>] __driver_attach+0x43/0x5f
[<c12f1c29>] bus_for_each_dev+0x38/0x68
[<c12f2d00>] driver_attach+0x19/0x1b
[<c12f2f60>] ? driver_probe_device+0x183/0x183
[<c12f2233>] bus_add_driver+0xb8/0x1e7
[<c12f346a>] driver_register+0x7c/0xeb
[<c128075e>] __pci_register_driver+0x2e/0x31
[<edc21123>] _scsih_init+0x123/0x145 [mpt2sas]
[<c1003087>] do_one_initcall+0x87/0x13d
[<edc21000>] ? 0xedc20fff
[<c10640f0>] do_init_module+0x3b/0x17a
[<c1065263>] load_module+0xc44/0xd93
[<c106549d>] sys_init_module+0x86/0x8a
[<c141feec>] syscall_call+0x7/0xb
Code: 8b 46 0c 89 01 8b 46 0c 89 4d d0 e8 5c d4 dd ff 8b 45 e8 03 46 18 8b 10 0f b7 46 02 89 56 08 83 ca 01 c1 e0 04 83 c0 0c 03 46 18 <89> 10 89 56 08 8b 76 10 8b 4d d0 83 ee 10 83 c1 08 8d 46 10 39
EIP: [<c128b521>] pci_enable_msix+0x281/0x336 SS:ESP 0069:c3a21b68
CR2: 00000000edbfe00c
---[ end trace 2b0966fc7a1e1c98 ]---
udevd-work[592]: '/sbin/modprobe -bv pci:v00001000d00000072sv00001000sd00003020bc01sc07i00' unexpected exit with status 0x0009

 

And when trying to destroy the VM, saw this on the console:

 

libxl: error: libxl_pci.c:1248:do_pci_remove: xc_domain_irq_permission irq=28
libxl: error: libxl_device.c:1127:libxl__wait_for_backend: Backend /local/domain/0/backend/pci/3/0 not ready
libxl: error: libxl_pci.c:1248:do_pci_remove: xc_domain_irq_permission irq=17
libxl: error: libxl_device.c:1127:libxl__wait_for_backend: Backend /local/domain/0/backend/pci/3/0 not ready

 

To rule out hardware issues, I installed ESXi 5.1u1 and was able to passthrough the same M1015 to unRAID on the first go - at least it's not my M1015. So I guess it could be Xen related for now and/or specific to my hardware or the unRAID vm.

 

Hopefully others with the same goal will have better success.

it could be Xen 4.3.

I had lots of issues trying to get passthrough working on xen 4.3 but with ubuntu 13.04

have been working with ddeeds and ironic.

 

I only managed it on ubuntu 12.04 + xen 4.2

try it one step back on xen version and see if you have the same issue.

I was jus wondering, do I need a kernel parameter to turn on IOMMU or something? Should I get something when I do this:

 

dmesg | grep -i amd-vi

 

I don't see any mention of AMD-Vi during boot up, but I was able to pass in the MV8, so dunno.

 

Yeah, good idea will try 4.2 and see what happens.

I was jus wondering, do I need a kernel parameter to turn on IOMMU or something? Should I get something when I do this:

 

dmesg | grep -i amd-vi

 

you know I did not get anything back for this command ,but my passthrough worked just fine.

 

I am telling you try Xen 4.2 instead. I think the 4.3 version is bugged.

I have followed the ddeeds guide for ubuntu+xen to the letter on 4.3 and it did not work.

but it was much easier with 4.2.

 

also some commands need to be run via root(sudo), even though in the guide they are given as is. I had issue running some of the commands until I tried it as "sudo"

 

Damn. Just tried 4.2.2, no go, same stack dump. I hoped 4.2.2 would work.

 

Maybe the unRAID vm needs some other option enabled in the kernel - I know, shooting in the dark here...

 

I have another two M1015 in another working server, I'm tempted to test with another one, just to rule out it out.

waight, just had a though here

are you using general unraid files from lime-tech?

if yes they will not work.

you need one of the recompiled ones.

go to my build post (link in the sig)

one of the referred links are for ddeeds thread where he have the  files you need to replace your stock files with. or search unraid on Xen or something...

he have recompiled unraid with xen support.

thanks vl1969, I've been using ddeeds' from the beginning with the xen options compiled in...

 

i tried another m1015 and same bug. so not the card. guess I'm out of luck.

thanks vl1969, I've been using ddeeds from the beginning with the xen options compiled it...

 

i tried another m1015 and same bug. so not the card. guess I'm out of luck.

 

Try rolling back the xen to version 4.2

 

 

Sent from my SGH-T889 using Tapatalk 4

 

 

Sorry I haven't had much time to be on here to help.

 

By any chance have googled to see if other people were able to passthrough a m1015 card in Xen (Any version)?

 

If so, any suggestions / advice / guidance on how? I will look when I can later this evening but any info you could find might help us come up with a solution.

 

No worries. Searched for mpt2sas, m1015, xen, passthrough, lsi2008 -  really didn't come across anything relevant to my particular issue. Usually when I come up empty with my searches/issues, the problem typically resides on my end...or maybe not many out there trying to passthrough a m1015 with xen in unRAID.

 

Another good thing to try would be to try with another VM, and seeing if the problem exists on other guests or just isolated with an unRAID guest.

 

Thinking about trying http://www.napp-it.org/ next.

The M1015 appears in the XenServer's HCL so it should work on Xen too.

i'm watching this thread closely, since i need to passthrough 3 m1015s

 

one question ... did you get a text menu during arch installation ... i get booted to a root shell and haven't found any references to the text menu on the interwebs

i'm watching this thread closely, since i need to passthrough 3 m1015s

 

one question ... did you get a text menu during arch installation ... i get booted to a root shell and haven't found any references to the text menu on the interwebs

 

No text menu in my case, used the efi boot directly to a root shell. No need for a text menu, bit more manual but you can follow the following:

 

https://wiki.archlinux.org/index.php/Beginners'_Guide

http://www.youtube.com/watch?v=jFvJ09LWwZA

The M1015 appears in the XenServer's HCL so it should work on Xen too.

 

Yeah the m1015 works with Xen kernel, the host sees all the disks attached to the m1015, uses the same mp2tsas. Problem could reside with the unRAID guest + hardware combination.

I tried creating an openindiana vm, to the best of my limited knowledge with solaris, i do not see the disks on the m1015 and see to notice the same error when I destory the vm:

libxl: error: libxl_pci.c:1247:do_pci_remove: xc_domain_irq_permission irq=28

libxl: error: libxl_device.c:1085:libxl__wait_for_backend: Backend /local/domain/0/backend/pci/25/0 not ready

 

PV vs HVM, could that be the issue? Nevermind, hvm only allow passthrough.

 

This is how the correctly loaded driver ( mpt2sas ) appears in unRAID on ESXi:

mpt2sas version 14.100.00.00 loaded
mpt2sas0: 32 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (2073208 kB)
mpt2sas 0000:03:00.0: irq 72 for MSI/MSI-X
mpt2sas0-msix0: PCI-MSI-X enabled: IRQ 72
mpt2sas0: iomem(0x00000000d2400000), mapped(0xf84c8000), size(16384)
mpt2sas0: ioport(0x0000000000004000), size(256)
mpt2sas0: Allocated physical memory: size(4938 kB)
mpt2sas0: Current Controller Queue Depth(3307), Max Controller Queue Depth(3432)
mpt2sas0: Scatter Gather Elements per IO(128)
mpt2sas0: LSISAS2008: FWVersion(15.00.00.00), ChipRevision(0x02), BiosVersion(00.00.00.00)
mpt2sas0: Protocol=(Initiator,Target), Capabilities=(TLR,EEDP,Snapshot Buffer,Diag Trace Buffer,Task Set Full,NCQ)
mpt2sas0: sending port enable !!
mpt2sas0: host_add: handle(0x0001), sas_addr(0x500605b001cd6620), phys(
mpt2sas0: port enable: SUCCESS

 

Based on the above I see MSI/MSI-X enabled in ESXi, maybe I need to enable for xen as well.

 

The wiki says, to enable:

kernel xen.gz msi=1

 

Have I correctly enable it in my grub?

multiboot       /xen-4.2.2.gz /xen-4.2.2.gz msi=1 dom0_mem=768M,max:768M xsave=1

 

Never mind MSI seems to be enabled by default

As a last ditch effort I decided to try ubuntu 13.04 with Xen 4.2.1, the minute I used the pciback.sh script to passthrough the m1015 I got an error on the console:

 

[    117.068743] pciback 0000:02:00.0: MSI-X preparation failed (-38)

 

Booting unRAID, I did not get the dump, however, I still don't see the disks...

pt2sas version 14.100.00.00 loaded
scsi0 : Fusion MPT SAS Host
mpt2sas 0000:00:00.0: enabling device (0000 -> 0002)
mpt2sas 0000:00:00.0: Xen PCI mapped GSI28 to IRQ27
mpt2sas0: 32 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (1030432 kB)
mpt2sas0-msix0: PCI-MSI-X enabled: IRQ 28
mpt2sas0: iomem(0x00000000d0540000), mapped(0xedc30000), size(16384)
mpt2sas0: ioport(0x000000000000d000), size(256)
usb 1-2: new high-speed USB device number 3 using ehci-pci
scsi1 : usb-storage 1-2:1.0
mpt2sas0: sense pool: pci_pool_alloc failed
mpt2sas0: failure at drivers/scsi/mpt2sas/mpt2sas_scsih.c:8120/_scsih_probe()!

 

EDIT1:

 

1) this could be related or not, the error is quite similar to what I got in 4.2.1 running ubuntu, in all honesty not quite sure I understand the the problem below

http://markmail.org/thread/2njw43iaurmvzlys, it mentions something about option ROM, not sure if that refers to the boot ROM, which i didn't flash on the m1015

 

2) also found this: http://xen.1045712.n5.nabble.com/LSI-SAS-2008-controller-in-a-pv-domu-td5674233.html. There's talk of hiding/blacklisting the module is question from dom0, any thoughts on how to accomplish that at the OS/Xen level?

 

EDIT2:

 

I tried hiding the device with the following at boot, same result:

In order to do this, edit your /etc/mkinitcpio.conf and make the following changes:

 

* Add 'xen-pciback' into the MODULES array. This will make sure that the module is included in the initrd.

* Add 'modconf' into the 'HOOKS' array . This will make sure that all the files in /etc/modprobe.d/ get included in the initrd.

 

The only other thing I did was use the same name as the module for my modprobe.d file:

 

$ cat /etc/modprobe.d/xen-pciback.conf

options xen-pciback hide=(0000:02:00.0)/quote]

do you have any updates joelones ?

 

i'm waiting for you to fix this issue before starting my migration

Hey man,

 

Few questions just to make sure you've done the basics - I'm sure you have!

 

So as I understand it, you've installed Arch and Xen 4.2.1 (from where? and how?).

You're using the files ddeeds provides for unRAID with Xen kernel level support?

What about the IOMMU or VT-D switch in your BIOS, is that enabled?

Would you be opposed to installing XenServer and getting the PCI passthrough working there - just so you know if your hardware truly supports the passthrough. If it works in XS it'll work anywhere else.

 

Finally, once you've edited your grub file to tell Xen to grab the PCI devices at boot when you run

xl pci-assignable-list

do you get any output?

 

Let me know...

Thanks for the suggestions ironicbadger.  I do get output running that command, the pci devices are assigned correctly. VT-D or IOMMU in the AMD world, is enabled on my board. I know the M1015 is fine and it works well in ESXi, I also tried a second M1015 with the same dump, so it's not the M1015. And no, I did not try XenServer - only so much tinkering you could get away with without having the wifey leave you for an investment banker or something.

 

So to clarify the goal was to get the M1015 card, also know as the LSI2008 chipset (already flashed to IT mode, version 15) working with Xen with modified unRAID that ddeeds posted in another other thread . And as I bonus, I would of liked to try VGA passthrough (as I hear that Xen excels in that department) working with windows 8.

 

From Arch Linux, specially AUR, I installed the following:

[*]system xen                 4.2.2-1 Compiled, same stack dump launching unRAID

[*]system xen-4.3-git 20130730-1 Did not compile at first, but found a working tarball somewhere in AUR (and also tried yours ironicbadger, same dump)

[*]system xen-git         20130824-1 Did not compile at all

 

I then tried ubuntu 13.04 with Xen 4.2.1, still didn't work but with another error.

 

[    117.068743] pciback 0000:02:00.0: MSI-X preparation failed (-38)

 

(Keep in mind, i tried the above in EFI boot mode, not sure if that makes a difference though.)

 

Finally, before pulling out what little hair I had left, I tried Linux Mint 14. Two versions of 4.1.3 were avaiable in the repo. This thread has a lot of info http://forums.linuxmint.com/viewtopic.php?t=112013&f=42

 

The Xen kernel would panic if I didn't have this, exactly as follows as a boot parameter

"dom0_mem=min:2048M,max:4192M"

That version of Xen seems to be sensitive about the dom0_mem parameter, who knows...

 

To my surprise the most recent one of the 4.1.3 versions worked and I was finally able to see the disks attached to the M1015. Wow, I thought, it's a miracle. I then tried VGA passthrough and the Windows 8 VM would not start.

 

I would get:

Error: (22, 'Invalid argument')

I think this was a known bug which was fixed in subsequent versions of Xen. I then downgraded to another version of 4.1.3 and VGA passthrough worked, I was able to pass through the HD 6450 card and Windows 8 provided the drivers but when trying to install the AMD Catalyst drivers the whole box froze up and I had to reboot. Nice.

 

At that point, I sort of threw in the towel - for now at least. Oh BTW,  ESXi 5.1 was a breeze to get working with the above, freakin' night 'n' day compared to Xen. I was even able to passthrough the 6450 to a Windows 8 VM with the following in the .vmx file:

pciHole.start = "1200"
pciHole.end = "2200"

And stable to boot, with the AMD proprietary drivers. Installed Openindiana/Win8 in an hour and was up 'n running.

 

Oh, your RAM *must* be reserved for your VM if you’re passing through a video card in ESXi.

 

Conclusion:

I can't tell whether Xen has a problem specifically with the M1015 or the combination of the M1015 with my hardware (see sig for complete specs), although, I was able to pass through a MV8 with no issues. I don't think my hardware is that esoteric though, my AMD chipset should be well supported. I hate to have this taint my experience with Xen, but as of now, my experience has been quite negative. For a 4.x product, it sure doesn't feel like one. I know, my issue is highly specific and in no way can I make a generalization. I'm sure the VGA passthrough is much improved in later versions of Xen but the M1015 with unRAID dump was a somewhat big issue and could be if others on this forum attempt the same.

 

One thing for sure, if new to the world of Xen you may want to budget some time to get things working, although, you may be looking for a new soulmate once you're up and running.

(Keep in mind, i tried the above in EFI boot mode, not sure if that makes a difference though.)

 

This makes a HUGE difference!!!!

 

Did you compile a xen.efi file and edit the grub2 so it can boot Xen correctly with all the various parameters you need in EFI mode?

 

I did not compile a xen.efi file, didn't know I needed to. I figured I had no boot issues with Xen kernel. Was able to run xl commands and what not. Successful passed in the unraid USB and the MV8, so its not clear to me how the efi parameters would affect the m1015 with respect to solving my issue.

do you have any updates joelones ?

 

i'm waiting for you to fix this issue before starting my migration

 

As you see, at the moment, I have not been successful with passing through the M1015 with Xen. I would suggest you attempt to test whether you can passthrough the M1015 with your hardware prior to doing a full migration.

do you have any updates joelones ?

 

i'm waiting for you to fix this issue before starting my migration

 

As you see, at the moment, I have not been successful with passing through the M1015 with Xen. I would suggest you attempt to test whether you can passthrough the M1015 with your hardware prior to doing a full migration.

 

 

try it with Ubuntu 12.04 + Xen 4.2 I had many issues on my hardware with 13.04

but passthrough worked in 12.04. (different SAS card but who knows)

I think there is a bug in 13.04 some where in virtualization modules...

 

To rule out hardware issues, I installed ESXi 5.1u1 and was able to passthrough the same M1015 to unRAID on the first go

 

No USB/PCI pass-through issues with 5.1U1?  Did you patch it at all?  I've been using 5.0U2 as I've heard a lot of horror stories with USB pass-through on 5.1/U1

To rule out hardware issues, I installed ESXi 5.1u1 and was able to passthrough the same M1015 to unRAID on the first go

 

No USB/PCI pass-through issues with 5.1U1?  Did you patch it at all?  I've been using 5.0U2 as I've heard a lot of horror stories with USB pass-through on 5.1/U1

 

I don't get it, if esxi works, why bother with xen?

Ther is an esxi vm image somewhere on this board.

 

Sent from my SGH-T889 using Tapatalk 4

 

 

Because vmware "just works", there's no challenge/fun.

Because vmware "just works", there's no challenge/fun.

 

oh yeahh  :-), you tell this to me? who spend 5 days trying to setup unRaid on ESXi starting

from version 5.1 and going to version 4.0 in version increments * -1

and NOTHING worked.