SpaceInvaderOne Posted March 20, 2017 Share Posted March 20, 2017 PLEASE NOT THIS GUIDE IS OUTDATED NOW. NEW VERSIONS OF QEMU/UNRAID ALLOW BOOTING A VM DIRECTLY FROM THE NVME CONTROLLER SO NO NEED TO USE CLOVER AS SUGGESTED IN THIS GUIDE. YOU CAN JUST PASSTHROUGH THE CONTROLLER AND THE VM WILL BOOT FROM IT. Hi Guys, this Video is a tutorial on how to pass through an NVMe controller then boot a VM from it in unRAID. This method uses a modified clover boot loader with the NvmExpressDxe-64.EFI added to it to be able to use the NVMe drive then boot. You will also see how to copy your existing windows VM from a vdisk directly onto the NVMe drive. This method of pass through will also allow unRAID to have access to the device when the VM isn't using it as it isn't stubbed from the host. Hope you find it useful How to passthrough and boot from an NVMe controller for bare metal performance in unRAID Link to comment
psparks Posted March 20, 2017 Share Posted March 20, 2017 Hey Gridrunner, First off thanks for the videos, they are super helpful and very relevant to things I'm interested in doing. I've tried your tutorial out, as it's something I've been trying to solve constantly for the last few weeks. I am getting stuck hover on the clover bootloader piece. I tried using your clover img file to boot windows directly from my nvme disk, but it seems like there is some problem with that file for me, as it only loads to a grey screen that says clover boot manager, and no further. I started to dive down the road of building my own boot loader with the clover configurator, but quickly turned back as that program is daunting and somewhat out of my league at the moment. any suggestions on ways to get windows to load fully, or why i might be running into this issue in the first place? Link to comment
planetwilson Posted March 20, 2017 Share Posted March 20, 2017 Brilliant video as usual I am actually surprised that the non-pass through speeds are as high as they are. I am assuming that there is no way for this to work with OSX currently? It certainly wouldn't put me off grabbing an NVMe drive anyway with the the first set of speeds to be honest. Link to comment
SpaceInvaderOne Posted March 20, 2017 Author Share Posted March 20, 2017 Yes, this can be done with osx. but you have to patch the IONVMeFamily.kext so we can use non-apple nvme drives https://github.com/RehabMan/patch-nvme I haven't tried myself yet but in my VM but it should work fine. Link to comment
SpaceInvaderOne Posted March 20, 2017 Author Share Posted March 20, 2017 1 hour ago, psparks said: Hey Gridrunner, First off thanks for the videos, they are super helpful and very relevant to things I'm interested in doing. I've tried your tutorial out, as it's something I've been trying to solve constantly for the last few weeks. I am getting stuck hover on the clover bootloader piece. I tried using your clover img file to boot windows directly from my nvme disk, but it seems like there is some problem with that file for me, as it only loads to a grey screen that says clover boot manager, and no further. I started to dive down the road of building my own boot loader with the clover configurator, but quickly turned back as that program is daunting and somewhat out of my league at the moment. any suggestions on ways to get windows to load fully, or why i might be running into this issue in the first place? @psparksUm, thats strange its not working for you. You shouldn't even see the clover bootloader screen (the grey screen) The config.plist in my file should boot the bootmgfw.efi from the windows EFI partition and boot windows. Do you see any options on the clover screen to boot from? Any hard drives listed. If so you should be able to select it and press enter to boot. Also, how have you installed the windows os to the nvme drive. Did you install it directly to the nvme or did you copy across the install from your windows vdisk like i did in the video? It will not boot from a windows vdisk just stored on the nvme drive. It must be a proper partitioned disk. Also one other thing to check. Your VM must be OVMF it cant be seabios. You have to be using the EFI boot windows 10. Hope this may help. Link to comment
psparks Posted March 20, 2017 Share Posted March 20, 2017 @gridrunner Yea it is super weird. I followed the tutorial to the T, converting a working vdisk using the easeus todo software, then formatting the leftover partition with unassigned devices. On the grey screen there are no machines to boot from, just the clover icon. the VM is also ovmf. I'm going to start fresh from the begining and try it again. I think my issue may have been the formatting of the disk going into the whole process. I'm going to try to do again it by formatting it with unassigned devices and not giving it any fs/partitions (it may have had a single partition going into the process). currently it's showing the file systems types in unassigned devices for those windows partitions that the software made, and I saw that it wasn't showing those in your video. Link to comment
billington.mark Posted March 20, 2017 Share Posted March 20, 2017 You can boot to windows with a passed through NVMe without using clover at all. I've been doing this for a while since the first 6.3 RC. Bear in mind this assumes you're doing a fresh install. The only reason you cant do this natively is that Unraid ships with an older OVMF bootloader for VMs which is before it started supporting NVMe devices. We will just be downloading a referencing a different bootloader so we add support for direct NVMe booting. So basically, build the VM (dont start it yet) and pass through the NVMe device: IOMMU group 41 [144d:a802] 81:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM951/PM951 (rev 01) would be: <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x81' slot='0x00' function='0x0'/> </source> </hostdev> Download OVMF firmware from here https://www.kraxel.org/repos/jenkins/edk2/. You want the edk2.git-ovmf-x64-x-xxxxxxxx.xxxxx.xxxxxxxx.noarch.rpm file open with 7zip and drill down until you get to the edk2.git\ovmf-x64\ folder and pull out the OVMF-pure-efi.fd and OVMF_VARS-pure-efi.fd files and put them on an accessible share somewhere. Preferably on a cache only share (be sure to do both). in the xml below, Ive put them in /mnt/user/VMData/Windows10VM. reference the new files in XML (replace the default values): <os> <type arch='x86_64' machine='pc-q35-2.7'>hvm</type> <loader readonly='yes' type='pflash'>/mnt/user/VMData/Windows10VM/OVMF-pure-efi.fd</loader> <nvram>/mnt/user/VMData/Windows10VM/OVMF_VARS-pure-efi.fd</nvram> <boot dev='hd'/> </os> Boot the VM and go to the boot menu. Change the boot order to the DVD drive as top, NVMe as 2nd. Save and boot from the windows installation ISO. the NVMe drive should show as an available installation target in windows setup. Run through the installation as you normally would and you're done Link to comment
planetwilson Posted March 20, 2017 Share Posted March 20, 2017 If you have the NVMe drive attached through a PCI card, would that make much of a difference? or do the PCI adapter cards basically pass through the NVMe card? Link to comment
JorgeB Posted March 20, 2017 Share Posted March 20, 2017 2 minutes ago, planetwilson said: or do the PCI adapter cards basically pass through the NVMe card? PCI adapter is transparent to unRAID. Link to comment
billington.mark Posted March 20, 2017 Share Posted March 20, 2017 11 minutes ago, johnnie.black said: PCI adapter is transparent to unRAID. +1. I use my NVMe device in a PCIe adapter Link to comment
SpaceInvaderOne Posted March 20, 2017 Author Share Posted March 20, 2017 1 hour ago, billington.mark said: You can boot to windows with a passed through NVMe without using clover at all. I've been doing this for a while since the first 6.3 RC. Bear in mind this assumes you're doing a fresh install. The only reason you cant do this natively is that Unraid ships with an older OVMF bootloader for VMs which is before it started supporting NVMe devices. We will just be downloading a referencing a different bootloader so we add support for direct NVMe booting. So basically, build the VM (dont start it yet) and pass through the NVMe device: IOMMU group 41 [144d:a802] 81:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM951/PM951 (rev 01) would be: <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x81' slot='0x00' function='0x0'/> </source> </hostdev> Download OVMF firmware from here https://www.kraxel.org/repos/jenkins/edk2/. You want the edk2.git-ovmf-x64-x-xxxxxxxx.xxxxx.xxxxxxxx.noarch.rpm file open with 7zip and drill down until you get to the edk2.git\ovmf-x64\ folder and pull out the OVMF-pure-efi.fd and OVMF_VARS-pure-efi.fd files and put them on an accessible share somewhere. Preferably on a cache only share (be sure to do both). in the xml below, Ive put them in /mnt/user/VMData/Windows10VM. reference the new files in XML (replace the default values): <os> <type arch='x86_64' machine='pc-q35-2.7'>hvm</type> <loader readonly='yes' type='pflash'>/mnt/user/VMData/Windows10VM/OVMF-pure-efi.fd</loader> <nvram>/mnt/user/VMData/Windows10VM/OVMF_VARS-pure-efi.fd</nvram> <boot dev='hd'/> </os> Boot the VM and go to the boot menu. Change the boot order to the DVD drive as top, NVMe as 2nd. Save and boot from the windows installation ISO. the NVMe drive should show as an available installation target in windows setup. Run through the installation as you normally would and you're done 4 @billington.mark Wow thats awesome. i didnt know that was in the ovmf bootloader now. i will try that asap. Guess i will be making an update to that video today I guess we have to put <boot order="1"> in the xml of the pci device ie <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x81' slot='0x00' function='0x0'/> </source> <boot order="1"> </hostdev> Link to comment
billington.mark Posted March 20, 2017 Share Posted March 20, 2017 7 minutes ago, gridrunner said: @billington.mark Wow thats awesome. i didnt know that was in the ovmf bootloader now. i will try that asap. Guess i will be making an update to that video today I guess we have to put <boot order="1"> in the xml of the pci device ie <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x81' slot='0x00' function='0x0'/> </source> <boot order="1"> </hostdev> Ive found that didnt make any difference after booting the VM (may have been fixed since i last tested though?). I've always had to manually edit the boot order in the actual VM by invoking the OVMF setup (del \ F2 i think?) after the first boot. Link to comment
psparks Posted March 20, 2017 Share Posted March 20, 2017 @billington.mark Thanks for the great tip on your passthrough technique. I got it working, but for some reason I need to edit the boot order every time the VM restarts. Is that expected or is there a way to ensure that boot order stays the same? Link to comment
SpaceInvaderOne Posted March 20, 2017 Author Share Posted March 20, 2017 Hi, I tried using the OVMF firmware. I find I don't get the windows symbol when windows loads it has the Tianocore logo now. Doesn't make any difference though but anyone else have this? Personally, I found it better to just replace just the OVMF_CODE-pure-efi.fd in unraid. Just copy it to /usr/share/qemu/ovmf-x64. I found it works fine with only this file but you can copy all three if you want (OVMF-pure-efi.fd , OVMF_VARS-pure-efi.fd and OVMF_CODE-pure-efi.fd) The advantage is you can just create a VM without having to do any custom edits. And each VM gets its own NVRAM file as normal. Just make a user script to run at the start of array and it will update the ovmf firmware each time the server boots. I have attached my script below. Along with the new OVMF firmware. Just put the files in a share and set its location in the script. #!/bin/bash #Set location of ovmf ovmfstore=/mnt/user/test/ovmf #copy to unraid rsync -a $ovmfstore/* /usr/share/qemu/ovmf-x64 exit ovmf.zip user script.zip Link to comment
billington.mark Posted March 21, 2017 Share Posted March 21, 2017 12 hours ago, gridrunner said: Hi, I tried using the OVMF firmware. I find I don't get the windows symbol when windows loads it has the Tianocore logo now. Doesn't make any difference though but anyone else have this? Personally, I found it better to just replace just the OVMF_CODE-pure-efi.fd in unraid. Just copy it to /usr/share/qemu/ovmf-x64. I found it works fine with only this file but you can copy all three if you want (OVMF-pure-efi.fd , OVMF_VARS-pure-efi.fd and OVMF_CODE-pure-efi.fd) The advantage is you can just create a VM without having to do any custom edits. And each VM gets its own NVRAM file as normal. Just make a user script to run at the start of array and it will update the ovmf firmware each time the server boots. I have attached my script below. Along with the new OVMF firmware. Just put the files in a share and set its location in the script. #!/bin/bash #Set location of ovmf ovmfstore=/mnt/user/test/ovmf #copy to unraid rsync -a $ovmfstore/* /usr/share/qemu/ovmf-x64 exit ovmf.zip user script.zip This is much neater I dont have the no windows logo issue on the version im running... If you want the version i'm using, drop me a PM. Link to comment
psparks Posted March 22, 2017 Share Posted March 22, 2017 @gridrunner I do have the logo 'issue' but everything else runs just fine, so seems like it's just an oddity. and to answer my own question from earlier about not needing to change boot order every time you start the vm...I figured out you just need to remove the other boot options in the xml. So remove the vdisk, driver iso file, any dvdrom ect that were booting before your nvme drive and it'll go straight to the only thing left. Link to comment
billington.mark Posted March 23, 2017 Share Posted March 23, 2017 @gridrunner This explains the 'no windows logo' 'issue'. https://github.com/tianocore/edk2/commit/6e5e544f227f031d0b45828b56cec5668dd1bf5b Also mentioned here: https://www.reddit.com/r/VFIO/comments/5yws5u/win10_logo_replaced_by_tianocore_logo_during_boot/ Link to comment
SpaceInvaderOne Posted March 24, 2017 Author Share Posted March 24, 2017 @billington.mark thanks for that link. Anyway maybe its a blessing not having to see that windows logo !! Link to comment
Enver Posted March 25, 2017 Share Posted March 25, 2017 On 20/03/2017 at 9:29 PM, billington.mark said: You can boot to windows with a passed through NVMe without using clover at all. I've been doing this for a while since the first 6.3 RC. Bear in mind this assumes you're doing a fresh install. The only reason you cant do this natively is that Unraid ships with an older OVMF bootloader for VMs which is before it started supporting NVMe devices. We will just be downloading a referencing a different bootloader so we add support for direct NVMe booting. So basically, build the VM (dont start it yet) and pass through the NVMe device: IOMMU group 41 [144d:a802] 81:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM951/PM951 (rev 01) would be: <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x81' slot='0x00' function='0x0'/> </source> </hostdev> Download OVMF firmware from here https://www.kraxel.org/repos/jenkins/edk2/. You want the edk2.git-ovmf-x64-x-xxxxxxxx.xxxxx.xxxxxxxx.noarch.rpm file open with 7zip and drill down until you get to the edk2.git\ovmf-x64\ folder and pull out the OVMF-pure-efi.fd and OVMF_VARS-pure-efi.fd files and put them on an accessible share somewhere. Preferably on a cache only share (be sure to do both). in the xml below, Ive put them in /mnt/user/VMData/Windows10VM. reference the new files in XML (replace the default values): <os> <type arch='x86_64' machine='pc-q35-2.7'>hvm</type> <loader readonly='yes' type='pflash'>/mnt/user/VMData/Windows10VM/OVMF-pure-efi.fd</loader> <nvram>/mnt/user/VMData/Windows10VM/OVMF_VARS-pure-efi.fd</nvram> <boot dev='hd'/> </os> Boot the VM and go to the boot menu. Change the boot order to the DVD drive as top, NVMe as 2nd. Save and boot from the windows installation ISO. the NVMe drive should show as an available installation target in windows setup. Run through the installation as you normally would and you're done @gridrunner Hello have attempted your method to pass-through the NVME drive but get the following error when I attempt to update the XML file. New VM never been booted. It seems that the VM template does not like the reference to "<boot dev='hd'/>" This is on UNRAID 6.3.2 Any ideas? Link to comment
SpaceInvaderOne Posted March 25, 2017 Author Share Posted March 25, 2017 2 hours ago, Enver said: @gridrunner Hello have attempted your method to pass-through the NVME drive but get the following error when I attempt to update the XML file. New VM never been booted. It seems that the VM template does not like the reference to "<boot dev='hd'/>" This is on UNRAID 6.3.2 Any ideas? 2 Yes you will have more than one boot in your xml thats why you are getting that error. You will have something like <os> <type arch='x86_64' machine='pc-i440fx-2.7'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader> <nvram>/etc/libvirt/qemu/nvram/0a0c0f94-12fc-d6f4-f5ad-3f141f92f861_VARS-pure-efi.fd</nvram> <boot dev='hd'/> </os> But you will also have a boot order in the xml somewhere else. <boot order='1'/> You can only use one or the other so you will need to remove one from the xml Link to comment
billington.mark Posted March 25, 2017 Share Posted March 25, 2017 7 hours ago, Enver said: @gridrunner Hello have attempted your method to pass-through the NVME drive but get the following error when I attempt to update the XML file. New VM never been booted. It seems that the VM template does not like the reference to "<boot dev='hd'/>" This is on UNRAID 6.3.2 Any ideas? just remove this part: <boot dev='hd'/> I tend to just set the boot device order in the actual VM in the OVMF bios (press F2 on boot i think?). There you can re-order boot devices as you would on a normal full fat PC BIOS. As far as im aware, you cant set the boot order in the XML when one of your boot devices is a passed through PCIe device, like you can when you use VIRTIO disks. EDIT: post your VMs XML Link to comment
cman9090 Posted April 22, 2017 Share Posted April 22, 2017 Any confirmed working cases for OSX? Link to comment
Tyranian Posted September 4, 2017 Share Posted September 4, 2017 Hi, Firstly thanks for the great guides they have been a real help getting to grips with unRaid. I followed your guide and got my Nvme passed through no issues with the clover boot images but every time I isolate CPU's and try to launch the VM it refuses to boot and hangs on the cursor in the top left corner I assume this is the clover boot loader, is there a way to move from the clover boot loader to the method of booting from a passed through nvme drive described above with the newer ovmf? Thanks Link to comment
Recommended Posts
Archived
This topic is now archived and is closed to further replies.