**VIDEO GUIDE** How to passthrough and boot from an NVMe controller for bare metal performance in unRAID


Recommended Posts

PLEASE NOT THIS GUIDE IS OUTDATED NOW. NEW VERSIONS OF QEMU/UNRAID ALLOW BOOTING A VM DIRECTLY FROM THE NVME CONTROLLER SO NO NEED TO USE

CLOVER AS SUGGESTED IN THIS GUIDE. YOU CAN JUST PASSTHROUGH THE CONTROLLER AND THE VM WILL BOOT FROM IT.

 

Hi Guys, this Video is a tutorial on how to pass through an NVMe controller then boot a VM from it in unRAID.
This method uses a modified clover boot loader with the NvmExpressDxe-64.EFI added to it to be able to use the NVMe drive then boot.
You will also see how to copy your existing windows VM from a vdisk directly onto the NVMe drive.
This method of pass through will also allow unRAID to have access to the device when the VM isn't using it as it isn't stubbed from the host.

Hope you find it useful :D

 

How to passthrough and boot from an NVMe controller for bare metal performance in unRAID

 

Edited by gridrunner
  • Upvote 1
Link to comment

Hey Gridrunner,

 

First off thanks for the videos, they are super helpful and very relevant to things I'm interested in doing. 

 

I've tried your tutorial out, as it's something I've been trying to solve constantly for the last few weeks. I am getting stuck hover on the clover bootloader piece.

 

I tried using your clover img file to boot windows directly from my nvme disk, but it seems like there is some problem with that file for me, as it only loads to a grey screen that says clover boot manager, and no further. I started to dive down the road of building my own boot loader with the clover configurator, but quickly turned back as that program is daunting and somewhat out of my league at the moment. 

 

any suggestions on ways to get windows to load fully, or why i might be running into this issue in the first place?

Edited by psparks
great > grey, load > manager
Link to comment
1 hour ago, psparks said:

Hey Gridrunner,

 

First off thanks for the videos, they are super helpful and very relevant to things I'm interested in doing. 

 

I've tried your tutorial out, as it's something I've been trying to solve constantly for the last few weeks. I am getting stuck hover on the clover bootloader piece.

 

I tried using your clover img file to boot windows directly from my nvme disk, but it seems like there is some problem with that file for me, as it only loads to a grey screen that says clover boot manager, and no further. I started to dive down the road of building my own boot loader with the clover configurator, but quickly turned back as that program is daunting and somewhat out of my league at the moment. 

 

any suggestions on ways to get windows to load fully, or why i might be running into this issue in the first place?

 

@psparksUm, thats strange its not working for you.

You shouldn't even see the clover bootloader screen (the grey screen)

The config.plist in my file should boot the bootmgfw.efi from the windows EFI partition and boot windows.

Do you see any options on the clover screen to boot from? Any hard drives listed. If so you should be able to select it and press enter to boot.

Also, how have you installed the windows os to the nvme drive. Did you install it directly to the nvme or did you copy across the install from your windows vdisk like i did in the video?

It will not boot from a windows vdisk just stored on the nvme drive. It must be a proper partitioned disk.

Also one other thing to check. Your VM must be OVMF it cant be seabios. You have to be using the EFI boot windows 10.

Hope this may help.

Link to comment

@gridrunner Yea it is super weird. I followed the tutorial to the T, converting a working vdisk using the easeus todo software, then formatting the leftover partition with unassigned devices.

 

On the grey screen there are no machines to boot from, just the clover icon.  the VM is also ovmf. 

 

I'm going to start fresh from the begining and try it again. I think my issue may have been the formatting of the disk going into the whole process. I'm going to try to do again it by formatting it with unassigned devices and not giving it any fs/partitions (it may have had a single partition going into the process). currently it's showing the file systems types in unassigned devices for those windows partitions that the software made, and I saw that it wasn't showing those in your video.

Link to comment

You can boot to windows with a passed through NVMe without using clover at all. I've been doing this for a while since the first 6.3 RC. Bear in mind this assumes you're doing a fresh install.

 

The only reason you cant do this natively is that Unraid ships with an older OVMF bootloader for VMs which is before it started supporting NVMe devices. 

We will just be downloading a referencing a different bootloader so we add support for direct NVMe booting.

 

So basically, build the VM (dont start it yet) and pass through the NVMe device:

IOMMU group 41
	[144d:a802] 81:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM951/PM951 (rev 01)

would be:

 

<hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x81' slot='0x00' function='0x0'/>
      </source>
    </hostdev>

 

  • Download OVMF firmware from here https://www.kraxel.org/repos/jenkins/edk2/. You want the edk2.git-ovmf-x64-x-xxxxxxxx.xxxxx.xxxxxxxx.noarch.rpm file
  • open with 7zip and drill down until you get to the edk2.git\ovmf-x64\ folder and pull out the OVMF-pure-efi.fd and OVMF_VARS-pure-efi.fd files and put them on an accessible share somewhere. Preferably on a cache only share (be sure to do both). in the xml below, Ive put them in /mnt/user/VMData/Windows10VM.
  • reference the new files in XML (replace the default values):

 

<os>
    <type arch='x86_64' machine='pc-q35-2.7'>hvm</type>
    <loader readonly='yes' type='pflash'>/mnt/user/VMData/Windows10VM/OVMF-pure-efi.fd</loader>
    <nvram>/mnt/user/VMData/Windows10VM/OVMF_VARS-pure-efi.fd</nvram>
    <boot dev='hd'/>
  </os>

 

Boot the VM and go to the boot menu. Change the boot order to the DVD drive as top, NVMe as 2nd. Save and boot from the windows installation ISO. the NVMe drive should show as an available installation target in windows setup. Run through the installation as you normally would and you're done

 

Edited by billington.mark
  • Like 1
  • Upvote 3
Link to comment
1 hour ago, billington.mark said:

You can boot to windows with a passed through NVMe without using clover at all. I've been doing this for a while since the first 6.3 RC. Bear in mind this assumes you're doing a fresh install.

 

The only reason you cant do this natively is that Unraid ships with an older OVMF bootloader for VMs which is before it started supporting NVMe devices. 

We will just be downloading a referencing a different bootloader so we add support for direct NVMe booting.

 

So basically, build the VM (dont start it yet) and pass through the NVMe device:


IOMMU group 41
	[144d:a802] 81:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM951/PM951 (rev 01)

would be:

 

<hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x81' slot='0x00' function='0x0'/>
      </source>
    </hostdev>

 

  • Download OVMF firmware from here https://www.kraxel.org/repos/jenkins/edk2/. You want the edk2.git-ovmf-x64-x-xxxxxxxx.xxxxx.xxxxxxxx.noarch.rpm file
  • open with 7zip and drill down until you get to the edk2.git\ovmf-x64\ folder and pull out the OVMF-pure-efi.fd and OVMF_VARS-pure-efi.fd files and put them on an accessible share somewhere. Preferably on a cache only share (be sure to do both). in the xml below, Ive put them in /mnt/user/VMData/Windows10VM.
  • reference the new files in XML (replace the default values):

 


<os>
    <type arch='x86_64' machine='pc-q35-2.7'>hvm</type>
    <loader readonly='yes' type='pflash'>/mnt/user/VMData/Windows10VM/OVMF-pure-efi.fd</loader>
    <nvram>/mnt/user/VMData/Windows10VM/OVMF_VARS-pure-efi.fd</nvram>
    <boot dev='hd'/>
  </os>

 

Boot the VM and go to the boot menu. Change the boot order to the DVD drive as top, NVMe as 2nd. Save and boot from the windows installation ISO. the NVMe drive should show as an available installation target in windows setup. Run through the installation as you normally would and you're done

 

 
 
4

@billington.mark  Wow thats awesome. i didnt know that was in the ovmf bootloader now. i will try that asap. Guess i will be making an update to that video today :D

I guess we have to put

<boot order="1">  in the xml of the pci device

ie

<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x81' slot='0x00' function='0x0'/>
</source>
<boot order="1">
</hostdev>

 

 

Edited by gridrunner
Link to comment
7 minutes ago, gridrunner said:

@billington.mark  Wow thats awesome. i didnt know that was in the ovmf bootloader now. i will try that asap. Guess i will be making an update to that video today :D

I guess we have to put

<boot order="1">  in the xml of the pci device

ie


<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x81' slot='0x00' function='0x0'/>
</source>
<boot order="1">
</hostdev>

 

 

 

Ive found that didnt make any difference after booting the VM (may have been fixed since i last tested though?). I've always had to manually edit the boot order in the actual VM by invoking the OVMF setup (del \ F2 i think?) after the first boot.

 

Link to comment

Hi, I tried using the OVMF firmware. I find I don't get the windows symbol when windows loads it has the Tianocore logo now.

Doesn't make any difference though but anyone else have this?

 

Personally, I found it better to just replace just the OVMF_CODE-pure-efi.fd  in unraid.

Just copy it to /usr/share/qemu/ovmf-x64.  I found it works fine with only this file but you can copy all three if you want (OVMF-pure-efi.fd , OVMF_VARS-pure-efi.fd and OVMF_CODE-pure-efi.fd) The advantage is you can just create a VM without having to do any custom edits. And each VM gets its own NVRAM file as normal.

Just make a user script to run at the start of array and it will update the ovmf firmware each time the server boots.

I have attached my script below.  Along with the new OVMF firmware. Just put the files in a share and set its location in the script.

#!/bin/bash
#Set location of ovmf
ovmfstore=/mnt/user/test/ovmf
#copy to unraid
rsync -a $ovmfstore/* /usr/share/qemu/ovmf-x64

exit

 

 

ovmf.zip

user script.zip

Edited by gridrunner
Link to comment
12 hours ago, gridrunner said:

Hi, I tried using the OVMF firmware. I find I don't get the windows symbol when windows loads it has the Tianocore logo now.

Doesn't make any difference though but anyone else have this?

 

Personally, I found it better to just replace just the OVMF_CODE-pure-efi.fd  in unraid.

Just copy it to /usr/share/qemu/ovmf-x64.  I found it works fine with only this file but you can copy all three if you want (OVMF-pure-efi.fd , OVMF_VARS-pure-efi.fd and OVMF_CODE-pure-efi.fd) The advantage is you can just create a VM without having to do any custom edits. And each VM gets its own NVRAM file as normal.

Just make a user script to run at the start of array and it will update the ovmf firmware each time the server boots.

I have attached my script below.  Along with the new OVMF firmware. Just put the files in a share and set its location in the script.


#!/bin/bash
#Set location of ovmf
ovmfstore=/mnt/user/test/ovmf
#copy to unraid
rsync -a $ovmfstore/* /usr/share/qemu/ovmf-x64

exit

 

 

ovmf.zip

user script.zip

 

This is much neater :) 

I dont have the no windows logo issue on the version im running... If you want the version i'm using, drop me a PM. 

 

Edited by billington.mark
Link to comment

@gridrunner

I do have the logo 'issue' but everything else runs just fine, so seems like it's just an oddity.

 

and to answer my own question from earlier about not needing to change boot order every time you start the vm...I figured out you just need to remove the other boot options in the xml. So remove the vdisk, driver iso file, any dvdrom ect that were booting before your nvme drive and it'll go straight to the only thing left. 

  • Upvote 1
Link to comment
On 20/03/2017 at 9:29 PM, billington.mark said:

You can boot to windows with a passed through NVMe without using clover at all. I've been doing this for a while since the first 6.3 RC. Bear in mind this assumes you're doing a fresh install.

 

The only reason you cant do this natively is that Unraid ships with an older OVMF bootloader for VMs which is before it started supporting NVMe devices. 

We will just be downloading a referencing a different bootloader so we add support for direct NVMe booting.

 

So basically, build the VM (dont start it yet) and pass through the NVMe device:


IOMMU group 41
	[144d:a802] 81:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM951/PM951 (rev 01)

would be:

 

<hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x81' slot='0x00' function='0x0'/>
      </source>
    </hostdev>

 

  • Download OVMF firmware from here https://www.kraxel.org/repos/jenkins/edk2/. You want the edk2.git-ovmf-x64-x-xxxxxxxx.xxxxx.xxxxxxxx.noarch.rpm file
  • open with 7zip and drill down until you get to the edk2.git\ovmf-x64\ folder and pull out the OVMF-pure-efi.fd and OVMF_VARS-pure-efi.fd files and put them on an accessible share somewhere. Preferably on a cache only share (be sure to do both). in the xml below, Ive put them in /mnt/user/VMData/Windows10VM.
  • reference the new files in XML (replace the default values):

 


<os>
    <type arch='x86_64' machine='pc-q35-2.7'>hvm</type>
    <loader readonly='yes' type='pflash'>/mnt/user/VMData/Windows10VM/OVMF-pure-efi.fd</loader>
    <nvram>/mnt/user/VMData/Windows10VM/OVMF_VARS-pure-efi.fd</nvram>
    <boot dev='hd'/>
  </os>

 

Boot the VM and go to the boot menu. Change the boot order to the DVD drive as top, NVMe as 2nd. Save and boot from the windows installation ISO. the NVMe drive should show as an available installation target in windows setup. Run through the installation as you normally would and you're done

 

@gridrunner

 

Hello have attempted your method to pass-through the NVME drive but get the following error when I attempt to update the XML file.

 

New VM never been booted.

 

It seems that the VM template does not like the reference to "<boot dev='hd'/>"

 

This is on UNRAID 6.3.2

 

Any ideas?

 

Screen Shot 2017-03-25 at 16.45.55.png

Link to comment
2 hours ago, Enver said:

@gridrunner

 

Hello have attempted your method to pass-through the NVME drive but get the following error when I attempt to update the XML file.

 

New VM never been booted.

 

It seems that the VM template does not like the reference to "<boot dev='hd'/>"

 

This is on UNRAID 6.3.2

 

Any ideas?

 

Screen Shot 2017-03-25 at 16.45.55.png

2

 Yes you will have more than one boot in your xml thats why you are getting that error.

 You will have something like 

<os>
    <type arch='x86_64' machine='pc-i440fx-2.7'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/0a0c0f94-12fc-d6f4-f5ad-3f141f92f861_VARS-pure-efi.fd</nvram>
    <boot dev='hd'/>
  </os>

But you will also have a boot order in the xml somewhere else.

<boot order='1'/>

You can only use one or the other so you will need to remove one from the xml

Link to comment
7 hours ago, Enver said:

@gridrunner

 

Hello have attempted your method to pass-through the NVME drive but get the following error when I attempt to update the XML file.

 

New VM never been booted.

 

It seems that the VM template does not like the reference to "<boot dev='hd'/>"

 

This is on UNRAID 6.3.2

 

Any ideas?

 

 

 

just remove this part:

 

<boot dev='hd'/>

I tend to just set the boot device order in the actual VM in the OVMF bios (press F2 on boot i think?). There you can re-order boot devices as you would on a normal full fat PC BIOS.

As far as im aware, you cant set the boot order in the XML when one of your boot devices is a passed through PCIe device, like you can when you use VIRTIO disks. 

 

EDIT: post your VMs XML 

Edited by billington.mark
Link to comment
  • 4 weeks later...
  • 4 months later...

Hi, Firstly thanks for the great guides they have been a real help getting to grips with unRaid.

 

I followed your guide and got my Nvme passed through no issues with the clover boot images but every time I isolate CPU's and try to launch the VM it refuses to boot and hangs on the cursor in the top left corner I assume this is the clover boot loader, is there a way to move from the clover boot loader to the method of booting from a passed through nvme drive described above with the newer ovmf?

 

Thanks

  • Upvote 1
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.