Win 10 NVMe passthrough won't boot


Recommended Posts

Hi!

 

I just setup my new rig (specs below).

I installed a bare metal Win10 install on my NVMe m.2, installed some apps and ran some benchmarks for a comparison later.

 

I followed SI1 walkthrough here: 

https://www.youtube.com/watch?v=fnIn6GnA87c

 

I setup my bios to get IOMMU groups and it looks they are all nicely seperated from eath other.

Used VFIO-PCI Config from community applications to be able to pass through my NVMe in the VM.

I tried different combinations with gpu passthrough and just VNC, combination of CPU cores and RAM allocations, Hyper-V on/off.

 

The Problem is:

When I boot the VM is instantly comes up with "Start PXE over IP4" then "Start HTTP boot over IP4" and then into shell.

Intercepting it with F2 and clicking on the NVMe in bios manually wont do anything. (monitor goes black for a split second but nothing happens)

I tried to add boot order in the form view but it did not change anything.

 

My very uneducated guess is that the NVMe can’t be bootet. I checked and it is UEFI.

Bare Metal boot works fine.

 

Would love some help as I tried to rule out quite a few things on the weekend but don’t get anywhere atm.

<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm'>
  <name>Windows 10</name>
  <uuid>7fee0944-a204-83b3-7e75-f5c904b6aa45</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/>
  </metadata>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>7</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='2'/>
    <vcpupin vcpu='1' cpuset='14'/>
    <vcpupin vcpu='2' cpuset='4'/>
    <vcpupin vcpu='3' cpuset='16'/>
    <vcpupin vcpu='4' cpuset='6'/>
    <vcpupin vcpu='5' cpuset='18'/>
    <vcpupin vcpu='6' cpuset='20'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-i440fx-4.2'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/7fee0944-a204-83b3-7e75-f5c904b6aa45_VARS-pure-efi.fd</nvram>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='none'/>
    </hyperv>
  </features>
  <cpu mode='host-passthrough' check='none'>
    <topology sockets='1' cores='7' threads='1'/>
    <cache mode='passthrough'/>
    <feature policy='require' name='topoext'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/virtio-win-0.1.173-2.iso'/>
      <target dev='hdb' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:e1:31:7c'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' websocket='-1' listen='0.0.0.0' keymap='en-us'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
</domain>
 
Quote

IOMMU group 0:[1022:1482] 00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 1:[1022:1483] 00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge

IOMMU group 2:[1022:1483] 00:01.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge

IOMMU group 3:[1022:1482] 00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 4:[1022:1482] 00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 5:[1022:1483] 00:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge

IOMMU group 6:[1022:1483] 00:03.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge

IOMMU group 7:[1022:1482] 00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 8:[1022:1482] 00:05.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 9:[1022:1482] 00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 10:[1022:1484] 00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]

IOMMU group 11:[1022:1482] 00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 12:[1022:1484] 00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]

IOMMU group 13:[1022:790b] 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 61)

[1022:790e] 00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)

IOMMU group 14:[1022:1440] 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 0

[1022:1441] 00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 1

[1022:1442] 00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 2

[1022:1443] 00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 3

[1022:1444] 00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 4

[1022:1445] 00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 5

[1022:1446] 00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 6

[1022:1447] 00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 7

IOMMU group 15:[144d:a808] 01:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983

IOMMU group 16:[1022:57ad] 02:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse Switch Upstream

IOMMU group 17:[1022:57a3] 03:03.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge

IOMMU group 18:[1022:57a3] 03:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge

IOMMU group 19:[1022:57a4] 03:08.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge

[1022:1485] 06:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP

[1022:149c] 06:00.1 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller

[1022:149c] 06:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller

IOMMU group 20:[1022:57a4] 03:09.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge

[1022:7901] 07:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)

IOMMU group 21:[1022:57a4] 03:0a.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge

[1022:7901] 08:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)

IOMMU group 22:[8086:2723] 04:00.0 Network controller: Intel Corporation Wi-Fi 6 AX200 (rev 1a)

IOMMU group 23:[8086:1539] 05:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)

IOMMU group 24:[10de:1e84] 09:00.0 VGA compatible controller: NVIDIA Corporation TU104 [GeForce RTX 2070 SUPER] (rev a1)

[10de:10f8] 09:00.1 Audio device: NVIDIA Corporation TU104 HD Audio Controller (rev a1)

[10de:1ad8] 09:00.2 USB controller: NVIDIA Corporation TU104 USB 3.1 Host Controller (rev a1)

[10de:1ad9] 09:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU104 USB Type-C UCSI Controller (rev a1)

IOMMU group 25:[10de:1b82] 0a:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070 Ti] (rev a1)

[10de:10f0] 0a:00.1 Audio device: NVIDIA Corporation GP104 High Definition Audio Controller (rev a1)

IOMMU group 26:[1022:148a] 0b:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function

IOMMU group 27:[1022:1485] 0c:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP

IOMMU group 28:[1022:1486] 0c:00.1 Encryption controller: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP

IOMMU group 29:[1022:149c] 0c:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller

IOMMU group 30:[1022:1487] 0c:00.4 Audio device: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller

 

CPU Thread Pairings

Pair 1:cpu 0 / cpu 12

Pair 2:cpu 1 / cpu 13

Pair 3:cpu 2 / cpu 14

Pair 4:cpu 3 / cpu 15

Pair 5:cpu 4 / cpu 16

Pair 6:cpu 5 / cpu 17

Pair 7:cpu 6 / cpu 18

Pair 8:cpu 7 / cpu 19

Pair 9:cpu 8 / cpu 20

Pair 10:cpu 9 / cpu 21

Pair 11:cpu 10 / cpu 22

Pair 12:cpu 11 / cpu 23

 

USB Devices

Bus 001 Device 001:ID 1d6b:0002 Linux Foundation 2.0 root hub

Bus 001 Device 002:ID 048d:8297 Integrated Technology Express, Inc. ITE Device(8595)

Bus 001 Device 003:ID 05e3:0608 Genesys Logic, Inc. Hub

Bus 002 Device 001:ID 1d6b:0003 Linux Foundation 3.0 root hub

Bus 003 Device 001:ID 1d6b:0002 Linux Foundation 2.0 root hub

Bus 003 Device 002:ID 8087:0029 Intel Corp.

Bus 003 Device 003:ID 05e3:0608 Genesys Logic, Inc. Hub

Bus 003 Device 004:ID 03f0:094a Hewlett-Packard Optical Mouse [672662-001]

Bus 003 Device 005:ID 05ac:1006 Apple, Inc. Hub in Aluminum Keyboard

Bus 003 Device 006:ID 05ac:0220 Apple, Inc. Aluminum Keyboard (ANSI)

Bus 004 Device 001:ID 1d6b:0003 Linux Foundation 3.0 root hub

Bus 005 Device 001:ID 1d6b:0002 Linux Foundation 2.0 root hub

Bus 006 Device 001:ID 1d6b:0003 Linux Foundation 3.0 root hub

Bus 007 Device 001:ID 1d6b:0002 Linux Foundation 2.0 root hub

Bus 008 Device 001:ID 1d6b:0003 Linux Foundation 3.0 root hub

Bus 008 Device 002:ID 0781:5583 SanDisk Corp. Ultra Fit

 

SCSI Devices

[0:0:0:0]disk SanDisk Ultra Fit 1.00 /dev/sda 30.7GB

[3:0:0:0]disk ATA M4-CT128M4SSD2 0309 /dev/sdb 128GB

[5:0:0:0]disk ATA ST4000NE001-2MA1 EN01 /dev/sdc 4.00TB

[6:0:0:0]disk ATA ST4000NE001-2MA1 EN01 /dev/sdd 4.00TB

 

Specs of PC:

Ryzen 3900x

64GB Ram

Samsung NVME Evo Plus 1TB

Nvidia 1070ti

Nvidia 2070 super

 

Screen Shot 2020-07-05 at 8.54.10 pm.png

Screen Shot 2020-07-05 at 8.55.06 pm.png

Screen Shot 2020-07-05 at 8.55.25 pm.png

Edited by Grsh
added xml as code
Link to comment
<hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
  		<boot order='1'/>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </hostdev>

Try this. Hopefully the boot order line will help. 

Link to comment
7 hours ago, david279 said:

<hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
  		<boot order='1'/>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </hostdev>

Try this. Hopefully the boot order line will help. 

Thanks for the reply David.

I tried your version of the boot order but I am running in a VM creation error.

 

"internal error: unknown pci source type 'boot'"

 

I tried this version before (the boot flag is in a different line) but it wouldn't solve the problem either.

VM can be created. It will start directly in the shell and wont try booting in IP4 but still cant get the NVMe to boot.

<hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </hostdev>

 

Link to comment

have you try to boot with only passing you NVME drive?

 

i have similar setup with you,

 

when i setup i add one device then boot, shutdown, add another one then boot, 

repeat until all device added, that way i know which device is causing the problem

Edited by audias
Link to comment
6 minutes ago, audias said:

have you try to boot with only passing you NVME drive?

 

i have similar setup with you,

 

when i setup i add one device then boot, shutdown, add another one then boot, 

repeat until all device added, that way i know which device is causing the problem

Thanks for the reply!

 

I tried only the NVMe but same outcome.

End up in shell. NVMe is visible in bios but if I select it it still wont boot.

 

It looks like the NVMe is recognised but is not able to be booted. Is there any special trick I have to do?

Link to comment

Small update:

To my uneducated eye it looks like the NVMe cant be booted but it stills shows up.

I have just reinstalled Windows 10 on there as I had trouble even booting normally though bios (non VM).

 

Unfortunatly I still had no luck.

Link to comment

i did it the different way as you do, i am having different kind of trouble, but something i can live with. 

I have,

3950x

x570 taichi

nvme

2080 super.

 

I cannot passthough the whole nvme controller (the easy way), have to follow the old Spaceinvader guide.

setting the disk to manual and setting the location /dev/disk/by-id/YOURDISK 

 

i try adding the nvme as a pcie device and set the disk to none, the vm and unraid just stop/freeze and restart.

 

i have no trouble switching back and forth from baremetal to vm, but i get inaccessible boot device when switching from vm back to baremetal, it doesnt really bother me that much. 

 

it's normal again, when i boot into safe mode without doing anything and restart again. 

 

 

 

Edited by Unraid Newbie
Link to comment

OH YEAH. 

 

I did it. Successfully passed the nvme, not just virt disk nvme.

 

i studded the nvme, but then i saw the log with many FLR waiting message, google search reveal no power to nvme, so i kept the edited vm template, unstud the nvme, and restart~~~~~~~~ it's magical!! and reboot back to barebone doesnt have any issue, no more inaccessible boot device, straight to windows. 

 

 

Link to comment
6 hours ago, Unraid Newbie said:

OH YEAH. 

 

I did it. Successfully passed the nvme, not just virt disk nvme.

 

i studded the nvme, but then i saw the log with many FLR waiting message, google search reveal no power to nvme, so i kept the edited vm template, unstud the nvme, and restart~~~~~~~~ it's magical!! and reboot back to barebone doesnt have any issue, no more inaccessible boot device, straight to windows. 

 

 

Thats awesome.

I just had the same issue after playing around with unraid that my barebone would not boot anymore. So have to reinstall everything and create an image backup before I get back into it.

 

So you just unstubbed it and restarted? After that it worked without stubbing again? Anything you changed on the nvme barebone?

How did you get around the "no boot device found" if using barebone nvme?

Link to comment
2 hours ago, killasniff said:

You've installed windows as a MBR volume, so convert it to GPT. Lookup mbr2gpt conversion.

OMG!

Read you comment and did it straight away. mbr2gpt would not work for me so I reinstalled Win10 and just tested it and it worked!

Thanks so much for your help! I would have never thought og this. Coming from OSX/hackintosh I never really had to deal with this.

 

Thanks a bunch for everyones input.

Link to comment
Just now, Grsh said:

Thats awesome.

I just had the same issue after playing around with unraid that my barebone would not boot anymore. So have to reinstall everything and create an image backup before I get back into it.

 

So you just unstubbed it and restarted? After that it worked without stubbing again? Anything you changed on the nvme barebone?

How did you get around the "no boot device found" if using barebone nvme?

yah, without touching the vm template again, the pci-e device box stay checked even if the nvme is unstudded. that's how i get power to my nvme (not in some kind of power saving state). 

 

if you get no boot device found in barebone, i would try completely power off the pc, instead of restarting, that would be my first guess. 

but if you get "inaccessible boot device" (blue screen, windows BSOD) just restart again, go into safe mode and restart, windows should be able to fix it for you.

 

happy to post my vm xml(s) for you if you are interested.

right now, i have 3users 1pc build. the only thing left out is the onboard audio.

 

 

 

Link to comment
18 hours ago, Grsh said:

OMG!

Read you comment and did it straight away. mbr2gpt would not work for me so I reinstalled Win10 and just tested it and it worked!

Thanks so much for your help! I would have never thought og this. Coming from OSX/hackintosh I never really had to deal with this.

 

Thanks a bunch for everyones input.

I had the very same problem couldn't figure out why and then I saw a post that talked about UEFI and then it just clicked because I remembered that I installed windows wrong(not in UEFI ver.) but forgot about it. Happy it worked for you.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.