Cant passthrough nvme

[lusitopp]

Hi, my name is Stupid!

 

My VM was acting up so I decided to delete my VM and create a new. And I thought "I have reinstalled my VM so many times now so I remember how to passthough".... Yeah right!!!

 

 

Primary vDisk Location: Manual /dev/disk/by-id/ata-KINGSTON_SA2000M81000G_50026B72824BF6A7

 

This gives me "Not a valid block device location '/dev/disk/by-id/ata-KINGSTON_SA2000M81000G_50026B72824BF6A7'"

 

Any suggestions about this? And also, when creating a new VM some settings is predefined in the disk settings, 

 

Primary vDisk Size: 30G

Primary vDisk Type: raw

Primary vDisk Bus: VirtIO

 

Shall these be edited when doing the passthrough?

 

EDIT

It's solved, did run

ls /dev/disk/by-id/ and found that I should use nvme- and not ata-

 

"/dev/disk/by-id/nvme-KINGSTON_SA2000M81000G_50026B72824BF6A7"

 

But one question remain, which vdisk bus shall i use

 

 

Edited December 2, 2021 by lusitopp

[ghost82]

1 hour ago, lusitopp said:

Any suggestions about this? And also, when creating a new VM some settings is predefined in the disk settings, 

 

Primary vDisk Size: 30G

Primary vDisk Type: raw

Primary vDisk Bus: VirtIO

 

Shall these be edited when doing the passthrough?

 

You'd be better to modify the vm settings directly in the xml code instead of using the gui, to not mess things.

If you want to passthrough the whole disk, virtualizing the controller, you are doing it right and you should have something like this:

<disk type="block" device="disk">
  <driver name="qemu" type="raw"/>
  <source dev="/dev/disk/by-id/nvme-KINGSTON_SA2000M81000G_50026B72824BF6A7"/>
  <target dev="sda" bus="sata"/>
  <boot order="1"/>
  <address type="drive" controller="0" bus="0" target="0" unit="1"/>
</disk>

(the boot order line is not mandatory, the '1' value will have priority over other devices (other disks, cd rom, etc.))

 

This will attach your real nvme disk to the virtual sata controller, by default for a q35 machine is at 00:1f.2 (bus:slot.function)

 

You can also replace bus sata with virtio, but take into account that if you have your os already installed on the disk, and it hasn't the virtio driver installed for storage, it will fail to boot. On the other hand if you need to do a clean installation you can load from the virtio virtual cd rom that driver at the beginning of the installation. In the first case you need to inject the driver, search the forum, some solutions have been posted.

Bus sata will work "out of the box".

 

Note that a nvme disk has also a controller.

If you want to passthrough an nvme disk I would suggest to bind to vfio the nvme controller and passthrough it like you would do for a pci device, so something like:

      <hostdev mode='subsystem' type='pci' managed='yes'>
        <driver name='vfio'/>
        <source>
          <address domain='0x0000' bus='0x0a' slot='0x00' function='0x0'/>
        </source>
        <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </hostdev>

 

Where 0a:00.0 (bus:slot.function) is the source address of the nvme controller in the host (in other words where unraid sees the controller) and 01:00.0 (bus:slot.function) is the target address of the nvme controller (where the vm sees the controller).

In this case you don't need any <disk></disk> block in the xml

By passing through the controller you will eliminate the overhead of the virtual sata/virtio controller.

 

If you want more detailed info about your specific case, post your diagnostics file.

Edited December 2, 2021 by ghost82

[lusitopp]

5 hours ago, ghost82 said:

Note that a nvme disk has also a controller.

If you want to passthrough an nvme disk I would suggest to bind to vfio the nvme controller and passthrough it like you would do for a pci device, so something like:

      <hostdev mode='subsystem' type='pci' managed='yes'>
        <driver name='vfio'/>
        <source>
          <address domain='0x0000' bus='0x0a' slot='0x00' function='0x0'/>
        </source>
        <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </hostdev>

 

Where 0a:00.0 (bus:slot.function) is the source address of the nvme controller in the host (in other words where unraid sees the controller) and 01:00.0 (bus:slot.function) is the target address of the nvme controller (where the vm sees the controller).

In this case you don't need any <disk></disk> block in the xml

By passing through the controller you will eliminate the overhead of the virtual sata/virtio controller.

 

If you want more detailed info about your specific case, post your diagnostics file.

 

This was a good point, hadn't even thought about that, will do this when its time to reinstall next time 😃 this time i went for sata bus, last time i used virtio

[lusitopp]

On 12/2/2021 at 8:52 AM, ghost82 said:

 

You'd be better to modify the vm settings directly in the xml code instead of using the gui, to not mess things.

If you want to passthrough the whole disk, virtualizing the controller, you are doing it right and you should have something like this:

<disk type="block" device="disk">
  <driver name="qemu" type="raw"/>
  <source dev="/dev/disk/by-id/nvme-KINGSTON_SA2000M81000G_50026B72824BF6A7"/>
  <target dev="sda" bus="sata"/>
  <boot order="1"/>
  <address type="drive" controller="0" bus="0" target="0" unit="1"/>
</disk>

(the boot order line is not mandatory, the '1' value will have priority over other devices (other disks, cd rom, etc.))

 

This will attach your real nvme disk to the virtual sata controller, by default for a q35 machine is at 00:1f.2 (bus:slot.function)

 

You can also replace bus sata with virtio, but take into account that if you have your os already installed on the disk, and it hasn't the virtio driver installed for storage, it will fail to boot. On the other hand if you need to do a clean installation you can load from the virtio virtual cd rom that driver at the beginning of the installation. In the first case you need to inject the driver, search the forum, some solutions have been posted.

Bus sata will work "out of the box".

 

Note that a nvme disk has also a controller.

If you want to passthrough an nvme disk I would suggest to bind to vfio the nvme controller and passthrough it like you would do for a pci device, so something like:

      <hostdev mode='subsystem' type='pci' managed='yes'>
        <driver name='vfio'/>
        <source>
          <address domain='0x0000' bus='0x0a' slot='0x00' function='0x0'/>
        </source>
        <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </hostdev>

 

Where 0a:00.0 (bus:slot.function) is the source address of the nvme controller in the host (in other words where unraid sees the controller) and 01:00.0 (bus:slot.function) is the target address of the nvme controller (where the vm sees the controller).

In this case you don't need any <disk></disk> block in the xml

By passing through the controller you will eliminate the overhead of the virtual sata/virtio controller.

 

If you want more detailed info about your specific case, post your diagnostics file.

 

Hi there!

 

I tried to passthrough my nvme, and that worked, the windows installation found the drive and i could install windows but after the first reboot it seems that device is not bootable, i end up with "press ESC to skip startup.nsh" and then its just stops and im in the shell.

Any suggestions about this?

 

[ghost82]

Try this:

https://pve.proxmox.com/wiki/OVMF/UEFI_Boot_Entries

[lusitopp]

1 hour ago, ghost82 said:

Try this:

https://pve.proxmox.com/wiki/OVMF/UEFI_Boot_Entries

 

It does work but it is not persistent, after each reboot the VM my creatad boot record is gone.

I go to

Boot Maintenance Manager

Boot Options

Add Boot Option

Select my nvme drive where windows 11 is installed

EFI\Boot\biitx64.efi

Add Description

Commit Changes and Exit

 

Change boot order

Commit Changes and Exit

 

Continue.

 

This will boot my windows 11 VM. but just until next restart, then i have to do this all over again.

 

using 6.10-rc2, OVMF TPM bios, Q35-6.1

 

 

Edited December 6, 2021 by lusitopp

[ghost82]

57 minutes ago, lusitopp said:

 

It does work but it is not persistent, after each reboot the VM my creatad boot record is gone.

I go to

Boot Maintenance Manager

Boot Options

Add Boot Option

Select my nvme drive where windows 11 is installed

EFI\Boot\biitx64.efi

Add Description

Commit Changes and Exit

 

Change boot order

Commit Changes and Exit

 

Continue.

 

This will boot my windows 11 VM. but just until next restart, then i have to do this all over again.

 

using 6.10-rc2, OVMF TPM bios, Q35-6.1

 

 

 

Try to add the boot order line to your xml in the section of the nvme controller passthrough, something like this:

      <hostdev mode='subsystem' type='pci' managed='yes'>
        <driver name='vfio'/>
        <source>
          <address domain='0x0000' bus='0x0a' slot='0x00' function='0x0'/>
        </source>
        <boot order="1"/>
        <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </hostdev>

 

With the boot order line you may need to delete this line in your xml:

<boot dev='hd'/>

 

Edited December 6, 2021 by ghost82

[lusitopp]

31 minutes ago, ghost82 said:

 

Try to add the boot order line to your xml in the section of the nvme controller passthrough, something like this:

      <hostdev mode='subsystem' type='pci' managed='yes'>
        <driver name='vfio'/>
        <source>
          <address domain='0x0000' bus='0x0a' slot='0x00' function='0x0'/>
        </source>
        <boot order="1"/>
        <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </hostdev>

 

 

Great success, thank you!