Guide ASRock X570M Pro4 VM boot from baremetal NVME


Rhodius

Recommended Posts

Hi all,

 

I found Unraid last year and this past April set about to build my first PC/server for the purpose of data storage, Plex and VMs. I had a QNAP NAS that I’ll sync with the Unraid and when done probably use as a backup, Unraid being the main server.

 

I bought the following hardware in the beginning of June after waiting for a while on a particular motherboard and finally finished putting it together 2 weeks ago. I had spent on and off almost 1-2 weeks figuring out the VM part. Even though the main purpose was for data storage and Plex, I figured I don’t want issue down the road if I’m using it for VM that I couldn’t get to work in the beginning. As that may involve tearing apart and putting together new hardware, why spent all that time if the HW will be replaced anyway?

 

So the main purpose of this post is to save others from having to spend as much time as I did, hopefully this helps.

 

CPU: AMD Ryzen 3700X

RAM: Corsair LPX 32GB (2x16GB) 3200MHz C16 DDR4 DRAM

Mobo: ASRock X570M Pro4

GPU: NVIDIA Quadro P2200

Storage: Used/transplanted Toshiba NVME 512Gb from laptop, 2x 12TB Element

PSU: Corsair RM750x

 

Goal: Bare metal NVME Win 10 should be bootable and used as VM in the Unraid

The thinking is I only want to maintain one Windows system, I can use for benchmarking and other Windows related purpose such as gaming and others. Achieving that goal is how I spent on and off at least one week figuring out how to get it to work. After watching countless hours of SpaceInvaderOne and reading postings on the forum and countless hours of trying out and testing, I managed to achieve this goal.

 

For VM, the following BIOS settings must be done for ASRock X570M PRO4, BIOS is updated to 2.30 for AGESA compatibility:

Advanced -> CPU Configuration -> SVM Mode -> Enable

Advanced -> AMD CBS -> NBIO Common Options -> IOMMU -> Enable

Advanced -> AMD CBS -> NBIO Common Options -> ACS Enable -> Enable

Advanced -> AMD CBS -> NBIO Common Options -> Enable AER Cap -> Enable

 

Almost all X570 has similar backplane:

 

MB_backplane.png.0eda7193826dc334d32ba3255d04f0a4.png

 

I installed Windows 10 on the NVME, update all drivers, NVIDIA, Intel networking, AMD chipset. Then I took a copy of it using Reflect, this proves to be a wise move since the failed VM booting caused issues to the baremetal Windows installation. This step saves me countless hours since restoring my Windows take only 10 minutes. I have a copy of Win 10 complete with all bench-marking tools and device drivers before I took a snapshot.

 

I used Unraid 6.8.0-RC5 with X570 audio fix and didn’t add anything to Syslinux settings. This Unraid with audio fix released by Skittals is important to being able to have audio in the VM for gaming purpose.

 

I put Unraid where no. 3 is and my USB keyboard and mouse on no. 2. Originally I put Unraid on no.1 and USB can’t be passed onto VM without issue, I suspect no. 1 and 2 share the same USB bus. NVME is unassigned device and together with no. 2 are what I passed onto the VM. VM ACS setting is set to disabled and the following is my VFIO-PCI config settings (thanks Skittal).

 

2046702145_Screenshotfrom2020-06-2917-37-36.thumb.png.0a21a5bcd98d1b12310e3ccbcc5e3102.png

 

 

I had struggled to get the VM to boot to bare metal Windows installed on NVME, I tried re-installing Windows trying to get a legacy boot instead of UEFI thinking that was an issue, I even ordered a Samsung EVO 860 thinking it was the vendor ID issue etc. The issue turned out to be that I can’t use i440, I can only use Q35.

 

The following is the VM setup, I copied the UUID from the baremetal and inserted it here.

Compare to other examples I’ve followed, the biggest impact setting for my setup is Q35 below.

 

 

263412182_Screenshotfrom2020-06-2909-40-52.png.5fec7cb6114ad7df6a4b3a262f264121.png

 

 

For this next page the biggest issue is passing the USB, once I figured out which USB outlet works with what, then I read Skittal’s posting on his X570, that helps in finalizing the VFIO-PCI config which leads to the setting here. I dumped my own VBIOS using GPU-Z following SpaceInvaderOne’s instruction to remove the header from the VBIOS.

 

1940952405_Screenshotfrom2020-06-2917-40-12.png.1455ef5df6ad3a33c31558c34fec838c.png

 

Now I can use my VM the way I intended to use it for while booting from the same instance bare metal.

VM_lower.png

image.png

Screenshot from 2020-06-29 17-37-36.png

Edited by Rhodius
  • Like 1
  • Thanks 1
Link to comment

Your configuration of components is the same that I had in mind to assemble my new unraid build
Box I have thought fractal node 804

First of all Thank you, this type of threads detailing experiences with information and images help a lot of new users like me.
Sorry for my translation.

I wanted to ask you some basic novice doubts regarding the ASRock X570M Pro4 board,

 

- You received the board with 2.30 or earlier version, the update process can be done with usb from the bios itself or would it have to be done by asrock software in windows?

 

- It could start UNRAID without installing the "Quadro P ..." temporarily (it will take me a few more weeks to get it)

 

- Custom vbios can tell more about this.

 

- How about the temperatures with this asrock / ryzen? chipset fan very noisy?

Edited by kabuki14
Link to comment
On 7/5/2020 at 1:39 PM, kabuki14 said:

Your configuration of components is the same that I had in mind to assemble my new unraid build
Box I have thought fractal node 804



First of all Thank you, this type of threads detailing experiences with information and images help a lot of new users like me.
Sorry for my translation.

I wanted to ask you some basic novice doubts regarding the ASRock X570M Pro4 board,

 

- You received the board with 2.30 or earlier version, the update process can be done with usb from the bios itself or would it have to be done by asrock software in windows?

++ Yes I'm using the 804 as well. This case has really good air flow and with the dampener on the hard disks, it is not making any noise at all. I put it by my feet under my desk, the laptop cooler fan on my desk is noisier than the machine.

I used arctic fans that come in a box of 5, 2 140mm on top, 4 120 mm in the front and 2 in the back. Running Prime95 for 30 minutes, CPU temp barely reaches 58C and tops out at 72C with ambient temp of 24C. Unigine Heaven also runs at around 72C.

 

I installed Windows first and then Unraid then VM in the Unraid using the same Windows install. Windows is for the purpose of benchmarking and what if worst case scenario. Also in the case of BIOS upgrade, I did this from the Windows. I only upgraded the BIOS to 2.30 not the newest, although I just checked today there is a v2.80 out now that addressed the AGESA issue which impact IOMMU.

 

Quote

- It could start UNRAID without installing the "Quadro P ..." temporarily (it will take me a few more weeks to get it)

 

- Custom vbios can tell more about this.

 

- How about the temperatures with this asrock / ryzen? chipset fan very noisy?

 

You need a graphics card as Ryzen doesn't have graphics capability, this becomes a problem in my install since I did Windows first, you can't tell what happens then.

There's a youtube video from Space Invader One on extracting the graphics card BIOS. You need to use GPU-Z to save to a file then edit the file using hex editor to remove the header, exactly where is what the youtube video explains.

Regarding noise, I suppose with all the extra fans I throw at it, it's not making any noise at all, see my description above. I also used a Corsair RM series power supply that only runs the fan if it gets to certain point, otherwise the fan is not running. Since the power supply fan is immediately next to a case opening, I think it would make more noticeable noise compare to chipset fan, if that's what you worried about.

Edited by Rhodius
Link to comment
On 7/5/2020 at 2:48 PM, Unraid Newbie said:

can you share your xml?

 

<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm'>
  <name>Windows 10 NVME boot</name>
  <uuid>abc12345-yz09-0000-0000-000000000000</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/>
  </metadata>
  <memory unit='KiB'>16777216</memory>
  <currentMemory unit='KiB'>16777216</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>8</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='4'/>
    <vcpupin vcpu='1' cpuset='12'/>
    <vcpupin vcpu='2' cpuset='5'/>
    <vcpupin vcpu='3' cpuset='13'/>
    <vcpupin vcpu='4' cpuset='6'/>
    <vcpupin vcpu='5' cpuset='14'/>
    <vcpupin vcpu='6' cpuset='7'/>
    <vcpupin vcpu='7' cpuset='15'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-q35-4.0.1'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/fec28570-eb75-0000-0000-000000000000_VARS-pure-efi.fd</nvram>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='none'/>
    </hyperv>
  </features>
  <cpu mode='host-passthrough' check='none'>
    <topology sockets='1' cores='8' threads='1'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/virtio-win-0.1.185.iso'/>
      <target dev='hdb' bus='sata'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x8'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x9'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0xa'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0xb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0xc'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x4'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0xd'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x5'/>
    </controller>
    <controller type='pci' index='7' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='7' port='0xe'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x6'/>
    </controller>
    <controller type='pci' index='8' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='8' port='0xf'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x7'/>
    </controller>
    <controller type='pci' index='9' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='9' port='0x10'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='10' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='10' port='0x11'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:3f:d1:18'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/disk1/isos/Nvidia1660SuperOC.rom'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0a' slot='0x00' function='0x4'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x05' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x05' slot='0x00' function='0x3'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x08' slot='0x00' function='0x2'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x09' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x08' slot='0x00' function='0x3'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x0a' slot='0x00' function='0x0'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
</domain>

Link to comment
  • 4 weeks later...

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.