Unraid for the everyday with virtual machines

[Cross_5]

 

Hello, everybody,
I thought about running my normal PC with Unraid. Maybe one of you already had this idea. I mean to simply work with VMs and use all the advantages of Unraid.

My hardware:
CPU: AMD Ryzen 7 2700 Eight-Core 3.20 GHz
RAM: 32GB DDR4 3600-16 Trident Z Neo
Motherboard: X570 AORUS PRO
Hard disk: Samsung 970 EVO NVMe M.2 1TB
Graphics card1: NVIDIA NVS 300 1GB
Graphics card2: Radeon RX 570 4GB

Of course I own two graphics cards, because Unraid claims one only for itself. I want someone who doesn't know anything about the VM to not notice that my Windows is running on a VM. This means using Array Autostart, VM Autostart, On Board Sound Card and of course the automatic detection of USB devices.

Now I have already encountered two problems during the first test. With the sound card I was shown the following: See picture at the end.

The second problem is the graphics card:  After I installed Windows 10 and Windows tries to install the graphics card driver the picture freezes and the ram goes to maximum load + one core goes to 100%. This happens even when I manually try to install the graphics card drivers. I've read about this problem with some others as well. But if there is now a real solution I have not found out yet.

The VMs are not usable after they get into this freezing mode and you can delete them.

If someone has the same problems and has a solution I would be very happy about help.

Edited December 19, 2019 by Cross_5

[meep]

This is a very common use case here. I myself run two workstation grade ‘daily driver’ systems simultaneously with full device and gpu passthrough (both OS X and windows). I also run a few headless systems that I can vnc into.

 

you have a few issues there and for anyone to help, you’ll need to provide more details and info, such as your iommu grouping, debug fuels etc. however, a few pointers...

 

you don’t need to reserve one gpu for unraid. You can pass both Gpus to discrete VMs, you might need to pass through a rom file for the boot gpu, but it should work.

 

Ive had challenges using two identical gods (rx570s, as it happens). I could get the main one to pass through just fine, but the second always manifests the reset bug. I had to replace the second with an nvidia card to get stability.

 

you need to ensure that any device you pass through is in its own iommu group, or you’re passing all devices in a group. Look into ACS override if your mb is not separating devices to your liking.

 

your on board sound card will likely be grouped with a bunch of other devices and can be challenging to pass through. Consider outputting audio via the gpu hdmi connection.

 

thats all I can think of for now based on your post. Check out space invader one videos on multiple gpu passthrough, iommu groups etc. 

 

good luck!

[Cross_5]

13 hours ago, meep said:

This is a very common use case here. I myself run two workstation grade ‘daily driver’ systems simultaneously with full device and gpu passthrough (both OS X and windows). I also run a few headless systems that I can vnc into.

 

you have a few issues there and for anyone to help, you’ll need to provide more details and info, such as your iommu grouping, debug fuels etc. however, a few pointers...

 

you don’t need to reserve one gpu for unraid. You can pass both Gpus to discrete VMs, you might need to pass through a rom file for the boot gpu, but it should work.

 

Ive had challenges using two identical gods (rx570s, as it happens). I could get the main one to pass through just fine, but the second always manifests the reset bug. I had to replace the second with an nvidia card to get stability.

 

you need to ensure that any device you pass through is in its own iommu group, or you’re passing all devices in a group. Look into ACS override if your mb is not separating devices to your liking.

 

your on board sound card will likely be grouped with a bunch of other devices and can be challenging to pass through. Consider outputting audio via the gpu hdmi connection.

 

thats all I can think of for now based on your post. Check out space invader one videos on multiple gpu passthrough, iommu groups etc. 

 

good luck!

Thanks for the quick answer.

To the further information:
I would like to control 2-3 VMs with the graphics card, of course not simultaneously. The image output should be done normally on my 2 monitors which are directly connected to the graphics card. I also imagined to connect my speakers directly to the mainboard to get sound. Then I would like to run one or the other VM with VNC at the same time. Several Docker etc. should also run.

 

 

Further information:

Quote

PCI Devices and IOMMU Groups
IOMMU group 0:[1022:1452] 00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge[1022:1453] 00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge[1022:1453] 00:01.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge[144d:a808] 01:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983[1022:57ad] 02:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57ad[1022:57a3] 03:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57a3[1022:57a4] 03:08.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57a4[1022:57a4] 03:09.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57a4[1022:57a4] 03:0a.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57a4[8086:1539] 04:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)[1022:1485] 05:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP[1022:149c] 05:00.1 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller[1022:149c] 05:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller[1022:7901] 06:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)[1022:7901] 07:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)IOMMU group 1:[1022:1452] 00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host BridgeIOMMU group 2:[1022:1452] 00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge[1022:1453] 00:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge[1002:67df] 08:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] (rev ef)[1002:aaf0] 08:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere HDMI Audio [Radeon RX 470/480 / 570/580/590]IOMMU group 3:[1022:1452] 00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host BridgeIOMMU group 4:[1022:1452] 00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge[1022:1454] 00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B[1022:145a] 09:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function[1022:1456] 09:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor[1022:145f] 09:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Zeppelin USB 3.0 Host controllerIOMMU group 5:[1022:1452] 00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge[1022:1454] 00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B[1022:1455] 0a:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Renoir PCIe Dummy Function[1022:7901] 0a:00.2 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)[1022:1457] 0a:00.3 Audio device: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) HD Audio ControllerIOMMU group 6:[1022:790b] 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 59)[1022:790e] 00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)IOMMU group 7:[1022:1460] 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 0[1022:1461] 00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 1[1022:1462] 00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 2[1022:1463] 00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 3[1022:1464] 00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 4[1022:1465] 00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 5[1022:1466] 00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 6[1022:1467] 00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 7

Quote

CPU Thread Pairings
Pair 1:cpu 0 / cpu 8Pair 2:cpu 1 / cpu 9Pair 3:cpu 2 / cpu 10Pair 4:cpu 3 / cpu 11Pair 5:cpu 4 / cpu 12Pair 6:cpu 5 / cpu 13Pair 7:cpu 6 / cpu 14Pair 8:cpu 7 / cpu 15

Quote

USB Devices
Bus 001 Device 001:ID 1d6b:0002 Linux Foundation 2.0 root hubBus 001 Device 002:ID 048d:8297 Integrated Technology Express, Inc. ITE Device(8595)Bus 001 Device 003:ID 05e3:0608 Genesys Logic, Inc. HubBus 002 Device 001:ID 1d6b:0003 Linux Foundation 3.0 root hubBus 003 Device 001:ID 1d6b:0002 Linux Foundation 2.0 root hubBus 003 Device 002:ID 05e3:0608 Genesys Logic, Inc. HubBus 003 Device 003:ID 046d:c07c Logitech, Inc. M-R0017 [G700s Rechargeable Gaming Mouse]Bus 003 Device 004:ID 04b3:3025 IBM Corp. NetVista Full Width KeyboardBus 004 Device 001:ID 1d6b:0003 Linux Foundation 3.0 root hubBus 004 Device 002:ID 0781:5583 SanDisk Corp. Ultra FitBus 005 Device 001:ID 1d6b:0002 Linux Foundation 2.0 root hubBus 006 Device 001:ID 1d6b:0003 Linux Foundation 3.0 root

Quote

SCSI Devices
[0:0:0:0]disk SanDisk Ultra Fit 1.00 /dev/sda 61.5GB[N:0:4:1]disk Samsung SSD 970 EVO 1TB__1 /dev/nvme0n1 1.00TB

 

The NVida graphics card was not installed at this time, therefore it is not listed.

 

Does the rom file for the boot GPU mean the GPU firmware? I got it from AMD while searching for a boot GPU rom. Now I have downloaded two roms from AMD.

 

h338651_C8800100_FuryX_107_signed.rom
h338654_C8820200_Nano_107_signed.rom

Edited December 20, 2019 by Cross_5

[meep]

Here's one of your problems, the GPU is not isolated in its own IOMMU group;

 

 

And here, your on-board audio is also not isolated, and will not pass through;

 

 

If you can solve those issues, you'll be well on your way. 

 

look into IOMMU groups for your Montherboard and what best BIOS settings would be. look into ACS override for Unraid.

 

 

[Cross_5]

2 hours ago, meep said:

Here's one of your problems, the GPU is not isolated in its own IOMMU group;

 

 

And here, your on-board audio is also not isolated, and will not pass through;

 

 

If you can solve those issues, you'll be well on your way. 

 

look into IOMMU groups for your Montherboard and what best BIOS settings would be. look into ACS override for Unraid.

 

 

You mean that if the IOMMU groups are all single, everything works as desired? I ask because I currently only have the PC where unraid should run. The old one doesn't work anymore since the new one arrived. o.0

 

My new BIOS settings:

PCI Devices and IOMMU Groups

IOMMU group 0:[1022:1452] 00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge

IOMMU group 1:[1022:1453] 00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge

IOMMU group 2:[1022:1453] 00:01.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge

IOMMU group 3:[1022:1452] 00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge

IOMMU group 4:[1022:1452] 00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge

IOMMU group 5:[1022:1453] 00:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge

IOMMU group 6:[1022:1452] 00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge

IOMMU group 7:[1022:1452] 00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge

IOMMU group 8:[1022:1454] 00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B

IOMMU group 9:[1022:1452] 00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge

IOMMU group 10:[1022:1454] 00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B

IOMMU group 11:[1022:790b] 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 59)

[1022:790e] 00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)

IOMMU group 12:[1022:1460] 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 0

[1022:1461] 00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 1

[1022:1462] 00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 2

[1022:1463] 00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 3

[1022:1464] 00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 4

[1022:1465] 00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 5

[1022:1466] 00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 6

[1022:1467] 00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 7

IOMMU group 13:[144d:a808] 01:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983

IOMMU group 14:[1022:57ad] 02:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57ad

IOMMU group 15:[1022:57a3] 03:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57a3

IOMMU group 16:[1022:57a4] 03:08.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57a4

[1022:1485] 05:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP

[1022:149c] 05:00.1 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller

[1022:149c] 05:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller

IOMMU group 17:[1022:57a4] 03:09.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57a4

[1022:7901] 06:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)

IOMMU group 18:[1022:57a4] 03:0a.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57a4

[1022:7901] 07:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)

IOMMU group 19:[8086:1539] 04:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)

IOMMU group 20:[1002:67df] 08:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] (rev ef)

[1002:aaf0] 08:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere HDMI Audio [Radeon RX 470/480 / 570/580/590]

IOMMU group 21:[1022:145a] 09:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function

IOMMU group 22:[1022:1456] 09:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor

IOMMU group 23:[1022:145f] 09:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Zeppelin USB 3.0 Host controller

IOMMU group 24:[1022:1455] 0a:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Renoir PCIe Dummy Function

IOMMU group 25:[1022:7901] 0a:00.2 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)

IOMMU group 26:[1022:1457] 0a:00.3 Audio device: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) HD Audio Controller

 

USB Devices

 

Bus 001 Device 001:ID 1d6b:0002 Linux Foundation 2.0 root hub

Bus 001 Device 002:ID 048d:8297 Integrated Technology Express, Inc. ITE Device(8595)

Bus 001 Device 003:ID 05e3:0608 Genesys Logic, Inc. Hub

Bus 002 Device 001:ID 1d6b:0003 Linux Foundation 3.0 root hub

Bus 003 Device 001:ID 1d6b:0002 Linux Foundation 2.0 root hub

Bus 003 Device 002:ID 05e3:0608 Genesys Logic, Inc. Hub

Bus 003 Device 003:ID 046d:c07c Logitech, Inc. M-R0017 [G700s Rechargeable Gaming Mouse]

Bus 003 Device 004:ID 04b3:3025 IBM Corp. NetVista Full Width Keyboard

Bus 004 Device 001:ID 1d6b:0003 Linux Foundation 3.0 root hub

Bus 005 Device 001:ID 1d6b:0002 Linux Foundation 2.0 root hub

Bus 005 Device 002:ID 046d:0a4d Logitech, Inc. G430 Surround Sound Gaming Headset

Bus 006 Device 001:ID 1d6b:0003 Linux Foundation 3.0 root hub

Bus 006 Device 002:ID 0781:5583 SanDisk Corp. Ultra Fit

Bus 006 Device 003:ID 174c:1153 ASMedia Technology Inc. ASM2115 SATA 6Gb/s bridge

[meep]

That looks much better. Your GPU is in its own group, so should pass through for you now.

 

The onboard audio looks good, but you'll need to block it from loading on system boot.

 

Check out this thread for details;

 

[Cross_5]

 

I have now tried to run unraid with only 1 graphics card. For this I have edited in the syslinx.cfg

append initrd=/bzroot
to
append initrd=/bzroot video=efifb:off

This should make it possible to use a VM assigned to the graphics card with a screen in a normal way. I made a VM in autostart and you can see that when Unraid is loaded. Something happens but then I have only a black desktop.

I also added acpi_enforce_resources=lax to the config to see how fast my fans run. For this I also edited the /boot/config/go file. Extended with this line.
modprobe it87 force_id=0x8628

To be able to use my USB slots later normally, I created a new file /boot/config/vfio-pci.cfg
There then the Ids from the IOMMU group from my USB controller integrated.
BIND=05:00.0 05:00.1 05:00.3

So now I have all changes here. Maybe someone knows if I made a mistake or if something is not working together.

 

VM Settings:

 

VM Logs:

 

System  Logs:

tower-syslog-20191221-2052.zip

Edited December 22, 2019 by Cross_5