2 vms passed through x570


9 posts in this topic Last Reply

Recommended Posts

Yes I have this working, trying to get a 3rd VM working but running into issues. 2 VMs with passthrough actually perform very well simultaneously in my opinion (4 isolated cores, 4 HTs (makes 8 total) and 16gb of reserved memory per VM). Have it running for a couple of months now daily and did not encounter much problems. Guess I had 2 or 3 incidental freezes and can not really remember it crashing completely. Problems for me start at the point I am trying to use a 3rd gaming vm with passthrough. I am updating the bios to version P3.61 on short notice. I don't use any other things on my rig currently.

 

Motherboard: AsRock x570 Taichi, bios version P3.40

CPU: AMD Ryzen 9 3950x

Memory: 128gb DDR4 (4x 32gb Samsung M378A4G43MB1-CTD)

GPU: 3x GTX 1080 Ti gaming X 11gb

 

Edited by Timothyy
Link to post
9 hours ago, Timothyy said:

Yes I have this working, trying to get a 3rd VM working but running into issues. 2 VMs with passthrough actually perform very well simultaneously in my opinion (4 isolated cores, 4 HTs (makes 8 total) and 16gb of reserved memory per VM). Have it running for a couple of months now daily and did not encounter much problems. Guess I had 2 or 3 incidental freezes and can not really remember it crashing completely. Problems for me start at the point I am trying to use a 3rd gaming vm with passthrough. I am updating the bios to version P3.61 on short notice. I don't use any other things on my rig currently.

 

Motherboard: AsRock x570 Taichi, bios version P3.40

CPU: AMD Ryzen 9 3950x

Memory: 128gb DDR4 (4x 32gb Samsung M378A4G43MB1-CTD)

GPU: 3x GTX 1080 Ti gaming X 11gb

 

Are the graphic cards identical?  In the old days, the graphic cards had to be different.

Edited by Jessie
Link to post
  • 1 month later...

I'll answer my own question in case someone else encounters the same problem.

 

Recently, I built a machine on a Gigabyte X570 gaming x platform and a Ryzen 7 3770x.  On this board, you can change a setting in BIOS and the IOMMU groups pretty well work in unraid for usb/gpu passthrough without the need to set ACS override in unraid.  It worked flawlessly.

 

Then I tried a Gigabyte B550 Gaming X motherboard with the same processor and tried to pass through a Gigabyte GT130 GPU (No fans) and a gpu bios dump from tech powerup.

Black screen.

I've built a few machines using this GPU and BIOS dump without problems on series 2 ryzen and intel boards.

So I blamed the B550 board.

Then I built a machine on an x570 board.

Black screen again.

So I dumped the BIOS from the actual card as per Spaceinvaders tutorial and it worked.

I then added another GT130 card and was able to create a second Windows vm complete with keyboards/ mice and usb passthrough.

 

I am assuming the B550 failed for the same reason.

I will do the experiment in the near future.

Not really sure why it didn't work. on the series 3 hardware but did on the second gen servers.

Maybe the firmware changed in the GT130 card.

Edited by Jessie
Link to post
  • 4 weeks later...
On 12/15/2020 at 10:12 AM, Jessie said:

I will do the experiment in the near future.

I'm looking at the B550 too, I usally use a VM as my daily driver, but heard a lot of people criticizing the B550 for not beening able to passthrough.

Would love to hear what you recomend :) The B550 or x570 c:

Link to post
1 hour ago, Michelle Bausager said:

I'm looking at the B550 too, I usally use a VM as my daily driver, but heard a lot of people criticizing the B550 for not beening able to passthrough.

Would love to hear what you recomend :) The B550 or x570 😄

I've built quite a few Unraid systems over the past few years.  I recently went to Ryzen.  The first Ryzen3 was on a X570 platform (Gigabyte X570 Gaming X) Brilliant.  You can change a setting in Cmos and it optimises the IOMMU allocations for Unraid passthrough.

 

The next one I built was on a B550 platform. (Gigabyte B550 Gaming X)

I went that way because the requirements of this machine didn't warrant an X570 and the B550's have a lower cost.

I had trouble passing through a single Nvidia GT1030.  I had previously followed Spaceinvaders methods for modifying a downloaded BIOS image, but this didn't work.

In the end I resorted to the old method which was to install a cheap graphics card to boot unraid and then I passed the vm through to the second GPU.

The problem with that was that I lost a full sized PCIe slot that could have been used for something else.

 

Then I built another machine on a X570 and had the same problem.  This machine had the same hardware specs as the first one, which worked straight off.

I then decided to extract the BIOS from the same card and it worked.

I suspect there might have been a firmware change in the graphic card which made the bios image incompatible.

 

So with this new found knowledge, I returned to the B550 machine, extracted the BIOS and it now works perfectly on a single card.

BIOS extraction was done using the GPU as a secondary card and command lines.

Spaceinvader has now released a script which will do all the hard work for you. You don't even require the second GPU for the extraction.  He is a very clever man.  (If you don't know who he is, go to youtube and discover his brilliant tutorials on unraid, pfsense and others)

 

So far I have been using the Ryzen 7 3700x and previously 2700 processors.  I chose the 3700x over the 3800 because it draws less power and isn't much slower than the 3800.

 

In the next couple of weeks I will build one on a B550 and a Ryzen r5 3600?? The r5 has 12 threads vs 16. 

12 will be enough for the anticipated needs of its user.
 

For the near future, I will build on either of these platforms depending on their anticipated workload.

 

To date, I have been holding off going to 6.9 until it went mainstream.  I changed my mind last week when chasing a problem on my current machine.  6.9 rc2 seems very stable and the features are too good to go back again.

 

B550 Gaming X has 2 x m.2 slots and 4 Sata ports, so using 4tb drives you can have 12tb with parity plus 2 m.2 cache drives.  

X570 Gaming X has 2 x .m2 slots and 6 Sata ports, so using 4tb drives you  can have up to 20tb with parity plus 2 x m.2 cache drives or with v6.9 12tb and 4 cache drives.

I used to use coolermaster cases (Mainly silencio 550's) I feel the new generation doesn't suit unraid.

So I have opted for Phantecs P400s cases.  There is a gaming version of the P400 but I mainly build these machines for business so "light shows" are not really appropriate.  I like the phantec case design.  They provide for 2 sata ssd drives, 2 x 3.5" bays and then you can install up to 4 extra 3.5 bays to provision the max number of drives for the unraid setup. They have plenty of room behind the motherboard to hide the cables.

 

A typical spec might use one of the above motherboards with a Ryzen 7 3700x, 32 or 16gb ddr4 3200 ram, gigabyte or asus GT-1030 graphics cards (no fans).

possibly an addon pcie usb card for extra passed through usb slots, a pcie lan adapter if the machine is going to run a pfsense firewall vm.

 

Then it will typically run at least 1 operating system. (usually win 10) In the background, a nextcloud / Collabora instance, maybe a pfsense vm, maybe a open vpn docker so they can securely access their system remotely and then the sky seems to be the limit for other docker applications.

 

 

 

Link to post
  • 2 months later...
On 11/11/2020 at 6:21 PM, Timothyy said:

Yes I have this working, trying to get a 3rd VM working but running into issues. 2 VMs with passthrough actually perform very well simultaneously in my opinion (4 isolated cores, 4 HTs (makes 8 total) and 16gb of reserved memory per VM). Have it running for a couple of months now daily and did not encounter much problems. Guess I had 2 or 3 incidental freezes and can not really remember it crashing completely. Problems for me start at the point I am trying to use a 3rd gaming vm with passthrough. I am updating the bios to version P3.61 on short notice. I don't use any other things on my rig currently.

 

Motherboard: AsRock x570 Taichi, bios version P3.40

CPU: AMD Ryzen 9 3950x

Memory: 128gb DDR4 (4x 32gb Samsung M378A4G43MB1-CTD)

GPU: 3x GTX 1080 Ti gaming X 11gb

 

Planning on doing the same. Could you give us ur complete parts list?

I intend on using:

Processor: AMD Ryzen 9 3900X with a be quiet! Dark Rock Pro 4 

MB: Asus TUF GAMING X570-PLUS (WI-FI) 

RAM: Corsair Vengeance LPX 32 GB (4 x 8 GB) DDR4-3200 

GPU: 2x RTX 3070 8 GB Founders Edition

PSU: Corsair RMx (2018) 850 W 80+ Gold

Case: Lian Li LANCOOL II-X

 

Your thoughts/advice on the above parts will be much appreciated☺️

Edited by Kich902
Link to post
  • 2 months later...

I am looking to do the same thing.

 

Need 2 GPU and 1 HBA 9211-8i.  Do I need a PCIe x1 GPU(I see some GT710 x1 ones on Amazon) for unRaid to boot into so I still had the 3 full size slots?

 

I looked at the ASUS AMD AM4 Pro WS X570-Ace MB but don't know how good the IOMMU groups are like.  $470 CAD 3 PCIe 4. 0 x16 (3-Way x8/x8/x8), 4 SATA ports.

 

I looked at the AsRock X570 Creator but it is really expensive at $662 CAD.  AQUANTIA 10G LAN, Intel Gigabit LAN, 3 PCIe 4.0 x16(x8,x8,x4), 3 PCIe 2.0 x1, 8 SATA3, 1 Hyper M.2 (PCIe Gen4 x4 & SATA3), 1 Hyper M.2 (PCIe Gen4 x4)

 

Looking at AsRock x570 Taichi $440 CAD

ASRock X570 TAICHI RAZER Edition $570 CAD same as regular Taichi but has 2.5G LAN, wifi(not needed), BT(not needed)

 

Other bit unsure is ECC memory really needed?  I use ECC in my current system.  It just gets more confusing as to how they describe ECC compatibility.

I found this on an ASUS but probably only applies to ASUS:

image.png.dda44019c5c941dc85e5fc22c2e294d1.png

Edited by Paul_Ber
Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.