Help with first attempt to pass-thru a graphics card to a VM

[Mattaton]

I haven't gotten into the nitty-gritty of this yet, but I wanted some input about sequencing.

 

My motherboard/CPU does not have on-board graphics, so I have to have a graphics card for basic output. I'm assuming I'll need to have one, cheapo graphics card just for that basic output and then add my better graphics card to be passed-thru.

 

At the moment, I only have the graphics card to be passed thru in the system.

 

So, how would you guys go about switching things around? Can the better card stay in the top x16 slot on my mobo and then add the lesser card in the next slot? How do I get BIOS and unRAID to use the second-slot card by default?

 

Thanks!

[ghost82]

30 minutes ago, Mattaton said:

My motherboard/CPU does not have on-board graphics, so I have to have a graphics card for basic output. I'm assuming I'll need to have one, cheapo graphics card just for that basic output and then add my better graphics card to be passed-thru.

 

Although gpu passthrough is possible with a single gpu, it is recommended to have a cheaper primary gpu for unraid video output (igpu or pcie), because:

1. no video output for unraid when the vm with the gpu passthrough runs

2. when the vm stops not all gpus are able to bind again to the host, and in any case you need some scripting to reattach the gpu to the host from vfio

3. when the vm runs you can only control the host with some other access, ssh for example, from an external device or from the vm, with a bridge connection

4. passthorugh of a single primary gpu may not be possible at all, depending on motherboards

 

Personally In my 2 servers I'm using:

1. a quadro 600 gpu for host video (top slot, pcie gen 3 x16) and a gtx titan black for passthrough (3rd slot, pcie gen 3 x16)

2. a very very very old geforce 8400 gs for host video (top slot, pcie gen 3 x16) and a 6900 xt for passthrough (3rd slot, pcie gen 3 x16)

 

For a single gpu passthrough, best practice is:

1. disable efifb or vesafb, or better look at /proc/iomem to see what's using the gpu if you have errors in syslog like 'BAR X: Can't reserve' and/or black screen

2. attach to vfio at boot the gpu (no video output for unraid --> it will look like the screen freezes, but it gets simply detached from the host and attached to vfio)

3. pass a rom to the gpu, especially if you have the gpu in the primary slot

 

Note: this is best practice, the passthrough can work even without that.

 

45 minutes ago, Mattaton said:

Can the better card stay in the top x16 slot on my mobo and then add the lesser card in the next slot?

 

Usually, if you don't have any bios option to choose the primary gpu, the system will usually give priority from top to bottom (at least that's what I'm seeing in my systems): so if you have 2 gpus, one in slot 1 and another let's say in slot 3, the primary gpu will be that in the 1st (top) slot.

So, usually, it's better to put the primary gpu (for the host) in the top slot without any bios option.

 

 

[Mattaton]

3 hours ago, ghost82 said:

 

Although gpu passthrough is possible with a single gpu, it is recommended to have a cheaper primary gpu for unraid video output (igpu or pcie), because:

1. no video output for unraid when the vm with the gpu passthrough runs

2. when the vm stops not all gpus are able to bind again to the host, and in any case you need some scripting to reattach the gpu to the host from vfio

3. when the vm runs you can only control the host with some other access, ssh for example, from an external device or from the vm, with a bridge connection

4. passthorugh of a single primary gpu may not be possible at all, depending on motherboards

 

Personally In my 2 servers I'm using:

1. a quadro 600 gpu for host video (top slot, pcie gen 3 x16) and a gtx titan black for passthrough (3rd slot, pcie gen 3 x16)

2. a very very very old geforce 8400 gs for host video (top slot, pcie gen 3 x16) and a 6900 xt for passthrough (3rd slot, pcie gen 3 x16)

 

For a single gpu passthrough, best practice is:

1. disable efifb or vesafb, or better look at /proc/iomem to see what's using the gpu if you have errors in syslog like 'BAR X: Can't reserve' and/or black screen

2. attach to vfio at boot the gpu (no video output for unraid --> it will look like the screen freezes, but it gets simply detached from the host and attached to vfio)

3. pass a rom to the gpu, especially if you have the gpu in the primary slot

 

Note: this is best practice, the passthrough can work even without that.

 

 

Usually, if you don't have any bios option to choose the primary gpu, the system will usually give priority from top to bottom (at least that's what I'm seeing in my systems): so if you have 2 gpus, one in slot 1 and another let's say in slot 3, the primary gpu will be that in the 1st (top) slot.

So, usually, it's better to put the primary gpu (for the host) in the top slot without any bios option.

 

 

Gotcha. Once my data-rebuild is done on this new server build (old hardware), I'll check the BIOS to see if I can select the default graphics. Otherwise, I'll just drop the better card to slot 2.

 

You have me beat by a little bit, I'm putting an 8600 GTS in for the host gpu. 🙂 

 

Here's my old-skool system, I'm interested to know your opinion on how screwed I am in getting a Big Sur VM to use the graphics card.

Asus P9X79 LE

i7 4820K

MSI Twin Frozr GeForce GTX 760

24GB of RAM (3x8GB until I replace a bad 8GB stick)

 

I'm an extreme novice (as you've seen on the MIAB thread), but from reading, it sounds like Big Sur can still work with a GTX 760. Is that correct?

 

I'm going to try to get my hands on an AMD RX580 from work. That looks like it'll work up through Monterey.

 

How monumental of a task am I about to tackle??? 😄 

 

Thanks, @ghost82. You've been MOST helpful!

[ghost82]

10 hours ago, Mattaton said:

I'm interested to know your opinion on how screwed I am in getting a Big Sur VM to use the graphics card

I usually like asus motherboards, I have 2, I would say you can reach your target, the msi should run well in big sur.

Well, in theory, it can run also in monterey, but not without breaking the system seal and injecting the old kepler drivers from big sur.

In monterey the drivers were only removed.

Breaking the os seal basically means no delta updates (only full), no secure boot, custom sip value.

 

10 hours ago, Mattaton said:

I'm going to try to get my hands on an AMD RX580 from work. That looks like it'll work up through Monterey

That should work in monterey. Note the brands to avoid (dortania): 

Quote

The only brands you should avoid with the Polaris series would be XFX, PowerColour, HIS and VisionTek

 

You will need the amd reset patch (plugin in unraid 6.10.x, or unraid kernel helper).

[Mattaton]

7 hours ago, ghost82 said:

I usually like asus motherboards, I have 2, I would say you can reach your target, the msi should run well in big sur.

Well, in theory, it can run also in monterey, but not without breaking the system seal and injecting the old kepler drivers from big sur.

In monterey the drivers were only removed.

Breaking the os seal basically means no delta updates (only full), no secure boot, custom sip value.

 

That should work in monterey. Note the brands to avoid (dortania): 

 

You will need the amd reset patch (plugin in unraid 6.10.x, or unraid kernel helper).

As this is all new to me, I will refrain from breaking the OS seal for now. 😄 

 

Ah, so I'll need to switch my unRAID branch to get the 6.10 version in order to get the reset patch in order to get the AMD card to work? Gotcha. Not a problem since this is my sandbox server. 🙂 

 

I'll swap out the graphics cards, add some 1TB cache drives, and upgrade to 6.10 tonight. Then we'll see how the graphics pass-thru goes.

 

Thanks again for your help and insight!

[ghost82]

Just now, Mattaton said:

so I'll need to switch my unRAID branch to get the 6.10 version in order to get the reset patch in order to get the AMD card to work?

Not needed, you can use 6.9.2 but with the kernel helper (search the forum): I never used it but I think it creates a new bzroot file to be replaced in the root of the unraid usb pendrive; basically it patches the kernel to include the amd reset patch, and the kernel is into the bzroot file, that's why you need to replace it.

For testing only you don't need any amd patch: you will only see that you can't boot the vm after a vm shutdown or reboot.

This is because the gpu is not able to fully reinitialize and you need to shutdown the full host for a proper initialization.

That patch adds instructions to reset gpu after the vm is shutdown or rebooted.

 

5 minutes ago, Mattaton said:

Then we'll see how the graphics pass-thru goes.

Remember to look carefully at the xml code, as you will need to set the gpu as multifunction, otherwise, probably you won't be able to boot, mac os is very sensitive to this.

Other optimizations are also needed (first I have in mind is changing the bus of the virtual network to 0 so that it results built-in for apple services to work), but to see if the passthrough works they are not a must.

 

Always start with vnc, if it boots proceed further.

Remember to have always custom qemu args at the bottom of the xml.

Remember to use always latest opencore and kexts: you can find the updated opencore img with kexts in the macinabox github repository, in the pull request tab: at the time of writing it's here:

https://github.com/SpaceinvaderOne/Macinabox/pulls

https://github.com/SpaceinvaderOne/Macinabox/pull/61

https://github.com/SpaceinvaderOne/Macinabox/raw/84a82e54ae5f596f492ead2f2d98767029e2826a/bootloader/OpenCore.img.zip

[ghost82]

18 hours ago, Mattaton said:

I'm going to try to get my hands on an AMD RX580 from work

Somebody will not be able to get a video signal on boot, or...he/she will not be able to play cyberpunk 2077 with a replaced gpu of the 90s ahahahha

 

[Mattaton]

5 minutes ago, ghost82 said:

Not needed, you can use 6.9.2 but with the kernel helper (search the forum): I never used it but I think it creates a new bzroot file to be replaced in the root of the unraid usb pendrive; basically it patches the kernel to include the amd reset patch, and the kernel is into the bzroot file, that's why you need to replace it.

For testing only you don't need any amd patch: you will only see that you can't boot the vm after a vm shutdown or reboot.

This is because the gpu is not able to fully reinitialize and you need to shutdown the full host for a proper initialization.

That patch adds instructions to reset gpu after the vm is shutdown or rebooted.

 

Remember to look carefully at the xml code, as you will need to set the gpu as multifunction, otherwise, probably you won't be able to boot, mac os is very sensitive to this.

Other optimizations are also needed (first I have in mind is changing the bus of the virtual network to 0 so that it results built-in for apple services to work), but to see if the passthrough works they are not a must.

 

Always start with vnc, if it boots proceed further.

Remember to have always custom qemu args at the bottom of the xml.

Remember to use always latest opencore and kexts: you can find the updated opencore img with kexts in the macinabox github repository, in the pull request tab: at the time of writing it's here:

https://github.com/SpaceinvaderOne/Macinabox/pulls

https://github.com/SpaceinvaderOne/Macinabox/pull/61

https://github.com/SpaceinvaderOne/Macinabox/raw/84a82e54ae5f596f492ead2f2d98767029e2826a/bootloader/OpenCore.img.zip

Excellent. Thanks so much for the explanations! Helps me, "get it." 🙂 

 

I've been using the latest OpenCore.img you sent me in the MIAB thread and have become accustom to getting the MacOS VMs to run without MIAB. I have no idea why using MIAB was proving so troublesome. I can do it manually without issue. The kexts are already in the OpenCore image? Is that right? I don't have to add kexts manually?

 

I have a working Monterey VM on my main server (VNC only), and a Big Sur VM on this sandbox server. That's currently VNC only until I give the pass-thru a shot. If I can get the RX580, I'll just nuke the Big Sur VM and start over with Monterey. I want to up the size of the vdisk anyway. It's currently only 130GB, but I'm replacing a tiny 180GB cache drive with two 1TB cache drives. So, I'll up it to 500GB or so. 🙂 

[Mattaton]

4 minutes ago, ghost82 said:

Somebody will not be able to get a video signal on boot, or...he/she will not be able to play cyberpunk 2077 with a replaced gpu of the 90s ahahahha

 

HAHAHA! Nobody games on a Mac VM, do they? 😄 That's just asking for trouble! 😄 

[ghost82]

Just now, Mattaton said:

I have no idea why using MIAB was proving so troublesome

That's because it's been a year macinabox came out and with apple all is in continuous development, for example:

1. most of the time on every upgrade the product id of the os will change --> macinabox has it hardcoded and this will result in no product found

2. every year, when new macs come out in stores with a new apple os, board-ids change and if 1 day before your mac was able to get big sur, the day after it will be able to get monterey, or it won't be supported anymore, so no more updates; that's why the unpached macinabox tries to install monterey, because the hardcoded board-id is able to get monterey

3. on every minor upgrade, one should follow the opencore development, because it can be that the actual bootloader is able to boot v. x.x.1, but not x.x.2

 

When macinabox came out, it was really very easy to setup all, all was working as expected, but maintaining the container requires a lot of efforts, to follow all these things and update the code.

 

So yes, I agree that at the time of writing, it's easier to setup a mac os vm manually!

 

7 minutes ago, Mattaton said:

The kexts are already in the OpenCore image? Is that right? I don't have to add kexts manually?

Correct, the img contains all the basic kexts to have a running vm, with some others for gpu passthrough.

 

Note that the image contains "standard" smbios data, so you may want to backup your current smbios data (if you changed them) and replace them in the new image once the vm is booted.

Have a look once a month at the pull requests page, opencore updates once a month, and every month I will try to update the image with the latest releases (opencore and kexts).

[Mattaton]

5 minutes ago, ghost82 said:

That's because it's been a year macinabox came out and with apple all is in continuous development, for example:

1. most of the time on every upgrade the product id of the os will change --> macinabox has it hardcoded and this will result in no product found

2. every year, when new macs come out in stores with a new apple os, board-ids change and if 1 day before your mac was able to get big sur, the day after it will be able to get monterey, or it won't be supported anymore, so no more updates; that's why the unpached macinabox tries to install monterey, because the hardcoded board-id is able to get monterey

3. on every minor upgrade, one should follow the opencore development, because it can be that the actual bootloader is able to boot v. x.x.1, but not x.x.2

 

When macinabox came out, it was really very easy to setup all, all was working as expected, but maintaining the container requires a lot of efforts, to follow all these things and update the code.

 

So yes, I agree that at the time of writing, it's easier to setup a mac os vm manually!

 

Correct, the img contains all the basic kexts to have a running vm, with some others for gpu passthrough.

 

Note that the image contains "standard" smbios data, so you may want to backup your current smbios data (if you changed them) and replace them in the new image once the vm is booted.

Have a look once a month at the pull requests page, opencore updates once a month, and every month I will try to update the image with the latest releases (opencore and kexts).

You're an invaluable wealth of knowledge, sir! Thanks so much for the detailed explanations. I have learned a lot in just a few days!

 

Sidenote question for you about the OpenCore bootloader...

Via VNC, have you ever had the issue where the bootloader picker screen goes black as the mouse runs over the disk icons? Eventually, it's all just black and I have to blindly press Enter and hope the default drive is correct. I thought it was isolated to one instance, but it does the same thing on a different server. Same OpenCore.img, but different OS.

[ghost82]

5 minutes ago, ghost82 said:

so you may want to backup your current smbios data (if you changed them) and replace them in the new image once the vm is booted.

This is valid mostly if you login with an apple id.

If you change smbios data frequently, apple will think that your account is hacked and it will lock it server side.

That happened to me at the beginning (because as all of us, knowing nothing, one has to start experimenting with something..) and the only way to unlock it was to call apple by phone, and after the 3rd call I met a good guy able to unlock my account.

[ghost82]

2 minutes ago, Mattaton said:

Via VNC, have you ever had the issue where the bootloader picker screen goes black as the mouse runs over the disk icons?

Not black screen, but I was in contact with another user having issues with opencanopy (opencanopy is the name of the picker with the icons).

You can read about this issue here:

https://github.com/acidanthera/bugtracker/issues/1876

 

The solution was to upgrade the ovmf files, with the attached one, for example.

Just point in the xml the paths to the new files:

      <loader readonly='yes' type='pflash'>/path/to/OVMF_CODE.fd</loader>
      <nvram>/path/to/OVMF_VARS.fd</nvram>

 

edk2-OVMF-202011-Stable-RELEASE-GCC5-Kali.zip

[Mattaton]

15 minutes ago, ghost82 said:

This is valid mostly if you login with an apple id.

If you change smbios data frequently, apple will think that your account is hacked and it will lock it server side.

That happened to me at the beginning (because as all of us, knowing nothing, one has to start experimenting with something..) and the only way to unlock it was to call apple by phone, and after the 3rd call I met a good guy able to unlock my account.

I haven't logged in with any Apple ID yet. Am I good to create a new Monterey VM (with a new SMBIOS) if I get the AMD card, or should I try to use the same SMBIOS from my current Big Sur VM?

 

Clear some (more) confusion for me...you say to check once a month on the OpenCore updates. Does that mean I should be updating the bootloader image monthly on an existing/working VM? Or once I get a VM going and working properly, I can just use it as a normal computer without worrying about the bootloader?

[ghost82]

7 minutes ago, Mattaton said:

Am I good to create a new Monterey VM (with a new SMBIOS) if I get the AMD card, or should I try to use the same SMBIOS from my current Big Sur VM?

Unless you log in with an apple id you can do what you want and create/use all the smbios data you want.

 

7 minutes ago, Mattaton said:

Does that mean I should be updating the bootloader image monthly on an existing/working VM? Or once I get a VM going and working properly, I can just use it as a normal computer without worrying about the bootloader?

If you freeze your system (i.e.) you don't update it, you are good to go with the initial bootloader.

If you update mac os, even minor updates, you should be better to update the bootloader and kexts too.

[Mattaton]

1 minute ago, ghost82 said:

Unless you log in with an apple id you can do what you want and create/use all the smbios data you want.

 

If you freeze your system (i.e.) you don't update it, you are good to go with the initial bootloader.

If you update mac os, even minor updates, you should be better to update the bootloader and kexts too.

So I understand correctly, the SMBIOS is basically the identity of the Mac, it contains model, serial number, and hardware info that identifies it on Apple servers? Does it matter what model you pick from the lineup?

 

Gotcha, so if you ever let the OS update, you need to update the bootloader. Is that just as simple as swapping out the OpenCore image in the VM with the latest one? I have the OpenCore image as a separate image as you suggested. I did not modify the EFI on the OS image. And as we touched on earlier, the OpenCore image already has the kexts, right?

 

So, I would also assume you never want to check the box in MacOS to automatically apply updates!

 

Just getting a feel for what I'm getting into with an ongoing MacOS VM. 🙂 

[ghost82]

19 minutes ago, Mattaton said:

So I understand correctly, the SMBIOS is basically the identity of the Mac, it contains model, serial number, and hardware info that identifies it on Apple servers?

Correct

19 minutes ago, Mattaton said:

Does it matter what model you pick from the lineup?

Yes, imacpro1,1 is the best choice you can have, especially if you will want to have hevc decoding on hardware level.

 

19 minutes ago, Mattaton said:

Gotcha, so if you ever let the OS update, you need to update the bootloader

It's not a must, a bootloader can work for months, even a year, it can even work with different major oses, but usually having the most recent version of the bootloader only helps to not have an unbootable system.

 

19 minutes ago, Mattaton said:

Is that just as simple as swapping out the OpenCore image in the VM with the latest one?

Yes, but remember that if you replace the whole img you will loose your custom changes (if any), included the smbios data.

 

A second way to update the bootloader is within the vm: just mount the efi partition from within the vm (your actual bootloader), download latest opencore stable (debug version should be preferred), from here:

https://github.com/acidanthera/OpenCorePkg/releases

 

Extract it from the zip, and replace all the files with the .efi extension.

 

The same for the kexts:

https://github.com/acidanthera/lilu/releases

https://github.com/acidanthera/whatevergreen/releases

https://github.com/acidanthera/applealc/releases

https://github.com/acidanthera/virtualsmc/releases

 

Your settings are saved in the /EFI/OC/config.plist file

ATTENTION: before upgrading the files of the bootloader (the .efi files) validate your config.plist with ocvalidate, which is included in the zip package you will download in the Utilities/ocvalidate folder.

For example, from your mac os vm, in a terminal:

cd /path/to/ocvalidate
./ocvalidate /path/to/your/config.plist

 

This will validate your config.plist file and you could see if the new version of the bootloader requires new entries in the config.plist or it requires to delete old entries.

Each version of opencore has its own ocvalidate.

Edited January 10, 2022 by ghost82

[Mattaton]

3 minutes ago, ghost82 said:

Correct

Yes, imacpro1,1 is the best choice you can have, especially if you will want to have hevc decoding on hardware level.

 

It's not a must, a bootloader can work for months, even a year, it can even work with different major oses, but usually having the most recent version of the bootloader only helps to don't have an unbootable system.

 

Yes, but remember that if you replace the whole img you will loose your custom changes (if any), included the smbios data.

 

A second way to update the bootloader is within the vm: just mount the efi partition from within the vm (your actual bootloader), download latest opencore stable (debug version should be preferred), from here:

https://github.com/acidanthera/OpenCorePkg/releases

 

Extract it from the zip, and replace all the files with the .efi extension.

 

The same for the kexts:

https://github.com/acidanthera/lilu/releases

https://github.com/acidanthera/whatevergreen/releases

https://github.com/acidanthera/applealc/releases

https://github.com/acidanthera/virtualsmc/releases

 

Your settings are saved in the /EFI/OC/config.plist file

ATTENTION: before upgrading the files of the bootloader (the .efi files) validate your config.plist with ocvalidate, which is included in the zip package you will download in the Utilities/ocvalidate folder.

For example, from your mac os vm, in a terminal:

cd /path/to/ocvalidate
./ocvalidate /path/to/your/config.plist

 

This will validate your config.plist file and you could see if the new version of the bootloader requires new entries in the config.plist or it requires to delete old entries.

Each version of opencore has its own ocvalidate.

I'm gonna have to buy you your drink of choice for all this knowledge you're slapping me with. 😄 

Very much appreciated, sir!

 

I'll be putting all this to use this evening! Well, most of it anyway. 🙂