VM confusion - Single Machine


ne10g

Recommended Posts

Hi All

 

I'm sure I'm missing something obvious, searched around but my heads now spinning!

 

I have one machine I would like to use with unRAID, as my main machine.

 

Main machine has:

 

2 x GPU's: 1 x Nvidia 2080ti and 1 x Nvidia 1650

1 x USB-C HUB (from the 2080ti)

2 x NIC

1 x Monitor - multiple inputs

 

Plan is, as well as the "usual" stuff such as NAS, I would like:

 

1) WIN10 Gaming VM - I would like this to use the 2080ti and be playable direct from the unRAID server 

 

2) Ubuntu VM - for Plex, radarr, sonarr etc - I would like this to use the 1650 GPU - maybe display on monitor using separate port (not essential)? Use its own NIC, 1 of the 2 available.

 

I'm sure I am being thick, but I cannot get my head around how to make this work...

 

Thank you!

 

Link to comment
1 minute ago, trurl said:

What is the relevance of this? I hope you are not planning to use it for your disks. You need permanent SATA / SAS connections to your disks.

I think I read in one of the many threads that this could be used in a VM as a plug n play hub for USB devices - as it was passed through as part of the 2080ti graphics card?

 

Not to be used for disks, all connected normally.

 

I should also say:

 

2 x 2TB NVME

1 x 512GB SSD

4 x 14TB WD RED HDD's

 

For speed - would it be best practice to pass through 1 x NVME to each of those 2 VM's? Then use the 4 x HDD's in the array, 1 x parity and 3 x disk?

 

I need those two VM's to have access to at least 1 NVME each, especially the one for Ubuntu, as there'll be lots of IO on that one.

Link to comment
15 minutes ago, ne10g said:

2) Ubuntu VM - for Plex, radarr, sonarr etc - I would like this to use the 1650 GPU - maybe display on monitor using separate port (not essential)? Use its own NIC, 1 of the 2 available.

# 1 is the easy one.  For #2, why not run these natively in unraid with docker?  That way you aren't tying up resources for that VM.

Link to comment
Just now, civic95man said:

# 1 is the easy one.  For #2, why not run these natively in unraid with docker?  That way you aren't tying up resources for that VM.

That leads me nicely onto my next question :)

 

Would there be any issues with using the 1650 for hardware transcoding in Plex if I was to run natively with docker instead of an Ubuntu VM?  I'm not precious about the VM, docker would be better if it would do the job.

Link to comment
1 minute ago, ne10g said:

Would there be any issues with using the 1650 for hardware transcoding in Plex if I was to run natively with docker instead of an Ubuntu VM?  I'm not precious about the VM, docker would be better if it would do the job.

1650 only supports 2 transcodes, thats the only issue. and you would need to use the nVidia build of unraid (many of us do that).

 

with a VM, you are setting aside X cpu cores and Y memory for that virtual computer; whereas with docker, you share the system's cpu and memory between unraid and the different containers (plex, sonarr, etc). so those processes can use as much or less as needed without wasting memory and resources. 

Link to comment
1 minute ago, civic95man said:

1650 only supports 2 transcodes, thats the only issue. and you would need to use the nVidia build of unraid (many of us do that).

 

with a VM, you are setting aside X cpu cores and Y memory for that virtual computer; whereas with docker, you share the system's cpu and memory between unraid and the different containers (plex, sonarr, etc). so those processes can use as much or less as needed without wasting memory and resources. 

Yeah thats a good point. So I guess in a VM, there could be more options, which I wont mention as I read the notes in the nvidia build thread ;)

 

So maybe the VM route is the one I need to take for now if I want dedicated GPU for transcodes?

 

Or....could I just let the CPU (i9-10940X) do hardware transcoding in the plex docker? If it even can?

 

Sorry for all the queries, please bare with me! :D

 

Link to comment
4 minutes ago, ne10g said:

So I guess in a VM, there could be more options

I was referring to using everything (plex, sonarr, radarr, etc) within docker so that your system (unraid) can better utilize the resources. 

 

5 minutes ago, ne10g said:

which I wont mention as I read the notes in the nvidia build thread

I'm assuming you are referring to bypassing the 2 transcode limits, yeah, you are on your own, or get the p2000 card to handle unlimited transcodes.  It really depends on what you are trying to transcode and how many simultaneous streams you expect to encounter. If you don't anticipate any 4k stuff then gpu/cpu would be fine (2 via gpu, the rest with cpu).

9 minutes ago, ne10g said:

Or....could I just let the CPU (i9-10940X) do hardware transcoding in the plex docker? If it even can?

you can but gpu is always better than cpu (unless you REALLY care about quality, but thats another debate)

 

10 minutes ago, ne10g said:

Sorry for all the queries, please bare with me!

no problem, we were all there at one point.

  • Thanks 1
Link to comment
1 minute ago, civic95man said:

I was referring to using everything (plex, sonarr, radarr, etc) within docker so that your system (unraid) can better utilize the resources. 

 

I'm assuming you are referring to bypassing the 2 transcode limits, yeah, you are on your own, or get the p2000 card to handle unlimited transcodes.  It really depends on what you are trying to transcode and how many simultaneous streams you expect to encounter. If you don't anticipate any 4k stuff then gpu/cpu would be fine (2 via gpu, the rest with cpu).

you can but gpu is always better than cpu (unless you REALLY care about quality, but thats another debate)

 

no problem, we were all there at one point.

Not many transcodes tbh, you know what...maybe none! Only a handful of people, I will block 4k for family who cant direct stream.

 

Ok, so docker it is!

 

So with that in mind, I just pass the 2080ti to the Win10 VM and leave the 1650 for unRAID's sole use?  

 

This may sound silly, but how would I use the Win10 VM, direct on the server? Just want the one machine for all.

Link to comment
1 minute ago, ne10g said:

So with that in mind, I just pass the 2080ti to the Win10 VM and leave the 1650 for unRAID's sole use?  

 

This may sound silly, but how would I use the Win10 VM, direct on the server? Just want the one machine for all.

I don't know if you've even setup your server yet, let alone booted unraid yet.  I would suggest looking up spaceinvaderOne's videos.  He has videos on just about everything which are extremely helpful.

 

You would probably want to start with getting the nvidia build (through CA).  Then you would want to isolate the 2080ti card that you want the VM to use (again, look up his videos) - this just tells unraid not to use this card/ ignore it.  Then, when you setup the VM (again, watch the video) you are given the option to use the 2080ti card.

 

Now for general unraid use, with the nvidia build, you will be able to manage unraid via the 1650 card (assuming you are booting into the gui mode) and still use it for transcoding.  This will give you an option to manage the system if you don't have another computer available.  

 

In my situation, i have a monitor with two inputs - one from my win10 VM and another from the igpu that unraid uses.  So I can manage unraid with one input and interact with windows on the other input.  I also pass through several USB ports to the vm and attach the keyboard and mouse to those for win10 use.  I run my unraid "headless" in a sense where I don't really use the monitor and have no keyboard for unraid use.

 

Hope this gives you a little information to get started.

 

Good luck!

  • Thanks 1
Link to comment
1 minute ago, civic95man said:

I don't know if you've even setup your server yet, let alone booted unraid yet.  I would suggest looking up spaceinvaderOne's videos.  He has videos on just about everything which are extremely helpful.

 

You would probably want to start with getting the nvidia build (through CA).  Then you would want to isolate the 2080ti card that you want the VM to use (again, look up his videos) - this just tells unraid not to use this card/ ignore it.  Then, when you setup the VM (again, watch the video) you are given the option to use the 2080ti card.

 

Now for general unraid use, with the nvidia build, you will be able to manage unraid via the 1650 card (assuming you are booting into the gui mode) and still use it for transcoding.  This will give you an option to manage the system if you don't have another computer available.  

 

In my situation, i have a monitor with two inputs - one from my win10 VM and another from the igpu that unraid uses.  So I can manage unraid with one input and interact with windows on the other input.  I also pass through several USB ports to the vm and attach the keyboard and mouse to those for win10 use.  I run my unraid "headless" in a sense where I don't really use the monitor and have no keyboard for unraid use.

 

Hope this gives you a little information to get started.

 

Good luck!

Yeah I've been watching the videos for a week or so now, lots to get through, very informative.

 

I've been dabbling with the server for a few days, then realised I needed another GPU, so just got that today and will try again. I was using the nvidia build funnily enough, stumbled upon that in the CA, seems very good!

 

The penny has dropped now, I should just run it headless, then I can manage it either from inside the W10 VM, or from an ipad or something on the same network.  Then (the bit I was missing!) when I start W10 with the 2080ti dedicated to it, I can use the main monitor, just use the other input?

 

I knew I was being thick haha!

Link to comment

Actually, one more quick query for now, if you don't mind!

 

Am I right in setting up like:

 

1 x 14TB Parity disk

3 x 14TB Disks

 

Then what would be the best way to utilise the 2 x 2TB NVME's and 1x512GB SSD please?

 

I'd like Plex and the dockers to have use of most of the fast space on the NVME's. 

 

Should I leave them unallocated, or use them as cache disks?

Link to comment
8 minutes ago, ne10g said:

Then (the bit I was missing!) when I start W10 with the 2080ti dedicated to it, I can use the main monitor, just use the other input?

That's what I do.  Granted, I just use mine headless - I still have the monitor attached to the igpu for diagnostics and to mess with the BIOS.  I don't even boot up into the GUI mode - all of my admin work is done on another computer (or even my phone in a pinch).  Most of the time I have win10 running within the VM and output going to the monitor with it's dedicated mouse/keyboard.

 

11 minutes ago, ne10g said:

I've been dabbling with the server for a few days, then realised I needed another GPU, so just got that today and will try again. I was using the nvidia build funnily enough, stumbled upon that in the CA, seems very good!

I'm assuming you're running the beta version - 6.9? with that new of hardware, I'm sure you must be.

12 minutes ago, ne10g said:

The penny has dropped now, I should just run it headless, then I can manage it either from inside the W10 VM, or from an ipad or something on the same network.  Then (the bit I was missing!) when I start W10 with the 2080ti dedicated to it, I can use the main monitor, just use the other input?

Funny enough, i don't remember if there is an option to make the VM autostart on boot, otherwise you will need a way to start the VM - but your ipad will work.  I personally don't have anything on mine set to autostart - simply because if the server restarted, it wasn't because of me and I want to track down what caused it before potentially making things worse.

 

14 minutes ago, ne10g said:

I knew I was being thick haha!

It's always better to ask questions when in doubt rather than [potentially] making reckless mistakes, such as when it comes to your data

  • Thanks 1
Link to comment
12 minutes ago, ne10g said:

Am I right in setting up like:

 

1 x 14TB Parity disk

3 x 14TB Disks

yeah, if you want parity (i recommend it, but remember that parity is not a backup) then that looks good.  With parity, you will want to make sure turbo write is on (don't know if that is default by now but most likely is).  If you are initially transferring data from another source then some people don't bother with parity until everything has been transferred to speed things up. if you don't have anything to transfer or very little then it doesn't hurt to setup parity now.

 

15 minutes ago, ne10g said:

Then what would be the best way to utilise the 2 x 2TB NVME's and 1x512GB SSD please?

 

I'd like Plex and the dockers to have use of most of the fast space on the NVME's. 

You can setup the NVMe(s) as your cache and then run docker/VM from there.  It's really your preference.  If you decide to use both NVMes as cache then you can choose raid0 or raid1 (raid0 will get you 4tb storage, whereas raid1 will give you a 2tb mirror which can protect you if lose one of the devices, raid0 cannot).  I would recommend the backup plugin on CA which can backup your docker containers to your array so if things do go south then you have a backup of your containers.

 

As for the single SSD, its your call. You could utilize that to run your VM, or with the beta version of unraid, you can set it up as another pool

  • Thanks 1
Link to comment
36 minutes ago, civic95man said:

That's what I do.  Granted, I just use mine headless - I still have the monitor attached to the igpu for diagnostics and to mess with the BIOS.  I don't even boot up into the GUI mode - all of my admin work is done on another computer (or even my phone in a pinch).  Most of the time I have win10 running within the VM and output going to the monitor with it's dedicated mouse/keyboard.

 

I'm assuming you're running the beta version - 6.9? with that new of hardware, I'm sure you must be.

Funny enough, i don't remember if there is an option to make the VM autostart on boot, otherwise you will need a way to start the VM - but your ipad will work.  I personally don't have anything on mine set to autostart - simply because if the server restarted, it wasn't because of me and I want to track down what caused it before potentially making things worse.

 

It's always better to ask questions when in doubt rather than [potentially] making reckless mistakes, such as when it comes to your data

 

So much clearer now! I've just popped back to update.

 

1) The trial is no good as i now have too many disks...lol! i will purchase a license anyway, it's gonna be worth it!

2) I ran out of SATA data connectors, typical! Managed to scrape 3 together, so for now just put the 3x 14Tb drives in. Assuming I can put the 4th in and set as the parity disk at a later date?

 

The journey begins properly now, thanks again!

Link to comment
27 minutes ago, civic95man said:

yeah, if you want parity (i recommend it, but remember that parity is not a backup) then that looks good.  With parity, you will want to make sure turbo write is on (don't know if that is default by now but most likely is).  If you are initially transferring data from another source then some people don't bother with parity until everything has been transferred to speed things up. if you don't have anything to transfer or very little then it doesn't hurt to setup parity now.

 

You can setup the NVMe(s) as your cache and then run docker/VM from there.  It's really your preference.  If you decide to use both NVMes as cache then you can choose raid0 or raid1 (raid0 will get you 4tb storage, whereas raid1 will give you a 2tb mirror which can protect you if lose one of the devices, raid0 cannot).  I would recommend the backup plugin on CA which can backup your docker containers to your array so if things do go south then you have a backup of your containers.

 

As for the single SSD, its your call. You could utilize that to run your VM, or with the beta version of unraid, you can set it up as another pool

Perfect, thanks again! I'm about to jump right back in!

Link to comment

I hit the hurdle that I had with only one gpu, I go into device settings, then select to bind the 2080ti (it selects all 4 devices, including sound etc..) then as soon as I reboot it hangs.

 

I guess I need to tell unraid to use the 1650 as the main gpu before doing this? How would I do that please?

 

EDIT: i remember reading I could wipe the vfio-cfg.cfg if this happened, so I did.

 

The contents, with the change I made previously were:

 

BIND=0000:65:00.0|10de:1e07 0000:65:00.1|10de:10f7 0000:65:00.2|10de:1ad6 0000:65:00.3|10de:1ad7
 

thats the 2080ti identifiers

Edited by ne10g
Link to comment
18 minutes ago, ne10g said:

So much clearer now! I've just popped back to update.

 

1) The trial is no good as i now have too many disks...lol! i will purchase a license anyway, it's gonna be worth it!

2) I ran out of SATA data connectors, typical! Managed to scrape 3 together, so for now just put the 3x 14Tb drives in. Assuming I can put the 4th in and set as the parity disk at a later date?

 

The journey begins properly now, thanks again!

1.  Trial should be good for more than 3 disks. Do you have internet connection to your server so that it can "phone home?"  It only contacts unraid's servers while under the trial license.  You If you aren't already on the beta, then you may need to for the network drivers.  Else, maybe connect to a standard 1gb port instead of a 2.5gbe if your board has those - this is all assuming that your server isn't visible on the network, if it is then disregard.

 

2.  Yes that is perfectly fine.  You can easily add new drive as you please, including parity.  Granted, it will take a good part of a day to create the parity so be aware of that when the time comes.  Not so much critical now since you don't have parity, but you want to ensure all of your disk connections are high quality (power and data).  Don't want to use questionable sata cables as they can cause problems, same for power. 

Link to comment
6 minutes ago, ne10g said:

I hit the hurdle that I had with only one gpu, I go into device settings, then select to bind the 2080ti (it selects all 4 devices, including sound etc..) then as soon as I reboot it hangs.

What do you mean by it hangs?  Do you get to the boot menu to choose the GUI/non-GUI/safemode and then nothing appears on the screen?

 

Try switching to the other gpu (1650) when that happens.  When you isolate the 2080ti, unraid essentially doesn't see it so it can't use it.  Therefore it will fall back to its other gpu (1650).  Failing that, it will go into "headless" mode and not use any video card, in which case you should still see it on the network.

Link to comment
4 minutes ago, civic95man said:

What do you mean by it hangs?  Do you get to the boot menu to choose the GUI/non-GUI/safemode and then nothing appears on the screen?

 

Try switching to the other gpu (1650) when that happens.  When you isolate the 2080ti, unraid essentially doesn't see it so it can't use it.  Therefore it will fall back to its other gpu (1650).  Failing that, it will go into "headless" mode and not use any video card, in which case you should still see it on the network.

I'll do it again and grab a pic of where it hangs.

Link to comment

That is what I expect, I'm assuming that is the output from the 2080ti.  So basically, at that point the kernel binds that video card at the specified hardware location (based on the vfio-cfg.cfg file) to the vfio-pci driver. Thats basically just a placeholder for when the VM starts and it gets passed through.  But once that card binds with that driver, unraid can no longer see or talk to it.

 

So I expect that and your system is still functioning, not hanging.

 

3 minutes ago, ne10g said:

Interestingly, if I unplug the DP cable from the 2080ti then I can boot, certainly headless. Will try GUI now.

The 2080ti must be checking if the monitor is attached, if not then it excludes itself from being available. Does the other 1650 takes over?

 

So you have three options it sounds like, see if you can specify in the bios which card is the primary, rearrange your gpus on the motherboard so the 1650 appears as the primary, or find a kernel boot parameter to specify which card to treat as the primary (haven't checked if possible but I'm sure it it)

Link to comment
14 minutes ago, civic95man said:

That is what I expect, I'm assuming that is the output from the 2080ti.  So basically, at that point the kernel binds that video card at the specified hardware location (based on the vfio-cfg.cfg file) to the vfio-pci driver. Thats basically just a placeholder for when the VM starts and it gets passed through.  But once that card binds with that driver, unraid can no longer see or talk to it.

 

So I expect that and your system is still functioning, not hanging.

 

The 2080ti must be checking if the monitor is attached, if not then it excludes itself from being available. Does the other 1650 takes over?

 

So you have three options it sounds like, see if you can specify in the bios which card is the primary, rearrange your gpus on the motherboard so the 1650 appears as the primary, or find a kernel boot parameter to specify which card to treat as the primary (haven't checked if possible but I'm sure it it)

Thank you.

 

It is the output from the 2080ti yep. I thought it was hanging as when it gets to that point the machine never comes up, I can’t access it even remotely.

 

If I unplug it, then I can get in headless and see the machine remotely. 
 

Looked around in the bios and can’t see anyway to set the primary gpu. If it helps the motherboard is an ASUS TUF X299 MK2.

 

I can’t boot with the 1650 only, can’t get any display it seems to need the primary gpu for any sort of display; even bios?

 

Cant really swap the cards around due to space i wonder if there’s any way to edit via a kernel boot parameter as you say?

Link to comment

Correction. The machine DOES come up. I had it set to dynamic IP, changed that now!

 

So the machine doesn’t hang, but the DisplayPort (from the 2080ti which is now out of bounds for unraid) sticks on the console screen.

 

No output at all from the 1650 via hdmi, tried all ports and also tried DisplayPort to be safe. Assuming I somehow need to ‘tell’ unraid about it?

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.