Multiple OS Build Advice


Guan

Recommended Posts

Hi All,

 

I am starting a new build to run multiple OS (namely two ubuntu and windows machine on my server). As I am trying to source from a server supplier for the hardware required (EXALIT), we came up with a few enquiries regarding it.

 

The hardware that we are going to run are:

CPU: 2 x Xeon Gold 5218

Motherboard: PCI-E lanes without PLX chip

RAM: 8 x 32GB DDR4-2933 ECC RDIMM 

Storage: 4 x 960GB Enterprise SATA SSD, 2.5”, 1DWPD endurance, 2M MTBF

GPU: 3 x RTX 2080Ti equivalent (Might be RTX 4000 as it is a single slot GPU)

USB PCI-E Cards: TBD (decided upon query answered)

GIGABIT LAN PCI-E Cards: TBD (decided upon query answered)

 

As we would like to run multiple OS at the same time, the queries are:

1) Can unRAID assign GPUs to each OS if they are running on a PLX chip?
2) Can unRAID assign a couple of USB ports for all the different OS from the same USB controller? (same question for LAN ports)
3) Can unRAID use the motherboard usb and gigabit LAN port whilst the other OS use the PCI-E usb and LAN ports?

 

Thank you for assisting me on the queries!

 

Cheers,

-Guan-
 

Link to comment
8 hours ago, Guan said:

1) Can unRAID assign GPUs to each OS if they are running on a PLX chip?

I would need to test to verify, but I don't believe PLX prevents GPU assignment, though it may impact IOMMU groupings and the rules for those remain true.  You would need to boot up a server with all your PCI-e add-on cards installed to see the IOMMU groupings to determine what assignments are possible.

 

8 hours ago, Guan said:

2) Can unRAID assign a couple of USB ports for all the different OS from the same USB controller? (same question for LAN ports)

There is a specific USB add-on controller that you can purchase which is a single physical controller but has four discrete USB controllers built inside of it.  We use this controller and assign an individual port to each VM, using a USB hub to create multiple ports out of the one for each user.

 

You cannot assign individual USB ports themselves to VMs in Unraid yet.  This is something we have looked into in the past, but haven't prioritized for development just yet.

 

8 hours ago, Guan said:

3) Can unRAID use the motherboard usb and gigabit LAN port whilst the other OS use the PCI-E usb and LAN ports?

Yes, but while the benefits are obvious for using PCI-e USB controllers for VMs, using virtual NICs for VMs is far more efficient than trying to do NIC passthrough (though there are of course use-cases for wanting that).

Link to comment
14 hours ago, trurl said:

SSDs are not recommended in the parity array but can be used in the cache pool. At least one data disk is required in the array. 

Thanks Trurl, will look into getting data disks as storage devices. The usage of SSDs was to make it as the primary drive for OS booting and software launching. May I know if I could mount it outside of the array to do as such, rather than combining them into one array as a cache pool?

 

6 hours ago, jonp said:

I would need to test to verify, but I don't believe PLX prevents GPU assignment, though it may impact IOMMU groupings and the rules for those remain true.  You would need to boot up a server with all your PCI-e add-on cards installed to see the IOMMU groupings to determine what assignments are possible.

Thanks jonp for replying. I think I will tell the supplier to provide me with a motherboard and cpu combo that does not need a PLX chip for my usage as it poses a lot of unkowns to me.

 

6 hours ago, jonp said:

There is a specific USB add-on controller that you can purchase which is a single physical controller but has four discrete USB controllers built inside of it.  We use this controller and assign an individual port to each VM, using a USB hub to create multiple ports out of the one for each user.

May I know if this is the similar to the one you were mentioning? http://techsupportlb.com/product/pvu3-5o2u-usb3-0-7-ports-desktop-pci-express-card-with-vl800-vl812-usb-3-0-controller/

Do you have a link to the specific usb add-on controller that you mentioned? That would be help tremendously with my serach.

 

6 hours ago, jonp said:

Yes, but while the benefits are obvious for using PCI-e USB controllers for VMs, using virtual NICs for VMs is far more efficient than trying to do NIC passthrough (though there are of course use-cases for wanting that).

 Noted with the suggestion. Thanks a lot for helping!

Link to comment
7 hours ago, Guan said:

May I know if there are any LAN cards with four different controllers like the USB cards that @jonp mentioned? It will be of great help! Thanks!

I use the Intel I350 ethernet controller.

IOMMU group 14:	[8086:1521] 07:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
IOMMU group 15:	[8086:1521] 07:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
IOMMU group 16:	[8086:1521] 07:00.2 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
IOMMU group 17:	[8086:1521] 07:00.3 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)

 

Link to comment
  • 1 month later...
On 10/11/2019 at 1:20 AM, bonienl said:

I use the Intel I350 ethernet controller.

Thank you for your help!

 

I have another question regarding USB3 cards. 

After researching for a bit, I realised that PEXUS Startech PEXUSB3S44V does not play well with unRAID as seen in the links below:

 

 

 

 

 

 

 

 

 

May I know if there are any usb controllers that have 4 usb port that work well with unRAID with Ubuntu and Windows VM?

 

I saw using people using Sonnet Allegro Pro having some success and some failures as well. Can someone advise me on whether this work for four VMs?

Link to comment
53 minutes ago, Guan said:

I saw using people using Sonnet Allegro Pro having some success and some failures as well. Can someone advise me on whether this work for four VMs?

I saw John Bartlett (@jbartlett) using RocketU 1144D and Vantec Quad Controller with success for his VMs, may I know if this is true for everyone?

 

Thanks for your help again!

Link to comment

The RocketU 1144D card groups all of their controllers in one IOMMU group so you are limited to attaching them to just one VM without using the PCIe Override switch. I did not test performance with the override set and multiple VM's.

 

The Vantec Quad controller put each controller in their own IOMMU group but I had issues using a high speed device.

 

I've had good success on using unraid's USB passthrough via the VM GUI if it's not a high data device (audio worked) but devices that function as a hub such as a security dongle won't work.

 

In my current VM build, I had better luck with running two Brio's at 1080p@60fps passing through a MB USB controller than a PCIe card - less flicker which represented (to my guess) data not flowing at a consistent rate. I almost had to ditch unraid as a VM host in favor of Hyper-V because of it until I retried the MB USB controller - I'm limited to the number of cams I can plug in but it's smooth.

 

You *WILL* need to pay close attention to what PCIe slots/USB controllers are connected to what CPU, that will potentially limit your hardware configuration.

Link to comment
8 minutes ago, jbartlett said:

I've had good success on using unraid's USB passthrough via the VM GUI if it's not a high data device (audio worked) but devices that function as a hub such as a security dongle won't work.

@jbartlett, thanks a lot for replying to this thread.

 

Just another query, if I were to pass through a USB port per VM using the Vantec Quad Controller (essentially 4 VMs), I cannot use that usb port to connect up to a USB hub that connects my keyboard and mouse?

Link to comment
50 minutes ago, Guan said:

Just another query, if I were to pass through a USB port per VM using the Vantec Quad Controller (essentially 4 VMs), I cannot use that usb port to connect up to a USB hub that connects my keyboard and mouse?

You can use anything you want hooked up to that one port. For example, a hub with even several other hubs connected to that hub.

 

You may run into speed issues on that bus. My DiskSpeed docker app also maps out the USB tree and reports the bus speed on each one - can help if you have issues.

Link to comment

I see the confusion. Not every USB device will function properly when passed through by itself though if plugged into a USB port that is passed through by itself or by a parent port, then it should work.

 

ETA: I get confused just reading my less confused explanation. Too many "passed" and "through"s

Edited by jbartlett
Link to comment
2 hours ago, jbartlett said:

I see the confusion. Not every USB device will function properly when passed through by itself though if plugged into a USB port that is passed through by itself or by a parent port, then it should work.

Okay, thanks a lot for the explaination! 

Will go ahead with the Vantec quad port then. Hopefully will manage to find one..

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.