Hello Forum,

 

currently I am planning my first unraid build. I like to have UNRAID and on top :

 

-2 gaming VMs with GPU passthrough  ( Racing, FPS, Strategy, maybe VR HTC Vive )

-1 VM media server with TV Tuner passthrough, TV headend

-1 VM workstation (Win or Linux)

-1 VM workstation with Win 7 or 10.

 

 

besides that I like to play around with Docker, maybe run a few containers. Would be nice if some of the Dockers can run parallel to the VMs

And some software testing with ´throw away´ VMs but most likely lower load and usually not in parallel to any Gaming VM.

Either I test and crash systems or I game and crash cars. But not both the same time

 

 

I would like to be able to run 2 VMs with passthrough GPU in parallel, but usually no more than one Gaming VM at a time.

 

I am very worried about proper virtualization, clean IOMMU grouping, ACS support, root port isolation and all this.

So at the very beginning I was looking at Skylake i7 but now decided against it because of PCIe lane restrictions and occasionally reported ACS issues.

 

I am thinking to get this CPU

 

-E5-1650v3  6 cores, 40 lanes, ECC support and (afaik) full ACS support

-32 GB ECC

 

 

But for the mainboard I have a hard time figuring it out which one support my needs best.

Some manuals explain that there is an ACS enable/disable switch in the BIOS but say nothing more and

some other manuals do not mention ACS but mention things like :

 

"relaxed ordering, interrupt remapping, SR-IOV support" (Supermicro c612)

"ACS Control, SR-IOV" (Asrock Rack c612)

"IO per DMAR-ACPI table, Interrupt Remapping, coherency support- ISOCH/non-ISOCH"  (ASUS x99 Deluxe)

Options according to manual are Vt-x / VT-d enable/disable    (Asrock x99 WS)

 

 

So, the boards I have currently an eye on:

 

x99 chipset

asrock x99 http://www.asrock.com/mb/Intel/X99%20WS/

asus x99 deluxe/3.1  https://www.asus.com/Motherboards/X99DELUXEU31/

c7x99-oce-f http://www.supermicro.com/products/motherboard/Core/X99/C7X99-OCE-F.cfm

 

c612 chipset

x10sra-f http://www.supermicro.com/products/motherboard/Xeon/C600/X10SRA-F.cfm

epc612d8(or a (audio onboard) http://www.asrockrack.com/general/productdetail.asp?Model=EPC612D8#

 

I dont need fancy features but like to build a rock solid and powerful system.

 

 

Which one would you consider for my use case ?

Any experiences how they group their PCI devices?

 

I am scared that I will end up not having enough PCI(e) slots for all my use cases and that I may not be able to isolate and passthrough individual components such as TV card, USB hub or extension cards.

 

btw, would I be able to run UNRAID without GPU and keep the 2 dedicated cards always available for spin up of one of the 4 VMs that need a video output ? Stop and start of VMs with GPU without reboot of UNRAID ?

Has IPMI and its VGA output any use other than administrating the host hardware ?  Can I use the VGA onboard HW for a third VM ?

 

Your comment and sharing of opinion is most appreciated. 

 

Thanks 

 

unrateable

 

Dear Forum, I like to pull the trigger and would like to start my UNRAID build.

Plenty of time after Xmas 

 

No one here with a proper ACS/IOMMU build with multiple GPU passthrough

who would like to share which HW is in use ?

 

Cheers

 

 

Hi! I believe I have a proper ACS/IOMMU build with multiple GPU passthrough, but I'm not sure it really helps you, because I am using older hardware (See my sig).

 

That said, it works great, and though I've had some teething pains as the VM world is completely new to me, I think my end result is a pretty solid dual-vm with multiple GPU passthrough .

 

Now, I think I can answer some of your questions:

 

I am scared that I will end up not having enough PCI(e) slots for all my use cases and that I may not be able to isolate and passthrough individual components such as TV card, USB hub or extension cards.

It looks like all those boards you posted have 5 or more PCI-e slots. I don't think running out of slots is an issue. 2 GPUs and a TV Tuner still leaves you with slots to spare. With 8+ Sata ports on these boards, and the size of drives being what they are today, one extra PCI-e 8-port controller is probably all you'll ever need (Unless you have a bunch of old small drives you want to recycle, in which case, you could still probably get a second controller in there... or use an expander).

 

That said, it might be worthwhile to find someone actually using the hardware you are looking at to confirm, as there are so many permutations and combinations.... I always recommend that unless you already own the hardware, or you have a very limited selection, go with parts that others are already using and vetted for you...

 

 

btw, would I be able to run UNRAID without GPU and keep the 2 dedicated cards always available for spin up of one of the 4 VMs that need a video output ?

Yup. If the board has a built in vga for IPMI, you should be good to go (in my experience)... I have had a number of combinations of nvidia and AMD GPUs (and a built in vga for IPMI), and never had a problem with unRaid wanting to take over a discrete GPU... The only thing I needed to do was make sure that the onboard IPMI gpu was set as primary in the bios.

 

Stop and start of VMs with GPU without reboot of UNRAID ?

Shouldn't be a problem. I do it all the time.

 

Has IPMI and its VGA output any use other than administrating the host hardware ?  Can I use the VGA onboard HW for a third VM ?

Nope! I tried once and my server crashed hard. Needed to do a hard reboot. Again... My experience ... Perhaps one of the VM gurus knows a trick, but I doubt it...

 

Hopefully that helps you a bit. Again, NOT A DOCTOR . Just speaking from my experience.

 

Were I you, my first step would be identifying a board that is popular with other VM users and build my system around that...

 

@DoeBoye

 

Thanks for contributing !  Nice to hear your HW works out for you.

 

Well, I´ll keep digging into the technicalities for a short while.

 

Right now I am leaning towards a Supermicro x10sra-f  board for the beginning and to see if I can get it to work properly. Second choice would be the ASRock X99 WS (but its targeting overclockers  )

 

PCIe slots: yeah all of them got enough in total, just worried if IOMMU grouping will fit to my VM configs.

 

Haven´t found one particluar 2011-3 board here in the forum that seems to be preferred for what I plan. Guess overall Supermicro builds good grade server stuff, however the http://www.newegg.com/Product/Product.aspx?Item=N82E16813182959 NEWEGG Review and one forum experience here I came across read a bit scary 

 

cheers