October 10, 20169 yr Running Dual 2670's on an ASRock EP2C602 Motherboard (The one with only 8 memory slots, if that helps) I am trying to sort out what I can effectively use on this board and what the performance will be when I add VM's and pass through GPU's to them. Currently, I do not have any VM's running that I do not use via VNC, but my goal is to pass through a Windows VM and possibly a MacOS VM as well which will require two GPU's if I want to run them at the same time. I also have two M1015's and a 4 port Intel nic, but I can lose the nic if needed and drop one of the M1015's and add an expander (which I already have, but not using) or some of the Sata ports on the MB itself. I pasted the slot mapping to get advice on if I can run all or some of this all on the same box: 49 PCI Express 3.0 x 4 Slot (PCIE1, White) from CPU_BSP1 50 PCI Slot (PCI2, White) 51 Intel C602 Chipset 52 PCI Express 3.0 x16 Slot (PCIE3, Blue) from CPU_BSP1 53 PCI Express 3.0 x16 Slot (PCIE4, Blue) from CPU_BSP1 54 PCI Express 3.0 x16 Slot (PCIE5, Blue) from CPU_BSP1 55 PCI Express 3.0 x16 Slot (PCIE6, Blue) from CPU_AP1 56 PCI Express 3.0 x16 Slot (PCIE7, Blue) from CPU_AP1 My concern is adding two many cards on the lanes will nuke performance either on the data side or the GPU side. The end goal is to move away from multiple towers and have all my windows and mac instances on this server in my garage and use extenders (Cat6) to send video to my desk on another floor of the home. Any advice on what I can do and which way to head would be a big help before I buy new GPU's for passthrough.
October 10, 20169 yr With dual CPUs all slots can be used, and since all are CPU slots there shouldn't be any performance issues, 2 slots share a x16 link, so it's x8/x8 when both are used, if possible use those for the controllers, the quad port NIC can be used in the x4 slot.
October 10, 20169 yr Hey there, writing from a vm right now, running on an asrock EP2C602 (4Lan+16 Dim) Model. I use two Xeon 2680 which are different in clock but nothing else of importance in comparisson with the 2670. I use a gtx 770 and a USB 3.1 Controller card per vm at a total of 2 vms. I run them at the same time with good gaming performance and overall feel and speed - I would describe the performance and overall feel and appearance as 99% native. Each CPU is capable of 40 PCIE lanes totaling to 80 Lanes, so you got some lanes to play with, in my case the user manual was not 100% accurate so do not count on it and make your own observations. The two vms of mine are in daily use by multiple people at once and errors have yet to occur. Pushing data over the network sometimes reaches >100mbs to the array of 2*3tb WD Reds + parity but currently I only get around 60 mbs (because the drive is >90% full. Caching is off for everything except for the stuff that stays on the cache, so the write speeds are actual long term values. The virtual disks of the multiple vms are stored on the cache in unraid which in my case consists of 2 ssds. Write and read speeds are very good, boot time of the vm is faster then boot times of a (whole) native system. I am happy with my system and currently some IT people I work with want to follow my build. Considerations: You probably want usb controllers passed through, I tested usb device-based passtrough on multiple motherboards with bad experience, but whole pcie controllers work fine. If you do use device based passtrough, basic microsoft stuff worked fine (for me) while logitech crashed the machine. Sleep states might have to be turned off, or on, in bios, depending on the hardware you use, and if you are going to stub controllers. A monitor with a headphonejack to output the gpus sound would be good, there are external adapters from hdmi to hdmi+audiojack as well. If there isn't a display connected to a vm with a passed trough GPU, it might not be able to start correctly. Some screens do not correctly listen to the ports and might cause issues. I had one such monitor. I'd recomend UEFI GPUs only. Fan Speed is way of what I would like to achieve for a silent system, so I do not use the fan headers on the board and use an external fan controler instead(except for the cpu fans). Whatever I set in BIOS, I always get 100% rpm on my fans. Heat emissions of the xeons is much less then I expected, this might be due to the fact that I never realy even used 1/3 of the horsepower even while benchmarking, Idle temperature is pretty cool aswell even on semi passive cpu heatsinks. You need to check if your CPU model is of the right stepping, so that it supports vt-d. Easiest way to check for that support is to press the info button on unraid itself looking for IOMMU = enabled. If this is not enabled yet, you might have to find that entry in the BIOS, if it is not possible to set it there, you might have an incompatible cpu. If it odes not support vt-d, you can not pass trough the pcie devices correctly. What is your main reason to have the server located in the garage? As the cost factor of extenders might be so big, that you might be able to upgrade your system instead to xeon 2687 v3 and passively cool them or something like that. Can you explain in detail how you want to setup your "clients" and how you want to route the connections to there? If you want to game on the machines, I would take "latency" into consideration. Network latency to an internet server is a different thing from input latency in the local network.
October 11, 20169 yr Author With dual CPUs all slots can be used, and since all are CPU slots there shouldn't be any performance issues, 2 slots share a x16 link, so it's x8/x8 when both are used, if possible use those for the controllers, the quad port NIC can be used in the x4 slot. That is good news. Now I just need to figure out how to use all the slots when the newer video cards need two. Risers might be in order if I can find room in the case.
October 11, 20169 yr Author Hey there, writing from a vm right now, running on an asrock EP2C602 (4Lan+16 Dim) Model. I use two Xeon 2680 which are different in clock but nothing else of importance in comparisson with the 2670. I use a gtx 770 and a USB 3.1 Controller card per vm at a total of 2 vms. I run them at the same time with good gaming performance and overall feel and speed - I would describe the performance and overall feel and appearance as 99% native. Each CPU is capable of 40 PCIE lanes totaling to 80 Lanes, so you got some lanes to play with, in my case the user manual was not 100% accurate so do not count on it and make your own observations. The two vms of mine are in daily use by multiple people at once and errors have yet to occur. Pushing data over the network sometimes reaches >100mbs to the array of 2*3tb WD Reds + parity but currently I only get around 60 mbs (because the drive is >90% full. Caching is off for everything except for the stuff that stays on the cache, so the write speeds are actual long term values. The virtual disks of the multiple vms are stored on the cache in unraid which in my case consists of 2 ssds. Write and read speeds are very good, boot time of the vm is faster then boot times of a (whole) native system. I plan on running two as well. One windows 7 or 10 (can't decide which) and a MacOS VM. Do you use virtual nic's on your VM's? I have read they are faster as they do not have to pass through the card, cable and the switch which makes sense. I assume there are no data collisons when you use virtual nic's? I am happy with my system and currently some IT people I work with want to follow my build. Considerations: You probably want usb controllers passed through, I tested usb device-based passtrough on multiple motherboards with bad experience, but whole pcie controllers work fine. If you do use device based passtrough, basic microsoft stuff worked fine (for me) while logitech crashed the machine. So your USB devices are on a PCIE card? Do you have seperate cards for each VM or can you use one with a lot of ports and spilt them up? I also ran into the Logitech issue the other day while setting up my dads VM. Sleep states might have to be turned off, or on, in bios, depending on the hardware you use, and if you are going to stub controllers. I don't use sleep, but good to know. A monitor with a headphonejack to output the gpus sound would be good, there are external adapters from hdmi to hdmi+audiojack as well. I have looked at those displays and will need to get something like that to get sound to my desk If there isn't a display connected to a vm with a passed trough GPU, it might not be able to start correctly. Some screens do not correctly listen to the ports and might cause issues. I had one such monitor. I'd recomend UEFI GPUs only. I will keep that in mind Fan Speed is way of what I would like to achieve for a silent system, so I do not use the fan headers on the board and use an external fan controler instead(except for the cpu fans). Whatever I set in BIOS, I always get 100% rpm on my fans. Heat emissions of the xeons is much less then I expected, this might be due to the fact that I never realy even used 1/3 of the horsepower even while benchmarking, Idle temperature is pretty cool aswell even on semi passive cpu heatsinks. I have swapped out all the fans on my SM case and have dual fan Noctua's on the 2670's. It's pretty quiet now, compared to how it came. My PFSense box makes the most noise as it is a repurposed internet security appliance. Not a whole lot of options for 1U cooling that is quiet. You need to check if your CPU model is of the right stepping, so that it supports vt-d. Easiest way to check for that support is to press the info button on unraid itself looking for IOMMU = enabled. If this is not enabled yet, you might have to find that entry in the BIOS, if it is not possible to set it there, you might have an incompatible cpu. If it odes not support vt-d, you can not pass trough the pcie devices correctly. Both of my 2670's have a SR0KX stepping and IOMMU is active and working. What is your main reason to have the server located in the garage? As the cost factor of extenders might be so big, that you might be able to upgrade your system instead to xeon 2687 v3 and passively cool them or something like that. Can you explain in detail how you want to setup your "clients" and how you want to route the connections to there? If you want to game on the machines, I would take "latency" into consideration. Network latency to an internet server is a different thing from input latency in the local network. I bought a Tam's 4U SM chassis years ago and picked up a 42U rack for free. I have always liked the rack setup even though I know many have moved away from that now due to the smaller MOBO's and larger capacity drives. Plus my wife is a neat freak and I don't have a closet with good air flow near my desk that I could hide the rack (or build one) so she does not have to look at it. I know as long as I keep it out of sight, she doesn't say anything about the bill for server parts. Thank you for the reply! It seems like your setup is very similar to what I am trying to do so this info is a big help so I don't run in circles trying different parts that will not work in the first place.
October 11, 20169 yr Regarding OS: I strongly recomend windows 10 instead of windows 7. From a stability and performance standpoing windows 10 on OVMF boot is the way to go. I can't help you with mac...but Linux distros worked fine for me aswell. Regarding NICS: In unraid you can choose btw. multiple modes of virtual nics. There is an option that acts like a default virtual nic and one that lets the vm kind of use the nic directly. One can be found in the net by others and one does not. It is possible to, say, have two VMs join the same locale gameserver for instance, or to not let them know of eachother, even if both are conected to the internet through unraid on the same nic. I think with the newer version(s) of unraid you can also pass one of the nics you have to the VM, but I do not know what is the difference between this and sharing a nic (beside obviously that the load gets distributed to multiple Ports). Well, you can aswell just put a PCI(E) ethernet card into your system and pass that one trough aswell. From my experience the network is stable and I did not have Issues with multiple VMs on multiple ips on the same NIC. Regarding USB: 1. You can Pass trough a single device, like a mouse, it's what you do if you pass torugh USB stuff in the gui. 2. You can Pass trough a USB controller, like if you got multiple controllers on your motherboard (do not mix port with controllers here) 3. and you can pass trough a PCIE device, which can be/inherit a USB controller aswell. 1 does work for simple devices like basic keyboards, mouse, sometimes flash sticks, but compatiblity issues are not rare, and your system (whole unraid) might crash or hang especially if you use multiple vms with this kind of passtorugh at once. Also sometimes devices can't reconect once a vm is shutdown and restartet and other stuff. Many devices that reconect or have different power or action states do not work, like bluetooth dongles for instance. Hotplug does not work. I strongly doubt you will be happy with this option. option 2 works just fine, if you got enough controllers, you probably do not, and USB 2 might not be enough anyway. Keep in mind to not passtrough the usb controller to a vm, that the unraid boot stick is on option 3 is the way to go i'd say. it costs though, uses up some pci(e) slots and adds power draw. I bought two usb 3.1 controllers, so one per vm. normaly there is only one controller per card, so you can not split them, also you pass torugh the whole pcie device, so whatever is on that card is for one vm only. I can confirm the asrock usb 3.1 usb controller to work with my board w hich basicaly is the same as yours. Regarding sleep:I never use sleep states on windows, i'd rather hibernate, however, unused devices such as a gpu or a usb controller that is not currently in use by a vm IS in sleep mode. So you do neeed to care for that option. The asrock usb controller mentioned even has jumpers to set behavior like power draw at sleep state. Regarding your Pfsense you can run pfsense on unraid, but you probably know that already? Regarding placement/case: I use a Fractal Define XL 2, it is a tower case that fits SSI-EEB boards. It has 8 drive bays and 4 additional optical drive bays, so it is easy to cram a bunch of drives in it. It is super silent / 0db according to what it is doing at the moment and sits just in the living room. You seem to know a thing or two about Wiring out the in and outputs per Cat6, do you want to share your options here or per pm? I might be interested and do not have experience with that. Some more recomendations: modern GPU which has 0 rpm fans in idle/sleep such as a gtx 1060 if someone can confirm that series to work well with passthrough some cheap old screen with vga to output the text console to unraid itself. I rarely use it but it is nice to have just in case. You do have IPMI on your board as do I so that's not super important. A set of input/output devices per VM and additionaly some cheapo keyboard and maybe mouse for your unraid itself. Wlan Router, so you can easily acces your unraid gui per phone. If I am done streaming stuff in bed on the phone I do not have to get up to power down my pc and such. Set static ip adresses - boot time will be cut immensely depending on your network. I save about 5-20 seconds on boot just with static ip on a Fritz box 4790 Router (maybe set a static ip for ipmi also and configure both ip's in your router) If you run pfsense you probably got all that figured out anyway.
October 12, 20169 yr Author Regarding your Pfsense you can run pfsense on unraid, but you probably know that already? Regarding placement/case: I use a Fractal Define XL 2, it is a tower case that fits SSI-EEB boards. It has 8 drive bays and 4 additional optical drive bays, so it is easy to cram a bunch of drives in it. It is super silent / 0db according to what it is doing at the moment and sits just in the living room. You seem to know a thing or two about Wiring out the in and outputs per Cat6, do you want to share your options here or per pm? I might be interested and do not have experience with that. Some more recomendations: modern GPU which has 0 rpm fans in idle/sleep such as a gtx 1060 if someone can confirm that series to work well with passthrough some cheap old screen with vga to output the text console to unraid itself. I rarely use it but it is nice to have just in case. You do have IPMI on your board as do I so that's not super important. A set of input/output devices per VM and additionaly some cheapo keyboard and maybe mouse for your unraid itself. Wlan Router, so you can easily acces your unraid gui per phone. If I am done streaming stuff in bed on the phone I do not have to get up to power down my pc and such. Set static ip adresses - boot time will be cut immensely depending on your network. I save about 5-20 seconds on boot just with static ip on a Fritz box 4790 Router (maybe set a static ip for ipmi also and configure both ip's in your router) If you run pfsense you probably got all that figured out anyway. I keep pfsense on a stand alone box so the rest of the family can use the WiFi while I am playing with the unRaid server or in the event of a crash that I can't fix until I get home. I thought about including it as I would love to have one less box, but the recycled security appliance is only 1U in the rack and has more than enough power for my home needs. I think I will end up with CAT6 extenders for the HDMI to my double/triple monitor setup. I would love to use one Displayport cable from the garage to an MST hub that would allow for up to three monitors, but I cannot find a cable that long that I would trust. Most reviews say the longer cables cannot handle the data correctly and the monitors do not respond or don't display the correct resolution. I have enough room in my rack currently to have a 19" monitor sitting on a shelf with an old keyboard to see the server boot. I might add a KVM setup down the road, but for now, I don't need it and I have the room for a monitor to sit there. I have IPMI working as well. Now off to find some deals on GPU's and also pickup a USB card or two. Thanks again for the suggestions.
Archived
This topic is now archived and is closed to further replies.