May 4, 20188 yr Hi, I am putting together a system for machine learning. The main requirement is the ability to use four NVIDIA Titan V cards on 16x PCI-E lanes. This is needed because we're processing high res images so that bandwidth is important. SLI is not necessary. There is no display needed for the VMs because we'll only be accessing them through ssh. Use Cases: Allocate a single GPU for a VM (ex: Ubuntu for Titan V card 1, Windows 10 for Titan V card 2, CentOS for Titan V card 3, Fedora for Titan V card 4) Allocate multiple GPUs for a VM (ex: Ubuntu with access to Titan V card 1+2, Windows 10 with access to Titan V card 3+4) The VMs can be turned off to reallocate the GPUs All VMs need to be able to read and write to the same shared storage (NAS) Around ~600MB of data every 5 minutes will be generated by each VM (heavy writing on HDDs) This process runs for a couple hours or days. A majority of this data will be transferred to somewhere else and then deleted. (lots of deleting on HDDs) Enclosed chassis (safety requirement from management) What is your budget? Around 20k USD How many drives do you want your server to be able to support and how much capacity do you need? 10TB of regular HDs. I think RAID5 is okay for our usage. Is expandability important to you? If so, what's your long term goal? Yes, it would be nice to add more storage space in the future. Are you interested in running any unRAID Add Ons (see here)? If so, which ones? Be specific. I haven't used unRAID before so I don't know what I really need. Do you want to run green/low power drives or faster 7200 rpm drives? If you don't have a specific need for 7200 rpm drives, then choose green drives. What do you plan to run for hard drives? unRAID and VMs on SSDs, then regular HDs to hold all the generated data. I don't know what are good models of SSDs and HDs for our use cases. Do you have any spare parts laying around that you would like to apply towards your build? This includes drives. No If you already have parts in mind, please oh pretty please post links to them so that we don't have to look them up. Rough parts list, only requirement is the 4 X Titan V. Everything else can be modified: https://pcpartpicker.com/list/QQc9NQ I found a chassis that looks great for GPUs but I don't see a good spot for SSD/HDs: https://www.newegg.com/Product/Product.aspx?Item=N82E16811147277 Is it possible to do this with only unRAID? What are some good alternatives? I appreciate any advice and suggestions. Thanks!
May 14, 20188 yr I have a 2 titanV and 2 Xeon CPU setup for similar use. (no NAS though). I use unraid with ease with one gpu on a linux vm and one on a windows.. I also sometimes spin new clones of the vms without the gpu (if I need cpu only). it was quite easy to setup. At the mement Im setting up unraid on a k80 boxes I have.
May 14, 20188 yr Author 1 hour ago, daemonix said: I have a 2 titanV and 2 Xeon CPU setup for similar use. (no NAS though). I use unraid with ease with one gpu on a linux vm and one on a windows.. I also sometimes spin new clones of the vms without the gpu (if I need cpu only). it was quite easy to setup. At the mement Im setting up unraid on a k80 boxes I have. Thanks for your input, I appreciate it! Have you encountered any issues at all with unRAID when initially trying to allocate the primary Titan V to a VM? I read online that this could be a problem for 1080TI cards on CPUs without integrated graphics (Xeon CPUs). Did you store all your VM data on a single SSD/HD with partitions or have one physical drive for each VM? I guess this is all preference but I am interested in what you have done ? I'll be on the lookout for your k80 build, please let us know how that goes
May 14, 20188 yr SuperMicro MB has an extra VGA onboard (most of the PRO MBs seem to have one). Giving the GPU to the VM is all simple GUI setting, nothing extra.. no VBIOS... nothing. Just remember to get some "dummy" HDMI dongles to plug. (got mine from Amazon UK) K80s are a bit different as they dont have VGA port. So Im having some treble with using the VNC/qEMU one as primary and the K80 as CUDA only. EDIT: btw get some CPU with higher base clock. My older build have [email protected] and the new is [email protected] .. if you are not super CPU parallel in you coding (custom Cpp etc) go for the clock speed. V4s are slow compare to the new Scalable. My K80 are on V4, now Im on Scalable. Edited May 14, 20188 yr by daemonix
May 14, 20188 yr Author I see, it is great to hear that unRAID was easy to use and did not give you any problems. Wow, thank you for the tip about the dummy HDMI dongles! I didn't see anyone mention those before in their VM builds so that is really helpful. Yes that sounds like a good idea, I'll choose a Xeon with higher clock speed since I don't think more cores will be beneficial for my uses. If you're interested in a finished 4x Titan V GPU build, you can google 'hardforum prototype 4x titan v'. A user named venturi has put together a really amazing system with 4 Titan V cards but I am not sure if he is using unRAID.
Archived
This topic is now archived and is closed to further replies.