Some advice needed on a full SSD + VM build

[dmk]

I'm rebuilding a machine I have because I want to run two VMs with passthrough.

 

Current hardware:

 

Motherboard: SuperMicro X10SAT

CPU: Xeon E3-1275 V3 - 3.5Ghz

RAM: 32GB ECC unbuffered

PSU: 1500W Silverstone Strider

GPU 1: GTX 980Ti

GPU 2: HD7750 or HD6570

SSDs: 4 * 120GB + 1 * 64GB

 

The passthrough VMs I want to run are OSX El Capitan as workstation and Win10 for gaming.

 

Questions:

 

1. Which ATI gpu should I use for OSX? The HD7750 or the HD6570?

2. What setup for the array/cache and/or dedicated VM drives should I use?

 

So far I've come up with these scenarios:

 

Scenario 1

Array: 3 * 120GB

VM OSX: 1 * 120GB

VM Win: 1 * 64GB

 

Scenario 2

Array: 2 * 120GB + 1 * 64GB

VM OSX: 1 * 120GB

VM Win: 1 * 120GB

 

Scenario 3

Array: 4 * 120GB

Cache: 1 * 64GB

 

Scenario 4

Array: 4 * 120GB + 1 * 64GB

 

I'm hoping running VMs from an array as in scenario 4 in combination with a virtual 10Gb network would perform reasonably. I can accept some performance loss of the drive speeds but not all the way down to hard disk speeds.

 

3. Would this work or do I need to invest in some more and/or higher capacity SSDs?

 

Thanks in progress and: Hi all, first post

[Frank1940]

I think you will need to do a bit more research. 

 

First thing is that (as I recall) SSD's are not recommended at this point for array drives.  It has to do with the way that they 'Trim' themselves up to ensure that maximum drive life and performance.  However, they are recommended for use in a Btrfs cache array where those operations are supported.  As I seem to recall, this arrangement does provide parity protection for the cache.

 

Basically, unRAID has been designed for the storage of LARGE qualities of data for essentially archival purposes.  The data should be written once and read often in this scenario--- store of media files is typical of this type.  It is not really suited for a transactional type database use---- think air line ticketing database as an extreme example.  (Primarily because writing is much slower than reading.  Plus the maximum write speed is limited to the speed of the data drive itself.)  Of course, many people do use unRAID for uses that don't exactly fit these two extremes and they have accepted the shortcomings of unRAID. 

[dmk]

Thanks for the reply.

 

I've done the research, this is not my first go around The purpose for this build is having a workstation and a gaming machine all in one, with the added benefit of docker and securing the data I work with. Hence the choice for UnRaid. I'm not interested in storing large quantities of data.

 

Besides that I don't have HDs, only SSDs. My HDs are in a Synology, which is mainly used for backups.

 

As I understand it, it is possible to run nightly TRIMs.

[CHBMB]

So you essentially want to use Unraid as a hypervisor? 

 

What are you planning on storing on the array drives and are you going to use a parity disk?  Which I think may not be a great idea, can't explain why, just doesn't seem quite right given what has been said above...

 

I'm inclined towards options 1 or 2 depending on how much space you want for the Windows VM..

 

As for the gfx cards, archedraft's thread on OSX VM would be a good place to ask.  I know the 6450 is compatible and that ATI is preferable to Nvidia, but that's about all I know.

 

 

 

 

[garycase]

For what you want to do, I'd assign the 64GB SSD as the only array drive, no parity, and the 4 120GB drives as a btrfs cache pool.

 

If 3 120GB drives in the cache pool will provide enough storage, you could use one of them as a parity drive ... but I suspect all of your data is going to be on the cache, and the pool is fault tolerant, so you really aren't even going to use the array for storage.

 

[dmk]

So @garycase, what you're saying is to basically use btrfs raid5 with 4 120GB SSDs to create a 360GB cache pool and run the VMs from the cache?

Would you consider btrfs raid5 as safe right now? Any considerations I need to make regarding the write hole issue?

 

Just remembered I have a few 2.5" HDDs (1TB + 750GB + 500GB). I might as well use those as an array to move games to when not playing them, with the 1TB drive as parity drive. Sounds about right.

 

I wrote "I'm not interested in storing large quantities of data." a few posts back. What I mean is I don't have a need for tens of TB of storage, the only reason I can think of is storing loads of video and music files but services like Netflix and Spotify do that for me. Stuff I would store are photo's, documents, git repos, etc. To be honest even that isn't more than 150GB. Games would add to it though.

 

 

[garycase]

Yes, I'd just store all the VM's on the cache.

 

Using your spare HDD's as a protected array would be a good use for them as well.

 

[dmk]

Thanks @garycase

 

Final question: does the cache pool support btrfs raid-5 or only btrfs raid-1?

[Frank1940]

Stuff I would store are photo's, documents, git repos, etc. To be honest even that isn't more than 150GB. Games would add to it though.

 

You will be amazed at how fast 'stuff' accumulates.  I can remember when I first started out, I thought, I would NEVER fill a 3TB RAID5 array...  (Look at my signature for what I have now!)

[dmk]

Well, I've accumulated about 150GB in the last 15 years. I've had a Freenas zfs system but I just wasn't using it. It was way more time efficient to just buy a Synology for what I needed it for in the first place: backups.

 

Right now my interest is running a passthrough OSX VM with (relatively safe) data protection, the Windows VM I care less about and I consider it as a throwaway VM just for games. Having the protected data array as a user share to save games and my 150GB of data to is fine for me. If I need more space I'll delete some games, no biggie, I can always download them again. The 150GB of important data is also synchronized to a rMBP, a Synology in RAID1 and an offsite backup.

[dmk]

Well, that was easy.

 

Had to use ACS override for the passthrough to work but after that, easy peasy, Windows is installed. Even the Nvidia driver installed without issues. Color me impressed! Thanks everyone, much appreciated!

 

Now to install OSX...

[peter_sm]

 

Now to install OSX...

Please let us know it you are successful to install OSX, we have found some issue on 10.11.2 iso where we don't get into the install GUI

//Peter

[dmk]

Please let us know it you are successful to install OSX, we have found some issue on 10.11.2 iso where we don't get into the install GUI

I installed 10.11.1 as that was the one already in my Applications folder. It didn't get to the GUI as well but after a reboot it did. Installed fine after that. HD7750 passthrough also working, at least to the boot screen. I need to add keyboard/mouse to get past it. Maybe later tonight.

 

For a Christmas project in between family obligations everything is going very well so far.

[peter_sm]

Did You reboot UnRaid to get 10.11.1 to the installer GUI?

[dmk]

No, rebooted the VM.

[dmk]

I've got Win8.1 and OSX 10.11.1 running both.

 

USB and GPU passthrough working on both but OSX has issues with the GPU.

 

HD7750 results in garbled display, can't find a solution anywhere that works with enoch. Tried to figure out how to change to clover but I'm lost here.

 

HD6570 does work but is stuck to 1024x768. Not sure how to fix it.

[dmk]

Upgraded 10.11.1 to 10.11.2, no issues but GPU still stuck at 1024x768 (somehow also thinks it's a projector).

 

Did a disk speed test from OSX, pretty good although the 4K blocks are relatively slow.

 

Results 800.30

System Info

Xbench Version 1.3

System Version 10.11.2 (15C50)

Physical RAM 4096 MB

Model Macmini2,1

Drive Type QEMU HARDDISK

Disk Test 800.30

Sequential 501.97

Uncached Write 2103.98 1291.81 MB/sec [4K blocks]

Uncached Write 1970.72 1115.03 MB/sec [256K blocks]

Uncached Read 151.69 44.39 MB/sec [4K blocks]

Uncached Read 2541.54 1277.36 MB/sec [256K blocks]

Random 1972.62

Uncached Write 688.20 72.85 MB/sec [4K blocks]

Uncached Write 3760.43 1203.85 MB/sec [256K blocks]

Uncached Read 5612.70 39.77 MB/sec [4K blocks]

Uncached Read 7657.28 1420.86 MB/sec [256K blocks]

 

[dmk]

The Radeon GPUs aren't working out. I'm selling them to get some cash for a GTX 960, that would also be powerful enough for OSX to drive a 4K monitor and two 1600x1200 side monitors which I currently drive from my rMBP with the help of a Kensington USB3 docking station via DisplayLink.

[rhinoman]

To the OP. I like the idea of what you are doing here and have been contemplating doing someting similar myself so just wanted to throw another suggestion into the ring for discussion.

 

Why not just set up an SSD array using BTRFS as the file system or at least for one drive for docker. One of your 120gb drives as parity. Drive one for dockers and appdata, drive 2 for windows vm, drive 3 for osx vm, the other for data.

 

My logic for this is that it gives the benefit of unraid, the parity protection and the ability to easily resize disks to meet new needs. Keeping the vm's to seperate disks means minimal overlap of logic and disk writes. It seems in your senario you may have some dockers that run all the time with the os's largely in either/or use.

 

Can the parity drive keep up is my only question and maybe worth having a newer model with the best in/out speeds for this? With an individual machine VM running I'm sure it will be fine as its no worse than raid1 protection. Add the dockers running to the equation and there wil be possibly some slowdown but only if running a downloader that is wrting all the time which is probably not the point here.

[dmk]

Thanks for the reply rhinoman,

 

I have it set up this way now:

 

• BTRFS cache pool of 4 * 120GB SSDs (see performance stats a few posts back).
  
• Storage array of 1TB (parity), 750GB, 500GB.
  

 

The cache pool is in the default as UnRaid sets it up in BTRFS RAID1 (which is somewhat different than regular RAID1).

 

I run the VMs from the cache pool and while the 4K performance is less than perfect the real world performance is great.

 

[dmk]

I've ditched the Radeon cards and installed a GTX 960 for OSX.

 

Now I can't get the Nvidia webdriver to work, probably due to the model identifier of the VM is iMac (Summer 2001). I've asked a question about how to change the model identifier in the Mac OS X 10.11 El Capitan – VM on unRAID guide:

https://lime-technology.com/forum/index.php?topic=43816.msg429679#msg429679

 

 

[dmk]

So as a recap...

 

I've got the Nvidia webdriver working by adding org.chameleon.boot.plist and smbios.plist to the /Extra folder on the Mac HD.

 

 

 

This is the org.chameleon.boot.plist I used:

<?xml version="1.0" encoding="UTF-8"?>  
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">  
<plist version="1.0">  
<dict>  
    <key>Timeout</key>
    <string>5</string>
    <key>EthernetBuiltIn</key>
    <string>Yes</string>
    <key>PCIRootUID</key>
    <string>1</string>
    <key>KernelBooter_kexts</key>
    <string>Yes</string>
    <key>nvda_drv</key>
    <string>1</string>
</dict>

 

And this is the smbios.plist:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>SMbiosdate</key>
    <string>10/07/2013</string>
    <key>SMbiosvendor</key>
    <string>Apple Inc.</string>
    <key>SMbiosversion</key>
    <string>IM141.88Z.0118.B00.1309031249</string>
    <key>SMboardproduct</key>
    <string>Mac-27ADBB7B4CEE8E61</string>
    <key>SMfamily</key>
    <string>iMac</string>
    <key>SMmanufacturer</key>
    <string>Apple Inc.</string>
    <key>SMproductname</key>
    <string>iMac14,2</string>
    <key>SMserial</key>
    <string>C02LH068FLHH</string>
    <key>SMsystemversion</key>
    <string>1.0</string>
</dict>
</plist>?

 

With <key>nvda_drv</key><string>1</string> in org.chameleon.boot.plist it still won't boot to the Nvidia webdriver so I need to add nvda_drv=1 as a boot option every time I boot the VM. This is annoying but at least it works.

 

Real world performance is awesome. Compared to my late 2014 Core i7 MacBook Pro Retina it feels much more responsive and snappier.

 

Being able to work on my "HackVM" while simultaneously run Windows to test games and VR applications on my Oculus is great. No more rebooting.