Hello Lime Technology, and Hello Community

I am a fellow Hardware Enthusiast, and since Long i have been looking for a way to use my Server Computer in a "Multi seat mode", that is a single Desktop PC to power two stations.

 

I have tried until now Aster, and it Kind of works, but in actuality it has Limitation, since everything uses the same Windows, with multiple accounts and i cant Play a game on the Internet with my friend from the two separate stations, only different ones, which defeats the purpose of multiseating.

 

Then i came across i dont know how to the Linus tech Tips Video "2 Gamers One CPU" and my god, that was exactly what i was aiming to achieve.

 

However the Situation is not as simple as it might seem, even if i do follow the steps described, step by step, as i have a Special Configuration, which is one might argue, a bit of a rarity among Computer users.

 

Digging deeper i learned all is possible thanks to a Custom OS, the UNRAID OS, that you guys provide, and apart from creating the 2 stations i always wanted, at its main core, UNRAID is an OS that allows for mix and maching HDD SSD to create that what you Need concerning data storage, reliability and Speed. Which i also wanted to incorporate somehow in my build, due to the number of 3.5" HDD and 2.5" SSD the whole Desktop case is suppose to incorporate. Thus there are Major Questions that Need to be answered first, before i even start to attempt this on my own, due to the nature of the Situation. Before i list my questions, let me tell you about my Hardware configuration.

 

1.Motherboard Gigabyte MZ31 AR0, latest bios F06, Support for AMD EPYC single socket CPUs, out of which there are only 3 models available.

http://b2b.gigabyte.com/Server-Motherboard/MZ31-AR0-rev-10#ov

2)CPU: AMD Epyc 7551P, 32core CPU with hyperthreading

3)Ram, 16x4 Memory sticks, for a total of 64gb ecc registered ddr4 Memory, Motherboard has 16 Slots.

4)Onboard Video Card, with VGA Output, i can Output htis through an Adapter to a HDMI capable Monitor. Plus 2 Radeon Vega Frontier Edition Watercooled variants

5)1 NVME SSD on motherboard, second NVME ssd through Expansion Card.

6)PCI express 4x Card with 4 USB 3.0 ports

6)1 WD Gold 10 TB hdd

7)2 SATA CDROMs.

8)As Network there are 2 SFP+ 10 gbit ports, and one Maintance IPMI rj45 port. all three can be connected to router.

 

The Motherboard has only 2 USB 3.0 ports and 2 USB 2.0 ports at the back, the internal USB 2.0/3.0 is unusable since the Video Cards are so Long, so i am left with only the ports at the back to connect the peripherals.

 

1)USB 2.0 for UNRAID Memory stick, the other USB 2.0 for the Adapter of the VGA to HDMI of the onboard graphics, Needs power USB

2)One USB 3.0 for a external USB hub, placed outside of case. the other one connects to an internal USB 3.0 HUB that drives front USB, basically replaces the internal USB 3.0 Header.

 

Peripherals are connected to the external USB 3.0 hub, and to the PCI express Card that Outputs the 4 USB 3.0 ports.

 

Now here are my questions, regarding stuff that i do not yet understand.

1)Does my CPU AMD Epyc 7551P 32 core CPU Support IOMMU or AMD-Vi ?. From what i've read, without this perk, i cannot do what Linus did in the Video, with my two discrete Video Cards. I know for a fact, that it does have Support for Virtualization, AMD-V. I find it hard to believe that the latest AMD high end Server CPU doesnt Support IOMMU/AMD-Vi, if that is the case, my dreams are shattered, and None of this works.

 

2)I didnt get very much the UNRAID disk System, and how it can be set up. I did read, there is Support for max 6 parity drives, 24 HDD, 6 Cache drives, but then i've read that the PRO Version of the Software supports unlimited devices., so which one is it ?

3)In my example, how do i go about using my SSDs and HDD ?. What i was imagining, was that i would Need a two partitions for Windows, and a storage for the 10 TB hdd. But then i've read about the SSD Cache pool.

Can i create a 768GB SSD Cache pool, and use for instance 160gb for a Windows partion on machine 1, 160gb for Windows Partition of machine 2, and 448GB for a gaming Partition ? or is this not how it is suppose to work ? Wanted the Windows and games on fast drives, SSD, and storage on HDD.

What if i have 8 SATA SSD times 4 TB each, can i create a pool of 32 GB SSD Cache, out of which, 1TB for each Windows Partition, 18 TB gaming Partition, and 12 TB Caching Pool, would that work, or the cache can consist of Maximum 6 drives ?

what happens if i mix NVME ssds in the Thing, is it possible at all, because These are connected through PCI express...

3)Then the storage Array. Say i have 10 3.5" hdd, 12 TB each. Is there an Advantage to Setting up more than 1 Drive for Parity ? Could i use one for parity, and 9 for raid 5/6 OR 2 for Parity, and 8 for RAID 5/6. Isnt having a RAID 6 Array allready going to provide redundancy, why would i Need parity drive separately ?

 

I only got 16 Slots for SATA on Motherboard, what if i want to connect more than 16 drives ?

To be honest i was planning for:

2 SATA SSDs, 2x3.5"HDD, 4x2.5"SSD, thats 8 drives on the Motherboard.

Another 8 drives, either 8x3.5" HDD, or 8x2.5" SSD on the Motherboard, filling up the 16 Motherboard Slots.

And i would still have 8 Slots left in the case, they would Need an Expansion Card. But having drives connected to a raid Expansion Card, would those be able to be used in the UNRAID Array ? or every PCI Card is suppose to be designated to each Virtual Machine ? because if that is the case, then the drives connected through the PCI express Raid Card, would be assigned to either one or the other Virtual machine, and not be available to both, when creating the UnRaid Array. Or am i wrong ?

 

The same question applies to the internal PCI express USB 3.0 Card for instance. If the ports on it, cannot be shared to both the Virtual machines, then i would Need two of them, one for each Virtual machine. And two raid Cards, one for each virtual machine.

 

4)Since my Audio is given from an external USB Card, i cannot share it between virtual machine. Could the HDMI Sound pastrough from the videocards be shared also through Display ports ? Because my Monitors are connected through their displa ports.The Monitors have AUDIO passthrough, and each Video Card could Output ist own Sound to the Monitors, and in the Monitors i could connect the Speakers, if that were the case. If this Option is available only through HDMI, then i would Need to use HDMI, but i only get 144hz, on the Monitors when using the Display port.

 

5)If i were to install 512GB of ram on the Motherboard, would each virtual machine be able to use 256gb, or does the Memory stick OS Need ist own RAM Memory Designation, for instance 250GB per machine of ram, and 12GB Memory ram for the OS on the Memory stick. Is this how it is suppose to work, or not. If so, is 12gb enough, or more should be enabled ?

 

 

I Need to know how this stuff is suppose to work, before i even try it, esspecially since i plan to buy more drives, i Need to know if raid Controllers Need to be bought, a second PCI express USB 3.0 Adapter, or if how the Sound works, would i Need a second PCI express soundCard for the second Virtual machine if the Sound through Display port doesnt work ? Or If i should give up the NVME ssds alltogether and go with only SATA ones, etc.

 

I am sorry for such the lengthy POST, but you guys are the only one that can clear this questions for me, since you build the OS, and know what is what. And ofcourse to all knowing community, as i would like to call it.

 

Apart from that i have data on the 10TB hdd, and i cannot even start to try setting UP UnRAID OS, without Messing it up becasue from what i understand it, the drives will be formated, I will probably Need to buy more HDD, set the storage Array, and see afterwards how i will be able to copy the data from this 10Tb Hdd on the ARRAY ?

 

If my CPU cant do IOMMU, ist all for nothing ! Isnt it ?

I was thinking on using Windows Server 2016 for the two virtual machines, i would Need to purchase two 16 core licence for each virtual machine, since the Motherboard doesnt officially suport Windows 10, which im running now. What would you advise when you take this into conideration, would i be better of, or not ?. Because i allready have a Windows 10 licence, i would Need to get only one extra. Two 16 core Windows Server 2016 Standard is Close to 800 euros. If it would be worth it, i would do it. But i Need to know first. Probably will try Windows Server 2016 as a Trial see how everything goes.

 

Hopefully the questions will be answered and i will then know what is what and what i need to buy, and what i need to sell.

 

 

 

 

 

I'll let others tackle some of this, especially the VM stuff. I suspect you may have some assumptions and other information you have gotten elsewhere. The LimeTech website has a good overview of unRAID: https://lime-technology.com/ After you learn a little more you might decide to reconsider some aspects of your proposed design.

 

5 minutes ago, Bytales said:

2)I didnt get very much the UNRAID disk System, and how it can be set up. I did read, there is Support for max 6 parity drives, 24 HDD, 6 Cache drives, but then i've read that the PRO Version of the Software supports unlimited devices., so which one is it ?

Don't know what you were reading, the max parity drives is 2.

 

8 minutes ago, Bytales said:

3)Then the storage Array. Say i have 10 3.5" hdd, 12 TB each. Is there an Advantage to Setting up more than 1 Drive for Parity ? Could i use one for parity, and 9 for raid 5/6 OR 2 for Parity, and 8 for RAID 5/6. Isnt having a RAID 6 Array allready going to provide redundancy, why would i Need parity drive separately ?

 

unRAID is not RAID. See the LimeTech link I gave earlier.

 

1) All AMD Zen-based processors support both AMD-V and AMD-Vi.

 

5) unRAID needs some memory for itself. 4 to 8 GB should suffice.

What would be the Advantage of having two parity drives ? Two drive must be used if say you have 8 tb drives in unraid, and 2x4 tb drives for parity ?

Or can you use 2x12tb for parity, and 8x12 TB for data ?

So basically, there is no talk of Raid 5 or 6, that is being implemented, but the Custom Raid of the Unraid OS.

 

What i didnt get was how the cache pool works. Does it act like a cache per se, being completly trasnparent to the user ? Or you could partition a part of it, to use as a system drive, or gaming partition ? Or there wouldnt be any need for it, such fast "system partitions", and "gaming partitions" can be created out of the Data drives, giving you basically lots of space, and fastness of a ssd, when you have proper space in the cache pool ?

 

I could pool my 256 and 512 gb NVME ssds, for a 768 ssd cache pool, and use lets say for the sake of example, 5x12tb drives, one for parity, and i would be left with 4x12tb as DATA storage ?

This 48 TB, can be then partitioned for each Virtual Machine, like give 20 TB to each, and 8 TB as common space, could that be done ?

 

Good to know that the EPYC CPU support AMD-Vi, now i can start considering using UnRAID. just need further planing.

 

One worry i had, say i made such example. 1x12tb for parity, 4x12tb for DATA. Could i then ADD anotherr 10 TB to the data, without formating it, just adding it to the DATA Pool ?

You must let unRAID completely manage any drives in the parity array or you will break parity, including partitioning and formatting. For your VMs, you can create virtual disks and use those since these are basically just files as far as unRAID is concerned.

Does anyone know if the NVME SSDs can be used for the SSD Cache ?

Likewise, can one use or create a full Data Array of 2.5" sata SSDs, or the Data Array should only be done with HDD ?

Or could one create two Data Arrays, a SSD Data Array and a HDD Data Array ?

Or it wouldnt make any sence in implementing a SSD Data Array ?

 

What would happen if i were to have 4SSD x7.63 TB in the SSD Cache pool, and 4x6TB hdd in the DATA Array (plus one 6tb hdd for parity) ? Would using the shares on the Data Array always run at SSD Speeds ? since the SSD Cache pool is larger than the HDD Data Array ?

NVMe SSDs can be used for the cache pool.

SSDs do work in the array but you can't use trim on them, so their use is unsupported. I haven't tried myself. At least one person built an array using only SSDs for test purposes. I don't know if it is of practical use. I want storage capacity rather than speed. I use hard disks.

You can create one data array, one cache pool and any disks remaining unassigned can be mounted individually using the Unassigned Devices plugin.

I use magnetic HDDs for my data array and SATA SSDs for my cache pool, as do most people.

 

I don't understand the rest of your questions.

So i could live the Data Array without a SSD Cache, and use the NVME SSDs individualy.

What i dont get is the following.

If i were to use the NVME SSDs for Cache, would the whole Datta Array behave like it were a SSD ? Because like so it would be worth it. I would be sacrifing the NVME SSDs, and would gain a huge Data Array with NVME SSD like Speeds. I would create Windows partitions out of the DATA Array, and they would load as fast as the NVME SSDs, because they would be cached. And everything going on with the DATA Array would benefit from the NVME SSD Speeds!

 

Am i correct in my reasoning ?

If PCI express NVME SSDs can be used for Cache, would i be free to assume that any PCI express SSD could be used for Cache ?

How about building a DATA Array with SSD like Speeds using only SSD. Normal Raid can also be done on SSDs.

But isnt this what the Cache pool is suppose to do ? How much Cache SSD capacity would you Need to be ok compared to HDD Data Array ?

The user share caching feature is designed to allow faster writes to a user share, since writing to the parity array is slower than to a single disk. At a scheduled time, the cached data is moved to the parity array. Read speed is always at the speed of the single disk, since any file is stored completely on a single disk.

 

People also use the cache pool for apps (dockers), VMs, and anything else they want. So you could have SSDs in cache and have some user shares stay on cache so you get the read/write speed of SSD. Most people only have 1GB ethernet and normal HDD read speed is good enough for that.

 

There is some versatility to the ways in which user shares use the cache disk, but there is nothing exactly "smart" about it, as in caching things for reading as needed. Here is a good explanation of the results of different user share cache settings:

 

https://lime-technology.com/forums/topic/46802-faq-for-unraid-v6/?page=2#comment-537383

 

So basically i can make user Shares, as in partitions, Folders, out of the Cache pool, and leave some for Caching as well. Am i right ?

Like if i have 4x4tb SSD in Cache pool, i could assign 1TB x2 for two Windows partitions, 8TB for a gaming Partition, 2Tb for a work Partition, and still have 4TB for the Caching pool ?Right ?

Also i could make out of the NVME SSD, a share or Partition that is standalone, right ?

If that is so, that is very important info that will help me plan the future frive buys.

Not partitions, but you can set up cache-only shares. So yes, essentially you're right.

 

Again, not partitions, but the cache pool is a very good place to store disk images used by VMs. Whatever space is left empty can be used for caching writes to the main storage array. With four SSDs in the cache pool I would be considering providing it with some redundancy. BTRFS RAID level 10 comes to mind, though that would cost you half of your capacity.

 

You could leave an NVMe SSD unassigned (meaning not a member of the main storage array, and not a member of the cache pool either) and format and mount it using the Unassigned Devices plugin.

Or i should get SSD with high Endurance Rating that should work for 20 years. the Thing is, how will i migrate, if i Need to Change the SSDs and HDD, that i havent considered, because everything is suppose to be connected to all the 16 Sata ports of the Motherboard.

 

i could get 4x3.84 tb ssds, with 17 PT endurance, or 4x7.68TB  SSDs with 5 PT endurance and raid them, in BTFS RAID as you said, if it would be possible to raid them from the UNRAID OS, inside the Caching pool.

 

would probably go for the 17 PT endurances SSD, will probably last me a life time, but what would i do, when i Need to migrate the SSD Cache pool to a new one ? would Need some sort of external backup drive, right ? or is somehow possible without external drives, because all 16 sata Slots will be occupied, as i plan to occupy all of them.

 

Could These Arrays (SSD Cache pool, and Data storage Array) be configured off of drives attached to PCI express Raid Controllers as JBOD set up ?

 

i would probably backup the whole Cache Array on the Data Array, and then create a new Cache Array, and new capacious Shares, and copy the cloned data to the new bigger Shares. Right ?

 

The next Problem would be then migrating data from 10x12 TB Data Array (excluding parity) thats 120 TB. Would Need an external backup worth of 120 TB and copy everthing there, or, i couldnt imagine how in other way ?!

 

Hopefully will have 256 TB Memory sticks by the time i Need to Change my Data Array, and copy everything on the Memory stick.

Edited March 5, 20188 yr by Bytales

Do i Need to asign a parity disk at all, or can i live it out for now ?

SSDs fail for more reasons than just wearing out. If you need more SATA ports you could add an HBA - not a RAID controller, it needs to be in JBOD mode.

 

When you set up a cache pool the default configuration is BTRFS RAID level 1. You can easily change this to level 10 if you want to.

 

There is a procedure for upgrading your cache pool. You can search the board for it.

 

You could import the data one disk at a time?

 

256 TB memory sticks? What?

No. You don't need to use a parity disk but why wouldn't you? It seems foolish to me to build such a big budget server and not use parity. In fact, I'd use two parity disks and I'd keep a spare one on the shelf.

If you have finished with the overview from the LimeTech website link I gave earlier you might find some more answers in the V6 FAQ here:

 

https://lime-technology.com/forums/topic/46802-faq-for-unraid-v6/?page=2&tab=comments#comment-554741