TL;DR - It just works, and it works fine. Epyc is a great platform for UnRaid, virtualizing the Hackintosh was challenging, and successful at the end.
Why Epyc (and not Ryzen / Threadripper)
Because I wanted more PCIe lanes and have a choice of CPUs with less than 180W TDP, to build a decent host for a virtual Hackintosh.
UnRaid OS: 6.10 RC2
Case: Fractal Design Define 7
PSU: Be Quiet! Straight Power 550W
Board: AsRockrack RomeD8-2T Full ATX w/ Bios 3.20; latest version as of 2021/8
CPU: AMD Epyc 7232P (8C/16T, 120W, Zen 2)
Case Cooling: 2x Arctic P14 PWM fans in front to cool HDDs, 1x at the bottom of Case; fans are ultra silent
CPU Cooling: BeQuiet! Silent Sloop 2 360 w/ 2x Noctua NF-S12B redux-1200 PWM, 1x NF-S12B redux-700
Memory: 64 GB Registed ECC RDIMMs (2x32 GB) Kingston KSM32RD4/32MEI @ 3200Mhz (Per Memory QVL)
Data disks: 3x 4TB WD40EFRX + 1x 4TB WD40EFRX for Parity (all same disks, same size)
Cache 0: 2x 512GB Transcend MTE220S NVME SSDs Raid 0, 1TB cache drive specifically for backup share
Cache 1: 4x 960GB Corsair MP510 NVME SSDs Raid 1+0. 1,92TB cache drive, set up with ASUS Hyper M.2 in PCIE X16 Slot (BIOS PCIe Slot 1 Bifurcation config: 4x4x4x4x; primary purpose is for appdata and VM Vdisks)
GPU 1: MSI RX560 Aero ITX (To be used in Hackintosh once figured out how to)
Hackintosh VM specific software / hardware (passed through); Big thanks to @RiDDiX & @ghost82 for your support!
GPU 2: Nvidia GT710 (Passive, passed through to macOS VM)
USB PCIe Controller: Inateck RedComets U26 (For Keyboard & Mouse; USB Sticks)
WIFI & BT: Fenvi T919 WIFI / BT PCIe Controller, internally connected to Inateck controller (card provides Wifi & Bluetooth for macOS VM to enable Airplay, Airdrop etc.)
Peripherals: HDMI & USB over Cat extender (UnRaid hosts sits in a closet, 10m away from actual desk)
MX Keys for Business + Logi Bolt / MX Anywhere 3 + Logi Unify
Plus approx 15 Docker (Nextcloud, Photoprism, Pihole etc.) and two Windows 11 VMs (Insider + Latest Public release)
Switching from the Ryzen platform to the new board, CPU and Memory went smooth. Just replaced Board, CPU, Memory and cooling, plugged in the USB stick and booted. At the beginning of the build it hasn’t been clear to me that I had to buy registered memory (RDIMMs), which I did at the end after mailing with the support. The board does not boot with standard ECC memory which I have been using for the Ryzen (Debug LED gets stuck at 46). At the end I purchased the memory based on the RomeD8-2T QVL.
As you can see, it was a full swap. Old hardware has been sold.
Cooling - I picked the Silent Loop 2 360 and replaced the stock fans with Noctua models, 2 PWMs and one fixed 700RPM fan. The cooling is exceptional quiet, the pump isn’t noticeable at all. To mount the pump unit on the SP3 socket I used the TRX4 bracket which bequiet sells. The original mounting bracket has to be pulled out of the pump (slides out) and the new bracket can be slided in - a 5 second job and the cooling is SP3 compatible. I used the stock by termalpaste that comes with the silent loop.
I also removed the rear fan since I do not see a need for it. Air is pulled into the case from front and bottom and pushed out at the top through the radiator. I am aware that this might cause slightly higher temps, but that’s no problem for me.
Radiator - The Radiator fits well into the case, however in combination with this board and both ATX & 12V Power Connectors I had to make compromises with the cabling. It still looks good, but I think I had to bend the cables a bit to much. The radiator in combination with the fans is basically to low and it is almost impossible to service the memory, cabling, cooling etc. once everything is fitted and mounted. Hence I‘ll get a Define XL, which has more head room above the board.
Everything fits nicely into the case, except when one wants to connect the case header (Power, Reset etc.) at the bottom of the board. Since the Pins are angled by 90° it is rather difficult to connect the header , see photo. It worked out at the end, but as written earlier I will swap the case with a Define XL in a few weeks to address these space limits.
The onboard ASpeed VGA allows a headless setup and the board has a remote management interface, so Bios adjustments etc. can be done remotely. The BIOS is well structured but does not provide (at least I didn’t find it yet) any overclocking / under-volting options - The X570D4U does. What’s cool is that every PCI slot can be configured and in general terms I‘d say the BIOS allows tailoring the config to one‘s needs. I had to enable IOMMU, enabled Bifurcation for the Asus Adapter card and set the onboard VGA to standard. That was it more or less in the BIOS (for now). BIOS photos in the attachment section.
Once UnRaid was running with all the additional cards I noticed that 1 - 2 CPU cores constantly peaked at 100%, causing 10W higher draw and glitches in the VMs (Audio was hanging sometimes).
I pulled the cards and swapped them back and forth, at the end PCIe 6 (2nd slot from top) turned out to trigger this behavior, once it was occupied with a card like the Inateck or T919. So I adjusted the card setup and so far everything works fine. The slot is now covered by the GPU cooler. All additional cards, except the Asus Hyper M.2 adapter, are bind to VFIO and passed through the Hackintosh VM resp. the RX560 to one Win VM.
One of the main reason to move from Ryzen to Epyc was also to have more flex for the hardware passthrough for the VMs. Now that I have enough lanes and slots available it was easy to add a GPU, WIFI, BT and a USB controller for the peripherials so the virtual Mac acts close to a physical.
It took a few attempts to get the VM to boot (PCIe config was often wrong) and I must admit that getting a Hackintosh to work isn’t simple, but it is obviously doable.
My macOS VM is a MacPro 7,1 (2019) with a Penryn CPU and the basic kexts. I‘d say nothing fancy, but the process was much more complicated than installing & configuring a Linux or Win host.
Up and running / Notes & Todos :
Idle draw is between 93 - 100W
CPU Temperature is 37-39°C when idling, haven’t stressed the CPU yet.
SSDs 34 - 35°C (onboard slots), 4 NVMEs in the Asus Adapter 38 - 40°C
RX560 fan seems to be louder when GPU is bind to VFIO (have to verify, could be also due to higher temps in the closet)
Case will be replaced by a Define XL to have more head room above the board to enable better cabling, allow less bending of the cables. And to allow connecting the case header at the bottom of the board without having to stress the header physically.
Replace the two Transcend SSDs with Phison based PCIe 4 SSDs to enable stable passthrough of the SSDs to macOS and a bare metal installation of macOS. Right now I am using a vdisk for the OS on the Raid 0+1 cache disk. Works fine, no glitches however I prefer bare metal installation.
Apparently macOS does not boot (install yes) from the Transcend SSDs.
Get the RX560 working with BigSur. I can pass the GPU to the VM, but the boot process gets stuck once the GPU is being initialized. Not a big deal atm, because I can use the GT710 which however only does 30hz at 3440x1440. Works, but refresh rates could be better. Passing the RX560 to the Win 11 VM works flawless.
Add more Memory (+64GB)
Maybe swap the 3.5“ disk with 2.5“ disks. I find the WD Reds rather noisy and I don’t need larger drives. Most of my UnRaid action is happening on the cache drive anyhow and 3x 5TB (max. 2.5“ disk size) + Parity works for me and there is another MiniSAS connector on the board waiting to serve.
Replace the 7232P CPU with a 7313P - 8 more cores and so much more L3 cache with a slighty higher TDP (155W vs. 120W)
Maybe replace the Arctic case fans with Noctua PWM or 700RPM Redux to gain even more silence & optics.
Was it worth? Definitely. I am happy with the setup and with a few tweaks (Case, disks, GPU fix for macOS) this should become a perfect UnRaid setup soon.
… To be extended and edited over time. Hope it helps others and inspires your builds.