AMD EPYC | Watercooled | ASROCK RomeD8-2T | Virtualized Hackintosh


Recommended Posts

TL;DR - It just works, and it works fine. Epyc is a great platform for UnRaid, virtualizing the Hackintosh was challenging, and successful at the end.

 

5D4B1A56-C8BC-4666-9289-8D102A8D8911.thumb.jpeg.b7ed3cf71b3a770d28def2eba6f1818a.jpeg

 

Why Epyc (and not Ryzen / Threadripper)

Because I wanted more PCIe lanes and have a choice of CPUs with less than 180W TDP, to build a decent host for a virtual Hackintosh.

 

Configuration:

  • UnRaid OS: 6.10 RC2
  • Case: Fractal Design Define 7
  • PSU: Be Quiet! Straight Power 550W
  • Board: AsRockrack RomeD8-2T Full ATX w/ Bios 3.20; latest version as of 2021/8
  • CPU: AMD Epyc 7232P (8C/16T, 120W, Zen 2)
  • Case Cooling: 2x Arctic P14 PWM fans in front to cool HDDs, 1x at the bottom of Case; fans are ultra silent
  • CPU Cooling: BeQuiet! Silent Sloop 2 360 w/ 2x Noctua NF-S12B redux-1200 PWM, 1x NF-S12B redux-700
  • Memory: 64 GB Registed ECC RDIMMs (2x32 GB) Kingston KSM32RD4/32MEI @ 3200Mhz (Per Memory QVL)
  • Data disks: 3x 4TB WD40EFRX + 1x 4TB WD40EFRX for Parity (all same disks, same size)

  • OS disk (passed through) for Hackintosh: 500GB Seagate FireCuda 520 M.2 PCIe 4.0 x4 SSD

  • UnRaid Cache 0: WD 1000GB WD SSD M.2 SN750SE specifically for backup share

  • CoolSierra EC360® PLATINUM Thermal paste for OS disk and Cache 0 NVMEs

  • UnRaid Cache 1: 4x 960GB Corsair MP510 NVME SSDs Raid 1+0. 1,92TB cache drive, set up with ASUS Hyper M.2 in PCIE X16 Slot (BIOS PCIe Slot 1 Bifurcation config: 4x4x4x4x; primary purpose is for appdata and VM Vdisks)

  • GPU 1: Gigabyte Radeon RX 460 WindForce 2X OC 4GB
  • GPU 2: Zotac GeForce GT 710 PCIe x1 (Backup GPU)

 

        Hackintosh VM specific software / hardware (passed through); Big thanks to @RiDDiX & @ghost82 for your support!

 

  • Monterey 12.0.1
  • 16GB Memory
  • 8 virtual CPU cores, Penryn
  • Physical OS disk (passed through): 500GB Seagate FireCuda 520 M.2 PCIe 4.0 x4 SSD
  • GPU 1: Gigabyte Radeon RX 460 WindForce 2X OC 4GB (passed through to macOS VM)
  • USB PCIe Controller: Inateck RedComets U26 (For Keyboard & Mouse; USB Sticks)
  • WIFI & BT: Fenvi T919 WIFI / BT passed through PCIe Controller, internally connected via USB-Header-to-USB2.0 adapter cable to passed through Inateck RedComets U26 controller (Fenvi T919 provides Wifi & Bluetooth for macOS VM to enable Airplay, Airdrop etc.)
  • Peripherals: HDMI & USB over Cat extender (UnRaid hosts sits in a closet, 10m away from actual desk)
  • MX Keys for Business + Logi Bolt / MX Anywhere 3 + Logi Unify

 

        Plus approx 15 Docker (Nextcloud, Photoprism, Pihole etc.) and two Windows 11 VMs (Insider + Latest Public release)

 

Build Process:

Switching from the Ryzen platform to the new board, CPU and Memory went smooth. Just replaced Board, CPU, Memory and cooling, plugged in the USB stick and booted. At the beginning of the build it hasn’t been clear to me that I had to buy registered memory (RDIMMs), which I did at the end after mailing with the support. The board does not boot with standard ECC memory which I have been using for the Ryzen (Debug LED gets stuck at 46). At the end I purchased the memory based on the RomeD8-2T QVL. 

 

40B2B18F-7340-49D2-B82F-5B5815E17111.thumb.jpeg.4a3287b042bf047dafafcb8f09218948.jpeg

 

A8E00653-079B-490B-88C1-603C3E51BE5E.thumb.jpeg.bfb4a17c02f729c66b5045dd231d9b7a.jpeg

 

81FE8A63-56D4-46FA-AA00-B42737B0B810.thumb.jpeg.089d474eefdc6271c413a34577a3fafb.jpeg

As you can see, it was a full swap. Old hardware has been sold. 

 

Cooling - I picked the Silent Loop 2 360 and replaced the stock fans with Noctua models, 2 PWMs and one fixed 700RPM fan. The cooling is exceptional quiet, the pump isn’t noticeable at all. To mount the pump unit on the SP3 socket I used the TRX4 bracket which bequiet sells. The original mounting bracket has to be pulled out of the pump (slides out) and the new bracket can be slided in - a 5 second job and the cooling is SP3 compatible. I used the stock by termalpaste that comes with the silent loop.

 

978BA3B7-4105-4AA1-8B66-4E2CF6F0AF4C.thumb.jpeg.2a9a61adeacd6eb8d3dcdc3cc7096275.jpeg

 

I also removed the rear fan since I do not see a need for it. Air is pulled into the case from front and bottom and pushed out at the top through the radiator. I am aware that this might cause slightly higher temps, but that’s no problem for me. 

 

Radiator - The Radiator fits well into the case, however in combination with this board and both ATX & 12V Power Connectors I had to make compromises with the cabling. It still looks good, but I think I had to bend the cables a bit to much. The radiator in combination with the fans is basically to low and it is almost impossible to service the memory, cabling, cooling etc. once everything is fitted and mounted. Hence I‘ll get a Define XL, which has more head room above the board.

 

Everything fits nicely into the case, except when one wants to connect the case header (Power, Reset etc.) at the bottom of the board. Since the Pins are angled by 90° it is rather difficult to connect the header , see photo. It worked out at the end, but as written earlier I will swap the case with a Define XL in a few weeks to address these space limits.

 

05382D42-C68D-4647-9CF6-E1217BDEAE14.thumb.jpeg.295ee93f7322c6e068610894f7540e94.jpeg

 

The onboard ASpeed VGA allows a headless setup and the board has a remote management interface, so Bios adjustments etc. can be done remotely. The BIOS is well structured but does not provide (at least I didn’t find it yet) any overclocking / under-volting options - The X570D4U does. What’s cool is that every PCI slot can be configured and in general terms I‘d say the BIOS allows tailoring the config to one‘s needs. I had to enable IOMMU, enabled Bifurcation for the Asus Adapter card and set the onboard VGA to standard. That was it more or less in the BIOS (for now). BIOS photos in the attachment section.

 

Once UnRaid was running with all the additional cards I noticed that 1 - 2 CPU cores constantly peaked at 100%, causing 10W higher draw and glitches in the VMs (Audio was hanging sometimes). 

I pulled the cards and swapped them back and forth, at the end PCIe 6 (2nd slot from top) turned out to trigger this behavior, once it was occupied with a card like the Inateck or T919. So I adjusted the card setup and so far everything works fine. The slot is now covered by the GPU cooler. All additional cards, except the Asus Hyper M.2 adapter, are bind to VFIO and passed through the Hackintosh VM resp. the RX560 to one Win VM. 

 

Hackintosh related:

One of the main reason to move from Ryzen to Epyc was also to have more flex for the hardware passthrough for the VMs. Now that I have enough lanes and slots available it was easy to add a GPU, WIFI, BT and a USB controller for the peripherials so the virtual Mac acts close to a physical. 

 

It took a few attempts to get the VM to boot (PCIe config was often wrong) and I must admit that getting a Hackintosh to work isn’t simple, but it is obviously doable. 

My macOS VM is a MacPro 7,1 (2019) with a Penryn CPU and the basic kexts. I‘d say nothing fancy, but the process was much more complicated than installing & configuring a Linux or Win host. 

 

Up and running / Notes & Todos :

  • Idle draw is between 93 - 107W, RX460 increased the draw a bit compared with GT710
  • CPU Temperature is 37-39°C when idling, haven’t stressed the CPU yet. 
  • SSDs 34 - 35°C  (onboard slots), 4 NVMEs in the Asus Adapter 38 - 40°C
  • RX560 fan seems to be louder when GPU is bind to VFIO (have to verify, could be also due to higher temps in the closet😞 RX560 does not work in this setup, cannot bot BigSur/Monterey stable to login screen, boot process fails. Hence picked a Gigabyte RX460.
  • Case will be replaced by a Define XL to have more head room above the board to enable better cabling, allow less bending of the cables. And to allow connecting the case header at the bottom of the board without having to stress the header physically. 
  • Replace the two Transcend SSDs with Phison based PCIe 4 SSDs to enable stable passthrough of the SSDs to macOS and a bare metal installation of macOS. Right now I am using a vdisk for the OS on the Raid 0+1 cache disk. Works fine, no glitches however I prefer bare metal installation. 
  • Apparently macOS does not boot (install yes) from the Transcend SSDs.
  • Get the RX560 working with BigSur. I can pass the GPU to the VM, but the boot process gets stuck once the GPU is being initialized. Not a big deal atm, because I can use the GT710 which however only does 30hz at 3440x1440. Works, but refresh rates could be better. Passing the RX560 to the Win 11 VM works flawless. Decided to go with a RX460, works fine. 
  • Add more Memory (+64GB)
  • Maybe swap the 3.5“ disk with 2.5“ disks. I find the WD Reds rather noisy and I don’t need larger drives. Most of my UnRaid action is happening on the cache drive anyhow and 3x 5TB (max. 2.5“ disk size) + Parity works for me and there is another MiniSAS connector on the board waiting to serve. 
  • Replace the 7232P CPU with a 7313P - 8 more cores and so much more L3 cache with a slighty higher TDP (155W vs. 120W)
  • Maybe replace the Arctic case fans with Noctua PWM or 700RPM Redux to gain even more silence & optics. 

 

Conclusion:

Was it worth? Definitely. I am happy with the setup and with a few tweaks (Case, disks, GPU fix for macOS) this should become a perfect UnRaid setup soon. 

 

… To be extended and edited over time. Hope it helps others and inspires your builds. :) 

 

E22BC4B4-CDBC-4C30-B7C7-233B40B555AB.jpeg

C9F2D319-8381-4706-84FD-E05545D77886.jpeg

278324D3-291D-4F4D-B51E-18B3D331AA03.jpeg

9FF30949-6735-47EA-8EA9-E70CCC1968FC.jpeg

Edited by doesntaffect
Updated hardware list
  • Like 2
Link to comment

I have remote messaging through Pushover installed, however want to tweak the alerting and auto-shutdown in the coming weeks. I can shut down the host remotely though. Also have the idea to install a water sensor / sensors in the case and connect that to the alerting. The pump is running at 2600RPM / 9V afaik, something I also want to tinker with and see how the whole monitoring can be improved. 

 

I installed the watercooling primarily to learn and get much better temps (The 3900 had +60°C constantly) and to clean up the case. And I like water cooling :)

 

Re unattended - mostly, in the sense that I don’t want to look at it constantly since it sits at 2,5m above ground, however the closet is being accessed daily. 

Edited by doesntaffect
Link to comment

Have you run tests on how long it takes the CPU to heat up with no coolant? When running unattended I'd be afraid of not getting it shut down quick enough to avoid frying something. That's the major advantage of air cooling, typically when a fan fails you still have a big ole chunk of metal to absorb the heat so the CPU doesn't skyrocket temps.

Link to comment

It's a nice setup, the only thing I don't like is the radiator.

I had 2 of them in my 2 servers to cool the xeons since at that time I was using windows for heavy computational calculations (24 h/day for some months) with tdp of 150W for each cpu, and each server had 2 of them.

One of the radiators started to leak some drops...directly on the gpu...luckily I saved it and it's still running from some years without issues.

After this I trashed the radiator and replaced with air cooling, and I will never use any kind of liquid inside the cases.

Be careful on leaking, if it starts to leak damages can be serious.

Edited by ghost82
Link to comment

I share your point @JonathanM, still I wanted to set up a water cooled system to see how it goes over time. I think I am going to go for a custom solution at some point, where monitoring is easier due to availability of addons like sensors. @ghost82 At this point I picked an all-in-one solution because it is basically sealed, sold million times and its the 2nd gen of the silent loop product. First one had a few issues, and with version 2 a lot seems to have improved. I am surprised that we still have to deal with leaking radiators nowadays. 

  • Like 1
Link to comment

Nice build and nice set up. Clearly mentioned everything others my be finding usefull. (Like RDIMMS).

Like your build and even your AIO-Watercooled solution should be fine. 🙂

Also to mention not problem while helping you out with your hackintosh. It was a pleasure! 🙂

Good Luck with your new build. Mine is also already set up and running but atm just as a backup unraid 🙂 (Trial) cuz I want to get all working before I switch my "Working-Monster" xD

  • Like 1
Link to comment
On 11/23/2021 at 11:10 PM, doesntaffect said:

TL;DR - It just works, and it works fine. Epyc is a great platform for UnRaid, virtualizing the Hackintosh was challenging, and successful at the end.

 

5D4B1A56-C8BC-4666-9289-8D102A8D8911.thumb.jpeg.b7ed3cf71b3a770d28def2eba6f1818a.jpeg

 

 

But just for my inner monk... your bequit logo is up side down xD This would drive my crazy. 🤣

  • Haha 2
Link to comment

You noticed that? Amazing! 😂

For full disclosure, I destroyed one Silent Loop during the build because I couldn’t live with the logo being upside down. However, the mounting bracket does only slide in one way into the cooler/pump unit, which forces the upside down because the mounting holes are different on each side of the socket. Forcing the bracket from the other side into the unit cracks the sealing of the tubing.

Once I’ll have time early next year, I’ll go to a local CNC shop and get a custom bracket which allows “correct” mounting. 

 

Thrilled to see that I am not the only one who is picky on such things :)

 

Edited by doesntaffect
  • Like 1
Link to comment

Updated pictures and light config changes: 

 

Had to move the GPU to a riser card since, even if it emits relatively little heat, it caused the two SSDs in the onboard slots (MacOS & Backup cache) to heat up to 51° Celsius. 

 

I used the Fractal design riser cable. Quality of the cable is ok, however I had to remove the little slider which locks the card since I couldn't slide the card into the slot. That could be designed in a better way. The cable is relatively stiff. 

 

Temps for both SSDs dropped to 38 - 43, depending on ambient and load. 

 

1921664946_IMG_2823Large.thumb.jpeg.4cb4b52db54793b5cce71a7e5adff26a.jpeg

 

1587929460_IMG_2822Large.thumb.jpeg.a2286b00b9dc27814aaa98525e149cbb.jpeg

 

1908211742_IMG_2821Large.thumb.jpeg.fe073fe05512eb40886017a8ce07368f.jpeg

 

603170369_IMG_2820Large.thumb.jpeg.8bad1b33d5c9e41d5ecf2f5de46f9f46.jpeg

 

1442074493_IMG_2819Large.thumb.jpeg.28dd81ec2f888b805b7e5ad5c0952f33.jpeg

 

525814072_IMG_2818Large.thumb.jpeg.d82ec2fad72df3a617e9ab2e05f570f3.jpeg

Link to comment
  • 3 months later...

Hello,

your build is very nice 😀. My configuration is not that impressive, but I am using the small version of your mainboard (romed8u-2t). There is one issue I am facing and not able to resolve. Have you tried passing through a GPU in the first PCIE slot? I cant get it working. The host is using the internal BMC graphics for boot and when it is up.   Both external GPU are correctly taken over by vfio and can be passed throught to any Windows VMs, except when they are plugged into the first PCIE slot. In this case Windows is booting up only in save mode, without any graphics driver.

Thanks!

Link to comment
1 hour ago, doesntaffect said:

@raptwa did you open a support ticket with ASrock yet? To check if they have any limitations in their BIOS regarding PCIe 1.

Yep, I just got an answer from them. 

 

Q:

I am running a Windows 10 VM with GPU passthrough (KVM + VFIO). It is working fine, when the GPU is in PCIE slot 2 or 3. However it is not working, when the GPU is in PCIE slot 1. Is this a known issue?

 

A (ASRock Support):

Normally it should work. Maybe you can check the slot settings in the bios.

Link to comment
  • 2 months later...
On 11/23/2021 at 11:10 PM, doesntaffect said:
  • USB PCIe Controller: Inateck RedComets U26 (For Keyboard & Mouse; USB Sticks)
  • WIFI & BT: Fenvi T919 WIFI / BT passed through PCIe Controller, internally connected via USB-Header-to-USB2.0 adapter cable to passed through Inateck RedComets U26 controller (Fenvi T919 provides Wifi & Bluetooth for macOS VM to enable Airplay, Airdrop etc.)

Hey,

 

what cable are you using for the connection? It does not look like the original one which is being delivered and I can't find it on the internet.

The original has 4-pin to 9pin USB header. Yours is 4-pin to USB Type A.

 

Thanks,

syngin

Bild 2.jpeg

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.