Hi all,
I've been trying to build a Hackintosh VM in the last months.
After a lot of research and suggestions from the community, I've decided to buy a new "ASMedia Technology Inc. ASM1142 USB 3.1 Host Controller" to play nice and natively with my Mac setup.
Following the guides of @SpaceInvaderOne (thank you so much man), I have set up the VM and properly passed though the PCI card.
Things are going well and everything works fine until I ask the Mac VM to restart or shutdown.
If any of the 2 options are executed (either though the VM itself or from VM section on Unraid) the VM shuts down then the system (server) hangs.
I have to manually force reset the whole system (server) cause everything is unresponsive and cannot access WEB GUI or ssh etc.
I started investigating on the issue and checked the logs.
Seems that the problem has to do with the ASM1142 PCI card.
In the system log right after I execute a restart or shutdown I get the following:
QuoteFeb 10 14:47:03 Tower avahi-daemon[11251]: Registering new address record for fe80::fc54:ff:fe51:c393 on vnet2.*.
Feb 10 14:47:04 Tower kernel: vfio_ecap_init: 0000:02:00.0 hiding ecap 0x19@0x270
Feb 10 14:47:04 Tower kernel: vfio_ecap_init: 0000:02:00.0 hiding ecap 0x1b@0x2d0
Feb 10 14:47:04 Tower kernel: vfio_ecap_init: 0000:02:00.0 hiding ecap 0x1e@0x370
Feb 10 14:47:04 Tower kernel: pmd_set_huge: Cannot satisfy [mem 0xe0000000-0xe0200000] with a huge-page mapping due to MTRR override.
Feb 10 14:47:05 Tower kernel: vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x280
Feb 10 14:47:12 Tower kernel: vfio-pci 0000:02:00.0: No more image in the PCI ROM
Feb 10 14:47:12 Tower kernel: vfio-pci 0000:02:00.0: No more image in the PCI ROM
Feb 10 14:48:26 Tower kernel: DMAR: DRHD: handling fault status reg 2
Feb 10 14:48:26 Tower kernel: DMAR: [DMA Write] Request device [01:00.0] fault addr 4e0c9ce277000 [fault reason 04] Access beyond MGAW
Feb 10 14:48:26 Tower kernel: DMAR: [DMA Write] Request device [01:00.0] fault addr 4e0c9ce278000 [fault reason 04] Access beyond MGAW
Feb 10 14:48:26 Tower kernel: DMAR: [DMA Write] Request device [01:00.0] fault addr 4e0c9ce277000 [fault reason 04] Access beyond MGAW
Feb 10 14:48:26 Tower kernel: DMAR: [DMA Write] Request device [01:00.0] fault addr 4e0c9ce277000 [fault reason 04] Access beyond MGAW
Feb 10 14:48:26 Tower kernel: DMAR: [DMA Write] Request device [01:00.0] fault addr 4e0c9ce277000 [fault reason 04] Access beyond MGAW
Feb 10 14:48:26 Tower kernel: DMAR: [DMA Write] Request device [01:00.0] fault addr 4e0c9ce277000 [fault reason 04] Access beyond MGAW
Feb 10 14:48:26 Tower kernel: DMAR: [DMA Write] Request device [01:00.0] fault addr 4e0c9ce278000 [fault reason 04] Access beyond MGAW
Feb 10 14:48:26 Tower kernel: DMAR: [DMA Write] Request device [01:00.0] fault addr 4e0c9ce278000 [fault reason 04] Access beyond MGAW
Feb 10 14:48:26 Tower kernel: DMAR: [DMA Write] Request device [01:00.0] fault addr 4e0c9ce278000 [fault reason 04] Access beyond MGAW
Feb 10 14:48:26 Tower kernel: pcieport 0000:00:02.0: AER: Uncorrected (Fatal) error received: 0000:00:02.0
Feb 10 14:48:26 Tower kernel: pcieport 0000:00:02.0: PCIe Bus Error: severity=Uncorrected (Fatal), type=Transaction Layer, (Receiver ID)
Feb 10 14:48:26 Tower kernel: pcieport 0000:00:02.0: device [8086:2f04] error status/mask=00040000/00000000
Feb 10 14:48:26 Tower kernel: pcieport 0000:00:02.0: [18] MalfTLP (First)
Feb 10 14:48:26 Tower kernel: pcieport 0000:00:02.0: TLP Header: 60000005 01000078 578ce0c9 ce277ff0
Feb 10 14:48:26 Tower kernel: vfio-pci 0000:01:00.0: Relaying device request to user (#0)
Feb 10 14:48:31 Tower kernel: dmar_fault: 1134 callbacks suppressed
Feb 10 14:48:31 Tower kernel: DMAR: DRHD: handling fault status reg 100
Feb 10 14:48:32 Tower kernel: DMAR: DRHD: handling fault status reg 100
Feb 10 14:48:33 Tower kernel: DMAR: DRHD: handling fault status reg 100
Posting my IOMMU groups:
QuoteSystem Devices
PCI Devices and IOMMU GroupsIOMMU group 0: [8086:2f81] ff:0b.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring (rev 02)
[8086:2f36] ff:0b.1 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring (rev 02)
[8086:2f37] ff:0b.2 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring (rev 02)
IOMMU group 1: [8086:2fe0] ff:0c.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)
[8086:2fe1] ff:0c.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)
[8086:2fe2] ff:0c.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)
[8086:2fe3] ff:0c.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)
[8086:2fe4] ff:0c.4 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)
[8086:2fe5] ff:0c.5 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)
IOMMU group 2: [8086:2ff8] ff:0f.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Buffered Ring Agent (rev 02)
[8086:2ff9] ff:0f.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Buffered Ring Agent (rev 02)
[8086:2ffc] ff:0f.4 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers (rev 02)
[8086:2ffd] ff:0f.5 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers (rev 02)
[8086:2ffe] ff:0f.6 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers (rev 02)
IOMMU group 3: [8086:2f1d] ff:10.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCIe Ring Interface (rev 02)
[8086:2f34] ff:10.1 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCIe Ring Interface (rev 02)
[8086:2f1e] ff:10.5 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers (rev 02)
[8086:2f7d] ff:10.6 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers (rev 02)
[8086:2f1f] ff:10.7 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers (rev 02)
IOMMU group 4: [8086:2fa0] ff:12.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 0 (rev 02)
[8086:2f30] ff:12.1 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 0 (rev 02)
IOMMU group 5: [8086:2fa8] ff:13.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Target Address, Thermal & RAS Registers (rev 02)
[8086:2f71] ff:13.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Target Address, Thermal & RAS Registers (rev 02)
[8086:2faa] ff:13.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder (rev 02)
[8086:2fab] ff:13.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder (rev 02)
[8086:2fac] ff:13.4 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder (rev 02)
[8086:2fad] ff:13.5 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder (rev 02)
[8086:2fae] ff:13.6 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Channel 0/1 Broadcast (rev 02)
[8086:2faf] ff:13.7 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Global Broadcast (rev 02)
IOMMU group 6: [8086:2fb0] ff:14.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 0 Thermal Control (rev 02)
[8086:2fb1] ff:14.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 1 Thermal Control (rev 02)
[8086:2fb2] ff:14.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 0 ERROR Registers (rev 02)
[8086:2fb3] ff:14.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 1 ERROR Registers (rev 02)
[8086:2fbc] ff:14.4 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 (rev 02)
[8086:2fbd] ff:14.5 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 (rev 02)
[8086:2fbe] ff:14.6 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 (rev 02)
[8086:2fbf] ff:14.7 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 (rev 02)
IOMMU group 7: [8086:2fb4] ff:15.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 2 Thermal Control (rev 02)
[8086:2fb5] ff:15.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 3 Thermal Control (rev 02)
[8086:2fb6] ff:15.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 2 ERROR Registers (rev 02)
[8086:2fb7] ff:15.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 3 ERROR Registers (rev 02)
IOMMU group 8: [8086:2f68] ff:16.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Target Address, Thermal & RAS Registers (rev 02)
[8086:2f6e] ff:16.6 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Channel 2/3 Broadcast (rev 02)
[8086:2f6f] ff:16.7 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Global Broadcast (rev 02)
IOMMU group 9: [8086:2fd0] ff:17.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel 0 Thermal Control (rev 02)
[8086:2fb8] ff:17.4 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 (rev 02)
[8086:2fb9] ff:17.5 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 (rev 02)
[8086:2fba] ff:17.6 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 (rev 02)
[8086:2fbb] ff:17.7 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 (rev 02)
IOMMU group 10: [8086:2f98] ff:1e.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit (rev 02)
[8086:2f99] ff:1e.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit (rev 02)
[8086:2f9a] ff:1e.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit (rev 02)
[8086:2fc0] ff:1e.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit (rev 02)
[8086:2f9c] ff:1e.4 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit (rev 02)
IOMMU group 11: [8086:2f88] ff:1f.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 VCU (rev 02)
[8086:2f8a] ff:1f.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 VCU (rev 02)
IOMMU group 12: [8086:2f00] 00:00.0 Host bridge: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMI2 (rev 02)
IOMMU group 13: [8086:2f04] 00:02.0 PCI bridge: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 2 (rev 02)
IOMMU group 14: [8086:2f08] 00:03.0 PCI bridge: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 3 (rev 02)
IOMMU group 15: [8086:2f28] 00:05.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Address Map, VTd_Misc, System Management (rev 02)
[8086:2f29] 00:05.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Hot Plug (rev 02)
[8086:2f2a] 00:05.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 RAS, Control Status and Global Errors (rev 02)
[8086:2f2c] 00:05.4 PIC: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 I/O APIC (rev 02)
IOMMU group 16: [8086:8d7c] 00:11.0 Unassigned class [ff00]: Intel Corporation C610/X99 series chipset SPSR (rev 05)
[8086:8d62] 00:11.4 SATA controller: Intel Corporation C610/X99 series chipset sSATA Controller [AHCI mode] (rev 05)
IOMMU group 17: [8086:8d31] 00:14.0 USB controller: Intel Corporation C610/X99 series chipset USB xHCI Host Controller (rev 05)
IOMMU group 18: [8086:8d3a] 00:16.0 Communication controller: Intel Corporation C610/X99 series chipset MEI Controller #1 (rev 05)
IOMMU group 19: [8086:15a1] 00:19.0 Ethernet controller: Intel Corporation Ethernet Connection (2) I218-V (rev 05)
IOMMU group 20: [8086:8d2d] 00:1a.0 USB controller: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #2 (rev 05)
IOMMU group 21: [8086:8d20] 00:1b.0 Audio device: Intel Corporation C610/X99 series chipset HD Audio Controller (rev 05)
IOMMU group 22: [8086:8d10] 00:1c.0 PCI bridge: Intel Corporation C610/X99 series chipset PCI Express Root Port #1 (rev d5)
IOMMU group 23: [8086:8d14] 00:1c.2 PCI bridge: Intel Corporation C610/X99 series chipset PCI Express Root Port #3 (rev d5)
IOMMU group 24: [8086:8d26] 00:1d.0 USB controller: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #1 (rev 05)
IOMMU group 25: [8086:8d47] 00:1f.0 ISA bridge: Intel Corporation C610/X99 series chipset LPC Controller (rev 05)
[8086:8d02] 00:1f.2 SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)
[8086:8d22] 00:1f.3 SMBus: Intel Corporation C610/X99 series chipset SMBus Controller (rev 05)
IOMMU group 26: [1b21:1242] 01:00.0 USB controller: ASMedia Technology Inc. ASM1142 USB 3.1 Host Controller
IOMMU group 27: [1002:67df] 02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] (rev c7)
[1002:aaf0] 02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere HDMI Audio [Radeon RX 470/480 / 570/580/590]
IOMMU group 28: [1912:0015] 04:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02)
As you can see the "AER: Uncorrected (Fatal) error" on "[8086:2f04] 00:02.0" device that is mentioned in the system log is the PCI bus that the "ASM1142" PCI card is installed.
In order to completely be sure that its the card that has the issue I tried the following:
- Used the ASM1142 PCI card in a Windows 10 VM and got the same error and behaviour (without it everything works fine)
- Enabled (previously disabled) "PCIe ACS override" and tried all combinations
- Enabled (previously disabled) "VFIO allow unsafe interrupts" and tried all combinations.
- Passed through the "IOMMU group 13: [8086:2f04] 00:02.0 PCI bridge: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 2 (rev 02)" that system log mentioned that has the "AER: Uncorrected (Fatal) error"
Nothing of the above solved the problem.
I have to mention that sometimes the system doesn't hang completely, but the VM page cannot load up (other VMs work fine), dockers run fine. When that happens the RAM that was allocated for the VM that failed stays allocated, even after I manually turn off the VM Manager from settings. When I turn on the VM Manager again, the VM Manager is still unresponsive and cannot have access to it.
Then if I try to restart the system or shutdown, it hangs and I have to reset again.
Any help would be greatly appreciated.
BR