Catalina network bridging weirdness


Recommended Posts

tl;dr: My Catalina VM doesn't like the same bridged network configuration I used with High Sierra and Mojave, and I can't make any sense of it.

 

I've been running a simple (no GPU or other passthroughs) macOS VM on my Unraid server for years now--first High Sierra, then Mojave, and now Catalina.

 

Before Catalina, I always successfully used a bridged network configuration like this (phony MAC address below):

    <interface type='bridge'>
      <mac address='00:11:22:33:44:55'/>
      <source bridge='br0'/>
      <model type='e1000-82545em'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </interface>

When I boot Catalina with the configuration above, however, things go really strangely--the Ethernet cable is reported as unplugged, the system lags, the menu bar either never appears or is only ever partially rendered, sometimes the keyboard doesn't work at all, and I can't ever successfully shut down.

 

I use Clover as my bootloader, so I tried this configuration with both same older bootloader and UEFI driver versions that worked with Mojave, as well as with everything updated to Clover v2.5k 5070; I saw no difference either way.

 

So, I changed to a non-bridged configuration, like this...

    <interface type='network'>
      <mac address='00:11:22:33:44:55'/>
      <source network='default'/>
      <model type='e1000-82545em'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </interface>

...and everything works fine, which is surprising, given that it is still a virtual e1000-82545em NIC with the same MAC in the same virtual PCI slot. I wouldn't imagine that Catalina would even know the difference!

 

It's better than nothing for the moment, but I really do need the VM to be bridged for my purposes.

 

So, as another experiment, I tried bridging using the 'vmxnet3' virtual NIC, like this...

    <interface type='bridge'>
      <mac address='00:11:22:33:44:55'/>
      <source bridge='br0'/>
      <model type='vmxnet3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </interface>

...which sorta surprisingly does make Catalina work properly, except I get bumped off all Apple cloud services, and System Preferences asks me to log in again.

 

By itself, that isn't shocking, since I did change the (virtual) hardware out from under Catalina. The surprising part is that Catalina then can't reach the Apple ID servers:

 

681184911_ScreenShot2019-10-14at5_20_42.thumb.png.b5c713ba586887be5f3db2bfa75f5bef.png

 

I did a fair amount of Googling about this result, and while I couldn't find anyone really getting into a discussion or analysis, I did find a few offhand references to vmxnet3 causing iCloud login woes.

 

So, out of desperation, I tried installing Catalina again using vmxnet3 bridging from the start, but it made no difference.


So, does anyone have any idea why a straightforward e1000-82545em bridge that worked fine under High Sierra and Mojave might cause Catalina to melt down?

 

Or (though I'd much rather just get the e1000-82545em bridging working) does anyone have any experience getting communication with the Apple ID servers working properly with a vmxnet3 bridge?

 

Thanks for any help!

Edited by bland328
Clarified a couple points
Link to comment

Edit: I wasn't quite right about this! It's true that that some or all of Apple's online services require communication over an en0 device that appears to be "built in"...but vmxnet3 does appear as a built-in interface if the system configuration is right.

 

For my earlier testing, my virtual vmxnet3 interface was configured on bus 3/slot 0, in which case macOS decides it isn't built-in (IORegistry says IOBuiltin=False and IOPrimaryInterface=False, though I'm not clear if the latter matters).

 

When I move my vmxnet3 to bus 0/slot 5, macOS does see it as built-in (IORegistry says IOBuiltin=True and IOPrimaryInterface=True).

 

However, I still can't use it to log into to the App Store, as I get a "There was an error connecting to the Apple ID Server" error.

 

If I switch back to an unbridged e1000-82545em configuration, I can log into the App Store. But, then I can't bridge, or Catalina has a nervous breakdown.

 

Some progress on this...it looks like vmxnet3 simply won't work for Apple cloud services (including the App Store) by design, because macOS (for whatever security reasons, I suppose) requires a built-in Ethernet adapter for such things.

 

Since vmxnet3 is a paravirtualized NIC powered by an Apple driver, it makes sense that macOS would know it isn't a real, built-in NIC.

 

Just to confirm, with vmxnet3 networking active, I used IORegistryExplorer (an Xcode-related developers' tool) in the Catalina VM to search for 'en0' and, sure enough, found:

IOBuiltin	Boolean	False

So, vmxnet3 doesn't look like a good solution for those of us who use iCloud/iMessage/App Store/etc. At least, not without some low-level game playing that I'd rather not get into, if I can avoid it.

 

So, that focuses my question:

Why might bridging to br0 using e1000-82545em work under High Sierra and Mojave, but not Catalina?

 

Is there a macOS virtual networking guru out there (perhaps even @SpaceInvaderOne?) who might be able to point me to something to analyze or try?

Edited by bland328
I was wrong.
Link to comment

Just to add to your findings that I also have problems with Catalina and icloud login with e1000-82545em (only in the preference panel, it works ok with the web browser).

I cannot track what apple checks, since sometimes it works sometimes it doesn't work, and it fails with very generic errors.

iCloud was working with e1000-82545em and Catalina, iMessage didn't work, so I logout from all, reset icloud, removed the mac os vm from the list of "find my phone" and "my devices".

I also enabled the 2 factor authentication, the code is delivered than a message is displayed ("cannot verify your identity") than it hangs (preference panel).

I have to try the ethernet passthrough and wait for the mac pro 2019 to be on the market as my last chance.

Link to comment

@ghost82, when you turn on e1000-82545em bridging (to br0, I assume), does Catalina fundamentally function correctly?

 

If so, are you on Unraid 6.7.2, or a beta? If your QEMU version is newer than mine, it might include a change to the vmxnet3 implementation.

 

For me, it's still the case that when I shut down, switch from the default network to the br0 bridge, and reboot, Catalina won't even finish painting the menu bar, gives me lots of long beachball pauses, and won't successfully shut down.

 

Also, regarding your "cannot verify your identity" issues, are they possibly because your Model, MAC address, ROM, MLB, SMBIOS Board Serial Number and SmUUID aren't all carefully chosen and in sync?

 

If you aren't familiar with all these DRM-related issues, https://www.tonymacx86.com/threads/how-to-fix-imessage.110471/ is a great resource, but there's plenty to wade through that's more about issues specific to physical Hackintosh builds.

Edited by bland328
  • Like 1
Link to comment

@bland328 yes Catalina is working without issues and I'm on unraid 6.7.2, with QEMU 3.1.0.

I have 2 nics on my server, bonded together and bridged to br0, which the VM is using.

This is actually the xml part referring to network:

    <interface type='bridge'>
      <mac address='RE:DA:CT:ED:XX:XX'/>
      <source bridge='br0'/>
      <target dev='vnet0'/>
      <model type='e1000-82545em'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>

I never had the problems you described, I'm sorry for your issues.

 

Thank you for the link to the tutorial to try to make icloud/imessage/etc working.

Link to comment

@ghost82, thanks for that. The only differences between your working configuration and my non-working configuration are:

  1. Your br0 is running over bonded interfaces, whereas mine is running over a single interface.
  2. My PCI address is bus='0x00' slot='0x05'.

You may (in my experience so far, probably will) need to move yours to bus='0x00' in order to get it to look to macOS like a built-in Ethernet port (which can be confirmed by searching for 'en0' in the IORegistryBrowser app); the App Store and iCloud stuff only works over a built-in Ethernet port.

 

Meaning that if you had a MacBook with a broken Ethernet port and plugged in a third-party USB-to-Ethernet adapter to get back online...you then couldn't use iCloud (at least, not without some icky low-level spoofing trickery)! Sounds crazy, but it is discussed online by people who've run into precisely that. That's a serious DRM/security pain point! 😅

Link to comment
14 minutes ago, bland328 said:

@ghost82, thanks for that. The only differences between your working configuration and my non-working configuration are:

  1. Your br0 is running over bonded interfaces, whereas mine is running over a single interface.
  2. My PCI address is bus='0x00' slot='0x05'.

You may (in my experience so far, probably will) need to move yours to bus='0x00' in order to get it to look to macOS like a built-in Ethernet port (which can be confirmed by searching for 'en0' in the IORegistryBrowser app); the App Store and iCloud stuff only works over a built-in Ethernet port.

 

Meaning that if you had a MacBook with a broken Ethernet port and plugged in a third-party USB-to-Ethernet adapter to get back online...you then couldn't use iCloud (at least, not without some icky low-level spoofing trickery)! Sounds crazy, but it is discussed online by people who've run into precisely that. That's a serious DRM/security pain point! 😅

I solved my problems with icloud/appstore, I need to fix imessage/facetime but they're working on it :D (I will update if it will get fixed (hopefully by tomorrow) in the new Tips thread opened by Leoyzen.

Even if my ethernet is in bus 1 slot 0 it results as en0 and built in (checked in ioreg both IOBuiltin and IoPrimaryInterface are "true"), so no problems with it.

The tutorial you linked about imessage fix was very helpful, thanks again; everyone should read it BEFORE attempting to make the mac os to work, because something can be fixed on the user side, and something not :P

Moreover, internet should be disconnected when installing updates and reconnected only after checking all the basic parameters apple is checking for icloud/imessage/etc

Edited by ghost82
Link to comment

For anyone running into the e1000-82545em bridging-to-br0 weirdness under Catalina, I have a workaround that's working fine for me:

  1. Install AppleIntelE1000e.kext (I'm using the latest build from the fork at https://github.com/chris1111/AppleIntelE1000e) either to /Library/Extensions (the advantage being simplicity; you can install it manually or with the simple KextBeast utility) or by injecting it with Clover (the advantage being that it will likely work while installing macOS or when booted into Recovery Mode).
  2. Change the Interface definition in your XML to use the 'e1000e' virtual NIC:
<model type='e1000e'/>

Having done this, I can bridge to br0 under Catalina without issue, and even access the App Store and use iCloud services.

 

I'm hoping to be able to make 'virtio-net-pci' work one of these days, but no luck so far.

  • Like 3
Link to comment
Just now, ghost82 said:

everyone should read it BEFORE attempting to make the mac os to work

That's a very good point. If you get everything together before installing macOS, it's all painless.

 

This morning, I carefully chose a MAC address, ROM, MLB, SMBIOS Board Serial Number and SmUUID for a new virtual macOS install in about 10 minutes (I've done it before, which helps!), installed Catalina, and logged right into iCloud.

  • Like 1
Link to comment
On 10/21/2019 at 10:10 PM, bland328 said:

I'm hoping to be able to make 'virtio-net-pci' work one of these days, but no luck so far.

I've give out the steps to make virtio work but it still buggy and I make no progress there.Here is the link I've made looking for help but nobody response.

I'm not facing the "e1000-82545em" problem which work for me all the time so maybe there is something/tips we are missing, but in my hackintosh vm, I've already change to passthrough a X550 nic instead.

Edited by Leoyzen
  • Like 1
Link to comment
I've already change to passthrough a X550 nic instead.

unfortunately that is the only real solid solution to proper network in osx vm. Fought with virtual nics for months . A setup i used for a while was use the vmnet3 for the main network card and the e1000 defined as en0 but disabled. Only its existance as en0 is relevant enough for icloud/appstore, does not have to be active.
But eventualy it was just too slow and moved to a cheap 4 port intel 1gb nic and give every vm its own physical network post passed thru. A world of difference in stability, speed (full line speed) and struggling with virtual io has looking back been a big waste of time.
Link to comment
On 10/27/2019 at 4:51 PM, ghost82 said:

Yes, but I'm not sure for its support, just try.

It was worth a shot but no, it does not work with any.

 

Also I see this behavior, Macos reporting Cable unplugged. on a brand new install of Catalina using Macinabox with default settings (except using qcow2). Changing virtual network device to e1000-82545em after install.

 

That means it's something about my unRAID configuration (using v6.7.2 and bridging). But it works with High Sierra and Mojave.

Seems like a very reproducible bug.

 

 

EDIT: a New Macinabox install works with e1000-82545em on unRAID v6.8.0 rc5, Tested on two machines, both not working with 6.7.2 and both working with 6.8.0 rc5.

My solution: just abandon my old install and use the new Macinabox Catalina install.

 

 

@glennv What you say is perfectly sensible, but unfortunately I do not have a free PCIe slot. I also tried a USB to Ethernet adapter I had laying around but no success with that.

Edited by Jagadguru
additional info
Link to comment
On 10/25/2019 at 11:06 AM, Jagadguru said:

@bland328 After injecting that kext, networking works great, but there is a process called kernel_task which now has constant 115% CPU usage. Is your VM doing that as well?

Sorry for being so slow to respond, @Jagadguru! Yes...turns out I also have that going on. Odd. And unfortunate.

On 10/30/2019 at 4:27 PM, Jagadguru said:

EDIT: a New Macinabox install works with e1000-82545em on unRAID v6.8.0 rc5, Tested on two machines, both not working with 6.7.2 and both working with 6.8.0 rc5.

Are you saying that with MacinaBox & Unraid v6.8.0rc5, e1000-82545 is successfully providing both bridging and Apple ID/iCloud/App Store functionality?

Link to comment
  • 3 weeks later...
On 10/21/2019 at 10:10 PM, bland328 said:

For anyone running into the e1000-82545em bridging-to-br0 weirdness under Catalina, I have a workaround that's working fine for me:

  1. Install AppleIntelE1000e.kext (I'm using the latest build from the fork at https://github.com/chris1111/AppleIntelE1000e) either to /Library/Extensions (the advantage being simplicity; you can install it manually or with the simple KextBeast utility) or by injecting it with Clover (the advantage being that it will likely work while installing macOS or when booted into Recovery Mode).
  2. Change the Interface definition in your XML to use the 'e1000e' virtual NIC:

<model type='e1000e'/>

Having done this, I can bridge to br0 under Catalina without issue, and even access the App Store and use iCloud services.

 

I'm hoping to be able to make 'virtio-net-pci' work one of these days, but no luck so far.

 

Thanks for sharing this! After hours and hours of troubleshooting.. I finally got it working.

 

I downloaded this kext, and installed it using KextBeast. Rebooted the VM and got it working with iCloud, iMessage, everything.

 

I had trouble from the get-go with e1000-82545em, where the taskbar didn't show up, and under network setting the cable was "unplugged". VMXNET3 was slow and couldn't connect to iCloud. I'm happy to finally getting this to work.

 

Thanks again!

 

 

Link to comment
  • 3 weeks later...

I was having the exact same issue as OP, and updating Unraid to 6.8.0 RC9 resolved it.  Before that, I tried switching from VMXNET3 (which was working minus apple server connectivity) to e1000-82545em, which just gave me a never ending beach ball.  Haven't tried imessage, but app store is working.

Link to comment

For the record, I'm currently experimenting with running Catalina under qemu/kvm/libvirt on a non-Unraid Linux box (still an Unraid fan here...this is just a side project!), and I find that using when qemu-4.1.1_1, a virtual e1000-82545em NIC is working just great with br0: bridging.

 

To be fair, this br0: bridge is one I configured, and I'm not currently quite savvy enough to know if that could be the difference...but I doubt it.

 

So, I'm looking forward to Unraid 6.8.0, suspecting an updated qemu will fix everything for me, as it did for @Gitchu.

 

Assuming it does, I'll stop using qemu's virtual e1000 NIC+e1000 kext, and return to using e1000-82545em.

  • Like 1
Link to comment

@bland328 I'm using VMXNET3 on the 6.8.0 RC9 build.  Seems to be working well still, including iMessage.  I don't think i tried the e1000 NIC after switching to RC9, since everything just started working, but I know none NIC options were working before the update, for one reason or another.  I was looking into using other kexts as well, but didn't end up trying it before moving to the RC. I have my mac VM set up as a Q35-3.1 machine, if that makes a difference.

Link to comment

Thanks for the info, @Gitchu.

 

I've upgraded to Unraid 6.8.0 final, and now find that VMXNET3 (+Q35-3.1, which may or may not have anything to do with it) is working great with Catalina, as well.

 

Though, to be fair, I haven't freshly logged into iCloud services (iMessage, App Store, iCloud Drive, etc.) recently; for anyone else reading this, I've run into problems in the past with VMXNET3 working fine with those services only after I've successfully logged in with a different (e1000 or passed-through) NIC.

 

Fingers crossed that those days are now behind us, but I don't feel like logging out of what's now working just to test it. 😉

  • Like 1
Link to comment
  • 4 months later...

I had to switch from the VMXNET3 to e1000 because for some strange reason, osx would steal and lockup the NIC, with the e1000 it seems fine.

So hopefully if anyone else see's OSx kill your network, try changing to e1000-82545em, no idea why, but much more solid now I'm using that...

 

Edit: FYI I'm on Catalina and Unraid 6.8.3

Edited by chrisjenx
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.