Problems solved. The card was bad with the port that wasn't linking. A new card replacement verified it.
Installing the new card, I had to do the following.
* delete the `/boot/config/network.cfg`
* delete the `/boot/config/network-rules.cfg`
Upon reboot, the `network-rules.cfg` was regenerated with the new hardware assigned to `eth0` and `eth1`. Through the UI, I reconfigured `bond0` to use static IP as well as bonding type to use `802.3ad`. During the reboot, I configured my switch to also aggregate the ports that are servicing `eth0` and `eth1`. And voila, this problem is fixed!