[SOLVED] 10G network works in VM after Unraid reboot, but not after VM restart

[DANgerous25]

Short story: I have an ASRock x570 motherboard that has an on board Aquantia Aqtion 10Gbit network adapter. I've successfully passed through the adapter card to the VM (leaving the other onboard adapter for Unraid). If I reboot Unraid and then start the VM, the adapter in Windows gets an IP address via DHCP and the connection remains stable and I am happy. However, if I stop the VM [nicely] and then restart it, then the adapter doesn't pick up an IP address and its status fluctuates between the three states below. Nothing I've tried other than rebooting Unraid seems to solve the issue.

 

Identifying -> Network cable unplugged -> Enabled -> ... repeat ...

 

 

Longer story; Here are some of the things I've tried:

• Updated the Aquantia drivers in Window with a) Those listed on the ASRock site, b) From the Aquantia site. -> No joy.
• Tried both static and dynamic IP with DHCP. -> No difference.
• Disabled the other adapters in Windows. -> No difference apart from making it impossible for me to log in to the VM without rebooting!
• Reset the adapter, uninstall/re-installed in device manager, enabled/disabled in device manager, manually set link speed to 10G, turned off power features. -> Nope.
• Checked to see if the adapter port is overheating on the motherboard (just using my finger, so not massively scientific, but doesn't seem very hot). -> Inconclusive.
• Left the Unraid server and VM running while in the working state for hours to see if at some point the connection drops (it doesn't). -> Inconclusive, it has only been hours.
• Changed the CAT 7 network cable for another cable. -> No difference.
• Reset the SPF+ module in the switch, tried a different and definitely working SPF+ module in the switch. -> That's right, no change.
• Run the VM from the Unraid array and from its own unassigned drive. -> Clutching at straws I know, but I was doing this for another unrelated purpose.
• Checked for errors coming up when I start the array. -> No errors come up (but maybe there are errors behind the scenes?)
• Stop VM / start VM nicely or Force Stop VM / Start VM from the Unraid VM page. -> Windows starts nicely, but the 10G adapter says meh.
• Searched the internet/forums for similar issues. -> I'm yet to find anything similar.
• Checked for flashing lights on the interface. -> Yep, lights are on.
• Rebooted Unraid and start the array then VM -> Works.

 

My setup:

• I'm using Unraid Version 6.9.0-beta1 2020-03-06, Windows 10 May 2020 Update VM on its own SSD, Ryzen 3950X / 128 GB RAM / Asrock x570. Pretty much vanilla Unraid install, I've only just set it up so it's just this VM and an Ubuntu server I'm playing with.
• The Unraid server connects to a Mikrotik 10Gb switch via cat 7 RJ/45 -> SPF+ adapter module in the switch. Another computer connects to the switch in the same way without issues. The rest of the network is 1Gb LAN / ethernet over power / Wifi 5.

 

I'm pretty new to Unraid and so I'm really stuck as to how to diagnose this issue beyond what I've done above. I am also very aware that this could be a Windows issue or unrelated to Unraid. Any help would be appreciated.

 

[Jessie]

Sounds like the 10g adapter is not releasing itself when you shut down the vm.  What if you made the 10g adapter the unraid adapter, then bridge that through to the vm?  A link which might be a similar scenareo. 

 

[DANgerous25]

Superb, @Jessie! I made some simple modifications to the "AMD reset bug" script like below, which enables a successful workaround. Indeed the same solution works for the network adapter. Thanks so much! Also thanks to the SpaceInvader One video for the great overview this.

 

My solution script:

 

#!/bin/bash

#

#replace xx\:xx.x with the number of your PCIE device that you want to reset

#

echo "disconnecting Aquantia network card"

echo "1" | tee -a /sys/bus/pci/devices/0000\:2e\:00.0/remove

#echo "1" | tee -a /sys/bus/pci/devices/0000\:xx\:xx.x/remove

echo "Entered suspended state... press power button to continue!"

echo -n mem > /sys/power/state

echo "Reconnecting Aquantia network card..."

echo "1" | tee -a /sys/bus/pci/rescan

echo "Aquantia network card reset, all done!"

 

Regarding the suggestion for making the 10G adapter for Unraid; the reason I didn't is because I wasn't getting 10G out of the virtual driver, I had actually thought it was limited to 100Mb. It looks like I didn't do this properly the first time, as I've just done this again and here are the results using some simple iperf3 tests:

 

• From a client computer to the Unraid server I am getting 8-9GB/s, f
• From a client computer to the Win10 VM using the bridged adapter in Unraid I am getting ~1.25Gb/s.
• For comparison, when passing through the 10G adapater to the VM I get ~7Gb/s from the client computer but have to use the reset script above if I shut down the VM without rebooting.

= So the bridged adapter setup significantly reduces the bandwidth, but actually 1.25Gb/s is fine for my use case and is an acceptable tradeoff to the "AMD reset bug".

 

Edit: Updated with results of the bridging suggestion. Thanks again @Jessie.

Edited June 9, 2020 by DANgerous25
Updated with further test results

[Jessie]

You're welcome. 

I found this regarding connecting 10gb via bridge.  Not sure whether relevant to your situation.

 

 

[DANgerous25]

 

Thanks as always @Jessie from the response. Actually this doesn't exactly cover my use case, as I'm not sure if this is an Unraid question or a network question, I posted in another forum but I'd be happy to take any advice if you have it. https://forums.serverbuilds.net/t/guide-remote-gaming-on-unraid/4248/124

Edited June 10, 2020 by DANgerous25