[System hangs when VM Changes states (shutdown, restart etc)]

*UPDATE*

Changed Machine Type to Q35-2.12 as suggested from @Karatekid and the issue is now gone.

Everything seems to work just fine, with the GPU as a multifunction device and the ASM2142 USB 3.1 Host Controller PCI card, passed through.

Next step is to test the new PCI card, a WTXUP for Broadcom BCM94360CS2 Wifi and Bluetooth card that works natively with Mac OS and brings all the features of a Mac PC to Hackintosh like Airdrop and Continuity.

I'll keep you updated for any changes regarding the original subject as I'm testing it through the next weeks.

Thanks again

[System hangs when VM Changes states (shutdown, restart etc)]

Hi guys,

Hope you are all well and healthy.

I have finally found some time (now with the COVID-19 we all have more free time in our hands ) to test the boot parameter "pcie_no_flr=1022:149c,1022:1487" .

VM restarts without crashing the Server. *VICTORY*

BUT after the VM restarts, it doesnt recognise the ASM2142 USB 3.1 Host Controller PCI card.
In order to make it work again i have to restart the server.

The system log:

Mar 22 13:02:17 Tower avahi-daemon[11015]: Joining mDNS multicast group on interface veth0cef2f5.IPv6 with address fe80::a4ad:faff:fe5b:dd4.
Mar 22 13:02:17 Tower avahi-daemon[11015]: New relevant interface veth0cef2f5.IPv6 for mDNS.
Mar 22 13:02:17 Tower avahi-daemon[11015]: Registering new address record for fe80::a4ad:faff:fe5b:dd4 on veth0cef2f5.*.
Mar 22 13:02:18 Tower avahi-daemon[11015]: Joining mDNS multicast group on interface veth11e7458.IPv6 with address fe80::d4d5:efff:feca:be58.
Mar 22 13:02:18 Tower avahi-daemon[11015]: New relevant interface veth11e7458.IPv6 for mDNS.
Mar 22 13:02:18 Tower avahi-daemon[11015]: Registering new address record for fe80::d4d5:efff:feca:be58 on veth11e7458.*.
Mar 22 13:02:20 Tower kernel: vfio-pci 0000:01:00.0: Relaying device request to user (#50)
Mar 22 13:04:03 Tower kernel: vfio-pci 0000:01:00.0: Relaying device request to user (#60)
Mar 22 13:04:11 Tower crond[2524]: failed parsing crontab for user root: 01:00 /usr/local/emhttp/plugins/ca.turbo/scripts/turboSchedule.php disable 480 > /dev/null 2>&1
Mar 22 13:05:45 Tower kernel: vfio-pci 0000:01:00.0: Relaying device request to user (#70)
Mar 22 13:07:28 Tower kernel: vfio-pci 0000:01:00.0: Relaying device request to user (#80)
Mar 22 13:09:10 Tower kernel: vfio-pci 0000:01:00.0: Relaying device request to user (#90)
Mar 22 13:10:52 Tower kernel: vfio-pci 0000:01:00.0: Relaying device request to user (#100)

The VM Log:

-device pcie-root-port,port=0x8,chassis=5,id=pci.5,bus=pcie.0,multifunction=on,addr=0x1 \
-device pcie-root-port,port=0x9,chassis=6,id=pci.6,bus=pcie.0,addr=0x1.0x1 \
-device pcie-pci-bridge,id=pci.7,bus=pci.1,addr=0x0 \
-device nec-usb-xhci,p2=15,p3=15,id=usb,bus=pcie.0,addr=0x7 \
-device virtio-serial-pci,id=virtio-serial0,bus=pci.2,addr=0x0 \
-blockdev '{"driver":"file","filename":"/mnt/user/domains/MacinaboxCatalina/Clover.qcow2","node-name":"libvirt-3-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-3-format","read-only":false,"cache":{"direct":false,"no-flush":false},"driver":"qcow2","file":"libvirt-3-storage","backing":null}' \
-device ide-hd,bus=ide.2,drive=libvirt-3-format,id=sata0-0-2,bootindex=1,write-cache=on \
-blockdev '{"driver":"file","filename":"/mnt/user/domains/MacinaboxCatalina/Catalina-install.img","node-name":"libvirt-2-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-2-format","read-only":false,"cache":{"direct":false,"no-flush":false},"driver":"raw","file":"libvirt-2-storage"}' \
-device ide-hd,bus=ide.3,drive=libvirt-2-format,id=sata0-0-3,write-cache=on \
-blockdev '{"driver":"host_device","filename":"/dev/sdc","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":false,"cache":{"direct":false,"no-flush":false},"driver":"raw","file":"libvirt-1-storage"}' \
-device ide-hd,bus=ide.4,drive=libvirt-1-format,id=sata0-0-4,write-cache=on \
-netdev tap,fd=36,id=hostnet0 \
-device e1000-82545em,netdev=hostnet0,id=net0,mac=52:54:00:8a:cb:a9,bus=pci.3,addr=0x0 \
-chardev pty,id=charserial0 \
-device isa-serial,chardev=charserial0,id=serial0 \
-chardev socket,id=charchannel0,fd=37,server,nowait \
-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 \
-device usb-tablet,id=input0,bus=usb.0,port=1 \
-vnc 0.0.0.0:1,websocket=5701 \
-k en-us \
-device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pci.7,addr=0x1 \
-device vfio-pci,host=0000:02:00.0,id=hostdev0,bus=pci.4,addr=0x0 \
-device vfio-pci,host=0000:02:00.1,id=hostdev1,bus=pci.5,addr=0x0 \
-device vfio-pci,host=0000:01:00.0,id=hostdev2,bus=pci.6,addr=0x0 \
-usb \
-device usb-kbd,bus=usb-bus.0 \
-device '************************' \
-smbios type=2 \
-cpu Penryn,kvm=on,vendor=GenuineIntel,+invtsc,vmware-cpuid-freq=on,+pcid,+ssse3,+sse4.2,+popcnt,+avx,+aes,+xsave,+xsaveopt,check \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
2020-03-21 20:44:33.416+0000: Domain id=2 is tainted: high-privileges
2020-03-21 20:44:33.416+0000: Domain id=2 is tainted: custom-argv
2020-03-21 20:44:33.416+0000: Domain id=2 is tainted: host-cpu
char device redirected to /dev/pts/1 (label charserial0)
usb_desc_get_descriptor: 1 unknown type 33 (len 10)
2020-03-22T09:53:43.992097Z qemu-system-x86_64: warning: guest updated active QH

The VM settings:


Any suggestions?
Do anyone have had the same issue?

Thanks so much for your help guys.

[System hangs when VM Changes states (shutdown, restart etc)]

Thanks so much for sharing your tests.

I've been away for a while.
Tonight i'm ginna try all your suggestions and will inform you.

 

[System hangs when VM Changes states (shutdown, restart etc)]

In the last screenshot you’ve posted, it’s a different device on a different IOMMU group.

Here you’re showing us an AMD USB controller (possibly your motherboards usb controller).

 

In the previous screenshot you were showing us a PCI bridge that had issues with the VM.

 

let me get this straight.

you said you’re trying to passthough the following:

On 2/11/2020 at 3:54 PM, dboris said:

ASMedia Technology Inc. ASM2142 USB 3.1 Host Controller | USB controller (08:00.0)

Is the ASM2142 a PCI USB controller connected on a PCI slot? 
 

 

[System hangs when VM Changes states (shutdown, restart etc)]

23 hours ago, dboris said:

I think I had to enable PCI ACS override because the IOMMU group was containing the GPU + other devices 🤨

I'll update you in the coming days.

Also the log screen capture I previously posted isn't a generality, after reboot, I got that instead :

I see.
The 1822:1453 is the PCI bridge that runs the ASM2142?
Did i got it right?

[System hangs when VM Changes states (shutdown, restart etc)]

4 minutes ago, dboris said:

Thanks, I corrected my post. It's been a few weeks for me in the unraid world. I got to so many parity repairs, which are caused by hard reset. It seems like a similar behaviour.

What would be the best test to confirm it ?

Should I remove passing this usb pci controler in Unraid OS options and try to see if windows VM accept to reboot?

Usually never works for me.

Be aware I also ran in a case where no VM were working anymore ; I got to reinstall unraid.

First and foremost I would recommend turning off PCI ACS override, reboot then post the IOMMU groups here in a quote in order to see what have you passed through.

 

Things to try:

1) enable "VFIO allow unsafe interrupts" 

2) try to boot ur VM with the PCI passed though and restart/shutdown

3) try to boot ur VM without any devices passed though (except the GPU) and restart/shutdown

4) remove the PCI devices from the passthrough config

5) try to boot ur vm with GPU passed though and the USB devices selected on your PCI USB Controller and restart/shutdown

 

Every time you restart or shutdown your VM, keep a tab open with the System logs so that you can see what is going on.

 

Then next to each try (2,3,5) post your results so that we can compare and better understand your situation.

 

[System hangs when VM Changes states (shutdown, restart etc)]

Hi @dboris,

 

The PCI USB controller you are using is different than mine and the others that were mentioned in the posts linked by @peter_sm.
So I’m guessing there is a bigger issue here regarding the passthrough of PCI USB controllers (maybe in the latest unraid build?).

 

Lets hope someone from @limetech will see this and collect all the data posted and start debugging the issue. 
 

Crossing fingers 🤞

[System hangs when VM Changes states (shutdown, restart etc)]

Hi peter,

 

I have read both yours and the other guy’s posts, and that’s why I posted this here as a bug. 
 

Let’s hope we’re gonna draw some attention and someone actually help us resolve our issues.