No local terminal access but SSH works? (v6.12)

[johnsanc]

I upgraded to v6.12 last week. Last night I had an issue where all of a sudden unRAID was completely inaccessible and neither the Web GUI or SSH worked. I checked my monitor that is locally connected and I noticed that the terminal window had no command prompt and seemed to just hang during processing of USB devices.

 

I hard rebooted the server since I had no other option and noticed the terminal hangs up in the same place again, but SSH and everything else works fine. What is causing this and why did not occur with previous versions?

 

I've attached my diagnostics. Any thoughts are appreciated.

tower-diagnostics-20230625-1205.zip

[ljm42]

I'm not sure why the console would stop updating during boot, what happens if you press <enter> a few times?

 

The syslog has call traces related to macvlan, that is likely why it crashed. See the release notes here: 

https://docs.unraid.net/unraid-os/release-notes/6.12.0#call-traces-related-to-macvlan

[johnsanc]

Thanks I’ll try switching to ipvlan.

 

regarding the terminal it just does nothing and is cut off as shown in the photo. I’ll try a different keyboard just to make sure it’s not an issue with the wireless dongle I normally use.

[johnsanc]

17 hours ago, ljm42 said:

I'm not sure why the console would stop updating during boot, what happens if you press <enter> a few times?

 

The syslog has call traces related to macvlan, that is likely why it crashed. See the release notes here: 

https://docs.unraid.net/unraid-os/release-notes/6.12.0#call-traces-related-to-macvlan

OK so I confirmed its not a keyboard issue. The screen just freezes completely and I never get to the login screen. I can even sometimes see the cursor frozen in an "displayed" state depending on when the screen freezes.

 

Also I changed my docker settings from macvlan to ipvlan and my server still crashed. Was not even up for a single day this time. Can you see anything in my diagnostics that explains either of these issues?

 

I'm hesitant to stay on 6.12.x since this does not seem stable compared to 6.11.5 which would be up for months at a time without any issues.

 

tower-diagnostics-20230627-1333.zip

[johnsanc]

I just noticed this in the syslog which is the line directly after where my local display freezes. What is this?
I am not passing through that graphics card or anything and I have it specifically for local terminal access in case there is an issue with SSH.
 

Jun 27 13:30:40 Tower kernel: Console: switching to colour dummy device 80x25
Jun 27 13:30:40 Tower kernel: vfio-pci 0000:33:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none

 

[JorgeB]

1 minute ago, johnsanc said:

I am not passing through that graphics card

It's bound to vfio-pci, remove this part from /boot/config/vfio-pci.cfg and reboot

0000:33:00.0|10de:1e84 0000:33:00.1|10de:10f8 0000:33:00.2|10de:1ad8 0000:33:00.3|10de:1ad9

 

[johnsanc]

@JorgeB - Apologies, I misspoke. Yes that is my GTX 2070 Super I have for my Windows VM. I do want to pass that through. I have another card that is a crappy little GT710 just for console access which is not passed through. Not sure why my display no longer works with this GT710 and stops at that line. Works fine on 6.11.5.

Edited June 27 by johnsanc

[JorgeB]

I assume the monitor is connected to the other GPU? You could still try to temporarily unbind that GPU just to test.

[johnsanc]

@JorgeB - Yes the monitor is connected to the crappy GT710 that was not bound. When I removed the RTX2070 from being bound, the GT710 worked and I could see the terminal get to the login prompt. For some reason passing through one graphics card caused the other to stop working. This was not an issue with any prior unraid release.

[JorgeB]

@SimonFany idea how to avoid this?

[SimonF]

3 hours ago, JorgeB said:

@SimonFany idea how to avoid this?

Would more logs around when the vm starts to see if any issues

[johnsanc]

Is that not in the diagnostics I posted?

 

Note I also tried not auto-starting any VMs and the issue still occurs during boot.

[johnsanc]

@SimonF - Not sure if this helps or not, but here is my windows 10 VM log:

 

2023-06-28 13:48:31.461+0000: Starting external device: TPM Emulator
/usr/bin/swtpm socket --ctrl 'type=unixio,path=/run/libvirt/qemu/swtpm/1-Windows 10-swtpm.sock,mode=0600' --tpmstate dir=/var/lib/libvirt/swtpm/691a4041-8a00-7085-a6e4-0ba8106d2f44/tpm2,mode=0600 --log 'file=/var/log/swtpm/libvirt/qemu/Windows 10-swtpm.log' --terminate --tpm2
2023-06-28 13:48:31.503+0000: starting up libvirt version: 8.7.0, qemu version: 7.1.0, kernel: 6.1.34-Unraid, hostname: Tower
LC_ALL=C \
PATH=/bin:/sbin:/usr/bin:/usr/sbin \
HOME='/var/lib/libvirt/qemu/domain-1-Windows 10' \
XDG_DATA_HOME='/var/lib/libvirt/qemu/domain-1-Windows 10/.local/share' \
XDG_CACHE_HOME='/var/lib/libvirt/qemu/domain-1-Windows 10/.cache' \
XDG_CONFIG_HOME='/var/lib/libvirt/qemu/domain-1-Windows 10/.config' \
/usr/local/sbin/qemu \
-name 'guest=Windows 10,debug-threads=on' \
-S \
-object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-1-Windows 10/master-key.aes"}' \
-blockdev '{"driver":"file","filename":"/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi-tpm.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"}' \
-blockdev '{"driver":"file","filename":"/etc/libvirt/qemu/nvram/691a4041-8a00-7085-a6e4-0ba8106d2f44_VARS-pure-efi-tpm.fd","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw","file":"libvirt-pflash1-storage"}' \
-machine pc-q35-7.1,usb=off,dump-guest-core=off,memory-backend=pc.ram,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-format \
-accel kvm \
-cpu host,migratable=on,topoext=on,hv-time=on,hv-relaxed=on,hv-vapic=on,hv-spinlocks=0x1fff,hv-vendor-id=none,host-cache-info=on,l3-cache=off \
-m 36864 \
-object '{"qom-type":"memory-backend-ram","id":"pc.ram","size":38654705664}' \
-overcommit mem-lock=off \
-smp 16,sockets=1,dies=1,cores=8,threads=2 \
-uuid 691a4041-8a00-7085-a6e4-0ba8106d2f44 \
-display none \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=36,server=on,wait=off \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=localtime \
-no-hpet \
-no-shutdown \
-boot strict=on \
-device '{"driver":"pcie-root-port","port":8,"chassis":1,"id":"pci.1","bus":"pcie.0","multifunction":true,"addr":"0x1"}' \
-device '{"driver":"pcie-root-port","port":9,"chassis":2,"id":"pci.2","bus":"pcie.0","addr":"0x1.0x1"}' \
-device '{"driver":"pcie-root-port","port":10,"chassis":3,"id":"pci.3","bus":"pcie.0","addr":"0x1.0x2"}' \
-device '{"driver":"pcie-root-port","port":11,"chassis":4,"id":"pci.4","bus":"pcie.0","addr":"0x1.0x3"}' \
-device '{"driver":"pcie-root-port","port":12,"chassis":5,"id":"pci.5","bus":"pcie.0","addr":"0x1.0x4"}' \
-device '{"driver":"pcie-root-port","port":13,"chassis":6,"id":"pci.6","bus":"pcie.0","addr":"0x1.0x5"}' \
-device '{"driver":"pcie-root-port","port":14,"chassis":7,"id":"pci.7","bus":"pcie.0","addr":"0x1.0x6"}' \
-device '{"driver":"pcie-root-port","port":15,"chassis":8,"id":"pci.8","bus":"pcie.0","addr":"0x1.0x7"}' \
-device '{"driver":"pcie-root-port","port":16,"chassis":9,"id":"pci.9","bus":"pcie.0","multifunction":true,"addr":"0x2"}' \
-device '{"driver":"pcie-root-port","port":17,"chassis":10,"id":"pci.10","bus":"pcie.0","addr":"0x2.0x1"}' \
-device '{"driver":"pcie-pci-bridge","id":"pci.11","bus":"pci.1","addr":"0x0"}' \
-device '{"driver":"qemu-xhci","p2":15,"p3":15,"id":"usb","bus":"pcie.0","addr":"0x7"}' \
-device '{"driver":"virtio-serial-pci","id":"virtio-serial0","bus":"pci.2","addr":"0x0"}' \
-blockdev '{"driver":"file","filename":"/mnt/user/domains/iso/Windows_10_Pro-20170304.iso","node-name":"libvirt-3-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-3-format","read-only":true,"driver":"raw","file":"libvirt-3-storage"}' \
-device '{"driver":"ide-cd","bus":"ide.0","drive":"libvirt-3-format","id":"sata0-0-0","bootindex":1}' \
-blockdev '{"driver":"file","filename":"/mnt/user/domains/drivers/virtio-win-0.1.225.iso","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-2-format","read-only":true,"driver":"raw","file":"libvirt-2-storage"}' \
-device '{"driver":"ide-cd","bus":"ide.1","drive":"libvirt-2-format","id":"sata0-0-1"}' \
-blockdev '{"driver":"file","filename":"/mnt/cache/domains/loaders/spaces_win_clover.img","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":false,"cache":{"direct":false,"no-flush":false},"driver":"raw","file":"libvirt-1-storage"}' \
-device '{"driver":"ide-hd","bus":"ide.2","drive":"libvirt-1-format","id":"sata0-0-2","bootindex":2,"write-cache":"on"}' \
-netdev tap,fd=37,id=hostnet0 \
-device '{"driver":"virtio-net","netdev":"hostnet0","id":"net0","mac":"52:54:00:1f:a0:a3","bus":"pci.3","addr":"0x0"}' \
-chardev pty,id=charserial0 \
-device '{"driver":"isa-serial","chardev":"charserial0","id":"serial0","index":0}' \
-chardev socket,id=charchannel0,fd=35,server=on,wait=off \
-device '{"driver":"virtserialport","bus":"virtio-serial0.0","nr":1,"chardev":"charchannel0","id":"channel0","name":"org.qemu.guest_agent.0"}' \
-chardev 'socket,id=chrtpm,path=/run/libvirt/qemu/swtpm/1-Windows 10-swtpm.sock' \
-tpmdev emulator,id=tpm-tpm0,chardev=chrtpm \
-device '{"driver":"tpm-tis","tpmdev":"tpm-tpm0","id":"tpm0"}' \
-device '{"driver":"usb-tablet","id":"input0","bus":"usb.0","port":"1"}' \
-audiodev '{"id":"audio1","driver":"none"}' \
-device '{"driver":"vfio-pci","host":"0000:33:00.0","id":"hostdev0","bus":"pci.4","addr":"0x0"}' \
-device '{"driver":"vfio-pci","host":"0000:33:00.1","id":"hostdev1","bus":"pci.5","addr":"0x0"}' \
-device '{"driver":"vfio-pci","host":"0000:2f:00.1","id":"hostdev2","bus":"pci.6","addr":"0x0"}' \
-device '{"driver":"vfio-pci","host":"0000:32:00.0","id":"hostdev3","bus":"pci.7","addr":"0x0"}' \
-device '{"driver":"vfio-pci","host":"0000:33:00.2","id":"hostdev4","bus":"pci.8","addr":"0x0"}' \
-device '{"driver":"vfio-pci","host":"0000:33:00.3","id":"hostdev5","bus":"pci.9","addr":"0x0"}' \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
char device redirected to /dev/pts/0 (label charserial0)

 

[johnsanc]

Anyone have any ideas on this? I also noticed that with 6.12.3 my Windows 10 VM starts but but there is no graphics nor does remote desktop work. I have to force stop it and restart in order for it to work properly. I never had these issues prior to 6.12.x. Was there some kind of change in the binding process that could be causing this?