October 22, 20241 yr Hi, Really hoping someone can help. I have been running a stable server for a long time, and I was running a VM gaming machine on a NVME but not properly passed though. I decided tonight to pass it through and reinstall windows. It went ok, install went fine and installed all the drivers etc. Was just starting to download some games and then the wholes server crashed! Restarted it and done the same thing! so I've left the VM off for now. 20:15:22 Jeeves kernel: pcieport 0000:00:1b.3: AER: Corrected error received: 0000:05:00.0 Oct 22 20:15:22 Jeeves kernel: vfio-pci 0000:05:00.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID) Oct 22 20:15:22 Jeeves kernel: vfio-pci 0000:05:00.0: device [1b21:2142] error status/mask=00000001/00002000 Oct 22 20:15:22 Jeeves kernel: vfio-pci 0000:05:00.0: [ 0] RxErr Oct 22 20:15:22 Jeeves kernel: pcieport 0000:00:1b.3: AER: Corrected error received: 0000:05:00.0 I found this in the logs and I've attahced the diags, has anyone seen this? Many thanks Dave jeeves-diagnostics-20241022-2020.zip
October 23, 20241 yr is the nvme drive in the top slot by the cpu / x16 GPU slot? That slot is PCIE iommu standalone and is recommend if you are going to pcie pass a nvme inot a VM Next you will need to check vm settings and VFIO Bind settings. Under Unraid > Setting > Vm reboot unraid then under tools > System devices You want to bind and have a green dot and a check mark next to your NVME drive, Click Main > Flash add this to your syslinx grub command: nvme_core.default_ps_max_latency_us=5500 default_hugepagesz=1G hugepagesz=1G transparent_hugepage=always pci=noaer pcie_aspm=off pcie_acs_override=downstream,multifunction reboot to apply Then in VM settings, add the pcie device at the bottom. This way the NVME is not touched by unraid at boot and is slated for the VM only as a bare metal HD. the "AER: Corr" is a pcie power error you may need to fix nvme latency and adjust power settings... Edited October 23, 20241 yr by bmartino1
October 23, 20241 yr Author Hi bmartino1, Thanks for the info, much appreciated. So I checked everything and found all the settings was like your images and the NVME was right at the bottom of the list. Added the command to the syslinx and rebooted but nothing showed up in the VM settings. It looks loke this: default menu.c32 menu title Lime Technology, Inc. prompt 0 timeout 50 label Unraid OS menu default kernel /bzimage append pcie_acs_override=downstream,multifunction isolcpus=2-7,10-15 initrd=/bzroot label Unraid OS GUI Mode kernel /bzimage append pcie_acs_override=downstream,multifunction isolcpus=2-7,10-15 initrd=/bzroot,/bzroot-gui label Unraid OS Safe Mode (no plugins, no GUI) kernel /bzimage append initrd=/bzroot unraidsafemode label Unraid OS GUI Safe Mode (no plugins) kernel /bzimage append initrd=/bzroot,/bzroot-gui unraidsafemode label Memtest86+ kernel /memtest label Unraid OS (GPUs left alone) menu default kernel /bzimage append initrd=/bzroot earlymodules=vfio-pci video=efifb:off,vesafb:off isolcpus=2-7,10-15 gfxpayload=text nvme_core.default_ps_max_latency_us=5500 default_hugepagesz=1G hugepagesz=1G transparent_hugepage=always pci=noaer pcie_aspm=off pcie_acs_override=downstream,multifunction Thank You Dave
October 23, 20241 yr whats the output of your vfio bind for pcie? would need picture. as we may have to add an advance modpobe to make sure the kerneal drive in use is vfio output of lspci -v ? would need unraid diagnostic to check and see settings. unraid > webui > tools > system devices there should be a log, is the nvme device green?
October 23, 20241 yr Author Hi bmartino1, Here is the vfio-pci log: Loading config from /boot/config/vfio-pci.cfg BIND=0000:03:00.0|1002:7480 0000:03:00.1|1002:ab30 0000:05:00.0|1b21:2142 0000:0a:00.0|144d:a808--- Processing 0000:03:00.0 1002:7480 Vendor:Device 1002:7480 found at 0000:03:00.0 IOMMU group members (sans bridges): /sys/bus/pci/devices/0000:03:00.0/iommu_group/devices/0000:03:00.0 /sys/bus/pci/devices/0000:03:00.0/iommu_group/devices/0000:03:00.1 Binding... success... Device 1002:7480 at 0000:03:00.0 bound to vfio-pci --- Processing 0000:03:00.1 1002:ab30 Vendor:Device 1002:ab30 found at 0000:03:00.1 IOMMU group members (sans bridges): /sys/bus/pci/devices/0000:03:00.1/iommu_group/devices/0000:03:00.0 /sys/bus/pci/devices/0000:03:00.1/iommu_group/devices/0000:03:00.1 Binding... 0000:03:00.0 already bound to vfio-pci 0000:03:00.1 already bound to vfio-pci success... Device 1002:ab30 at 0000:03:00.1 bound to vfio-pci --- Processing 0000:05:00.0 1b21:2142 Vendor:Device 1b21:2142 found at 0000:05:00.0 IOMMU group members (sans bridges): /sys/bus/pci/devices/0000:05:00.0/iommu_group/devices/0000:05:00.0 Binding... Unbound 0000:05:00.0 from xhci_hcd success... Device 1b21:2142 at 0000:05:00.0 bound to vfio-pci --- Processing 0000:0a:00.0 144d:a808 Vendor:Device 144d:a808 found at 0000:0a:00.0 IOMMU group members (sans bridges): /sys/bus/pci/devices/0000:0a:00.0/iommu_group/devices/0000:0a:00.0 Binding... success... Device 144d:a808 at 0000:0a:00.0 bound to vfio-pci --- Devices listed in /sys/bus/pci/drivers/vfio-pci: lrwxrwxrwx 1 root root 0 Oct 23 05:12 0000:03:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:00.0/0000:03:00.0 lrwxrwxrwx 1 root root 0 Oct 23 05:12 0000:03:00.1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:00.0/0000:03:00.1 lrwxrwxrwx 1 root root 0 Oct 23 05:12 0000:05:00.0 -> ../../../../devices/pci0000:00/0000:00:1b.3/0000:05:00.0 lrwxrwxrwx 1 root root 0 Oct 23 05:12 0000:0a:00.0 -> ../../../../devices/pci0000:00/0000:00:1d.0/0000:0a:00.0 vfio-pci binding complete Loading config from /boot/config/vfio-pci.cfg BIND=0000:03:00.0|1002:7480 0000:03:00.1|1002:ab30 0000:05:00.0|1b21:2142 0000:0a:00.0|144d:a808--- Processing 0000:03:00.0 1002:7480 Vendor:Device 1002:7480 found at 0000:03:00.0 IOMMU group members (sans bridges): /sys/bus/pci/devices/0000:03:00.0/iommu_group/devices/0000:03:00.0 /sys/bus/pci/devices/0000:03:00.0/iommu_group/devices/0000:03:00.1 Binding... success... Device 1002:7480 at 0000:03:00.0 bound to vfio-pci --- Processing 0000:03:00.1 1002:ab30 Vendor:Device 1002:ab30 found at 0000:03:00.1 IOMMU group members (sans bridges): /sys/bus/pci/devices/0000:03:00.1/iommu_group/devices/0000:03:00.0 /sys/bus/pci/devices/0000:03:00.1/iommu_group/devices/0000:03:00.1 Binding... 0000:03:00.0 already bound to vfio-pci 0000:03:00.1 already bound to vfio-pci success... Device 1002:ab30 at 0000:03:00.1 bound to vfio-pci --- Processing 0000:05:00.0 1b21:2142 Vendor:Device 1b21:2142 found at 0000:05:00.0 IOMMU group members (sans bridges): /sys/bus/pci/devices/0000:05:00.0/iommu_group/devices/0000:05:00.0 Binding... Unbound 0000:05:00.0 from xhci_hcd success... Device 1b21:2142 at 0000:05:00.0 bound to vfio-pci --- Processing 0000:0a:00.0 144d:a808 Vendor:Device 144d:a808 found at 0000:0a:00.0 IOMMU group members (sans bridges): /sys/bus/pci/devices/0000:0a:00.0/iommu_group/devices/0000:0a:00.0 Binding... success... Device 144d:a808 at 0000:0a:00.0 bound to vfio-pci --- Devices listed in /sys/bus/pci/drivers/vfio-pci: lrwxrwxrwx 1 root root 0 Oct 23 05:12 0000:03:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:00.0/0000:03:00.0 lrwxrwxrwx 1 root root 0 Oct 23 05:12 0000:03:00.1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:00.0/0000:03:00.1 lrwxrwxrwx 1 root root 0 Oct 23 05:12 0000:05:00.0 -> ../../../../devices/pci0000:00/0000:00:1b.3/0000:05:00.0 lrwxrwxrwx 1 root root 0 Oct 23 05:12 0000:0a:00.0 -> ../../../../devices/pci0000:00/0000:00:1d.0/0000:0a:00.0 vfio-pci binding complete It's currently green and looks like this: Thanks Dave
October 23, 20241 yr Author Solution Hi, Strangely I just added pcie_aspm=off to the Syslinux and its been stable since. This seems fixed now. Thanks
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.