Jump to content

bastl

Members
  • Posts

    1,267
  • Joined

  • Last visited

  • Days Won

    3

Posts posted by bastl

  1. On 4/27/2020 at 5:52 AM, RickD00 said:

    I tried it with VNC it was on UEFI Shell window

    Type "exit" in the UEFI shell window and "continue" in the BIOS that opens. VM reboots and now you should see the the hint to "press any button to boot from the install media". Keep in mind this hint only shows a couple seconds and without pressing any button you will end in the UEFI shell again. Starting a VM and not quickly enough connecting with VNC you will miss this part. Also your boot order might be wrong. Post the xml of the VM.

  2. 7 hours ago, Madman2012 said:

    I planned to mount the Synology drives inside on Ubuntu but do it in a VM as detailed in Synology's recovery instructions

    Setting up a Ubuntu VM and passing through the physical drives shouldn't be a big deal.

     

    Create a share in Unraid where you wanna copy your data to. Depending on the amount of data you wanna recover, you need space on the Unraid array. Having the array up and running with parity already configured is the way I would go. Keep in mind, no disk in the array can be bigger than the parity drive. So don't start with a let's say 1TB parity + 1TB+500GB+250GB array drive when you wanna add bigger drives to the array anyways.

     

    Setup the VM first without any drives from the Synology attached. Make sure it is working and has access to the unraid shares. Shutdown the VM and attach the drives to the VM. You have a couple options. You can either passthrough the controller where the drives are connected to in case you have an extra raid card or hba or you can pass them through by-id. Let me explain.

     

    First check under Unassigned devices the ID/Name of the drives. Make sure they are not mounted and make notes for all drive members of that Synology SHR raid. For example

    grafik.png.bf4803502911ba72ee830411dc3b27b8.png

    Open up a terminal and navigate to /dev/disk/by-id/ and list all the devices

    grafik.png.cf0c23fa0e5803e7fcc60021be692fd6.png

     

    You should see all drives and partitions in your Unraid box. In my example you see 2 entries for the drive we wanna use. 

     

    "ata-ST2000LM007-1R8174_WDZ5J72F" is the full device

    "ata-ST2000LM007-1R8174_WDZ5J72F-part1" is only one partition of the disk.

     

    You need the full drive passed through to the VM. Make notes for all the drives you need and go to the next step. Shutdown the VM and edit the template.

    Click the small + button below the already existing vdisk and add the first disk. Change it from "Auto" to "Manual" and enter the full path

     

    grafik.png.a2140dcad6944bb10e401b644e2fb962.png

     

    Again, don't passthrough individual partitions, only use the full disk ID. Repeat the steps and add all the SHR drive members. If you start your VM you should have access to all the drives and can use the Synology tutorial to recover the data and copy it to the Unraid share.

     

  3. @Leoyzen Thanks for that hint, but it is not in the same group with other devices.

    IOMMU group 29:	[1022:148a] 22:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
    IOMMU group 30:	[1022:1485] 23:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
    IOMMU group 31:	[1022:1486] 23:00.1 Encryption controller: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP
    IOMMU group 32:	[1022:148c] 23:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller
    IOMMU group 33:	[1022:1487] 23:00.4 Audio device: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller
    IOMMU group 34:	[1022:1482] 40:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
    IOMMU group 35:	[1022:1483] 40:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
    IOMMU group 36:	[1022:1483] 40:01.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
    IOMMU group 37:	[1022:1482] 40:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
    IOMMU group 38:	[1022:1482] 40:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
    IOMMU group 39:	[1022:1483] 40:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge

     

  4. Not really sure what you did. I guess you ticked a box in the webui for the VM, right? The only USB device connected to the VM is the following in the XML

        <hostdev mode='subsystem' type='usb' managed='no'>
          <source>
            <vendor id='0x046d'/>
            <product id='0xc063'/>
          </source>
          <address type='usb' bus='0' port='1'/>
        </hostdev>

    Check under tools > system devices > USB devices what this device is (046d:c063)

    You can remove it from the XML by deleting the part above.

    5 hours ago, clambert said:

    I enabled a connected mouse on my laptop

    How did you do that? You only can enable devices physically connected to unraid itself.

     

    • Like 1
  5. In case someone stumbles across this and needs the info. I recently upgraded from a 1950x to a 3960x on a Aorus Extreme TRX40 and struggled the last couple of days getting my VMs back up running. Passing through both GPUs (1080ti, 1050ti) to different VMs, no issues, onboard USB controller also no issue. The problems started with the onboard audio controller. The common way is to find the controller in the devices list, stub/bind them and you should be good to go. In my case the controller is reported as

    [1022:1487] 23:00.4 Audio device: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller

    BUT

    this isn't the real audio controller that can be passed to a VM. I tried the 6.9 beta 1, tried different patched kernels for 6.8.3 with fixes for 3rd gen Ryzens (AMD onboard audio/usb controller flr patch, Vega-Navi-Patch etc.). During all my testings I completely missed, that this board provides 2 separate USB audio devices. There is no need for passing through the Starship/Matisse controller or using a patched kernel. It works straight out of the box with Unraid 6.8.3. No ACS patch or unsafe interrupts needed. Simply pick one of the 2 USB audio devices in the VM template and you're good to go. Windows will recognize it as "Realtek USB Audio" and on Linux (Zorin15.1) it also works straight out of the box. 👍

     

    grafik.png.677d7a041b2034e9b966ea244972a8a5.png

     

    The first device is for the front panel audio and the second one for the back panel 5.1/7.1 links. And guess what....

    you can use them in 2 different VMs and run them simultaneously without issues or any extra tweaking needed. 😍

     

    For reference my initial posting, trying to get help in the custom kernel thread:

     

  6. How could I missed that????WTF!!1!!!1

     

    Ok, so during all my testings I completely missed that the Aorus Extreme TRX40 provides 2 USB audio devices

     

    grafik.png.1f557b23f927e5d098b95fbcad07b67f.png

     

    The first one is for the front panel audio and the second for the rear 5.1/7.1 audio links. I always tried to passthrough the onboard "[AMD] Starship/Matisse HD Audio Controller" without success. 😑

     

    Both these controllers are connected to the chipset and can be passed through to different VMs at the same time.

     

    RTFM 😂

     

    grafik.png.5c07f883f8e03853634eda09ac997bdb.png

     

    I reverted back to default 6.8.3 kernel and had no issues to passthrough both controllers to different vms, running them simultaneously with 5.1 speakers at the back and headset attached to the front without any issues. No patched kernel needed 👍

     

    • Like 1
  7. @Leoyzen First of all thanks for your work to provide custom kernels. 👍

     

    Let me explain the issue i have. I upgraded from a TR4 1950x to a Threadripper 3960x on an Gigabyte Aorus Extreme TRX40 and kinda see the same issues people having with passing through the onboard audio. In my case it's the following device:

    IOMMU group 42:	[1022:1487] 23:00.4 Audio device: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller
    root@UNRAID:~# lspci -v -s 23:00.4
    23:00.4 Audio device: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller
            Subsystem: Advanced Micro Devices, Inc. [AMD] Device d102
            Flags: bus master, fast devsel, latency 0, IRQ 154
            Memory at b1400000 (32-bit, non-prefetchable) [size=32K]
            Capabilities: [48] Vendor Specific Information: Len=08 <?>
            Capabilities: [50] Power Management version 3
            Capabilities: [64] Express Endpoint, MSI 00
            Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
            Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
            Capabilities: [150] Advanced Error Reporting
            Capabilities: [2a0] Access Control Services
            Capabilities: [370] Transaction Processing Hints
            Kernel driver in use: vfio-pci

    With the default 6.8.3 Kernel I have the same issue like other users with an 3rd gen Ryzen. Passing through the onboard audio controller to a VM and starting it up causing the server to freeze/lockup. No matter if I bind the device to vfio or not. Only way to get the server back to a working state is to force a shutdown.

    Short snippet in the state when the server locks up. I can't pull the full diagnostic at this point, not via ssh nor via webui.

    Apr 25 15:46:29 UNRAID kernel: vfio-pci 0000:23:00.4: not ready 1023ms after FLR; waiting
    Apr 25 15:46:31 UNRAID kernel: vfio-pci 0000:23:00.4: not ready 2047ms after FLR; waiting
    Apr 25 15:46:34 UNRAID kernel: vfio-pci 0000:23:00.4: not ready 4095ms after FLR; waiting
    Apr 25 15:46:39 UNRAID kernel: vfio-pci 0000:23:00.4: not ready 8191ms after FLR; waiting
    Apr 25 15:46:48 UNRAID kernel: vfio-pci 0000:23:00.4: not ready 16383ms after FLR; waiting
    Apr 25 15:47:06 UNRAID kernel: vfio-pci 0000:23:00.4: not ready 32767ms after FLR; waiting
    Apr 25 15:47:44 UNRAID kernel: vfio-pci 0000:23:00.4: not ready 65535ms after FLR; giving up
    Apr 25 15:48:16 UNRAID nginx: 2020/04/25 15:48:16 [error] 11486#11486: *1680 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 10.0.0.7, server: , request: "POST /plugins/dynamix.vm.manager/include/VMajax.php HTTP/2.0", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "unraid.local", referrer: "https://unraid.local/VMs"
    Apr 25 15:48:43 UNRAID kernel: rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
    Apr 25 15:48:43 UNRAID kernel: rcu: 	31-....: (59 ticks this GP) idle=a66/1/0x4000000000000000 softirq=131201/131201 fqs=14340 
    Apr 25 15:48:43 UNRAID kernel: rcu: 	(detected by 32, t=60002 jiffies, g=412949, q=631295)
    Apr 25 15:48:43 UNRAID kernel: Sending NMI from CPU 32 to CPUs 31:
    Apr 25 15:48:43 UNRAID kernel: NMI backtrace for cpu 31
    Apr 25 15:48:43 UNRAID kernel: CPU: 31 PID: 47678 Comm: qemu-system-x86 Tainted: G           O      4.19.107-Unraid #1
    Apr 25 15:48:43 UNRAID kernel: Hardware name: Gigabyte Technology Co., Ltd. TRX40 AORUS XTREME/TRX40 AORUS XTREME, BIOS F4d 03/05/2020
    Apr 25 15:48:43 UNRAID kernel: RIP: 0010:pci_mmcfg_read+0x98/0xa6
    Apr 25 15:48:43 UNRAID kernel: Code: 83 fe 02 74 15 41 83 fe 04 74 1a 41 ff ce 75 1d 48 01 d8 8a 00 0f b6 c0 eb 10 48 01 d8 66 8b 00 0f b7 c0 eb 05 48 01 d8 8b 00 <89> 45 00 31 c0 5b 5d 41 5c 41 5d 41 5e c3 81 fa ff 00 00 00 41 56
    Apr 25 15:48:43 UNRAID kernel: RSP: 0018:ffffc900122e3cc8 EFLAGS: 00000286
    Apr 25 15:48:43 UNRAID kernel: RAX: 00000000ffffffff RBX: 0000000000000ffc RCX: 0000000000000ffc
    Apr 25 15:48:43 UNRAID kernel: RDX: 000000000000007f RSI: 0000000000000023 RDI: ffffc90008000000
    Apr 25 15:48:43 UNRAID kernel: RBP: ffffc900122e3cfc R08: 0000000000000004 R09: ffffc900122e3cfc
    Apr 25 15:48:43 UNRAID kernel: R10: 0000000000000004 R11: 0000000000000084 R12: 0000000002304000
    Apr 25 15:48:43 UNRAID kernel: R13: 0000000000000004 R14: 0000000000000004 R15: ffff888ebcb25b40
    Apr 25 15:48:43 UNRAID kernel: FS:  00001524a07b4e00(0000) GS:ffff88902d5c0000(0000) knlGS:0000000000000000
    Apr 25 15:48:43 UNRAID kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
    Apr 25 15:48:43 UNRAID kernel: CR2: 000014f97fed62b0 CR3: 0000000f3cf9a000 CR4: 0000000000340ee0
    Apr 25 15:48:43 UNRAID kernel: Call Trace:
    Apr 25 15:48:43 UNRAID kernel: pci_bus_read_config_dword+0x44/0x65
    Apr 25 15:48:43 UNRAID kernel: pci_find_next_ext_capability+0x9e/0xc9
    Apr 25 15:48:43 UNRAID kernel: ? _raw_spin_unlock_irqrestore+0xc/0x12
    Apr 25 15:48:43 UNRAID kernel: pci_restore_vc_state+0x20/0x5c
    Apr 25 15:48:43 UNRAID kernel: pci_restore_state+0xd0/0x26e
    Apr 25 15:48:43 UNRAID kernel: pci_dev_restore+0x18/0x34
    Apr 25 15:48:43 UNRAID kernel: pci_try_reset_function+0x3f/0x4e
    Apr 25 15:48:43 UNRAID kernel: vfio_pci_open+0x7e/0x3af
    Apr 25 15:48:43 UNRAID kernel: vfio_group_fops_unl_ioctl+0x355/0x42e
    Apr 25 15:48:43 UNRAID kernel: vfs_ioctl+0x19/0x26
    Apr 25 15:48:43 UNRAID kernel: do_vfs_ioctl+0x533/0x55d
    Apr 25 15:48:43 UNRAID kernel: ? __se_sys_newlstat+0x48/0x6b
    Apr 25 15:48:43 UNRAID kernel: ksys_ioctl+0x37/0x56
    Apr 25 15:48:43 UNRAID kernel: __x64_sys_ioctl+0x11/0x14
    Apr 25 15:48:43 UNRAID kernel: do_syscall_64+0x57/0xf2
    Apr 25 15:48:43 UNRAID kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
    Apr 25 15:48:43 UNRAID kernel: RIP: 0033:0x1524a1fa24b7
    Apr 25 15:48:43 UNRAID kernel: Code: 00 00 90 48 8b 05 d9 29 0d 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d a9 29 0d 00 f7 d8 64 89 01 48
    Apr 25 15:48:43 UNRAID kernel: RSP: 002b:00007fff5878e818 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
    Apr 25 15:48:43 UNRAID kernel: RAX: ffffffffffffffda RBX: 000015241dfc14e0 RCX: 00001524a1fa24b7
    Apr 25 15:48:43 UNRAID kernel: RDX: 000015241c9a1520 RSI: 0000000000003b6a RDI: 000000000000001b
    Apr 25 15:48:43 UNRAID kernel: RBP: 00001524181605c0 R08: 000015241c9a1520 R09: 00007fff5878c783
    Apr 25 15:48:43 UNRAID kernel: R10: 0000000000000018 R11: 0000000000000246 R12: 00001524181605c0
    Apr 25 15:48:43 UNRAID kernel: R13: 000015241c9a1520 R14: 00007fff5878fa30 R15: 000015241dfc0c00

     

    At this point, the server becomes basically unusable.

     

    If I use your custom Kernel "6.8.3-5.5.8-2" you posted 2 pages back and the kernel agument "pcie_no_flr"

    append pcie_no_flr=1022:1487 vfio-pci.ids=1022:1487,1b21:2142 isolcpus=12-23,36-47 initrd=/bzroot

    I'am able to passthrough the device to a Windows VM without freezing the whole server, BUT the device isn't shown in the device manager as new device. 1b21:2142 is one of the USB onboard controllers and passthrough is working fine. Drivers are installed in a bare metal Windows install on an NVME which I use in this VM. I can't trigger Windows to show it in the device list. No errors in the VM log except for some warning:

    2020-04-25T15:24:26.974831Z qemu-system-x86_64: vfio: Cannot reset device 0000:23:00.4, depends on group 39 which is not owned.
    2020-04-25T15:24:29.155826Z qemu-system-x86_64: vfio: Cannot reset device 0000:23:00.4, depends on group 39 which is not owned.
    IOMMU group 39:	[1022:1485] 23:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP

    The "Starship/Matisse Reserved SPP" isn't bind to vfio or used for passthrough either. Do I have to bind and passthrough that device as well???

     

    Any idea for an workaround to have the onboard audio device to show up in the Windows VM? I really wanna figure this out and maybe try to help @limetech to implement the fix for the onboard audio issue in the next builds hopefully. I'am willing to test to find a solution for this. From all I've read so far, a couple people have issues passing these devices to a VM on the latest AMD platforms. Maybe we can workout a solution thats worth to implement in the future Unraid builds. I've not ried the 6.9 beta build yet. It's kinda the next step for me to try, but I think this won't make any difference.

     

    Quote

    This initial -beta1 release is exactly the same code set of 6.8.3 except that the Linux kernel has been updated to 5.5.8 (latest stable as of this date).  We have only done basic testing on this release; normally we would not release 'beta' code but some users require the hardware support offered by the latest kernel along with security updates added into 6.8.

     

    I'll report back soon. Thanks for the help ❤️

     

    syslog_683_default_kernel.txt syslog_patched_5.5.8-2_kernel.txt Win10_VM_log_patched.txt 01_W10.xml IOMMU.txt

     

     

     

    EDIT:

     

    Tried the 6.9 beta 1, same issue as with unpatched 6.8.3 kernel. Same freezes, same errors. No passthrough of the onboard audio possible. After reverting back to 6.8.3 with the patched 5.5.8-2 Kernel I tried a couple combinations of binding and passing through the device in group 39 with no success. Either the "FLR; waiting" error occures and server freezes or VM starts fine without any extra audio device showing up in the device manager.

     

     

     

  8. @XiuzSu Where do you got that Windows Iso from? From the naming it looks like a modified version. You can bake in drivers into an ISO and maybe the one which is included has some issues. For example a guy a couple days ago had issues installing a newer Nvidia driver into a 1809 Windows install. If the author of that image used an old image with newer not supported drivers this might can be your issue. Try it with an up-to-date ISO directly from Microsoft.

  9. 12 hours ago, joecool169 said:

    device, stuck in D3

    This indicates the device is stuck in a lower power setting and isn't fully reset. The only way to recover from it is to reboot the server. After reboot the passthrough might only work once and a restart or power down and power up of the VM won't brings back the card and you have to restart the server again. This is usually an issue on newer AMD cards starting with the Navi series. Your card is a really old model and I'am not sure if someone managed it to get it working somehow. I guess not.

  10. 16 hours ago, Han1 said:

    would it be possible to put the PGU in another PCIE slot?

    Worth a try.

     

    Is your Intel GPU enabled and set to primary GPU? If not enable it and set it as primary in the BIOS so Unraid itself uses it. In case unraid picks up your 1080 as primary all the time, put it down in another slot and use another GPU in the first slot. In my case 1080ti in first slot and 1050ti in third slot are both able to passthrough. If I switch them only the 1080ti in third slot as working. Remember I'am on AMD TR4 and have no GPU onboard or in the CPU itself.

     

    Using another slot may change your IOMMU groupings and also the IDs might change. Check if using another slot may throw the card in a group with other devices. Also keep in mind, often the lower slots are shared with other devices like USB or Sata controllers.

  11. Remove the following section

        <hostdev mode='subsystem' type='pci' managed='yes'>
          <driver name='vfio'/>
          <source>
            <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
          </source>
          <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
        </hostdev>
        

     

    3 hours ago, Underscoreus said:

    Device 0000:01:00.1 not found: could not access /sys/bus/pci/devices/0000:01:00.1/config: No such file or directory

    For explanation, the error is complaining to not find the device on bus 01, slot 00 with function 1. As you can see in the <source> section above this is the culprit

    • Like 1
    • Thanks 2
  12. 3 hours ago, Underscoreus said:

    I might have had the GPU checked

    You propably had. When editing your VM you can switch to xml view, top right corner, and remove the section. Post your xml and the VM logs. In case you don't know which hostdev section to remove. Are there any other devices passed through to the VM?

  13. 2 minutes ago, mwoods98 said:

    Did you just remove the device that it was sharing to fix it?

    By accident I passed through the onboard controller where the Unraid flash drive is plugged in to the VM. Starting the VM removed it from the host and I only could fix it by restarting the server. By using another controller for the VM with other ports on the board the issue never happened again.

     

    In your case I would try to use another port for the Unraid USB flash drive, USB 2 if you have multiple + restart of the server. If the same issue appears again check the flash drive on a Windows machine for errors and let it repair.

×
×
  • Create New...