sixsicsix

Members
  • Posts

    3
  • Joined

  • Last visited

Posts posted by sixsicsix

  1. I have a use case where I need to modify limits.conf on the Unraid host in order to allow my Docker containers to have more than the default number of files open. I've seen a lot of hacky methods tried online, but the proper way is to modify this file instead and reboot the Unraid host. It would be great if Unraid team were to move the contents of the directory into flash under /boot/config somewhere and symlink it from /etc/security -> /boot/config/security to allow changes.

     

    EDIT: As an aside, being able to apply this OS-wide could also help with SMB/shfs performance.

  2. On 1/27/2020 at 1:45 AM, uek2wooF said:

    I had a heck of a time getting pfsense to see my 4 port intel nic.  Linux could see it fine.  Anyway, I ended up modifying the XML like this:

     

        <hostdev mode='subsystem' type='pci' managed='yes'>
          <driver name='vfio'/>
          <source>
            <address domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
          </source>
          <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
        </hostdev>
        <hostdev mode='subsystem' type='pci' managed='yes'>
          <driver name='vfio'/>
          <source>
            <address domain='0x0000' bus='0x06' slot='0x00' function='0x1'/>
          </source>
          <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
        </hostdev>
        <hostdev mode='subsystem' type='pci' managed='yes'>
          <driver name='vfio'/>
          <source>
            <address domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
          </source>
          <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0' multifunction='on'/>
        </hostdev>
        <hostdev mode='subsystem' type='pci' managed='yes'>
          <driver name='vfio'/>
          <source>
            <address domain='0x0000' bus='0x07' slot='0x00' function='0x1'/>
          </source>
          <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x1'/>
        </hostdev>

     

     

    Notice now the slot 5s do a function 0x0 and 0x1 and also the same for the slot 6s.  Where before it was just slot 5 6 7 8 and 0x0 for the function.  I hope this makes sense.  The multifunction parts got added on automatically for me.

     

    I did pci-stub.ids=8086:10bc in the append line in syslinux config (from flash link on main page) to get the card ready for passthrough.

     

    Also if you want to bridge br0 or virbr0 make sure it is using <model type='vmxnet3'/>

     

    Hope this helps.  Good luck.

    Sorry to quote the entire post and stretch the thread but I wanted to be sure this information is saved for the future. Out of all the information in this thread, this post is the one that got me up and running with my Intel PRO/1000 quad-port NIC. The main difference with this post is that the network card's ports are placed on bus 0 and separated using slots. In the IOMMU groups, the card is split into two pairs on different buses, so I did the whole multifunction='on' thing for both pairs, and set the functions to 0 and 1 for each pair.

     

    I _also_ tried this by putting all four ports on the same bus, same slot, different functions, with multifunction on for the two leader ports (0,2 are leaders and 1,3 are secondary to them respectively). Or something. I have no idea what I'm doing but it works. :)

     

    So anyway here's my own XML:

     

        <hostdev mode='subsystem' type='pci' managed='yes'>
          <driver name='vfio'/>
          <source>
            <address domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
          </source>
          <alias name='hostdev0'/>
          <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
        </hostdev>
        <hostdev mode='subsystem' type='pci' managed='yes'>
          <driver name='vfio'/>
          <source>
            <address domain='0x0000' bus='0x04' slot='0x00' function='0x1'/>
          </source>
          <alias name='hostdev1'/>
          <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
        </hostdev>
        <hostdev mode='subsystem' type='pci' managed='yes'>
          <driver name='vfio'/>
          <source>
            <address domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
          </source>
          <alias name='hostdev2'/>
          <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2' multifunction='on'/>
        </hostdev>
        <hostdev mode='subsystem' type='pci' managed='yes'>
          <driver name='vfio'/>
          <source>
            <address domain='0x0000' bus='0x05' slot='0x00' function='0x1'/>
          </source>
          <alias name='hostdev3'/>
          <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x3'/>
        </hostdev>

    Anyone having trouble should start here and work backwards ^^

    • Like 1
    • Thanks 1
  3. Let's start with my setup:

     

    Motherboard: ASUS X470-F Gaming (Socket AM4)
    CPU: AMD Ryzen 7 2700X (does not have integrated graphics)
    Memory: 32GB
    Storage: 2 x 512GB NVME SSDs (no RAID)
    GPU0 (top PCIE slot): EVGA GTX 1050
    GPU1 (secondary PCIE slot): EVGA GTX 1080Ti Hybrid
    Network: Single onboard gigabit nic

     

    Everyone's got something different, but the above it working for me just fine.

     

    Here are the hurdles I faced during this journey:

     

    * My Native Instruments Komplete Audio 6 would panic and reset itself in a Windows 10 guest.
    - To fix this, I had to look at my IOMMU groups and find a USB controller that doesn't share the same IOMMU group as anything else important, then "stub" it to keep the Unraid underlying OS from claiming it during boot.

     

    * As a result of the above, I became limited in which USB ports I could use for VMs
    - But, to be fair, I can now unplug/replug anything I want on the Windows 10 guest.  That's the benefit of binding an entire USB controller.

     

    * I found I was completely unable to pass through my primary graphics card to a Linux guest.
    - To fix this, I had to stub the card, AND - and this is the most important bit - I had to disable EFI frame buffering (or something like that).  These bits are all done in the Unraid kernel startup line.

     

    * My Linux guest had stuttery, demonic audio over HDMI and video would lag when audio was playing.
    - To fix this, I had to enable some intel sound options inside the Linux guest, which I found amusing since nothing in my machine is Intel (something about the virt layer needing this).

     

    Let's dig into each.

     

    USB audio interface runs terribly when passed through

     

    This happens because of the emulation layer.  The only way will be to find out which USB controller can be stubbed.

     

    To figure out which one, get on Unraid and open a terminal and run:

    lspci | grep USB
    

    Take note of the IDs there, then run:

    for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d);do echo "IOMMU group $(basename "$iommu_group")"; for device in $(\ls -1 "$iommu_group"/devices/); do if [[ -e "$iommu_group"/devices/"$device"/reset ]]; then echo -n "[RESET]"; fi; echo -n $'\t';lspci -nns "$device"; done; done
    

    In this big list you're looking for something like this:

    IOMMU group 20
    0b:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] USB 3.0 Host controller [1022:145f]
    

    Notice how this USB controller is the ONLY device in this IOMMU group (which just so happens to be group 20 on my machine).  This means that we should be able to completely isolate it from the Unraid host OS so we can let the VM claim it completely and without question.

     

    To stub it, we modify the kernel startup line.  You can either use terminal and modify /boot/syslinux/syslinux.cfg or you can use the Unraid web UI and go to Main -> click "Flash" under 'Boot Device' -> then edit the 'Unraid OS' section under 'Syslinux Configuration'

     

    Here's what it looks like when I stub ONLY the above USB controller:

    kernel /bzimage
    append initrd=/bzroot vfio-pci.ids=1022:145f
    

    Save that change, then reboot Unraid.  You can now edit your VM and at the bottom, uncheck the USB devices and instead pass through the USB hub itself.  That should fix it.

     

    I can't pass through my GPU in the top slot because Unraid is using it

     

    Correct, but we can be aggressive about it and do it anyway.  Start by finding the device IDs using lspci:

    root@funraid:~# lspci | grep VGA -A1
    09:00.0 VGA compatible controller: NVIDIA Corporation GP107 [GeForce GTX 1050 3GB] (rev a1)
    09:00.1 Audio device: NVIDIA Corporation GP107GL High Definition Audio Controller (rev a1)
    0a:00.0 VGA compatible controller: NVIDIA Corporation GP102 [GeForce GTX 1080 Ti] (rev a1)
    0a:00.1 Audio device: NVIDIA Corporation GP102 HDMI Audio Controller (rev a1)

    Above, you can see that my GTX 1050 and its associated audio interface are on 09:00.(0|1)

     

    So now I need to find out the device IDs:

    root@funraid:~# lspci -n | grep 09:00.
    09:00.0 0300: 10de:1c83 (rev a1)
    09:00.1 0403: 10de:0fb9 (rev a1)

    Awesome.  Let's stub those, too... but remember!  We need to also disable EFI frame buffering (I think that's what it's called).  This means that when you boot Unraid next, you WILL NOT have any console output after the main bootloader (the blue screen) completes - your main GPU that you're passing through (in this case, my GTX 1050) will appear to freeze at that bootloader.  If you see the OS even remotely start to boot, you didn't do this right.

     

    In addition to the USB hub, we'll also stub the GPU and its audio interface and disable EFI FB:

    kernel /bzimage
    append initrd=/bzroot vfio-pci.ids=1022:145f,10de:1c83,10de:0fb9 video=efifb:off
    

    Save that and reboot Unraid, and all will be well.  You might - and this is very unlikely, but - you might need to supply a VGA bios from techpowerup for your card, or worst case put the card you are trying to use here into PCIE slot 2 and dump its BIOS from the Unraid OS, but I really don't think you'll have to.

     

    My Linux guest has demonic, stuttery audio and video when trying to play videos and audio is coming over HDMI through my nVidia card.

     

    Me too, and I dug for a bit and found out that I can simply give the Linux guest a modprobe line to clear up this behavior.

     

    Inside of the Linux guest's OS (NOT UNRAID!), create a conf file in /etc/modprobe.d/... and use the EXACT same data I use below.  Here's mine:

    [root@archer ~]# cat /etc/modprobe.d/snd-hda-intel.conf 
    options snd-hda-intel enable_msi=1
    

    That's it.  Literally one line.  Yeah, you have to type that in word for word.  Don't worry, I know, you're not running anything Intel - but trust me, it works.

    Save that and reboot the Linux guest OS.  Everything should be in order.

     

    I really hope this all helps someone.  Thanks! :o)

    • Like 2