• Posts

  • Joined

  • Last visited

Everything posted by sixsicsix

  1. I have a use case where I need to modify limits.conf on the Unraid host in order to allow my Docker containers to have more than the default number of files open. I've seen a lot of hacky methods tried online, but the proper way is to modify this file instead and reboot the Unraid host. It would be great if Unraid team were to move the contents of the directory into flash under /boot/config somewhere and symlink it from /etc/security -> /boot/config/security to allow changes. EDIT: As an aside, being able to apply this OS-wide could also help with SMB/shfs performance.
  2. Sorry to quote the entire post and stretch the thread but I wanted to be sure this information is saved for the future. Out of all the information in this thread, this post is the one that got me up and running with my Intel PRO/1000 quad-port NIC. The main difference with this post is that the network card's ports are placed on bus 0 and separated using slots. In the IOMMU groups, the card is split into two pairs on different buses, so I did the whole multifunction='on' thing for both pairs, and set the functions to 0 and 1 for each pair. I _also_ tried this by putting all four ports on the same bus, same slot, different functions, with multifunction on for the two leader ports (0,2 are leaders and 1,3 are secondary to them respectively). Or something. I have no idea what I'm doing but it works. So anyway here's my own XML: <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </source> <alias name='hostdev0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x04' slot='0x00' function='0x1'/> </source> <alias name='hostdev1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </source> <alias name='hostdev2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2' multifunction='on'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x05' slot='0x00' function='0x1'/> </source> <alias name='hostdev3'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x3'/> </hostdev> Anyone having trouble should start here and work backwards ^^
  3. Let's start with my setup: Motherboard: ASUS X470-F Gaming (Socket AM4) CPU: AMD Ryzen 7 2700X (does not have integrated graphics) Memory: 32GB Storage: 2 x 512GB NVME SSDs (no RAID) GPU0 (top PCIE slot): EVGA GTX 1050 GPU1 (secondary PCIE slot): EVGA GTX 1080Ti Hybrid Network: Single onboard gigabit nic Everyone's got something different, but the above it working for me just fine. Here are the hurdles I faced during this journey: * My Native Instruments Komplete Audio 6 would panic and reset itself in a Windows 10 guest. - To fix this, I had to look at my IOMMU groups and find a USB controller that doesn't share the same IOMMU group as anything else important, then "stub" it to keep the Unraid underlying OS from claiming it during boot. * As a result of the above, I became limited in which USB ports I could use for VMs - But, to be fair, I can now unplug/replug anything I want on the Windows 10 guest. That's the benefit of binding an entire USB controller. * I found I was completely unable to pass through my primary graphics card to a Linux guest. - To fix this, I had to stub the card, AND - and this is the most important bit - I had to disable EFI frame buffering (or something like that). These bits are all done in the Unraid kernel startup line. * My Linux guest had stuttery, demonic audio over HDMI and video would lag when audio was playing. - To fix this, I had to enable some intel sound options inside the Linux guest, which I found amusing since nothing in my machine is Intel (something about the virt layer needing this). Let's dig into each. USB audio interface runs terribly when passed through This happens because of the emulation layer. The only way will be to find out which USB controller can be stubbed. To figure out which one, get on Unraid and open a terminal and run: lspci | grep USB Take note of the IDs there, then run: for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d);do echo "IOMMU group $(basename "$iommu_group")"; for device in $(\ls -1 "$iommu_group"/devices/); do if [[ -e "$iommu_group"/devices/"$device"/reset ]]; then echo -n "[RESET]"; fi; echo -n $'\t';lspci -nns "$device"; done; done In this big list you're looking for something like this: IOMMU group 20 0b:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] USB 3.0 Host controller [1022:145f] Notice how this USB controller is the ONLY device in this IOMMU group (which just so happens to be group 20 on my machine). This means that we should be able to completely isolate it from the Unraid host OS so we can let the VM claim it completely and without question. To stub it, we modify the kernel startup line. You can either use terminal and modify /boot/syslinux/syslinux.cfg or you can use the Unraid web UI and go to Main -> click "Flash" under 'Boot Device' -> then edit the 'Unraid OS' section under 'Syslinux Configuration' Here's what it looks like when I stub ONLY the above USB controller: kernel /bzimage append initrd=/bzroot vfio-pci.ids=1022:145f Save that change, then reboot Unraid. You can now edit your VM and at the bottom, uncheck the USB devices and instead pass through the USB hub itself. That should fix it. I can't pass through my GPU in the top slot because Unraid is using it Correct, but we can be aggressive about it and do it anyway. Start by finding the device IDs using lspci: root@funraid:~# lspci | grep VGA -A1 09:00.0 VGA compatible controller: NVIDIA Corporation GP107 [GeForce GTX 1050 3GB] (rev a1) 09:00.1 Audio device: NVIDIA Corporation GP107GL High Definition Audio Controller (rev a1) 0a:00.0 VGA compatible controller: NVIDIA Corporation GP102 [GeForce GTX 1080 Ti] (rev a1) 0a:00.1 Audio device: NVIDIA Corporation GP102 HDMI Audio Controller (rev a1) Above, you can see that my GTX 1050 and its associated audio interface are on 09:00.(0|1) So now I need to find out the device IDs: root@funraid:~# lspci -n | grep 09:00. 09:00.0 0300: 10de:1c83 (rev a1) 09:00.1 0403: 10de:0fb9 (rev a1) Awesome. Let's stub those, too... but remember! We need to also disable EFI frame buffering (I think that's what it's called). This means that when you boot Unraid next, you WILL NOT have any console output after the main bootloader (the blue screen) completes - your main GPU that you're passing through (in this case, my GTX 1050) will appear to freeze at that bootloader. If you see the OS even remotely start to boot, you didn't do this right. In addition to the USB hub, we'll also stub the GPU and its audio interface and disable EFI FB: kernel /bzimage append initrd=/bzroot vfio-pci.ids=1022:145f,10de:1c83,10de:0fb9 video=efifb:off Save that and reboot Unraid, and all will be well. You might - and this is very unlikely, but - you might need to supply a VGA bios from techpowerup for your card, or worst case put the card you are trying to use here into PCIE slot 2 and dump its BIOS from the Unraid OS, but I really don't think you'll have to. My Linux guest has demonic, stuttery audio and video when trying to play videos and audio is coming over HDMI through my nVidia card. Me too, and I dug for a bit and found out that I can simply give the Linux guest a modprobe line to clear up this behavior. Inside of the Linux guest's OS (NOT UNRAID!), create a conf file in /etc/modprobe.d/... and use the EXACT same data I use below. Here's mine: [root@archer ~]# cat /etc/modprobe.d/snd-hda-intel.conf options snd-hda-intel enable_msi=1 That's it. Literally one line. Yeah, you have to type that in word for word. Don't worry, I know, you're not running anything Intel - but trust me, it works. Save that and reboot the Linux guest OS. Everything should be in order. I really hope this all helps someone. Thanks! :o)