AnnabellaRenee87 Posted October 15, 2018 Share Posted October 15, 2018 White rebooting I seen this screen for a few moments before getting back to the previous screenshot too. Sent from my SM-G955U using Tapatalk Quote Link to comment
AnnabellaRenee87 Posted October 15, 2018 Share Posted October 15, 2018 Sorry for posting so much, I was able to downgrade with iLO. I used Midnight Commander and moved all the files from the "Previous" folder back in to the /boot folder and was able to get back to 6.6.0 My Web GUI wouldn't load when I attempted 6.6.2 Quote Link to comment
1812 Posted October 16, 2018 Author Share Posted October 16, 2018 2 hours ago, AnnabellaRenee87 said: Sorry for posting so much, I was able to downgrade with iLO. I used Midnight Commander and moved all the files from the "Previous" folder back in to the /boot folder and was able to get back to 6.6.0 My Web GUI wouldn't load when I attempted 6.6.2 No worries, I’ll run it again tomorrow am using the older script 1 Quote Link to comment
1812 Posted October 16, 2018 Author Share Posted October 16, 2018 11 hours ago, 1812 said: No worries, I’ll run it again tomorrow am using the older script New 6.6.2 bzimage available in top of post. Quote Link to comment
AnnabellaRenee87 Posted October 16, 2018 Share Posted October 16, 2018 8 hours ago, 1812 said: New 6.6.2 bzimage available in top of post. Looks to be working now!! Seeing these error on the server's screen tho. Cant remember if vnstatd is a Plugin or not, don't know what cat is. Quote Link to comment
1812 Posted October 17, 2018 Author Share Posted October 17, 2018 23 hours ago, AnnabellaRenee87 said: Looks to be working now!! Seeing these error on the server's screen tho. Cant remember if vnstatd is a Plugin or not, don't know what cat is. ¯\_(ツ)_/¯ Quote Link to comment
1812 Posted November 3, 2018 Author Share Posted November 3, 2018 (edited) 6.6.3 now available in first post. Edited November 10, 2018 by 1812 Quote Link to comment
1812 Posted November 10, 2018 Author Share Posted November 10, 2018 6.6.5 now available in first post Quote Link to comment
1812 Posted December 6, 2018 Author Share Posted December 6, 2018 6.6.6 is up in the top of the thread. 1 Quote Link to comment
m0ngr31 Posted December 18, 2018 Share Posted December 18, 2018 I tried to upgrade to 6.6.6 and it's broken my server networking. Having an issue with networking coming up: Quote Link to comment
1812 Posted December 18, 2018 Author Share Posted December 18, 2018 13 hours ago, m0ngr31 said: I tried to upgrade to 6.6.6 and it's broken my server networking. Having an issue with networking coming up: is this with the 6.6.6 bzimage in the top? and if so, are you certain you downloaded the correct one? it does not modify your network settings. Quote Link to comment
m0ngr31 Posted December 18, 2018 Share Posted December 18, 2018 Yeah, I tried it twice yesterday. Both did the same thing. Unraid worked fine (minus my Windows VM) before I copied over the new bzimage Quote Link to comment
m0ngr31 Posted December 18, 2018 Share Posted December 18, 2018 Switched back to the stock bzimage and everything but my VM is working okay. After looking at the screenshots some more, it seems to be an issue with loading the bonding module. I'm not sure if that got removed for some reason when it recompiled? Quote Link to comment
Creslinunx Posted December 19, 2018 Share Posted December 19, 2018 Hi this info in this thread with a couple other sources helped me enormously. I managed to get my 1080 working in KVM on ubuntu 18.04 with my bulleted steps / configs. This was not an unraid build, but standard ubuntu 18.04 - but the overlap is huge so hopefully this may be of help to others It took me a couple days of playing with lots of options before found this fit where the KVM would come up clean, the card not fall of the bus etc. PCIe Passthrough on DL360/380 Gen 7 for KVM for GTX 1080 Linux Kernel 4.15.x Ubuntu 18.04 LTS A) Host BIOS ============ 1) In the host bios check both VT-x and VT-d are enabled. Press "F9" to enter the bios at boot B) Host Operating system ======================== 0) Install KVM, vfio as standard(outside the scope of these bullets) 1) Take a copy of the GTX vbios, this is needed later when loading the KVM - Download nvflash_linux from https://www.techpowerup.com/download/nvidia-nvflash/ - save the vbios with `./nvflash_linux --save mygtx.rom` 2) Blacklist any drivers that may attach to the card - edit `/etc/modprobe.d/blacklist.conf` - add: ``` blacklist nouveau blacklist nvidiafb blacklist nvidia blacklist nvidia_drm blacklist snd-pcsp ``` - run `update-initramfs -u` 3) Take note of PCIe IDs for the GTX, in this example they are "10de:1b80" and "10de:10f0" - `lspci --nnk` ** VGA compatible controller [0300]: NVIDIA Corporation GP104 [GeForce GTX 1080] [10de:1b80] ** Audio device [0403]: NVIDIA Corporation GP104 High Definition Audio Controller [10de:10f0] 4) Assign these to VFIO - edit `cat /etc/modprobe.d/vfio.conf` - add(replace the IDs with your own): ``` vfio vfio_iommu_type1 vfio_pci ids=10de:1b80,10de:10f0 vfio_virqfd vhost-net ``` 5) create modprobe.d vfio.conf to ensure vfio takes the IDs on boot - edit `/etc/modprobe.d/vfio.conf` - add: ``` options vfio-pci ids=10de:1b80,10de:10f0 softdep nouveau pre: vfio-pci ``` - and set to load on boot `echo 'vfio-pci' > /etc/modules-load.d/vfio-pci.conf` 6) Update GRUB to use iommu and allow unsafe interupts: - edit `/etc/defaults/grub` - update/append entry "GRUB_CMDLINE_LINUX_DEFAULT" with additional arguments: GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1" - update grub `update-grub` 7) Recompile Linux to disable RMRR check - apt-get install fakeroot, kernel-package, linux-source - download and untar linux src under /usr/src, as example `tar xf /usr/src/linux-source-4.15.tar.xz` - cd `/usr/src/linux-4.15.0/` (or your src) - edit `/usr/src/linux-4.15.0/drivers/iommu/intel-iommu.c` to remove the `-EPERM` return - Replace: ``` if (device_is_rmrr_locked(dev)) { dev_warn(dev, "Device is ineligible for IOMMU domain attach due to platform RMRR requirement. Contact your platform vendor.\n"); return -EPERM; } ``` With: ``` if (device_is_rmrr_locked(dev)) { dev_warn(dev, "HP should release a BIOS update for G7 hardware. Ignoring RMRR.\n"); } ``` - Compile the image ("this will take a while...") ` fakeroot make-kpkg --initrd --revision=1.0.custom kernel_image` - Instal the image: `dpkg -i ../linux-image-4.15-.....<< your sub arch number >>_1.0.custom_amd64.deb` Create your KVM in virt manager - The KVM should use UEFI i.e type 35, not BIOS - Add hardware, find the GTX PCIe and add If unsure you can find your PCI channel with: `lspci -nn` Here I am interested in `09:00:0` 0.9:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP104 [GeForce GTX 1080] [10de:1b80] (rev a1) 09:00.1 Audio device [0403]: NVIDIA Corporation GP104 High Definition Audio Controller [10de:10f0] (rev a1) - set "rom to On" - If using headless for CUDA work, you may want to add a QLX video card and spice display to access the console in virt manager - apply / save 9) Edit the KVM XLM directly: - vish edit <your kvm name> - Update the PCIe lane added from host with you card to include the PATH to the cards ROM dumped earlier. - From: ``` <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x09' slot='0x00' function='0x0'/> </source> <rom bar='on'/> <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/> </hostdev> ``` - To: ``` <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x09' slot='0x00' function='0x0'/> </source> <rom bar='on' file='/home/creslin/gtx/mygtx.rom'/> <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/> </hostdev> ``` 10) Reboot! 11) Check the driver in use for the GT care is `vfio-pci` - `lspci -nnk` 09:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP104 [GeForce GTX 1080] [10de:1b80] (rev a1) Subsystem: Gigabyte Technology Co., Ltd GP104 [GeForce GTX 1080] [1458:3730] Kernel driver in use: vfio-pci Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia 09:00.1 Audio device [0403]: NVIDIA Corporation GP104 High Definition Audio Controller [10de:10f0] (rev a1) Subsystem: Gigabyte Technology Co., Ltd GP104 High Definition Audio Controller [1458:3730] Kernel driver in use: vfio-pci Kernel modules: snd_hda_intel 12) Check Intel IOMMU is loaded, VT-D is working: - ` dmesg | grep -E "DMAR|IOMMU" ` [ 0.000000] ACPI: DMAR 0x00000000C762FE80 000150 (v01 HP ProLiant 00000001 \xd2? 0000162E) [ 0.000000] DMAR: IOMMU enabled [ 0.000000] DMAR-IR: This system BIOS has enabled interrupt remapping [ 1.319488] DMAR: Host address width 39 [ 1.319492] DMAR: DRHD base: 0x000000cfffe000 flags: 0x1 [ 1.319513] DMAR: dmar0: reg_base_addr cfffe000 ver 1:0 cap c90780106f0462 ecap f0207e [ 1.319517] DMAR: RMRR base: 0x000000c77fc000 end: 0x000000c77fdfff [ 1.319520] DMAR: RMRR base: 0x000000c77f5000 end: 0x000000c77fafff [ 1.319523] DMAR: RMRR base: 0x000000c763e000 end: 0x000000c763ffff [ 1.319526] DMAR: ATSR flags: 0x0 [ 1.319841] DMAR: dmar0: Using Queued invalidation [ 1.319858] DMAR: Setting RMRR: [ 1.320205] DMAR: Setting identity map for device 0000:02:00.0 [0xc763e000 - 0xc763ffff] [ 1.320562] DMAR: Setting identity map for device 0000:02:00.2 [0xc763e000 - 0xc763ffff] [ 1.320844] DMAR: Setting identity map for device 0000:05:00.0 [0xc763e000 - 0xc763ffff] [ 1.321190] DMAR: Setting identity map for device 0000:09:00.0 [0xc763e000 - 0xc763ffff] [ 1.321564] DMAR: Setting identity map for device 0000:09:00.1 [0xc763e000 - 0xc763ffff] [ 1.321893] DMAR: Setting identity map for device 0000:00:1d.0 [0xc77f5000 - 0xc77fafff] [ 1.322178] DMAR: Setting identity map for device 0000:00:1d.1 [0xc77f5000 - 0xc77fafff] [ 1.322513] DMAR: Setting identity map for device 0000:00:1d.2 [0xc77f5000 - 0xc77fafff] [ 1.322870] DMAR: Setting identity map for device 0000:00:1d.3 [0xc77f5000 - 0xc77fafff] [ 1.322891] DMAR: Setting identity map for device 0000:02:00.0 [0xc77f5000 - 0xc77fafff] [ 1.322896] DMAR: Setting identity map for device 0000:02:00.2 [0xc77f5000 - 0xc77fafff] [ 1.323218] DMAR: Setting identity map for device 0000:02:00.4 [0xc77f5000 - 0xc77fafff] [ 1.323570] DMAR: Setting identity map for device 0000:00:1d.7 [0xc77fc000 - 0xc77fdfff] [ 1.323596] DMAR: Prepare 0-16MiB unity mapping for LPC [ 1.323918] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff] [ 1.324132] DMAR: Intel(R) Virtualization Technology for Directed I/O 13) Check the GTX and sound if multifunctional are in their own IOMMU group - run: `for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU Group %s ' "$n"; lspci -nns "${d##*/}"; done;` As example, mine are in group 21 without other devices: IOMMU Group 21 09:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP104 [GeForce GTX 1080] [10de:1b80] (rev a1) IOMMU Group 21 09:00.1 Audio device [0403]: NVIDIA Corporation GP104 High Definition Audio Controller [10de:10f0] (rev a1) 14) Start and install your KVM! C) On KVM ======================== 1) Check host sees the GTX card: - `lspci --nnk` 2) Install Nvidia GTX drivers, CUDA drivers (outside the scope of these bullets 3) Benchmark, i used gpu-burn - i got 100% the CUDA acceleration in guest as did in host Quote Link to comment
1812 Posted December 20, 2018 Author Share Posted December 20, 2018 On 12/18/2018 at 5:59 PM, m0ngr31 said: Switched back to the stock bzimage and everything but my VM is working okay. After looking at the screenshots some more, it seems to be an issue with loading the bonding module. I'm not sure if that got removed for some reason when it recompiled? I don't know why it would have been. Can you try to reconfigure the bond in GUI? or make a copy of the network.cfg file form the working version to your desktop, update to 6.6.6, put in new bzimage, see if standard networking is functional, then move the older network.cfg back and reboot to retest? Quote Link to comment
AnnabellaRenee87 Posted December 20, 2018 Share Posted December 20, 2018 Yeah, I tried it twice yesterday. Both did the same thing. Unraid worked fine (minus my Windows VM) before I copied over the new bzimageStrange, I'm using the same patch on my server with 6.6.6 on my DL370 G6.Sent from my SM-G955U using Tapatalk Quote Link to comment
m0ngr31 Posted December 21, 2018 Share Posted December 21, 2018 Well I don't know what changed, but I tried it again today and it worked fine! Thanks for this patch! 2 Quote Link to comment
oneclear Posted January 14, 2019 Share Posted January 14, 2019 I want to say thank you to everyone involved in achieving this computer saving modification. I have spent quite some time trying to passthrough an Intel 4 port nic for a Pfsense VM on my Microserver G8, yesterday morning I was contemplating selling it off and building something else. I came across this thread and swapped the bzimage, saved me a lot of money and trouble. I am on 6.6.6 Stable. I really do appreciate the time and effort put in and just wanted to say thank you. 1 Quote Link to comment
aterfax Posted January 30, 2019 Share Posted January 30, 2019 On 1/14/2019 at 12:18 PM, oneclear said: I want to say thank you to everyone involved in achieving this computer saving modification. I have spent quite some time trying to passthrough an Intel 4 port nic for a Pfsense VM on my Microserver G8, yesterday morning I was contemplating selling it off and building something else. I came across this thread and swapped the bzimage, saved me a lot of money and trouble. I am on 6.6.6 Stable. I really do appreciate the time and effort put in and just wanted to say thank you. +1 from me. 1 Quote Link to comment
1812 Posted February 11, 2019 Author Share Posted February 11, 2019 6.7 rc3 now available in the top post if you're feeling like playing with the latest release candidate. Quote Link to comment
m0ngr31 Posted February 16, 2019 Share Posted February 16, 2019 (edited) On 7/7/2018 at 11:52 AM, 1812 said: There are some bios options to try to fix this on newer models with some success. Do you know what settings I could check for in my BIOS? Currently running a DL380p Gen8. I'm worried that this patch won't be compatible with the new Nvidia Plugin, and I'm trying to give myself the best chance at making everything work at once. Edited February 16, 2019 by m0ngr31 Quote Link to comment
AnnabellaRenee87 Posted February 16, 2019 Share Posted February 16, 2019 Do you know what settings I could check for in my BIOS? Currently running a DL380p Gen8. I'm worried that this patch won't be compatible with the new Nvidia Plugin, and I'm trying to give myself the best chance at making everything work at once.It works with the Nvidia build.Install the plugin, install RC3, go to your flash drive and drop in bzimage then reboot.Only reboot that one time. Working fine for me!Sent from my SM-G955U using Tapatalk Quote Link to comment
1812 Posted February 16, 2019 Author Share Posted February 16, 2019 20 minutes ago, m0ngr31 said: Do you know what settings I could check for in my BIOS? Currently running a DL380p Gen8. Hp released some procedures for newer machines that allow for a workaround. You’d have to google “proliant rmrr “ and look for their service advisory to see if one is available for the g8’s (I don’t remember off the top of my head.) Quote Link to comment
m0ngr31 Posted February 16, 2019 Share Posted February 16, 2019 4 hours ago, AnnabellaRenee87 said: It works with the Nvidia build. Install the plugin, install RC3, go to your flash drive and drop in bzimage then reboot. Only reboot that one time. Working fine for me! Sent from my SM-G955U using Tapatalk That's awesome, thanks! Quote Link to comment
CHBMB Posted February 16, 2019 Share Posted February 16, 2019 It works with the Nvidia build.Install the plugin, install RC3, go to your flash drive and drop in bzimage then reboot.Only reboot that one time. Working fine for me!Sent from my SM-G955U using TapatalkI had a suspicion it would work fine, but nice to see you confirm it@AnnabellaRenee87Sent from my Mi A1 using Tapatalk 1 Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.