Jump to content
1812

unRaid HP Proliant Edition - RMRR Error Patching

86 posts in this topic Last Reply

Recommended Posts

Sorry for posting so much, I was able to downgrade with iLO. I used Midnight Commander and moved all the files from the "Previous" folder back in to the /boot folder and was able to get back to 6.6.0

My Web GUI wouldn't load when I attempted 6.6.2

Share this post


Link to post
2 hours ago, AnnabellaRenee87 said:

Sorry for posting so much, I was able to downgrade with iLO. I used Midnight Commander and moved all the files from the "Previous" folder back in to the /boot folder and was able to get back to 6.6.0

My Web GUI wouldn't load when I attempted 6.6.2

No worries, I’ll run it again tomorrow am using the older script

Share this post


Link to post
11 hours ago, 1812 said:

No worries, I’ll run it again tomorrow am using the older script

 

New 6.6.2 bzimage available in top of post.

Share this post


Link to post
8 hours ago, 1812 said:

 

New 6.6.2 bzimage available in top of post.

Looks to be working now!! Seeing these error on the server's screen tho.

image.thumb.png.216cdf058ab587dad54b44c1a2a32752.png

 

Cant remember if vnstatd is a Plugin or not, don't know what cat is.

Share this post


Link to post
23 hours ago, AnnabellaRenee87 said:

Looks to be working now!! Seeing these error on the server's screen tho.

image.thumb.png.216cdf058ab587dad54b44c1a2a32752.png

 

Cant remember if vnstatd is a Plugin or not, don't know what cat is.

 

¯\_(ツ)_/¯

Share this post


Link to post

6.6.3 now available in first post.

Edited by 1812

Share this post


Link to post

I tried to upgrade to 6.6.6 and it's broken my server networking. Having an issue with networking coming up:

 

gWi4FCS.png

 

uLwrrRV.png

Share this post


Link to post
13 hours ago, m0ngr31 said:

I tried to upgrade to 6.6.6 and it's broken my server networking. Having an issue with networking coming up:

 

gWi4FCS.png

 

uLwrrRV.png

is this with the 6.6.6 bzimage in the top? and if so, are you certain you downloaded the correct one? it does not modify your network settings.

Share this post


Link to post

Yeah, I tried it twice yesterday. Both did the same thing. Unraid worked fine (minus my Windows VM) before I copied over the new bzimage

Share this post


Link to post

Switched back to the stock bzimage and everything but my VM is working okay. After looking at the screenshots some more, it seems to be an issue with loading the bonding module. I'm not sure if that got removed for some reason when it recompiled?

Share this post


Link to post

Hi this info in this thread with a couple other sources helped me enormously. 

I managed to get my 1080 working in KVM on ubuntu 18.04 with my bulleted steps / configs. 


This was not an unraid build, but standard ubuntu 18.04 - but the overlap is huge so hopefully this may be of help to others
It took me a couple days of playing with lots of options before found this fit where the KVM would come up clean, the card not fall of the bus etc. 

 

PCIe Passthrough on DL360/380 Gen 7 for KVM for GTX 1080
Linux Kernel 4.15.x
Ubuntu 18.04 LTS

A) Host BIOS
============
1) In the host bios check both VT-x and VT-d are enabled. Press "F9" to enter the bios at boot

B) Host Operating system 
========================
0) Install KVM, vfio as standard(outside the scope of these bullets)

1) Take a copy of the GTX vbios, this is needed later when loading the KVM
 - Download nvflash_linux from https://www.techpowerup.com/download/nvidia-nvflash/
 - save the vbios with `./nvflash_linux --save mygtx.rom`

2) Blacklist any drivers that may attach to the card 
 - edit `/etc/modprobe.d/blacklist.conf`
 - add: ```
      blacklist nouveau
      blacklist nvidiafb
      blacklist nvidia
      blacklist nvidia_drm
      blacklist snd-pcsp

```
 - run `update-initramfs -u`

3) Take note of PCIe IDs for the GTX, in this example they are "10de:1b80" and "10de:10f0"
 - `lspci --nnk`
 ** VGA compatible controller [0300]: NVIDIA Corporation GP104 [GeForce GTX 1080] [10de:1b80]
 ** Audio device [0403]: NVIDIA Corporation GP104 High Definition Audio Controller [10de:10f0]

4) Assign these to VFIO 
 - edit `cat /etc/modprobe.d/vfio.conf`
 - add(replace the IDs with your own): ```
      vfio
      vfio_iommu_type1
      vfio_pci ids=10de:1b80,10de:10f0
      vfio_virqfd
      vhost-net

```

5) create modprobe.d vfio.conf to ensure vfio takes the IDs on boot
 - edit `/etc/modprobe.d/vfio.conf`
 - add: ```
   options vfio-pci ids=10de:1b80,10de:10f0
   softdep nouveau pre: vfio-pci

```
 - and set to load on boot 
   `echo 'vfio-pci' > /etc/modules-load.d/vfio-pci.conf`


6) Update GRUB to use iommu and allow unsafe interupts:
 - edit `/etc/defaults/grub`
 - update/append entry "GRUB_CMDLINE_LINUX_DEFAULT" with additional arguments:
   GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1"
 - update grub `update-grub`

7) Recompile Linux to disable RMRR check 
 - apt-get install  fakeroot, kernel-package, linux-source
 - download and untar linux src under /usr/src, as example `tar xf /usr/src/linux-source-4.15.tar.xz`
 - cd `/usr/src/linux-4.15.0/` (or your src)
 - edit `/usr/src/linux-4.15.0/drivers/iommu/intel-iommu.c` to remove the `-EPERM` return
 - Replace: ```
   if (device_is_rmrr_locked(dev)) {
               dev_warn(dev, "Device is ineligible for IOMMU domain attach due to platform RMRR requirement.  Contact your platform vendor.\n");
               return -EPERM;
       }

```

   With: ```
   if (device_is_rmrr_locked(dev)) {
                dev_warn(dev, "HP should release a BIOS update for G7 hardware. Ignoring RMRR.\n");
        }

```
 - Compile the image ("this will take a while...")
   ` fakeroot make-kpkg --initrd --revision=1.0.custom kernel_image`
 - Instal the image:
   `dpkg -i ../linux-image-4.15-.....<< your sub arch number  >>_1.0.custom_amd64.deb` 
 
8) Create your KVM in virt manager
  - The KVM should use UEFI i.e type 35, not BIOS
  - Add hardware, find the GTX PCIe and add 
   If unsure you can find your PCI channel with: `lspci -nn` Here I am interested in `09:00:0`
      0.9:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP104 [GeForce GTX 1080] [10de:1b80] (rev a1)
      09:00.1 Audio device [0403]: NVIDIA Corporation GP104 High Definition Audio Controller [10de:10f0] (rev a1)   
  - set "rom to On"
  - If using headless for CUDA work, you may want to add a QLX video card and spice display to access the console in virt manager
  - apply / save 
 
9) Edit the KVM XLM directly: 
  - vish edit <your kvm name>
  - Update the PCIe lane added from host with you card to include the PATH to the cards ROM dumped earlier. 
  - From: ```
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x09' slot='0x00' function='0x0'/>
      </source>
      <rom bar='on'/>
      <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
    </hostdev>

```
  - To: ```
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x09' slot='0x00' function='0x0'/>
      </source>
      <rom bar='on' file='/home/creslin/gtx/mygtx.rom'/>
      <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
    </hostdev>

```

10) Reboot!


11) Check  the driver in use for the GT care  is `vfio-pci`
  - `lspci -nnk`

09:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP104 [GeForce GTX 1080] [10de:1b80] (rev a1)
    Subsystem: Gigabyte Technology Co., Ltd GP104 [GeForce GTX 1080] [1458:3730]
    Kernel driver in use: vfio-pci
    Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia
09:00.1 Audio device [0403]: NVIDIA Corporation GP104 High Definition Audio Controller [10de:10f0] (rev a1)
    Subsystem: Gigabyte Technology Co., Ltd GP104 High Definition Audio Controller [1458:3730]
    Kernel driver in use: vfio-pci
    Kernel modules: snd_hda_intel


12) Check Intel IOMMU is loaded, VT-D is working:
  - `  dmesg | grep -E "DMAR|IOMMU" `
[    0.000000] ACPI: DMAR 0x00000000C762FE80 000150 (v01 HP     ProLiant 00000001 \xd2?   0000162E)
[    0.000000] DMAR: IOMMU enabled
[    0.000000] DMAR-IR: This system BIOS has enabled interrupt remapping
[    1.319488] DMAR: Host address width 39
[    1.319492] DMAR: DRHD base: 0x000000cfffe000 flags: 0x1
[    1.319513] DMAR: dmar0: reg_base_addr cfffe000 ver 1:0 cap c90780106f0462 ecap f0207e
[    1.319517] DMAR: RMRR base: 0x000000c77fc000 end: 0x000000c77fdfff
[    1.319520] DMAR: RMRR base: 0x000000c77f5000 end: 0x000000c77fafff
[    1.319523] DMAR: RMRR base: 0x000000c763e000 end: 0x000000c763ffff
[    1.319526] DMAR: ATSR flags: 0x0
[    1.319841] DMAR: dmar0: Using Queued invalidation
[    1.319858] DMAR: Setting RMRR:
[    1.320205] DMAR: Setting identity map for device 0000:02:00.0 [0xc763e000 - 0xc763ffff]
[    1.320562] DMAR: Setting identity map for device 0000:02:00.2 [0xc763e000 - 0xc763ffff]
[    1.320844] DMAR: Setting identity map for device 0000:05:00.0 [0xc763e000 - 0xc763ffff]
[    1.321190] DMAR: Setting identity map for device 0000:09:00.0 [0xc763e000 - 0xc763ffff]
[    1.321564] DMAR: Setting identity map for device 0000:09:00.1 [0xc763e000 - 0xc763ffff]
[    1.321893] DMAR: Setting identity map for device 0000:00:1d.0 [0xc77f5000 - 0xc77fafff]
[    1.322178] DMAR: Setting identity map for device 0000:00:1d.1 [0xc77f5000 - 0xc77fafff]
[    1.322513] DMAR: Setting identity map for device 0000:00:1d.2 [0xc77f5000 - 0xc77fafff]
[    1.322870] DMAR: Setting identity map for device 0000:00:1d.3 [0xc77f5000 - 0xc77fafff]
[    1.322891] DMAR: Setting identity map for device 0000:02:00.0 [0xc77f5000 - 0xc77fafff]
[    1.322896] DMAR: Setting identity map for device 0000:02:00.2 [0xc77f5000 - 0xc77fafff]
[    1.323218] DMAR: Setting identity map for device 0000:02:00.4 [0xc77f5000 - 0xc77fafff]
[    1.323570] DMAR: Setting identity map for device 0000:00:1d.7 [0xc77fc000 - 0xc77fdfff]
[    1.323596] DMAR: Prepare 0-16MiB unity mapping for LPC
[    1.323918] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
[    1.324132] DMAR: Intel(R) Virtualization Technology for Directed I/O


13) Check the GTX and sound if multifunctional are in their own IOMMU group 
 - run: 
   `for d in /sys/kernel/iommu_groups/*/devices/*; do      n=${d#*/iommu_groups/*}; n=${n%%/*};     printf 'IOMMU Group %s ' "$n";     lspci -nns "${d##*/}"; done;`

  As example, mine are in group 21 without other devices:
  IOMMU Group 21 09:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP104 [GeForce GTX 1080] [10de:1b80] (rev a1)
  IOMMU Group 21 09:00.1 Audio device [0403]: NVIDIA Corporation GP104 High Definition Audio Controller [10de:10f0] (rev a1)


14) Start and install your KVM!


C) On KVM 
========================
1) Check host sees the GTX card:
 - `lspci --nnk`
2) Install Nvidia GTX drivers, CUDA drivers 
   (outside the scope of these bullets 
3) Benchmark, i used gpu-burn - i got 100% the CUDA acceleration in guest as did in host
 

Share this post


Link to post
On 12/18/2018 at 5:59 PM, m0ngr31 said:

Switched back to the stock bzimage and everything but my VM is working okay. After looking at the screenshots some more, it seems to be an issue with loading the bonding module. I'm not sure if that got removed for some reason when it recompiled?

 

I don't know why it would have been. Can you try to reconfigure the bond in GUI? or make a copy of the network.cfg file form the working version to your desktop, update to 6.6.6, put in new bzimage, see if standard networking is functional, then move the older network.cfg back and reboot to retest?

Share this post


Link to post
Yeah, I tried it twice yesterday. Both did the same thing. Unraid worked fine (minus my Windows VM) before I copied over the new bzimage
Strange, I'm using the same patch on my server with 6.6.6 on my DL370 G6.

Sent from my SM-G955U using Tapatalk

Share this post


Link to post

Well I don't know what changed, but I tried it again today and it worked fine! Thanks for this patch!

Share this post


Link to post

I want to say thank you to everyone involved in achieving this computer saving modification. I have spent quite some time trying to passthrough an Intel 4 port nic for a Pfsense VM on my Microserver G8, yesterday morning I was contemplating selling it off and building something else. I came across this thread and swapped the bzimage, saved me a lot of money and trouble. I am on 6.6.6 Stable. I really do appreciate the time and effort put in and just wanted to say thank you.

Share this post


Link to post
On 1/14/2019 at 12:18 PM, oneclear said:

I want to say thank you to everyone involved in achieving this computer saving modification. I have spent quite some time trying to passthrough an Intel 4 port nic for a Pfsense VM on my Microserver G8, yesterday morning I was contemplating selling it off and building something else. I came across this thread and swapped the bzimage, saved me a lot of money and trouble. I am on 6.6.6 Stable. I really do appreciate the time and effort put in and just wanted to say thank you.

+1 from me.

Share this post


Link to post

6.7 rc3 now available in the top post if you're feeling like playing with the latest release candidate.

Share this post


Link to post
On 7/7/2018 at 11:52 AM, 1812 said:

There are some bios options to try to fix this on newer models with some success.

Do you know what settings I could check for in my BIOS? Currently running a DL380p Gen8.

 

I'm worried that this patch won't be compatible with the new Nvidia Plugin, and I'm trying to give myself the best chance at making everything work at once.

Edited by m0ngr31

Share this post


Link to post
Do you know what settings I could check for in my BIOS? Currently running a DL380p Gen8.
 
I'm worried that this patch won't be compatible with the new Nvidia Plugin, and I'm trying to give myself the best chance at making everything work at once.
It works with the Nvidia build.
Install the plugin, install RC3, go to your flash drive and drop in bzimage then reboot.

Only reboot that one time. Working fine for me!

Sent from my SM-G955U using Tapatalk

Share this post


Link to post
20 minutes ago, m0ngr31 said:

Do you know what settings I could check for in my BIOS? Currently running a DL380p Gen8.

Hp released some procedures for newer machines that allow for a workaround. You’d have to google “proliant rmrr “ and look for their service advisory to see if one is available for the g8’s (I don’t remember off the top of my head.)

Share this post


Link to post
4 hours ago, AnnabellaRenee87 said:

It works with the Nvidia build.
Install the plugin, install RC3, go to your flash drive and drop in bzimage then reboot.

Only reboot that one time. Working fine for me!

Sent from my SM-G955U using Tapatalk
 

That's awesome, thanks!

Share this post


Link to post
It works with the Nvidia build.
Install the plugin, install RC3, go to your flash drive and drop in bzimage then reboot.

Only reboot that one time. Working fine for me!

Sent from my SM-G955U using Tapatalk

I had a suspicion it would work fine, but nice to see you confirm it@AnnabellaRenee87



Sent from my Mi A1 using Tapatalk

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.