[Support] ich777 - AMD Vendor Reset, CoralTPU, hpsahba,...


Recommended Posts

1 hour ago, Abdul said:

Hi,

 

Does unraid kernel helper and Vendor-Reset work with AMD Radeon VII? AMD Reset bug? Anyone try it?

 

It may work, but that's no guarantee. Some vega20 cards seem to refuse baco reset (bus active chip off)...

 

If you dont mind, could you share your vendor/model with us and maybe try it out and share your results with us?

We would highly appreciate that :) 

  • Like 1
Link to comment

@ich777

 

If you have the time and possibility I would like to request the Mellanox 4.9-2.2.4.0 LTS driver:

https://www.mellanox.com/products/infiniband-drivers/linux/mlnx_ofed/#tabs-2

 

Release notes/Installation:

https://docs.mellanox.com/display/MLNXENv492240/Release+Notes

 

Linux Driver Compatibility Matrix:

https://www.mellanox.com/support/mlnx-ofed-matrix

 

Currently, UnRAID 6.9.1 ships with

kernel: mlx4_core: Mellanox ConnectX core driver v4.0-0
modinfo mlx4_en
filename:       /lib/modules/5.10.21-Unraid/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko.xz
version:        4.0-0
license:        Dual BSD/GPL
description:    Mellanox ConnectX HCA Ethernet driver
author:         Liran Liss, Yevgeny Petrilin
srcversion:     EE160E8DB5FA601160D41B2
depends:        mlx4_core
retpoline:      Y
intree:         Y
name:           mlx4_en
vermagic:       5.10.21-Unraid SMP mod_unload 
parm:           udp_rss:Enable RSS for incoming UDP traffic or disabled (0) (uint)
parm:           pfctx:Priority based Flow Control policy on TX[7:0]. Per priority bit mask (uint)
parm:           pfcrx:Priority based Flow Control policy on RX[7:0]. Per priority bit mask (uint)
parm:           inline_thold:Threshold for using inline data (range: 17-104, default: 104) (uint)

 

Link to comment
3 hours ago, sorano said:

If you have the time and possibility I would like to request the Mellanox 4.9-2.2.4.0 LTS driver

Isn't this a infiniband card or something like that? Unraid doesn't support Infiniband in general I think...

 

You always can compile them on your own with the Custom Build Mode.

 

EDIT: Which card are you having exactly?

  • Thanks 1
Link to comment
10 minutes ago, ich777 said:

Isn't this a infiniband card or something like that? Unraid doesn't support Infiniband in general I think...

 

You always can compile them on your own with the Custom Build Mode.

 

EDIT: Which card are you having exactly?

 

I have a ConnectX-3 Pro.

 

But I'm having issues with SR-IOV to a Windows VM as in this thread: 

 

 

I was discussing my issue with the moderators in VFIO discord and one of them said I should try with newer Mellanox drivers.

Link to comment
On 4/2/2021 at 4:19 PM, sorano said:

Mellanox 4.9-2.2.4.0 LTS

Sadly enough this driver is for Kernels pre 5.x and they are not working on recent versions of Unraid.

The newer drivers dropped, like you said, the ConnectX-3 cards and only supporting ConnectX-4 and the last recent version that I saw ConnectX-5 and up.

Link to comment
On 4/1/2021 at 9:36 PM, giganode said:

 

It may work, but that's no guarantee. Some vega20 cards seem to refuse baco reset (bus active chip off)...

 

If you dont mind, could you share your vendor/model with us and maybe try it out and share your results with us?

We would highly appreciate that :) 

@giganode

Vendor/Model: XFX AMD RADEON VII 16GB HBM2, 1750 MHz Boost, 1801 MHz Peak, 3xDP 1xHDMI PCI-E 3.0

 

I Try it for 3 days now Vendor-Reset kernel patch and it work partially. I get the reset bug after restarting VM 5 to 7. 

 

Ubuntu work great i can restart shutdown the VM from Unraid or VM it self with no problems.

 

Mac OS work only for 5 to 7 times with Force stop and then restart from unraid, If try to restart from the VM it self i will get AMD Reset Bug.

 

Windows 10 what ever i do it will crash and i have to restart my unraid server to use the GPU again.

 

@ich777 thanks a lot you super star. 🌟 🙃

  • Like 1
Link to comment

Thank you @ich777 for including the GPU reset for AMD cards in your "something for everyone" docker.  When I return to an unRAID machine I will definitely be using unless Limetech do the same and fix AMD card reset for us ;-)

 

Could I ask if you could apply your wizardry to enabling Audio support in unRAID, please? 😍 I'd love to try using the -sb16 and -ac97 switches in VMs but they only simulate a Soundblaster16 or whatever and pass the audio back to the host Linux system and unRAID has none (I think!). Such a change would make VMs for older OS like Win9x/2000 and DOS possible.

 

Another "limitation" of the KVM-QEMU we have (again not sure as not running unRAID right now) is that it does not include qemu-ppc or qemu-arm which would enable VMs for PPC-Macs, PPC-Amigas and Arm-RaspberryPi2....  If your magic wand is able to fix that too I'd be a very happy user 😁

  • Like 1
Link to comment
18 hours ago, methanoid said:

If your magic wand is able to fix that too I'd be a very happy user 😁

The problem here is that I have to compile QEMU from source and I don't know how QEMU is compiled exactly for Unraid and since I don't wan't to break something I can't do this easily...

 

I tried it once and it compiled just fine but some VM's on my Dev machine won't fire up after installing my custom QEMU build since I'm pretty sure I don't compiled it with the right flags and options to work with Unraid.

 

I'm also looking for a way to compile QEMU with GL support since I'm currently looking into Intel GVT-g and this has a really big potential for HW accelerated VM's with vGPU's (for example iGPU's).

Currently the VM's run with HW accelerated Video if you use it over RDP or Parsec and you feel the difference that they have a "real" graphics card attached to them but the GL support for QEMU is missing and so I can't get SPICE with MESA EGL, QEMU Display,... not to work.

 

Eventually @eschultz can help here. 😃

 

I think they are planing to release ARM support if I'm not mistaken for quite some time but I think that there are some heavy modifications to the GUI needed otherwise this is all command line only or in terms of Unraid XML editing only.

 

The build options for QEMU would be really nice but I also understand that they (as a company) don't give it to us since you can do very bad things with it and if something isn't working the support for it can be very "frustrating" if the user does not tell them in the first place that something custom is installed.

 

I will definitely look into QEMU again but currently my time is very limited...

 

EDIT: Eventually I should rename the container from "Unraid-Kernel-Helper" to "Something-For-everyone"... :D

  • Like 2
Link to comment
55 minutes ago, ich777 said:

The problem here is that I have to compile QEMU from source and I don't know how QEMU is compiled exactly for Unraid and since I don't wan't to break something I can't do this easily...

 

I tried it once and it compiled just fine but some VM's on my Dev machine won't fire up after installing my custom QEMU build since I'm pretty sure I don't compiled it with the right flags and options to work with Unraid.

 

I'm also looking for a way to compile QEMU with GL support since I'm currently looking into Intel GVT-g and this has a really big potential for HW accelerated VM's with vGPU's (for example iGPU's).

Currently the VM's run with HW accelerated Video if you use it over RDP or Parsec and you feel the difference that they have a "real" graphics card attached to them but the GL support for QEMU is missing and so I can't get SPICE with MESA EGL, QEMU Display,... not to work.

 

Eventually @eschultz can help here. 😃

 

I think they are planing to release ARM support if I'm not mistaken for quite some time but I think that there are some heavy modifications to the GUI needed otherwise this is all command line only or in terms of Unraid XML editing only.

 

The build options for QEMU would be really nice but I also understand that they (as a company) give it to us since you can do very bad things with it and if something isn't working the support for it can be very "frustrating" if the user does not tell them in the first place that something custom is installed.

 

I will definitely look into QEMU again but currently my time is very limited...

 

EDIT: Eventually I should rename the container from "Unraid-Kernel-Helper" to "Something-For-everyone"... :D

 

I hear what you are saying about Qemu-ppc and Qemu-arm - hopefully @limetech will help us out soon, as would really make unRAID the goto VM platform as well as NASbox++ ;-)

 

You didnt mention any issues around audio (apart from lack of hours in the day I assume) - I saw we've had DVB versions of unRAID before so maybe audio support might not be as horrible to add as Qemu-PPC/Arm ?  Again, could be enabled by good old @limetech if they are feeling the love for us 🙂 

 

Don't get me wrong, I am VERY happy we have an easier FLR for Navi cards in unRAID - the rest are icing on the cake and cherries on top of that icing 😂

  • Like 1
Link to comment
20 minutes ago, methanoid said:

You didnt mention any issues around audio (apart from lack of hours in the day I assume)

I could theoretically make a plugin for this but I have to look into this since 'SOUND' isn't enabled for Unraid.

Is this enough for QEMU to make it work or that what you want?

 

Also shouldn't it be possible to emulate sound and give it back to the passed through Graphics Card with integrated HDMI audio (if you do it like that)?

I have to look this all up because I'm not really into QEMU.

Link to comment
34 minutes ago, ich777 said:

I could theoretically make a plugin for this but I have to look into this since 'SOUND' isn't enabled for Unraid.

Is this enough for QEMU to make it work or that what you want?

 

Also shouldn't it be possible to emulate sound and give it back to the passed through Graphics Card with integrated HDMI audio (if you do it like that)?

I have to look this all up because I'm not really into QEMU.

I dont claim to understand HOW but adding "-soundhw sb16" to Qemu command line emulates a Sound Blaster 16 at
IRQ 5, DMA 1, high DMA 5, and port0x220 and passes the emulated sound to the "native host audio" I THINK through ALSA or Pulse Audio or something like that!! 🙂

Link to comment
2 hours ago, jonathanm said:

That's the issue, Unraid doesn't have any "native host audio".

Yes, I thought it would be also somehow possible to hand the audio over to a device that is passed through but after thinking again that can't work. :D

 

3 hours ago, methanoid said:

I dont claim to understand HOW but adding "-soundhw sb16" to Qemu command line emulates a Sound Blaster 16

I will look into this ASAP.

  • Like 1
Link to comment
14 hours ago, jortan said:

Any chance the iSCSI Target plugin could be modified to create thin-provisioned qcow2 files instead of raw?

 


qemu-img create -f qcow2 file-path -o size=Size-in-MB,preallocation=metadata

https://www.geekpills.com/operating-system/debian/ubuntu18-04/how-to-setup-iscsi-storage-server-on-ubuntu-18-04-lts

Sorry seen your post just now.

 

That should be possible but you have to create the assignment by hand, like in the tutorial you linked.

Or do you mean the UI?

The UI is more or less in Beta...

Link to comment
On 3/31/2021 at 6:32 AM, ich777 said:

I think so.

Don't have any hardware on hand to test...

I have migrated from intel i7 4790k to intel i5 11400 (GPU UHD 730), it doesn't seem to be supported

Edited by xXx
Link to comment

Hey!

I'm building a kernel that contains vendor-reset for @plis halp. Is it failing to build vendor-reset?:
 

---Copying Kernel Image to output folder---
---Downloading gnif/vendor-reset from Github---
---Source: https://github.com/gnif/vendor-reset ---
Cloning into 'vendor-reset'...
Already on 'master'
Your branch is up to date with 'origin/master'.
make -C /lib/modules/5.10.21-Unraid/build M=/usr/src/vendor-reset/vendor-reset modules
make[1]: *** /lib/modules/5.10.21-Unraid/build: No such file or directory. Stop.
make: *** [Makefile:8: build] Error 2

make -C /lib/modules/5.10.21-Unraid/build M=/usr/src/vendor-reset/vendor-reset INSTALL_MOD_PATH= modules_install
make[1]: *** /lib/modules/5.10.21-Unraid/build: No such file or directory. Stop.
make: *** [Makefile:11: install] Error 2

---Generating bzmodules in output folder---

logs.txt

Link to comment
3 hours ago, lnxd said:

I'm building a kernel that contains vendor-reset for @plis halp. Is it failing to build vendor-reset?:

Please upgrade to 6.9.2 and then try to do a custom build again.

From what I see you are trying to build on 6.9.1 for 6.9.2 (the Kernel-Helper is always configured to only support the latest version).

  • Thanks 1
Link to comment
1 hour ago, ich777 said:

Please upgrade to 6.9.2 and then try to do a custom build again.

From what I see you are trying to build on 6.9.1 for 6.9.2 (the Kernel-Helper is always configured to only support the latest version).

Thank you! Should have been obvious to me, but I didn't know the new version was out yet 😅

  • Haha 1
Link to comment

I am running 6.9.2 and built the kernel for Gnif's FLR patch.  Backed up files. Copied the new files over the old ones and rebooted. 

 

My Win10 VM boots fine on a fresh boot but any restart causes me to lose the VM and display. I have a Powercolor Red Dragon RX5700, primary card stubbed, efifb=off and amd_iommu=on in syslinux config and pass through with GPU Rom included. I have another GPU for 2nd VMs but not used it yet. 

 

Any ideas where I am going wrong, please?

Link to comment
38 minutes ago, methanoid said:

My Win10 VM boots fine on a fresh boot but any restart causes me to lose the VM and display. I have a Powercolor Red Dragon RX5700, primary card stubbed, efifb=off and amd_iommu=on in syslinux config and pass through with GPU Rom included. I have another GPU for 2nd VMs but not used it yet. 

Hopefully @giganode has not too much to do so he can answer here soon since he is the guy who can help with the AMD cards.

  • Like 1
  • Thanks 1
Link to comment
4 hours ago, methanoid said:

I am running 6.9.2 and built the kernel for Gnif's FLR patch.  Backed up files. Copied the new files over the old ones and rebooted. 

 

My Win10 VM boots fine on a fresh boot but any restart causes me to lose the VM and display. I have a Powercolor Red Dragon RX5700, primary card stubbed, efifb=off and amd_iommu=on in syslinux config and pass through with GPU Rom included. I have another GPU for 2nd VMs but not used it yet. 

 

Any ideas where I am going wrong, please?

 

Hey!

 

Let me share my syslinux config which works on my system...

 

label Unraid OS
  menu default
  kernel /bzimage
  append video=vesafb:off,efifb:off vfio_iommu_type1.allow_unsafe_interrupts=1 isolcpus=4-15,20-31 pcie_acs_override=downstream,multifunction initrd=/bzroot

    ( Do not just copy & paste this)

 

 

I guess you do not allow unsafe interrupts?

 

What is the behaviour if you try a linux vm?

Edited by giganode
  • Thanks 2
Link to comment
14 hours ago, giganode said:

 

Hey!

 

Let me share my syslinux config which works on my system...

 


label Unraid OS
  menu default
  kernel /bzimage
  append video=vesafb:off,efifb:off vfio_iommu_type1.allow_unsafe_interrupts=1 isolcpus=4-15,20-31 pcie_acs_override=downstream,multifunction initrd=/bzroot

    ( Do not just copy & paste this)

 

 

I guess you do not allow unsafe interrupts?

 

What is the behaviour if you try a linux vm?

 

Hi, thanks for any help here

 

I have efifb:off but not vesafb - is that needed?

No, no unsafe interrupts

I have tried with and without acs_override... no difference

 

Not tried Linux (will later) but also tried i440FX instead of Q35 (with existing NVME so wasnt likely to work) - worse

 

Checked my logs and interesting here

 

ErrorWarningSystemArrayLogin


-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=localtime \
-no-hpet \
-no-shutdown \
-boot strict=on \
-device pcie-root-port,port=0x8,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x1 \
-device pcie-root-port,port=0x9,chassis=2,id=pci.2,bus=pcie.0,addr=0x1.0x1 \
-device pcie-root-port,port=0xa,chassis=3,id=pci.3,bus=pcie.0,addr=0x1.0x2 \
-device pcie-root-port,port=0xb,chassis=4,id=pci.4,bus=pcie.0,addr=0x1.0x3 \
-device pcie-root-port,port=0xc,chassis=5,id=pci.5,bus=pcie.0,addr=0x1.0x4 \
-device pcie-root-port,port=0xd,chassis=6,id=pci.6,bus=pcie.0,addr=0x1.0x5 \
-device pcie-root-port,port=0xe,chassis=7,id=pci.7,bus=pcie.0,addr=0x1.0x6 \
-device pcie-root-port,port=0xf,chassis=8,id=pci.8,bus=pcie.0,addr=0x1.0x7 \
-device pcie-root-port,port=0x10,chassis=9,id=pci.9,bus=pcie.0,addr=0x2 \
-device ich9-usb-ehci1,id=usb,bus=pcie.0,addr=0x7.0x7 \
-device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pcie.0,multifunction=on,addr=0x7 \
-device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pcie.0,addr=0x7.0x1 \
-device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pcie.0,addr=0x7.0x2 \
-device virtio-serial-pci,id=virtio-serial0,bus=pci.2,addr=0x0 \
-netdev tap,fd=33,id=hostnet0 \
-device virtio-net,netdev=hostnet0,id=net0,mac=52:54:00:ed:cd:78,bus=pci.1,addr=0x0 \
-chardev pty,id=charserial0 \
-device isa-serial,chardev=charserial0,id=serial0 \
-chardev socket,id=charchannel0,fd=34,server,nowait \
-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 \
-device vfio-pci,host=0000:10:00.0,id=hostdev0,bus=pci.3,addr=0x0,romfile=/boot/GPU_Roms/Powercolor.RX5700.rom \
-device vfio-pci,host=0000:10:00.1,id=hostdev1,bus=pci.4,addr=0x0 \
-device vfio-pci,host=0000:0a:00.0,id=hostdev2,bus=pci.5,addr=0x0 \
-device vfio-pci,host=0000:0a:00.1,id=hostdev3,bus=pci.6,addr=0x0 \
-device vfio-pci,host=0000:0a:00.3,id=hostdev4,bus=pci.7,addr=0x0 \
-device vfio-pci,host=0000:0d:00.0,id=hostdev5,bus=pci.8,addr=0x0 \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
2021-04-15 09:07:11.798+0000: Domain id=2 is tainted: high-privileges
2021-04-15 09:07:11.798+0000: Domain id=2 is tainted: host-cpu
char device redirected to /dev/pts/0 (label charserial0)
2021-04-15T09:07:15.788543Z qemu-system-x86_64: vfio: Cannot reset device 0000:10:00.1, no available reset mechanism.
2021-04-15T09:07:15.932195Z qemu-system-x86_64: vfio: Cannot reset device 0000:10:00.1, no available reset mechanism.
2021-04-15T09:08:52.266352Z qemu-system-x86_64: terminating on signal 15 from pid 11862 (/usr/sbin/libvirtd)
2021-04-15 09:08:58.100+0000: shutting down, reason=shutdown


ErrorWarningSystemArrayLogin


-nodefaults \
-chardev socket,id=charmonitor,fd=31,server,nowait \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=localtime \
-no-hpet \
-no-shutdown \
-boot strict=on \
-device pcie-root-port,port=0x8,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x1 \
-device pcie-root-port,port=0x9,chassis=2,id=pci.2,bus=pcie.0,addr=0x1.0x1 \
-device pcie-root-port,port=0xa,chassis=3,id=pci.3,bus=pcie.0,addr=0x1.0x2 \
-device pcie-root-port,port=0xb,chassis=4,id=pci.4,bus=pcie.0,addr=0x1.0x3 \
-device pcie-root-port,port=0xc,chassis=5,id=pci.5,bus=pcie.0,addr=0x1.0x4 \
-device pcie-root-port,port=0xd,chassis=6,id=pci.6,bus=pcie.0,addr=0x1.0x5 \
-device pcie-root-port,port=0xe,chassis=7,id=pci.7,bus=pcie.0,addr=0x1.0x6 \
-device pcie-root-port,port=0xf,chassis=8,id=pci.8,bus=pcie.0,addr=0x1.0x7 \
-device pcie-root-port,port=0x10,chassis=9,id=pci.9,bus=pcie.0,addr=0x2 \
-device ich9-usb-ehci1,id=usb,bus=pcie.0,addr=0x7.0x7 \
-device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pcie.0,multifunction=on,addr=0x7 \
-device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pcie.0,addr=0x7.0x1 \
-device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pcie.0,addr=0x7.0x2 \
-device virtio-serial-pci,id=virtio-serial0,bus=pci.2,addr=0x0 \
-netdev tap,fd=33,id=hostnet0 \
-device virtio-net,netdev=hostnet0,id=net0,mac=52:54:00:ed:cd:78,bus=pci.1,addr=0x0 \
-chardev pty,id=charserial0 \
-device isa-serial,chardev=charserial0,id=serial0 \
-chardev socket,id=charchannel0,fd=34,server,nowait \
-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 \
-device vfio-pci,host=0000:10:00.0,id=hostdev0,bus=pci.3,addr=0x0,romfile=/boot/GPU_Roms/Powercolor.RX5700.rom \
-device vfio-pci,host=0000:10:00.1,id=hostdev1,bus=pci.4,addr=0x0 \
-device vfio-pci,host=0000:0a:00.0,id=hostdev2,bus=pci.5,addr=0x0 \
-device vfio-pci,host=0000:0a:00.1,id=hostdev3,bus=pci.6,addr=0x0 \
-device vfio-pci,host=0000:0a:00.3,id=hostdev4,bus=pci.7,addr=0x0 \
-device vfio-pci,host=0000:0d:00.0,id=hostdev5,bus=pci.8,addr=0x0 \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
2021-04-15 09:09:54.742+0000: Domain id=3 is tainted: high-privileges
2021-04-15 09:09:54.742+0000: Domain id=3 is tainted: host-cpu
char device redirected to /dev/pts/0 (label charserial0)
2021-04-15T09:10:04.876682Z qemu-system-x86_64: vfio: Cannot reset device 0000:10:00.1, no available reset mechanism.
2021-04-15T09:10:05.020320Z qemu-system-x86_64: vfio: Cannot reset device 0000:10:00.1, no available reset mechanism.

 

It hangs on 2nd boot (this is both listed here)

 

Looks to me like the Reset ISNT working, since that is where it hangs!

Link to comment
  • ich777 changed the title to [Support] ich777 - AMD Vendor Reset, CoralTPU, hpsahba,...

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.