jc179

Hi, On Unraid 7.2.3, I upgraded from open source nvidia kernel module 580.95.05 to open source nvidia kernel module 590.48.01, and run into problems with ollama not working with both GPU (3060, and 5060). See the below in the dmesg: [ 1816.413718] NVRM: iovaspaceDestruct_IMPL: 1 left-over mappings in IOVAS 0x400 [ 1816.435493] NVRM: GPU1 nvAssertFailedNoLog: Assertion failed: pIOVAS != NULL @ io_vaspace. c:592 [ 1816.435576] NVRM: GPU1 nvAssertFailedNoLog: Assertion failed: pIOVAS != NULL @ io_vaspace. c:601 [ 2374.905012] NVRM: GPU1 kbifInitLtr_GB202: LTR is disabled in the hierarchy I would like to go back to the previous version of open source nvidia module 580.95.05 but it is no longer available as a selection under " open source driver " in the nvidia driver settings page, and has been replaced with 590.48.01, which I am having problems with. Selecting the 580.95.05 version there seems to load the nvidia proprietary version and not the open source version. I did some searching, and it found the some notes on editing /boot/config/plugins/nvidia-driver/settings.cfg and setting driver_version=580.95.05 however that does not seem to be working either. Is there anyway to just undo / go back to the previous driver version for now? edit: figured it out: downloaded both of the below wget https://api.github.com/repos/unraid/unraid-nvidia-driver/releases/tags/6.12.54-Unraid/nvos-580.95.05-6.12.54-Unraid-1.txz https://github.com/unraid/unraid-nvidia-driver/releases/download/6.12.54-Unraid/nvos-580.95.05-6.12.54-Unraid-1.txz.md5 and copy both into /boot/config/plugins/nvidia-driver/packages/6.12.54 next edit /boot/config/plugins/nvidia-driver/settings.cfg first_installation=no driver_version=latest_nos disable_xconfig=false update_check=false finally reboot

I understand - thank you .

I understand I should upgrade but presently its missing some addons I need, and my dockers don't start automatically, and only manually start works. Of course eventually I will have no choice but to solve the problem and upgrade.... Appreciate you taking the time to do this for me, hopefully one day I can figure out how to do this.

OK, I was able to get the unraid-kernel docker setup, specified gcc_11.2.0. I wasn't sure if need to run the installscript.sh, but tried with and without, both times ran into some issues with missing dependencies for cmake ( openssl-solibs, libarchive, libxml2). Manually installed them (from slackware 15), think is OK for now. I took the kernel ".config" file from 6.1.125, and copy into 6.1.126 and was able to "make modules" successfully ! Good start! I cloned unraid-nvidia-driver to /usr/src, and ran into a few questions. I checked out the compile.sh, and noted the instructions and the bottom, but upon trying the example commands and even older versions, it doesn't seem to kick anything off. I see the wget looks like its designed to fetch the nvidia package and then run it with command line options you've figured out, I'd really love to use that if I can figure out how its intended to be run? I am not sure what ${DATA_DIR} should be, variable not defined in the docker, can only assume there is some sort of setup I missed or inadvertently skipped? 1. compile.sh - what should DATA_DIR be defined as ? Assume /usr/src? The kernel helper docker by default doesn't have it defined and I've probably missed where that step somewhere? "root@0052624a534f:/usr/src# echo $DATA_DIR root@0052624a534f:/usr/src# set | grep -i data root@0052624a534f:/usr/src# env | grep -i data " 2. Following the examples in: bash compile.sh "580.126.09" "opensource" only sets a few variables, and doesn't do anything; bash compile.sh "580.126.09" "opensource" I added some echo statements { "branches": { "production": { "current": "580.126.09", "last_prb": "580.119.02" }, "newfeature": { "current": "590.48.01", "last_nfb": "575.64.05" }, "beta": { "current": "590.44.01" }, "legacy": { "current": "470.256.02" } }, "runtimes": { "containertoolkit": { "current": "1.18.1" }, "libnvidia": { "current": "1.18.1" } } } lib and contv 1.18.1 and 1.18.1 I of course tried calling with just bash compile.sh "580.126.09" and even "580", and "580.126" ... ? No matter it quits, compile.sh has a function called nvidia_driver() inside, but it never gets called? I manually called it at the bottom of the script which of course forced some action to happen, but not the expected. I assume this isn't how its intended to be run? Any hints? thanks very much!

Ah, thank you so much ich777! This is very helpful info honestly, years ago I compiled a custom kernel for Unraid ~6.8 for the Fusion-IO cards, and was heart broken when that was no longer possible with the newer versions. I'm not exactly sure on how to build the Nvidia driver, as this likely more complex than compiling the driver into the kernel, or making a loadable .ko module, none the less it will be a fine adventure ..... o7 . Really appreciate you taking the time to put all that together over the years and share the link and the environment to compile a driver. The installscript.sh alone is very helpful to get all the proper autoconf and make, gcc, etc. That was always such a pain with slackware not having a package manager that can just pull and or build (eg emerge). I will need to figure out lots, but intend to follow though on this until Unraid V7 has more what I'm after. Thanks, Jonathan

Hi, Is it possible to upgrade open kernel driver for unraid 6.12.15 ? I have a 5000 series GPU I want to use, and Unraid 7.2 doesn't startup my dockers correctly (rc.docker: Docker daemon... Failed. but they startup OK manually, etc /etc/rc.d/rc.docker start) , lacks nerdtools and everything else I want to use, as a result I backed out to 6.12.15 until 7.x is improved. Presently on 6.12.15 I have only the option of Open Source Driver: v565.57.01 . In Unraid 7.2.3, "NVIDIA UNIX Open Kernel Mode Setting Driver for x86_64 580.95.05 " worked. I see driver 580.95 supports kernel 6.1. Is it possible I can modify the nvidia plugin startup script to install a newer packaged driver other than the old nvos-565.57.01-6.1.126-Unraid-1.txz ? Or run the NVIDIA-Linux-x86_64-580.95.05.run thanks, Jonathan

ich777 - They have a few different series; ioDrive ioDrive DUO ioDrive2 Probably more. As these get cheaper it might be possible to get you one, I saw some 640gb ones on kijiji here not too long ago. igi - Yes, under windows they're working well, and with VMWare ESX too, perhaps what I should have done a long time ago instead of going the other path I chose to at the time... The TBW lifespan of the black SN700 isn't very good for my use though; ENDURANCE (TBW): 2TB: 1,200 1TB: 600 500GB: 300 250GB: 200 (https://documents.westerndigital.com/content/dam/doc-library/en_us/assets/public/western-digital/product/internal-drives/wd-black-ssd/product-brief-wd-black-sn770-nvme-ssd.pdf) So Fill up the 1 TB, 600 times and it has already reached the end of what they will cover in 5 year warranty, pretty easy to do that in 2 years in a server like environment, not so good for a server, workload depending that is. Even if it was 2000 TBW, that's only 1TB written 2000 times. With heavy swap and VM-IO this goes quickly. These 'enterprise' drives are rated to write in the PBW - petabyte, 3 extra 0's , pretend for a moment the 1TB drive was rated at just 6 PBW, then you'd have ~half a million full write cycles vs 600 before it hits the rated drive lifetime. I think I might just try a P3600 and report back weather it works or not... I am leaning more towards a working situation than not at the moment... Jonathan

Welp, I had been using this on the old Unraid 6.8, but finally upgraded to 6.12 and with the amount of time I want to spend trying again, it seems you can no longer compile a module for the kernel anymore? So annoying, just going to move on until I decide to replace unraid. In the mean time, I just want to find a solition that works - does anyone know or can confirm if Intel P3520 or P3600 series will work within unraid? These intel drives seem to support the NVME command set, so maybe ? ... Or any other sort of "durable" MLC NAND flash ? ( do NOT want to use an actual "NVME drive on a pci-express adapter as those seem far too fragile, overheat, poop out, wear out easily.). I enjoy using enterprise class hardware that will outlive its useful life and still work for years...lol. Speaking of which my 3 fusion-io adapters are all still over 95% health, and at least work in Windows. What a fantastic durable product. thanks, Jonathan

Hi there, Hoping someone might be able to help with this... I have unraid 6.10.3 with the latest devtools installed, and cc1 found under /usr/libexec/gcc/x86_64-slackware-linux/9.2.0 seems to be wanting the libisl.so.22 , and not the newer libisl.so.23 I did turn off all addins from within the Dev Pack tool, and apply, reboot, and turn them all backon, apply, and reboot. See below output for LDD root@cerebellum:/usr/libexec/gcc/x86_64-slackware-linux/9.2.0# cc1 cc1: error while loading shared libraries: libisl.so.22: cannot open shared object file: No such file or directory root@cerebellum:/usr/libexec/gcc/x86_64-slackware-linux/9.2.0# root@cerebellum:/usr/libexec/gcc/x86_64-slackware-linux/9.2.0# ldd cc1 linux-vdso.so.1 (0x00007ffeb848b000) libisl.so.22 => not found libmpc.so.3 => /usr/lib64/libmpc.so.3 (0x00001485e4b9f000) libmpfr.so.6 => /usr/lib64/libmpfr.so.6 (0x00001485e48f4000) libgmp.so.10 => /usr/lib64/libgmp.so.10 (0x00001485e487b000) libdl.so.2 => /lib64/libdl.so.2 (0x00001485e4876000) libz.so.1 => /lib64/libz.so.1 (0x00001485e485c000) libm.so.6 => /lib64/libm.so.6 (0x00001485e4712000) libc.so.6 => /lib64/libc.so.6 (0x00001485e4533000) /lib64/ld-linux-x86-64.so.2 (0x00001485e4bca000) root@cerebellum:/usr/libexec/gcc/x86_64-slackware-linux/9.2.0# See LIBISL /usr/lib64/libisl.so.23 /usr/lib64/libisl.so.23.1.0 root@cerebellum:/usr/libexec/gcc/x86_64-slackware-linux/9.2.0# ldconfig -p | grep -i isl libisl.so.23 (libc6,x86-64) => /usr/lib64/libisl.so.23 thanks for any advice tips or tricks to resolve this... Jonathan

Messing around further, I tried using a loop device back to the /dev/fioa, but even *that* doesn't show up in the Unraid GUI, lsscsi also doesn't see it, probably because it is major #7, (major #8, hdd block devices, losetup won't do it ). again, all a guess... mknod /dev/sdn b 7 200 losetup /dev/sdn /dev/fioa udevadm control --reload-rules udevadm trigger root@cerebellum:/# fdisk -l /dev/sdn Disk /dev/sdn: 596.5 GiB, 640000000000 bytes, 1250000000 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: gpt Disk identifier: 5D116A01-F6AC-45FB-94A9-C3881252FA51 Device Start End Sectors Size Type /dev/sdn1 34 32767 32734 16M Microsoft reserved /dev/sdn2 32768 1249996799 1249964032 596G Microsoft basic data root@cerebellum:/# Perhaps if we could modify the error checking of losetup (opensource recompile), so that this would work, then this might show up, with the caveat of a hefty performance impact, but I'd still use it honestly... crude quick speed test: root@cerebellum:/# dd if=/dev/fioa of=/dev/null bs=1024k count=20000 20000+0 records in 20000+0 records out 20971520000 bytes (21 GB, 20 GiB) copied, 14.4936 s, 1.4 GB/s root@cerebellum:/# dd if=/dev/sdn of=/dev/null bs=1024k count=20000 20000+0 records in 20000+0 records out 20971520000 bytes (21 GB, 20 GiB) copied, 24.5716 s, 853 MB/s root@cerebellum:/# This fails... root@cerebellum:/dev# mknod /dev/sdo b 8 200 root@cerebellum:/dev# losetup /dev/sdo /dev/fioa losetup: /dev/sdo: failed to set up loop device: No such device or address Maybe someone else has another idea to try, maybe emhttpd could be patched to recognize the fioa device or something else... I don't know... sucks... Just want to make this work, id even take half speed... lol. Is like pandora's magic box here... Jonathan

You know what, I was so fixated with that I forgot to look at what the script is actually doing, and looked in the wrong place. I do see the card shows up under /dev/disk/by-id !!! root@cerebellum:/dev/disk/by-id# ls -al total 0 drwxr-xr-x 2 root root 1100 Jan 5 14:59 ./ drwxr-xr-x 7 root root 140 Dec 30 13:17 ../ lrwxrwxrwx 1 root root 10 Jan 5 14:59 fio-0630fa34-0505-1441-9d5b-6d79afe74709 -> ../../fioa lrwxrwxrwx 1 root root 11 Jan 5 14:59 fio-0630fa34-0505-1441-9d5b-6d79afe74709-part1 -> ../../fioa1 lrwxrwxrwx 1 root root 11 Jan 5 14:59 fio-0630fa34-0505-1441-9d5b-6d79afe74709-part2 -> ../../fioa2 (I remove all other unecessary info here, eg other disks we're not interested in). Checking in the Unraid GUI, there is nothing new. I didn't have any new unassigned disks, I even stopped the array to have a look, no changes... dang. Was really hoping this would work! Maybe Unraid 6.9 might behave differently? Will see if I can give that a try at some point... There are indeed two partitions on my fio card, and this has mapped them correctly - I can mount and talk to the fio card a-ok as well. The issue might lie that even using the by-id, when mounted, seems to revert back to a /dev/fioa device.... mount /dev/disk/by-id/fio-0630fa34-0505-1441-9d5b-6d79afe74709-part2 /a /dev/fioa2 597G 113M 596G 1% /a /dev/fioa2 on /a type fuseblk (rw,allow_other,blksize=4096) Wish I had better news... Jonathan

Tried compiling with the latest GIT release (v4.2) as I am kernel 4.19 on unraid 6.8.3 still. Managed to compile it, but needed to modify the .sh scripts - I assume no one else has run into this, so it must be something weird with my unraid build env. Basically some instances of "cat" seems to be broken reporting 'standard output: No such file or directory'.... Very odd, not going to dwell on it for now, managed to work around this, and load the module (left some notes below as to what I did) . I added the script you linked to my /lib/udev/rules.d/95-fio.rules , but don't see any new devices show up. There wasn't any info on how exactly to use this either, so I may need to run something to kick off the scirpt and create the new udev devices. I am a little tempted to manually populate and run the below and just see, but I will wait : /usr/bin/fio-status --field iom.format_uuid /dev/%k", SYMLINK+="disk/by-id/fio-%c /usr/bin/fio-status --field iom.format_uuid /dev/%P", SYMLINK+="disk/by-id/fio-%c-part%n If interested, here is the error output from the make make[1]: Leaving directory '/mnt/user/dev/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16' ./kfio_config.sh -a x86_64 -o include/fio/port/linux/kfio_config.h -k /lib/modules/4.19.107-Unraid/build -p -d /mnt/user/dev/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16/kfio_config -l 0 cat: standard output: No such file or directory 1609869599.051 Exiting Preserving configdir due to '-p' option: /mnt/user/dev/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16/kfio_config make: *** [Makefile:88: include/fio/port/linux/kfio_config.h] Error 1 kfio_config.sh complains that "cat: standard output: No such file or directory" which is ridiculous, as cat certainly exists, and is in the path... root@cerebellum:/mnt/user/dev/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16# env | grep PATH MANPATH=/usr/local/man:/usr/man NODE_PATH=/usr/lib64/node_modules PATH=.:/usr/local/sbin:/usr/sbin:/sbin:/usr/local/bin:/usr/bin:/bin root@cerebellum:/mnt/user/dev/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16# which cat /usr/bin/cat Here is the line which it's stopping at, I added an echo to identify: : "${TMP_OUTPUTFILE:=${CONFIGDIR}/kfio_config.h}" echo "cat instance 5" cat <<EOF if I modify the line with the cat commands in kfio_config.sh: from this "cat <<EOF" to "echo <<EOF" ... to use echo instead of cat, we progress !! So the change now looks like: : "${TMP_OUTPUTFILE:=${CONFIGDIR}/kfio_config.h}" echo "cat instance 5" #cat <<EOF echo <<EOF and re-run the line from the make file script, it completes now! ./kfio_config.sh -a x86_64 -o include/fio/port/linux/kfio_config.h -k /lib/modules/4.19.107-Unraid/build -p -d /mnt/user/dev/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16/kfio_config -l 0 So, stop and resume the original make command, which finally finishes: Ok: altered version for iomemory-vsl.ko appended 2a18ca0 ### I don't see any new /dev/sdX devices, I assume the linked script goes somewhere like /lib/udev/rules.d/95-fio.rules ### insmod iomemory-vsl.ko ## dmesg -T | tail -100 [Tue Jan 5 13:38:33 2021] <6>fioinf VSL configuration hash: 8f82ea05bdf1195cb400fb48e4ef09fc49b3c1aa [Tue Jan 5 13:38:33 2021] <6>fioinf [Tue Jan 5 13:38:33 2021] <6>fioinf Copyright (c) 2006-2014 Fusion-io, Inc. (acquired by SanDisk Corp. 2014) [Tue Jan 5 13:38:33 2021] <6>fioinf Copyright (c) 2014-2016 SanDisk Corp. and/or all its affiliates. (acquired by Western Digital Corp. 2016) [Tue Jan 5 13:38:33 2021] <6>fioinf Copyright (c) 2016-2018 Western Digital Technologies, Inc. All rights reserved. [Tue Jan 5 13:38:33 2021] <6>fioinf For Terms and Conditions see the License file included [Tue Jan 5 13:38:33 2021] <6>fioinf with this driver package. [Tue Jan 5 13:38:33 2021] <6>fioinf [Tue Jan 5 13:38:33 2021] <6>fioinf ioDrive driver 3.2.16.1731 4870ad45b7ea 2a18ca0 a loading... [Tue Jan 5 13:38:33 2021] <6>fioinf ioDrive 0000:04:00.0: mapping controller on BAR 5 [Tue Jan 5 13:38:33 2021] <6>fioinf ioDrive 0000:04:00.0: MSI enabled [Tue Jan 5 13:38:33 2021] <6>fioinf ioDrive 0000:04:00.0: using MSI interrupts [Tue Jan 5 13:38:33 2021] <6>fioinf ioDrive 0000:04:00.0.0: Starting master controller [Tue Jan 5 13:38:34 2021] <6>fioinf ioDrive 0000:04:00.0.0: Adapter serial number is 602555 [Tue Jan 5 13:38:34 2021] <6>fioinf ioDrive 0000:04:00.0.0: Board serial number is 602555 [Tue Jan 5 13:38:34 2021] <6>fioinf ioDrive 0000:04:00.0.0: Adapter serial number is 602555 [Tue Jan 5 13:38:34 2021] <6>fioinf ioDrive 0000:04:00.0.0: Default capacity 640.000 GBytes [Tue Jan 5 13:38:34 2021] <6>fioinf ioDrive 0000:04:00.0.0: Default sector size 512 bytes [Tue Jan 5 13:38:34 2021] <6>fioinf ioDrive 0000:04:00.0.0: Rated endurance 10.00 PBytes [Tue Jan 5 13:38:34 2021] <6>fioinf ioDrive 0000:04:00.0.0: 85C temp range hardware found [Tue Jan 5 13:38:34 2021] <6>fioinf ioDrive 0000:04:00.0.0: Firmware version 7.1.17 116786 (0x700411 0x1c832) [Tue Jan 5 13:38:34 2021] <6>fioinf ioDrive 0000:04:00.0.0: Platform version 10 [Tue Jan 5 13:38:34 2021] <6>fioinf ioDrive 0000:04:00.0.0: Firmware VCS version 116786 [0x1c832] [Tue Jan 5 13:38:34 2021] <6>fioinf ioDrive 0000:04:00.0.0: Firmware VCS uid 0xaeb15671994a45642f91efbb214fa428e4245f8a [Tue Jan 5 13:38:34 2021] <6>fioinf ioDrive 0000:04:00.0.0: Powercut flush: Enabled [Tue Jan 5 13:38:34 2021] <6>fioinf ioDrive 0000:04:00.0.0: PCIe power monitor enabled (master). Limit set to 24.750 watts. [Tue Jan 5 13:38:34 2021] <6>fioinf ioDrive 0000:04:00.0.0: Thermal monitoring: Enabled [Tue Jan 5 13:38:34 2021] <6>fioinf ioDrive 0000:04:00.0.0: Hardware temperature alarm set for 85C. [Tue Jan 5 13:38:34 2021] <6>fioinf ioDrive 0000:04:00.0: Found device fct0 (Fusion-io ioDrive 640GB 0000:04:00.0) on pipeline 0 [Tue Jan 5 13:38:36 2021] <6>fioinf Fusion-io ioDrive 640GB 0000:04:00.0: probed fct0 [Tue Jan 5 13:38:36 2021] <6>fioinf Fusion-io ioDrive 640GB 0000:04:00.0: sector_size=512 [Tue Jan 5 13:38:36 2021] <6>fioinf Fusion-io ioDrive 640GB 0000:04:00.0: setting channel range data to [2 .. 4095] [Tue Jan 5 13:38:36 2021] <6>fioinf Fusion-io ioDrive 640GB 0000:04:00.0: Found metadata in EBs 168-169, loading... [Tue Jan 5 13:38:36 2021] <6>fioinf Fusion-io ioDrive 640GB 0000:04:00.0: setting recovered append point 169+198180864 [Tue Jan 5 13:38:36 2021] <6>fioinf Fusion-io ioDrive 640GB 0000:04:00.0: Creating device of size 640000000000 bytes with 1250000000 sectors of 512 bytes (213422 mapped). [Tue Jan 5 13:38:36 2021] fioinf Fusion-io ioDrive 640GB 0000:04:00.0: Creating block device fioa: major: 254 minor: 0 sector size: 512... [Tue Jan 5 13:38:36 2021] fioa: fioa1 fioa2 [Tue Jan 5 13:38:36 2021] <6>fioinf Fusion-io ioDrive 640GB 0000:04:00.0: Attach succeeded. root@cerebellum:/mnt/user/dev# /usr/bin/fio-status Found 1 ioMemory device in this system Driver version: 3.2.16 build 1731 Adapter: Single Adapter Fusion-io ioDrive 640GB, Product Number:FS1-004-640-CS, SN:602555, FIO SN:602555 External Power: NOT connected PCIe Power limit threshold: 24.75W Connected ioMemory modules: fct0: Product Number:FS1-004-640-CS, SN:602555 fct0 Attached ioDrive 640GB, Product Number:FS1-004-640-CS, SN:602555 PCI:04:00.0, Slot Number:5 Firmware v7.1.17, rev 116786 Public 640.00 GBytes device size Internal temperature: 54.14 degC, max 54.63 degC Reserve space status: Healthy; Reserves: 100.00%, warn at 10.00% Contained VSUs: fioa: ID:0, UUID:0630fa34-0505-1441-9d5b-6d79afe74709 fioa State: Online, Type: block device ID:0, UUID:0630fa34-0505-1441-9d5b-6d79afe74709 640.00 GBytes device size

deleted - redundant info