kubed_zero

Community Developer
  • Posts

    166
  • Joined

  • Last visited

Posts posted by kubed_zero

  1. 1 hour ago, DjangoDurano said:

     

    Thanks for your help.

    I google for a long time but it's really hard to find older slackware packages. Older "stable" versions are can be found for example glibc 2.33 but other versions not.

    I know https://slackware.uk/slackware/ but there are also only newer versions. The only thing is, that there are slackBuild scripts, maybe it is possible to build a previous version with that scripts.


    https://slackware.uk/search?p=%2F&q=glibc-2.37
    https://slackware.uk/cumulative/slackware64-current/slackware64/l/glibc-2.37-x86_64-1.txz

    https://slackware.uk/cumulative/slackware64-current/slackware64/l/glibc-2.37-x86_64-2.txz
    https://slackware.uk/cumulative/slackware64-current/slackware64/l/glibc-2.37-x86_64-3.txz

    Behold! With some digging through that website, old versions can be found

  2. I just finished building Python 3.12.1 using similar instructions to above.

     

    • I now run "find usr/lib*/python* -name '*.so' | xargs strip --strip-unneeded" instead of "find . -print0 | xargs -0 file | grep -e "executable" -e "shared object" | grep ELF  | cut -f 1 -d : | xargs strip --strip-unneeded 2> /dev/null || true"
    • I used a longer list of packages installed at compile time, some of these not needed for Python but needed for BorgBackup
    • I found that using newer versions of OpenSSL and glibc would allow for successful compilation, but failures during executions of Python with errors such as "python3: /lib64/libm.so.6: version GLIBC_2.38' not found (required by python3)" and sure enough, Unraid 6.12.6 has only "/lib64/libm-2.37.so"
    Python-3.12.1.tar.xz             libffi-3.4.4-x86_64-1.txz
    acl-2.3.1-x86_64-1.txz           libmpc-1.3.1-x86_64-1.txz
    binutils-2.41-x86_64-1.txz       libzip-1.10.1-x86_64-1.txz
    bzip2-1.0.8-x86_64-3.txz         lzlib-1.13-x86_64-1.txz
    expat-2.5.0-x86_64-1.txz         make-4.4.1-x86_64-1.txz
    fuse3-3.15.0-x86_64-1.txz        openssl-3.1.1-x86_64-1.txz
    gc-8.2.4-x86_64-1.txz            openssl-solibs-3.1.1-x86_64-1.txz
    gcc-13.2.0-x86_64-1.txz          openssl11-1.1.1w-x86_64-1.txz
    gcc-g++-13.2.0-x86_64-1.txz      openssl11-solibs-1.1.1w-x86_64-1.txz
    gdbm-1.23-x86_64-1.txz           pkg-config-0.29.2-x86_64-4.txz
    git-2.43.0-x86_64-1.txz          readline-8.2.001-x86_64-1.txz
    glibc-2.37-x86_64-2.txz          xz-5.4.3-x86_64-1.txz
    guile-3.0.9-x86_64-1.txz         zlib-1.2.13-x86_64-1.txz
    kernel-headers-6.1.64-x86-1.txz


    I've attached the compiled python3-3.12.1 TXZ file, as well as the compiled pyfuse3-3.3.0 and borgbackup-1.2.7 WHL files that can be installed with Pip and are not otherwise available for Unraid. As far as I've tested in my use cases, they work as expected. 

    Below are my notes on the Python Wheel compilation for borgbackup and pyfuse3:

     

    ## 2023-12-10 Pyfuse3 Update
    - Installing pyfuse3 before BorgBackup, as it's a dependency of Borg
    - Installing some Slackware packages first:
    ```
    binutils-2.41-x86_64-1.txz  glibc-2.37-x86_64-2.txz
    fuse3-3.16.2-x86_64-1.txz   kernel-headers-6.1.64-x86-1.txz
    gcc-13.2.0-x86_64-1.txz     pkg-config-0.29.2-x86_64-4.txz
    ```

       - Then run `pip3 install pyfuse3` which is building 3.3.0
       - It installs a few pip3 packages as dependencies:
    ```
    trio-0.23.1-py3-none-any.whl
    attrs-23.1.0-py3-none-any.whl
    sortedcontainers-2.4.0-py2.py3-none-any.whl
    idna-3.6-py3-none-any.whl
    outcome-1.3.0.post0-py2.py3-none-any.whl
    sniffio-1.3.0-py3-none-any.whl

    Successfully installed attrs-23.1.0 idna-3.6 outcome-1.3.0.post0 pyfuse3-3.3.0 sniffio-1.3.0 sortedcontainers-2.4.0 trio-0.23.1
    ```

    - Permanently fetch all these packages (other than pyfuse3) with `pip3 download attrs==23.1.0 idna==3.6 outcome==1.3.0.post0 sniffio==1.3.0 sortedcontainers==2.4.0 trio==0.23.1` and put them in `/boot/python_wheels/` for offline installation at boot time
    - Copy the `pyfuse3` wheel file from the directory stated in the build logs `cp /root/.cache/pip/wheels/b0/c1/b2/bd1f9969742c3b690d74ae13233287eec544c5a9135497443e/pyfuse3-3.3.0-cp312-cp312-linux_x86_64.whl /boot/python_wheels/` 
        - This would have appeared to be something similar to `Created wheel for pyfuse3: filename=pyfuse3-3.3.0-cp312-cp312-linux_x86_64.whl size=1283409 sha256=8471a14517ee73366b6b7fd7dc2921c57327830f0bb2d58a4569abf0e2ca50ad Stored in directory: /root/.cache/pip/wheels/b0/c1/b2/bd1f9969742c3b690d74ae13233287eec544c5a9135497443e`
        - Note that renaming needs to follow some strict guidelines https://peps.python.org/pep-0491/#file-name-convention otherwise Pip will error out and give responses such as `ERROR: pyfuse3-3.3.0-cp312-cp312-linux_x86_64-kubed20231210.whl is not a valid wheel filename` or `ERROR: pyfuse3-3.3.0-cp312-cp312-linux_x86_64_kubed20231210.whl is not a supported wheel on this platform`
    - Add a line to the `/boot/config/go` file to automatically install `pyfuse3` at boot, or just run it adhoc to confirm it works: `/usr/bin/pip3 install /boot/python_wheels/pyfuse3* --no-index --find-links file:///boot/python_wheels`
    ## 2023-12-10 Borg 1.2.7 Update
    - OK, I'm now running Python 3.12.1 as compiled above
    - I also have pyfuse3 installed in Pip in preparation
    - (Sidenote) https://forums.unraid.net/topic/129200-plug-in-nerdtools/?do=findComment&comment=1291205 are my old instructions for compiling Borg
    - (Sidenote) there's now a SlackBuild for installing Borg on Slackware! Updated in 2023Q3 which is more recent than the last time I updated this https://slackbuilds.org/slackbuilds/15.0/system/borgbackup/borgbackup.SlackBuild
    - Following https://borgbackup.readthedocs.io/en/stable/installation.html#pip-installation generally
    - First it notes to follow https://borgbackup.readthedocs.io/en/stable/installation.html#source-install and get some Slackware dependencies installed
        - Intentionally skipping the installation of `libxxhash` and `libzstd` and `liblz4` as the desire is to use the bundled code instead of the system-provided libraries
        - The instructions call for `acl` and `pkg-config` only, but the others are needed based on trial and error
    ```
    acl-2.3.1-x86_64-1.txz      glibc-2.37-x86_64-2.txz
    binutils-2.41-x86_64-1.txz  kernel-headers-6.1.64-x86-1.txz
    gcc-13.2.0-x86_64-1.txz     pkg-config-0.29.2-x86_64-4.txz
    ```
    - Then it asks to install some Pip dependencies JUST for the build process: `wheel`, `setuptools`, `pkgconfig`
        - This can be done with `pip3 download pkgconfig setuptools wheel && pip3 install *.whl`
    ```
    pkgconfig-1.5.5-py3-none-any.whl    wheel-0.42.0-py3-none-any.whl
    setuptools-69.0.2-py3-none-any.whl
    ```
    - Now run `pip3 install "borgbackup[pyfuse3]"` to install borgbackup and build the wheel. Note that this adds `pyfuse3` integration, but that can be skipped by running the simplified install command `pip3 install borgbackup`
    - It installs a few pip3 packages as dependencies:
    ```
    msgpack-1.0.7-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
    packaging-23.2-py3-none-any.whl

    Successfully installed borgbackup-1.2.7 msgpack-1.0.7 packaging-23.2
    ```

    - Permanently fetch these packages (other than borgbackup) with `pip3 download msgpack==1.0.7 packaging==23.2` and put them in `/boot/python_wheels/` for offline installation at boot time
    - Copy the `borgbackup` wheel file from the directory stated in the build logs `cp /root/.cache/pip/wheels/b0/c1/b2/bd1f9969742c3b690d74ae13233287eec544c5a9135497443e/pyfuse3-3.3.0-cp312-cp312-linux_x86_64.whl /boot/python_wheels/` 
        - This would have appeared to be something similar to `Created wheel for borgbackup: filename=borgbackup-1.2.7-cp312-cp312-linux_x86_64.whl size=6248226 sha256=830ec3fbcc5c74922f7e269e8378aa08052c8c9cec51f407c53166d989d62e7c Stored in directory: /root/.cache/pip/wheels/64/90/02/c60bc19558d5e1e7c5ed13041bc27d25d41518ddfdb2def852`
        - Note that renaming needs to follow some strict guidelines https://peps.python.org/pep-0491/#file-name-convention otherwise Pip will error out
    - Add a line to the `/boot/config/go` file to automatically install `pyfuse3` at boot, or just run it adhoc to confirm it works: `/usr/bin/pip3 install /boot/python_wheels/borgbackup* --no-index --find-links file:///boot/python_wheels`
    - The final list of wheel files ready to install at boot time are:
    ```
    attrs-23.1.0-py3-none-any.whl
    borgbackup-1.2.7-cp312-cp312-linux_x86_64.whl
    idna-3.6-py3-none-any.whl
    msgpack-1.0.7-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
    outcome-1.3.0.post0-py2.py3-none-any.whl
    packaging-23.2-py3-none-any.whl
    pyfuse3-3.3.0-cp312-cp312-linux_x86_64.whl
    sniffio-1.3.0-py3-none-any.whl
    sortedcontainers-2.4.0-py2.py3-none-any.whl
    trio-0.23.1-py3-none-any.whl
    ```

    pyfuse3-3.3.0-cp312-cp312-linux_x86_64.whl python3-3.12.1-x86_64-1-kubed20231210.txz borgbackup-1.2.7-cp312-cp312-linux_x86_64.whl

  3. On 9/29/2023 at 6:10 PM, kesm said:

    Thanks a lot for this, 

    Is there any particular reason for this PR to not be added to nerdtools? I'm trying to use a script that rely on 3.11 so I can't use it right now. Hope to see python3.11 soon in nerdtools



    I'm not aware of any reason for it not to be added. The repo owner seems to be MIA, not responding to the PR, not updating the repo, not commenting here. 

     

    On 10/12/2023 at 2:19 AM, HellScytheX said:

    Is it possible to add RHash into the Nerdtools pack?

    https://github.com/rhash/RHash 

     

    I'm unable to build and compile the tool myself as it requires a gcc compiler, which I haven't been able to mange to install in my unRaid system (the old DevPack plugin is incompatible with unRaid versions > 6.10.3).

     

    Otherwise if there is perhaps any other tool for doing md5 hash checks, which is perhaps a little more versatile than the default md5sum utility.

     

    Thanks!

    I have notes in my PR for installing the necessary compiling tools to compile on Unraid https://github.com/UnRAIDES/unRAID-NerdTools/pull/84

    I also made an application I call DiffLens, which uses Blake3 hashes instead of MD5 hashes to do file integrity validations on Unraid shares https://github.com/UnRAIDES/unRAID-NerdTools/pull/84

  4. Right now, any disk's Identity page allows defining a YYYYMMDD for purchase date and a YYYYMMDD for manufacture date. There is a category for Warranty Period, but it is preconfigured to only allow from 6-50 months. There is no option to add a YYYYMMDD, and no option to add a custom number of months.

     

    2035907523_Unraid6_12.3DiskIdentity.thumb.png.86d6cd2d033b3d62a63b8422a6473bb0.png

    This should be a straightforward UI change to allow for more than just the pre-configured dropdown. 

     

    This would be useful because Seagate, Western Digital, and I'm sure other HDD manufacturers all have warranty pages that state the exact end date of warranty periods. For those buying used drives or RMA'd drives, warranty expiry does not always align nicely with manufacture date or purchase date. 

     

     

    Thanks for considering this change! 

  5. And as a side note, I was able to get Borg and PyFuse3 Pip wheels compiled and functional with no additional system packages needed. They're attached here. They can be installed with "pip3 install /path/to/wheel" and can be used immediately after. Note that pyfuse3 is only needed if a Borg repository needs to be mounted. Otherwise it can be skipped. 

     

    This falls out of the purview of NerdTools, but I figure it would be of interest to the group here since the Borgbackup system package is the only other way of getting it working on Unraid. This is easborgbackup-1.2.4-cp311-cp311-linux_x86_64.whlpyfuse3-3.2.3-cp311-cp311-linux_x86_64.whlier and more reliable, in my opinion, as Pip manages the dependency installation and upgrading can be done by replacing only the Wheel file.

     

    Here are my development notes:

     



    ## Compiling Borg 1.2.4 on Unraid 6.12.3
    - Starting with a fresh reboot, plus the openSSL and python 3.11.4 package installations I figured out earlier
    - `pip3 install borgbackup`
    - Installing the following packages as precursors

    pkg-config-0.29.2-x86_64-4.txz
    gcc-13.2.0-x86_64-1.txz
    glibc-2.37-x86_64-2.txz
    kernel-headers-6.1.42-x86-1.txz
    binutils-2.41-x86_64-1.txz
    acl-2.3.1-x86_64-1.txz
    
    
    python:
    msgpack-1.0.5-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
    packaging-23.1-py3-none-any.whl
    pip        23.2.1
    setuptools 65.5.0


    - Errors seen and their package fix if there was an issue:
        - `OSError: pkg-config probably not installed: FileNotFoundError(2, 'No such file or directory')` is fixed by installing pkg-config
        - `error: command 'gcc' failed: No such file or directory` is fixed by installing gcc
        - `/usr/include/openssl/evp.h:22:12: fatal error: stdio.h: No such file or directory` is fixed by installing glibc
        - `/usr/include/bits/local_lim.h:38:10: fatal error: linux/limits.h: No such file or directory` is fixed by kernel-headers
        - `gcc: fatal error: cannot execute ‘as’: execvp: No such file or directory` is fixed by installing binutils
        - ` src/borg/platform/linux.c:1104:10: fatal error: sys/acl.h: No such file or directory`  is fixed by installing acl
        - That was it. After installing those dependencies, the build succeeded. I rebooted and reinstalled to confirm. 


    Compiling PyFUSE3 on Unraid 6.12.3
    - Starting with a fresh reboot, plus the openSSL and python 3.11.4 package installations I figured out earlier
    - `pip3 install pyfuse3`
    - Installing the following packages as precursors

    pkg-config-0.29.2-x86_64-4.txz
    fuse3-3.15.0-x86_64-1.txz
    gcc-13.2.0-x86_64-1.txz
    glibc-2.37-x86_64-2.txz
    kernel-headers-6.1.42-x86-1.txz
    binutils-2.41-x86_64-1.txz
    
    python:
    trio-0.22.2-py3-none-any.whl
    attrs-23.1.0-py3-none-any.whl
    sortedcontainers-2.4.0-py2.py3-none-any.whl
    idna-3.4-py3-none-any.whl
    outcome-1.2.0-py2.py3-none-any.whl
    sniffio-1.3.0-py3-none-any.whl


    - Errors seen and their package fix if there was an issue:
        - `FileNotFoundError: [Errno 2] No such file or directory: 'pkg-config'` fixed by installing pkg-config
        - `Package fuse3 was not found in the pkg-config search path` fixed by installing fuse3
        - `error: command 'gcc' failed: No such file or directory` fixed by installing gcc
        - `/usr/include/python3.11/Python.h:23:12: fatal error: stdlib.h: No such file or directory` fixed by installing glibc
        - `/usr/include/bits/errno.h:26:11: fatal error: linux/errno.h: No such file or directory` fixed by installing kernel-headers
        - `gcc: fatal error: cannot execute ‘as’: execvp: No such file or directory` fixed by installing binutils
        - And then it built! I rebooted and confirmed it worked as well. Seems good. 

  6. I've submitted a PR to the NerdTools GitHub repo to upgrade Python 3.9 to Python 3.11. https://github.com/UnRAIDES/unRAID-NerdTools/pull/84 As far as I could tell, not a single person had shared a Python 3.11 build for Slackware anywhere on the internet, so I took it upon myself to get an Unraid development environment functional so I could compile Python 3.11 specially for Unraid.

    I also included Setuptools 65.5.0 and Pip 23.2.1 as part of the package, so there is no longer a need to install them separately. They can always be updated using Pip if desired, for example "pip3 install --upgrade pip" 

     

    I've tested it with a couple different Python projects, including my Difflens project as well as Borgbackup 1.2.4, and as far as I can see it's working well for my uses. There are a few modules that are not compiled into the Python package. For example, OpenSSL now needs >1.1.1 to be installed on the system as of Python 3.10, which Unraid does not meet by default. For that reason, the OpenSSL package will need to be installed alongside the Python package for Python/Pip to work correctly. I've updated the NerdTools dependency list to include OpenSSL, and it should get installed automatically when installing Python 3.11. 

     

    Some other modules that I didn't compile were "`_dbm  _tkinter    _uuid nis    readline" so they will need to be installed via Pip or another method if necessary.

     

    See 

    for more detail on development 

  7. All right, I got things working. 

     

    Starting with Python 3.10, OpenSSL 1.1.1 or newer is required https://peps.python.org/pep-0644/ which means the OpenSSL 1.1 installed by default in Unraid 6.12.3 does not meet the requirements. In other words, using Python 3.10 or newer will require OpenSSL to also be installed as a prerequisite. I got OpenSSL 3.1.2 from https://slackware.uk/slackware/slackware64-current/slackware64/n/openssl-3.1.2-x86_64-1.txz and added it to the /boot/extra/ directory to install at boot. 

     

    I also spent a bunch of time working through the different settings and libraries needed to configure and build/compile Python 3.11.4 on Unraid. I eventually succeeded, and am happy to share the 3.11.4 TXZ file. I put this in the /boot/extra/ directory alongside OpenSSL and am happy to report that Python seems to be working for my needs. I included Pip in the TXZ as well, so Pip  will install automatically next to Python and does not need a separate package to install

     

    As far as I'm aware, I'm the only person to have successfully/publicly shared a Python 3.10 or 3.11 built for Slackware or Unraid. With that in mind, here are some notes I took along the way:

     

     

     

    I read through http://www.slackware.com/~alien/slackbuilds/python3/build/python3.SlackBuild for inspiration for a SlackBuild file, but did not fully end up using it


    My required package dependency for building steps
        - gcc https://slackware.uk/slackware/slackware64-current/slackware64/d/gcc-13.2.0-x86_64-1.txz and `installpkg gcc-13.2.0-x86_64-1.txz ` to fix `configure: error: no acceptable C compiler found in $PATH`
        - binutils https://slackware.uk/slackware/slackware64-current/slackware64/d/binutils-2.41-x86_64-1.txz to fix `configure: error: C compiler cannot create executables` which stems from `gcc: fatal error: cannot execute 'as': execvp: No such file or directory` https://stackoverflow.com/questions/56801179/fatal-error-cannot-execute-as-execvp-no-such-file-or-directory
        - glibc https://slackware.uk/slackware/slackware64-current/slackware64/l/glibc-2.37-x86_64-2.txz to fix `configure: error: C compiler cannot create executables`  with sub-error  `/ld: cannot find crt1.o: No such file or directory`
        - kernel-headers https://slackware.uk/slackware/slackware64-current/slackware64/d/kernel-headers-6.1.42-x86-1.txz to fix `C preprocessor "/lib/cpp" fails sanity check` with sub-error `linux/limits.h: No such file or directory`
        - **Everything above here is the minimal set needed to get `./configure` running end to end**
        - make https://slackware.uk/slackware/slackware64-current/slackware64/d/make-4.4.1-x86_64-1.txz obviously just to run make
        - guile https://slackware.uk/slackware/slackware64-current/slackware64/d/guile-3.0.9-x86_64-1.txz to fix the make-time error `make: error while loading shared libraries: libguile-3.0.so.1: cannot open shared object file: No such file or directory`
        - gc https://slackware.uk/slackware/slackware64-current/slackware64/l/gc-8.2.4-x86_64-1.txz to fix the make-time error `make: error while loading shared libraries: libgc.so.1: cannot open shared object file: No such file or directory`
        - zlib https://slackware.uk/slackware/slackware64-current/slackware64/l/zlib-1.2.13-x86_64-1.txz RIP after 45m of compiling I saw an error `zipimport.ZipImportError: can't decompress data; zlib not available`
        - libffi https://slackware.uk/slackware/slackware64-current/slackware64/l/libffi-3.4.4-x86_64-1.txz to fix compilation error `Failed to build these modules: _ctypes` and `_ctypes.c:118:10: fatal error: ffi.h: No such file or directory`
        - openssl (not openssl11, which corresponds to 1.1 and not the 3.x of normal openssl) https://slackware.uk/slackware/slackware64-current/slackware64/n/openssl-3.1.1-x86_64-1.txz to let `./configure` know that SSL exists so that hopefully Python builds the SSL module `checking for openssl/ssl.h in /usr... yes / checking whether compiling and linking against OpenSSL works... yes`
            - I see `/usr/bin/openssl` and `/usr/lib64/libevent_openssl.so` and `/usr/lib64/libssl.so` seem to exist on the test Unraid, so maybe it's safe to assume it's available by default? I suspect pkg-config will aid in detecting that it's available
            - Nope, I see `checking whether compiling and linking against OpenSSL works... no` when running `/configure`
            - Yeah, even when running make I see the same `Could not build the ssl module! Python requires a OpenSSL 1.1.1 or newer` error
            - I see that `./configure` has a log line `checking for stdlib extension module _ssl... missing` which makes me suspect that SSL won't be built into this Python package installer, and thus pip is going to fail once I get to that point. I bet I could get around that by installing SSL using the instructions from earlier, but then I suspect it won't really be bundled and will instead just look to the system for an SSL implementation. That said, I wonder if I could install pkg-config AND SSL and set the https://docs.python.org/3/using/configure.html#cmdoption-with-openssl-rpath `--with-openssl-rpath=auto` flag during configuration and then it would just use whatever SSL is available? The alternative is to just build this, realize SSL is missing, install the SSL package as well, and call it a day. I think it should technically be fine if the python installer also depends on SSL
        - pkg-config to resolve a general compiler warning https://slackware.uk/slackware/slackware64-current/slackware64/d/pkg-config-0.29.2-x86_64-4.txz that `WARNING: pkg-config is missing. Some dependencies may not be detected correctly.`
        - bzip2 https://slackware.uk/slackware/slackware64-current/slackware64/a/bzip2-1.0.8-x86_64-3.txz to fix `checking for stdlib extension module _bz2... missing` in the `./configure`
        - xz https://slackware.uk/slackware/slackware64-current/slackware64/a/xz-5.4.3-x86_64-1.txz to fix to fix `checking for stdlib extension module _lzma... missing` in the `./configure`
        - gdbm https://slackware.uk/slackware/slackware64-current/slackware64/l/gdbm-1.23-x86_64-1.txz to fix `checking for stdlib extension module _gdbm... missing` in the `./configure`
        - ncurses https://slackware.uk/slackware/slackware64-current/slackware64/l/ncurses-6.4_20230610-x86_64-1.txz to fix failed compilation of the `_ncurses` module



    **My steps**
        - Download python 3 source in xz format to `/boot/buildPython/Python-3.11.4.tar.xz`
        - Create a temp directory to do work in, and move there: `rm -rf /tmp/buildPython | exit 0 && mkdir /tmp/buildPython && cd /tmp/buildPython`
        - Install the dependencies needed to run `./configure` and `make` in `/boot/buildPython`
        - Extract the Python source to the temp directory `tar xf /boot/buildPython/Python-3.11.4.tar.xz` which creates a subdirectory `/tmp/buildPython/Python-3.11.4/`
        - Move into that subdirectory `cd /tmp/buildPython/Python-3.11.4/`
        - Try to run `./configure --build=x86_64-slackware-linux --with-ensurepip=upgrade --prefix=/usr --libdir=/usr/lib64 --with-platlibdir=lib64 --enable-optimizations --with-lto --with-pkg-config=yes --disable-test-modules --without-static-libpython`
        - Run make with multiple jobs (assuming multiple CPUs) to compile faster: `make -j6`
        - Clear and create the directory that `make install` will output into `rm -rf /tmp/package-python-make-output | exit 0 && mkdir -p /tmp/package-python-make-output`
        - Run make with `make install DESTDIR=/tmp/package-python-make-output`
        - Now remove some of the files in `/tmp/package-python-make-output` that should not be packaged:
            - `cd /tmp/package-python-make-output/`
            - `find . \( -name '*.exe' -o -name '*.bat' \) -exec rm -f '{}' \+` to remove windows stuff 
            - `find . -type d -exec chmod 755 "{}" \+` to update permissions
            - `find . -perm 640 -exec chmod 644 "{}" \+`
            - `find . -perm 750 -exec chmod 755 "{}" \+`
            - `find . -print0 | xargs -0 file | grep -e "executable" -e "shared object" | grep ELF  | cut -f 1 -d : | xargs strip --strip-unneeded 2> /dev/null || true` I have no idea what this does but it seemed to find a couple files such as `./usr/lib64/python3.11/lib-dynload/termios.cpython-311-x86_64-linux-gnu.so` or `./usr/lib64/python3.11/lib-dynload/_codecs_tw.cpython-311-x86_64-linux-gnu.so` so maybe it's removing extended attributes? IDK
            - `strip -s usr/lib/* usr/lib64/* usr/bin/*` as recommended by https://docs.slackware.com/howtos:slackware_admin:building_a_package
        - `mkdir install && cd install && wget https://www.slackbuilds.org/slackbuilds/14.2/python/python3/slack-desc` to add a `slack-desc` file to the package
        - edit the usr/bin/pip3 and other pip files to point to python3 instead of just python, as this package doesn't install a default python
        - `cd /tmp/package-python-make-output && /sbin/makepkg -l y -c n /tmp/python3-3.11.4-x86_64-1.txz` to actually make the output Slackware package using all the files in the `/tmp/package-python-make-output` directory

    python3-3.11.4-x86_64-20230802kubedwithpip.txz

  8. I've been trying to get Python 3.11 installed on Unraid 6.11 or 6.12. I looked around for pre-compiled TXZ packages, but only found 3.9.x packages from @dlandon in https://github.com/dlandon/python3 and @EUGENI_CAT in https://github.com/UnRAIDES/unRAID-NerdTools/blob/main/packages/6.11/python3-3.9.16-x86_64-1.txz

     

    Usually for Slackware package compilation, a SlackBuild file is provided to configure the system before calling the package's make file. The newest SlackBuild I could find out there was for 3.9.5 http://www.slackware.com/~alien/slackbuilds/python3/build/python3.SlackBuild and it is not Unraid specific. 

    I was able to install a bunch of support packages on Unraid 6.12.3 and get the SlackBuild working:

     

    Python-3.11.4.tar.xz             libmpc-1.3.1-x86_64-1.txz
    binutils-2.40-x86_64-1.txz       libzip-1.10.0-x86_64-1.txz
    expat-2.5.0-x86_64-1.txz         lzlib-1.13-x86_64-1.txz
    gc-8.2.4-x86_64-1.txz            make-4.4.1-x86_64-1.txz
    gcc-13.1.0-x86_64-2.txz          openssl-3.1.1-x86_64-1.txz
    gcc-g++-13.1.0-x86_64-2.txz      openssl-solibs-3.1.1-x86_64-1.txz
    git-2.41.0-x86_64-1.txz          openssl11-1.1.1u-x86_64-1.txz
    glibc-2.37-x86_64-2.txz          openssl11-solibs-1.1.1u-x86_64-1.txz
    guile-3.0.9-x86_64-1.txz         pkg-config-0.29.2-x86_64-4.txz
    kernel-headers-6.1.35-x86-1.txz  python3.SlackBuild
    libffi-3.4.4-x86_64-1.txz        zlib-1.2.13-x86_64-1.txz

     
    However, I've had some issues after installing this custom-built Python. I bundled Pip into this install since that's supported with a makefile flag, but I started seeing  "Can't connect to HTTPS URL because the SSL module is not available." after rebooting Unraid. I suspect this is because the SSL package and system libraries were installed during compilation, so Python doesn't reference its internal SSL library and instead uses the system library, which after an Unraid reboot no longer exists. 

    I've seen a couple other issues along this line as well.

    I'm hoping someone here is aware of an Unraid-compatible  Python 3.11 TXZ file, or a SlackBuild file that could get Python compiled and functional on Unraid. Or if someone here simply has more experience with compiling/SlackBuild/makefiles on Unraid, maybe that would also work. 

     

    Thanks!

  9. 1 hour ago, d0rh4ns said:

    Thanks a bunch, kubed_zero! I ended up modifying that disk_temps.sh script to also include the drives that were missing before and pointed SNMP to that modified copy. 

     

    Man, if only there was a proper UnRAID API to not have to deal with hacky workarounds like this.

    Nice. Just in case you weren't aware, the modifications will be reset on each reboot, so you'll probably want to make a copy of the modified script on the /boot USB drive and then reference that script in your SNMP config instead of the default one.

    Also, if your fix is generalized, feel free to submit a pull request and I can see about adding this for wider usage 😇

    • Upvote 1
  10. 8 hours ago, d0rh4ns said:

    First of all, thanks for the super quick reply!!!

     

    So what I'm getting from this is that mdcmd status only outputs array disks and no cache pools etc. 

    What I don't quite understand is if there's a way to get those drives in there and if so, how. 

    That is an Unraid-specific executable as far as I'm aware, so I think trying to get it showing up there would be a dead end.

    On the other hand, you don't actually need mdcmd to get disk temp. That responsibility is held by smartctl, which is not unraid-specific. You just give it a disk mount point (aka /dev/sdb) and then it will report back temperature and SMART information https://github.com/kubedzero/unraid-snmp/blob/main/source/usr/local/emhttp/plugins/snmp/disk_temps.sh#L120

    So you could make a script that determines what the non-pool drives are mounted as, and then calls smartctl for those /dev/sdX  paths. And then if you added that script to your SNMP config, it would start to show up in SNMP.

    smartctl can actually do a scan for attached devices, aka `smartctl --scan` so you could always start there as well.

     

    Hope that makes sense! Feel free to browse through the Github code for the disk temp script for further insight.

    • Like 2
  11. 9 minutes ago, d0rh4ns said:

    I'm hoping I haven't missed this information somewhere, but is there a way to get disk temperature of disks that are not in the array (i.e. in pools) ? I can only see array drives when I do

     

    snmpwalk -v 2c localhost -O n -c public 'NET-SNMP-EXTEND-MIB::nsExtendOutLine."disktemp"'

     

    Already tried adding 

     

    disk /mnt/pool

     

    to the SNMP configuration (the pool is called pool 🙂) but nothing...

     

    Any advice?


    There is a separate standalone script that parses the output of "mdcmd status" to determine what disks are installed: https://github.com/kubedzero/unraid-snmp/blob/main/source/usr/local/emhttp/plugins/snmp/disk_temps.sh#L52C18-L52C30
     

     here is some more information on the output of the mdcmd command

    The SNMP configuration only will adjust the free/used disk space output and won't add on disk temperature. 

  12. Ah, I found the true root cause of everyone's issue. https://linux.die.net/man/8/snmpd notes the different flags. Looking more closely at the output from ps, I see the default settings for snmpd are to call with the -a flag which "Logs the source addresses of incoming requests" hence the huge log files.

     

    root@Test:~# ps -ef | grep snmp
    root      2070     1  0 18:54 ?        00:00:00 /usr/sbin/snmpd -A -p /var/run/snmpd -a -c /etc/snmp/snmpd.conf
    root      2933  2715  0 18:55 pts/0    00:00:00 grep snmp


    So the root cause is twofold: I didn't have the doinst.sh script built in a way that was compatible with SNMP 5.9.3, and this allowed the default flag to log all addresses and fill up people's disks. 

    This is fixed in the version I just released, 2023.04.20 (blaze it). https://github.com/kubedzero/unraid-snmp/releases/tag/2023.04.20 Now the ps output should look the same as what I have below:

     

    root@Test:~# ps -ef | grep snmp
    root      4749     1  0 19:20 ?        00:00:00 /usr/sbin/snmpd -A -p /var/run/snmpd -LF 0-5 /var/log/snmpd.log -c /etc/snmp/snmpd.conf
    root      5247  5167  0 19:21 pts/0    00:00:00 grep snmp


    One can also verify the correct behavior by looking at the update log output in the plugin update popup window.

    This is wrong, because the desired flags don't match the actual flags:

    Editing SNMP startup options in rc.snmpd to be [-LF 0-5 /var/log/snmpd.log -A -p /var/run/snmpd -a]
    Restart SNMP daemon now that we've adjusted how rc.snmpd starts it
    Starting snmpd: /usr/sbin/snmpd -A -p /var/run/snmpd -a -c /etc/snmp/snmpd.conf


    This is the correct output:

    Writing extra SNMPD_OPTIONS into /etc/default/snmpd to configure logging
    Start SNMP daemon back up now that snmpd.conf and /etc/default/snmpd modifications are done
    Starting snmpd: /usr/sbin/snmpd -A -p /var/run/snmpd -LF 0-5 /var/log/snmpd.log -c /etc/snmp/snmpd.conf


    Thanks all and let me know if you have issues!

    • Like 1
  13. On 4/16/2023 at 12:33 PM, Uncore said:

    when I take a look at /etc/rc.d/rc.snmpd

     

    9 hours ago, Uncore said:

    version I'm running is the latest stable 6.11.5


    OK, I've figured out the issue and will have a fix out soon. The newest version of the SNMP plugin (2023.02.19) declares a dependency on net-snmp-5.9.3-x86_64-1.txz. Meanwhile the old version (2021.05.21) uses net-snmp-5.9-x86_64-1.txz. 

    The problem is that the net-snmp package differs in its /etc/rc.d/rc.snmpd file from 5.9 to 5.9.3, so the do-inst.sh file in my own unraid-snmp plugin fails to modify the file to adjust the logging. 

     

    Old /etc/rc.d/rc.snmpd snippet:

    OPTIONS="-A -p /var/run/snmpd -a"
    
    start() {
    	if [ -x /usr/sbin/snmpd -a -f /etc/snmp/snmpd.conf ]; then
    		echo -n "Starting snmpd: "
    		/usr/sbin/snmpd $OPTIONS -c /etc/snmp/snmpd.conf
    		echo " /usr/sbin/snmpd $OPTIONS -c /etc/snmp/snmpd.conf"
    	fi
    }

     

    New /etc/rc.d/rc.snmpd snippet:

    [ -r /etc/default/snmpd ] && . /etc/default/snmpd
    SNMPD_OPTIONS=${SNMPD_OPTIONS:-"-A -p /var/run/snmpd -a"}
    
    start() {
    	if [ -x /usr/sbin/snmpd -a -f /etc/snmp/snmpd.conf ]; then
    		echo -n "Starting snmpd: "
    		/usr/sbin/snmpd $SNMPD_OPTIONS -c /etc/snmp/snmpd.conf
    		echo " /usr/sbin/snmpd $SNMPD_OPTIONS -c /etc/snmp/snmpd.conf"
    	fi
    }



    My modification code, compatible with 5.9 but not 5.9.3:

        new_flags="-LF 0-5 /var/log/snmpd.log "
        options=$(grep "OPTIONS=" /etc/rc.d/rc.snmpd | cut -d'"' -f 2)
        if [[ $options != *"-L"* ]]; then
            options=$new_flags$options
            echo "Editing SNMP startup options in rc.snmpd to be [$options]"
            sed --in-place=.bak --expression "s|^OPTIONS=.*|OPTIONS=\"$options\"|" /etc/rc.d/rc.snmpd
        else
            echo "SNMP logging flag already present in rc.snmpd, skipping modification"
        fi


    I need to change this code to something that can handle the new syntax. Some thoughts I'll be exploring:

    • I saw /etc/default/snmpd included in the new SNMP package but not the old, and perhaps that could be used
    • I need to check the syntax, but I might be able to just set an environment variable SNMPD_OPTIONS as the install script now seems to merge whatever it's set to with the values declared in /etc/rc.d/rc.snmpd
    • I could update my script to just do the text modification inside the /etc/rc.d/rc.snmpd file, the same as it was before but with updated find and replace logic

     

    Hopefully I'll have a plugin update out in the next week. Thanks @Uncore for the details!

  14. On 4/16/2023 at 12:33 PM, Uncore said:

    I look a look at the do-inst.sh script and tried tracing its steps

    You didn't need to do this, I explicitly noted this in an earlier message: 

     

     

    On 4/2/2023 at 3:29 PM, kubed_zero said:

    Specifically, there is a line that sets up the logging https://github.com/kubedzero/unraid-snmp/blob/main/source/install/doinst.sh#L58

     

     

    anyway...

     

    On 4/16/2023 at 12:33 PM, Uncore said:

    when I take a look at /etc/rc.d/rc.snmpd the variable doesn't appear to be updated and is missing the logging syntax

    Odd. I'm running 6.9.2 and have no issues:

     

    root@Unraid:~# cat /etc/rc.d/rc.snmpd 
    #!/bin/sh
    #
    # rc.snmpd      This shell script takes care of starting and stopping
    #               the net-snmp SNMP daemon
    
    OPTIONS="-LF 0-5 /var/log/snmpd.log -A -p /var/run/snmpd -a"

     

    root@Unraid:~# ps -ef | grep snmp
    root      3744     1  0 Apr15 ?        00:01:35 /usr/sbin/snmpd -LF 0-5 /var/log/snmpd.log -A -p /var/run/snmpd -a -c /etc/snmp/snmpd.conf
    root     19665 19643  0 17:33 pts/0    00:00:00 grep snmp

     

     

     

     

     

    53 minutes ago, irishjd said:

    every time I reboot, the config reverts.

    Yes, this is expected. A clean install of SNMP would be:
     

    - Uninstall SNMP plugin, and ensure it fully completes and removes all the SNMP-related files in /boot/config/plugins
    - Reboot Unraid
    - Reinstall SNMP after reboot

     

    On 4/16/2023 at 12:33 PM, Uncore said:

    the snmpd.log file is no longer growing.


    That's great news. That means the issue people are seeing seems to be from SNMP's logging modification not installing properly. Again, odd that I can't reproduce this and haven't seen this myself. What version of Unraid? 

  15. 1 hour ago, stuartcarrison said:

    Did anyone discover a way to turn off verbose logging

    So more people are having this issue, odd. What's in your logs, the same thing as @irishjd

    We were chatting in PMs, but I guess this applicable to more people. I've never had luck with log output.

     

    - What version of Unraid are you using?

    - Did you apply any customizations to the SNMP config?

    - Have you rebooted recently?

    - Has uninstalling and reinstalling the SNMP plugin made any difference?


    I suspect I know the line that is causing this. Inside the unraid-snmp TXZ file there is the do-inst.sh script that configures and modifies everything. Specifically, there is a line that sets up the logging https://github.com/kubedzero/unraid-snmp/blob/main/source/install/doinst.sh#L58

     

    Right now it outputs levels 1=a=alert, 2=c=crit, 3=e=err, 4=w=warn, 5=n=notice into /var/log/snmpd.log. I'd suspect that changing this line to output something else (0-4, 1-4, 1-3, etc) would probably cut down on your log output. That said I've never gotten ANYTHING to show up in /var/log/snmpd.log on my own installations so I've never been able to test. I need reproduction steps to get a better idea of what's happening. 

  16. 13 minutes ago, irishjd said:

    Has anyone seen the issue where the /var/log/snmpd.log fills up the root volume? I can always delete it and restart the service. It would be nice if some logic was added to the plugging where this was done automatically when the volume capacity is less than 20%.

    What is it filled with? The file has always been empty whenever I checked or did development on it. 

  17. Thanks for that info @MVLP. I realized I could optimize the way I'm grabbing CPU MHz so I applied that change and stopped using cat, hopefully it won't happen anymore.

     

    @cjhammel I also added the change to read the Community string from the snmpd.conf file, so changing the Community should no longer fail the SNMP plugin installation after boot.

     

    These changes are a part of 2023.02.19 😄 

    • Like 1
  18. Thanks all for the info. I just released an update 2023.02.18 that should fix this, now reading from /proc/cpuinfo instead of lscpu. I found during testing Unraid 6.12 that my VMs with an updated version of lscpu (2.38 vs 2.36) were no longer outputting CPU MHz at all, much less min and max.

    Granted, my implementation only reads the speed of CPU core 0, but I figure that's better than nothing. Technically someone could submit a PR that grabs the CPU MHz from all the different cores and averages them out, or gets the max (or min) from all of them.


    Let me know how install/update and testing goes @mattie112@MVLP @irishjd

    • Like 1