kubed_zero Posted July 31, 2023 Share Posted July 31, 2023 I've been trying to get Python 3.11 installed on Unraid 6.11 or 6.12. I looked around for pre-compiled TXZ packages, but only found 3.9.x packages from @dlandon in https://github.com/dlandon/python3 and @EUGENI_CAT in https://github.com/UnRAIDES/unRAID-NerdTools/blob/main/packages/6.11/python3-3.9.16-x86_64-1.txz Usually for Slackware package compilation, a SlackBuild file is provided to configure the system before calling the package's make file. The newest SlackBuild I could find out there was for 3.9.5 http://www.slackware.com/~alien/slackbuilds/python3/build/python3.SlackBuild and it is not Unraid specific. I was able to install a bunch of support packages on Unraid 6.12.3 and get the SlackBuild working: Python-3.11.4.tar.xz libmpc-1.3.1-x86_64-1.txz binutils-2.40-x86_64-1.txz libzip-1.10.0-x86_64-1.txz expat-2.5.0-x86_64-1.txz lzlib-1.13-x86_64-1.txz gc-8.2.4-x86_64-1.txz make-4.4.1-x86_64-1.txz gcc-13.1.0-x86_64-2.txz openssl-3.1.1-x86_64-1.txz gcc-g++-13.1.0-x86_64-2.txz openssl-solibs-3.1.1-x86_64-1.txz git-2.41.0-x86_64-1.txz openssl11-1.1.1u-x86_64-1.txz glibc-2.37-x86_64-2.txz openssl11-solibs-1.1.1u-x86_64-1.txz guile-3.0.9-x86_64-1.txz pkg-config-0.29.2-x86_64-4.txz kernel-headers-6.1.35-x86-1.txz python3.SlackBuild libffi-3.4.4-x86_64-1.txz zlib-1.2.13-x86_64-1.txz However, I've had some issues after installing this custom-built Python. I bundled Pip into this install since that's supported with a makefile flag, but I started seeing "Can't connect to HTTPS URL because the SSL module is not available." after rebooting Unraid. I suspect this is because the SSL package and system libraries were installed during compilation, so Python doesn't reference its internal SSL library and instead uses the system library, which after an Unraid reboot no longer exists. I've seen a couple other issues along this line as well. I'm hoping someone here is aware of an Unraid-compatible Python 3.11 TXZ file, or a SlackBuild file that could get Python compiled and functional on Unraid. Or if someone here simply has more experience with compiling/SlackBuild/makefiles on Unraid, maybe that would also work. Thanks! Quote Link to comment
kubed_zero Posted August 3, 2023 Author Share Posted August 3, 2023 All right, I got things working. Starting with Python 3.10, OpenSSL 1.1.1 or newer is required https://peps.python.org/pep-0644/ which means the OpenSSL 1.1 installed by default in Unraid 6.12.3 does not meet the requirements. In other words, using Python 3.10 or newer will require OpenSSL to also be installed as a prerequisite. I got OpenSSL 3.1.2 from https://slackware.uk/slackware/slackware64-current/slackware64/n/openssl-3.1.2-x86_64-1.txz and added it to the /boot/extra/ directory to install at boot. I also spent a bunch of time working through the different settings and libraries needed to configure and build/compile Python 3.11.4 on Unraid. I eventually succeeded, and am happy to share the 3.11.4 TXZ file. I put this in the /boot/extra/ directory alongside OpenSSL and am happy to report that Python seems to be working for my needs. I included Pip in the TXZ as well, so Pip will install automatically next to Python and does not need a separate package to install As far as I'm aware, I'm the only person to have successfully/publicly shared a Python 3.10 or 3.11 built for Slackware or Unraid. With that in mind, here are some notes I took along the way: I read through http://www.slackware.com/~alien/slackbuilds/python3/build/python3.SlackBuild for inspiration for a SlackBuild file, but did not fully end up using it My required package dependency for building steps - gcc https://slackware.uk/slackware/slackware64-current/slackware64/d/gcc-13.2.0-x86_64-1.txz and `installpkg gcc-13.2.0-x86_64-1.txz ` to fix `configure: error: no acceptable C compiler found in $PATH` - binutils https://slackware.uk/slackware/slackware64-current/slackware64/d/binutils-2.41-x86_64-1.txz to fix `configure: error: C compiler cannot create executables` which stems from `gcc: fatal error: cannot execute 'as': execvp: No such file or directory` https://stackoverflow.com/questions/56801179/fatal-error-cannot-execute-as-execvp-no-such-file-or-directory - glibc https://slackware.uk/slackware/slackware64-current/slackware64/l/glibc-2.37-x86_64-2.txz to fix `configure: error: C compiler cannot create executables` with sub-error `/ld: cannot find crt1.o: No such file or directory` - kernel-headers https://slackware.uk/slackware/slackware64-current/slackware64/d/kernel-headers-6.1.42-x86-1.txz to fix `C preprocessor "/lib/cpp" fails sanity check` with sub-error `linux/limits.h: No such file or directory` - **Everything above here is the minimal set needed to get `./configure` running end to end** - make https://slackware.uk/slackware/slackware64-current/slackware64/d/make-4.4.1-x86_64-1.txz obviously just to run make - guile https://slackware.uk/slackware/slackware64-current/slackware64/d/guile-3.0.9-x86_64-1.txz to fix the make-time error `make: error while loading shared libraries: libguile-3.0.so.1: cannot open shared object file: No such file or directory` - gc https://slackware.uk/slackware/slackware64-current/slackware64/l/gc-8.2.4-x86_64-1.txz to fix the make-time error `make: error while loading shared libraries: libgc.so.1: cannot open shared object file: No such file or directory` - zlib https://slackware.uk/slackware/slackware64-current/slackware64/l/zlib-1.2.13-x86_64-1.txz RIP after 45m of compiling I saw an error `zipimport.ZipImportError: can't decompress data; zlib not available` - libffi https://slackware.uk/slackware/slackware64-current/slackware64/l/libffi-3.4.4-x86_64-1.txz to fix compilation error `Failed to build these modules: _ctypes` and `_ctypes.c:118:10: fatal error: ffi.h: No such file or directory` - openssl (not openssl11, which corresponds to 1.1 and not the 3.x of normal openssl) https://slackware.uk/slackware/slackware64-current/slackware64/n/openssl-3.1.1-x86_64-1.txz to let `./configure` know that SSL exists so that hopefully Python builds the SSL module `checking for openssl/ssl.h in /usr... yes / checking whether compiling and linking against OpenSSL works... yes` - I see `/usr/bin/openssl` and `/usr/lib64/libevent_openssl.so` and `/usr/lib64/libssl.so` seem to exist on the test Unraid, so maybe it's safe to assume it's available by default? I suspect pkg-config will aid in detecting that it's available - Nope, I see `checking whether compiling and linking against OpenSSL works... no` when running `/configure` - Yeah, even when running make I see the same `Could not build the ssl module! Python requires a OpenSSL 1.1.1 or newer` error - I see that `./configure` has a log line `checking for stdlib extension module _ssl... missing` which makes me suspect that SSL won't be built into this Python package installer, and thus pip is going to fail once I get to that point. I bet I could get around that by installing SSL using the instructions from earlier, but then I suspect it won't really be bundled and will instead just look to the system for an SSL implementation. That said, I wonder if I could install pkg-config AND SSL and set the https://docs.python.org/3/using/configure.html#cmdoption-with-openssl-rpath `--with-openssl-rpath=auto` flag during configuration and then it would just use whatever SSL is available? The alternative is to just build this, realize SSL is missing, install the SSL package as well, and call it a day. I think it should technically be fine if the python installer also depends on SSL - pkg-config to resolve a general compiler warning https://slackware.uk/slackware/slackware64-current/slackware64/d/pkg-config-0.29.2-x86_64-4.txz that `WARNING: pkg-config is missing. Some dependencies may not be detected correctly.` - bzip2 https://slackware.uk/slackware/slackware64-current/slackware64/a/bzip2-1.0.8-x86_64-3.txz to fix `checking for stdlib extension module _bz2... missing` in the `./configure` - xz https://slackware.uk/slackware/slackware64-current/slackware64/a/xz-5.4.3-x86_64-1.txz to fix to fix `checking for stdlib extension module _lzma... missing` in the `./configure` - gdbm https://slackware.uk/slackware/slackware64-current/slackware64/l/gdbm-1.23-x86_64-1.txz to fix `checking for stdlib extension module _gdbm... missing` in the `./configure` - ncurses https://slackware.uk/slackware/slackware64-current/slackware64/l/ncurses-6.4_20230610-x86_64-1.txz to fix failed compilation of the `_ncurses` module **My steps** - Download python 3 source in xz format to `/boot/buildPython/Python-3.11.4.tar.xz` - Create a temp directory to do work in, and move there: `rm -rf /tmp/buildPython | exit 0 && mkdir /tmp/buildPython && cd /tmp/buildPython` - Install the dependencies needed to run `./configure` and `make` in `/boot/buildPython` - Extract the Python source to the temp directory `tar xf /boot/buildPython/Python-3.11.4.tar.xz` which creates a subdirectory `/tmp/buildPython/Python-3.11.4/` - Move into that subdirectory `cd /tmp/buildPython/Python-3.11.4/` - Try to run `./configure --build=x86_64-slackware-linux --with-ensurepip=upgrade --prefix=/usr --libdir=/usr/lib64 --with-platlibdir=lib64 --enable-optimizations --with-lto --with-pkg-config=yes --disable-test-modules --without-static-libpython` - Run make with multiple jobs (assuming multiple CPUs) to compile faster: `make -j6` - Clear and create the directory that `make install` will output into `rm -rf /tmp/package-python-make-output | exit 0 && mkdir -p /tmp/package-python-make-output` - Run make with `make install DESTDIR=/tmp/package-python-make-output` - Now remove some of the files in `/tmp/package-python-make-output` that should not be packaged: - `cd /tmp/package-python-make-output/` - `find . \( -name '*.exe' -o -name '*.bat' \) -exec rm -f '{}' \+` to remove windows stuff - `find . -type d -exec chmod 755 "{}" \+` to update permissions - `find . -perm 640 -exec chmod 644 "{}" \+` - `find . -perm 750 -exec chmod 755 "{}" \+` - `find . -print0 | xargs -0 file | grep -e "executable" -e "shared object" | grep ELF | cut -f 1 -d : | xargs strip --strip-unneeded 2> /dev/null || true` I have no idea what this does but it seemed to find a couple files such as `./usr/lib64/python3.11/lib-dynload/termios.cpython-311-x86_64-linux-gnu.so` or `./usr/lib64/python3.11/lib-dynload/_codecs_tw.cpython-311-x86_64-linux-gnu.so` so maybe it's removing extended attributes? IDK - `strip -s usr/lib/* usr/lib64/* usr/bin/*` as recommended by https://docs.slackware.com/howtos:slackware_admin:building_a_package - `mkdir install && cd install && wget https://www.slackbuilds.org/slackbuilds/14.2/python/python3/slack-desc` to add a `slack-desc` file to the package - edit the usr/bin/pip3 and other pip files to point to python3 instead of just python, as this package doesn't install a default python - `cd /tmp/package-python-make-output && /sbin/makepkg -l y -c n /tmp/python3-3.11.4-x86_64-1.txz` to actually make the output Slackware package using all the files in the `/tmp/package-python-make-output` directory python3-3.11.4-x86_64-20230802kubedwithpip.txz Quote Link to comment
kubed_zero Posted December 11, 2023 Author Share Posted December 11, 2023 I just finished building Python 3.12.1 using similar instructions to above. I now run "find usr/lib*/python* -name '*.so' | xargs strip --strip-unneeded" instead of "find . -print0 | xargs -0 file | grep -e "executable" -e "shared object" | grep ELF | cut -f 1 -d : | xargs strip --strip-unneeded 2> /dev/null || true" I used a longer list of packages installed at compile time, some of these not needed for Python but needed for BorgBackup I found that using newer versions of OpenSSL and glibc would allow for successful compilation, but failures during executions of Python with errors such as "python3: /lib64/libm.so.6: version GLIBC_2.38' not found (required by python3)" and sure enough, Unraid 6.12.6 has only "/lib64/libm-2.37.so" Python-3.12.1.tar.xz libffi-3.4.4-x86_64-1.txz acl-2.3.1-x86_64-1.txz libmpc-1.3.1-x86_64-1.txz binutils-2.41-x86_64-1.txz libzip-1.10.1-x86_64-1.txz bzip2-1.0.8-x86_64-3.txz lzlib-1.13-x86_64-1.txz expat-2.5.0-x86_64-1.txz make-4.4.1-x86_64-1.txz fuse3-3.15.0-x86_64-1.txz openssl-3.1.1-x86_64-1.txz gc-8.2.4-x86_64-1.txz openssl-solibs-3.1.1-x86_64-1.txz gcc-13.2.0-x86_64-1.txz openssl11-1.1.1w-x86_64-1.txz gcc-g++-13.2.0-x86_64-1.txz openssl11-solibs-1.1.1w-x86_64-1.txz gdbm-1.23-x86_64-1.txz pkg-config-0.29.2-x86_64-4.txz git-2.43.0-x86_64-1.txz readline-8.2.001-x86_64-1.txz glibc-2.37-x86_64-2.txz xz-5.4.3-x86_64-1.txz guile-3.0.9-x86_64-1.txz zlib-1.2.13-x86_64-1.txz kernel-headers-6.1.64-x86-1.txz I've attached the compiled python3-3.12.1 TXZ file, as well as the compiled pyfuse3-3.3.0 and borgbackup-1.2.7 WHL files that can be installed with Pip and are not otherwise available for Unraid. As far as I've tested in my use cases, they work as expected. Below are my notes on the Python Wheel compilation for borgbackup and pyfuse3: ## 2023-12-10 Pyfuse3 Update - Installing pyfuse3 before BorgBackup, as it's a dependency of Borg - Installing some Slackware packages first: ``` binutils-2.41-x86_64-1.txz glibc-2.37-x86_64-2.txz fuse3-3.16.2-x86_64-1.txz kernel-headers-6.1.64-x86-1.txz gcc-13.2.0-x86_64-1.txz pkg-config-0.29.2-x86_64-4.txz ``` - Then run `pip3 install pyfuse3` which is building 3.3.0 - It installs a few pip3 packages as dependencies: ``` trio-0.23.1-py3-none-any.whl attrs-23.1.0-py3-none-any.whl sortedcontainers-2.4.0-py2.py3-none-any.whl idna-3.6-py3-none-any.whl outcome-1.3.0.post0-py2.py3-none-any.whl sniffio-1.3.0-py3-none-any.whl Successfully installed attrs-23.1.0 idna-3.6 outcome-1.3.0.post0 pyfuse3-3.3.0 sniffio-1.3.0 sortedcontainers-2.4.0 trio-0.23.1 ``` - Permanently fetch all these packages (other than pyfuse3) with `pip3 download attrs==23.1.0 idna==3.6 outcome==1.3.0.post0 sniffio==1.3.0 sortedcontainers==2.4.0 trio==0.23.1` and put them in `/boot/python_wheels/` for offline installation at boot time - Copy the `pyfuse3` wheel file from the directory stated in the build logs `cp /root/.cache/pip/wheels/b0/c1/b2/bd1f9969742c3b690d74ae13233287eec544c5a9135497443e/pyfuse3-3.3.0-cp312-cp312-linux_x86_64.whl /boot/python_wheels/` - This would have appeared to be something similar to `Created wheel for pyfuse3: filename=pyfuse3-3.3.0-cp312-cp312-linux_x86_64.whl size=1283409 sha256=8471a14517ee73366b6b7fd7dc2921c57327830f0bb2d58a4569abf0e2ca50ad Stored in directory: /root/.cache/pip/wheels/b0/c1/b2/bd1f9969742c3b690d74ae13233287eec544c5a9135497443e` - Note that renaming needs to follow some strict guidelines https://peps.python.org/pep-0491/#file-name-convention otherwise Pip will error out and give responses such as `ERROR: pyfuse3-3.3.0-cp312-cp312-linux_x86_64-kubed20231210.whl is not a valid wheel filename` or `ERROR: pyfuse3-3.3.0-cp312-cp312-linux_x86_64_kubed20231210.whl is not a supported wheel on this platform` - Add a line to the `/boot/config/go` file to automatically install `pyfuse3` at boot, or just run it adhoc to confirm it works: `/usr/bin/pip3 install /boot/python_wheels/pyfuse3* --no-index --find-links file:///boot/python_wheels` ## 2023-12-10 Borg 1.2.7 Update - OK, I'm now running Python 3.12.1 as compiled above - I also have pyfuse3 installed in Pip in preparation - (Sidenote) https://forums.unraid.net/topic/129200-plug-in-nerdtools/?do=findComment&comment=1291205 are my old instructions for compiling Borg - (Sidenote) there's now a SlackBuild for installing Borg on Slackware! Updated in 2023Q3 which is more recent than the last time I updated this https://slackbuilds.org/slackbuilds/15.0/system/borgbackup/borgbackup.SlackBuild - Following https://borgbackup.readthedocs.io/en/stable/installation.html#pip-installation generally - First it notes to follow https://borgbackup.readthedocs.io/en/stable/installation.html#source-install and get some Slackware dependencies installed - Intentionally skipping the installation of `libxxhash` and `libzstd` and `liblz4` as the desire is to use the bundled code instead of the system-provided libraries - The instructions call for `acl` and `pkg-config` only, but the others are needed based on trial and error ``` acl-2.3.1-x86_64-1.txz glibc-2.37-x86_64-2.txz binutils-2.41-x86_64-1.txz kernel-headers-6.1.64-x86-1.txz gcc-13.2.0-x86_64-1.txz pkg-config-0.29.2-x86_64-4.txz ``` - Then it asks to install some Pip dependencies JUST for the build process: `wheel`, `setuptools`, `pkgconfig` - This can be done with `pip3 download pkgconfig setuptools wheel && pip3 install *.whl` ``` pkgconfig-1.5.5-py3-none-any.whl wheel-0.42.0-py3-none-any.whl setuptools-69.0.2-py3-none-any.whl ``` - Now run `pip3 install "borgbackup[pyfuse3]"` to install borgbackup and build the wheel. Note that this adds `pyfuse3` integration, but that can be skipped by running the simplified install command `pip3 install borgbackup` - It installs a few pip3 packages as dependencies: ``` msgpack-1.0.7-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl packaging-23.2-py3-none-any.whl Successfully installed borgbackup-1.2.7 msgpack-1.0.7 packaging-23.2 ``` - Permanently fetch these packages (other than borgbackup) with `pip3 download msgpack==1.0.7 packaging==23.2` and put them in `/boot/python_wheels/` for offline installation at boot time - Copy the `borgbackup` wheel file from the directory stated in the build logs `cp /root/.cache/pip/wheels/b0/c1/b2/bd1f9969742c3b690d74ae13233287eec544c5a9135497443e/pyfuse3-3.3.0-cp312-cp312-linux_x86_64.whl /boot/python_wheels/` - This would have appeared to be something similar to `Created wheel for borgbackup: filename=borgbackup-1.2.7-cp312-cp312-linux_x86_64.whl size=6248226 sha256=830ec3fbcc5c74922f7e269e8378aa08052c8c9cec51f407c53166d989d62e7c Stored in directory: /root/.cache/pip/wheels/64/90/02/c60bc19558d5e1e7c5ed13041bc27d25d41518ddfdb2def852` - Note that renaming needs to follow some strict guidelines https://peps.python.org/pep-0491/#file-name-convention otherwise Pip will error out - Add a line to the `/boot/config/go` file to automatically install `pyfuse3` at boot, or just run it adhoc to confirm it works: `/usr/bin/pip3 install /boot/python_wheels/borgbackup* --no-index --find-links file:///boot/python_wheels` - The final list of wheel files ready to install at boot time are: ``` attrs-23.1.0-py3-none-any.whl borgbackup-1.2.7-cp312-cp312-linux_x86_64.whl idna-3.6-py3-none-any.whl msgpack-1.0.7-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl outcome-1.3.0.post0-py2.py3-none-any.whl packaging-23.2-py3-none-any.whl pyfuse3-3.3.0-cp312-cp312-linux_x86_64.whl sniffio-1.3.0-py3-none-any.whl sortedcontainers-2.4.0-py2.py3-none-any.whl trio-0.23.1-py3-none-any.whl ``` pyfuse3-3.3.0-cp312-cp312-linux_x86_64.whl python3-3.12.1-x86_64-1-kubed20231210.txz borgbackup-1.2.7-cp312-cp312-linux_x86_64.whl Quote Link to comment
DjangoDurano Posted January 31 Share Posted January 31 Hello. I like your python builds for unraid, and I founded it at the right time. So I had a question, is it possible that you provide your build packages? As you mentioned it is necessary to have the correct packages to build, even not after build it will not run. I tried to build some own python package, but the main problem for me is, to find older slackware packages, like openssl 3.1 or glibc 2.37 and so on. If you know where I can find this older packages it's also fine. Best regards Quote Link to comment
kubed_zero Posted January 31 Author Share Posted January 31 14 minutes ago, DjangoDurano said: If you know where I can find this older packages it's also fine I think I was able to find everything I needed on https://slackware.uk/slackware/ but some general googling allowed me to find the older versions. I've also uploaded some of the older packages as part of this post: glibc-2.37-x86_64-2.txzopenssl-3.1.1-x86_64-1.txz Let me know if you need anything else! 1 Quote Link to comment
DjangoDurano Posted February 1 Share Posted February 1 22 hours ago, kubed_zero said: I think I was able to find everything I needed on https://slackware.uk/slackware/ but some general googling allowed me to find the older versions. I've also uploaded some of the older packages as part of this post: glibc-2.37-x86_64-2.txzopenssl-3.1.1-x86_64-1.txz Let me know if you need anything else! Thanks for your help. I google for a long time but it's really hard to find older slackware packages. Older "stable" versions are can be found for example glibc 2.33 but other versions not. I know https://slackware.uk/slackware/ but there are also only newer versions. The only thing is, that there are slackBuild scripts, maybe it is possible to build a previous version with that scripts. Quote Link to comment
kubed_zero Posted February 1 Author Share Posted February 1 1 hour ago, DjangoDurano said: Thanks for your help. I google for a long time but it's really hard to find older slackware packages. Older "stable" versions are can be found for example glibc 2.33 but other versions not. I know https://slackware.uk/slackware/ but there are also only newer versions. The only thing is, that there are slackBuild scripts, maybe it is possible to build a previous version with that scripts. https://slackware.uk/search?p=%2F&q=glibc-2.37 https://slackware.uk/cumulative/slackware64-current/slackware64/l/glibc-2.37-x86_64-1.txz https://slackware.uk/cumulative/slackware64-current/slackware64/l/glibc-2.37-x86_64-2.txz https://slackware.uk/cumulative/slackware64-current/slackware64/l/glibc-2.37-x86_64-3.txz Behold! With some digging through that website, old versions can be found Quote Link to comment
DjangoDurano Posted February 2 Share Posted February 2 23 hours ago, kubed_zero said: https://slackware.uk/search?p=%2F&q=glibc-2.37 https://slackware.uk/cumulative/slackware64-current/slackware64/l/glibc-2.37-x86_64-1.txz https://slackware.uk/cumulative/slackware64-current/slackware64/l/glibc-2.37-x86_64-2.txz https://slackware.uk/cumulative/slackware64-current/slackware64/l/glibc-2.37-x86_64-3.txz Behold! With some digging through that website, old versions can be found Oh. Didn't see the search field. Thanks for notice me. Is there a problem, if I add your python packages as dependency to my python sleep script at github? https://github.com/DjangoDurano/Unraid_Python_Sleep_Script Quote Link to comment
kubed_zero Posted February 2 Author Share Posted February 2 36 minutes ago, DjangoDurano said: Is there a problem, if I add your python packages as dependency to my python sleep script at github? Thanks for checking! No problem, feel free to use it. 1 Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.