bmartino1 Posted February 17 Share Posted February 17 (edited) Not sure if this is the correct space to ask. I'm receiving error 127 with unraid in a python venv. But not in an Ubuntu vm... *sure I can just run the vm but trying to cut resource overhead and want a bit more functionally from just running on unraid... For context and setup to reproduce as I'm more than willing to share my venv / how I run users scripting and pythons code. My goal is to get a python script on unraid to pull data for ameren power smart pricing. There website: https://www.ameren.com/illinois/account/customer-service/bill/power-smart-pricing/prices This website must render its JavaScript to grab the table below. This data changes daily. I later plan to have it export the table to a messag.txt and have something else email / sent it to locations I want to use it for...etc.. per other users for home automation and other use of this data: https://community.home-assistant.io/t/ameren-illinois-psp-price-tracking/505671/6 https://github.com/danieldotnl/ha-multiscrape I have dlandons apps for python 2 (for some other community apps and python 3 to fix unraid / nerdtools python3: I also have nerd tools install to install python 3: I run a user script at startup of array: a python 3.10 venv environment with the requirements for this python environment: root@ubuntu-pythongpull:/amerenweb# cat requirements.txt appdirs==1.4.4 attrs==23.2.0 beautifulsoup4==4.12.3 behave==1.2.6 certifi==2024.2.2 chardet==5.2.0 charset-normalizer==3.3.2 colorama==0.4.6 cssselect==1.2.0 exceptiongroup==1.2.0 execnet==2.0.2 fasteners==0.19 filelock==3.13.1 h11==0.14.0 idna==3.6 importlib-metadata==7.0.1 iniconfig==2.0.0 markdown-it-py==3.0.0 mdurl==0.1.2 outcome==1.3.0.post0 packaging==23.2 parameterized==0.9.0 parse==1.20.1 parse-type==0.6.2 pdbp==1.5.0 platformdirs==4.2.0 pluggy==1.4.0 py==1.11.0 pyee==8.2.2 Pygments==2.17.2 pynose==1.4.8 pyotp==2.9.0 pyppeteer==1.0.2 PySocks==1.7.1 pytest==8.0.0 pytest-html==2.0.1 pytest-metadata==3.1.1 pytest-ordering==0.6 pytest-rerunfailures==13.0 pytest-xdist==3.5.0 python-dotenv==1.0.1 PyVirtualDisplay==3.0 PyYAML==6.0.1 requests==2.31.0 rich==13.7.0 sbvirtualdisplay==1.3.0 selenium==4.17.2 seleniumbase==4.23.4 six==1.16.0 sniffio==1.3.0 sortedcontainers==2.4.0 soupsieve==2.5 tabcompleter==1.3.0 tomli==2.0.1 tqdm==4.66.2 trio==0.24.0 trio-websocket==0.11.1 typing_extensions==4.9.0 urllib3==1.26.18 webdriver-manager==4.0.1 websockets==10.4 wsproto==1.2.0 zipp==3.17.0 I have a venv setup missing pip installs on disk array startup: #!/bin/bash sleep 30s # Waits 30 seconds after array starts cd /mnt/user/DS-9/PythonScripts/amerenweb/venv/bin source activate python3 -m pip install -r requirements.txt #python3 -m pip install mysql-connector-python flask pytz gunicorn requests beautifulsoup4 psycopg2-binary egauge-python egauge-async influxdb grafana-api influxdb-client deactivate with this, I can now run a script with cron on unraid to use my python VENV (I have ran other python scripts in this configuration before.) more for pi temp login to influx docker/grafana and other home lab stuff. I run this script in another user script to run a python command example: #!/bin/bash cd /mnt/user/DS-9/PythonScripts/amerenweb/venv/bin source activate python3 pythonscript name.py deactivate ^This is the script I cron to run my python scripts on unraid at intervals throughout the day or manually when needed. Now then in an ubuntu VM as I'm a kat kitty and have some knowledge of scripting and getting solutions. From what I can tell, unraids error 127 for this is related to missing lib files from waht I can tell. the python script I run to achieve this is: root@ubuntu-pythongpull:/amerenweb# cat run_ameren.py import os from selenium import webdriver from selenium.webdriver.common.by import By from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.support import expected_conditions as EC def scrape_hourly_prices(url): # Configure Chrome options for headless mode chrome_options = webdriver.ChromeOptions() chrome_options.add_argument('--headless') chrome_options.add_argument('--no-sandbox') chrome_options.add_argument('--disable-dev-shm-usage') # Initialize Chrome WebDriver instance driver = webdriver.Chrome(options=chrome_options) try: # Navigate to the URL driver.get(url) # Wait for the table to be visible WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.CSS _SELECTOR, 'table.table'))) # Find the table table = driver.find_element(By.CSS_SELECTOR, 'table.table') # Extract data from the table rows = table.find_elements(By.TAG_NAME, 'tr') for row in rows: cells = row.find_elements(By.TAG_NAME, 'td') if cells: hour = cells[0].text price = cells[1].text print(f"Hour: {hour}, Price: {price}") except Exception as e: print(f"Error: {e}") finally: # Close the browser driver.quit() # Specify the URL to scrape url_to_scrape = 'https://www.ameren.com/illinois/account/customer-service/bill/p ower-smart-pricing/prices' # Call the function to scrape hourly prices scrape_hourly_prices(url_to_scrape) root@ubuntu-pythongpull:/amerenweb# in the vm when I run this I get a good output: root@ubuntu-pythongpull:/amerenweb# source venv/bin/activate (venv) root@ubuntu-pythongpull:/amerenweb# python run_ameren.py Hour: 12 AM, Price: 2.0¢ Hour: 1 AM, Price: 2.0¢ Hour: 2 AM, Price: 2.1¢ Hour: 3 AM, Price: 2.1¢ Hour: 4 AM, Price: 2.2¢ Hour: 5 AM, Price: 2.6¢ Hour: 6 AM, Price: 2.8¢ Hour: 7 AM, Price: 2.7¢ Hour: 8 AM, Price: 2.7¢ Hour: 9 AM, Price: 2.6¢ Hour: 10 AM, Price: 2.4¢ Hour: 11 AM, Price: 2.3¢ Hour: 12 PM, Price: 2.4¢ Hour: 1 PM, Price: 2.2¢ Hour: 2 PM, Price: 2.2¢ Hour: 3 PM, Price: 2.2¢ Hour: 4 PM, Price: 2.7¢ Hour: 5 PM, Price: 2.9¢ Hour: 6 PM, Price: 2.7¢ Hour: 7 PM, Price: 2.4¢ Hour: 8 PM, Price: 2.3¢ Hour: 9 PM, Price: 2.2¢ Hour: 10 PM, Price: 2.1¢ Hour: 11 PM, Price: 0.0¢ (venv) root@ubuntu-pythongpull:/amerenweb# However, on unriad same venv, same script I get: Thoughts to get this python script running on unraid host and not in the vm? Edited February 17 by bmartino1 Fix weblinks to homeautomation info Quote Link to comment
bmartino1 Posted February 17 Author Share Posted February 17 Sad panda getting a new error on a python script that worked before. Seems like unraid is missing or changed quite a few libs that I would otherwise see and have acccess to in other environments... Missing libs I would require atm - so far... debain package libs i need atm that are missing form UNRAID...: libffi-dev libnss3 libnspr4 (venv) root@SGC:/mnt/user/DS-9/PythonScripts/discord/venv/bin# python3 botsend.py Traceback (most recent call last): File "/mnt/user/DS-9/PythonScripts/discord/venv/bin/botsend.py", line 1, in <module> import discord File "/mnt/user/DS-9/PythonScripts/discord/venv/lib64/python3.9/site-packages/discord/__init__.py", line 23, in <module> from .client import * File "/mnt/user/DS-9/PythonScripts/discord/venv/lib64/python3.9/site-packages/discord/client.py", line 51, in <module> from .user import User, ClientUser File "/mnt/user/DS-9/PythonScripts/discord/venv/lib64/python3.9/site-packages/discord/user.py", line 29, in <module> import discord.abc File "/mnt/user/DS-9/PythonScripts/discord/venv/lib64/python3.9/site-packages/discord/abc.py", line 59, in <module> from .voice_client import VoiceClient, VoiceProtocol File "/mnt/user/DS-9/PythonScripts/discord/venv/lib64/python3.9/site-packages/discord/voice_client.py", line 49, in <module> from . import opus, utils File "/mnt/user/DS-9/PythonScripts/discord/venv/lib64/python3.9/site-packages/discord/opus.py", line 30, in <module> import ctypes File "/usr/lib64/python3.9/ctypes/__init__.py", line 8, in <module> from _ctypes import Union, Structure, Array ImportError: libffi.so.7: cannot open shared object file: No such file or directory (venv) root@SGC:/mnt/user/DS-9/PythonScripts/discord/venv/bin# ########### Again, works fine in the vm ... would like to have it run native on host again... sadly I had this working on unraid two version ago... Now I'm getting a lib error.... This script last ran on unraid version 6.12.4 similar setup with script issues above... ?I guess Unraid / Nerd pack tools are moving away from libs or recreating for a more minimal approach to their Linux environment it runs??? Quote Link to comment
primeval_god Posted February 18 Share Posted February 18 (edited) Trying to run python scripts directly on unRAID is just asking for a headache. Save yourself some trouble and just run your scripts in a docker container or an LXC container (LXC requires a plugin). For docker there are several ways to handle it, you could build your whole script into a container, or your could create a very simple container from one of the python base containers that has only your venv in it and keep the script external. You could run it from a script in the user scripts plugin on a schedule using a docker run command to launch the container, pass through your python script directory and run your python script. Edited February 18 by primeval_god Quote Link to comment
bmartino1 Posted February 18 Author Share Posted February 18 (edited) I tried python dockers before with no success. And I'm beyond lost on making my own docker. I have many ideas, but nothing that forms or helps me build them. I saw the lxc plugin in the app page, but I have no experience with LXC setup. I would need a resource page / man page or some more info on setting up a lxc environment. Last I heard that term was back with Ubuntu 6 with LVM and while I didn't set it up a little interaction with LXC as an experience was more with separating multiple environments. Like having LXC running separate services(samba), Apache, SSH, etc... but that was more then 15 years ago. I can see a LXC working. but would need a bit more guidance on setting up and configuring one. My experience was interacting at that time, not its configuration and setup. ATM I have a temp proxmox system Ubuntu mate VM running this script. I have a script to get today date(posted above) and tomorrow(finshed earlier today in testing) to grab the table form psp from ameren. I think the LXC container would act the same with a nfs / folder share to host with a ?full layer distro to have a package manager in a folder run an environment... I tried to use containers in proxmox in the past without success for this and similar but couldn't get past errors and access restrictions. Issue without end, where running a VM was easier and less resource intensive. I appreciate your response and insight in this matter. Any Guidance to setup a lxc. Edited February 18 by bmartino1 Quote Link to comment
primeval_god Posted February 18 Share Posted February 18 10 hours ago, bmartino1 said: ATM I have a temp proxmox system Ubuntu mate VM running this script. I have a script to get today date(posted above) and tomorrow(finshed earlier today in testing) to grab the table form psp from ameren. I think the LXC container would act the same with a nfs / folder share to host with a ?full layer distro to have a package manager in a folder run an environment... Yeah LXC is a bit like running a VM without the overhead. You can get a full featured linux distro within the container much like a vm. There is no need for an nfs share to get access to files on the host system. LXC containers allow you to pass folders from the host system in a similar manner to docker. As for the specific error you posted, it would be best to ask about that in the LXC plugin support page. I am not an expert in LXC, I have never had such an error setting up container on unraid. Regarding python in docker, it can actually be quite simple to setup if you are only using the container to hold the environment. A dockerfile, based on one of the official python images, that installs your env file should be pretty simple to write. Build the image locally then you can use it to run scripts from the unRAID command line or from a user script with a command something like this docker run --rm -it -v /host/scripts:/container/scripts -w /container/scripts local/python:3.10 python myscript.py -some-arg which launches a container, bind mounts the scripts directory from the host into the container, sets the container working directory to the scripts directory and then runs the command python myscript.py -some-arg 1 Quote Link to comment
bmartino1 Posted February 18 Author Share Posted February 18 As I'm unable to use lxc atm. if you can point me toward a docker solution. Docker python would need to be able to set up missing libs and pip installs vai the requirements from a venv pip freeze command. I can use a script to then start the docker. To then run the python script. the script I have don't have arguments or options, they just run and done. I'm not aware of a docker where I can pass my generated VENV and have the docker python open my venv and run similar to how I script to call and open/activate the source to then call my python environment. as I would need its repository and docker run / compose info to make a solution to happen. Quote Link to comment
primeval_god Posted February 18 Share Posted February 18 (edited) Create a Dockerfile something like this (basically copied off of the dockerhub page for the official python container) FROM python:3.10 COPY requirements.txt ./ RUN pip install --no-cache-dir -r requirements.txt WORKDIR /scripts #Uncomment below if you want to include your script inside the container #COPY . . #CMD ["python", "./yourscript.py"] put you requirements file in the same directory. If you want to bundle your script into the container put that in the same directory as well and uncomment the bottom 2 lines in the docker file. Build a local image with this command (from the same directory) docker build -t local/mypythonimage:latest . when finished you can run the script in one of the following ways. If you included your script in the images just run the following form anywhere on the system docker run --rm -it local/mypythonimage:latest If you did not include the script in the image run docker run --rm -it -v /mnt/user/path/to/your/script/dir:/scripts local/mypythonimage:latest python yourscript.py If your script needs access to other files on your system you will need to bind mount the directories into the system with additional -v flags. Edited February 18 by primeval_god 1 Quote Link to comment
bmartino1 Posted February 19 Author Share Posted February 19 Thank you! primeval_god this has helped alot! I have LXC capable now and have learned quite a bit from you with using python and docker. I will try LXC first having dealt with a similar instance then try a docker instances Quote Link to comment
bmartino1 Posted February 19 Author Share Posted February 19 (edited) For prosperity of a python config for PSP ameren web scraping: As I need to clean up my Python venv will most likely start over the VENV to grab only what I need. Output of the 2 python scripts: I also am using a discord bot to send the 2 txt files: Here are my finished scripts: I run a cron job daily at 6 PM: using nano /etc/crontab and have this working by running the cron and python scripts as root only... #!/bin/bash #connect to nfs share on unraid cd /unraid/venv/bin/ echo "Cron Script Running... opening python venv to run multiple python scripts" # Activate the Python virtual environment source /unraid/venv/bin/activate # Run the Python script to export current day for bot message python /unraid/venv/bin/todays-ameren-psp.py sleep 3 #Run the Python Script to export next day for bot message python /unraid/venv/bin/tomorrows-ameren-psp.py echo "Website gather data Scripts completed and scrape sessions closed" sleep 5 #Run same script to botmaker with bot api token and servers channel ID python /unraid/venv/bin/bot_logan.py echo "Script completed and message send to discord" # Deactivate the Python virtual environment deactivate echo "Script completed successfully and Ameren PSP captured and notified" #cleanup data folder #rm /unraid/data/*.* This is the script that activates my 2 script and send them off to be noticed will alter setup home automation for dashboard and other at this time discord seemed easier to set up as a delivery method for a python only environment... Script 1 this pulls the Ameren Data for current day: import os from selenium import webdriver from selenium.webdriver.common.by import By from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.support import expected_conditions as EC from datetime import datetime def scrape_hourly_prices(url, output_directory): # Configure Chrome options for headless mode chrome_options = webdriver.ChromeOptions() chrome_options.add_argument('--headless') chrome_options.add_argument('--no-sandbox') chrome_options.add_argument('--disable-dev-shm-usage') # Initialize Chrome WebDriver instance driver = webdriver.Chrome(options=chrome_options) try: # Navigate to the URL driver.get(url) # Wait for the table to be visible WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.CSS_SELECTOR, 'table.table'))) # Find the table table = driver.find_element(By.CSS_SELECTOR, 'table.table') # Extract data from the table rows = table.find_elements(By.TAG_NAME, 'tr') # Get today's date in the desired format today_date = datetime.now().strftime('%B %d, %Y') # Construct the output file path output_file_path = os.path.join(output_directory, f"todaymessagefor_{today_date}.txt") # Save the extracted data to the output file with open(output_file_path, 'w') as f: f.write(f"Here is Ameren PSP for {today_date}:\n") for row in rows: cells = row.find_elements(By.TAG_NAME, 'td') if cells: hour = cells[0].text price = cells[1].text f.write(f"Hour: {hour}, Price: {price}\n") print(f"Output saved to {output_file_path}") except Exception as e: print(f"Error: {e}") finally: # Close the browser driver.quit() # Specify the URL to scrape url_to_scrape = 'https://www.ameren.com/illinois/account/customer-service/bill/power-smart-pricing/prices' # Specify the output directory output_directory = '/unraid/data' # Call the function to scrape hourly prices and save to the output file scrape_hourly_prices(url_to_scrape, output_directory) Script 2 this pulls the ameren data for next day / tomorrow: import os import time from datetime import datetime, timedelta from selenium import webdriver from selenium.webdriver.common.by import By from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.support import expected_conditions as EC def scrape_tomorrow_prices(url, output_directory): # Configure Chrome options for headless mode chrome_options = webdriver.ChromeOptions() chrome_options.add_argument('--headless') chrome_options.add_argument('--no-sandbox') chrome_options.add_argument('--disable-dev-shm-usage') # Initialize Chrome WebDriver instance driver = webdriver.Chrome(options=chrome_options) try: # Navigate to the URL driver.get(url) # Wait for page to load completely driver.implicitly_wait(10) # Click on the "Tomorrow" button tomorrow_button = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.ID, "rtp-tomorrow"))) tomorrow_button.click() # Wait for 3 seconds for the JavaScript to load and the table to render time.sleep(3) # Extract data from the table table = driver.find_element(By.XPATH, "//table[@aria-label='Hourly Prices']") rows = table.find_elements(By.TAG_NAME, 'tr')[1:] # Exclude the header row # Calculate tomorrow's date tomorrow_date = datetime.now() + timedelta(days=1) tomorrow_date_str = tomorrow_date.strftime('%B %d, %Y') # Construct the output file path output_file_path = os.path.join(output_directory, f"tomorrowmessagefor_{tomorrow_date_str}.txt") # Save the extracted data to the output file with open(output_file_path, 'w') as f: f.write(f"PSP for Tomorrow ({tomorrow_date_str}):\n") for row in rows: cells = row.find_elements(By.TAG_NAME, 'td') hour = cells[0].text price = cells[1].text f.write(f"Hour: {hour}, Price: {price}\n") print(f"Output saved to {output_file_path}") except Exception as e: print(f"Error: {e}") finally: # Close the browser driver.quit() # Specify the URL to scrape url_to_scrape = 'https://www.ameren.com/illinois/account/customer-service/bill/power-smart-pricing/prices' # Specify the output directory output_directory = '/unraid/data' # Call the function to scrape tomorrow's prices and save to the output file scrape_tomorrow_prices(url_to_scrape, output_directory) Script 3 send and deliver the 2 files created to notify me / use with other things... ?maybe later send to a database... I want physical files atm for ease of data access testing and file manipulation... Discord BOT - You need your account set to developer mode. add a bot created at the development page, add that bot to your server, and reset api token to copy the token for use in this script. Right mouse click the "text" channel to copy channel id for this script. This bot sits offline until cron runs to connect and write to the channel. import discord import datetime import os TOKEN = 'grab from dicord development of your bot. api Token' CHANNEL_ID = ReplaceMEWITHServerChannelIDNumber URL = 'https://www.ameren.com/illinois/account/customer-service/bill/power-smart-pricing/prices' DATA_FOLDER = '/unraid/data' # Get current date and format it as Month Day, Year current_date = datetime.datetime.now().strftime('%B %d, %Y') next_date = (datetime.datetime.now() + datetime.timedelta(days=1)).strftime('%B %d, %Y') TODAY_FILE_NAME = f'todaymessagefor_{current_date}.txt' TOMORROW_FILE_NAME = f'tomorrowmessagefor_{next_date}.txt' TODAY_FILE_PATH = os.path.join(DATA_FOLDER, TODAY_FILE_NAME) TOMORROW_FILE_PATH = os.path.join(DATA_FOLDER, TOMORROW_FILE_NAME) intents = discord.Intents.default() intents.messages = True client = discord.Client(intents=intents) @client.event async def on_ready(): print(f'Logged in as {client.user.name} ({client.user.id})') print('------') channel = client.get_channel(CHANNEL_ID) if channel: try: # Read the contents of today's file if os.path.exists(TODAY_FILE_PATH): with open(TODAY_FILE_PATH, 'r') as file: file_contents = file.read() else: file_contents = f'Error: No PSP data available for today ({current_date}). Please check and fix scripts.' # Read the contents of tomorrow's file if os.path.exists(TOMORROW_FILE_PATH): with open(TOMORROW_FILE_PATH, 'r') as file: tomorrow_file_contents = file.read() else: tomorrow_file_contents = f'Error: No PSP data available for tomorrow ({next_date}). PSP data is released at 5:30 PM.' # Compose the message message_content = f'Hello, this is your bot! Check the website: {URL}\n\n{file_contents}\n{tomorrow_file_contents}' # Send the message await channel.send(message_content) print('Message sent successfully!') except Exception as e: print(f'Error: {str(e)}') else: print(f'Channel with ID {CHANNEL_ID} not found.') await client.close() client.run(TOKEN) As you can see in the script, the python environment needs some dependency. I'm using this command to create my venv and installing the python packages to run them pip install discord.py beautifulsoup4 requests selenium webdriver-manager Now to test in a Ubuntu jammy lxc and then a docker to see which is less resource intensive. As I can cron a start LXC container or docker at 6 PM and run these python scripts. Then leaving an Ubuntu VM running 24/7 for a cron and script to run... Edited March 2 by bmartino1 Quote Link to comment
Solution bmartino1 Posted February 21 Author Solution Share Posted February 21 (edited) On 2/17/2024 at 11:02 PM, primeval_god said: Trying to run python scripts directly on unRAID is just asking for a headache. Save yourself some trouble and just run your scripts in a docker container or an LXC container (LXC requires a plugin). For docker there are several ways to handle it, you could build your whole script into a container, or your could create a very simple container from one of the python base containers that has only your venv in it and keep the script external. You could run it from a script in the user scripts plugin on a schedule using a docker run command to launch the container, pass through your python script directory and run your python script. Thank you again primeval_god and ich777 !!! I was unable to get lxc to run at unraid terminal for userscripts plugin for container start stop cron.... I was unable to use lxc config to mount a unraid host folder to the container... I was able to fstab mount the drive path via nfs / samba.... LXC is so much less intensive then running the VM... Here is a LXC solutions: Install LXC app: Be sure to visit the support form for LXC. set up your lxc path options. I chose to put them under the appdata where docker data is stored. run to fix an issue with unraid /var and lxc path Thanks LXC support team and ICH777 for their assistance: Go to setting LXC set you options: run script (4th pin) to be able to use lxc on unraid: *OPTIONAL! - This May or May not be required! LXC_PATH=$(cat /boot/config/plugins/lxc/lxc.conf | grep "lxc.lxcpath=" | cut -d '=' -f2 | head -1) rm -rf /var/cache/lxc ${LXC_PATH}/cache mkdir -p ${LXC_PATH}/cache ln -s ${LXC_PATH}/cache /var/cache/lxc now go to LXC tab similar to VM and add a container: In my setup for this LXC container, I chose Ubuntu (jammy template) as I already had a similar VM made to be mimicked in this container. Next was to start and open the terminal to run and install some things... Be sure to set up your timezone (fix date and time) by running dpkg-reconfigure tzdata with lxc, you are root with no users setup in this instance. as away apt-get update and upgrade before install packages. (should be done with latest template container...) and set a root password: passwd Install needed packages. We will need x11 libs and files following similar container: https://gist.github.com/Roadmaster/0de007826485d0e5a9c856171a9a1e9c Recommend packages to install apt-get install mc nano vim python3 python3-pip openssh-server x11-apps mesa-utils alsa-utils nfs-common cifs-utils libffi-dev libnss3 libnspr4 We will be leveraging chrome driver with my python script above following the lxc Google Chrome instance we will also need to install chrome stable via dpkg: wget -c https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb dpkg --install google-chrome-stable_current_amd64.deb || true apt install -fy We now have a working chrome / chrome driver instance to call with selenium / web manger in python... Lets add our host to container path... [I was unable to get this to work lets connect ot a samba share / nfs share instead...] https://gist.github.com/julianlam/07abef272136ea14a627 #lxc.mount.entry = /mnt/user/Program-Installers-PCRepair/Other_OS/PythonScripts/python /unraid none bind 0 0 #lxc config device add python sharename disk path=/mnt/user/Program-Installers-PCRepair/Other_OS/PythonScripts/python source="/unraid" It is recommended to use NFS for file sharing for linux to linux... https://www.ibm.com/docs/en/zos/2.3.0?topic=client-displaying-server-mount-informationshowmount lets see what the container sees for nfs: (192.168.1.254 is my unriad a share can be seen if nfs is enabled and share is set to public) showmount -e 192.168.1.254 lets make a path mkdir /unraid edit fstab nano /etc/fstab Example: # UNCONFIGURED FSTAB FOR BASE SYSTEM #NFS Shares: 192.168.1.254:/mnt/user/python/lxcpython /unraid nfs defaults 0 0 user your Unraids IP for 192.168.1.254 : /path/to folder/passed for nfs now we can see the nfs at a container restart and set up our python environment: python3 -m venv nameofenviroment cd /nameofenviroment/bin source activate pip install the need commands "discory.py beautifulsoup4 requests selenium webdriver-manager" my venv requirement to run root@python:/unraid/amerenpsp/bin# cat requirments.txt aiohttp==3.9.3 aiosignal==1.3.1 async-timeout==4.0.3 attrs==23.2.0 beautifulsoup4==4.12.3 certifi==2024.2.2 charset-normalizer==3.3.2 discord.py==2.3.2 exceptiongroup==1.2.0 frozenlist==1.4.1 h11==0.14.0 idna==3.6 multidict==6.0.5 outcome==1.3.0.post0 packaging==23.2 PySocks==1.7.1 python-dotenv==1.0.1 requests==2.31.0 selenium==4.18.1 sniffio==1.3.0 sortedcontainers==2.4.0 soupsieve==2.5 trio==0.24.0 trio-websocket==0.11.1 typing_extensions==4.9.0 urllib3==2.2.1 webdriver-manager==4.0.1 wsproto==1.2.0 yarl==1.9.4 make a python script: import os import time from datetime import datetime, timedelta from selenium import webdriver from selenium.webdriver.common.by import By from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.support import expected_conditions as EC import discord TOKEN = 'Discord bot Token API added to your server' CHANNEL_ID = DisocrdChannel ID URL = 'https://www.ameren.com/illinois/account/customer-service/bill/power-smart-pricing/prices' DATA_FOLDER = '/unraid/amerenpsp/bin/data/' #I recommend putting the data path in the bin folder def scrape_hourly_prices(url, output_directory): output_file_path = os.path.join(output_directory, f"todaymessagefor_{datetime.now().strftime('%B %d, %Y')}.txt") chrome_options = webdriver.ChromeOptions() chrome_options.add_argument('--headless') chrome_options.add_argument('--no-sandbox') chrome_options.add_argument('--disable-dev-shm-usage') driver = webdriver.Chrome(options=chrome_options) try: driver.get(url) WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.CSS_SELECTOR, 'table.table'))) table = driver.find_element(By.CSS_SELECTOR, 'table.table') rows = table.find_elements(By.TAG_NAME, 'tr') with open(output_file_path, 'w') as f: f.write(f"Here is Ameren PSP for {datetime.now().strftime('%B %d, %Y')}:\n") for row in rows: cells = row.find_elements(By.TAG_NAME, 'td') if cells: hour = cells[0].text price = cells[1].text f.write(f"Hour: {hour}, Price: {price}\n") print(f"Output saved to {output_file_path}") except Exception as e: print(f"Error: {e}") finally: driver.quit() return output_file_path def scrape_tomorrow_prices(url, output_directory): output_file_path = os.path.join(output_directory, f"tomorrowmessagefor_{(datetime.now() + timedelta(days=1)).strftime('%B %d, %Y')}.txt") chrome_options = webdriver.ChromeOptions() chrome_options.add_argument('--headless') chrome_options.add_argument('--no-sandbox') chrome_options.add_argument('--disable-dev-shm-usage') driver = webdriver.Chrome(options=chrome_options) try: driver.get(url) driver.implicitly_wait(10) tomorrow_button = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.ID, "rtp-tomorrow"))) tomorrow_button.click() time.sleep(10) # Check if the table is available table_present = driver.find_elements(By.XPATH, "//table[@aria-label='Hourly Prices']") if not table_present: raise Exception("No table found for tomorrow's prices") table = driver.find_element(By.XPATH, "//table[@aria-label='Hourly Prices']") rows = table.find_elements(By.TAG_NAME, 'tr')[1:] # Exclude the header row with open(output_file_path, 'w') as f: f.write(f"PSP for Tomorrow ({(datetime.now() + timedelta(days=1)).strftime('%B %d, %Y')}):\n") for row in rows: cells = row.find_elements(By.TAG_NAME, 'td') hour = cells[0].text price = cells[1].text f.write(f"Hour: {hour}, Price: {price}\n") print(f"Output saved to {output_file_path}") except Exception as e: print(f"Error: {e}") finally: driver.quit() return output_file_path def send_discord_message(token, channel_id, url, today_file_path, tomorrow_file_path): intents = discord.Intents.default() intents.messages = True client = discord.Client(intents=intents) @client.event async def on_ready(): print('Logged in as') print(client.user.name) print(client.user.id) print('------') channel = client.get_channel(channel_id) if channel: try: if os.path.exists(today_file_path): with open(today_file_path, 'r') as file: file_contents = file.read() else: file_contents = f'Error: No PSP data available for today ({datetime.now().strftime("%B %d, %Y")}). Please check and fix scripts.' if os.path.exists(tomorrow_file_path): with open(tomorrow_file_path, 'r') as file: tomorrow_file_contents = file.read() else: tomorrow_file_contents = f'Error: No PSP data available for tomorrow ({(datetime.now() + timedelta(days=1)).strftime("%B %d, %Y")}). PSP data is released at 5:30 PM.' message_content = f'Hello, this is your bot! Check the website: {url}\n\n{file_contents}\n{tomorrow_file_contents}' await channel.send(message_content) print('Message sent successfully!') except Exception as e: print(f'Error: {str(e)}') else: print(f'Channel with ID {channel_id} not found.') await client.close() client.run(token) # Specify the URL to scrape url_to_scrape = URL # Call the function to scrape hourly prices and save to the output file today_file_path = scrape_hourly_prices(url_to_scrape, DATA_FOLDER) # Pause before executing the next script time.sleep(10) # Call the function to scrape tomorrow's prices and save to the output file try: tomorrow_file_path = scrape_tomorrow_prices(url_to_scrape, DATA_FOLDER) except Exception as e: print(f"Error occurred during scraping tomorrow's prices: {e}") tomorrow_file_path = None # Call the function to send a Discord message with the scraped data send_discord_message(TOKEN, CHANNEL_ID, URL, today_file_path, tomorrow_file_path) #Uncomment if you don't want to keep the 2 text files made when scraping... # Clean up files #for file in os.listdir(DATA_FOLDER): # file_path = os.path.join(DATA_FOLDER, file) # try: # if os.path.isfile(file_path): # os.unlink(file_path) # except Exception as e: # print(f"Error: {e}") all in 1 to get today's, tomorrows and send to discord... next cron. LXC is quite less resource intensive. Lets make a cron scrip file: #!/bin/bash #connect to nfs share on unraid cd /unraid/amerenpsp/bin echo "Cron Script Running... opening python venv to run multiple python scripts" # Activate the Python virtual environment source activate # Run the Python scrip python runme_scrapeandsend.py deactivate echo "Script completed successfully and Ameren PSP captured and notified" now we can leave the lxc running and let its cron run ?-later use unraids user script to start the lxc container and set the cron job to run at startup root@python:/unraid/amerenpsp/bin# nano /etc/crontab root@python:/unraid/amerenpsp/bin# cat /etc/crontab # /etc/crontab: system-wide crontab # Unlike any other crontab you don't have to run the `crontab' # command to install the new version when you edit this file # and files in /etc/cron.d. These files also have username fields, # that none of the other crontabs do. SHELL=/bin/sh # You can also override PATH, but by default, newer versions inherit it from the environment #PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin # Example of job definition: # .---------------- minute (0 - 59) # | .------------- hour (0 - 23) # | | .---------- day of month (1 - 31) # | | | .------- month (1 - 12) OR jan,feb,mar,apr ... # | | | | .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat # | | | | | # * * * * * user-name command to be executed 17 * * * * root cd / && run-parts --report /etc/cron.hourly 25 6 * * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) 47 6 * * 7 root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly ) 52 6 1 * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.monthly ) # 0 18 * * * root /unraid/amerenpsp/bin/cronrunpython.sh root@python:/unraid/amerenpsp/bin# here I have added a line to run my sh script at 6 PM daily. The root user will run it. This sh script will start our python venv and run the python script. lets not forget to start and enable the corn service, so the task will run as long as the lxc container is running. systemctl start cron.service systemctl enable cron.service In the lxc if you encounter an error in running the scritp you may need to fix file permission setting under unraid. Since I did a NFS share as normally you would need to chmod +x a sh scirpt to make it executable... #root@BMM-Unraid:/mnt/user/python/lxcpython# chmod -R 777 * #root@BMM-Unraid:/mnt/user/python/lxcpython# chown nobody:users -R * cd /mnt/user/python/lxcpython chmod -R 777 * chown nobody:users -R * To fix nfs file permission so the ubuntu lxc can access and execute the commands This appears to be the correct way to run a python script on Unraid. Testing a docker system next Edited February 22 by bmartino1 Quote Link to comment
ich777 Posted February 21 Share Posted February 21 11 minutes ago, bmartino1 said: run script (4th pin) to be able to use lxc on unraid: Please mark that as maybe necessary if users have issues setting up the first container since this happens not to all users. 12 minutes ago, bmartino1 said: now go to LXC tab similar to VM and add a container: I have a question about the MAC address here... why does it look like that? The MAC address is generated randomly and automatically if you add a container. 13 minutes ago, bmartino1 said: edit fstab You could also mount a path directly from Unraid if you want to like described here: (Please note that it's not typo that there is no starting / for the path in the container) 1 Quote Link to comment
bmartino1 Posted February 21 Author Share Posted February 21 Hi ich777. Updated in edit. For the optional pin 4 if error. I was trying to grba a quick picture and wasn't sure if it was random, did that manually. Good to know about symlinking. 1 Quote Link to comment
ich777 Posted February 21 Share Posted February 21 2 minutes ago, bmartino1 said: Good to know about symlinking. Did I link the wrong post, it should have described how to make a host path mount into the container, so to speak mount a path from Unraid directly into the container itself via the LXC config from the container. Quote Link to comment
bmartino1 Posted February 21 Author Share Posted February 21 On 5/31/2022 at 12:58 AM, ich777 said: Yes! If you want to share anything with the container you have to keep in mind that you maybe mess up your permissions on that share if you don't created the appropriate user and group and assign the user in the container to this user and group, so I would recommend that you set up a test share to do this. The steps would be: Start up the container and create the directory where the files should be mounted (in this example "/hostshare" with the command "mkdir -p /hostshare") Stop the container Create a share on Unraid (in this example we will use "lxcshare" with the path "/mnt/user/lxcshare") Set the permissions from this share to 777 with "chmod -R 777 /mnt/user/lxcshare" Open up your config file for the container (you'll find the path by clicking on the container name in the first line) Add this line: "lxc.mount.entry = /mnt/user/lxcshare hostshare none rw,bind 0.0" (without double quotes) and save the file Start the container again and navigate to /hostshare Depending on how you set up the container and users you maybe can skip a few steps or have to do other things to make it work, but the above is the most basic test scenario which will work with for almost any configuration. Quote Link to comment
ich777 Posted February 21 Share Posted February 21 @bmartino1 yes, these are the steps to mount a path from the hsot to the container, like you can do it for Docker with a bind mount. 1 Quote Link to comment
bmartino1 Posted March 2 Author Share Posted March 2 (edited) I tried the docker option, similar to running commands directly on the host of unraid. While docker can do basic python only takes and scripts, it's not the same environment from the VM/ LXC. Thanks again to ich777 and primeval_god for their work and contributions! Edited March 2 by bmartino1 Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.