RoninTech Posted March 20, 2019 Share Posted March 20, 2019 (edited) Hi all, I've had an Unraid cron python script (invtrack - investment tracker) running for about a year that scrapes some web info every day, puts it into a dockered Influxdb and makes it accessible via a dockered Grafana. All of this has been working great within Unraid. Recently the website I am scraping battened down the hatches to obfuscated JavaScript dynamic content. This meant I had to migrate my python parser to use Selenium that requires a browser installation. I found an existing python/selenium/chrome docker here: https://hub.docker.com/r/joyzoursky/python-chromedriver To get it working at first I simply created a /mnt/user/appdata/invtrack folder, put my python scraper script in it, then ran the docker run command they suggest (slightly modified for TZ etc.) from the new folder: docker run --name scraper -e TZ="America/Tijuana" -e HOST_OS="Unraid" -it -v $(pwd):/usr/workspace joyzoursky/python-chromedriver:3.7-selenium bash I can then run my Selenium updated python scraper and everything works as intended. That being said there are some obvious weaknesses to not having been added to the Unraid docker page correctly. The Unraid docker page entry for the scraper does not allow me to edit/update the docker. Hitting the "update ready" link for the docker doesn't work. My Python scraper has to be manually moved to /mnt/user/appdata/invtrack, it's not part of the docker image. I have to manually execute an init script when the docker starts to setup some python dependencies for my scraper. I have to setup a daily Unraid cron job to kick the scraper inside the docker. I enabled docker hub searches in Unraid's Community Apps and found and downloaded "python chromedriver". I then edited the docker parameters as shown in the attached image. When I attempt to start the docker with network set to host I get nothing in the logs. If I set it to bridge I see the following (2 start attempts) but the docker does not come up. Interestingly when I got this docker working outside of Community Apps by just doing the above docker run command, the Unraid docker page shows it as having a "default" network which isn't an option when I add it through Community apps.: Mar 19 22:04:09 unraid kernel: vethef16f86: renamed from eth0 Mar 19 22:04:09 unraid kernel: docker0: port 7(vetha506275) entered disabled state Mar 19 22:04:09 unraid kernel: docker0: port 7(vetha506275) entered disabled state Mar 19 22:04:09 unraid kernel: device vetha506275 left promiscuous mode Mar 19 22:04:09 unraid kernel: docker0: port 7(vetha506275) entered disabled state Mar 19 22:08:15 unraid kernel: docker0: port 7(veth5e15951) entered blocking state Mar 19 22:08:15 unraid kernel: docker0: port 7(veth5e15951) entered disabled state Mar 19 22:08:15 unraid kernel: device veth5e15951 entered promiscuous mode Mar 19 22:08:15 unraid kernel: IPv6: ADDRCONF(NETDEV_UP): veth5e15951: link is not ready Mar 19 22:08:17 unraid kernel: eth0: renamed from veth247fa63 Mar 19 22:08:17 unraid kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth5e15951: link becomes ready Mar 19 22:08:17 unraid kernel: docker0: port 7(veth5e15951) entered blocking state Mar 19 22:08:17 unraid kernel: docker0: port 7(veth5e15951) entered forwarding state Mar 19 22:08:18 unraid kernel: veth247fa63: renamed from eth0 Mar 19 22:08:18 unraid kernel: docker0: port 7(veth5e15951) entered disabled state Mar 19 22:08:18 unraid kernel: docker0: port 7(veth5e15951) entered disabled state Mar 19 22:08:18 unraid kernel: device veth5e15951 left promiscuous mode Mar 19 22:08:18 unraid kernel: docker0: port 7(veth5e15951) entered disabled state Mar 19 22:08:30 unraid kernel: docker0: port 7(veth7c2c127) entered blocking state Mar 19 22:08:30 unraid kernel: docker0: port 7(veth7c2c127) entered disabled state Mar 19 22:08:30 unraid kernel: device veth7c2c127 entered promiscuous mode Mar 19 22:08:30 unraid kernel: IPv6: ADDRCONF(NETDEV_UP): veth7c2c127: link is not ready Mar 19 22:08:30 unraid kernel: docker0: port 7(veth7c2c127) entered blocking state Mar 19 22:08:30 unraid kernel: docker0: port 7(veth7c2c127) entered forwarding state Mar 19 22:08:30 unraid kernel: docker0: port 7(veth7c2c127) entered disabled state Mar 19 22:08:33 unraid kernel: eth0: renamed from veth7af64bb Mar 19 22:08:33 unraid kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth7c2c127: link becomes ready Mar 19 22:08:33 unraid kernel: docker0: port 7(veth7c2c127) entered blocking state Mar 19 22:08:33 unraid kernel: docker0: port 7(veth7c2c127) entered forwarding state Mar 19 22:08:33 unraid kernel: veth7af64bb: renamed from eth0 Mar 19 22:08:33 unraid kernel: docker0: port 7(veth7c2c127) entered disabled state Mar 19 22:08:34 unraid kernel: docker0: port 7(veth7c2c127) entered disabled state Mar 19 22:08:34 unraid kernel: device veth7c2c127 left promiscuous mode Mar 19 22:08:34 unraid kernel: docker0: port 7(veth7c2c127) entered disabled state So I'm looking for some feedback on any potential errors I've made in adding the docker via Community Apps that are causing the docker to not start. Would also appreciate any tips on how to kick a script inside my docker once it is running, and how to run a setup script (to pip python modules) inside a docker on startup. Thanks, and apologies if this is in the wrong place or has wrong/missing info. I will correct them as soon as I am aware. 😃 Edited March 20, 2019 by RoninTech Added a point on network Quote Link to comment
Squid Posted March 20, 2019 Share Posted March 20, 2019 Your best course of action is to compare the docker run commands between when editing the container and what you used to run at the command line and then making adjustments accordingly Quote Link to comment
RoninTech Posted March 20, 2019 Author Share Posted March 20, 2019 Thanks squid. Turns out it was the "-it" options. Now the docker starts as expected. I also was able to hook it up to cron via the docker run command. My last issue I need to work out is how can I permanently add my python parser and init script into this other person's docker container? I see there is a "docker cp" but I don't think that is persistent. Currently I am manually adding these 2 files to the docker startup location /mnt/user/appdata/invtrack which seems a bit mickey mouse. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.