Questions on "properly" migrating an existing docker to Unraid: docker-python-chromedriver

[RoninTech]

Hi all,

 

I've had an Unraid cron python script (invtrack - investment tracker) running for about a year that scrapes some web info every day, puts it into a dockered Influxdb and makes it accessible via a dockered Grafana.  All of this has been working great within Unraid.  Recently the website I am scraping battened down the hatches to obfuscated JavaScript dynamic content.  This meant I had to migrate my python parser to use Selenium that requires a browser installation.   I found an existing python/selenium/chrome docker here:

 

https://hub.docker.com/r/joyzoursky/python-chromedriver

 

To get it working at first I simply created a /mnt/user/appdata/invtrack folder, put my python scraper script in it, then ran the docker run command they suggest (slightly modified for TZ etc.) from the new folder:

 

docker run --name scraper -e TZ="America/Tijuana" -e HOST_OS="Unraid" -it -v $(pwd):/usr/workspace joyzoursky/python-chromedriver:3.7-selenium bash

 

I can then run my Selenium updated python scraper and everything works as intended.  That being said there are some obvious weaknesses to not having been added to the Unraid docker page correctly.

 

1. The Unraid docker page entry for the scraper does not allow me to edit/update the docker.
2. Hitting the "update ready" link for the docker doesn't work.
3. My Python scraper has to be manually moved to /mnt/user/appdata/invtrack, it's not part of the docker image.
4. I have to manually execute an init script when the docker starts to setup some python dependencies for my scraper.
5. I have to setup a daily Unraid cron job to kick the scraper inside the docker.

 

I enabled docker hub searches in Unraid's Community Apps and found and downloaded "python chromedriver".  I then edited the docker parameters as  shown in the attached image.

 

When I attempt to start the docker with network set to host I get nothing in the logs.  If I set it to bridge I see the following (2 start attempts) but the docker does not come up.  Interestingly when I got this docker working outside of Community Apps by just doing the above docker run command, the Unraid docker page  shows it as having a "default" network which isn't an option when I add it through Community apps.:

Mar 19 22:04:09 unraid kernel: vethef16f86: renamed from eth0
Mar 19 22:04:09 unraid kernel: docker0: port 7(vetha506275) entered disabled state
Mar 19 22:04:09 unraid kernel: docker0: port 7(vetha506275) entered disabled state
Mar 19 22:04:09 unraid kernel: device vetha506275 left promiscuous mode
Mar 19 22:04:09 unraid kernel: docker0: port 7(vetha506275) entered disabled state
Mar 19 22:08:15 unraid kernel: docker0: port 7(veth5e15951) entered blocking state
Mar 19 22:08:15 unraid kernel: docker0: port 7(veth5e15951) entered disabled state
Mar 19 22:08:15 unraid kernel: device veth5e15951 entered promiscuous mode
Mar 19 22:08:15 unraid kernel: IPv6: ADDRCONF(NETDEV_UP): veth5e15951: link is not ready
Mar 19 22:08:17 unraid kernel: eth0: renamed from veth247fa63
Mar 19 22:08:17 unraid kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth5e15951: link becomes ready
Mar 19 22:08:17 unraid kernel: docker0: port 7(veth5e15951) entered blocking state
Mar 19 22:08:17 unraid kernel: docker0: port 7(veth5e15951) entered forwarding state
Mar 19 22:08:18 unraid kernel: veth247fa63: renamed from eth0
Mar 19 22:08:18 unraid kernel: docker0: port 7(veth5e15951) entered disabled state
Mar 19 22:08:18 unraid kernel: docker0: port 7(veth5e15951) entered disabled state
Mar 19 22:08:18 unraid kernel: device veth5e15951 left promiscuous mode
Mar 19 22:08:18 unraid kernel: docker0: port 7(veth5e15951) entered disabled state
Mar 19 22:08:30 unraid kernel: docker0: port 7(veth7c2c127) entered blocking state
Mar 19 22:08:30 unraid kernel: docker0: port 7(veth7c2c127) entered disabled state
Mar 19 22:08:30 unraid kernel: device veth7c2c127 entered promiscuous mode
Mar 19 22:08:30 unraid kernel: IPv6: ADDRCONF(NETDEV_UP): veth7c2c127: link is not ready
Mar 19 22:08:30 unraid kernel: docker0: port 7(veth7c2c127) entered blocking state
Mar 19 22:08:30 unraid kernel: docker0: port 7(veth7c2c127) entered forwarding state
Mar 19 22:08:30 unraid kernel: docker0: port 7(veth7c2c127) entered disabled state
Mar 19 22:08:33 unraid kernel: eth0: renamed from veth7af64bb
Mar 19 22:08:33 unraid kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth7c2c127: link becomes ready
Mar 19 22:08:33 unraid kernel: docker0: port 7(veth7c2c127) entered blocking state
Mar 19 22:08:33 unraid kernel: docker0: port 7(veth7c2c127) entered forwarding state
Mar 19 22:08:33 unraid kernel: veth7af64bb: renamed from eth0
Mar 19 22:08:33 unraid kernel: docker0: port 7(veth7c2c127) entered disabled state
Mar 19 22:08:34 unraid kernel: docker0: port 7(veth7c2c127) entered disabled state
Mar 19 22:08:34 unraid kernel: device veth7c2c127 left promiscuous mode
Mar 19 22:08:34 unraid kernel: docker0: port 7(veth7c2c127) entered disabled state

 

So I'm looking for some feedback on any potential errors I've made in adding the docker via Community Apps that are causing the docker to not start.  Would also appreciate any tips on how to kick a script inside my docker once it is running, and how to run a setup script (to pip python modules) inside a docker on startup. 

 

Thanks, and apologies if this is in the wrong place or has wrong/missing info.  I will correct them as soon as I am aware. 😃

Edited March 20, 2019 by RoninTech
Added a point on network

[Squid]

Your best course of action is to compare the docker run commands between when editing the container and what you used to run at the command line and then making adjustments accordingly

[RoninTech]

Thanks squid.  Turns out it was the "-it" options.  Now the docker starts as expected.  I also was able to hook it up to cron via the docker run command.

 

My last issue I need to work out is how can I permanently add my python parser and init script into this other person's docker container?  I see there is a "docker cp" but I don't think that is persistent.  Currently I am manually adding these 2 files to the docker startup location /mnt/user/appdata/invtrack which seems a bit mickey mouse.