Sn3akyP3t3 Posted December 10, 2017 Share Posted December 10, 2017 I've been living on a rocky road scripting headless selenium to automate firefox running from UnRaid hosted Docker containers . When things are stable its all good, but every now and then I run into a snag that tends to take awhile to identify and resolve. This time I believe I've run into failure that may have occurred due to exploring a RC version of UnRaid 6.4.0_rc15e. This would be a snippet of code and logs that resulted from a known working script. Either one of these two methods: display = Display(visible=0, size=(800, 600)) #Alternate pyvirtualdisplay way display = Xvfb() # This is throwing errors now in DEC 2017... results in this error: Traceback (most recent call last): File "./Selenium/DoSomeStuffOhGreatSeleniumAndFirefox.py", line 553, in <module> display.start() File "/usr/local/lib/python3.6/site-packages/xvfbwrapper.py", line 54, in start close_fds=True) File "/usr/local/lib/python3.6/subprocess.py", line 709, in __init__ restore_signals, start_new_session) File "/usr/local/lib/python3.6/subprocess.py", line 1344, in _execute_child raise child_exception_type(errno_num, err_msg, err_filename) FileNotFoundError: [Errno 2] No such file or directory: 'Xvfb': 'Xvfb' This area of the script hasn't broken like this ever before. I had previously ran into a networking nightmare which turned out to be the technologies provided by the container used, but this issue with Xvfb I believe was originally provided from UnRaid. I can't say that with absolute certainty because I did only recently recover my cache drive and appdata content because of a problem I encountered with cache pool maintenance. Could anyone confirm that this works for them on UnRaid 6.3.x? I'm running 6.4.0_rc15e. Xvfb -help I get this: -bash: Xvfb: command not found Quote Link to comment
Sn3akyP3t3 Posted December 10, 2017 Author Share Posted December 10, 2017 While the disappearance of xvfb is investigated I've been exploring alternative options. Not too long ago there were no alternative options, but both Chrome and Firefox have made it possible to conduct running headless without use of a display virtualization tool like xvfb. The documentation that reveals that is pretty straightforward. https://developer.mozilla.org/en-US/Firefox/Headless_mode In my exploration to achieve this nirvana I've encountered a series of other problems that appear to be partially UnRaid related and also my lack of understanding of Docker. My first problem is that pip installs conducted from inside a container fail due to permissions. I'm using python3 when I call a python script. I found I can resolve this issue if I make sure to operate from within a python:3.6.3 container. To get the best of both worlds then it stands to reason that I need both python:3.6.3 and selenium/node-firefox to co-exist. I threw together a custom Dockerfile to marry the two. FROM selenium/node-firefox FROM python:3.6.3 So while trying to explore that I'm running into yet another problem. Once I'm running inside the container I'm noticing that I'm missing the necessary geckodriver and firefox binaries. https://github.com/SeleniumHQ/docker-selenium/blob/master/NodeFirefox/Dockerfile I believe I have a gross misconception on what the FROM command is doing in the Dockerfile. It seems to be pulling in all the fixings, but why then does the resulting container not have the desired binares? Quote Link to comment
Sn3akyP3t3 Posted December 10, 2017 Author Share Posted December 10, 2017 Looks like I fell into a common mistake pit... https://runnable.com/blog/9-common-dockerfile-mistakes I'll have to find another way to resolve these permission issues. These also only recently showed up after updating to RC UnRaid and overlaps with my cache drive recovery fiasco as well. Quote Link to comment
Sn3akyP3t3 Posted December 10, 2017 Author Share Posted December 10, 2017 I was thinking I could solve this with multi-stage builds https://docs.docker.com/engine/userguide/eng-image/multistage-build/#use-multi-stage-builds but it looks like neither of the containers have a simple set of work that I can just scrape a few files from one place to another. I'm going to see if I can focus on the permissions issue that sent me down this path in the first place. Quote Link to comment
Sn3akyP3t3 Posted December 17, 2017 Author Share Posted December 17, 2017 I still don't understand what happened that caused the permission problems. For now, I'm getting around it by forcing ROOT user. I understand this is a bad practice and security risk, but it is literally the only way I'm able to make this work for now. I'm going to begin making use of virtual environments in python and hopefully, I can no dodge need of ROOT afterward. I also have no idea what caused xvfb to fail all the sudden either, but the new Firefox headless methods work without it. I solved merging docker containers by taking one docker container with the most complexity and fulfilled mostly everything needed and installed the missing components within. The resulting Dockerfile: FROM selenium/node-firefox USER root # Found that the python3 version on Unraid conflicted with the intended installation with PIP further on. RUN apt-get purge -y python* RUN apt-get update && apt-get install -y python3-venv && apt-get install -y python3 && apt-get install -y python3-pip RUN mkdir /tmp/Requirements ADD ./installRequirements.py /tmp/Requirements ADD ./requirements.txt /tmp/Requirements ADD ./pushbullet.py-0.10.0-py2.py3-none-any.whl /tmp/Requirements WORKDIR /tmp RUN pip3 install --user -r /tmp/Requirements/requirements.txt Quote Link to comment
Sn3akyP3t3 Posted December 24, 2017 Author Share Posted December 24, 2017 Just realized this probably belongs under Prerelease 6.4 Support Quote Link to comment
bonienl Posted December 24, 2017 Share Posted December 24, 2017 23 minutes ago, Sn3akyP3t3 said: Just realized this probably belongs under Prerelease 6.4 Support Why? It is not an unRAID issue. Permissions haven't changed between 6.3 and 6.4. Quote Link to comment
Sn3akyP3t3 Posted January 13, 2018 Author Share Posted January 13, 2018 On 12/24/2017 at 10:13 AM, bonienl said: Why? It is not an unRAID issue. Permissions haven't changed between 6.3 and 6.4. All I know is that in this case, the scripts failed to run after the update to 6.4. I'm not keen on the underlying changes made and what impacts they may or may not have. Xvfb worked before, but not after. Docker permissions likely were the same as before, but because Xvfb suddenly failed I was forced to seek out an alternative which uncovered the permissions issue. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.