I've been living on a rocky road scripting headless selenium to automate firefox running from UnRaid hosted Docker containers  .  When things are stable its all good, but every now and then I run into a snag that tends to take awhile to identify and resolve.  This time I believe I've run into failure that may have occurred due to exploring a RC version of UnRaid 6.4.0_rc15e.

 

This would be a snippet of code and logs that resulted from a known working script.

Either one of these two methods:
display = Display(visible=0, size=(800, 600)) #Alternate pyvirtualdisplay way
display = Xvfb() # This is throwing errors now in DEC 2017...

results in this error:
Traceback (most recent call last):
  File "./Selenium/DoSomeStuffOhGreatSeleniumAndFirefox.py", line 553, in <module>
    display.start()
  File "/usr/local/lib/python3.6/site-packages/xvfbwrapper.py", line 54, in start
    close_fds=True)
  File "/usr/local/lib/python3.6/subprocess.py", line 709, in __init__
    restore_signals, start_new_session)
  File "/usr/local/lib/python3.6/subprocess.py", line 1344, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'Xvfb': 'Xvfb'

This area of the script hasn't broken like this ever before.  I had previously ran into a networking nightmare which turned out to be the technologies provided by the container used, but this issue with Xvfb I believe was originally provided from UnRaid.  I can't say that with absolute certainty because I did only recently recover my cache drive and appdata content because of a problem I encountered with cache pool maintenance.

 

Could anyone confirm that this works for them on UnRaid 6.3.x?  I'm running 6.4.0_rc15e.

Xvfb -help

I get this:

-bash: Xvfb: command not found

 

While the disappearance of xvfb is investigated I've been exploring alternative options.  Not too long ago there were no alternative options, but both Chrome and Firefox have made it possible to conduct running headless without use of a display virtualization tool like xvfb.  The documentation that reveals that is pretty straightforward.  https://developer.mozilla.org/en-US/Firefox/Headless_mode

 

In my exploration to achieve this nirvana I've encountered a series of other problems that appear to be partially UnRaid related and also my lack of understanding of Docker.  My first problem is that pip installs conducted from inside a container fail due to permissions.  I'm using python3 when I call a python script.  I found I can resolve this issue if I make sure to operate from within a python:3.6.3 container.  To get the best of both worlds then it stands to reason that I need both python:3.6.3 and selenium/node-firefox to co-exist.  I threw together a custom Dockerfile to marry the two.

FROM selenium/node-firefox
FROM python:3.6.3

So while trying to explore that I'm running into yet another problem.  Once I'm running inside the container I'm noticing that I'm missing the necessary geckodriver and firefox binaries.  https://github.com/SeleniumHQ/docker-selenium/blob/master/NodeFirefox/Dockerfile   I believe I have a gross misconception on what the FROM command is doing in the Dockerfile.  It seems to be pulling in all the fixings, but why then does the resulting container not have the desired binares?

Looks like I fell into a common mistake pit... https://runnable.com/blog/9-common-dockerfile-mistakes  I'll have to find another way to resolve these permission issues.  These also only recently showed up after updating to RC UnRaid and overlaps with my cache drive recovery fiasco as well.

I was thinking I could solve this with multi-stage builds https://docs.docker.com/engine/userguide/eng-image/multistage-build/#use-multi-stage-builds but it looks like neither of the containers have a simple set of work that I can just scrape a few files from one place to another.  I'm going to see if I can focus on the permissions issue that sent me down this path in the first place.

I still don't understand what happened that caused the permission problems.  For now, I'm getting around it by forcing ROOT user.  I understand this is a bad practice and security risk, but it is literally the only way I'm able to make this work for now.  I'm going to begin making use of virtual environments in python and hopefully, I can no dodge need of ROOT afterward.  I also have no idea what caused xvfb to fail all the sudden either, but the new Firefox headless methods work without it.

 

I solved merging docker containers by taking one docker container with the most complexity and fulfilled mostly everything needed and installed the missing components within.  The resulting Dockerfile:

FROM selenium/node-firefox
USER root
# Found that the python3 version on Unraid conflicted with the intended installation with PIP further on.
RUN apt-get purge -y python*

RUN apt-get update && apt-get install -y python3-venv && apt-get install -y python3  && apt-get install -y python3-pip

RUN mkdir /tmp/Requirements
ADD ./installRequirements.py /tmp/Requirements
ADD ./requirements.txt /tmp/Requirements
ADD ./pushbullet.py-0.10.0-py2.py3-none-any.whl /tmp/Requirements
WORKDIR /tmp

RUN pip3 install --user -r /tmp/Requirements/requirements.txt

 

Just realized this probably belongs under Prerelease 6.4 Support

23 minutes ago, Sn3akyP3t3 said:

Just realized this probably belongs under Prerelease 6.4 Support

 

Why? It is not an unRAID issue. Permissions haven't changed between 6.3 and 6.4.

 

On 12/24/2017 at 10:13 AM, bonienl said:

 

Why? It is not an unRAID issue. Permissions haven't changed between 6.3 and 6.4.

 

All I know is that in this case, the scripts failed to run after the update to 6.4.  I'm not keen on the underlying changes made and what impacts they may or may not have.  Xvfb worked before, but not after.  Docker permissions likely were the same as before, but because Xvfb suddenly failed I was forced to seek out an alternative which uncovered the permissions issue.