[Support] Crocs - Tube Archivist


Crocs

Recommended Posts

3 hours ago, Ezrael said:

I just have one question (right now): What is the elasticsearch repo? It's longer than the text box and I can't copy it.
I can't see that repo on Github/Docker either...

 

Thank you!

Sorry about that, its "docker.elastic.co/elasticsearch/elasticsearch:7.14.1"

Link to comment
1 hour ago, muppie said:

I have other Redis and Elasticsearch dockers on my machine and it seems like they are connecting but Im getting an Internal Server Error when trying to use the web gui. Posting logs soon. Just not sure if I should exclude anything from the logs, should I?

Make sure you have the "http://" in front of your IP for the ElasticSearch variable in the template.

 

It will throw errors if you don't.

Link to comment

I have http in front but I’m still getting a Internal Server Error.

 

Do I have the wrong redis version? I don’t have any special for redis json.

Logs here:

 

Quote

ErrorWarningSystemArrayLogin


redis.exceptions.ResponseError: unknown command `JSON.GET`, with args beginning with: `config`,
[uWSGI] getting INI configuration from uwsgi.ini
*** Starting uWSGI 2.0.19.1 (64bit) on [Fri Sep 24 22:13:05 2021] ***
compiled with version: 10.2.1 20210110 on 08 September 2021 09:13:02
os: Linux-5.10.28-Unraid #1 SMP Wed Apr 7 08:23:18 PDT 2021
nodename: 4742eae369e2
machine: x86_64
clock source: unix
detected number of CPU cores: 8
current working directory: /app
writing pidfile to /tmp/project-master.pid
detected binary path: /usr/local/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
your processes number limit is 191529
your memory page size is 4096 bytes
detected max file descriptor number: 40960
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to TCP address :8080 fd 3
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
Python version: 3.9.7 (default, Sep 3 2021, 02:02:37) [GCC 10.2.1 20210110]
*** Python threads support is disabled. You can enable it with --enable-threads ***
Python main interpreter initialized at 0x557c139c7800
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 145808 bytes (142 KB) for 1 cores
*** Operational MODE: single process ***
Traceback (most recent call last):
File "/app/./config/wsgi.py", line 16, in <module>
application = get_wsgi_application()
File "/usr/local/lib/python3.9/site-packages/django/core/wsgi.py", line 12, in get_wsgi_application
django.setup(set_prefix=False)
File "/usr/local/lib/python3.9/site-packages/django/__init__.py", line 24, in setup
apps.populate(settings.INSTALLED_APPS)
File "/usr/local/lib/python3.9/site-packages/django/apps/registry.py", line 91, in populate
app_config = AppConfig.create(entry)
File "/usr/local/lib/python3.9/site-packages/django/apps/config.py", line 212, in create
mod = import_module(mod_path)
File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "/app/./home/__init__.py", line 7, in <module>
from home.src.index_management import index_check
File "/app/./home/src/index_management.py", line 159, in <module>
class ElasticIndex:
File "/app/./home/src/index_management.py", line 164, in ElasticIndex
CONFIG = AppConfig().config
File "/app/./home/src/config.py", line 18, in __init__
self.config = self.get_config()
File "/app/./home/src/config.py", line 22, in get_config
config = self.get_config_redis()
File "/app/./home/src/config.py", line 54, in get_config_redis
config = get_message("config")
File "/app/./home/src/helper.py", line 80, in get_message
reply = redis_connection.execute_command("JSON.GET", key)
File "/usr/local/lib/python3.9/site-packages/redis/client.py", line 901, in execute_command
return self.parse_response(conn, command_name, **options)
File "/usr/local/lib/python3.9/site-packages/redis/client.py", line 915, in parse_response
response = connection.read_response()
File "/usr/local/lib/python3.9/site-packages/redis/connection.py", line 756, in read_response
raise response
redis.exceptions.ResponseError: unknown command `JSON.GET`, with args beginning with: `config`,
unable to load app 0 (mountpoint='') (callable not found or import error)
*** no app loaded. going in full dynamic mode ***
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 13)
spawned uWSGI worker 1 (pid: 23, cores: 1)
Usage: celery [OPTIONS] COMMAND [ARGS]...

Error: Invalid value for '-A' / '--app':
Unable to load celery application.
While trying to load the module home.tasks the following error occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/celery/bin/celery.py", line 53, in convert
return find_app(value)
File "/usr/local/lib/python3.9/site-packages/celery/app/utils.py", line 384, in find_app
sym = symbol_by_name(app, imp=imp)
File "/usr/local/lib/python3.9/site-packages/kombu/utils/imports.py", line 56, in symbol_by_name
module = imp(module_name, package=package, **kwargs)
File "/usr/local/lib/python3.9/site-packages/celery/utils/imports.py", line 100, in import_from_cwd
return imp(module, package=package)
File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 850, in exec_module
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
File "/app/home/__init__.py", line 7, in <module>
from home.src.index_management import index_check
File "/app/home/src/index_management.py", line 159, in <module>
class ElasticIndex:
File "/app/home/src/index_management.py", line 164, in ElasticIndex
CONFIG = AppConfig().config
File "/app/home/src/config.py", line 18, in __init__
self.config = self.get_config()
File "/app/home/src/config.py", line 22, in get_config
config = self.get_config_redis()
File "/app/home/src/config.py", line 54, in get_config_redis
config = get_message("config")
File "/app/home/src/helper.py", line 80, in get_message
reply = redis_connection.execute_command("JSON.GET", key)
File "/usr/local/lib/python3.9/site-packages/redis/client.py", line 901, in execute_command
return self.parse_response(conn, command_name, **options)
File "/usr/local/lib/python3.9/site-packages/redis/client.py", line 915, in parse_response
response = connection.read_response()
File "/usr/local/lib/python3.9/site-packages/redis/connection.py", line 756, in read_response
raise response
redis.exceptions.ResponseError: unknown command `JSON.GET`, with args beginning with: `config`,

--- no python application found, check your startup logs for errors ---
[pid: 23|app: -1|req: -1/1] 10.0.8.2 () {40 vars in 650 bytes} [Fri Sep 24 20:13:09 2021] GET / => generated 21 bytes in 0 msecs (HTTP/1.1 500) 2 headers in 83 bytes (0 switches on core 0)
--- no python application found, check your startup logs for errors ---
[pid: 23|app: -1|req: -1/2] 10.0.8.2 () {40 vars in 650 bytes} [Fri Sep 24 20:13:11 2021] GET / => generated 21 bytes in 0 msecs (HTTP/1.1 500) 2 headers in 83 bytes (0 switches on core 0)
{
"name" : "87c285cb6eaf",
"cluster_name" : "docker-cluster",
"cluster_uuid" : "iaxRVLu4RTyqKTB-TN6A4Q",
"version" : {
"number" : "7.10.2",
"build_flavor" : "default",
"build_type" : "docker",
"build_hash" : "747e1cc71def077253878a59143c1f785afa92b9",
"build_date" : "2021-01-13T00:42:12.435326Z",
"build_snapshot" : false,
"lucene_version" : "8.7.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
Traceback (most recent call last):
File "/app/manage.py", line 23, in <module>
main()
File "/app/manage.py", line 19, in main
execute_from_command_line(sys.argv)
File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 395, in execute
django.setup()
File "/usr/local/lib/python3.9/site-packages/django/__init__.py", line 24, in setup
apps.populate(settings.INSTALLED_APPS)
File "/usr/local/lib/python3.9/site-packages/django/apps/registry.py", line 91, in populate
app_config = AppConfig.create(entry)
File "/usr/local/lib/python3.9/site-packages/django/apps/config.py", line 212, in create
mod = import_module(mod_path)
File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 850, in exec_module
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
File "/app/home/__init__.py", line 7, in <module>
from home.src.index_management import index_check
File "/app/home/src/index_management.py", line 159, in <module>
class ElasticIndex:
File "/app/home/src/index_management.py", line 164, in ElasticIndex
CONFIG = AppConfig().config
File "/app/home/src/config.py", line 18, in __init__
self.config = self.get_config()
File "/app/home/src/config.py", line 22, in get_config
config = self.get_config_redis()
File "/app/home/src/config.py", line 54, in get_config_redis
config = get_message("config")
File "/app/home/src/helper.py", line 80, in get_message
reply = redis_connection.execute_command("JSON.GET", key)
File "/usr/local/lib/python3.9/site-packages/redis/client.py", line 901, in execute_command
return self.parse_response(conn, command_name, **options)
File "/usr/local/lib/python3.9/site-packages/redis/client.py", line 915, in parse_response
response = connection.read_response()
File "/usr/local/lib/python3.9/site-packages/redis/connection.py", line 756, in read_response
raise response
redis.exceptions.ResponseError: unknown command `JSON.GET`, with args beginning with: `config`,
Traceback (most recent call last):
File "/app/manage.py", line 23, in <module>
main()
File "/app/manage.py", line 19, in main
execute_from_command_line(sys.argv)
File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 395, in execute
django.setup()
File "/usr/local/lib/python3.9/site-packages/django/__init__.py", line 24, in setup
apps.populate(settings.INSTALLED_APPS)
File "/usr/local/lib/python3.9/site-packages/django/apps/registry.py", line 91, in populate
app_config = AppConfig.create(entry)
File "/usr/local/lib/python3.9/site-packages/django/apps/config.py", line 212, in create
mod = import_module(mod_path)
File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 850, in exec_module
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
File "/app/home/__init__.py", line 7, in <module>
from home.src.index_management import index_check
File "/app/home/src/index_management.py", line 159, in <module>
class ElasticIndex:
File "/app/home/src/index_management.py", line 164, in ElasticIndex
CONFIG = AppConfig().config
File "/app/home/src/config.py", line 18, in __init__
self.config = self.get_config()
File "/app/home/src/config.py", line 22, in get_config
config = self.get_config_redis()
File "/app/home/src/config.py", line 54, in get_config_redis
config = get_message("config")
File "/app/home/src/helper.py", line 80, in get_message
reply = redis_connection.execute_command("JSON.GET", key)
File "/usr/local/lib/python3.9/site-packages/redis/client.py", line 901, in execute_command
return self.parse_response(conn, command_name, **options)
File "/usr/local/lib/python3.9/site-packages/redis/client.py", line 915, in parse_response
response = connection.read_response()
File "/usr/local/lib/python3.9/site-packages/redis/connection.py", line 756, in read_response
raise response
redis.exceptions.ResponseError: unknown command `JSON.GET`, with args beginning with: `config`,
[uWSGI] getting INI configuration from uwsgi.ini
*** Starting uWSGI 2.0.19.1 (64bit) on [Sat Sep 25 19:34:29 2021] ***
compiled with version: 10.2.1 20210110 on 08 September 2021 09:13:02
os: Linux-5.10.28-Unraid #1 SMP Wed Apr 7 08:23:18 PDT 2021
nodename: 4742eae369e2
machine: x86_64
clock source: unix
detected number of CPU cores: 8
current working directory: /app
writing pidfile to /tmp/project-master.pid
detected binary path: /usr/local/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
your processes number limit is 191529
your memory page size is 4096 bytes
detected max file descriptor number: 40960
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to TCP address :8080 fd 3
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
Python version: 3.9.7 (default, Sep 3 2021, 02:02:37) [GCC 10.2.1 20210110]
*** Python threads support is disabled. You can enable it with --enable-threads ***
Python main interpreter initialized at 0x5615eec8f800
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 145808 bytes (142 KB) for 1 cores
*** Operational MODE: single process ***
Traceback (most recent call last):
File "/app/./config/wsgi.py", line 16, in <module>
application = get_wsgi_application()
File "/usr/local/lib/python3.9/site-packages/django/core/wsgi.py", line 12, in get_wsgi_application
django.setup(set_prefix=False)
File "/usr/local/lib/python3.9/site-packages/django/__init__.py", line 24, in setup
apps.populate(settings.INSTALLED_APPS)
File "/usr/local/lib/python3.9/site-packages/django/apps/registry.py", line 91, in populate
app_config = AppConfig.create(entry)
File "/usr/local/lib/python3.9/site-packages/django/apps/config.py", line 212, in create
mod = import_module(mod_path)
File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "/app/./home/__init__.py", line 7, in <module>
from home.src.index_management import index_check
File "/app/./home/src/index_management.py", line 159, in <module>
class ElasticIndex:
File "/app/./home/src/index_management.py", line 164, in ElasticIndex
CONFIG = AppConfig().config
File "/app/./home/src/config.py", line 18, in __init__
self.config = self.get_config()
File "/app/./home/src/config.py", line 22, in get_config
config = self.get_config_redis()
File "/app/./home/src/config.py", line 54, in get_config_redis
config = get_message("config")
File "/app/./home/src/helper.py", line 80, in get_message
reply = redis_connection.execute_command("JSON.GET", key)
File "/usr/local/lib/python3.9/site-packages/redis/client.py", line 901, in execute_command
return self.parse_response(conn, command_name, **options)
File "/usr/local/lib/python3.9/site-packages/redis/client.py", line 915, in parse_response
response = connection.read_response()
File "/usr/local/lib/python3.9/site-packages/redis/connection.py", line 756, in read_response
raise response
redis.exceptions.ResponseError: unknown command `JSON.GET`, with args beginning with: `config`,
unable to load app 0 (mountpoint='') (callable not found or import error)
*** no app loaded. going in full dynamic mode ***
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 12)
spawned uWSGI worker 1 (pid: 22, cores: 1)
Usage: celery [OPTIONS] COMMAND [ARGS]...

Error: Invalid value for '-A' / '--app':
Unable to load celery application.
While trying to load the module home.tasks the following error occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/celery/bin/celery.py", line 53, in convert
return find_app(value)
File "/usr/local/lib/python3.9/site-packages/celery/app/utils.py", line 384, in find_app
sym = symbol_by_name(app, imp=imp)
File "/usr/local/lib/python3.9/site-packages/kombu/utils/imports.py", line 56, in symbol_by_name
module = imp(module_name, package=package, **kwargs)
File "/usr/local/lib/python3.9/site-packages/celery/utils/imports.py", line 100, in import_from_cwd
return imp(module, package=package)
File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 850, in exec_module
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
File "/app/home/__init__.py", line 7, in <module>
from home.src.index_management import index_check
File "/app/home/src/index_management.py", line 159, in <module>
class ElasticIndex:
File "/app/home/src/index_management.py", line 164, in ElasticIndex
CONFIG = AppConfig().config
File "/app/home/src/config.py", line 18, in __init__
self.config = self.get_config()
File "/app/home/src/config.py", line 22, in get_config
config = self.get_config_redis()
File "/app/home/src/config.py", line 54, in get_config_redis
config = get_message("config")
File "/app/home/src/helper.py", line 80, in get_message
reply = redis_connection.execute_command("JSON.GET", key)
File "/usr/local/lib/python3.9/site-packages/redis/client.py", line 901, in execute_command
return self.parse_response(conn, command_name, **options)
File "/usr/local/lib/python3.9/site-packages/redis/client.py", line 915, in parse_response
response = connection.read_response()
File "/usr/local/lib/python3.9/site-packages/redis/connection.py", line 756, in read_response
raise response
redis.exceptions.ResponseError: unknown command `JSON.GET`, with args beginning with: `config`,

 

Link to comment
1 hour ago, muppie said:

I have http in front but I’m still getting a Internal Server Error.

 

Do I have the wrong redis version? I don’t have any special for redis json.

Logs here:

 

 

Check the screenshots above, do you have Redis and ES mapped to the correct volumes inside of the TubeArchivsit directory?

Link to comment
11 hours ago, Crocs said:

Check the screenshots above, do you have Redis and ES mapped to the correct volumes inside of the TubeArchivsit directory?

I’ve filled out the redis IP and elasticsearch IP in the template and I’ve tried to review the screenshots again and to me they look the same. Attaching it here:

 

I cannot spot any errors. If I add port to the redis IP it is duplicated in the logs (as IP:6379:6379) so I removed it.

DCA796EC-E861-4C44-A1A2-D9266780E86B.jpeg

Edited by muppie
Link to comment
20 hours ago, muppie said:

I’ve filled out the redis IP and elasticsearch IP in the template and I’ve tried to review the screenshots again and to me they look the same. Attaching it here:

 

I cannot spot any errors. If I add port to the redis IP it is duplicated in the logs (as IP:6379:6379) so I removed it.

DCA796EC-E861-4C44-A1A2-D9266780E86B.jpeg

I'm not seeing anything obvious either. The only thing that I can think is that you're using different IP addresses for each container. Not being the dev of the project, I can only assume that this might be your issue.

Link to comment
12 hours ago, muppie said:

I posted mu question and logs in the github for the application and it appears that you need a special redis version that support json. As far as I can see, there is no template for the correct redis so we need to install it manuallly

I posted screenshots above for the correct templates needed.

Link to comment
  • 1 month later...
  • 1 month later...
On 10/4/2021 at 8:20 AM, muppie said:

I posted mu question and logs in the github for the application and it appears that you need a special redis version that support json. As far as I can see, there is no template for the correct redis so we need to install it manuallly

Nope, all three dependencies are in CA.

Check the notes in the TA container.

Link to comment
On 12/13/2021 at 12:57 AM, Douji said:

Hello all,

the current ElasticSearch container contains Log4j 2.11.

Anyone know whether a newer tag already has a fix built in?

 

Cheers

Great question,

 

ES has already addressed it here: https://discuss.elastic.co/t/apache-log4j2-remote-code-execution-rce-vulnerability-cve-2021-44228-esa-2021-31/291476

 

Basically, it was never vulnerable. However, we just pushed an update to 7.16.1, the latest stable release.

 

Thank you for bringing that up :)

Link to comment
  • 3 weeks later...

Getting the internal server error despite installing the two dependent packages and pointing to the local IP addresses in the setup.  Any advice here?  See link below for settings.  Im not seeing anything I need to change in the dependent packages but maybe Im missing something.  Thanks!

Screenshot 2022-01-25 104527.png

Link to comment
On 1/25/2022 at 9:43 AM, Vaslo said:

Getting the internal server error despite installing the two dependent packages and pointing to the local IP addresses in the setup.  Any advice here?  See link below for settings.  Im not seeing anything I need to change in the dependent packages but maybe Im missing something.  Thanks!

Screenshot 2022-01-25 104527.png

Your Redis variable is set up wrong.

 

It's just the IP address, your variable should be '192.168.1.70'.

 

No 'http://' or '6379'

 

Let me know if that works!

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.