[Support] Crocs - Tube Archivist


Crocs

Recommended Posts

4 minutes ago, msherman83 said:

Hello,

Same issue as the others above on a fresh install. It seems for some reason you need to change the ./es folder to 777 to get the es container to write to it.

 

Access the Unraid CLI and enter:

chmod -R 777 /mnt/user/appdata/TubeArchivist/es

 

I noticed a large java file permissions error in the ES container logs the first few times i tried to start it. It would die right away, so you couldn't login to it.

 

 

Looking into this... might be a difference between updating vs starting fresh then.  Thanks for the reports, we're looking into it.

Link to comment
53 minutes ago, Emilia said:

I'm trying to install tube archivist, not update from a previous version. I'm also just using all the default settings:

* also tried reinstalling it but that didn't fix anything

 

21 minutes ago, msherman83 said:

Hello,

Same issue as the others above on a fresh install. It seems for some reason you need to change the ./es folder to 777 to get the es container to write to it.

 

Access the Unraid CLI and enter:

chmod -R 777 /mnt/user/appdata/TubeArchivist/es

 

I noticed a large java file permissions error in the ES container logs the first few times i tried to start it. It would die right away, so you couldn't login to it.

 

 

 

Apparently this is a known potential issue that can affect new or updated installations.  The fix is documented under "Common Errors" on the projects' GitHub here: https://github.com/tubearchivist/tubearchivist#permissions-for-elasticsearch

  • Thanks 2
Link to comment

I finally fixed the permission issue (thanks), and the new error for TubeArchivist-ES is:

 

"Failure running machine learning native code"

 

Quote

{"@timestamp":"2022-12-16T23:57:30.532Z", "log.level":"ERROR", "message":"fatal exception while booting Elasticsearch", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.bootstrap.Elasticsearch","elasticsearch.node.name":"5a3593a3ad63","elasticsearch.cluster.name":"docker-cluster","error.type":"org.elasticsearch.ElasticsearchException","error.message":"Failure running machine learning native code. This could be due to running on an unsupported OS or distribution, missing OS libraries, or a problem with the temp directory. To bypass this problem by running Elasticsearch without machine learning functionality set [xpack.ml.enabled: false].","error.stack_trace":"org.elasticsearch.ElasticsearchException: Failure running machine learning native code. This could be due to running on an unsupported OS or distribution, missing OS libraries, or a problem with the temp directory. To bypass this problem by running Elasticsearch without machine learning functionality set [xpack.ml.enabled: false].\n\tat org.elasticsearch.xpack.ml.MachineLearning.createComponents(MachineLearning.java:950)\n\tat [email protected]/org.elasticsearch.node.Node.lambda$new$16(Node.java:709)\n\tat [email protected]/org.elasticsearch.plugins.PluginsService.lambda$flatMap$0(PluginsService.java:252)\n\tat java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273)\n\tat java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)\n\tat java.base/java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:722)\n\tat java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)\n\tat java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)\n\tat java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575)\n\tat java.base/java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)\n\tat java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616)\n\tat java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:622)\n\tat java.base/java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:627)\n\tat [email protected]/org.elasticsearch.node.Node.<init>(Node.java:724)\n\tat [email protected]/org.elasticsearch.node.Node.<init>(Node.java:318)\n\tat [email protected]/org.elasticsearch.bootstrap.Elasticsearch$2.<init>(Elasticsearch.java:214)\n\tat [email protected]/org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:214)\n\tat [email protected]/org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:67)\n"}

 

  • Like 1
Link to comment
  • 3 weeks later...

I can't get this running, I've followed the advice of this entire thread but now I've gotten stuck with all 3 containers running, but the ES container is erroring with: 

Been at this for hours now, appreciate any suggestions.
 

Quote

2023-01-03 12:40:25,312 process reaper (pid 194) ERROR Recursive call to appender rolling
{"@timestamp":"2023-01-03T20:40:25.285Z", "log.level":"ERROR", "message":"uncaught exception in thread [process reaper (pid 194)]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"process reaper (pid 194)","log.logger":"org.elasticsearch.bootstrap.ElasticsearchUncaughtExceptionHandler","elasticsearch.cluster.uuid":"tj8CjiO2TFS0To2q63O4dA","elasticsearch.node.id":"7Oj_m2-_Sc6mJnqmTQahMA","elasticsearch.node.name":"e48359223531","error.type":"java.security.AccessControlException","error.message":"access denied (\"java.lang.RuntimePermission\" \"modifyThread\")","error.stack_trace":"java.security.AccessControlException: access denied (\"java.lang.RuntimePermission\" \"modifyThread\")\n\tat java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:485)\n\tat java.base/java.security.AccessController.checkPermission(AccessController.java:1068)\n\tat java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:411)\n\tat [email protected]/org.elasticsearch.secure_sm.SecureSM.checkThreadAccess(SecureSM.java:166)\n\tat [email protected]/org.elasticsearch.secure_sm.SecureSM.checkAccess(SecureSM.java:120)\n\tat java.base/java.lang.Thread.checkAccess(Thread.java:2360)\n\tat java.base/java.lang.Thread.setDaemon(Thread.java:2308)\n\tat java.base/java.lang.ProcessHandleImpl.lambda$static$0(ProcessHandleImpl.java:103)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.<init>(ThreadPoolExecutor.java:637)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:928)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1021)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1158)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1589)\n\tat java.base/jdk.internal.misc.InnocuousThread.run(InnocuousThread.java:186)\n"}

 

Link to comment

Hi all, I have been running TubeArchivist now for a few weeks without issue, just subscribed to one channel or and downloading one video here or there. I added a few channels recently with ~500 videos or so ending up in my download queue.

 

Now all of a sudden every time I start the download process after a while my entire unraid server starts to become unresponsive, which is VERY weird considering this is just a container... 

 

I have tried limiting the download speed to 5000kbps (I have 500Mbps download connection) but the same thing still happens.

It's so bad that I can't even stop the container, unraid just says "server error" when I try and I can't get things like Plex to work, my NPM reverse proxy goes in/out (external monitoring on my public facing site will cite tons of up/down errors over night when the downloads run) and the only fix is to reboot my server because I can't even just stop the Tube Archivist container. 

 

My specs are pretty great, so it's not like it's killing the CPU or RAM or anything.

I'm just not sure what's going on, or what I can even do to try and fix it. 

Any advice or suggestions are appreciated! 

 

I should also note that while the issues were present and I was able to sometimes get the unraid webUI to load, it didn't show any spiked or pegged usages, CPU was at 20% at the time, RAM wasn't full, nothing seemed to be getting used excessively, but pages would fail to load, containers would fail to stop, etc, only recourse was to reboot from command line or web UI if I got lucky.

 

Specs

Processor: Dual Intel® Xeon® CPU E5-2650 v2 @ 2.60GHz

RAM: 128GB RAM

All docker containers run thru a single NIC interface eth1 - no VALNs on it, no bonding, they're just all using that single NIC (due to some previous issues I had with crashing on macvlan -- a known bug in unraid)

Link to comment

Upon investigating a little more - I did try pinning to just 2 cores, but no luck there, things still get wonky fast.

It does seem to be when it's finalizing/processing the video after it's finished downloading -- the download itself goes fine, and then if I monitor the Downloads page it will hit 100% completion then when it goes to processing the box showing the progress disappears and I lose all visibility on what TA is doing. -- this is also when things start to lock up on the whole server, other dockers become inaccessible, web UI and command line for unraid are terribly slow to respond.

 

TA doesn't seem to log TOO much in the docker logs but here's the logs I get...

 

download pending
6a80476e-9d66-4ba5-b673-e55367adc07f: set task id
[2023-01-07 19:00:47,564: INFO/MainProcess] Task download_pending[6a80476e-9d66-4ba5-b673-e55367adc07f] received
[2023-01-07 19:00:47,574: WARNING/ForkPoolWorker-1] Did not acquire download lock.
[2023-01-07 19:00:47,575: INFO/ForkPoolWorker-1] Task download_pending[6a80476e-9d66-4ba5-b673-e55367adc07f] succeeded in 0.006124248000560328s: None
download pending
0a949b28-3747-428b-9b05-729d0d8e24df: set task id
[2023-01-07 19:00:49,529: INFO/MainProcess] Task download_pending[0a949b28-3747-428b-9b05-729d0d8e24df] received
[2023-01-07 19:00:49,534: WARNING/ForkPoolWorker-1] Did not acquire download lock.
[2023-01-07 19:00:49,535: INFO/ForkPoolWorker-1] Task download_pending[0a949b28-3747-428b-9b05-729d0d8e24df] succeeded in 0.0038173840002855286s: None
{"error":{"root_cause":[{"type":"resource_not_found_exception","reason":"snapshot lifecycle policy or policies [ta_daily] not found, no policies are configured"}],"type":"resource_not_found_exception","reason":"snapshot lifecycle policy or policies [ta_daily] not found, no policies are configured"},"status":404}
Filter download view ignored only: False
Filter download view ignored only: True
Filter download view ignored only: True
Filter download view ignored only: False
Filter download view ignored only: True
zB-BRBmv34k: add single vid to download queue
Filter download view ignored only: False
Filter download view ignored only: True
Filter download view ignored only: False
[2023-01-07 19:08:12,452: WARNING/ForkPoolWorker-33] 6q6SXzmcqPY: get metadata from youtube
[2023-01-07 19:08:16,634: WARNING/ForkPoolWorker-33] UCXnNibvR_YIdyPs8PZIBoEw: get metadata from es
[2023-01-07 19:08:17,008: WARNING/ForkPoolWorker-33] 6q6SXzmcqPY: get ryd stats

 

Link to comment
Just now, Phuriousgeorge said:

This is very odd indeed.  I've been using TA extensively with just under 800k videos in my archive.

 

I would personally suggest using Discord or GitHub for support, as the dev of the app will respond there and it's just us users monitoring here

 

Rgr that, I agree it's a VERY strange issue. I download tons of data via Transmission and qBT as well and have never had issue.

Only thing TA is doing differently is it downloads to my cache drive first and then it gets procsesed/moved to the array.

 

I have NOT tried changing that config yet to put the /appdata/tubearchivist/downloads folder onto the array directly, but I have other friends using this to have over 10k videos as well already and have not run into this issue. 

 

Something very odd is going on, just can't pin down what!

I've copied the majority of my two posts into a GitHub issue

Here's the link if anyone wants to chime in w/ anything

https://github.com/tubearchivist/tubearchivist/issues/402

Link to comment

update incase anyone has similar issues later...

 

I'm pre-emptively calling this fixed because it's been going for over 2 hours now without locking up -- I certainly never got that yesterday when doing all my testing.

 

If anyone finds this in the future and has the same issue on unraid, then I added this path to resolve it, which prevents the download files from being put on your btrfs cache drive(s) and having to move them to the array -- this allows it to download straight to the array and then it will just move it to the other path when finished -- still on the same array and filesystem.

 

image.thumb.png.bf1916cecdc3cd01fd69b0e49ca53c4f.png

Link to comment
6 hours ago, N¿¿B said:

Has there been a fix to this that isn't reverting back to previous version? I'd like to be able to use the `latest` tag. This does work though.

 

Current redis does work, however the migration issue with corrupting the db hasn't been resolved that we're aware of.  This isn't an issue with TA itself & I've not seen a fix from redis.

 

What we've had to do to overcome is:

  1. Copy out any non-default settings from your settings page
  2. Update redis
  3. If the fault occurs, delete the TubeArchivist/redis/dump.rdb (path dependent on your installation settings)
  4. Restart redis
  5. Apply settings copied from step 1
  6. If using your YT cookie, you'll have to re-add it as well
Edited by Phuriousgeorge
Link to comment
  • 4 weeks later...

Hey all, pulling my hair out, and I don't have any left lol!

 

I've read the docs forward and backward and can't figure out the correct formatting.

 

Wondering if anyone can share a download format that would limit to 1080 and embed the thumbnail and subtitles. Given a codec choice I'd say 264 and container, mkv.

 

If this is at all possible that is.

 

 

I've tried this 

bestvideo[height<=1080] --remux-video mkv

 

 but I still get errors 

 

[2023-02-08 17:18:53,258: INFO/MainProcess] Task home.tasks.download_single[c4dbe92f-6c7a-4ba1-b2d5-3609fc5c1187] received
[2023-02-08 17:18:53,260: WARNING/ForkPoolWorker-16] Added to queue with priority: {'youtube_id': 'nMmvuNHHxnM', 'vid_type': 'videos'}
[2023-02-08 17:18:53,312: WARNING/ForkPoolWorker-16] Downloading type: VideoTypeEnum.VIDEOS
[2023-02-08 17:18:54,061: WARNING/ForkPoolWorker-16] ERROR: [youtube] nMmvuNHHxnM: Requested format is not available. Use --list-formats for a list of available formats
[2023-02-08 17:18:54,061: WARNING/ForkPoolWorker-16] nMmvuNHHxnM: failed to download.
[2023-02-08 17:18:54,151: INFO/ForkPoolWorker-16] Task home.tasks.download_single[c4dbe92f-6c7a-4ba1-b2d5-3609fc5c1187] succeeded in 0.8925375429680571s: None
{"error":{"root_cause":[{"type":"resource_not_found_exception","reason":"snapshot lifecycle policy or policies [ta_daily] not found, no policies are configured"}],"type":"resource_not_found_exception","reason":"snapshot lifecycle policy or policies [ta_daily] not found, no policies are configured"},"status":404}

 

Edited by aglyons
Link to comment
4 hours ago, aglyons said:

Hey all, pulling my hair out, and I don't have any left lol!

 

I've read the docs forward and backward and can't figure out the correct formatting.

 

Wondering if anyone can share a download format that would limit to 1080 and embed the thumbnail and subtitles. Given a codec choice I'd say 264 and container, mkv.

 

If this is at all possible that is.

 

 

You cannot choose the container.  Relevant section of the docs: https://github.com/tubearchivist/tubearchivist/wiki/Settings#download-format

 

I've got a pretty specific format preference, for 1080 it's:

(bestvideo[vcodec^=av01][height=1080][fps>30]/bestvideo[vcodec^=vp9.2][height=1080][fps>30]/bestvideo[vcodec^=vp9][height=1080][fps>30]/bestvideo[vcodec^=avc1][height=1080][fps>30]/bestvideo[height=1080][fps>30]/bestvideo[vcodec^=av01][height=1080]/bestvideo[vcodec^=vp9.2][height=1080]/bestvideo[vcodec^=vp9][height=1080]/bestvideo[vcodec^=avc1][height=1080]/bestvideo[height=1080]/bestvideo[vcodec^=av01][height=720][fps>30]/bestvideo[vcodec^=vp9.2][height=720][fps>30]/bestvideo[vcodec^=vp9][height=720][fps>30]/bestvideo[vcodec^=avc1][height=720][fps>30]/bestvideo[height=720][fps>30]/bestvideo[vcodec^=av01][height=720]/bestvideo[vcodec^=vp9.2][height=720]/bestvideo[vcodec^=vp9][height=720]/bestvideo[vcodec^=avc1][height=720]/bestvideo[height=720]/bestvideo[vcodec^=av01][height=480][fps>30]/bestvideo[vcodec^=vp9.2][height=480][fps>30]/bestvideo[vcodec^=vp9][height=480][fps>30]/bestvideo[vcodec^=avc1][height=480][fps>30]/bestvideo[height=480][fps>30]/bestvideo[vcodec^=av01][height=480]/bestvideo[vcodec^=vp9.2][height=480]/bestvideo[vcodec^=vp9][height=480]/bestvideo[vcodec^=avc1][height=480]/bestvideo[height=480]/bestvideo[vcodec^=av01][height=360][fps>30]/bestvideo[vcodec^=vp9.2][height=360][fps>30]/bestvideo[vcodec^=vp9][height=360][fps>30]/bestvideo[vcodec^=avc1][height=360][fps>30]/bestvideo[height=360][fps>30]/bestvideo[vcodec^=av01][height=360]/bestvideo[vcodec^=vp9.2][height=360]/bestvideo[vcodec^=vp9][height=360]/bestvideo[vcodec^=avc1][height=360]/bestvideo[height=360]/bestvideo[vcodec^=avc1][height=240][fps>30]/bestvideo[vcodec^=av01][height=240][fps>30]/bestvideo[vcodec^=vp9.2][height=240][fps>30]/bestvideo[vcodec^=vp9][height=240][fps>30]/bestvideo[height=240][fps>30]/bestvideo[vcodec^=avc1][height=240]/bestvideo[vcodec^=av01][height=240]/bestvideo[vcodec^=vp9.2][height=240]/bestvideo[vcodec^=vp9][height=240]/bestvideo[height=240]/bestvideo[vcodec^=avc1][height=144][fps>30]/bestvideo[vcodec^=av01][height=144][fps>30]/bestvideo[vcodec^=vp9.2][height=144][fps>30]/bestvideo[vcodec^=vp9][height=144][fps>30]/bestvideo[height=144][fps>30]/bestvideo[vcodec^=avc1][height=144]/bestvideo[vcodec^=av01][height=144]/bestvideo[vcodec^=vp9.2][height=144]/bestvideo[vcodec^=vp9][height=144]/bestvideo[height=144]/bestvideo[height<=?1080]/bestvideo)+(bestaudio[acodec^=opus]/bestaudio)/best

 

This sets my preferred codecs for 1080, then if that's not available, 720, and down all the way to give me whatever the #$&$^% you can

 

Embed metadata and thumbnail are completely separate settings:

image.png.9ba8b818b6f7504f7508bde9f9be8cac.png

  • Like 1
Link to comment
11 hours ago, Phuriousgeorge said:

 

You cannot choose the container.  Relevant section of the docs: https://github.com/tubearchivist/tubearchivist/wiki/Settings#download-format

 

I've got a pretty specific format preference, for 1080 it's:

(bestvideo[vcodec^=av01][height=1080][fps>30]/bestvideo[vcodec^=vp9.2][height=1080][fps>30]/bestvideo[vcodec^=vp9][height=1080][fps>30]/bestvideo[vcodec^=avc1][height=1080][fps>30]/bestvideo[height=1080][fps>30]/bestvideo[vcodec^=av01][height=1080]/bestvideo[vcodec^=vp9.2][height=1080]/bestvideo[vcodec^=vp9][height=1080]/bestvideo[vcodec^=avc1][height=1080]/bestvideo[height=1080]/bestvideo[vcodec^=av01][height=720][fps>30]/bestvideo[vcodec^=vp9.2][height=720][fps>30]/bestvideo[vcodec^=vp9][height=720][fps>30]/bestvideo[vcodec^=avc1][height=720][fps>30]/bestvideo[height=720][fps>30]/bestvideo[vcodec^=av01][height=720]/bestvideo[vcodec^=vp9.2][height=720]/bestvideo[vcodec^=vp9][height=720]/bestvideo[vcodec^=avc1][height=720]/bestvideo[height=720]/bestvideo[vcodec^=av01][height=480][fps>30]/bestvideo[vcodec^=vp9.2][height=480][fps>30]/bestvideo[vcodec^=vp9][height=480][fps>30]/bestvideo[vcodec^=avc1][height=480][fps>30]/bestvideo[height=480][fps>30]/bestvideo[vcodec^=av01][height=480]/bestvideo[vcodec^=vp9.2][height=480]/bestvideo[vcodec^=vp9][height=480]/bestvideo[vcodec^=avc1][height=480]/bestvideo[height=480]/bestvideo[vcodec^=av01][height=360][fps>30]/bestvideo[vcodec^=vp9.2][height=360][fps>30]/bestvideo[vcodec^=vp9][height=360][fps>30]/bestvideo[vcodec^=avc1][height=360][fps>30]/bestvideo[height=360][fps>30]/bestvideo[vcodec^=av01][height=360]/bestvideo[vcodec^=vp9.2][height=360]/bestvideo[vcodec^=vp9][height=360]/bestvideo[vcodec^=avc1][height=360]/bestvideo[height=360]/bestvideo[vcodec^=avc1][height=240][fps>30]/bestvideo[vcodec^=av01][height=240][fps>30]/bestvideo[vcodec^=vp9.2][height=240][fps>30]/bestvideo[vcodec^=vp9][height=240][fps>30]/bestvideo[height=240][fps>30]/bestvideo[vcodec^=avc1][height=240]/bestvideo[vcodec^=av01][height=240]/bestvideo[vcodec^=vp9.2][height=240]/bestvideo[vcodec^=vp9][height=240]/bestvideo[height=240]/bestvideo[vcodec^=avc1][height=144][fps>30]/bestvideo[vcodec^=av01][height=144][fps>30]/bestvideo[vcodec^=vp9.2][height=144][fps>30]/bestvideo[vcodec^=vp9][height=144][fps>30]/bestvideo[height=144][fps>30]/bestvideo[vcodec^=avc1][height=144]/bestvideo[vcodec^=av01][height=144]/bestvideo[vcodec^=vp9.2][height=144]/bestvideo[vcodec^=vp9][height=144]/bestvideo[height=144]/bestvideo[height<=?1080]/bestvideo)+(bestaudio[acodec^=opus]/bestaudio)/best

 

This sets my preferred codecs for 1080, then if that's not available, 720, and down all the way to give me whatever the #$&$^% you can

 

Embed metadata and thumbnail are completely separate settings:

image.png.9ba8b818b6f7504f7508bde9f9be8cac.png

 

 

 

HOLY CRAP!

 

Thanks so much for this! I doubt I would have had the time to figure out this entire config script.

 

This should be included as an example in the official docs as I think this covers what pretty much everyone would want. 

 

1080p is pretty much as high as I think anyone would want to go being that you're storing it locally. 4k videos can chew up space pretty quick.

Link to comment

Hey all, just getting started. Managed to get everything working but I had a thought and noticed something that made me go hmmmmm.

 

First, I see a few people talking about already having ES and REDIS already installed on their system, and because of that are having a hard time getting TA to talk to it. While I didn't have either installed prior, I had some challenges trying to figure out what needed to go where and what needed to be set to what.

 

I was curious, would it not be a good idea to create a docker image that includes the three components (TA, ES & REDIS) already installed in the one app? I get it that would be duplicating services in a way. But a dedicated ES and REDIS for TA might simplify setup and cut down on people running into problems. Just a thought.

 

The hmmmm I was referring to is if you select the "more info" link on the Unraid container for TubeArchivist-RedisJSON.

 

image.png.7d4ed725071e51b65d4d4e07e2809809.png

 

It brings you to the DockerHub page (https://hub.docker.com/r/redislabs/rejson/). That page states that "This docker image has been deprecated, please download the redis-stack docker.

 

Does this pose any concerns going forward?

 

Link to comment
  • 2 weeks later...
On 2/9/2023 at 10:38 AM, aglyons said:

It brings you to the DockerHub page (https://hub.docker.com/r/redislabs/rejson/). That page states that "This docker image has been deprecated, please download the redis-stack docker.

 

Does this pose any concerns going forward?

 

 

I just saw the following update posted yesterday to the TubeArchivist Discord cahnnel.

 

Quote

billy — Yesterday at 12:06 AM

Hello @everyone Has been some time since my last post here. v0.3.3 is out, update at your earliest convenience. This ships yt-dlp version 2023-02-17 released a few hours ago, where the amazing yt-dlp team fixed the Unable to extract uploader id error that probably everybody is experiencing by now.

 

Please read the release notes: https://github.com/tubearchivist/tubearchivist/releases/tag/v0.3.3, particularly:

- redislabs/rejson is deprecated by the redis team, use redis/redis-stack-server as a drop in replacement. Maintainers of installation instructions, please update your documentation to reflect that change.

- path.repo is now enforced at startup, if you didn’t already, make sure you set the environment variable path.repo=/usr/share/elasticsearch/data/snapshot to the ElasticSearch container and activate snapshots on the settings page.

 

This is also the first release while working on the list of high priorities here: https://github.com/tubearchivist/tubearchivist/issues/415, shipping a rewrite of the application startup functions with much better debug output and process handling. This list of blocking items will keep us busy for the foreseeable future, probably longer than the optimistic estimate there. Any help with an item on the list is very much welcome, splitting the workload will get us back to creating new awesome features for Tube Archivist sooner.

 

In any case, I’m glad to have you on board with this project, let’s keep these download queues rolling!

 

According to this we just change the redislabs/rejason to redis/redis-stack-server and it should be a drop-in replacement.

 

On startup I can confirm I see the following now with the change.

image.png.737b6c15a347ac6a2aed994678a12e05.png

 

So to confirm this is now my current setup.

 

image.thumb.png.42dedec217caea959026d87bc84b0571.png

 

 

  • Like 1
Link to comment

Thanks @CorneliousJD,

 

I rebuilt the container with the new image and I noticed something in the logs.

 

8:M 19 Feb 2023 16:43:48.852 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.

 

Did you see this as well? I wonder if this error is specifically for container use and could cause an issue or, if this issue is specifically for running REDIS on bare metal?

 

I also added the now required variable 'path.repo' to the containers config. I think I passed the path correctly. I didn't see any errors in any logs in either REDIS or ES. TA seems to work ok too.

 

EDIT: WOW, I tried to go though the bug report in the git and thought I was reading a different language lol. I'm not really sure if this issue was resolved or if it's still a problem or what!?!

Edited by aglyons
Link to comment
2 minutes ago, aglyons said:

Thanks @CorneliousJD,

 

I rebuilt the container with the new image and I noticed something in the logs.

 

8:M 19 Feb 2023 16:43:48.852 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.

 

Did you see this as well? I wonder if this error is specifically for container use and could cause an issue or, if this issue is specifically for running REDIS on bare metal?

 

I also added the now required variable 'path.repo' to the containers config. I think I passed the path correctly. I didn't see any errors in any logs in either REDIS or ES. TA seems to work ok too.

I honestly didn't look at the redis stack logs, I just let it do it's thing. 

 

The path.repo variable was already there for me in the ES container hidden behind the "show more" part. So no changes there for me, all I did was drop in the replacement redis container and let it ride. 

 

I have 128GB of RAM on my system so I doubt low memory would ever be an issue for me, but perhaps the maintainers of the template, or TA devs can shed more light on the error you're seeing.

Link to comment
On 2/19/2023 at 6:54 PM, aglyons said:

Thanks @CorneliousJD,

 

I rebuilt the container with the new image and I noticed something in the logs.

 

8:M 19 Feb 2023 16:43:48.852 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.

 

Did you see this as well? I wonder if this error is specifically for container use and could cause an issue or, if this issue is specifically for running REDIS on bare metal?

 

I also added the now required variable 'path.repo' to the containers config. I think I passed the path correctly. I didn't see any errors in any logs in either REDIS or ES. TA seems to work ok too.

 

EDIT: WOW, I tried to go though the bug report in the git and thought I was reading a different language lol. I'm not really sure if this issue was resolved or if it's still a problem or what!?!

Hi.

I also got this Warning but I didn't need to add the variable. As mentioned by @CorneliousJD it was already there.

Link to comment
7 minutes ago, mathomas3 said:

I tried to mimic the advice here... and blew up my entire setup... when I checked for updates for this docker it said there was "not available" 

 

in my panic... I forced an update... and blew up the whole thing... it wont start and on a fresh install it dies right away... 

 

suggestions?

Which container, as there are 3.

 

Did you replace the repo for the redisjson container?

 

Which of the 3 said it didn't have an update available? Forcing an update shouldn't break anything.

Link to comment
3 minutes ago, mathomas3 said:

I did the force refresh on the main tube archive docker... that was the one that said and update was "not available" which is odd 

 

I did change the repo on the redisjson container and that didnt help anything... thus I tried the force update on the main docker

 

A force update certainly shouldn't break anything - have you looked at logs - what do the logs of the main container say when it ends up stopping?

 

I think anyone here (myself included) will need some more info before we can really help.

 

Also, not that I suggest this yet, but if all else fails you could blow away the entire stack of 3 containers and start fresh and restore from a snapshot backup within TA itself?

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.