[[PLUGIN] SnapRAID on UNRAID]

Just wanted to say thank you @Rysz for putting this together, it's been on my wish list for awhile now as an alternative setup!  Time to put my concept to the test and though I'm horrible at it, hopefully I'll report back.

[LUKS Encryption Header Backup]

Actually had my header info lost on a drive not long after migrating data due to a power loss I believe and lost about half the data on the drive.  This would be very much appreciated.

[[Support] Crocs - Tube Archivist]

4 hours ago, aglyons said:

Hey all, pulling my hair out, and I don't have any left lol!

 

I've read the docs forward and backward and can't figure out the correct formatting.

 

Wondering if anyone can share a download format that would limit to 1080 and embed the thumbnail and subtitles. Given a codec choice I'd say 264 and container, mkv.

 

If this is at all possible that is.

 

 

You cannot choose the container.  Relevant section of the docs: https://github.com/tubearchivist/tubearchivist/wiki/Settings#download-format

 

I've got a pretty specific format preference, for 1080 it's:

(bestvideo[vcodec^=av01][height=1080][fps>30]/bestvideo[vcodec^=vp9.2][height=1080][fps>30]/bestvideo[vcodec^=vp9][height=1080][fps>30]/bestvideo[vcodec^=avc1][height=1080][fps>30]/bestvideo[height=1080][fps>30]/bestvideo[vcodec^=av01][height=1080]/bestvideo[vcodec^=vp9.2][height=1080]/bestvideo[vcodec^=vp9][height=1080]/bestvideo[vcodec^=avc1][height=1080]/bestvideo[height=1080]/bestvideo[vcodec^=av01][height=720][fps>30]/bestvideo[vcodec^=vp9.2][height=720][fps>30]/bestvideo[vcodec^=vp9][height=720][fps>30]/bestvideo[vcodec^=avc1][height=720][fps>30]/bestvideo[height=720][fps>30]/bestvideo[vcodec^=av01][height=720]/bestvideo[vcodec^=vp9.2][height=720]/bestvideo[vcodec^=vp9][height=720]/bestvideo[vcodec^=avc1][height=720]/bestvideo[height=720]/bestvideo[vcodec^=av01][height=480][fps>30]/bestvideo[vcodec^=vp9.2][height=480][fps>30]/bestvideo[vcodec^=vp9][height=480][fps>30]/bestvideo[vcodec^=avc1][height=480][fps>30]/bestvideo[height=480][fps>30]/bestvideo[vcodec^=av01][height=480]/bestvideo[vcodec^=vp9.2][height=480]/bestvideo[vcodec^=vp9][height=480]/bestvideo[vcodec^=avc1][height=480]/bestvideo[height=480]/bestvideo[vcodec^=av01][height=360][fps>30]/bestvideo[vcodec^=vp9.2][height=360][fps>30]/bestvideo[vcodec^=vp9][height=360][fps>30]/bestvideo[vcodec^=avc1][height=360][fps>30]/bestvideo[height=360][fps>30]/bestvideo[vcodec^=av01][height=360]/bestvideo[vcodec^=vp9.2][height=360]/bestvideo[vcodec^=vp9][height=360]/bestvideo[vcodec^=avc1][height=360]/bestvideo[height=360]/bestvideo[vcodec^=avc1][height=240][fps>30]/bestvideo[vcodec^=av01][height=240][fps>30]/bestvideo[vcodec^=vp9.2][height=240][fps>30]/bestvideo[vcodec^=vp9][height=240][fps>30]/bestvideo[height=240][fps>30]/bestvideo[vcodec^=avc1][height=240]/bestvideo[vcodec^=av01][height=240]/bestvideo[vcodec^=vp9.2][height=240]/bestvideo[vcodec^=vp9][height=240]/bestvideo[height=240]/bestvideo[vcodec^=avc1][height=144][fps>30]/bestvideo[vcodec^=av01][height=144][fps>30]/bestvideo[vcodec^=vp9.2][height=144][fps>30]/bestvideo[vcodec^=vp9][height=144][fps>30]/bestvideo[height=144][fps>30]/bestvideo[vcodec^=avc1][height=144]/bestvideo[vcodec^=av01][height=144]/bestvideo[vcodec^=vp9.2][height=144]/bestvideo[vcodec^=vp9][height=144]/bestvideo[height=144]/bestvideo[height<=?1080]/bestvideo)+(bestaudio[acodec^=opus]/bestaudio)/best

 

This sets my preferred codecs for 1080, then if that's not available, 720, and down all the way to give me whatever the #$&$^% you can

 

Embed metadata and thumbnail are completely separate settings:

[[Support] Crocs - Tube Archivist]

6 hours ago, N¿¿B said:

Has there been a fix to this that isn't reverting back to previous version? I'd like to be able to use the `latest` tag. This does work though.

 

Current redis does work, however the migration issue with corrupting the db hasn't been resolved that we're aware of.  This isn't an issue with TA itself & I've not seen a fix from redis.

 

What we've had to do to overcome is:

1. Copy out any non-default settings from your settings page
2. Update redis
3. If the fault occurs, delete the TubeArchivist/redis/dump.rdb (path dependent on your installation settings)
4. Restart redis
5. Apply settings copied from step 1
6. If using your YT cookie, you'll have to re-add it as well

[[Support] Crocs - Tube Archivist]

8 minutes ago, CorneliousJD said:

Upon investigating a little more...

 

This is very odd indeed.  I've been using TA extensively with just under 800k videos in my archive.

 

I would personally suggest using Discord or GitHub for support, as the dev of the app will respond there and it's just us users monitoring here

[[Support] Crocs - Tube Archivist]

53 minutes ago, Emilia said:

I'm trying to install tube archivist, not update from a previous version. I'm also just using all the default settings:

* also tried reinstalling it but that didn't fix anything

 

21 minutes ago, msherman83 said:

Hello,

Same issue as the others above on a fresh install. It seems for some reason you need to change the ./es folder to 777 to get the es container to write to it.

 

Access the Unraid CLI and enter:

chmod -R 777 /mnt/user/appdata/TubeArchivist/es

 

I noticed a large java file permissions error in the ES container logs the first few times i tried to start it. It would die right away, so you couldn't login to it.

 

 

 

Apparently this is a known potential issue that can affect new or updated installations.  The fix is documented under "Common Errors" on the projects' GitHub here: https://github.com/tubearchivist/tubearchivist#permissions-for-elasticsearch

[[Support] Crocs - Tube Archivist]

4 minutes ago, msherman83 said:

Hello,

Same issue as the others above on a fresh install. It seems for some reason you need to change the ./es folder to 777 to get the es container to write to it.

 

Access the Unraid CLI and enter:

chmod -R 777 /mnt/user/appdata/TubeArchivist/es

 

I noticed a large java file permissions error in the ES container logs the first few times i tried to start it. It would die right away, so you couldn't login to it.

 

 

Looking into this... might be a difference between updating vs starting fresh then.  Thanks for the reports, we're looking into it.

[[Support] Crocs - Tube Archivist]

Honestly can't fathom how adding this variable is "causing" permission errors within the container.  Several of us have updated with no issue so the assumption is user-sabotage by finagling inside the container.

 

My suggestion would be to destroy the container and re-create.  As long as your volumes are set up correctly (as with most docker deployments) all persistent data will be unaffected.

[[Support] Crocs - Tube Archivist]

2 hours ago, thx4nothing said:

Just to confirm, I only need the TubeArchivist, TubeArchivist-RedisJSON and TubeArchivist-ES containers, right? Do I need to install something else?

That's all you need.

 

How about your ElasticSearch Data field?  Is it nested inside the main TubeArchivist data folder?  Are you sure you've got sufficient space and write permission?

 

 

Otherwise uncertain how you can be getting the path.repo error with that both those set....

[[Support] Crocs - Tube Archivist]

19 hours ago, thx4nothing said:

I can't get ES to start. It can't find (access?) the path.repo. I didn't change it from the default as it said in the installation guide. It says to check the log at /usr/share/elasticsearch/logs/docker-cluster.log, but the file (and folders) are not there. I tried creating the elasticsearch folder and setting the permissions (chown 1000:0 -R) manually, but still the same error. I'm on unRAID Version: 6.10.3 and installed TubeArchivist for the first time.

 

Docker log:

@timestamp=2022-12-03T07:10:09.032Zecs.version=1.2.0elasticsearch.cluster.name=docker-clusterelasticsearch.node.name=3092b8c09c8aerror.message=Unable to access 'path.repo' (/usr/share/elasticsearch/data/snapshot)error.stack_trace=java.lang.IllegalStateException: Unable to access 'path.repo' (/usr/share/elasticsearch/data/snapshot) at [email protected]/org.elasticsearch.bootstrap.FilePermissionUtils.addDirectoryPath(FilePermissionUtils.java:66) at [email protected]/org.elasticsearch.bootstrap.Security.addFilePermissions(Security.java:252) at [email protected]/org.elasticsearch.bootstrap.Security.createPermissions(Security.java:178) at [email protected]/org.elasticsearch.bootstrap.Security.configure(Security.java:125) at [email protected]/org.elasticsearch.bootstrap.Elasticsearch.initPhase2(Elasticsearch.java:186) at [email protected]/org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:66) Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/snapshot at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) at java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:397) at java.base/java.nio.file.Files.createDirectory(Files.java:700) at java.base/java.nio.file.Files.createAndCheckIsDirectory(Files.java:807) at java.base/java.nio.file.Files.createDirectories(Files.java:793) at [email protected]/org.elasticsearch.bootstrap.Security.ensureDirectoryExists(Security.java:330) at [email protected]/org.elasticsearch.bootstrap.FilePermissionUtils.addDirectoryPath(FilePermissionUtils.java:64) ... 5 more error.type=java.lang.IllegalStateExceptionevent.dataset=elasticsearch.serverlog.level=ERRORlog.logger=org.elasticsearch.bootstrap.Elasticsearchmessage=fatal exception while booting Elasticsearchprocess.thread.name=mainservice.name=ES_ECS

03/12/2022 16:10:09 ERROR: Elasticsearch did not exit normally - check the logs at /usr/share/elasticsearch/logs/docker-cluster.log
03/12/2022 16:10:09 ERROR: Elasticsearch exited unexpectedly

 

Can you confirm you have a Variable setup like the following on your ElasticSearch container?  The template update got out a little slow for the update:

 

[[Support] Linuxserver.io - Calibre]

Just a note to anyone noticing their docker image size growing leaps and bounds.  Found this lovely thread featuring the Calibre developer and his flavor-full short-sightedness regarding an issue which is almost inevitable with Docker or image installs https://www.mobileread.com/forums/showthread.php?t=278484

 

I had 335GB in my *\appdata\calibre\.local\share\Trash directory.  Created a simple userscript to clean it out daily with a simple rm -rf

[Array disks unmountable after reboot]

I have no parity in this system at the moment.

[Array disks unmountable after reboot]

root@PGVault:~# fdisk -l /dev/sdx
Disk /dev/sdx: 9.1 TiB, 10000831348736 bytes, 19532873728 sectors
Disk model: WDC WD100EMAZ-00
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 8881A93C-7815-4B51-BB9C-B79A125EB4A8

 

I didn't include the other disk because it consisted of easily-replaceable "Linux ISOs", so I've been experimenting with its headers...and failing.

[Array disks unmountable after reboot]

Looking for assistance in recovering two of my data disks (all disks are xfs-encrypted) that have suddenly become "unmountable".  I was moving a bunch of small files to the array in Krusader that caused the user mount to go away.  I had seen this issue before, as this is a new server setup and I've been mass transferring files, so I made sure everything was stopped and restarted the machine.  After entering the decryption key and starting the array, two array disks with content are showing as unmountable and persists through subsequent restarts.

 

From looking at the diagnostics (attached), it appears it's related to:

 

PGVault emhttpd: error: ckmbr, 2263: No such file or directory (2): open: /dev/sdj1
...
PGVault emhttpd: error: ckmbr, 2263: No such file or directory (2): open: /dev/sdx1

 

Now into my speculations:

Somehow it appears these disks have lost their partition table?  This is my 1st time dealing with issues on encrypted disks, but I've had to conduct a similar rescue in the past.  Is it as simple as re-applying the partition table and viola, the data is accessible again?  If so, anyone have a good reference for doing so?  It's been awhile and I've got multiples of the same disk type to reference partition layout of.

 

Unraid v6.9.2

 

Spoiler

Unfortunately this was the last batch of information to move to the array before adding my parity disks (needed the raw write speed so I didn't spend 6 months transferring 205TB), so recovery via parity calculations isn't an option.

 

pgvault-diagnostics-20211116-1927.zip

[My Servers Early Access Plugin]

27 minutes ago, ljm42 said:

Thank you! this was super helpful, I see the issue.

 

If you delete the file from the flash are you back in business?

I'm still in the throws of massive transfers I don't want to bork up. I'll have to attempt again once things slow down.

[My Servers Early Access Plugin]

1 hour ago, ljm42 said:

If someone with the problem wouldn't mind PMing me a copy of the file before you delete it, it might give us an idea of how the file is being corrupted. Don't post it here though, it has sensitive info.

Sent mine

[My Servers Early Access Plugin]

14 minutes ago, OmgImAlexis said:

Just to confirm, are you seeing this issue on every page of the WebUI or just the my servers settings page?

Can you also check to see if your /boot/config/plugins/dynamix.my.servers/myservers.cfg looks similar to this?

[remote]
apikey="___REDACTED___"
regWizTime="___REDACTED___"
email="___REDACTED___"
username="OmgImAlexis"
avatar="https://ipsassets.unraid.net/uploads/___REDACTED___"
[upc]
apikey="___REDACTED___"

The config file looks exactly the same except an extra return between avatar and [upc]

 

Regarding pages, I was unable to navigate to any other pages as the page was refreshing constantly and tossing me back to the "Main" tab any time I attempted navigation.  Had to delete the .plg file to recover UI accessibility.  Everything's back to normal without the plugin running.

[My Servers Early Access Plugin]

Just updated to the new update of My Servers and now my page is refreshing like a madman. So much so that I cannot click anything before it spins through another refresh.  Attached the only thing I was able to grab as it flew by in the browser console.  Now have to wait for a huge file to finish transferring so I can attempt to stop the madness.