live4ever
-
Posts
19 -
Joined
-
Last visited
Content Type
Profiles
Forums
Downloads
Store
Gallery
Bug Reports
Documentation
Landing
Posts posted by live4ever
-
-
On 9/27/2022 at 5:51 PM, DZMM said:
Has anybody had any problems running the scripts on 6.11? Does the mergerfs install still work without NerdPack?
So after I upgraded my main uploading unRAID server from 6.9 series to 6.11 I noticed how permissions could mess up using the scripts from GitHub.
My mount script is a bit different - I create a “backup” folder acting as sort of Dropbox for uploads to gdrive at:
/mnt/user/local/gdrive_vfs/backupThis folder would get deleted after rclone_upload completes and then get recreated with rclone_mount (with root:root).
In DZMM rclone_mount the MountFolders bit:MountFolders=\{"downloads/complete,downloads/intermediate,downloads/seeds,movies,tv"\} # comma separated list of folders to create within the mount
In DZMM rclone_upload:Command1="--exclude downloads/**"
So the paths from MountFolders:/mnt/user/local/gdrive_vfs/movies /mnt/user/local/gdrive_vfs/tv
will get deleted (after uploading) - then the rclone_mount script will create them again with root:root (and dockers trying to use them will give permission errors.
Is there a way for the rclone_mount script to create the empty MountFolders with nobody:user permissions? -
The above was on unRAID 6.10.3 - another server I have on 6.9.3 doesn't have "dig" (not in nerd tools either) so the GES script won't run (in case anyone tries on the 6.9.x release).
*********************************************** ******** Google Endpoint Scanner (GES) ******** *********************************************** [GES] [INFO] The application started on Thu Jul 7 10:22:21 EDT 2022. [GES] [WARNING] Hosts backup file not found. Backing it up. [GES] [ERROR] The command 'dig' is not installed or not reachable in this user's $PATH. *********************************************** * Finished Google Endpoint Scanner (GES) * Message: Install dig or make sure its executable is reachable, then try again. *********************************************** Script Finished Jul 07, 2022 10:22.21 -
Some test from Canada this morning:
******** Google Endpoint Scanner (GES) ******** *********************************************** [GES] [INFO] The application started on Thu Jul 7 10:02:04 EDT 2022. [GES] [WARNING] Hosts backup file not found. Backing it up. [GES] [WARNING] Did not find a permanent blacklist at /root/blacklisted_google_ips. Creating a new one. [GES] [INFO] Please wait. Downloading the test file from 142.251.41.42... [GES] [INFO] Parsing connection with 142.251.41.42. [GES] [INFO] 1.538 MiB/s. Below criterion endpoint. Blacklisting IP '142.251.41.42'. [GES] [INFO] Please wait. Downloading the test file from 142.251.32.74... [GES] [INFO] Parsing connection with 142.251.32.74. [GES] [INFO] 8.334 MiB/s. Above criterion endpoint. Whitelisting IP '142.251.32.74'. [GES] [INFO] Please wait. Downloading the test file from 172.217.1.10... [GES] [INFO] Parsing connection with 172.217.1.10. [GES] [INFO] 1.564 MiB/s. Below criterion endpoint. Blacklisting IP '172.217.1.10'. [GES] [INFO] Please wait. Downloading the test file from 172.217.165.10... [GES] [INFO] Parsing connection with 172.217.165.10. [GES] [INFO] 1.545 MiB/s. Below criterion endpoint. Blacklisting IP '172.217.165.10'. [GES] [INFO] Please wait. Downloading the test file from 142.251.41.74... [GES] [INFO] Parsing connection with 142.251.41.74. [GES] [INFO] 10.32 MiB/s. Above criterion endpoint. Whitelisting IP '142.251.41.74'. [GES] [INFO] Please wait. Downloading the test file from 142.251.33.170... [GES] [INFO] Parsing connection with 142.251.33.170. [GES] [INFO] 1.566 MiB/s. Below criterion endpoint. Blacklisting IP '142.251.33.170'. [GES] [INFO] The fastest IP is 142.251.41.74. Putting into the hosts file. *********************************************** * Finished Google Endpoint Scanner (GES) * Message: Reached EOF without errors *********************************************** Script Finished Jul 07, 2022 10:04.31 Full logs for this script are available at /tmp/user.scripts/tmpScripts/Gdrive_api/log.txt Script Starting Jul 07, 2022 10:11.25 Full logs for this script are available at /tmp/user.scripts/tmpScripts/Gdrive_api/log.txt *********************************************** ******** Google Endpoint Scanner (GES) ******** *********************************************** [GES] [INFO] The application started on Thu Jul 7 10:11:25 EDT 2022. [GES] [INFO] Hosts backup file found. Restoring it. [GES] [INFO] Found permanent blacklist. Parsing it. [GES] [INFO] Please wait. Downloading the test file from 142.251.32.74... [GES] [INFO] Parsing connection with 142.251.32.74. [GES] [INFO] 33.19 MiB/s. Above criterion endpoint. Whitelisting IP '142.251.32.74'. [GES] [INFO] Please wait. Downloading the test file from 142.251.41.74... [GES] [INFO] Parsing connection with 142.251.41.74. [GES] [INFO] 39.40 MiB/s. Above criterion endpoint. Whitelisting IP '142.251.41.74'. [GES] [INFO] The fastest IP is 142.251.41.74. Putting into the hosts file. *********************************************** * Finished Google Endpoint Scanner (GES) * Message: Reached EOF without errors *********************************************** Script Finished Jul 07, 2022 10:11.39So initially it was the 50MB file then the 200MB file (at 10:11.25) - been having some buffering the past couple weeks.
-
On 5/20/2022 at 11:39 AM, jermdawg2017 said:
@DarthAlduin@EchoCharlieDelta Any detailed fixed for this? I'm running into the same issue! Thanks
Scroll down to the bottom of page “On a different network” here:
-
I’ve got two unRAIDs:
6.9 b22 47d uptime
6.9 b35 33d uptime
That have been still working fine here.
-
8 hours ago, M1kep_ said:
If the mount wasn't killed nicely there is most likely still the mount_running file in place. I'd also confirm with a ps to grep to see if the mount is indeed not running properly.
ps -aux | grep rcloneIf the mount scripts really isn't running, then you should run the rclone_unmount script as that will cleanup the necessary lock files. The deletion commands used by the script are:
find /mnt/user/appdata/other/rclone/remotes -name dockers_started* -delete find /mnt/user/appdata/other/rclone/remotes -name mount_running* -delete find /mnt/user/appdata/other/rclone/remotes -name upload_running* -delete@M1kep_ Thanks, the command:
root@Tower:~# find /mnt/user/appdata/other/rclone/remotes -name mount_running* /mnt/user/appdata/other/rclone/remotes/gdrive_media_vfs/mount_running find /mnt/user/appdata/other/rclone/remotes -name mount_running* -delete
allowed the rclone_mount script to start and download/install the mergerfs software.
-
@DZMM After a server reboot I can’t seem to get the the rclone_mount script (same one in my last post) to work all I’m seeing is:
————-
Script Starting Nov 14, 2020 23:51.50
Full logs for this script are available at /tmp/user.scripts/tmpScripts/rclone_mount/log.txt
14.11.2020 23:51:50 INFO: Creating local folders.
14.11.2020 23:51:50 INFO: *** Starting mount of remote gdrive_media_vfs
14.11.2020 23:51:50 INFO: Checking if this script is already running.
14.11.2020 23:51:50 INFO: Exiting script as already running.
Script Finished Nov 14, 2020 23:51.50Full logs for this script are available at /tmp/user.scripts/tmpScripts/rclone_mount/log.txt
_______
Thanks -
18 hours ago, DZMM said:
Nope, I added this to the script to try and help first-timers.
Do you do this before or after the mount script? It's a bad idea to do b4 the script runs, as you might get mounting issues - "mountpoint isn't empty" errors. Once rclone and mergerfs are mounted, it's 100% safe to 'create' folders in mergerfs (in reality, the folder is created in /local until uploaded to gdrive) and this is what you should do. That's the whole point - radarr/sonarr/manual rips etc being added to mergerfs that are accessible to plex regardless of what stage they are at - still residing locally or moved to gdrive.
@DZMMI do it after the mount script is up and running - and I did another test last night created two directories with the following paths:
/mnt/user/mount_mergerfs/gdrive_media_vfs/test/video.mkv /mnt/user/mount_mergerfs/gdrive_media_vfs/test2/
The directory ../test/video.mkv (with a file within it) was there in this morning while the ../test2/ directory (without any sub-folders or files) was deleted/removed. Not a big deal really though, was just a little concerned that (empty) folders I created in mergerfs were disappearing.
Also, I noticed that the mount script created shares/folders only on /mnt/disk14 and /mnt/disk15 (the least full disks of my array) - can I manually create the:
/mnt/local/gdrive_media_vfs
share on disks1-13? Or how do I get the mergerfs share to appear on all disks (to prevent moving files between disks before upload)?
Thanks again
-
On 10/23/2020 at 11:48 AM, DZMM said:
@live4ever all looks ok. Look in /mnt/user/mount_rclone and /mnt/user/local and you should see the source of the weird files - maybe you did a duff upload somewhere.
Either way - if you don't need them (unlikely), just delete and they should go away.
@DZMM had to reboot unRAID server to get it to clear up - unmounting with fusermount wasn't enough.
Quick question, does the mount script have to be used to create the subfolders within the mergerfs folder?
For example my mount script does:
MountFolders=\{"downloads/complete,downloads/incomplete,backup"\}and if I manually create a folder like /mnt/user/mount_mergerfs/gdrive_media_vfs/test/video.mkv
the /test/video.mkv disappears (I guess when the mount script runs after the next 10min?)
Thanks again
-
Using rclone 1.53.1
Here's the rclone_mount script:
#!/bin/bash ###################### #### Mount Script #### ###################### ### Version 0.96.8 ### ###################### ####### EDIT ONLY THESE SETTINGS ####### # INSTRUCTIONS # 1. Change the name of the rclone remote and shares to match your setup # 2. NOTE: enter RcloneRemoteName WITHOUT ':' # 3. Optional: include custom command and bind mount settings # 4. Optional: include extra folders in mergerfs mount # REQUIRED SETTINGS RcloneRemoteName="gdrive_media_vfs" RcloneMountShare="/mnt/user/mount_rclone" LocalFilesShare="/mnt/user/local" MergerfsMountShare="/mnt/user/mount_mergerfs" DockerStart="radarr sonarr" MountFolders=\{"downloads/complete,downloads/incomplete,backup"\} # OPTIONAL SETTINGS # Add extra paths to mergerfs mount in addition to LocalFilesShare LocalFilesShare2="ignore" # without trailing slash e.g. /mnt/user/other__remote_mount/or_other_local_folder. Enter 'ignore' to disable LocalFilesShare3="" LocalFilesShare4="" LocalFilesShare5="" LocalFilesShare6="" # Add extra commands or filters Command1="--rc" Command2="" Command3="" Command4="" Command5="" Command6="" Command7="" Command8="" CreateBindMount="N" # Y/N. Choose whether to bind traffic to a particular network adapter RCloneMountIP="192.168.1.252" # My unraid IP is 172.30.12.2 so I create another similar IP address NetworkAdapter="eth0" # choose your network adapter. eth0 recommended VirtualIPNumber="2" # creates eth0:x e.g. eth0:1. I create a unique virtual IP addresses for each mount & upload so I can monitor and traffic shape for each of them ####### END SETTINGS ####### ############################################################################### ##### DO NOT EDIT ANYTHING BELOW UNLESS YOU KNOW WHAT YOU ARE DOING ####### ############################################################################### ####### Preparing mount location variables ####### LocalFilesLocation="$LocalFilesShare/$RcloneRemoteName" RcloneMountLocation="$RcloneMountShare/$RcloneRemoteName" MergerFSMountLocation="$MergerfsMountShare/$RcloneRemoteName" ####### create directories for rclone mount and mergerfs mounts ####### mkdir -p /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName #for script files mkdir -p /mnt/user0/mount_rclone/cache/$RcloneRemoteName #for cache files if [[ $LocalFileShare == 'ignore' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Not creating local folders as requested." else echo "$(date "+%d.%m.%Y %T") INFO: Creating local folders." eval mkdir -p $LocalFilesLocation/"$MountFolders" fi mkdir -p $RcloneMountLocation mkdir -p $MergerFSMountLocation ####### Check if script is already running ####### echo "$(date "+%d.%m.%Y %T") INFO: *** Starting mount of remote ${RcloneRemoteName}" echo "$(date "+%d.%m.%Y %T") INFO: Checking if this script is already running." if [[ -f "/mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Exiting script as already running." exit else echo "$(date "+%d.%m.%Y %T") INFO: Script not running - proceeding." touch /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running fi ####### Create Rclone Mount ####### # Check If Rclone Mount Already Created if [[ -f "$RcloneMountLocation/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Success ${RcloneRemoteName} remote is already mounted." else echo "$(date "+%d.%m.%Y %T") INFO: Mount not running. Will now mount ${RcloneRemoteName} remote." # Creating mountcheck file in case it doesn't already exist echo "$(date "+%d.%m.%Y %T") INFO: Recreating mountcheck file for ${RcloneRemoteName} remote." touch mountcheck rclone copy mountcheck $RcloneRemoteName: -vv --no-traverse # Check bind option if [[ $CreateBindMount == 'Y' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: *** Checking if IP address ${RCloneMountIP} already created for remote ${RcloneRemoteName}" ping -q -c2 $RCloneMountIP > /dev/null # -q quiet, -c number of pings to perform if [ $? -eq 0 ]; then # ping returns exit status 0 if successful echo "$(date "+%d.%m.%Y %T") INFO: *** IP address ${RCloneMountIP} already created for remote ${RcloneRemoteName}" else echo "$(date "+%d.%m.%Y %T") INFO: *** Creating IP address ${RCloneMountIP} for remote ${RcloneRemoteName}" ip addr add $RCloneMountIP/24 dev $NetworkAdapter label $NetworkAdapter:$VirtualIPNumber fi echo "$(date "+%d.%m.%Y %T") INFO: *** Created bind mount ${RCloneMountIP} for remote ${RcloneRemoteName}" else RCloneMountIP="" echo "$(date "+%d.%m.%Y %T") INFO: *** Creating mount for remote ${RcloneRemoteName}" fi # create rclone mount rclone mount \ --allow-other \ --dir-cache-time 720h \ --log-level INFO \ --poll-interval 15s \ --cache-dir=/mnt/user0/mount_rclone/cache/$RcloneRemoteName \ --vfs-cache-mode full \ --vfs-cache-max-size 100G \ --vfs-cache-max-age 336h \ --bind=$RCloneMountIP \ $RcloneRemoteName: $RcloneMountLocation & # Check if Mount Successful echo "$(date "+%d.%m.%Y %T") INFO: sleeping for 5 seconds" # slight pause to give mount time to finalise sleep 10 echo "$(date "+%d.%m.%Y %T") INFO: continuing..." if [[ -f "$RcloneMountLocation/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Successful mount of ${RcloneRemoteName} mount." else echo "$(date "+%d.%m.%Y %T") CRITICAL: ${RcloneRemoteName} mount failed - please check for problems." docker stop $DockerStart find /mnt/user/appdata/other/rclone/remotes -name mount_running* -delete rm /mnt/user/appdata/other/scripts/running/fast_check exit fi fi ####### Start MergerFS Mount ####### if [[ $MergerfsMountShare == 'ignore' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Not creating mergerfs mount as requested." else if [[ -f "$MergerFSMountLocation/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Check successful, ${RcloneRemoteName} mergerfs mount in place." else # check if mergerfs already installed if [[ -f "/bin/mergerfs" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Mergerfs already installed, proceeding to create mergerfs mount" else # Build mergerfs binary echo "$(date "+%d.%m.%Y %T") INFO: Mergerfs not installed - installing now." mkdir -p /mnt/user/appdata/other/rclone/mergerfs docker run -v /mnt/user/appdata/other/rclone/mergerfs:/build --rm trapexit/mergerfs-static-build mv /mnt/user/appdata/other/rclone/mergerfs/mergerfs /bin # check if mergerfs install successful echo "$(date "+%d.%m.%Y %T") INFO: *sleeping for 5 seconds" sleep 10 if [[ -f "/bin/mergerfs" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Mergerfs installed successfully, proceeding to create mergerfs mount." else echo "$(date "+%d.%m.%Y %T") ERROR: Mergerfs not installed successfully. Please check for errors. Exiting." docker stop $DockerStart find /mnt/user/appdata/other/rclone/remotes -name mount_running* -delete rm /mnt/user/appdata/other/scripts/running/fast_check exit fi fi # Create mergerfs mount echo "$(date "+%d.%m.%Y %T") INFO: Creating ${RcloneRemoteName} mergerfs mount." # Extra Mergerfs folders if [[ $LocalFilesShare2 != 'ignore' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare2} to ${RcloneRemoteName} mergerfs mount." LocalFilesShare2=":$LocalFilesShare2" else LocalFilesShare2="" fi if [[ $LocalFilesShare3 != 'ignore' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare3} to ${RcloneRemoteName} mergerfs mount." LocalFilesShare3=":$LocalFilesShare3" else LocalFilesShare3="" fi if [[ $LocalFilesShare4 != 'ignore' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare4} to ${RcloneRemoteName} mergerfs mount." LocalFilesShare4=":$LocalFilesShare4" else LocalFilesShare4="" fi if [[ $LocalFilesShare5 != 'ignore' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare5} to ${RcloneRemoteName} mergerfs mount." LocalFilesShare5=":$LocalFilesShare5" else LocalFilesShare5="" fi if [[ $LocalFilesShare6 != 'ignore' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare6} to ${RcloneRemoteName} mergerfs mount." LocalFilesShare6=":$LocalFilesShare6" else LocalFilesShare6="" fi # mergerfs mount command mergerfs $LocalFilesLocation:$RcloneMountLocation$LocalFilesShare2$LocalFilesShare3$LocalFilesShare4$LocalFilesShare5$LocalFilesShare6 $MergerFSMountLocation -o rw,async_read=false,use_ino,allow_other,func.getattr=newest,category.action=all,category.create=ff,cache.files=partial,dropcacheonclose=true # check if mergerfs mount successful echo "$(date "+%d.%m.%Y %T") INFO: Checking if ${RcloneRemoteName} mergerfs mount created." if [[ -f "$MergerFSMountLocation/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Check successful, ${RcloneRemoteName} mergerfs mount created." else echo "$(date "+%d.%m.%Y %T") CRITICAL: ${RcloneRemoteName} mergerfs mount failed." docker stop $DockerStart find /mnt/user/appdata/other/rclone/remotes -name mount_running* -delete rm /mnt/user/appdata/other/scripts/running/fast_check exit fi fi fi ####### Starting Dockers That Need Mergerfs Mount To Work Properly ####### echo "$(date "+%d.%m.%Y %T") INFO: Starting dockers." docker start $DockerStart rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running echo "$(date "+%d.%m.%Y %T") INFO: ${RcloneRemoteName} Script complete" exitHere is rclone.conf
[gdrive] type = drive client_id = xxxx client_secret = xxxx scope = drive token = {"access_token":"xxxxx","token_type":"Bearer","refresh_token":"xxxx","expiry":"2020-10-23T11:50:00.72593565-04:00"} [gdrive_media_vfs] type = crypt remote = gdrive:crypt filename_encryption = standard directory_name_encryption = true password = xxxx password2 = xxxx -
Just set this up using the new mount script (posted a few pages back by @DZMM) seems to working and uploading fine.
Created two folders (via mount script backup and downloads) in gdrive_media_vfs crypt remote, and question is should I be seeing all these other folders in /mnt/user/mount_mergerfs/gdrive_media_vfs (attached images)?
Thanks
-
I can't seem to get the ADDITIONAL_PORTS variable to work.
Tried with Telly port 6077 and ChannelsDVR port 8089
Neither of these are accessible at unRAID.lan.ip:6077 or 8089
-
On 11/7/2018 at 2:39 PM, saarg said:
I'll check the build log if it complains about it and add if necessary.
Any updates to this? Still getting:
docker exec -it -u abc tvheadend /usr/bin/tv_grab_zz_sdjson_sqlite --configure Can't locate LWP/UserAgent/Determined.pm in @INC (you may need to install the LWP::UserAgent::Determined module) (@INC contains: /usr/local/lib/perl5/site_perl /usr/local/share/perl5/site_perl /usr/lib/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib/perl5/core_perl /usr/share/perl5/core_perl) at /usr/bin/tv_grab_zz_sdjson_sqlite line 4927. BEGIN failed--compilation aborted at /usr/bin/tv_grab_zz_sdjson_sqlite line 4927.Using the newest update from the weekend (4.2.7-34~g036b9cbab).
Thanks again.
-
Is anyone using the sqlite SchedulesDirect grabber?
Linux 4.14.49-unRAID. Last login: Mon Nov 5 07:26:05 -0500 2018 on /dev/pts/3. root@Tower:~# docker exec -it -u abc tvheadend /usr/bin/tv_grab_zz_sdjson_sqlite --configure Can't locate LWP/UserAgent/Determined.pm in @INC (you may need to install the LWP::UserAgent::Determined module) (@INC contains: /usr/local/lib/perl5/site_perl /usr/local/share/perl5/site_perl /usr/lib/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib/perl5/core_perl /usr/share/perl5/core_perl) at /usr/bin/tv_grab_zz_sdjson_sqlite line 4927. BEGIN failed--compilation aborted at /usr/bin/tv_grab_zz_sdjson_sqlite line 4927.
Maybe the docker image doesn't have the perl bits in it?
Thanks for all the ls-io dockers!
-
You just need to fill in something for the "Container Path:" field . . . like "/movies"
-
Thanks for posting this (disable fast startup). My Win10 VM was hanging with a black screen & cursor on first start after shutdown, thought it might be trying to boot off the install iso but it was still doing it after detaching from the VM. Thanks
-
How can I add two new disks to my already existing shares? I have disk6 and disk7 but they don't show in the menu to be included in the previously created shares.
They also can't be used in any new shares I want to create.
Guide: How To Use Rclone To Mount Cloud Drives And Play Files
in Plugins and Apps
Posted
Try installing google-auth-oauthlib:
pip install google-auth-oauthlib