axeman Posted August 30, 2021 Share Posted August 30, 2021 7 hours ago, sheldz8 said: The exclusions are on the upload script but I think it deletes the whole folder afterwards. Whatever is in the local folder will be copied over unless it is excluded. I wonder if the bandaid answer is to have a script that runs before the upload and move the metadata to the array. Probably not the best solution though. Would've been great if MergerFS can orchestrate which files go to which folder. Quote Link to comment
DZMM Posted August 30, 2021 Author Share Posted August 30, 2021 8 hours ago, sheldz8 said: The exclusions are on the upload script but I think it deletes the whole folder afterwards. Whatever is in the local folder will be copied over unless it is excluded. Excluded files aren't deleted - they stay local and aren't uploaded Quote Link to comment
sheldz8 Posted August 30, 2021 Share Posted August 30, 2021 I wonder if the bandaid answer is to have a script that runs before the upload and move the metadata to the array. Probably not the best solution though. Would've been great if MergerFS can orchestrate which files go to which folder. You can use radarr or sonarr to download the metadataSent from my SM-A305F using Tapatalk Quote Link to comment
francrouge Posted August 31, 2021 Share Posted August 31, 2021 (edited) Hi guys, Got a quick question is there a way to change the upload script to copy a specific folder and encrypt it and sync it with gdrive ? For exemple i wan to copy mtn/user/plex to gdrive with the option copy and not move how should i change the upload script ? thx Edited September 1, 2021 by francrouge Quote Link to comment
DZMM Posted September 1, 2021 Author Share Posted September 1, 2021 (edited) 7 hours ago, francrouge said: Hi guys, Got a quick question is there a way to change the upload script to copy a specific folder and encrypt it and sync it with gdrive ? For exemple i wan to copy mtn/user/plex to gdrive with the option copy and not move how should i change the upload script ? thx Easy: RcloneCommand="copy" All explained in the script Edited September 1, 2021 by DZMM Quote Link to comment
francrouge Posted September 1, 2021 Share Posted September 1, 2021 8 hours ago, DZMM said: Easy: RcloneCommand="copy" All explained in the script hi yes i changed that but i would like to upload a specific folder and not have to move it in mount_upload is it posssible ? thx again Quote Link to comment
DZMM Posted September 1, 2021 Author Share Posted September 1, 2021 1 hour ago, francrouge said: hi yes i changed that but i would like to upload a specific folder and not have to move it in mount_upload is it posssible ? thx again Probably, just create a new instance of the script maybe and --include only that path https://rclone.org/filtering/ 1 Quote Link to comment
francrouge Posted September 1, 2021 Share Posted September 1, 2021 1 hour ago, DZMM said: Probably, just create a new instance of the script maybe and --include only that path https://rclone.org/filtering/ thx Quote Link to comment
francrouge Posted September 1, 2021 Share Posted September 1, 2021 1 hour ago, DZMM said: Probably, just create a new instance of the script maybe and --include only that path https://rclone.org/filtering/ would it be better if i move my folder with mergefs ? does the upload script works with that also ? thx a lot Quote Link to comment
sheldz8 Posted September 1, 2021 Share Posted September 1, 2021 (edited) 6 hours ago, francrouge said: would it be better if i move my folder with mergefs ? does the upload script works with that also ? thx a lot when you upload; it will use "--filter" if you set it instead of "--exclude" the mount script works with mergerfs the upload script is optional. All your Apps (Radarr, Sonarr, Plex, etc) should be volume mounted to "/path/to/mergerfs, rw" Mergerfs creates a virtual directory which duplicate of the remote drive and also the local folder (temporary storage), whatever is in the local folder shows in mergerfs but not the remote folder until it uploads. Edited September 2, 2021 by sheldz8 1 Quote Link to comment
sheldz8 Posted September 2, 2021 Share Posted September 2, 2021 On 8/30/2021 at 1:30 AM, axeman said: I might have asked this before (can't seem to find it)... but are the exclusions on the upload script? Do you have something else to move them so that they are somewhere on the array - or do they basically stay in /local/ ? I have mergerFS joining my files from my Array: /mnt/user/Videos, Cache Drive: /mnt/user/local, and of course Rclone mount/cloud. Wondering if there's a way to force mergerFS to only copy certain files directly to /mnt/user/Videos. https://rclone.org/filtering/ 1 Quote Link to comment
francrouge Posted September 3, 2021 Share Posted September 3, 2021 (edited) On 9/1/2021 at 7:56 PM, sheldz8 said: when you upload; it will use "--filter" if you set it instead of "--exclude" the mount script works with mergerfs the upload script is optional. All your Apps (Radarr, Sonarr, Plex, etc) should be volume mounted to "/path/to/mergerfs, rw" Mergerfs creates a virtual directory which duplicate of the remote drive and also the local folder (temporary storage), whatever is in the local folder shows in mergerfs but not the remote folder until it uploads. so if i want to put a local folder i need to add it in this ? # Add extra paths to mergerfs mount in addition to LocalFilesShare LocalFilesShare2="ignore" # without trailing slash e.g. /mnt/user/other__remote_mount/or_other_local_folder. Enter 'ignore' to disable Wait i need to add my folders in MountFolders=\{"downloads/complete,downloads/intermediate,downloads/seeds,movies,tv"\} # comma separated list of folders to create within the mount option ? And can you explain how the files in merges fs are going to be uploaded since the upload script is optional ? please thx again Edited September 3, 2021 by francrouge Quote Link to comment
sheldz8 Posted September 3, 2021 Share Posted September 3, 2021 so if i want to put a local folder i need to add it in this ? # Add extra paths to mergerfs mount in addition to LocalFilesShare LocalFilesShare2="ignore" # without trailing slash e.g. /mnt/user/other__remote_mount/or_other_local_folder. Enter 'ignore' to disable And can you explain how the files in merges fs are going to be uploaded is the upload script is optional ? please thx againSorry I meant that it is optional if you don't plan on uploading anything. I misread your original post when I said that.LocalFileShare2 is for a 2nd remote drive and From what I read now again on your original post is that you want to copy plex to gdrive? Do you want to do this alongside the local share? I don't think that will work because I presume it will use move/copy/sync that you set for the first local Share. I would suggest using something like rsync to auto backup the plex folder periodically and point it to backup to the local folder.I haven't done this before because I know it usually corrupts the backup folder and plex restore is basically uselessSent from my SM-A305F using Tapatalk Quote Link to comment
francrouge Posted September 4, 2021 Share Posted September 4, 2021 23 hours ago, sheldz8 said: Sorry I meant that it is optional if you don't plan on uploading anything. I misread your original post when I said that. LocalFileShare2 is for a 2nd remote drive and From what I read now again on your original post is that you want to copy plex to gdrive? Do you want to do this alongside the local share? I don't think that will work because I presume it will use move/copy/sync that you set for the first local Share. I would suggest using something like rsync to auto backup the plex folder periodically and point it to backup to the local folder. I haven't done this before because I know it usually corrupts the backup folder and plex restore is basically useless Sent from my SM-A305F using Tapatalk Hi Main goal is to have one folder with all my local medias and gdrive So Main folder = Medias + Gdrive So i think mergefs would worked for that But the only issue that i'm trying to solved is if i want to move things from local media to gdrive automaticly how i do it. (uploaded) Exemple downloaded files that are in my local media and can be uploaded in gdrive thx again Quote Link to comment
francrouge Posted September 4, 2021 Share Posted September 4, 2021 @DZMMHi quick question for you. I always get api ban bust the 750 Go per day i thought with the command you add --drive-stop-on-upload-limit it would stop before. Do you know if its still working ? Thx a lot have a nice day Quote Link to comment
DZMM Posted September 4, 2021 Author Share Posted September 4, 2021 1 hour ago, francrouge said: @DZMMHi quick question for you. I always get api ban bust the 750 Go per day i thought with the command you add --drive-stop-on-upload-limit it would stop before. Do you know if its still working ? Thx a lot have a nice day my upload doesn't let me bust 750GB/day (moved home) anymore. stop on limit stops the download when the error occurs, not before i.e. it is not preventative - it just stops the script constantly hammering away for 24 hours to upload files. If you want to upload >750GB/day, use service accounts with the script. 1 Quote Link to comment
sheldz8 Posted September 4, 2021 Share Posted September 4, 2021 Hi Main goal is to have one folder with all my local medias and gdrive So Main folder = Medias + Gdrive So i think mergefs would worked for that But the only issue that i'm trying to solved is if i want to move things from local media to gdrive automaticly how i do it. (uploaded) Exemple downloaded files that are in my local media and can be uploaded in gdrive thx again Just use a cron job to run whenever you want on the user script page set to customExample every 6 hours use: 0 */6 * * * As long as your upload script is setup correctly it will auto run every 6 hours without failing if there are files to upload and if you used below 750gb.Sent from my SM-A305F using Tapatalk 1 Quote Link to comment
Lucka Posted September 13, 2021 Share Posted September 13, 2021 (edited) Hey, I got errors from the Upload script since this morning (been using it for more than a year now without any problems) : DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly.) DEBUG : pacer: Rate limited, increasing sleep to 2.62s BUT in the meantime, the script upload files to my drive perfectly. This cause me some issues because the script never ends (this error appear infinitely), so when new files are added, I need to restart my server to stop the script, and launch it again! Any Idea ? Also, is there a way to kill this script? I tried with grep but can't find it. Edited September 13, 2021 by Lucka Quote Link to comment
sheldz8 Posted September 14, 2021 Share Posted September 14, 2021 Hey, I got errors from the Upload script since this morning (been using it for more than a year now without any problems) : DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly.)DEBUG : pacer: Rate limited, increasing sleep to 2.62s BUT in the meantime, the script upload files to my drive perfectly. This cause me some issues because the script never ends (this error appear infinitely), so when new files are added, I need to restart my server to stop the script, and launch it again! Any Idea ? Also, is there a way to kill this script? I tried with grep but can't find it. You uploaded more than whatever amount of GB you set within 24 hours it should be fine now. Also ensure --drive-stop-on-upload-limit is set on the upload script because it will cause a loop if it hits 750gb it can cause issues with your gdrive api. You should be able to cancel the script on the plugin pagehttps://github.com/rclone/rclone/issues/3857Sent from my SM-A305F using Tapatalk Quote Link to comment
Playerz Posted September 16, 2021 Share Posted September 16, 2021 Hi all I'm having this strange issue where all the files in mount_rclone does not get deleted after upload? or am i missing something? this are uploading, but it would seem that it doesnt get cleaned up afterwards? Maybe something someone have an answer for? Mount: https://pastebin.com/dwKWJcuf Upload: https://pastebin.com/zmveVkA1 Quote Link to comment
sheldz8 Posted September 17, 2021 Share Posted September 17, 2021 Hi all I'm having this strange issue where all the files in mount_rclone does not get deleted after upload? or am i missing something? this are uploading, but it would seem that it doesnt get cleaned up afterwards? Maybe something someone have an answer for? Mount: https://pastebin.com/dwKWJcuf Upload: https://pastebin.com/zmveVkA1 What does the upload logs say after uploading is finished?Sent from my SM-A305F using Tapatalk Quote Link to comment
Playerz Posted September 17, 2021 Share Posted September 17, 2021 5 minutes ago, sheldz8 said: What does the upload logs say after uploading is finished? Sent from my SM-A305F using Tapatalk I have attached my upload log, it would seem that they are getting stuck under something called gdrive/VFS log.txt Quote Link to comment
sheldz8 Posted September 17, 2021 Share Posted September 17, 2021 I have attached my upload log, it would seem that they are getting stuck under something called gdrive/VFSlog.txtThe txt file is corruptSent from my SM-A305F using Tapatalk Quote Link to comment
Playerz Posted September 17, 2021 Share Posted September 17, 2021 4 minutes ago, sheldz8 said: The txt file is corrupt Sent from my SM-A305F using Tapatalk I have put it on pastebin: https://pastebin.com/wbtmhGP2 maybe it has something to do with it saying directory not empty? cannot delete Quote Link to comment
sheldz8 Posted September 17, 2021 Share Posted September 17, 2021 I have put it on pastebin: https://pastebin.com/wbtmhGP2 maybe it has something to do with it saying directory not empty? cannot deleteThe files are being deleted after each upload so it's probably excluded files preventing it from deleting the folder.If you want the folders deleted even though the media files are uploaded then you would need to edit the upload script to force remove directories that aren't empty.Sent from my SM-A305F using Tapatalk Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.