francrouge Posted December 29, 2018 Share Posted December 29, 2018 That totally depends on your download speed and your processing power of your server. And if you want to stream outside of your home it also depends on your upload speed. If you can mostly direct stream (no transcoding) it will depend 90% on your download/upload speed.Ok i understand thx for infoEnvoyé de mon Pixel 2 XL en utilisant Tapatalk Quote Link to comment
Spladge Posted January 3, 2019 Share Posted January 3, 2019 Back home after holiday travels and again my mount seems to have failed after a reboot / power outage. I've fixed it manually and haven't tested a reboot yet but now have a problem with ```/var/log is getting full (currently 100 % used)``` my syslog being full of rclone entries is all I can really notice. I've changed log level from DEBUG to WARN but do I need to reboot to test this or will it use changed log level on next run? I don't really want to reboot for a few days right now if I can avoid it. Or I should say, due to the size I do not really want to rebuild it if it fails again. Quote Link to comment
DZMM Posted January 3, 2019 Author Share Posted January 3, 2019 7 hours ago, Spladge said: Back home after holiday travels and again my mount seems to have failed after a reboot / power outage. I've fixed it manually and haven't tested a reboot yet but now have a problem with ```/var/log is getting full (currently 100 % used)``` my syslog being full of rclone entries is all I can really notice. I've changed log level from DEBUG to WARN but do I need to reboot to test this or will it use changed log level on next run? I don't really want to reboot for a few days right now if I can avoid it. Or I should say, due to the size I do not really want to rebuild it if it fails again. if you unmount and remount your log level will refresh Quote Link to comment
Kaizac Posted January 3, 2019 Share Posted January 3, 2019 2 hours ago, DZMM said: if you unmount and remount your log level will refresh What do you have your remount script set on? I think when an "de-mount" happens it's pretty hard to get a working mount again. Dockers will be moving files to the mount_unionfs folders and rclone will see it as not being empty thus failing. I've put my remount script on 1 minute, to try to mitigate this, but I think there is a big chance it would fail anyways. Quote Link to comment
Spladge Posted January 3, 2019 Share Posted January 3, 2019 (edited) Unmount then run mount script in background? Or will this require a server restart? Kaizac Is this the 'sleep 5' setting, ie sleep 60. Edited January 3, 2019 by Spladge Quote Link to comment
Kaizac Posted January 3, 2019 Share Posted January 3, 2019 4 minutes ago, Spladge said: Unmount then run mount script in background? Or will this require a server restart? Kaizac Is this the 'sleep 5' setting, ie sleep 60. When dockers are not active/moving files in the union folder then you can unmount and remount through running the script in the background. The script will notify you if it succeeded or not. Most reliable is a full restart of the server of course, but not necessary when you are just remounting previous mounts. The sleep is a waiting time build in the execution of the script. Some things need some time to load up properly (like the mounting). This depends on your own server speed and plugins you run. For example I run the Recycle Bin plugin which can take up to 5-10 minutes to load properly. So if you run docker start you need to put in sleeps for that period or that part of the script won't execute. So you just need to experiment a bit with that. What I was asking DZMM is the cron job timing for the mounting script after it started at array. So I have a mount script at start of array. Then I also run the same script every minute again in case of an unmount. Quote Link to comment
Spladge Posted January 3, 2019 Share Posted January 3, 2019 AH - thank you very much, answered a question I didn't even know I needed to ask yet. Quote Link to comment
DZMM Posted January 3, 2019 Author Share Posted January 3, 2019 (edited) When I want to manually remount I create a temp script to run in the background: fusermount -uz /mnt/user/mount_rclone/google_vfs rclone mount --allow-other --buffer-size 512M --dir-cache-time 72h --drive-chunk-size 512M --fast-list --log-level INFO --vfs-read-chunk-size 128M --vfs-read-chunk-size-limit off tdrive1_vfs: /mnt/user/mount_rclone/google_vfs & exit I don't add any sleeps in and I've not had any problems as the mount is remounted before any dockers realise. Ongoing uploads are fine as they don't use the mount. The sleep in the main mount script is just to avoid any potential problems - it's probably not needed anymore, but I keep it there for insurance. Like @Kaizac I have my script running every 5 mins to check if the mount has dropped. I think this has only happened once before to me in the early days, but it's there if I have e.g. a network failure in the future Edited January 3, 2019 by DZMM Quote Link to comment
Spladge Posted January 3, 2019 Share Posted January 3, 2019 What is the checker script? Quote Link to comment
DZMM Posted January 3, 2019 Author Share Posted January 3, 2019 1 hour ago, Spladge said: What is the checker script? just the mount script - it's designed to remount if the mount drops automatically if run on a schedule e.g. every 5 mins Quote Link to comment
NewDisplayName Posted January 4, 2019 Share Posted January 4, 2019 What you guys think about a small script which checkes if mountcheck file is there, and if not closes all related dockers (atleast plex)? Quote Link to comment
DZMM Posted January 4, 2019 Author Share Posted January 4, 2019 33 minutes ago, nuhll said: What you guys think about a small script which checkes if mountcheck file is there, and if not closes all related dockers (atleast plex)? I guess you could add that in, but if you set my mount script to run on a schedule it should remount automatically. I've been running my setup for over 6 months and I think my mount has only dropped once and that was probably my fault when I was figuring stuff out. Quote Link to comment
NewDisplayName Posted January 4, 2019 Share Posted January 4, 2019 (edited) 1 hour ago, DZMM said: I guess you could add that in, but if you set my mount script to run on a schedule it should remount automatically. I've been running my setup for over 6 months and I think my mount has only dropped once and that was probably my fault when I was figuring stuff out. Yes ive only use ur scripts, just changed here and there, like upload script, still the question is, if it fails to mount for whatever reason, and this whatever reason doenst get automaticly resolved, what happens e.g. to the plex library. Ive set the "dont auto remove" like u suggested, but i already found a flaw in it... if e.g. a series gets updated from xxxP to 1080p, plex shows it as missing (the old file) which is kinda annoying... so i wanna enable it again. I guess you could add it to your script like if not mounted, mount, if that still not helps, close dockers, sleep 10min. Since your script also starts the dockers, this would jsut make sense. Edited January 4, 2019 by nuhll Quote Link to comment
DZMM Posted January 4, 2019 Author Share Posted January 4, 2019 49 minutes ago, nuhll said: Yes ive only use ur scripts, just changed here and there, like upload script, still the question is, if it fails to mount for whatever reason, and this whatever reason doenst get automaticly resolved, what happens e.g. to the plex library. Ive set the "dont auto remove" like u suggested, but i already found a flaw in it... if e.g. a series gets updated from xxxP to 1080p, plex shows it as missing (the old file) which is kinda annoying... so i wanna enable it again. I just click 'empty trash' every day or so. IMO that's a much better solution than plex automatically removing all media if a mount fails and you loose all manual metadata corrections, manual artwork changes etc 51 minutes ago, nuhll said: if not mounted, mount, if that still not helps, close dockers, sleep 10min. Since your script also starts the dockers, this would jsut make sense. I don't think any script could do this fast enough 100% of the time before Plex decides a file is missing. As I stated I think my mount has only gone down once in 6 months, so if you hate the plex behaviour and want to take the chance then turn on auto remove. Quote Link to comment
NewDisplayName Posted January 4, 2019 Share Posted January 4, 2019 (edited) We could make a script which runs all the time and just checks for the mountcheck file, this would instant stop plex if neccessary. while [ ! -f /tmp/list.txt ]; do sleep 1; done; Would be like 1 or 2 sec delay, so fast he cant delete sometzhing, and even if, its unlikly its something gets deleted which was manually added. Edited January 4, 2019 by nuhll Quote Link to comment
slimshizn Posted January 5, 2019 Share Posted January 5, 2019 (edited) DZMM I noticed that you have two different rclone mount's on your github page and the first page of this discussion. Which one are you currently using? Secondly I'm having a issue where when I start my array, unionfs doesn't seem to be mounting, but when I run the script manually in the background it works fine. Here it is. Quote #!/bin/bash ####### Check if script already running ########## if [[ -f "/mnt/user/Media/Cloud/rclone_install_running" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Exiting script already running." exit else touch /mnt/user/Media/Cloud/rclone_install_running fi ## End Check ## mkdir -p /mnt/disks/crypt ####### Check if rclone vfs mount is mounted ########## if [[ -f "/mnt/disks/crypt/Movies/mountcheck" ]] && [[ -f "/mnt/disks/crypt/Series/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Check rclone_vfs mounted success." else echo "$(date "+%d.%m.%Y %T") INFO: mounting rclone." # Mount rclone vfs mount rclone mount --allow-other --dir-cache-time 72h --vfs-read-chunk-size 64M --vfs-read-chunk-size-limit off --buffer-size 1G --log-level INFO --stats 1m uploadcrypt: /mnt/disks/crypt --stats 1m & sleep 5 if [[ -f "/mnt/disks/crypt/Movies/mountcheck" ]] && [[ -f "/mnt/disks/crypt/Series/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: rclone_vfs mount success." else echo "$(date "+%d.%m.%Y %T") CRITICAL: rclone_vfs mount failed - please check for problems." rm /mnt/user/Media/Cloud/rclone_install_running exit fi fi ####### Mount unionfs ########## # check if mounted if [[ -f "/mnt/user/Media/Cloud/Series/mountcheck" ]] && [[ -f "/mnt/user/Media/Cloud/Movies/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Check successful, unionfs Movies & Series mounted." else # Unmount before remounting fusermount -uz /mnt/user/Media/Cloud/Series fusermount -uz /mnt/user/Media/Cloud/Movies unionfs -o cow,allow_other,direct_io,auto_cache,sync_read /mnt/user/Media/Cloud/Seriestmp=RW:/mnt/disks/crypt/Series=RO /mnt/user/Media/Cloud/Series unionfs -o cow,allow_other,direct_io,auto_cache,sync_read /mnt/user/Media/Cloud/Moviestmp=RW:/mnt/disks/crypt/Movies=RO /mnt/user/Media/Cloud/Movies if [[ -f "/mnt/user/Media/Cloud/Series/mountcheck" ]] && [[ -f "/mnt/user/Media/Cloud/Movies/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Check successful, unionfs Movies & Series mounted." else echo "$(date "+%d.%m.%Y %T") CRITICAL: unionfs Movies & Series Remount failed." rm /mnt/user/Media/Cloud/rclone_install_running exit fi fi ####### End Mount unionfs ########## rm /mnt/user/Media/Cloud/rclone_install_running exit Got this error after adding rclone mount --rc --allow-other --dir-cache-time 72h --vfs-read-chunk-size 128M --drive-chunk-size 256M --fast-list --vfs-read-chunk-size-limit off --buffer-size 1G --log-level INFO uploadcrypt: /mnt/disks/crypt & # Populate rclone dir-cache rclone rc --timeout=1h vfs/refresh recursive=true & exit Quote 2019/01/05 18:27:45 NOTICE: Serving remote control on http://127.0.0.1:5572/ 2019/01/05 18:27:45 ERROR : rc: "vfs/refresh": error: couldn't find method "vfs/refresh" 2019/01/05 18:27:45 Failed to rc: Failed to read rc response: 404 Not Found: { "error": "couldn't find method \"vfs/refresh\"", "input": { "recursive": "true" }, "path": "vfs/refresh", "status": 404 } Edited January 5, 2019 by slimshizn Quote Link to comment
slimshizn Posted January 5, 2019 Share Posted January 5, 2019 (edited) Okay I ran the unmount script and tried it again. This is what I'm getting. I ran the script, then saw it said please check for problems, I ran it a SECOND time and thats when it showed the syntax error...BUT I checked my mount and everything appeared. Quote Script Starting Sat, 05 Jan 2019 18:34:11 -0500 Full logs for this script are available at /tmp/user.scripts/tmpScripts/Rclone_install_DZMM/log.txt 05.01.2019 18:34:11 INFO: mounting rclone. 2019/01/05 18:34:11 NOTICE: Serving remote control on http://127.0.0.1:5572/ 05.01.2019 18:34:16 CRITICAL: rclone_vfs mount failed - please check for problems. Script Finished Sat, 05 Jan 2019 18:34:16 -0500 Full logs for this script are available at /tmp/user.scripts/tmpScripts/Rclone_install_DZMM/log.txt 2019/01/05 18:34:20 INFO : Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA - Errors: 0 Checks: 0 / 0, - Transferred: 0 / 0, - Elapsed time: 5h6m10.6s Script Starting Sat, 05 Jan 2019 18:34:57 -0500 Full logs for this script are available at /tmp/user.scripts/tmpScripts/Rclone_install_DZMM/log.txt 05.01.2019 18:34:57 INFO: Check rclone_vfs mounted success. fusermount: failed to unmount /mnt/user/Media/Cloud/Series: Invalid argument fusermount: failed to unmount /mnt/user/Media/Cloud/Movies: Invalid argument 05.01.2019 18:34:58 INFO: Check successful, unionfs Movies & Series mounted. /tmp/user.scripts/tmpScripts/Rclone_install_DZMM/script: line 85: syntax error near unexpected token `fi' /tmp/user.scripts/tmpScripts/Rclone_install_DZMM/script: line 85: `fi' Script Finished Sat, 05 Jan 2019 18:34:58 -0500 Full logs for this script are available at /tmp/user.scripts/tmpScripts/Rclone_install_DZMM/log.txt { "result": { "": "OK" } } I found the extra "fi" and removed it. Quote 05.01.2019 18:42:52 INFO: mounting rclone. 2019/01/05 18:42:52 NOTICE: Serving remote control on http://127.0.0.1:5572/ 05.01.2019 18:42:58 INFO: rclone_vfs mount success. fusermount: failed to unmount /mnt/user/Media/Cloud/Series: Invalid argument fusermount: failed to unmount /mnt/user/Media/Cloud/Movies: Invalid argument 05.01.2019 18:42:58 INFO: Check successful, unionfs Movies & Series mounted. Script Finished Sat, 05 Jan 2019 18:42:58 -0500 Full logs for this script are available at /tmp/user.scripts/tmpScripts/Rclone_install_DZMM/log.txt { "result": { "": "OK" } Oddly, the file that's created to show that it's running didn't delete. I had to go in and remove it manually. After that It ran like above and seems to be working fine again! Will report back if there's errors again after a reboot in the future. Edited January 5, 2019 by slimshizn Quote Link to comment
DZMM Posted January 5, 2019 Author Share Posted January 5, 2019 33 minutes ago, slimshizn said: DZMM I noticed that you have two different rclone mount's on your github page and the first page of this discussion Ignore the one in the first post - the github is the up to date version. 34 minutes ago, slimshizn said: Secondly I'm having a issue where when I start my array, how soon do you run the script after an array start? Mine's on a 5 min cycle - does it work the second time automatically? Maybe it starts too close to the array starting 35 minutes ago, slimshizn said: Got this error after adding rclone mount --rc --allow-other --dir-cache-time 72h --vfs-read-chunk-size 128M --drive-chunk-size 256M --fast-list --vfs-read-chunk-size-limit off --buffer-size 1G --log-level INFO uploadcrypt: /mnt/disks/crypt & # Populate rclone dir-cache rclone rc --timeout=1h vfs/refresh recursive=true & exit I've ditched rc as the refresh was just a nice to have and rc and wasn't worth the hassle 13 minutes ago, slimshizn said: I ran the script, then saw it said please check for problems, I ran it a SECOND time and thats when it showed the syntax error...BUT I checked my mount and everything appeared. what's on line 85? Quote Link to comment
slimshizn Posted January 6, 2019 Share Posted January 6, 2019 "fi" was on line 85, has been removed and working fine now. Seems to be all good, I'll set a custom time for it to run again after 10 minutes. Editing some scripts to update them and came across this portion of your upload.. Quote --delete-empty-src-dirs --fast-list --bwlimit 9500k --tpslimit 3 --min-age 30 Now I don't know if you remember our discussion where rclone would "download" files that were deleted from my server after an upload. Really odd thing that was going on. I believe you had me remove the delete portion and it seemed to work fine but I see it's changed to checkers 3, tpslimit 3 which I don't know what that is and min-age 30. Quote Link to comment
slimshizn Posted January 6, 2019 Share Posted January 6, 2019 Current rclone script Quote #!/bin/bash ####### Check if script already running ########## if [[ -f "/mnt/user/Media/Cloud/rclone_install_running" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Exiting script already running." exit else touch /mnt/user/Media/Cloud/rclone_install_running fi ## End Check ## mkdir -p /mnt/disks/crypt ####### Check if rclone vfs mount is mounted ########## if [[ -f "/mnt/disks/crypt/Movies/mountcheck" ]] && [[ -f "/mnt/disks/crypt/Series/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Check rclone_vfs mounted success." else echo "$(date "+%d.%m.%Y %T") INFO: mounting rclone." # Mount rclone vfs mount rclone mount --rc --allow-other --dir-cache-time 72h --vfs-read-chunk-size 128M --drive-chunk-size 256M --fast-list --vfs-read-chunk-size-limit off --buffer-size 1G --log-level INFO uploadcrypt: /mnt/disks/crypt & sleep 5 if [[ -f "/mnt/disks/crypt/Movies/mountcheck" ]] && [[ -f "/mnt/disks/crypt/Series/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: rclone_vfs mount success." else echo "$(date "+%d.%m.%Y %T") CRITICAL: rclone_vfs mount failed - please check for problems." rm /mnt/user/Media/Cloud/rclone_install_running exit fi fi ####### Mount unionfs ########## # check if mounted if [[ -f "/mnt/user/Media/Cloud/Series/mountcheck" ]] && [[ -f "/mnt/user/Media/Cloud/Movies/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Check successful, unionfs Movies & Series mounted." else # Unmount before remounting fusermount -uz /mnt/user/Media/Cloud/Series fusermount -uz /mnt/user/Media/Cloud/Movies unionfs -o cow,allow_other,direct_io,auto_cache,sync_read /mnt/user/Media/Cloud/Seriestmp=RW:/mnt/disks/crypt/Series=RO /mnt/user/Media/Cloud/Series unionfs -o cow,allow_other,direct_io,auto_cache,sync_read /mnt/user/Media/Cloud/Moviestmp=RW:/mnt/disks/crypt/Movies=RO /mnt/user/Media/Cloud/Movies if [[ -f "/mnt/user/Media/Cloud/Series/mountcheck" ]] && [[ -f "/mnt/user/Media/Cloud/Movies/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Check successful, unionfs Movies & Series mounted." else echo "$(date "+%d.%m.%Y %T") CRITICAL: unionfs Movies & Series Remount failed." rm /mnt/user/Media/Cloud/rclone_install_running exit fi fi # Populate rclone dir-cache rclone rc --timeout=1h vfs/refresh recursive=true & ####### End Mount unionfs ########## rm /mnt/user/Media/Cloud/rclone_install_running exit Current upload script Quote #!/bin/bash ####### Check if script already running ########## if [[ -f "/mnt/user/Media/Cloud/rclone_upload_running" ]]; then exit else touch /mnt/user/Media/Cloud/rclone_upload_running fi # set folders uploadfolderMovies="/mnt/user/Media/Cloud/Moviestmp" uploadfolderSeries="/mnt/user/Media/Cloud/Seriestmp" # move files rclone move $uploadfolderSeries uploadcrypt:/Series -vv --drive-chunk-size 512M --checkers 10 --fast-list --transfers 4 --exclude .unionfs/** --exclude *fuse_hidden* --exclude *_HIDDEN --exclude .recycle/** --exclude *.backup~* --exclude *.partial~* --bwlimit 1.25M rclone move $uploadfolderMovies uploadcrypt:/Movies -vv --drive-chunk-size 512M --checkers 10 --fast-list --transfers 4 --exclude .unionfs/** --exclude *fuse_hidden* --exclude *_HIDDEN --exclude .recycle/** --exclude *.backup~* --exclude *.partial~* --bwlimit 1.25M rm /mnt/user/Media/Cloud/rclone_upload_running exit Quote Link to comment
DZMM Posted January 6, 2019 Author Share Posted January 6, 2019 10 hours ago, slimshizn said: Now I don't know if you remember our discussion where rclone would "download" files that were deleted from my server after an upload. Really odd thing that was going on. I believe you had me remove the delete portion and it seemed to work fine but I see it's changed to checkers 3, tpslimit 3 which I don't know what that is and min-age 30. rclone fixed delete-empty-src-dirs a few releases ago, so I put it back in. min-age 30 stops files younger than 30 mins being uploaded. I reduced my checkers and tpslimits to reduce the odds of API bans - they probably could be increased. Quote Link to comment
NewDisplayName Posted January 6, 2019 Share Posted January 6, 2019 (edited) https://github.com/ncw/rclone/issues/2880 (we should add --drive-pacer-min-sleep=100ms ) Edited January 6, 2019 by nuhll 1 Quote Link to comment
DZMM Posted January 6, 2019 Author Share Posted January 6, 2019 15 minutes ago, nuhll said: https://github.com/ncw/rclone/issues/2880 good find. 1.4.6 will hopefully also include rclone union which should allow us to drop unionfs and have a all-in-one solution Quote Link to comment
NewDisplayName Posted January 6, 2019 Share Posted January 6, 2019 the create and remove directorys is also a known problem... Quote Link to comment
NewDisplayName Posted January 6, 2019 Share Posted January 6, 2019 https://github.com/ncw/rclone/issues/2869 Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.