Jump to content

neow

Members
  • Content Count

    47
  • Joined

  • Last visited

Everything posted by neow

  1. hello thanks for your reply i copy files via krusader from a share called archives to a share called unionfs (google drive rclone) please find enclosed the diagnostic file towerpen-diagnostics-20190709-1230.zip
  2. Hello, i need to copy some files from a share to another share, but this is very slow with up and down writing (up 50mo/sec down 5 mo/sec) this is large files, i don't understand why this is slow like this. When i copy from a syno to unraid i got 80 mo/sec, so why from a share to another one this is slow like this? my config : - last unraid version - ryzen 1800x - turbo write mode activated - 2 parity disks - 2 ssd cache thanks for your ideas
  3. neow

    Preclear

    i find the solution -> enable destructive mode thanks problem solved
  4. neow

    Preclear

    hello, i changed yesterday my ssd cache- everything work fine. Now i have ssd unassigned for vm i pre cleared them but impossible to format them or mount them : https://d.pr/i/7bj2Rz any solution? thanks
  5. Ok i will try. Just one question: when you start the array you mount check and the mount script work fine?
  6. Krusader. i can affirm i can wait 1hour. Without this manipulation unionfs will never mount i don’t have a cron task for mounting script like many if you. It’s just run at the array mounting
  7. 1h14 it’s the array mounted. The script stops here 16.06.2019 01:14:37 INFO: Exiting script already running. Script Finished Sun, 16 Jun 2019 01:14:37 +0200  Full logs for this script are available at /tmp/user.scripts/tmpScripts/rclone_mount_plugin/log.txt  After my manipulation 1h16 everything work fine 16.06.2019 01:16:51 INFO: Check successful, unionfs mounted.
  8. The problem is that when i turn on my array and after the mount check it doesn’t work. Impossible to see unionfs i have to do the procedure i wrote just before and then everything work. When you turn on your server everything work automatically after mountcheck? Or you have to do the same procedure as me?
  9. root@Tower:~# touch mountcheck root@Tower:~# rclone copy mountcheck gdrive_media_vfs: -vv --no-traverse 2019/06/16 01:17:02 DEBUG : rclone: Version "v1.48.0-001-g4549305f-beta" starting with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone-beta/.rclone.conf" "copy" "mountcheck" "gdrive_media_vfs:" "-vv" "--no-traverse"] 2019/06/16 01:17:02 DEBUG : Using config file from "/boot/config/plugins/rclone-beta/.rclone.conf" 2019/06/16 01:17:03 DEBUG : mountcheck: Modification times differ by -1m49.823031434s: 2019-06-16 01:17:01.382031434 +0200 CEST, 2019-06-15 23:15:11.559 +0000 UTC 2019/06/16 01:17:04 INFO : mountcheck: Copied (replaced existing) 2019/06/16 01:17:04 INFO : Transferred: 32 / 32 Bytes, 100%, 14 Bytes/s, ETA 0s Errors: 0 Checks: 0 / 0, - Transferred: 1 / 1, 100% Elapsed time: 2.2s 2019/06/16 01:17:04 DEBUG : 5 go routines active 2019/06/16 01:17:04 DEBUG : rclone: Version "v1.48.0-001-g4549305f-beta" finishing with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone-beta/.rclone.conf" "copy" "mountcheck" "gdrive_media_vfs:" "-vv" "--no-traverse"] #!/bin/bash ####### Check if script is already running ########## if [[ -f "/mnt/user/appdata/other/rclone/rclone_mount_running" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Exiting script already running." exit else touch /mnt/user/appdata/other/rclone/rclone_mount_running fi ####### End Check if script already running ########## ####### Start rclone mounts ########## # create directories for rclone mount and unionfs mount mkdir -p /mnt/user/appdata/other/rclone mkdir -p /mnt/user/mount_rclone/google_vfs mkdir -p /mnt/user/mount_unionfs/google_vfs mkdir -p /mnt/user/rclone_upload/google_vfs ####### Start rclone gdrive mount ########## # check if rclone mount already created if [[ -f "/mnt/user/mount_rclone/google_vfs/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Check rclone vfs mount success." else echo "$(date "+%d.%m.%Y %T") INFO: mounting rclone vfs." rclone mount --rc --allow-other --buffer-size 1G --dir-cache-time 72h --drive-chunk-size 256M --fast-list --log-level INFO --vfs-read-chunk-size 128M --vfs-read-chunk-size-limit off gdrive_media_vfs: /mnt/user/mount_rclone/google_vfs & # check if mount successful # slight pause to give mount time to finalise sleep 10 if [[ -f "/mnt/user/mount_rclone/google_vfs/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Check rclone gdrive vfs mount success." else echo "$(date "+%d.%m.%Y %T") CRITICAL: rclone gdrive vfs mount failed - please check for problems." rm /mnt/user/appdata/other/rclone/rclone_mount_running exit fi fi ####### End rclone gdrive mount ########## ####### End rclone mounts ########## ####### Start unionfs mount ########## if [[ -f "/mnt/user/mount_unionfs/google_vfs/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Check successful, unionfs mounted." else unionfs -o cow,allow_other,direct_io,auto_cache,sync_read /mnt/user/rclone_upload/google_vfs=RW:/mnt/user/mount_rclone/google_vfs=RO /mnt/user/mount_unionfs/google_vfs if [[ -f "/mnt/user/mount_unionfs/google_vfs/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Check successful, unionfs mounted." else echo "$(date "+%d.%m.%Y %T") CRITICAL: unionfs Remount failed." rm /mnt/user/appdata/other/rclone/rclone_mount_running exit fi fi ####### End Mount unionfs ########## ############### starting dockers that need unionfs mount ###################### # only start dockers once if [[ -f "/mnt/user/appdata/other/rclone/dockers_started" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: dockers already started" else touch /mnt/user/appdata/other/rclone/dockers_started echo "$(date "+%d.%m.%Y %T") INFO: Starting dockers." docker start plex docker start ombi docker start tautulli docker start radarr docker start sonarr fi ############### end dockers that need unionfs mount ###################### # populate rclone dir-cache echo "$(date "+%d.%m.%Y %T") Info: populating dir cache - this could take a while." rclone rc --timeout=1h vfs/refresh recursive=true echo "$(date "+%d.%m.%Y %T") Info: populating dir cache complete." exit results (see the difference at the boot and when i do it manually) Script Starting Sun, 16 Jun 2019 01:14:37 +0200 Full logs for this script are available at /tmp/user.scripts/tmpScripts/rclone_mount_plugin/log.txt 16.06.2019 01:14:37 INFO: Exiting script already running. Script Finished Sun, 16 Jun 2019 01:14:37 +0200 Full logs for this script are available at /tmp/user.scripts/tmpScripts/rclone_mount_plugin/log.txt Script Starting Sun, 16 Jun 2019 01:16:41 +0200 Full logs for this script are available at /tmp/user.scripts/tmpScripts/rclone_mount_plugin/log.txt 16.06.2019 01:16:41 INFO: mounting rclone vfs. 2019/06/16 01:16:41 NOTICE: Serving remote control on http://127.0.0.1:5572/ 16.06.2019 01:16:51 INFO: Check rclone gdrive vfs mount success. 16.06.2019 01:16:51 INFO: Check successful, unionfs mounted. 16.06.2019 01:16:51 INFO: Starting dockers. Error response from daemon: No such container: plex Error: failed to start containers: plex Error response from daemon: No such container: ombi Error: failed to start containers: ombi Error response from daemon: No such container: tautulli Error: failed to start containers: tautulli Error response from daemon: No such container: radarr Error: failed to start containers: radarr Error response from daemon: No such container: sonarr Error: failed to start containers: sonarr 16.06.2019 01:16:51 Info: populating dir cache - this could take a while. { "result": { "": "OK" } } 16.06.2019 01:16:52 Info: populating dir cache complete. Script Finished Sun, 16 Jun 2019 01:16:52 +0200 Full logs for this script are available at /tmp/user.scripts/tmpScripts/rclone_mount_plugin/log.txt For the containers no problem i didn't install them for the moment thanks for your help, i appreciate
  10. yes i solved 95% of the problems tonight we had an mounting of the unionfs everything worked fine but when i restart the server everything is mounted no error with the scripts but no visibility of the unionfs everytime i restart the server i do the mount check after the mounting but no connection with unionfs i don't understand why . Maybe it's krusader... the procedure to solve this for each reboot is: 1- erase the mount launched by the script at the start of the array 2- close krusder 3- relaunch the mount script 4- do the mountcheck 5- relaunch krusader All the other manipulations didn't work
  11. actually this is very strange i explain my self when i erase all the mounted files union/upload/rclone and erase theconfig file and after relaunch the script to mount it works but when i reboot all the things go bad again and i got to redo the procedure when i erase all the mounted files union/upload/rclone and erase theconfig file and after relaunch the script to mount it works very strange thing
  12. it's not simple as copy cut the script ii think because i tried several times and had several errors most of the time it was my fault because i am a noob but still some errors i can't explain myself
  13. i saw your posts at the beginning of the thread when you were not an expert you were nicer with people sorry for the mp i will not do it again btw i have a prblem with the upload script : 15.06.2019 20:32:55 INFO: rclone not installed - will try again later. Script Finished Sat, 15 Jun 2019 20:32:55 +0200 Full logs for this script are available at /tmp/user.scripts/tmpScripts/rclone_custom_plugin/log.txt upload script is : #!/bin/bash ####### Check if script already running ########## if [[ -f "/mnt/user/appdata/other/rclone/rclone_upload" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Exiting as script already running." exit else touch /mnt/user/appdata/other/rclone/rclone_upload fi ####### End Check if script already running ########## ####### check if rclone installed ########## if [[ -f "/mnt/user/mount_rclone/google_vfs/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: rclone installed successfully - proceeding with upload." else echo "$(date "+%d.%m.%Y %T") INFO: rclone not installed - will try again later." rm /mnt/user/appdata/other/rclone/rclone_upload exit fi ####### end check if rclone installed ########## # move files rclone move /mnt/user/rclone_upload/google_vfs/ gdrive_media_vfs: -vv --drive-chunk-size 512M --checkers 3 --fast-list --transfers 2 --exclude .unionfs/** --exclude *fuse_hidden* --exclude *_HIDDEN --exclude .recycle** --exclude *.backup~* --exclude *.partial~* --delete-empty-src-dirs --fast-list --bwlimit 9500k --tpslimit 3 --min-age 30m # remove dummy file rm /mnt/user/appdata/other/rclone/rclone_upload exit thanks for any help
  14. Is anybody can share is settings for docker for folders etc... plex radar sonarr? thanks Other question, can i use two servers unraid with the same scripts, same token, same folders?
  15. ok i didn't understand the main objective of your scripts sorry
  16. ok if i put files to mount_unionfs (not rclone_upload), it's still store locally even after the upload? (like dropbox with local copy synchronized?) i understood if i put directly my files to upload_rclone they will be erased once the upload is complete i am just afraid google close my account g drive and i lost everything
  17. To be understandable i wanted : 1- to update all my files to gdrive and keep them in local for secure them and shut down my big unraid server to save electricity 2- use a small nuc server connected to gdrive crypted to use with plex server (with unraid server shutted down) 3- if i see everything is ok erase my unraid files without erase gdrive crypted files When you say this : "All your 'media' dockers - plex, radarr, sonarr etc should be mapped to the unionfs folder" you talk about this file ? : root-> unraid->user->mount_unionfs->google_vfs i don't understand this sentence : "1. I think yes - as above this is the file 'view' that is what your dockers etc need to use" i don't have a file named "view" i think i understand the process : 1- mount_unionfs is gdrive decrypted ( store locally) 2- mount_rclone is gdrive encrypted (not stored locally) 3- Rclone_upload is the files who are updated to mount_rclone 1- which folder can i erase once it uploaded to earn space on unraid but keep the files in Gdrive as you did with your server? 2- A contrario, if i want to uploaded and keep files in unraid in just move plex files to mount_unionfs and i will keep them locally once it has been updated?
  18. ok thank you i have some stupid questions: 1- if i change my server to another place with another ip adress, is it a problem, should i configure the link to gdrive again? 2-if i want to upload (and keep them locallly) files from plex in my unraid server to gdrive uncrypted in which folder should i move them in unraid? 3-if i want to upload (and erase them locallly to earn space disk) files from plex in my unraid server to gdrive uncrypted in which folder should i move them in unraid? 4- with plex server, which files in unraid i need to track to use only gdrive uncrypted files? 5- can i use a small energy computer like nuc with windows 10 to install plex server and have access to gdrive data uncrypted? what software should i use to mount the gdrive uncrypted to windows 10 and decrypt it? because my unraid server actually costs me too much electricity 6- and please reexplain me the utility of clean script i don't understand at all what it does the procedure i use to upload to gdrive uncrypted is: 1- copy/moving file to root-> unraid->user->mount_unionfs->google_vfs 2- launch the upload script is it correct? am i right to think : 1- root-> unraid->user->mount_unionfs->google_vfs is store in local? 2- root->unraid->user->rclone_upload->google_vfs is just a passerelle, i don't have to put anything in it. When i launch script it takes new data from root-> unraid->user->mount_unionfs->google_vfs and delete it automatically by script when is done, am i right? 3- i don't understand the utility of the file root-> unraid->user-> mount_rclone-> google_vfs is it store in local or in gdrive? is it just the view decrypted of my gdrive? plex needs to be configure on this file? the question can be resume by, which file rclone is stored in my unraid server, which file rclone is gdrive connection and not stored in my unraid server i had my first upload today i am very happy thanks to you 😍
  19. hello i reboot this morning an before i cleared the files you told me, please fin enclosed the result: Script location: /tmp/user.scripts/tmpScripts/rclone_upload_plugin/script Note that closing this window will abort the execution of this script 13.06.2019 09:36:44 INFO: rclone installed successfully - proceeding with upload. 2019/06/13 09:36:44 DEBUG : --min-age 30m0s to 2019-06-13 09:06:44.438200373 +0200 CEST m=-1799.994050761 2019/06/13 09:36:44 DEBUG : rclone: Version "v1.47.0-123-g939b19c3-beta" starting with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone-beta/.rclone.conf" "move" "/mnt/user0/rclone_upload/google_vfs/" "gdrive_media_vfs:" "-vv" "--drive-chunk-size" "512M" "--checkers" "3" "--fast-list" "--transfers" "2" "--exclude" ".unionfs/**" "--exclude" "*fuse_hidden*" "--exclude" "*_HIDDEN" "--exclude" ".recycle**" "--exclude" "*.backup~*" "--exclude" "*.partial~*" "--delete-empty-src-dirs" "--fast-list" "--bwlimit" "9500k" "--tpslimit" "3" "--min-age" "30m"] 2019/06/13 09:36:44 DEBUG : Using config file from "/boot/config/plugins/rclone-beta/.rclone.conf" 2019/06/13 09:36:44 INFO : Starting bandwidth limiter at 9.277MBytes/s 2019/06/13 09:36:44 INFO : Starting HTTP transaction limiter: max 3 transactions/s with burst 1 2019/06/13 09:36:45 DEBUG : tl1q5apobimhco7a7eht612i0c: Skipping undecryptable file name: Bad PKCS#7 padding - too long 2019/06/13 09:36:45 INFO : Encrypted drive 'gdrive_media_vfs:': Waiting for checks to finish 2019/06/13 09:36:45 INFO : Encrypted drive 'gdrive_media_vfs:': Waiting for transfers to finish 2019/06/13 09:36:45 INFO : Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA - Errors: 0 Checks: 0 / 0, - Transferred: 0 / 0, - Elapsed time: 1.3s 2019/06/13 09:36:45 DEBUG : 6 go routines active 2019/06/13 09:36:45 DEBUG : rclone: Version "v1.47.0-123-g939b19c3-beta" finishing with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone-beta/.rclone.conf" "move" "/mnt/user0/rclone_upload/google_vfs/" "gdrive_media_vfs:" "-vv" "--drive-chunk-size" "512M" "--checkers" "3" "--fast-list" "--transfers" "2" "--exclude" ".unionfs/**" "--exclude" "*fuse_hidden*" "--exclude" "*_HIDDEN" "--exclude" ".recycle**" "--exclude" "*.backup~*" "--exclude" "*.partial~*" "--delete-empty-src-dirs" "--fast-list" "--bwlimit" "9500k" "--tpslimit" "3" "--min-age" "30m"] i think is better isn't it? i tried to run cleanup plugin he is the result: Script location: /tmp/user.scripts/tmpScripts/rclone_custom_plugin/script Note that closing this window will abort the execution of this script 13.06.2019 09:39:28 INFO: starting unionfs cleanup. find: '/mnt/user/mount_unionfs/google_vfs/.unionfs': No such file or directory find: '/mnt/user/mount_unionfs/google_vfs/.unionfs': No such file or directory
  20. hello i followed step by step be sure of that
  21. as you can see here this is a messup i don't know why i have one .appdata (i just deleted) one appdata one rclone_upload one mount_rclone at root unraid user one mount_rclone at root unraid user appdata other rclone i have the same kind of problem with script cleanup unionfs the problem is the script check root mnt user appdata and i have nothing there no?
  22. ok like this? 1- i run on background mount 2- i run script upload 3- i get this : Script location: /tmp/user.scripts/tmpScripts/rclone_upload_plugin/script Note that closing this window will abort the execution of this script 13.06.2019 00:05:22 INFO: rclone not installed - will try again later. 4- here is my upload script : #!/bin/bash ####### Check if script already running ########## if [[ -f "/mnt/user/appdata/other/rclone/rclone_upload" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Exiting as script already running." exit else touch /mnt/user/appdata/other/rclone/rclone_upload fi ####### End Check if script already running ########## ####### check if rclone installed ########## if [[ -f "/mnt/user/mount_rclone/google_vfs/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: rclone installed successfully - proceeding with upload." else echo "$(date "+%d.%m.%Y %T") INFO: rclone not installed - will try again later." rm /mnt/user/appdata/other/rclone/rclone_upload exit fi ####### end check if rclone installed ########## # move files rclone move /mnt/user/rclone_upload/google_vfs/ gdrive_media_vfs: -vv --drive-chunk-size 512M --checkers 3 --fast-list --transfers 2 --exclude .unionfs/** --exclude *fuse_hidden* --exclude *_HIDDEN --exclude .recycle** --exclude *.backup~* --exclude *.partial~* --delete-empty-src-dirs --fast-list --bwlimit 9500k --tpslimit 3 --min-age 30m # remove dummy file rm /mnt/user/appdata/other/rclone/rclone_upload exit thanks for your patience i will be lost without you guys edit: it's maybe because my folder is not at the good place like i said 1 page ago in script this is the location it checked /mnt/user/rclone_upload/google_vfs/ gdrive_media_vfs