workermaster

I tried running the same command directly on the server, but that made no differnce. I also tried removing all files in the project folder and start from scratch, to rule out any mistakes or other weird things since I first tried most commands on Windows. I got to the same point where I am now. Since I am stuck at the first step of this part: I decided to try and do step 2. Step 2 is completed and I think that the group I made, should work for the next few steps. I did make sure that the credentials file used in activating the Directory API is the correct one. I tried running step 3, but that gave me this error: I think that I first need to get step 1 to work. Or figure out how to manally add them to the group

I have already found the next problem. I did try and search for it on the Github link you gave me, but couldn't find it. I have now created a new api according to step 3 of the process (here ) and am trying to run the command: python quickstart.py that I have to run according to this page: https://developers.google.com/admin-sdk/directory/v1/quickstart/python I now get the error that there are insufficient permissions. I expected a screen where I had to give permission to rclone. Since I thought that the problem was most likely the previous command: python3 gen_sa_accounts.py --quick-setup -1 because I am not 100% if that was the option I should have used, I tried figuring out if the service accounts belong to that existing project. They appear to do so. Here is one of the accounts:

Thanks! That solved the problem. I am now going to try and follow the next steps and have dinner.

I ran that command and then ran the requirements again. It worked. The errors were gone. I then looked again at step 2 of creating the service accounts, and since I already have a project (you need to make one to enable the drive api) I thought this command: python3 gen_sa_accounts.py --quick-setup -1 was the best one to use. I entered that into the console and this time, it told me to go to a link to give access to the script. This is where I get my next problem. The request seems to be invalid: It says that the access is denied and that I should contact the developer for this problem. I tried doing this on a pc that has the Unraid UI open, and also on the server itself (booted into gui mode). I get the same error. I also tried running the other commands: python3 gen_sa_accounts.py --quick-setup 1 --new-only python3 gen_sa_accounts.py --quick-setup 1 to see if they gave a different result, but nothing helped. Do you know why the request is invalid?

I get this:

Right. I got pip working (just had to install it in the nerdpack) I am now getting a lot of errors while running the command. I didn't get these before on Windows. What could be wrong here?

I didn't know I could do that. I am going to try that in a moment. EDIT: Trying to run this now. I couldn't find a Python download in the community apps, but then remembered that the nerdpack could have it. I enabled and installed it there: I also moved the project from my windows PC to Disk 1 on the server. But I have no idea how to run the code (I know the directory is not correct in the above screenshot, but it doesn't even find the Python commands) I have never ran anything like this on the Unraid console before, so sorry if I make obvious mistakes. Should the above be enough to work, or should I run these commands in Unraid: EDIT2: I have figured out that I can do this; to start Python. I am now in the project folder, but can't run the needed commands: I now have to figure out how to get pip working

In the meantime, I have tried to copy all rclone files into the project folder and run the python3 gen_sa_accounts.py --quick-setup -1 command. It did not work and gave me the same module error. I then tried moving all files in the project folder into the Python installation folder, to rule out any problems with the Windows path. Still the same error. Then I tried to run step 1 and 2 again from github: https://github.com/xyou365/AutoRclone to make sure that I did everything right. I could not find a mistake anywhere. When I run the last confirmation step for the api, it doesn't ask me to login, but shows me a long string of letters and numbers in de console, but the website mentions that you only have to login the first time. As far as I can tell, everything should be setup correctly. This leaves me with the idea that the problem is with the rclone install, since the rclone I have, seems to be a portible one and not one that you need to install. But I did get it from the link in the manual (https://rclone.org/downloads/), so that should be the correct one.

I downloaded rclone, but it seems to not be a program that is installed. I see no installer, but only a .exe that tells me to access it from a cmd screen. Starting it from a cmd screen also does not give me the option to install it: Do I have the wrong rclone here? I do have a credentials file. Got it from here after enabling the api: Then renamed that file and it is in the project folder:

I tried that, but it made no difference. It seems to be installed, but for some reason, is not usable.

I have no programming knowledge, so to me, the missing module error I get in the post above, looks like it has someting to do with the script I am trying to execute, and not with the Python installation. That makes it impossible for me to Google what is going wrong, and since I can't read code, I am stuck.

I followed this link to try and add Python to the Windows path. My system looks a little different since I am using Windows 11, but I think that I managed to do what they asked. https://phoenixnap.com/kb/how-to-install-python-3-windows Do I even need to use the Python console? It seems that every command can be ran from the CMD. I then went back to the 4 steps to create service accounts. I am at step 2: But when I run that command, I still get this error: I have no idea how to troubleshoot this. Do you, or anyone else, know how to fix this, or help me with creating the file manually? EDIT: Could it be that you need to install rclone for creating the service accounts? One of the first steps mentions that you need to install it, but I skipped that since I have rclone running on Unraid and thought that was enough. Is it needed to install rclone on Windows in order to create the service accounts?

I'll try that tomorrow. I hope that solves my problem.

Quick question. I have given up trying to get the 100 Service Accounts to work, but is it possible to manually create 2 or 3? My whole problem is that I have no idea how to use Python and it keeps giving me weird error, but if I can manually make a few accounts, then I would be happy. If this is possible, how do I do this? Just add a few of my other Google accounts to the shared drive? Then create the accounts file myself that Python would normally do for me? How would that file look like?

I had to run the mount script twice to get it running. The first time, it generates a small logfile, and the second time, it does a lot of things and generates a large logfile. The problem is, that everytime I restart, it fails to start working again. I am going to let it keep running for now and try this again tomorrow (I will share all the logs tomorrow). I have limited the upload speed of the script to 8MB so it will keep going and not hit the 750GB limit.

I have a problem with the upload script. It no longer seems to do anything. This is the script: #!/bin/bash ###################### ### Upload Script #### ###################### ### Version 0.95.5 ### ###################### ####### EDIT ONLY THESE SETTINGS ####### # INSTRUCTIONS # 1. Edit the settings below to match your setup # 2. NOTE: enter RcloneRemoteName WITHOUT ':' # 3. Optional: Add additional commands or filters # 4. Optional: Use bind mount settings for potential traffic shaping/monitoring # 5. Optional: Use service accounts in your upload remote # 6. Optional: Use backup directory for rclone sync jobs # REQUIRED SETTINGS RcloneCommand="move" # choose your rclone command e.g. move, copy, sync RcloneRemoteName="gdrive_vfs" # Name of rclone remote mount WITHOUT ':'. RcloneUploadRemoteName="gdrive_upload_vfs" # If you have a second remote created for uploads put it here. Otherwise use the same remote as RcloneRemoteName. LocalFilesShare="/mnt/user/local" # location of the local files without trailing slash you want to rclone to use RcloneMountShare="/mnt/user/mount_rclone" # where your rclone mount is located without trailing slash e.g. /mnt/user/mount_rclone MinimumAge="15m" # sync files suffix ms|s|m|h|d|w|M|y ModSort="ascending" # "ascending" oldest files first, "descending" newest files first # Note: Again - remember to NOT use ':' in your remote name above # Bandwidth limits: specify the desired bandwidth in kBytes/s, or use a suffix b|k|M|G. Or 'off' or '0' for unlimited. The script uses --drive-stop-on-upload-limit which stops the script if the 750GB/day limit is achieved, so you no longer have to slow 'trickle' your files all day if you don't want to e.g. could just do an unlimited job overnight. BWLimit1Time="01:00" BWLimit1="8M" BWLimit2Time="08:00" BWLimit2="8M" BWLimit3Time="16:00" BWLimit3="8M" # OPTIONAL SETTINGS # Add name to upload job JobName="_daily_upload" # Adds custom string to end of checker file. Useful if you're running multiple jobs against the same remote. # Add extra commands or filters Command1="--exclude downloads/**" Command2="" Command3="" Command4="" Command5="" Command6="" Command7="" Command8="" # Bind the mount to an IP address CreateBindMount="N" # Y/N. Choose whether or not to bind traffic to a network adapter. RCloneMountIP="192.168.1.253" # Choose IP to bind upload to. NetworkAdapter="eth0" # choose your network adapter. eth0 recommended. VirtualIPNumber="1" # creates eth0:x e.g. eth0:1. # Use Service Accounts. Instructions: https://github.com/xyou365/AutoRclone UseServiceAccountUpload="N" # Y/N. Choose whether to use Service Accounts. ServiceAccountDirectory="/mnt/user/appdata/other/rclone/service_accounts" # Path to your Service Account's .json files. ServiceAccountFile="sa_gdrive_upload" # Enter characters before counter in your json files e.g. for sa_gdrive_upload1.json -->sa_gdrive_upload100.json, enter "sa_gdrive_upload". CountServiceAccounts="15" # Integer number of service accounts to use. # Is this a backup job BackupJob="N" # Y/N. Syncs or Copies files from LocalFilesLocation to BackupRemoteLocation, rather than moving from LocalFilesLocation/RcloneRemoteName BackupRemoteLocation="backup" # choose location on mount for deleted sync files BackupRemoteDeletedLocation="backup_deleted" # choose location on mount for deleted sync files BackupRetention="90d" # How long to keep deleted sync files suffix ms|s|m|h|d|w|M|y ####### END SETTINGS ####### ############################################################################### ##### DO NOT EDIT BELOW THIS LINE UNLESS YOU KNOW WHAT YOU ARE DOING ##### ############################################################################### ####### Preparing mount location variables ####### if [[ $BackupJob == 'Y' ]]; then LocalFilesLocation="$LocalFilesShare" echo "$(date "+%d.%m.%Y %T") INFO: *** Backup selected. Files will be copied or synced from ${LocalFilesLocation} for ${RcloneUploadRemoteName} ***" else LocalFilesLocation="$LocalFilesShare/$RcloneRemoteName" echo "$(date "+%d.%m.%Y %T") INFO: *** Rclone move selected. Files will be moved from ${LocalFilesLocation} for ${RcloneUploadRemoteName} ***" fi RcloneMountLocation="$RcloneMountShare/$RcloneRemoteName" # Location of rclone mount ####### create directory for script files ####### mkdir -p /mnt/user/appdata/other/rclone/remotes/$RcloneUploadRemoteName #for script files ####### Check if script already running ########## echo "$(date "+%d.%m.%Y %T") INFO: *** Starting rclone_upload script for ${RcloneUploadRemoteName} ***" if [[ -f "/mnt/user/appdata/other/rclone/remotes/$RcloneUploadRemoteName/upload_running$JobName" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Exiting as script already running." exit else echo "$(date "+%d.%m.%Y %T") INFO: Script not running - proceeding." touch /mnt/user/appdata/other/rclone/remotes/$RcloneUploadRemoteName/upload_running$JobName fi ####### check if rclone installed ########## echo "$(date "+%d.%m.%Y %T") INFO: Checking if rclone installed successfully." if [[ -f "$RcloneMountLocation/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: rclone installed successfully - proceeding with upload." else echo "$(date "+%d.%m.%Y %T") INFO: rclone not installed - will try again later." rm /mnt/user/appdata/other/rclone/remotes/$RcloneUploadRemoteName/upload_running$JobName exit fi ####### Rotating serviceaccount.json file if using Service Accounts ####### if [[ $UseServiceAccountUpload == 'Y' ]]; then cd /mnt/user/appdata/other/rclone/remotes/$RcloneUploadRemoteName/ CounterNumber=$(find -name 'counter*' | cut -c 11,12) CounterCheck="1" if [[ "$CounterNumber" -ge "$CounterCheck" ]];then echo "$(date "+%d.%m.%Y %T") INFO: Counter file found for ${RcloneUploadRemoteName}." else echo "$(date "+%d.%m.%Y %T") INFO: No counter file found for ${RcloneUploadRemoteName}. Creating counter_1." touch /mnt/user/appdata/other/rclone/remotes/$RcloneUploadRemoteName/counter_1 CounterNumber="1" fi ServiceAccount="--drive-service-account-file=$ServiceAccountDirectory/$ServiceAccountFile$CounterNumber.json" echo "$(date "+%d.%m.%Y %T") INFO: Adjusted service_account_file for upload remote ${RcloneUploadRemoteName} to ${ServiceAccountFile}${CounterNumber}.json based on counter ${CounterNumber}." else echo "$(date "+%d.%m.%Y %T") INFO: Uploading using upload remote ${RcloneUploadRemoteName}" ServiceAccount="" fi ####### Upload files ########## # Check bind option if [[ $CreateBindMount == 'Y' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: *** Checking if IP address ${RCloneMountIP} already created for upload to remote ${RcloneUploadRemoteName}" ping -q -c2 $RCloneMountIP > /dev/null # -q quiet, -c number of pings to perform if [ $? -eq 0 ]; then # ping returns exit status 0 if successful echo "$(date "+%d.%m.%Y %T") INFO: *** IP address ${RCloneMountIP} already created for upload to remote ${RcloneUploadRemoteName}" else echo "$(date "+%d.%m.%Y %T") INFO: *** Creating IP address ${RCloneMountIP} for upload to remote ${RcloneUploadRemoteName}" ip addr add $RCloneMountIP/24 dev $NetworkAdapter label $NetworkAdapter:$VirtualIPNumber fi else RCloneMountIP="" fi # Remove --delete-empty-src-dirs if rclone sync or copy if [[ $RcloneCommand == 'move' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: *** Using rclone move - will add --delete-empty-src-dirs to upload." DeleteEmpty="--delete-empty-src-dirs " else echo "$(date "+%d.%m.%Y %T") INFO: *** Not using rclone move - will remove --delete-empty-src-dirs to upload." DeleteEmpty="" fi # Check --backup-directory if [[ $BackupJob == 'Y' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: *** Will backup to ${BackupRemoteLocation} and use ${BackupRemoteDeletedLocation} as --backup-directory with ${BackupRetention} retention for ${RcloneUploadRemoteName}." LocalFilesLocation="$LocalFilesShare" BackupDir="--backup-dir $RcloneUploadRemoteName:$BackupRemoteDeletedLocation" else BackupRemoteLocation="" BackupRemoteDeletedLocation="" BackupRetention="" BackupDir="" fi # process files rclone $RcloneCommand $LocalFilesLocation $RcloneUploadRemoteName:$BackupRemoteLocation $ServiceAccount $BackupDir \ --user-agent="$RcloneUploadRemoteName" \ -vv \ --buffer-size 512M \ --drive-chunk-size 512M \ --tpslimit 8 \ --checkers 8 \ --transfers 4 \ --order-by modtime,$ModSort \ --min-age $MinimumAge \ $Command1 $Command2 $Command3 $Command4 $Command5 $Command6 $Command7 $Command8 \ --exclude *fuse_hidden* \ --exclude *_HIDDEN \ --exclude .recycle** \ --exclude .Recycle.Bin/** \ --exclude *.backup~* \ --exclude *.partial~* \ --drive-stop-on-upload-limit \ --bwlimit "${BWLimit1Time},${BWLimit1} ${BWLimit2Time},${BWLimit2} ${BWLimit3Time},${BWLimit3}" \ --bind=$RCloneMountIP $DeleteEmpty # Delete old files from mount if [[ $BackupJob == 'Y' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: *** Removing files older than ${BackupRetention} from $BackupRemoteLocation for ${RcloneUploadRemoteName}." rclone delete --min-age $BackupRetention $RcloneUploadRemoteName:$BackupRemoteDeletedLocation fi ####### Remove Control Files ########## # update counter and remove other control files if [[ $UseServiceAccountUpload == 'Y' ]]; then if [[ "$CounterNumber" == "$CountServiceAccounts" ]];then rm /mnt/user/appdata/other/rclone/remotes/$RcloneUploadRemoteName/counter_* touch /mnt/user/appdata/other/rclone/remotes/$RcloneUploadRemoteName/counter_1 echo "$(date "+%d.%m.%Y %T") INFO: Final counter used - resetting loop and created counter_1." else rm /mnt/user/appdata/other/rclone/remotes/$RcloneUploadRemoteName/counter_* CounterNumber=$((CounterNumber+1)) touch /mnt/user/appdata/other/rclone/remotes/$RcloneUploadRemoteName/counter_$CounterNumber echo "$(date "+%d.%m.%Y %T") INFO: Created counter_${CounterNumber} for next upload run." fi else echo "$(date "+%d.%m.%Y %T") INFO: Not utilising service accounts." fi # remove dummy file rm /mnt/user/appdata/other/rclone/remotes/$RcloneUploadRemoteName/upload_running$JobName echo "$(date "+%d.%m.%Y %T") INFO: Script complete" exit No matter what I try, it doesn't want to start. It keeps saying that it is already running, but there are no logs showing that it is running. I tried removing all rclone scripts and copying new ones from the Github page, but even after a reboot, I keep getting the message that it is already running, even though it hasn't been started before.

I made a post before, but deleted that since I managed to get a bit further. It has taken me a lot of stress, but have managed to do the first few steps. I now want to run the command to actually make the service accounts. I mean this step: But for some reason, the cmd screen says that Python is not installed. I then installed it again from the Microsoft store (even though it was already installed) but now it gives me this error: I do not understand what goes wrong here, since I did run this command multiple times. Do you know what is going wrong here? EDIT: I have given up. I just need something simple and I think this might be to much for me at the moment. I will just deal with the 750GB limt.

I have taken precautions to make sure I don't lose it. I now want to configure a shared drive. What do I need to do for that? Is the only thing I need to do, to change the remote to be a shared drive and add some account somehwere on Google? What happens to the data that I have already uploaded? Is that deleted?

I have uploaded a few files to Gdrive. They are all encrypted. When I look at my Gdrive from a browser, then I only see encrypted names of folders. Say that I were to lose the mount points configuration I have, can I still restore access to these files? Or can I never decrypt them? I would also like to upload files that don't have to be encrypted. I guess that I need to make another mountpoint and not encrypt that one? Then how can I add that new mount to the upload script?

Sounds interesting. How do I set that up?

Good point. I would like to take a look into making a team drive then. But that would mean another account and more costs I assume. I don't mind having a secord or third account to upload all the data, but after the upload is done, I want to keep costs low. How do I setup a team drive? I saw the option while making the remotes, but is that all that I need to do?

I rebooted the server, and now it is uploading. No idea why it wouldn't work at first. I am now uploading a few files to test the read speeds with Plex. What would you recommend for uploading all the data? Just move all the media I already have into the local folder? To add another question. I can upload 750GB per day. What happens if I am at 740Gb already and the script is trying to upload a file of 50GB? Will It just skip it, or upload half of it?

It sounds easy, but I have a talent for still making things go wrong. This case is not an exception. I have created 2 new mounts and gave them a different client_id. The ones I had: The ones I added: I hoped this was enough and tried to run the upload script manually to see what happened, and then I got this error: Somehow, the script thinks rclone is not installed. The plugin is clearly there. I have no idea how I messed this up.

I am going to try this tomorrow. I hope I can manage to do that, but fear that I might need more help. The first post didn't mention that we needed more mounts.

Does that mean that I need a second Google workspace account? That would double the expense. I don't know what a team drive is and does. Can't I just edit the upload script to work with the 2 mounts I already have: