Guide: How To Use Rclone To Mount Cloud Drives And Play Files


DZMM

Recommended Posts

7 minutes ago, privateer said:

I've been successfully running the original version (unionfs) for a while and finally decided to make the plunge to team drive, service accounts, and mergerfs.

 

While trying to upgrade, I ran the following command as listed on the AutoRclone git page:


sudo git clone https://github.com/xyou365/AutoRclone && cd AutoRclone && sudo pip3 install -r requirements.txt

The output for this command resulted in an error: 


sudo: pip3: command not found

The rest of the command worked fine. Any idea what's going on here?

You don't have pip installed on  your server. Get it through nerdpack.

Link to comment
5 minutes ago, Kaizac said:

You don't have pip installed on  your server. Get it through nerdpack.

python-pip-20.0.2-x86_64-1.txz is installed on my server. Only one I see with pip in it (unless I've missed something). Shouldn't need to reboot or anything after an install right?

 

pip3 returns this error:

Traceback (most recent call last):
  File "/usr/bin/pip3", line 6, in <module>
    from pkg_resources import load_entry_point
ModuleNotFoundError: No module named 'pkg_resources'

installed the python-setuptools package as well and now get this error when running  'sudo pip3 install -r requirements.txt'

 

ERROR: Command errored out with exit status 1:
     command: /usr/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-4jslat6w/progress/setup.py'"'"'; __file__='"'"'/tmp/pip-install-4jslat6w/progress/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-4jslat6w/progress/pip-egg-info
         cwd: /tmp/pip-install-4jslat6w/progress/
    Complete output (11 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/usr/lib64/python3.8/site-packages/setuptools/__init__.py", line 19, in <module>
        from setuptools.dist import Distribution, Feature
      File "/usr/lib64/python3.8/site-packages/setuptools/dist.py", line 36, in <module>
        from setuptools import windows_support
      File "/usr/lib64/python3.8/site-packages/setuptools/windows_support.py", line 2, in <module>
        import ctypes
      File "/usr/lib64/python3.8/ctypes/__init__.py", line 7, in <module>
        from _ctypes import Union, Structure, Array
    ImportError: libffi.so.7: cannot open shared object file: No such file or directory
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

 

Edited by privateer
Link to comment

 

4 minutes ago, privateer said:

python-pip-20.0.2-x86_64-1.txz is installed on my server. Only one I see with pip in it (unless I've missed something). Shouldn't need to reboot or anything after an install right?

 

pip3 returns this error:


Traceback (most recent call last):
  File "/usr/bin/pip3", line 6, in <module>
    from pkg_resources import load_entry_point
ModuleNotFoundError: No module named 'pkg_resources'

 

Then just open console and type "python3 install pip"

Link to comment

Newest problem:

Can't install everything in the requirements package. I went to install the packages manually, one at a time and I get the following error on several

Using cached rsa-4.2.tar.gz (46 kB)
    ERROR: Command errored out with exit status 1:
     command: /usr/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-8jf7sxhg/rsa/setup.py'"'"'; __file__='"'"'/tmp/pip-install-8jf7sxhg/rsa/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-8jf7sxhg/rsa/pip-egg-info
         cwd: /tmp/pip-install-8jf7sxhg/rsa/
    Complete output (11 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/usr/lib64/python3.8/site-packages/setuptools/__init__.py", line 19, in <module>
        from setuptools.dist import Distribution, Feature
      File "/usr/lib64/python3.8/site-packages/setuptools/dist.py", line 36, in <module>
        from setuptools import windows_support
      File "/usr/lib64/python3.8/site-packages/setuptools/windows_support.py", line 2, in <module>
        import ctypes
      File "/usr/lib64/python3.8/ctypes/__init__.py", line 7, in <module>
        from _ctypes import Union, Structure, Array
    ImportError: libffi.so.7: cannot open shared object file: No such file or directory
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Am I missing something else from the nerdpack? I've got python2, python 3, python-pip, and python-setuptools. Thanks for hand holding me along - can't believe I'm struggling so much...

 

 

Once I'm done I'll condense my posts down to a single post with steps I took to solve, so if others run into these issues they can see how I solved it, and hopefully find the post using the search function.

Link to comment
18 minutes ago, privateer said:

Newest problem:

Can't install everything in the requirements package. I went to install the packages manually, one at a time and I get the following error on several


Using cached rsa-4.2.tar.gz (46 kB)
    ERROR: Command errored out with exit status 1:
     command: /usr/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-8jf7sxhg/rsa/setup.py'"'"'; __file__='"'"'/tmp/pip-install-8jf7sxhg/rsa/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-8jf7sxhg/rsa/pip-egg-info
         cwd: /tmp/pip-install-8jf7sxhg/rsa/
    Complete output (11 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/usr/lib64/python3.8/site-packages/setuptools/__init__.py", line 19, in <module>
        from setuptools.dist import Distribution, Feature
      File "/usr/lib64/python3.8/site-packages/setuptools/dist.py", line 36, in <module>
        from setuptools import windows_support
      File "/usr/lib64/python3.8/site-packages/setuptools/windows_support.py", line 2, in <module>
        import ctypes
      File "/usr/lib64/python3.8/ctypes/__init__.py", line 7, in <module>
        from _ctypes import Union, Structure, Array
    ImportError: libffi.so.7: cannot open shared object file: No such file or directory
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Am I missing something else from the nerdpack? I've got python2, python 3, python-pip, and python-setuptools. Thanks for hand holding me along - can't believe I'm struggling so much...

 

 

Once I'm done I'll condense my posts down to a single post with steps I took to solve, so if others run into these issues they can see how I solved it, and hopefully find the post using the search function.

Not sure this will help you at all, but I just ran all the optional/pre-req stuff in a Linux Mint VM, and just copied over the service accounts files. Only because I wanted to test this thing out without really making too many changes to my UnRaid install. Just a thought.

Link to comment

Just tried that route. I get much further, but when I put the auth key in from google I get an error:

 

File "gen_sa_accounts.py", line 23, in <module>

  download_keys=args.download_keys

File "gen_sa_accounts.py", line 175, in serviceaccountfactory

  with open(token, 'wb') as t:

PermissionError: [Errno 13] Permission denied: 'token.pickle'

Link to comment
3 minutes ago, privateer said:

Just tried that route. I get much further, but when I put the auth key in from google I get an error:

 

File "gen_sa_accounts.py", line 23, in <module>

  download_keys=args.download_keys

File "gen_sa_accounts.py", line 175, in serviceaccountfactory

  with open(token, 'wb') as t:

PermissionError: [Errno 13] Permission denied: 'token.pickle'

did you run as sudo?

Link to comment
1 hour ago, privateer said:

well that was foolish.

 

I just can't seem to win today...  Now I get this when running 'python3 add_to_google_group.py -g sa@[mydomain]'

 

File "add_to_google_group.py", line 43, in <module>

  flow = InstalledAppFlow.from_client_secrets_file(credentials[0], scopes=[IndexError: list index out of range

 

I made a google group with the email address sa@[mydomain].

Sorry - that's about the end of my knowledge in this arena. 

 

@DZMM - when using the scripts with the Backup option, is there a way to mount it so that it doesn't look like this? mount_rclone/gdrive_media_vfs/backup/gdrive_media_vfs/Movies/Movie1/ ?

 

 

Link to comment

Hopefully progressing onward...but with a new issue.

 

Where should I get a copy of the sa_gdrive.json file? I have the remotes but not sure about that file...

 

I don't know what the SharedTeamDriveSrcID or the SharedTeamDriveDstID are. Is the DstID the folder inside the teamdrive where I'm going to store things (e.g. teamdrivefolder/crypt)? What should go here...wondering if this is why I don't have the .json file.

Edited by privateer
added to the end
Link to comment
2 hours ago, privateer said:

Hopefully progressing onward...but with a new issue.

 

Where should I get a copy of the sa_gdrive.json file? I have the remotes but not sure about that file...

 

I don't know what the SharedTeamDriveSrcID or the SharedTeamDriveDstID are. Is the DstID the folder inside the teamdrive where I'm going to store things (e.g. teamdrivefolder/crypt)? What should go here...wondering if this is why I don't have the .json file.

Did you read the github of DZMM?

https://github.com/BinsonBuzz/unraid_rclone_mount


 

Quote

 

Optional: Create Service Accounts (follow steps 1-4).To mass rename the service accounts use the following steps:

Place Auto-Genortated Service Accounts into /mnt/user/appdata/other/rclone/service_accounts/

Run the following in terminal/ssh

Move to directory: cd /mnt/user/appdata/other/rclone/service_accounts/

Dry Run:

n=1; for f in *.json; do echo mv "$f" "sa_gdrive_upload$((n++)).json"; done

Mass Rename:

n=1; for f in *.json; do mv "$f" "sa_gdrive_upload$((n++)).json"; done

 

 

Link to comment
5 hours ago, Kaizac said:

Did you read the github of DZMM?

https://github.com/BinsonBuzz/unraid_rclone_mount


 

 

Yes I read it, and yes I ran those commands.

 

the files I have are named sa_gdrive_upload[X].json, but there's no sa_gdrive.json file in there. They are in the correct folder and there's 100 of them. This is the error I've been getting:

 

Failed to create file system for "gdrive_media_vfs:": failed to make remote gdrive:"crypt" to wrap: drive: failed when making oauth client: error opening service account credentials file: open /mnt/user/appdata/other/rclone/service_accounts/sa_gdrive.json: no such file or directory

Link to comment
4 minutes ago, privateer said:

Yes I read it, and yes I ran those commands.

 

the files I have are named sa_gdrive_upload[X].json, but there's no sa_gdrive.json file in there. They are in the correct folder and there's 100 of them. This is the error I've been getting:

 

Failed to create file system for "gdrive_media_vfs:": failed to make remote gdrive:"crypt" to wrap: drive: failed when making oauth client: error opening service account credentials file: open /mnt/user/appdata/other/rclone/service_accounts/sa_gdrive.json: no such file or directory

You are renaming them to gdrive_upload.json through the renaming DZMM mentions. So if you want them to be called sa_gdrive.json you have to define that in your rename script.

Dry Run:
n=1; for f in *.json; do echo mv "$f" "sa_gdrive_upload$((n++)).json"; done

Mass Rename:
n=1; for f in *.json; do mv "$f" "sa_gdrive_upload$((n++)).json"; done

 

Don't just copy and paste the codes in githubs, but also try to understand what they are doing. Otherwise you have to idea where to troubleshoot and end up breaking your setup.

Link to comment

Thanks - this helped. I was expecting a file with that name was going to tell rclone how to use the numbered files. I didn't understand that the cycling of remotes didn't require it's own json file.

 

The first time this came out I figured it out quickly, but adding team drives and moving from unionfs turned me into a struggler! Thanks for yours (and everyone else's) patience. It's appreciated. Let me know if you have any thoughts on how I can help pay all the help I've recently been given forward.

  • Like 1
Link to comment

@DZMM 

Sorry I haven't been posting much (still in a cross-country move).

 

Currently I've dropped mergerfs to play around with union rclone backend. But my stable servers are now using "shortcuts" to combine all my team drives to a single my drive mount point. (As recommended by some developers of cloudbox/cloudplow). 

 

Once rclone union is a bit more mature/ironed-out I'll probably drop shortcuts and have the rclone union manage it all

 

 

Link to comment
16 hours ago, watchmeexplode5 said:

@DZMM 

Sorry I haven't been posting much (still in a cross-country move).

 

Currently I've dropped mergerfs to play around with union rclone backend. But my stable servers are now using "shortcuts" to combine all my team drives to a single my drive mount point. (As recommended by some developers of cloudbox/cloudplow). 

 

Once rclone union is a bit more mature/ironed-out I'll probably drop shortcuts and have the rclone union manage it all

 

 

@watchmeexplode5Are you having any joy with union?  I just had another go and I've encountered 2 show-stopping issues:

 

1. Changes to the local folder aren't picked up in a timely manner

2. Changes to remotes, EVEN if via rclone e.g. via the mount aren't picked up

 

16 hours ago, watchmeexplode5 said:

But my stable servers are now using "shortcuts" to combine all my team drives to a single my drive mount point.

Have you got a link for this please?

Link to comment

@DZMM

I've encountered the same issues with union. The changes not being picked up on mounts being the most problematic issue. 

 

Regarding shortcuts: there is a few posts here Rclone forum and on Google's blog.

 

Currently shortcuts "work" but I wouldn't recommend it. It adds an unnecessary layer of complication and you have to be very careful with file vs shortcut removal. I know a few people that have lost data when they thought they were deleting a shortcut but actually deleted files. 

 

I was experimenting with it because some higher ups at cloudbox use that setup. Currently it's easier/safer to just user mergerfs. I don't see downside to merger vs. shortcuts. 

Link to comment
On 6/6/2020 at 9:51 AM, DZMM said:

@TecneoI haven't had a chance to play with union this week - have you had any progress before I start?

Nope I didn't, I've been drowned at work lately.

 

I can't even get sonarr nor radarr to write properly to the mount (on the local side) because of the partials files thing they do so I gave up.

 

At the moment I'm using union mount though and it's rock solid.

I therefore lost the flexibility of your philosophy which I miss because I'm now writing straight to google drive when I liked the ability to stream latest downloaded files from disks until I decieded to move them...

 

 

Link to comment
8 minutes ago, Tecneo said:

Nope I didn't, I've been drowned at work lately.

 

I can't even get sonarr nor radarr to write properly to the mount (on the local side) because of the partials files thing they do so I gave up.

 

At the moment I'm using union mount though and it's rock solid.

I therefore lost the flexibility of your philosophy which I miss because I'm now writing straight to google drive when I liked the ability to stream latest downloaded files from disks until I decieded to move them...

 

 

 
 
 

I had time to play at the weekend and there's unfortunately a few showstoppers - 1) it doesn't poll the local folder for changes and 2) it doesn't poll the remotes either.   2) might be because 1) isn't working, so if 1) can be fixed 2) might be fixed.  @ncw from rclone is having a look, so we might have a fix/workaround.  I hope he finds one as playback is definitely better than mergerfs I guess from removing the middleman.

Link to comment

Yep I saw the feed on rclone forums.

 

An idea came up :

I'll try to give union a gdrive and an sftp remote which will actually be the host sftp to see if it's enough to workaround 1)

 

EDIT :

Nope, not picking up local changes... that would have been too easy

Definitly hope ncw find a fix as well

Edited by Tecneo
Avoid double post
Link to comment
On 6/15/2020 at 5:11 PM, DZMM said:

I had time to play at the weekend and there's unfortunately a few showstoppers - 1) it doesn't poll the local folder for changes and 2) it doesn't poll the remotes either.   2) might be because 1) isn't working, so if 1) can be fixed 2) might be fixed.  @ncw from rclone is having a look, so we might have a fix/workaround.  I hope he finds one as playback is definitely better than mergerfs I guess from removing the middleman.

2 Questions. 

 

1) When you say it doesnt poll local folder and remotes for changes, does it still pickup files added to the merged folder instantly? My current set up means I only ever add to the mount_mergerfs folder, so could I use union for this case?

 

2) What are the advantages of using union over Mergerfs? I notice you say quicker load times but is that all? my current times dont seem too bad. 

 

EDIT:

I tested it myself and got 1 to work, so for my use case this seems fine.

As a bonus question then will you modify a version of your scripts for union?

 

I tried a hardlink too in filebot and it came back with "Function not implemented"

Edited by JimmyFigus
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.