Amazon Cloud Drive docker - ACD_CLI and Encfs


Recommended Posts

Has anyone managed to mount Amazon Cloud Drive as an encrypted drive?  I've found various interesting posts on the internet like this one https://ryanclouser.com/2015/12/01/Amazon-Cloud-Drive-EncFS-and-acd-cli/ and I've attempted to do this on windows a few times (e.g using Netdrive to mount amazon and a windows port of EncFS to mirror it), but the performance was terrible.  There's an all-in-one solution in development https://stablebit.com/ , but their amazon implementation is throttled at the moment as they are not getting on well with Amazon.

 

I've found a few dockers like this one https://hub.docker.com/r/joshhogle/ecloudfs/ - I just wanted to know if anyone has any experience in this area?  It's one of the first things I want to try when I get my server built (just got 2 E5-2670s so far - other parts on the way/being selected) as for $60 for unlimited storage with Amazon I could do my backup jobs rather than using Crashplan, as well as having 'unlimited' file storage, use it for Plex Cloud Sync etc etc

Link to comment
  • 2 weeks later...
  • 2 weeks later...

Interested in this as well. I dropped Crashplan since it was not just working for me.  Have about 25TB of data already, Crashplan stopped for me at 10TB, saying everything completed. 

 

I saw someone on Amazon Cloud with 96TB saying it's awesome AND fast !

 

having the rclone using encryption (seems to be implemented already?) is just even more better!

Link to comment
  • 4 weeks later...

I've been looking at this for a while, tried some options like the OP but nothing practical. I stated looking at it again with the recent announcement of Plex Cloud. I was thinking about trying the acd-cli+encfs route but it sounds like rclone might be better. Anything tangible yet?

 

Edit: Answered my own question. I will be testing this.

https://lime-technology.com/forum/index.php?topic=52033.0

Link to comment

Hi,

 

I have been playing around with ACD options for the last couple of weeks, I am mostly looking at the mount options, as I want to be able to mount a drive and have a merged view of local and cloud storage.

 

rclone for me was too slow, just browsing folders is not quick at all, acd_cli maintains a local db of the folder tree so browses much faster. rclone mount also can only do sequential reads (at least at the moment) and this was an issue for me.

 

At the moment I have acd_cli and encfs working, I am still testing but so far the performance has been good, you can copy to\from the cloud mount, also most people recommend against it to ensure file integrity, so if I want to upload big chunks I unmount the acd folder and do the upload using acd_cli upload.

 

I am still trying to work out how to do a merged mount of the local\cloud data. I will basically have the same data in the cloud and local, but I will then remove\archive some local content, so will reference the cloud if it doesn't exist locally. I found a guide for using UnionFS-Fuse, but since it is a similar process to the local user folder and how it references cache and local discs I am trying to work out if I can do it without installing unionfs-fuse.

 

I made a post in another thread for anyone interested (http://lime-technology.com/forum/index.php?topic=45338.msg502980#msg502980)

 

Cheers,

Wob

Link to comment

well since Marcus hasn't yet got around to updating his stuff, I went ahead and got this all working following the amc.ovh tutorials. I pass the union mount to docker volume and therefore able to see the files in docker while having to perform the rest of the operations on the host. Not ideal but gets the job done. I wrote some of the commands as zsh functions to keep it simple.

 

EDIT: Marcus did https://hub.docker.com/r/msh100/acd-docker/ start to do this update using a new single-container. However, I am not sure how this works as I need to pass something (the union mount I presume) to my other containers for them to write and delete items.

Link to comment

Hi 2devnull,

 

Can I ask how you went about installing unionfs? I couldn't find a nice slackware install for it and didn't want to go installing make just for that task.

 

I ended up using the overlayfs built into the kernel, it works fine for reading, but it doesn't seem to like fuse mounts as RW so my merged mount is just Read Only.

 

I scripted my install with a basic start up script, but it doesn't seem to like mapping the acd_cli during boot. I have already copied the oauth in place, but still get

 

For the one-time authentication a browser (tab) will be opened at https://tensile-runway-92512.appspot.com/.
Please accept the request and save the plaintext response data into a file called "oauth_data" in the directory "/.cache/acd_cli".

 

Running the script after boot it all works fine, I have tried adding some delays, but no luck yet.

 

I am running acd_cli with -fg (foreground) using screen, as I was seeing the mount being a little buggy, timing out etc, as this was a noted fix on the issues on github.

 

Here is a copy of my startup script for anyone interested.

 

#!/usr/bin/bash
LOGFILE=/boot/acd_cli/logs/cloudstore-$(date "+%Y%m%d").log
echo CloudSotre log $(date) $'\r'$'\r' >> $LOGFILE 2>&1
echo "Starting Cloud Mounts" $'\r'>> $LOGFILE 2>&1

#Copy oauth file to system
mkdir -p /root/.cache/acd_cli/
cp /boot/acd_cli/config/oauth_data /root/.cache/acd_cli/oauth_data &&

#Install dependancies
upgradepkg --install-new /boot/acd_cli/install/boost-1.59.0-x86_64-1.txz >> $LOGFILE 2>&1
upgradepkg --install-new /boot/acd_cli/install/rlog-1.4-x86_64-1pw.txz >> $LOGFILE 2>&1
upgradepkg --install-new /boot/acd_cli/install/slocate-3.1-x86_64-4.txz >> $LOGFILE 2>&1

#Install encfs
upgradepkg --install-new /boot/acd_cli/install/encfs-1.8.1-x86_64-1gv.txz >> $LOGFILE 2>&1

#Install acd_cli
pip3 install --upgrade git+https://github.com/yadayada/acd_cli.git >> $LOGFILE 2>&1

#Sleep for 10s and then run a acd_cli sync
sleep 10s &&
acdcli sync >> $LOGFILE 2>&1

#Mount Amazon Cloud Drive (using screen)
echo Mounting Amazon Cloud Drive >> $LOGFILE 2>&1
screen -S acdcli -d -m /usr/bin/acd_cli -nl mount -fg -ao --uid 99 --gid 100 --modules="subdir,subdir=/Plex" /mnt/cache/cloud/.acd >> $LOGFILE 2>&1

#Mount Decrypted view of ACD
echo Mounting ENCFS points >> $LOGFILE 2>&1
echo <password> | ENCFS6_CONFIG='/boot/acd_cli/config/encfs.xml' encfs -S -o ro -o allow_other -o uid=99 -o gid=100 /mnt/cache/cloud/.acd/ /mnt/cache/cloud/acd/ >> $LOGFILE 2>&1

#Mount Encrypted view of Local Media (Use for uploading Data to ACD)
echo <password>| ENCFS6_CONFIG='/boot/acd_cli/config/encfs.xml' encfs -S --reverse -o ro -o allow_other -o uid=99 -o gid=100 /mnt/user/Media/ /mnt/cache/cloud/.local/ >> $LOGFILE 2>&1

#Overlay Mount with Local Data taking preference. (Read Only)
echo Mounting Overlay point >> $LOGFILE 2>&1
mount -t overlay -o lowerdir=/mnt/user/Media/:/mnt/user/cloud/acd/ overlay /mnt/user/cloud/media/ >> $LOGFILE 2>&1

#Restart the plex docker (so it can see data in the mount point)
docker restart plex

 

Cheers,

Wob

Link to comment

Thanks for the response, are you running RacherOS on another box to your unRAID, or in a VM?

 

Regards,

Wob

 

@Wob76 - I use RancherOS as the base to run my dockers which provides a selection of console (I'm using the Ubuntu one). Installing the unionfs therefore is just the same as installing on Ubuntu (install the unionfs-fuse package)

Link to comment

I am using the --reverse option (the second encfs mount in my script) that mount is purely there for uploading to ACD, and yeah, then you are only hit with the encryption overhead during the upload process.

 

I end up with the following mount points from my script

encfs on /mnt/cache/cloud/acd type fuse.encfs (rw,nosuid,nodev,allow_other,default_permissions) - The decrypted view of ACD
encfs on /mnt/cache/cloud/.local type fuse.encfs (ro,nosuid,nodev,allow_other,default_permissions) - The encrypted view of my local media folder *the reverse mount*
overlay on /mnt/user/cloud/media type overlay (ro,lowerdir=/mnt/user/Media/:/mnt/user/cloud/acd/) - The overlay mount with local media being the top layer and acd below that.
ACDFuse on /mnt/cache/cloud/.acd type fuse.ACDFuse (rw,nosuid,nodev,allow_other) - The encrypted ACD mount point

 

I solved the issue with my script not working at start up.

 

I enabled verbose on the acd_cli sync command and it was looking in /.cache/acd_cli for the oauth, so no home variable was being parsed.

 

I could move the file to that location, but it I want to be able to call acd_cli from the command line for uploading etc.

 

So I just added the following to the start of my script.

 

HOME=/root

 

If anyone has been able to install unionfs-fused on unRAID I would love to hear about it.

 

Wob

Link to comment

Glad you got to the bottom of that problem. I presume the unionfs-fuse mount will only need to combine the media directory (RW) with the decrypted view of the acd directory (RO).

 

Yep, that would do the trick.

 

The solution I have is fine for the most part, but if I want to track the collections (including ACD only files) with the likes of CP and SR then I need them to point to a mount that is RW. At the moment they will only see local content, as they will point just to local data as they need RW access for renaming. It is not a big issue for SR as it will just tag files that are moved to ACD as archived, but CP would no longer see track that are moved to ACD.

 

At the moment it is mostly theoretical, I only have a small amount of data in ACD for testing, it appears to stream fine. I am also hesitant to have ACD the only copy of something, so if I do want to remove something local I will be looking at a backup of some kind before I remove my local copy. At the moment, local space is not an issue.

 

My plan is to maintain local data for a least a few years, then have older stuff pulled from ACD on the rare occasion that someone views it.

Link to comment
  • 2 weeks later...

Glad you got to the bottom of that problem. I presume the unionfs-fuse mount will only need to combine the media directory (RW) with the decrypted view of the acd directory (RO).

 

Yep, that would do the trick.

 

The solution I have is fine for the most part, but if I want to track the collections (including ACD only files) with the likes of CP and SR then I need them to point to a mount that is RW. At the moment they will only see local content, as they will point just to local data as they need RW access for renaming. It is not a big issue for SR as it will just tag files that are moved to ACD as archived, but CP would no longer see track that are moved to ACD.

 

At the moment it is mostly theoretical, I only have a small amount of data in ACD for testing, it appears to stream fine. I am also hesitant to have ACD the only copy of something, so if I do want to remove something local I will be looking at a backup of some kind before I remove my local copy. At the moment, local space is not an issue.

 

My plan is to maintain local data for a least a few years, then have older stuff pulled from ACD on the rare occasion that someone views it.

 

Are there updates to your scripts? I would like to test this but if there are updated scripts, I would like to test.

 

Thanks.

Link to comment

I manually run this command in screen.

 

acd_cli -nl mount -fg -ao --uid 99 --gid 100 --modules="subdir,subdir=/encfs" /mnt/user/Amazon/.acd

 

Nothing happens.

 

What is the expected response? Is it usual to take a long time? How long does it usually take?

 

Edit:

 

Okay, I checked the directory. The command works.

Link to comment

I found this if it helps anyone.  Hoping for some kind of docker to make everything easier if possible.

 

https://enztv.wordpress.com/2016/10/19/using-amazon-cloud-drive-with-plex-media-server-and-encrypting-it/

 

That looks interesting but I wonder what the performance is like, especially considering how many programs it uses. It seems unnecessarily complex. I've seem tutorials use just rclone and others use a combination on acd_cli and encfs. What benefit is there from combining them all and throwing unionfs on top of it?

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.