Guide: How To Use Rclone To Mount Cloud Drives And Play Files


DZMM

Recommended Posts

Can I get some help testing please.  V1.5.3 of rclone (remember you have to remove and reinstall the plugin to update it) now supports better caching where files can be cached locally.  I'll add a variable in for setting the cache location once it's all working, but for now can a few people try these settings in the mount script:

 

# create rclone mount
	rclone mount \
	--allow-other \
	--dir-cache-time 720h \
	--log-level INFO \
	--poll-interval 15s \
	--cache-dir=/mnt/user/downloads/rclone/tdrive_vfs/cache \
	--vfs-cache-mode full \
	--vfs-cache-max-size 500G \
	--vfs-cache-max-age 336h \
	--bind=$RCloneMountIP \
	$RcloneRemoteName: $RcloneMountLocation &

set the cache-dir to wherever is convenient.   The settings above will keep up to 500GB of files downloaded from gdrive for up to 2 weeks, with the oldest removed first when full.  I think this will work well with my kids who keep stopping and starting the same file, or when plex is indexing or doing other operations.  However, I don't think it will help majorly with playback for my setup, unless a user tries to open the same file within a few hours.  Dunno.

 

There's another new setting --vfs-read-ahead that could potentially help with forward skipping/smoother playback by downloading more data ahead of the current stream position, that we can play with as well.

 

Edit: poll-interval shortens the default 1m, so should hopefully add a bit more butter to updates.

 

Edit 2:. Initial launch times are much faster even before the cache kicks in!!

Edited by DZMM
  • Thanks 1
Link to comment
18 hours ago, DZMM said:

Can I get some help testing please.  V1.5.3 of rclone (remember you have to remove and reinstall the plugin to update it) now supports better caching where files can be cached locally.  I'll add a variable in for setting the cache location once it's all working, but for now can a few people try these settings in the mount script:

 


# create rclone mount
	rclone mount \
	--allow-other \
	--dir-cache-time 720h \
	--log-level INFO \
	--poll-interval 15s \
	--cache-dir=/mnt/user/downloads/rclone/tdrive_vfs/cache \
	--vfs-cache-mode full \
	--vfs-cache-max-size 500G \
	--vfs-cache-max-age 336h \
	--bind=$RCloneMountIP \
	$RcloneRemoteName: $RcloneMountLocation &

set the cache-dir to wherever is convenient.   The settings above will keep up to 500GB of files downloaded from gdrive for up to 2 weeks, with the oldest removed first when full.  I think this will work well with my kids who keep stopping and starting the same file, or when plex is indexing or doing other operations.  However, I don't think it will help majorly with playback for my setup, unless a user tries to open the same file within a few hours.  Dunno.

 

There's another new setting --vfs-read-ahead that could potentially help with forward skipping/smoother playback by downloading more data ahead of the current stream position, that we can play with as well.

 

Edit: poll-interval shortens the default 1m, so should hopefully add a bit more butter to updates.

 

Edit 2:. Initial launch times are much faster even before the cache kicks in!!

I am testing this now and it seems to work really well.

I also have a few users with kids that repeat the same shows all the time.

 

Does it do full read-ahead of the file to cache or only cache the contents that it downloads as per request? IE, if a user watches the first 5 min of a move, will only that 5 min be cached or will the internet connection go wild and pull the full file? Hopefully the former.

 

I had wanted to implemented a smart read-ahead cache process that would monitor the last epp watched and pre-fetch the next epp and store it in local, but now this makes managing that file thereafter so much easier. I can pre"read" the next file and it will then be stored in the cache and will be read from there as required, or removed if not consumed in x days.

 

Anything in particular you want me to test, seems to be working out of the box.

  • Like 1
Link to comment

I have been testing the VFS settings fairly extensively for the past few months on the betas. Not with unraid but just my rclone mount scripts on straight ubuntu - what caching will really help with is jumping around - fwds and backwards in a file and as you mentioned with stuff like your kids watching the same file over and over - if you were sharing with a few people then the benefits would be stuff like the game of thrones linux iso final where everyone tunes in to watch on release day. Essentially you are keeping the file local. I keep jumping between full and writes.

 

If there is anything you want tested maybe I can help.

  • Like 1
Link to comment
20 minutes ago, Spladge said:

If there is anything you want tested maybe I can help.

 

Thanks.  Can you help me play around with --buffer-size and --vfs-read-ahead please.  From the rclone docs:

 

Quote

When reading a file rclone will read --buffer-size plus --vfs-read-ahead bytes ahead. The --buffer-size is buffered in memory whereas the --vfs-read-ahead is buffered on disk.

When using this mode it is recommended that --buffer-size is not set too big and --vfs-read-ahead is set large if required.

 
 

Previously I had --buffer-size at 256MB, but based on the above I've gone with the smaller default of 16MB - I'm not sure what --vfs-read-ahead defaults to.  Whatever the defaults are, my first play times seem about 50% faster (1-2 seconds rather than 2-3 seconds) with the settings above - I want to see if setting a higher --vfs-read-ahead makes skipping forward even smoother.  I'm a bit worried though that setting it too high will reduce the hit-rate on the cache if it's filled up with content that isn't used.

 

32 minutes ago, markrudling said:

Does it do full read-ahead of the file to cache or only cache the contents that it downloads as per request? IE, if a user watches the first 5 min of a move, will only that 5 min be cached or will the internet connection go wild and pull the full file? Hopefully the former.

 

It only caches 16MB to the memory buffer (--buffer-size) and to disk whatever the --vfs-read-ahead is set to - as above I don't know what the default is. so it's the former.  I'm curious to see if setting --vfs-read-ahead to say 128MB is enough to buffer the next xx seconds so skipping is a bit smoother.

 

34 minutes ago, markrudling said:

I had wanted to implemented a smart read-ahead cache process that would monitor the last epp watched and pre-fetch the next epp and store it in local, but now this makes managing that file thereafter so much easier. I can pre"read" the next file and it will then be stored in the cache and will be read from there as required, or removed if not consumed in x days.

 

I think this might leave you with a lot of unnecessary transfer as you only need the first say 5-10 seconds of each file to get virtually instant playback.  I think using --vfs-read-ahead will give you a better result.

 

Looking at my caches, my adult TV cache has already hit 500GB in just over 12 hours - although this might be because I'm building a new test Plex server which is currently scanning that folder.

root@Highlander:/mnt/cache/downloads/rclone# du -h --max-depth=1 | sort -hr
756G    .
499G    ./tdrive_t_adults_vfs
219G    ./tdrive_vfs
39G     ./tdrive_uhd_vfs
0       ./registry_vfs
0       ./gdrive_media_vfs

I've bumped up tdrive_t_adults_vfs to 1TB anyway to see what happens and I'm going to break my kids tv out of tdrive_vfs into tdrive_t_kids_vfs, so I can see how many of their shows I can cache e.g. my son tends to watch the same bits of Adventures of Gumball over and over again.

Link to comment

`--vfs-read-ahead` defaults to zero, no read ahead so if you have not specified a value it will still be 0 I think, or false or off. I am not sure, that is the one thing I have not messed with. 

As far as improving seek I think whatever you have in `vfs-read-ahead` will be the maximum it will help by, if you want to FFWD to a point 600mb ahead of where you are and the read ahead value is 400mb then I think it will make no difference, but it might also make a difference of 400mb depending on other settings.

I think the best thing for me to do is create two different teamdrive mounts with different settings and compare that way. I will try to set up a small VPS, biggest problem though may be that my home internet connection is not that fast and connections from most remote servers are poor to Australia.

Link to comment

@Spladge my ff seek experience with the new settings is so good, I can't really see what improvement vfs-read--ahead could make.  It feels like local playback as the restart times are so fast - in normal usage I wouldn't be able to tell the difference as I wouldn't be monitoring so closely.

 

I think I'm just going to leave it all alone so as not to rock the boat.

Link to comment

I think the value --vfs-read-ahead will vary largely based on your use case as well as hardware and internet connection.

 

In my testing, it does not add any extra load time to the start time of videos. Im in SA and it takes between 2-5 seconds to start most videos. Faster than spinning up a sleeping local drive. 

 

I have a 1 gig internet connection, but single thread downloads from google drive are limited to about 150-220 megabits. So even if you set the read ahead value very high, it will still take a few min to complete the file and have it in cache. Most users will skip ahead during the first few moments of loading a new video. How many times will you skip when 15 min into your content?

 

If download speeds from google maxed out my line, I would be inclined to pre-read 10-15gig and have 99% of my content pre downloaded. At full speed it would not take long and the forward skipping buffer issue is mitigated.

 

I will leave it my current setting (1gig) because the cache is going to manage its self fairly well, and I have fast enough internet to support many streams at the same time. HDD speed will become an issue if there are a lot of reads and writes to the single cache disk though, so perhaps turning it off completely is best(again, depends on your connection).

 

I think the cache as it is though, is fantastic for the more likely use case. Most users will skip back a few seconds, when they miss something. Cache here is fantastic and the backward skips are instant. Im very happy with the setup at the moment!

Edited by markrudling
Link to comment
1 hour ago, markrudling said:

I will leave it my current setting (1gig) because the cache is going to manage its self fairly well, and I have fast enough internet to support many streams at the same time. HDD speed will become an issue if there are a lot of reads and writes to the single cache disk though, so perhaps turning it off completely is best(again, depends on your connection).

I think I'm going to try raising my --buffer-size to 256MB or 512MB, rather than using --vfs-read-ahead as I'm nervous about multiple disk writes wiping out any potential benefit.

Link to comment
22 minutes ago, DZMM said:

I think I'm going to try raising my --buffer-size to 256MB or 512MB, rather than using --vfs-read-ahead as I'm nervous about multiple disk writes wiping out any potential benefit.

--Buffer-size will be how much of the file is stored in ram?

The cache will still write whatever was buffered from memory onto disk?

 

Agree though with read ahead removing items from cache prematurely, unless you have the space for large cache. 

 

I think where Im landing is that cache is great for rewinds. Bonus for content that is watched more than once. Buffer will help a bit with forward skips, but only a few seconds. 2-5 seconds to wait for a long skip is more than acceptable. :)

Link to comment
2 hours ago, markrudling said:

The cache will still write whatever was buffered from memory onto disk?

That's the bit I'm not sure about - if it's kept once playback stops.

 

Update:

 

Quote

The --buffer-size data gets written to the disk when the stream is closed so it isn't wasted.

--vfs-read-ahead data is written straight to the disk.

 

 

Edited by DZMM
Link to comment

I have been running with no buffer and no read-ahead. Playback is still pretty much instant on all files, including 4k. Skipping backward is a dream, skipping forward is, well, good enough.

 

The read-ahead when using plex is kinda mitigated if its set fairly small, as plex does its own magic between trans-coding and buffering etc.

 

Im quite happy like this. Less disk read/writes, stable, fast. Me like :) 

Link to comment
21 minutes ago, markrudling said:

I have been running with no buffer and no read-ahead. Playback is still pretty much instant on all files, including 4k. Skipping backward is a dream, skipping forward is, well, good enough.

 

The read-ahead when using plex is kinda mitigated if its set fairly small, as plex does its own magic between trans-coding and buffering etc.

 

Im quite happy like this. Less disk read/writes, stable, fast. Me like :) 

I've come to the same conclusion i.e. the defaults are fine and tinkering is only needed if experiencing problems.  

 

I'll try and update the scripts this week - I tried last week, but I had a few problems including a pull from @watchmeexplode5

Link to comment

Hi all,

My unraid logs are being filled with the below error, every second. The one file i can recall changing last was the rclone config credential. All files in my mergerfs mount are viewable but error when trying to open.
 

Quote

Sep 15 22:37:40 HTPC-unRAID emhttpd: error: share_luks_status, 6256: Operation not supported (95): getxattr: /mnt/user/mount_mergerfs

 

Edited by KeyBoardDabbler
Link to comment
2 hours ago, jonathanm said:

Are you sure you have the correct drive selected on the left side of the google page? Mine has separate entries on the left for the main drive and shared drives.

Yes but my folders are crypted so i don't know what file i want to delete that why i was wondering if krusder works. because i'm able to delete a file in krusader but i'm not sure if it is really deleted on the drive account

 

thx

Link to comment
11 hours ago, francrouge said:

Yes but my folders are crypted so i don't know what file i want to delete that why i was wondering if krusder works. because i'm able to delete a file in krusader but i'm not sure if it is really deleted on the drive account

 

thx

if i delete a file from my mount_mergerfs folder, i see the change in my google drive within afew seconds. 

Check the folder activity on the right side.

Also you can "rename" a file locally, then see which encrypted name changes in your drive. Then rename it back or delete knowing you have identified the correct file/folder

Edited by KeyBoardDabbler
  • Like 1
Link to comment
6 hours ago, KeyBoardDabbler said:

if i delete a file from my mount_mergerfs folder, i see the change in my google drive within afew seconds. 

Check the folder activity on the right side.

Also you can "rename" a file locally, then see which encrypted name changes in your drive. Then rename it back or delete knowing you have identified the correct file/folder

thx for the info

Link to comment

Is anybody using the built in rclone union for their GDrive and Local mounts with vfs caching? I seem to be struggling with it in a strange way. 

 

I have three rclone remotes:

 

1. Gsuite:

2. Crypt: (this wraps Gsuite:)

3. Union: (this wraps crypt:media and /mnt/disks/media/) (this is also what I am caching w/ vfs shown below)

 

My issue seems to be when I go to issue 

rclone move /mnt/user/media/movies crypt:media/movies --delete-src-dirs

that it moves successfully but then Plex cannot play the file. It can see it just fine but I get an input/output error unless I issue a vfs/refresh rclone rc command. Should I be caching crypt before unionizing it?

rclone mount \ 
       --allow-other \
       --dir-cache-time 720h \
       --poll-interval 15s \
       --buffer-size 256M \
       --cache-dir=/mnt/disk3/system/rclone/cache \
       --vfs-cache-mode writes \
       --vfs-cache-max-size 100G \
       --vfs-cache-max-age 168h \
       --vfs-read-chunk-size 128M \
       --vfs-read-chunk-size-limit off \
       --rc \
       --rc-addr 192.168.1.200:5572 \
       --syslog \
       union: /mnt/disks/media &

 

 

Edited by MowMdown
Link to comment
4 hours ago, MowMdown said:

Is anybody using the built in rclone union for their GDrive and Local mounts with vfs caching? I seem to be struggling with it in a strange way. 

Nope - I was having different problems (I can't remember what right now) so I'm sticking with mergerfs.

Link to comment

@DZMM

 

I am having the same issue I think @HonkyKONG22  was having.

 

/local/gdrive_media_vfs

                        /gdrive_media_vfs

                       /tv

/mount_mergerfs/gdrive_media_vfs

                       /gdrive_media_vfs
                       /tv

/mount_rclone/gdrive_media_vfs

                       /gdrive_media_vfs
                       /tv

 

Any help would be appreciated. I can't figure out why I have the extra gdrive_media_vfs folder.

log.txt

rclone_config.txt

mount script.txt

Edited by lzrdking71
Link to comment
7 hours ago, lzrdking71 said:

@DZMM

 

I am having the same issue I think @HonkyKONG22  was having.

 

/local/gdrive_media_vfs

                        /gdrive_media_vfs

                       /tv

/mount_mergerfs/gdrive_media_vfs

                       /gdrive_media_vfs
                       /tv

/mount_rclone/gdrive_media_vfs

                       /gdrive_media_vfs
                       /tv

 

Any help would be appreciated. I can't figure out why I have the extra gdrive_media_vfs folder.

log.txt 2.54 kB · 0 downloads

rclone_config.txt 378 B · 0 downloads

mount script.txt 10.27 kB · 0 downloads

I think the script doesn't like it it when you have only one folder specified in:

 

MountFolders=\{"tv"\}

Try adding a second folder e.g movies

 

MountFolders=\{"movies,tv"\}

 

Link to comment
1 hour ago, DZMM said:

I think the script doesn't like it it when you have only one folder specified in:

 


MountFolders=\{"tv"\}

Try adding a second folder e.g movies

 


MountFolders=\{"movies,tv"\}

 

I went through and did:

 

fusermount -uz /mnt/user/mount_rclone/gdrive_media_vfs
fusermount -uz /mnt/user/mount_mergerfs/gdrive_media_vfs

looked at the folders and removed the remaining gdrive_media_vfs folder from /local, /mount_mergerfs, and /mount_rclone

made the script modification you suggested and added MountFolders=\{"movies,tv"}

re- ran the script and now I again have the extra /gdrive_media_vfs in all of the folders I listed I removed it from above + the added movies folder

Edited by lzrdking71
Link to comment
58 minutes ago, DZMM said:

The extra folder could be because after being created the upload script added them to gdrive.  Delete and they shouldn't come back if my theory is correct.

I have not run the upload script? I deleted the extra gdrive_media_vfs folder from the /local location and it disappeared from the other two locations and did not re-appear after running the mount script again. Is this how it should be? It doesn't look like /mount_rclone/gdrive_media_vfs has the /tv or /movie folders now?

Edited by lzrdking71
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.